Ensure we get the non-devel module from OVAL data #859

javihernandez · 2024-06-10T11:14:21Z

If the first match in the affected regex search is the devel module, we fail when trying to map errata packages later.

If the first match in the affected regex search is the devel module, we fail when trying to map errata packages later. Fixes: AlmaLinux/build-system#304

github-actions · 2024-06-10T11:16:35Z

Code Coverage Summary

Package	Line Rate
alws	76%
alws.auth	77%
alws.auth.oauth	100%
alws.crud	40%
alws.dramatiq	60%
alws.middlewares	93%
alws.perms	86%
alws.routers	57%
alws.schemas	80%
alws.utils	41%
Summary	56% (5650 / 10003)

Linter reports

Pylint report

************* Module alws.crud.errata
alws/crud/errata.py:115:0: C0301: Line too long (86/80) (line-too-long)
alws/crud/errata.py:116:0: C0301: Line too long (88/80) (line-too-long)
alws/crud/errata.py:146:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:183:0: C0301: Line too long (85/80) (line-too-long)
alws/crud/errata.py:232:0: C0301: Line too long (87/80) (line-too-long)
alws/crud/errata.py:326:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:340:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:350:0: C0301: Line too long (88/80) (line-too-long)
alws/crud/errata.py:371:0: C0301: Line too long (83/80) (line-too-long)
alws/crud/errata.py:373:0: C0301: Line too long (84/80) (line-too-long)
alws/crud/errata.py:474:0: C0301: Line too long (88/80) (line-too-long)
alws/crud/errata.py:601:0: C0301: Line too long (83/80) (line-too-long)
alws/crud/errata.py:644:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:676:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:816:0: C0301: Line too long (85/80) (line-too-long)
alws/crud/errata.py:835:0: C0301: Line too long (82/80) (line-too-long)
alws/crud/errata.py:848:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:850:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:856:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:984:0: C0301: Line too long (86/80) (line-too-long)
alws/crud/errata.py:987:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:1047:0: C0301: Line too long (87/80) (line-too-long)
alws/crud/errata.py:1061:0: C0301: Line too long (84/80) (line-too-long)
alws/crud/errata.py:1098:0: C0301: Line too long (83/80) (line-too-long)
alws/crud/errata.py:1349:0: C0301: Line too long (81/80) (line-too-long)
alws/crud/errata.py:1436:0: C0301: Line too long (84/80) (line-too-long)
alws/crud/errata.py:1:0: C0302: Too many lines in module (1665/1000) (too-many-lines)
alws/crud/errata.py:70:5: W0511: FIXME: ovallib dependency should stay optional (fixme)
alws/crud/errata.py:217:29: W0511: TODO: Add test mapping here (fixme)
alws/crud/errata.py:1287:1: W0511: TODO: Check db_record (fixme)
alws/crud/errata.py:625:29: W1401: Anomalous backslash in string: '\d'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
alws/crud/errata.py:625:31: W1401: Anomalous backslash in string: '\w'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
alws/crud/errata.py:625:33: W1401: Anomalous backslash in string: '\-'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
alws/crud/errata.py:625:35: W1401: Anomalous backslash in string: '\_'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
alws/crud/errata.py:625:41: W1401: Anomalous backslash in string: '\d'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
alws/crud/errata.py:625:43: W1401: Anomalous backslash in string: '\.'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
alws/crud/errata.py:625:45: W1401: Anomalous backslash in string: '\w'. String constant might be missing an r prefix. (anomalous-backslash-in-string)
alws/crud/errata.py:96:0: R0903: Too few public methods (1/2) (too-few-public-methods)
alws/crud/errata.py:153:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:153:0: R0914: Too many local variables (34/15) (too-many-locals)
alws/crud/errata.py:171:4: R1702: Too many nested blocks (7/5) (too-many-nested-blocks)
alws/crud/errata.py:153:0: R0912: Too many branches (46/12) (too-many-branches)
alws/crud/errata.py:153:0: R0915: Too many statements (118/50) (too-many-statements)
alws/crud/errata.py:402:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:402:0: R0912: Too many branches (13/12) (too-many-branches)
alws/crud/errata.py:485:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:485:0: R0914: Too many local variables (25/15) (too-many-locals)
alws/crud/errata.py:605:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:605:0: R0914: Too many local variables (27/15) (too-many-locals)
alws/crud/errata.py:768:11: W0718: Catching too general exception Exception (broad-exception-caught)
alws/crud/errata.py:605:0: R0915: Too many statements (51/50) (too-many-statements)
alws/crud/errata.py:802:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:830:23: E1102: func.count is not callable (not-callable)
alws/crud/errata.py:899:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:899:0: R0914: Too many local variables (25/15) (too-many-locals)
alws/crud/errata.py:1028:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:1093:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:1155:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:1155:0: R0914: Too many local variables (19/15) (too-many-locals)
alws/crud/errata.py:1208:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:1338:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:1368:15: W0718: Catching too general exception Exception (broad-exception-caught)
alws/crud/errata.py:1396:19: W0718: Catching too general exception Exception (broad-exception-caught)
alws/crud/errata.py:1404:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:1436:0: W1404: Implicit string concatenation found in call (implicit-str-concat)
alws/crud/errata.py:1452:19: W0718: Catching too general exception Exception (broad-exception-caught)
alws/crud/errata.py:1478:23: W0718: Catching too general exception Exception (broad-exception-caught)
alws/crud/errata.py:1496:0: C0116: Missing function or method docstring (missing-function-docstring)
alws/crud/errata.py:1496:0: R0914: Too many local variables (19/15) (too-many-locals)
alws/crud/errata.py:1535:16: E1205: Too many arguments for logging format string (logging-too-many-args)
alws/crud/errata.py:1538:24: E0602: Undefined variable 'exc' (undefined-variable)
alws/crud/errata.py:1610:34: E1101: Module 'createrepo_c' has no 'SHA256' member (no-member)
alws/crud/errata.py:1496:0: R0912: Too many branches (18/12) (too-many-branches)

-----------------------------------
Your code has been rated at 8.72/10

Black report

--- alws/crud/errata.py	2024-06-10 11:14:37.301457+00:00
+++ alws/crud/errata.py	2024-06-10 11:15:33.806719+00:00
@@ -110,12 +110,17 @@
         if (
             self.parent is not None
             and len(self.criteria["criterion"]) == 1
             and len(self.criteria["criteria"]) == 0
         ):
-            self.parent.criteria["criterion"].append(self.criteria["criterion"].pop())
-        if len(self.criteria["criteria"]) == 0 and len(self.criteria["criterion"]) == 0:
+            self.parent.criteria["criterion"].append(
+                self.criteria["criterion"].pop()
+            )
+        if (
+            len(self.criteria["criteria"]) == 0
+            and len(self.criteria["criterion"]) == 0
+        ):
             return True
         return False
 
 
 async def get_oval_xml(
@@ -141,11 +146,12 @@
 
     query = query.filter(models.NewErrataRecord.platform_id == platform.id)
 
     if only_released:
         query = query.filter(
-            models.NewErrataRecord.release_status == ErrataReleaseStatus.RELEASED
+            models.NewErrataRecord.release_status
+            == ErrataReleaseStatus.RELEASED
         )
 
     records = (await db.execute(query)).scalars().all()
     return errata_records_to_oval(records, platform_name)
 
@@ -178,11 +184,13 @@
                 for albs_pkg in pkg.albs_packages
                 if albs_pkg.status == ErrataPackageStatus.released
             ]
             for albs_pkg in albs_pkgs:
                 rhel_evra = f"{pkg.epoch}:{pkg.version}-{pkg.release}"
-                albs_evra = f"{albs_pkg.epoch}:{albs_pkg.version}-{albs_pkg.release}"
+                albs_evra = (
+                    f"{albs_pkg.epoch}:{albs_pkg.version}-{albs_pkg.release}"
+                )
                 arch = albs_pkg.arch
                 if arch == "noarch":
                     arch = pkg.arch
                 rhel_evra_mapping[rhel_evra][arch] = albs_evra
                 rhel_name_mapping[rhel_evra].add(albs_pkg.name)
@@ -227,11 +235,13 @@
                     r"is signed with AlmaLinux OS",
                     criterion_list[0]["comment"],
                 ):
                     criterion_list = []
                 criteria["criterion"] = criterion_list
-                links_tracking.update(criterion["ref"] for criterion in criterion_list)
+                links_tracking.update(
+                    criterion["ref"] for criterion in criterion_list
+                )
             criteria_list = new_criteria_list
         for criteria in record.original_criteria:
             criteria_node = CriteriaNode(criteria, None)
             criteria_node.simplify()
         if record.oval_title:
@@ -321,11 +331,13 @@
                 )
             test["object_ref"] = debrand_id(test["object_ref"])
             if test.get("state_ref"):
                 test["state_ref"] = debrand_id(test["state_ref"])
             links_tracking.update([test["object_ref"], test["state_ref"]])
-            oval.append_object(get_test_cls_by_tag(test["type"]).from_dict(test))
+            oval.append_object(
+                get_test_cls_by_tag(test["type"]).from_dict(test)
+            )
         for obj in record.original_objects:
             obj["id"] = debrand_id(obj["id"])
             if obj["id"] in objects:
                 continue
             if obj["id"] not in links_tracking:
@@ -335,21 +347,25 @@
                 obj["instance_var_ref"] = debrand_id(obj["instance_var_ref"])
                 links_tracking.add(obj["instance_var_ref"])
             if get_object_cls_by_tag(obj["type"]) == RpmverifyfileObject:
                 if obj["filepath"] == "/etc/redhat-release":
                     obj["filepath"] = "/etc/almalinux-release"
-            oval.append_object(get_object_cls_by_tag(obj["type"]).from_dict(obj))
+            oval.append_object(
+                get_object_cls_by_tag(obj["type"]).from_dict(obj)
+            )
         for state in record.original_states:
             state["id"] = debrand_id(state["id"])
             if state["id"] in objects:
                 continue
             if state["id"] not in links_tracking:
                 continue
             if state.get("evr"):
                 if state["evr"] in rhel_evra_mapping:
                     if state["arch"]:
-                        state["arch"] = "|".join(rhel_evra_mapping[state["evr"]].keys())
+                        state["arch"] = "|".join(
+                            rhel_evra_mapping[state["evr"]].keys()
+                        )
                     state["evr"] = rhel_evra_mapping[state["evr"]][
                         next(iter(rhel_evra_mapping[state["evr"]].keys()))
                     ]
             objects.add(state["id"])
             state_cls = get_state_cls_by_tag(state["type"])
@@ -366,13 +382,18 @@
             if var["id"] in objects:
                 continue
             if var["id"] not in links_tracking:
                 continue
             objects.add(var["id"])
-            oval.append_object(get_variable_cls_by_tag(var["type"]).from_dict(var))
+            oval.append_object(
+                get_variable_cls_by_tag(var["type"]).from_dict(var)
+            )
             for obj in record.original_objects:
-                if obj["id"] != var["arithmetic"]["object_component"]["object_ref"]:
+                if (
+                    obj["id"]
+                    != var["arithmetic"]["object_component"]["object_ref"]
+                ):
                     continue
                 if obj["id"] in objects:
                     continue
                 objects.add(obj["id"])
                 oval.append_object(
@@ -469,11 +490,13 @@
         if update_record.title == record.original_title:
             record.title = None
         else:
             record.title = update_record.title
         if record.title:
-            record.oval_title = get_oval_title(record.title, record.id, record.severity)
+            record.oval_title = get_oval_title(
+                record.title, record.id, record.severity
+            )
     if update_record.description is not None:
         if update_record.description == record.original_description:
             record.description = None
         else:
             record.description = update_record.description
@@ -596,11 +619,14 @@
             nevra = parse_rpm_nevra(pulp_rpm_package.rpm_sourcerpm)
             errata_package.source_srpm = nevra.name
         items_to_insert.append(mapping)
         errata_package.albs_packages.append(mapping)
         errata_record_ids.add(errata_package.errata_record_id)
-    package_type = {"type": ErrataPackagesType.BUILD, "build_ids": list(build_ids)}
+    package_type = {
+        "type": ErrataPackagesType.BUILD,
+        "build_ids": list(build_ids),
+    }
     return items_to_insert, package_type
 
 
 async def create_errata_record(db: AsyncSession, errata: BaseErrataRecord):
     platform = await db.execute(
@@ -639,11 +665,13 @@
         issued_date=errata.issued_date,
         updated_date=errata.updated_date,
         description=None,
         original_description=errata.description,
         title=None,
-        oval_title=get_oval_title(errata.title, alma_errata_id, errata.severity),
+        oval_title=get_oval_title(
+            errata.title, alma_errata_id, errata.severity
+        ),
         original_title=get_verbose_errata_title(errata.title, errata.severity),
         contact_mail=platform.contact_mail,
         status=errata.status,
         version=errata.version,
         severity=errata.severity,
@@ -671,11 +699,13 @@
     self_ref_exists = False
     for ref in errata.references:
         db_cve = None
         if ref.cve:
             db_cve = await db.execute(
-                select(models.ErrataCVE).where(models.ErrataCVE.id == ref.cve.id)
+                select(models.ErrataCVE).where(
+                    models.ErrataCVE.id == ref.cve.id
+                )
             )
             db_cve = db_cve.scalars().first()
             if db_cve is None:
                 db_cve = models.ErrataCVE(
                     id=ref.cve.id,
@@ -811,11 +841,13 @@
     status: Optional[ErrataReleaseStatus] = None,
 ):
     options = []
     if compact:
         options.append(
-            load_only(models.NewErrataRecord.id, models.NewErrataRecord.updated_date)
+            load_only(
+                models.NewErrataRecord.id, models.NewErrataRecord.updated_date
+            )
         )
     else:
         options.extend([
             selectinload(models.NewErrataRecord.packages)
             .selectinload(models.NewErrataPackage.albs_packages)
@@ -830,11 +862,13 @@
         query = select(func.count(models.NewErrataRecord.id))
         if not count:
             query = select(models.NewErrataRecord).options(*options)
             query = query.order_by(models.NewErrataRecord.id.desc())
         if errata_id:
-            query = query.filter(models.NewErrataRecord.id.like(f"%{errata_id}%"))
+            query = query.filter(
+                models.NewErrataRecord.id.like(f"%{errata_id}%")
+            )
         if errata_ids:
             query = query.filter(models.NewErrataRecord.id.in_(errata_ids))
         if title:
             query = query.filter(
                 or_(
@@ -843,19 +877,25 @@
                 )
             )
         if platform:
             query = query.filter(models.NewErrataRecord.platform_id == platform)
         if cve_id:
-            query = query.filter(models.NewErrataRecord.cves.like(f"%{cve_id}%"))
+            query = query.filter(
+                models.NewErrataRecord.cves.like(f"%{cve_id}%")
+            )
         if status:
-            query = query.filter(models.NewErrataRecord.release_status == status)
+            query = query.filter(
+                models.NewErrataRecord.release_status == status
+            )
         if page and not count:
             query = query.slice(10 * page - 10, 10 * page)
         return query
 
     return {
-        "total_records": (await db.execute(generate_query(count=True))).scalar(),
+        "total_records": (
+            await db.execute(generate_query(count=True))
+        ).scalar(),
         "records": (await db.execute(generate_query())).scalars().all(),
         "current_page": page,
     }
 
 
@@ -979,14 +1019,18 @@
             }
     collection_name = (
         f"{platform.name.lower()}-for-{arch}-{repo_stage}-"
         f"rpms__{platform_version}_default"
     )
-    default_summary = clean_errata_title(record.get_title(), severity=record.severity)
+    default_summary = clean_errata_title(
+        record.get_title(), severity=record.severity
+    )
     pulp_record = {
         "id": record.id,
-        "updated_date": datetime.datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S"),
+        "updated_date": datetime.datetime.utcnow().strftime(
+            "%Y-%m-%d %H:%M:%S"
+        ),
         "issued_date": record.issued_date.strftime("%Y-%m-%d %H:%M:%S"),
         "description": record.get_description(),
         "fromstr": record.contact_mail,
         "status": record.status,
         "title": record.get_title(),
@@ -1042,13 +1086,13 @@
                     select(models.BuildTaskArtifact)
                     .where(
                         models.BuildTaskArtifact.href == pkg_href,
                     )
                     .options(
-                        selectinload(models.BuildTaskArtifact.build_task).selectinload(
-                            models.BuildTask.rpm_modules
-                        )
+                        selectinload(
+                            models.BuildTaskArtifact.build_task
+                        ).selectinload(models.BuildTask.rpm_modules)
                     )
                 )
             )
             .scalars()
             .all()
@@ -1056,11 +1100,12 @@
         for db_pkg in db_pkg_list:
             errata_pkgs = await db.execute(
                 select(models.NewErrataToALBSPackage)
                 .where(
                     or_(
-                        models.NewErrataToALBSPackage.albs_artifact_id == db_pkg.id,
+                        models.NewErrataToALBSPackage.albs_artifact_id
+                        == db_pkg.id,
                         models.NewErrataToALBSPackage.pulp_href == pkg_href,
                     )
                 )
                 .options(
                     selectinload(models.NewErrataToALBSPackage.errata_package),
@@ -1093,11 +1138,13 @@
 def append_update_packages_in_update_records(
     pulp_db: Session,
     errata_records: List[Dict[str, Any]],
     updateinfo_mapping: DefaultDict[
         str,
-        List[Tuple[models.BuildTaskArtifact, dict, models.NewErrataToALBSPackage]],
+        List[
+            Tuple[models.BuildTaskArtifact, dict, models.NewErrataToALBSPackage]
+        ],
     ],
 ):
     for record in errata_records:
         record_uuid = uuid.UUID(record["pulp_href"].split("/")[-2])
         packages = updateinfo_mapping.get(record["id"])
@@ -1344,11 +1391,13 @@
     async with open_async_session(key=get_async_db_key()) as session:
         session: AsyncSession
         query = generate_query_for_release([record_id])
         query = query.filter(models.NewErrataRecord.platform_id == platform_id)
         db_record = await session.execute(query)
-        db_record: Optional[models.NewErrataRecord] = db_record.scalars().first()
+        db_record: Optional[models.NewErrataRecord] = (
+            db_record.scalars().first()
+        )
         if not db_record:
             logging.info("Record with %s id doesn't exists", record_id)
             return
 
         logging.info("Record release %s has been started", record_id)
@@ -1431,11 +1480,12 @@
                 records_ids,
             )
             return
 
         logging.info(
-            "Starting bulk errata release, the following records are" " locked: %s",
+            "Starting bulk errata release, the following records are"
+            " locked: %s",
             [rec.id for rec in db_records],
         )
         for db_record in db_records:
             logging.info("Preparing data for %s", db_record.id)
             search_params = prepare_search_params(db_record)

Bandit report

Run started:2024-06-10 11:15:35.350319

Test results:
	No issues identified.

Code scanned:
	Total lines of code: 1541
	Total lines skipped (#nosec): 0
	Total potential issues skipped due to specifically being disabled (e.g., #nosec BXXX): 0

Run metrics:
	Total issues (by severity):
		Undefined: 0
		Low: 0
		Medium: 0
		High: 0
	Total issues (by confidence):
		Undefined: 0
		Low: 0
		Medium: 0
		High: 0
Files skipped (0):

View full reports on the Job Summary page.

If the first match in the affected regex search is the devel module, we fail when trying to map errata packages later. Fixes: AlmaLinux/build-system#304

* origin/master: Fix MissingGreenlet in getting new test tasks Bump fastapi-sqla from 3.1.2 to 3.3.0 Bump pylint from 3.2.2 to 3.2.3 Bump pytest from 8.2.1 to 8.2.2 Bump pydantic-settings from 2.3.1 to 2.3.2 Disable expire_on_commit in some interactions with pulp db Bump structlog from 24.1.0 to 24.2.0 Bump pydantic-settings from 2.2.1 to 2.3.1 Bump pydantic from 2.7.1 to 2.7.3 Bump sentry-sdk[fastapi] from 2.2.1 to 2.5.1 Ensure we get the non-devel module from OVAL data (AlmaLinux#859) Separated packages list in failed errata release Add platform_id in /errata/all/ endpoint (AlmaLinux/build-system#207) Bump uvicorn from 0.29.0 to 0.30.1 Bump errata2osv from 0.0.3 to 0.0.4 (AlmaLinux#856)

Ensure we get the non-devel module from OVAL data

6ed69fc

If the first match in the affected regex search is the devel module, we fail when trying to map errata packages later. Fixes: AlmaLinux/build-system#304

soksanichenko approved these changes Jun 10, 2024

View reviewed changes

maccelf approved these changes Jun 10, 2024

View reviewed changes

javihernandez merged commit c196774 into AlmaLinux:master Jun 10, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ensure we get the non-devel module from OVAL data #859

Ensure we get the non-devel module from OVAL data #859

javihernandez commented Jun 10, 2024

github-actions bot commented Jun 10, 2024

Ensure we get the non-devel module from OVAL data #859

Ensure we get the non-devel module from OVAL data #859

Conversation

javihernandez commented Jun 10, 2024

github-actions bot commented Jun 10, 2024

Code Coverage Summary

Linter reports