Migrate Vulnerability models to Advisory models

Validate and deploy advisories dedupe

Use same models for staging and production
Take backup of production, copy to staging and restore on staging
Deploy advisory dedupe, merged with Fast content ID migration #1795
Run improver only to dedupe advisories
Review that everything is okay and see if advisories are deduped (reduced). We had 119 million advisories earlier now we have 18 million advisories after running the dedupe pipeline
And deploy on production

Add advisory ID

Add advisory ID field to Advisory model, create schema migration
Move url field position just below the advisory_id field.
Add improver pipeline to populate advisory ID, each advisory created_by different importers implies a different treatment to determine the advisory ID in one of the aliases, the URL or the references.
Update all importers and improvers to account for the new advisory ID field. (import_runner and improve_runner as well)
Test improver on staging and deploy on production

Add other fields ...

Aliases: Create a new model for AdvisoryAlias, we migrate aliases from advisory models to the new models with improver. Ignoring the alias that are part of advisory ID. VCIO-next: Migrate Advisory aliases from JSON field to M2M relationship #1777
Affected Packages: Create a relationship between a package and advisory and migrate
References: Create AdvisoryReferences, and migrate
Severities: Severities needs to be refactored. Create new advisory severities. So they do not go through references. WIll be like VulnerabilitySeverity but will be directly associated with an advisory
Weakness: Create AdvisoryWeakness, and migrate.

Design how to relate to a vulnerability

Update API (v2) and UI.

Remove old models, old fields and old data.

QnA

How to decide advisory ID when all importers share exact same aliases. for example 2 importers only have alias: CVE-XXXX-YYYY, then what should be the heuristic?
Ans: Advisory ID will not be a unique field, but will be part of a unique together: (url, advisory_id, created_by etc...)
Complete the migration and API on the basis of data models.

We are planning to use these respective aliases for these importers for advisory IDs for the advisories generated by these importers

Apache HTTPD - CVE
Apache Kafka - CVE
Apache Tomcat - CVE
Arch Linux - CVE
Curl - CVE
Github - GHSA -> CVE
Github OSV - GHSA -> CVE
Gitlab - GMS -> GHSA -> CVE
Debian - CVE
Elixir - CVE
EPSS - CVE
Fireye - MNDT, CVE
Gentoo - CVE
Istio - CVE
Mozilla - CVE
OpenSSL - VC-OPENSSL, CVE
Postgresql - CVE
Xen - CVE
Ubuntu USN - CVE
Suse - CVE
Ruby - CVE
OSV - CVE -> GHSA -> PYSEC
Redhat - CVE
NVD - CVE
NPM - CVE

Fast content ID migration (#1795)

#1796 (comment) looks good

Add V2Advisory Model.
V2Advisory Model should have relationships between other models like aliases, affected packages, references, severities and weaknesses.
V2AdvisoyModel will have advisory ID. advisory ID will be a natural ID for example Redhat importer will have RHSA, NVD will have CVE, when there is no ID we will create one. For example openSSL
Refactor all importers to change according to the latest advisory model.

Make consistent changes and in smaller steps.

I was adding the models for AdvisoryDataV2, I have a question about what shall be our approach for storing AffectedPackages in our new models. Historically, we used to store affected packages' data like this in our Advisory.

package: PackageURL
affected_version_range: Optional[VersionRange] = None
fixed_version: Optional[Version] = None

Then we unfurl those version ranges using Improver and store them in these models.

class Package(PackageURLMixin):
    """
    A software package with related vulnerabilities.
    """

    # Remove the `qualifers` and `set_package_url` overrides after
    # https://github.com/package-url/packageurl-python/pull/35
    # https://github.com/package-url/packageurl-python/pull/67
    # gets merged

    affected_by_vulnerabilities = models.ManyToManyField(
        to="Vulnerability",
        through="AffectedByPackageRelatedVulnerability",
    )

    fixing_vulnerabilities = models.ManyToManyField(
        to="Vulnerability",
        through="FixingPackageRelatedVulnerability",
        related_name="fixed_by_packages",  # Unique related_name
    )

    package_url = models.CharField(
        max_length=1000,
        null=False,
        help_text="The Package URL for this package.",
        db_index=True,
    )

class FixingPackageRelatedVulnerability(PackageRelatedVulnerabilityBase):
    class Meta(PackageRelatedVulnerabilityBase.Meta):
        verbose_name_plural = "Fixing Package Related Vulnerabilities"


class AffectedByPackageRelatedVulnerability(PackageRelatedVulnerabilityBase):

    severities = models.ManyToManyField(
        VulnerabilitySeverity,
        related_name="affected_package_vulnerability_relations",
    )

    objects = BaseQuerySet.as_manager()

    class Meta(PackageRelatedVulnerabilityBase.Meta):
        verbose_name_plural = "Affected By Package Related Vulnerabilities"

class PackageRelatedVulnerabilityBase(models.Model):
    """
    Abstract base class for package-vulnerability relations.
    """

    package = models.ForeignKey(
        Package,
        on_delete=models.CASCADE,
        db_index=True,
        # related_name="%(class)s_set",  # Unique related_name per subclass
    )

    vulnerability = models.ForeignKey(
        Vulnerability,
        on_delete=models.CASCADE,
        db_index=True,
        # related_name="%(class)s_set",  # Unique related_name per subclass
    )

    created_by = models.CharField(
        max_length=100,
        blank=True,
        help_text=(
            "Fully qualified name of the improver prefixed with the module name "
            "responsible for creating this relation. Eg: vulnerabilities.importers.nginx.NginxBasicImprover"
        ),
    )

I was thinking to use the same thing in our new models, but now having some reservations that if we are going with an advisory only model, will this be our best approach since if we have a version range and not concrete versions how will someone query advisories? So what do you guys think ?

The style and type relationships we have between Vulnerability and Package should be carried forward to new relationship between Advisory V2 and Package

@TG1999

@TG1999 what about something like this as a base:

class PackageV2(PackageURLMixin):
    """
    A software package with related advisories.
    """

    # Remove the `qualifers` and `set_package_url` overrides after
    # https://github.com/package-url/packageurl-python/pull/35
    # https://github.com/package-url/packageurl-python/pull/67
    # gets merged

    affected_by_advisories = models.ManyToManyField(
        to="AdvisoryV2",
        through="AffectedByPackageRelatedAdvisory",
    )

    fixing_advisories = models.ManyToManyField(
        to="AdvisoryV2",
        through="FixingPackageRelatedAdvisory",
        related_name="fixed_by_packages",  # Unique related_name
    )

    package_url = models.CharField(
        max_length=1000,
        null=False,
        help_text="The Package URL for this package.",
        db_index=True,
    )

With this approach, the affected packages old JSON field would be replaced by these relationships:

    affected_packages = models.JSONField(
        blank=True, default=list, help_text="A list of serializable AffectedPackage objects"
    )

There is another thing: we cannot migrate or maintain something clean with old pipelines.

Before a pipeline would: 1. insert advisories, 2. from these inserted data, import/create vulnerabilities and relationships.

But now, this will be only a single step: 1. import/create advisories, packages and relationships.

Therefore IMHO:

we do not need to have intermediate dataclass data structures.
we should do at once the advisory creation, and its relationship including packages

And also: old importers cannot be reused as is, we need a v2 for each.
We should add new pipelines for each importer in existing scripts and try to reuse code as much as possible between old and v2.

In all cases we will only switch to v2 when we have migrated importers and added API and UI, so this is all internal for now.

Here is the schedule and list of importers/pipelines we are planning to migrate:

This is done now!

PRs for references:
#1866

To test this:
Set up Vulnerablecode on your system locally and run a new V2 importer pipeline for example:

./manage.py import gitlab_importer_v2

Then do ./manage.py shell to test newly introduced models. AdvisoryV2, PackageV2

AdvisoryV2.objects.all()
PackageV2.objects.all()

Navigation Menu

Repository navigation

Migrate Vulnerability models to Advisory models #1796

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Migrate Vulnerability models to Advisory models #1796

Description

Activity

TG1999 commented on Mar 6, 2025

pombredanne commented on Mar 18, 2025

TG1999 commented on Apr 17, 2025

TG1999 commented on Apr 25, 2025

pombredanne commented on Apr 25, 2025

pombredanne commented on Apr 25, 2025

pombredanne commented on Apr 25, 2025

TG1999 commented on May 21, 2025

TG1999 commented on Jul 11, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions