VCIO-next: Design new Improver

[pombredanne]

Context

Improvers are scripts that take a bunch of objects as an input and can selectively improve the data about packages and vulnerabilities. The current design is that Importers create Advisories and then a "Default Improver" does an actual proper Import of the data through an intermediate "Inference" object.

Other improvers have the same overall model:

• An Improver subclass implements an "interesting_advisories" method
• And then implements a "get_inferences" method that takes AvisoryData (a plain object not a model instance) and yields Inferences

The management command will control all this.

Problem

With the current design, we cannot do several things:

• We cannot improve if there are no advisories. For instance we cannot loop through some Package or Vulnerabilities
• We cannot do arbitrary operations on the data: we must go through an Inference
• We cannot control the frequency/periodicity at which we run improvers

Because of all this, there is also confusion in the community about how to add new improvers.

Solution

We should simplify improvers to the max.

An Improver should be a class that has a label, description and some run frequency and should have just an improve() method that can do ANYTHING it likes.

The default improver should be retired and its code merged in the Importer loop instead.

See also these related issues:

• Improve Improvers #701
• Package managers may not contain/report all versions related to a package #1144
• Add improver to validate that collected purl are real. #644
• Use the SCIO Pipeline Mechanism for Importers/Improvers #1509

[pombredanne]

@TG1999 @Hritik14 @ziadhany @keshav-space @DennisClark @johnmhoran FYI

[pombredanne]

Should we wait for the model refactoring?

• VCIO-next: Design new Advisory -> Package -> Vulnerability models relationhips #1393

I think not. In all cases we will have to amend improvers to work with the new models. We have many improvement work blocked by this current issue #1395 ... we may end up doing a bit more work if we do not wait, but this is marginal IMHO compared to waiting

[armijnhemel]

Another solution is to create "pseudo-advisories", for example, one per purl.

[pombredanne]

@keshav-space is this what you are working on with the pipelines?

[keshav-space]

@keshav-space is this what you are working on with the pipelines?

@pombredanne yes!

[pombredanne]

The design has been completed: we are now using the aboutcode.pipelines library extracted from scancode.io for both importers and improvers. Migrations is in progress and tracked in:

• Use the SCIO Pipeline Mechanism for Importers/Improvers #1509