back2source: Report on many d2d projects at once

[pombredanne]

I would like to report on many projects at once.

Given a bunch of "map_deploy_to_devel" projects that share a tag or name pattern, I would like to create a consolidated report across all the projects that will list the issues and problem that need attention in the d2d runs of these projects.

The exact details are exactly these:

• foremost "TODO requires review" status https://staging.scancode.io/project/test-ruby-devise-b32614ad/resources/?status=requires-review
• possibly later the "Not mapped" "To" files https://staging.scancode.io/project/test-ruby-devise-b32614ad/resources/?tag=to&relation_map_type=none

I would like to:

• Filter projects that have a given pipeline/labels
• Select these projects to run a report Add a TODOs sheet containing on REQUIRES_REVIEW resources in XLSX #1524 #1527

I would like this XLSX report would contain these columns:

• the project name
• the resource columns from a UI list filtered with resources/?status=requires-review

Later on we could refine that to look like this:
https://github.com/aboutcode-org/back2source-data/blob/main/d2d-summary.csv
but generic

[tdruez]

@pombredanne Current progress is ready to be tested:

1. Search a label, such as "back2source-Cpp", in the main project search bar
2. Click the "Select all" checkbox on the top left header column of the table
3. "Action" drop down on the right side, select the "Report" action
4. Select the "TODOs" type and click the "Download XLSX" button

-> be patient, it may take a while to build the report (loading screen will be added to prevent the user from clicking elsewhere). You may want to try on a smaller set a project in case of timeout, we will work on an async report generation next but I'd like feedback on the content of the XLSX first.

[tdruez]

@pombredanne The "Report" feature is now complete and ready for your review.
It is available from the UI, the CLI, and the API.

For the following usage example, we'll assume you have run multiple d2d pipelines using the batch-create command.
All your projects for that batch are tagged with the label "back2source-ruby".

UI

From the project list:

• Put the label "back2source-ruby" in the search box and
• Click on the "select all" checkbox in the table header
• The "Action" dropdown will be activated, click on it and select "Report"
• Select the "TODOs" type and check the "Include all xxx projects" to select all projects for your current search.

A new tab will open and once the report is created, the download will start. Note: It may take a while if too much data is selected to be included in the report.

CLI

Documented at https://scancodeio.readthedocs.io/en/latest/command-line-interface.html#scanpipe-report-model-model

$ scanpipe report --model todo --label back2source-ruby

REST API

Documented at https://scancodeio.readthedocs.io/en/latest/rest-api.html#xlsx-report

GET /api/projects/report/?model=todo&label=back2source-ruby

---

@pombredanne Let me know if that whole feature needs adjustments, otherwise you can close this issue.