Add download archiving system with LocalFilesystem provider

[VarshaUN]

As per discussed with mentors , I have added the following ,

• archiving.py
  Added DownloadStore abstract base class and implementations (LocalFilesystemProvider, S3LikeProvider, SftpProvider) for storing downloads with SHA256-based deduplication and metadata.

• settings.py
  Initialized download_store based on DOWNLOAD_ARCHIVING_PROVIDER (localstorage, s3, sftp) with configuration validation and error logging.

• input.py
  Added add_input_from_url and add_input_from_upload to archive URL downloads and uploaded files using download_store, with fallback to project input directory when archiving is disabled. Integrate with InputSource model for metadata storage.

Enhances input handling for pipelines, supporting deduplicated storage and retrieval of inputs across local, S3, and SFTP backends.

Still in progress ,

Signed-off-by : Varsha U N [email protected]

[AyanSinhaMahapatra]

@VarshaUN thanks for the PR, you need to address a few overall issues before we can start reviewing the code in more details, see comments below for these:

• there are no tests for the new functionality at all, we don't know if your code works
• the CI tests fail as you've added dependencies but not added them to pyproject.toml
• as the CI tests are not even running we don't have any idea about whether the code changes even pass existing tests
• the PR has no link to previous issues/discussions/PRs (please do not use As per discussed with mentors, link to an issue/discussion)
• please also consider all reviews/comments from add support to store packages/archives locally #1685, I feel many of them are also applicable here.
• please do a basic test run on your local machine/CI as mentioned in https://scancodeio.readthedocs.io/en/latest/installation.html#local-development and https://scancodeio.readthedocs.io/en/latest/installation.html#tests and preferably this passes on your PR (or there are expected failures which you can explain). I've also mentioned this before in add support to store packages/archives locally #1685 (review)

Couple of issues with general direction of the PR as discussed in #1685 (comment):

We could have a simple base class to get/put files in the archive and a local file system implementation for now, enable with a global settings.

• we asked for a working implementation of the features with local filesystem only, then only go for S3 (it's fine to create data/abstract classes) as this is much harder to test. Functionality that cannot be tested, is much likely to not be merged.

CORE: have a feature in the base Pipeline class and settings to enable that

• Your code does not interact with any pipelines at all, and no unittests/overall tests are added, so no way for us to test the functions/features.

The local storage looks like this:

• We need some tests so show this is being stored correctly.

Presently input archives are downloaded with

scancode.io/scanpipe/pipelines/__init__.py

Line 130 in 9d41ad3

def download_missing_inputs(self):

and stored in the /input/ directory for each project as specified in

scancode.io/scanpipe/models.py

Line 573 in 9d41ad3

WORK_DIRECTORIES = ["input", "output", "codebase", "tmp"]

, this needs to move into a central location for each instance of scancode.io and there is a lot of code changes required to do that. This is not addressed yet in the PR.