Some package-versions are missing files but still have modules and callables

[MagielBruntink]

Describe the bug

In the metadata DB, some package-versions do not have their files listed in the files table, but they still have modules and callables. This causes eg. the quality metrics plugin to fail, because it queries first the files table to match module_ids later.

To Reproduce

Example: https://api.fasten-project.eu/api/mvn/packages/org.bouncycastle:bcprov-jdk15on/1.54/files
https://api.fasten-project.eu/api/mvn/packages/org.bouncycastle:bcprov-jdk15on/1.54/callables

Expected behavior

Package-versions should have complete information in the metadata DB.

[proksch]

Which plugin is actually responsible for adding this information? Is it the metadata-db-extension?

[mir-am]

Which plugin is actually responsible for adding this information? Is it the metadata-db-extension?

Yes, it is the job of the metadata DB plugin.

[proksch]

Didn't @ashkboos and @gdrosos introduce a simplification/change there? Could this behavior be caused by that?

edit: Wrong Georgios

[proksch]

Also, when we have to touch this anyways, we could also think about addressing #102 in the same go.

[ashkboos]

I investigated this example and the problem was not metadata-db-extension plugin. The reason for not having any file was that OPAL could not find the source file information from the bytecode. As you can see in another example OPAL found the source file information and metadata-extension also inserted it to DB and rest-api shows the file information: https://api.fasten-project.eu/api/mvn/packages/org.apache.commons:commons-compress/1.9/files

Moreover, this is also not an OPAL bug. I manually investigated 3 random .class files from the package that was mentioned in this issue and none of them had the sourceFile information so we cannot extract a piece of information that is not there. Here you can see an example of a file with source file information (our synthetic jar) and one without it (example in this issue). The command to view the bytecode is also in the screenshots.

[MagielBruntink]

Ok, that clears it up. What I think might be going on is that we previously did not even get these coordinates in the DB at all because of missing info? Now we do and we have a bit more error rate in downstream plugins. Let’s accept that though. Thanks, Mehdi.

[ashkboos]

Ok, that clears it up. What I think might be going on is that we previously did not even get these coordinates in the DB at all because of missing info? Now we do and we have a bit more error rate in downstream plugins. Let’s accept that though. Thanks, Mehdi.

Yeah, that might be. We have way more dependencies now.
Your welcome.