Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not fail extraction if inspector cannot find the root project and minor doc update #1396

Merged
merged 2 commits into from
Mar 21, 2025
Merged

Do not fail extraction if inspector cannot find the root project and minor doc update #1396

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c66f087
Do not fail extraction if inspector cannot find the root project and …
andrian-sevastyanov Mar 20, 2025
998ed05
Log info message when no package matching project name can be found
andrian-sevastyanov Mar 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...com/blackduck/integration/detectable/detectables/pip/inspector/PipInspectorExtractor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ public Extraction extract(

for (Path requirementFilePath : requirementsPaths) {
List<String> inspectorOutput = runInspector(directory, pythonExe, pipInspector, projectName, requirementFilePath);
Optional<NameVersionCodeLocation> result = pipInspectorTreeParser.parse(inspectorOutput, directory.toString(), StringUtils.isNotEmpty(projectName));
Optional<NameVersionCodeLocation> result = pipInspectorTreeParser.parse(inspectorOutput, directory.toString());
if (result.isPresent()) {
codeLocations.add(result.get().getCodeLocation());
String potentialProjectVersion = result.get().getProjectVersion();
Expand Down
10 changes: 4 additions & 6 deletions ...kduck/integration/detectable/detectables/pip/inspector/parser/PipInspectorTreeParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ public PipInspectorTreeParser(ExternalIdFactory externalIdFactory) {
this.externalIdFactory = externalIdFactory;
}

public Optional<NameVersionCodeLocation> parse(List<String> pipInspectorOutputAsList, String sourcePath, boolean projectNameWasGiven) {
public Optional<NameVersionCodeLocation> parse(List<String> pipInspectorOutputAsList, String sourcePath) {
NameVersionCodeLocation parseResult = null;

DependencyGraph graph = new BasicDependencyGraph();
Expand All @@ -48,14 +48,16 @@ public Optional<NameVersionCodeLocation> parse(List<String> pipInspectorOutputAs
|| trimmedLine.startsWith(UNKNOWN_REQUIREMENTS_PREFIX)
|| trimmedLine.startsWith(UNPARSEABLE_REQUIREMENTS_PREFIX)
|| trimmedLine.startsWith(UNKNOWN_PACKAGE_PREFIX)
|| trimmedLine.startsWith(UNKNOWN_PROJECT_NAME) && projectNameWasGiven
) {
boolean wasUnresolved = parseErrorsFromLine(trimmedLine);
if (wasUnresolved) {
unResolvedPackageCount++;
}
continue;
}
if (trimmedLine.startsWith(UNKNOWN_PROJECT_NAME)) {
logger.info("Pip inspector did not find a package matching project name");
}
Dependency currentDependency = parseDependencyFromLine(trimmedLine, sourcePath);
adjustForIndentLevel(history, line);
project = addDependencyToGraph(graph, history, project, currentDependency);
Expand Down Expand Up @@ -118,10 +120,6 @@ private boolean parseErrorsFromLine(String trimmedLine) {
unResolvedPackage = true;
}

if (trimmedLine.startsWith(UNKNOWN_PROJECT_NAME)) {
logger.error("Pip inspector could not resolve the project");
unResolvedPackage = true;
}
Copy link
Contributor

@devmehtabd devmehtabd Mar 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a warning message be useful if we are not erroring out? In the message, we can ask users to use detect.pip.project.name property.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition could be encountered even if users pass in --detect.pip.project.name.
UNKNOWN_PROJECT_NAME is when the inspector outputs n? which means that the given project name (wherever it came from) was not found in installed PIP packages.
This is also encountered when there is no project name at all, and that is actually an OK condition for many projects that could be scanned.

So, I am somewhat on the fence about this. Maybe another reviewer can chime in. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ended up adding an info message after all.

return unResolvedPackage;
}

Expand Down
24 changes: 5 additions & 19 deletions ...uck/integration/detectable/detectables/pip/inspector/unit/PipInspectorTreeParserTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ public void validTest() {
" test==4.0.0"
);

Optional<NameVersionCodeLocation> validParse = parser.parse(pipInspectorOutput, "", true);
Optional<NameVersionCodeLocation> validParse = parser.parse(pipInspectorOutput, "");
Assertions.assertTrue(validParse.isPresent());
Assertions.assertEquals("projectName", validParse.get().getProjectName());
Assertions.assertEquals("projectVersionName", validParse.get().getProjectVersion());
Expand All @@ -59,7 +59,7 @@ void projectNotFoundWhenNameNotGivenTest() {
" test==4.0.0"
);

Optional<NameVersionCodeLocation> validParse = parser.parse(pipInspectorOutput, "src/path", false);
Optional<NameVersionCodeLocation> validParse = parser.parse(pipInspectorOutput, "src/path");
Assertions.assertTrue(validParse.isPresent());
Assertions.assertEquals("", validParse.get().getProjectName());
Assertions.assertEquals("", validParse.get().getProjectVersion());
Expand All @@ -73,20 +73,6 @@ void projectNotFoundWhenNameNotGivenTest() {
graphAssert.hasRootSize(3);
}

@Test
void projectNotFoundWhenNameGivenTest() {
List<String> pipInspectorOutput = Arrays.asList(
"n?==v?",
" with-dashes==1.0.0",
" Uppercase==2.0.0",
" child==3.0.0",
" test==4.0.0"
);

Optional<NameVersionCodeLocation> invalidParse = parser.parse(pipInspectorOutput, "src/path", true);
Assertions.assertFalse(invalidParse.isPresent());
}

@Test
public void packageNotFoundTest() {
List<String> pipInspectorOutput = Arrays.asList(
Expand All @@ -98,7 +84,7 @@ public void packageNotFoundTest() {
" test==4.0.0"
);

Optional<NameVersionCodeLocation> result = parser.parse(pipInspectorOutput, "src/path", true);
Optional<NameVersionCodeLocation> result = parser.parse(pipInspectorOutput, "src/path");
Assertions.assertFalse(result.isPresent());
}

Expand All @@ -107,7 +93,7 @@ public void invalidParseTest() {
List<String> invalidText = new ArrayList<>();
invalidText.add("i am not a valid file");
invalidText.add("the status should be optional.empty()");
Optional<NameVersionCodeLocation> invalidParse = parser.parse(invalidText, "", true);
Optional<NameVersionCodeLocation> invalidParse = parser.parse(invalidText, "");
Assertions.assertFalse(invalidParse.isPresent());
}

Expand All @@ -117,7 +103,7 @@ public void errorTest() {
invalidText.add(PipInspectorTreeParser.UNKNOWN_PACKAGE_PREFIX + "probably_an_internal_dependency_PY");
invalidText.add(PipInspectorTreeParser.UNPARSEABLE_REQUIREMENTS_PREFIX + "/not/a/real/path/encrypted/requirements.txt");
invalidText.add(PipInspectorTreeParser.UNKNOWN_REQUIREMENTS_PREFIX + "/not/a/real/path/requirements.txt");
Optional<NameVersionCodeLocation> invalidParse = parser.parse(invalidText, "", true);
Optional<NameVersionCodeLocation> invalidParse = parser.parse(invalidText, "");
Assertions.assertFalse(invalidParse.isPresent());
}
}
2 changes: 1 addition & 1 deletion documentation/src/main/markdown/packagemgrs/python.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Pip Native Inspector runs the [pip-inspector.py script](https://github.com/black
python setup.py install
pip install -r requirements.txt
````
* Pip detector derives your project name using your setup.py file if you have one. If you do not have a setup.py file, you must provide the correct project name using the propety --detect.pip.project.name.
* Pip detector attempts to derive the project name using your setup.py file if you have one. If you do not have a setup.py file, you can provide the correct project name using the propety `--detect.pip.project.name`.
Copy link
Contributor Author

@andrian-sevastyanov andrian-sevastyanov Mar 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not fully related to the code change, but as I was skimming through the doc I figured that "must" is too strong of a word here. There are many types of projects for which the PIP Inspector will produce complete results even if there is no root project name.

* If there are any dependencies specified in requirements.txt that are not specified in setup.py, then provide the requirements.txt file using the [detect_product_short] property.
<note type="tip">If you are using a virtual environment, be sure to switch to that virtual environment when you run [detect_product_short]. This also applies when you are using a tool such as Poetry that sets up a Python virtual environment.</note>

Expand Down