Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(npm): Remove unused parallelization constructs #9316

Merged
merged 2 commits into from
Oct 23, 2024
Merged

refactor(npm): Remove unused parallelization constructs #9316

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0f941ae
refactor(npm): Remove unused parallelization constructs
fviernau Oct 22, 2024
697903b
refactor(npm): Drop a slightly misleading log output
fviernau Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 7 additions & 24 deletions plugins/package-managers/node/src/main/kotlin/Npm.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,11 +24,6 @@ package org.ossreviewtoolkit.plugins.packagemanagers.node
import java.io.File
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit message:
Nit: "...strictly sequentially."
So, if I understand correctly, the whole fiddling with async had no real effect, since only a single package was processed, and the parsePackage function is only called sequentially? Maybe make this a bit more explicit in the commit message?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since only a single package was processed

hm, I made the observation when running the fun tests, so not a single package but multiple package were involved. (A single package cannot be processed in parallel, so I guess this is a mis-understanding)

and the parsePackage function is only called sequentially

I in fact verified that getRemotePackageDetails() executes only sequentally. But, parsePackage() is the only caller of that one, so I guess the same applied to parsePackage().

What change to the commit message do you suggest?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From what I understand, there are multiple aspects:

  • The withContext construct in getRemotePackageDetailsAsync is only exited after all coroutines started within it are finished. Therefore, the async invocations are useless, since the function waits until the asynchronous processing is complete.
  • But even on a higher level, no parallelization is applied; parsePackage is invoked sequentially for each package and waits for the result. So, even if asynchronous processing worked, it would not help here.

Do you agree with this?

Regarding the commit message, it is probably not necessary to describe this in detail. So, your text is probably fine.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you agree with this?

Yes, makes sense to me.

import java.util.concurrent.ConcurrentHashMap

import kotlinx.coroutines.Deferred
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.async
import kotlinx.coroutines.withContext

import org.apache.logging.log4j.kotlin.logger

import org.ossreviewtoolkit.analyzer.AbstractPackageManagerFactory
Expand Down Expand Up @@ -120,7 +115,7 @@ open class Npm(

private val graphBuilder by lazy { DependencyGraphBuilder(NpmDependencyHandler(this)) }

private val npmViewCache = ConcurrentHashMap<String, Deferred<PackageJson>>()
private val npmViewCache = mutableMapOf<String, PackageJson>()

protected open fun hasLockfile(projectDir: File) = NodePackageManager.NPM.hasLockfile(projectDir)

Expand Down Expand Up @@ -249,11 +244,7 @@ open class Npm(
* Construct a [Package] by parsing its _package.json_ file and - if applicable - querying additional
* content via the `npm view` command. The result is a [Pair] with the raw identifier and the new package.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit message nit: "...would have searched..."

*/
internal suspend fun parsePackage(workingDir: File, packageJsonFile: File): Package {
val packageDir = packageJsonFile.parentFile

logger.debug { "Found a 'package.json' file in '$packageDir'." }

internal fun parsePackage(workingDir: File, packageJsonFile: File): Package {
val packageJson = parsePackageJson(packageJsonFile)

// The "name" and "version" fields are only required if the package is going to be published, otherwise they are
Expand Down Expand Up @@ -289,7 +280,7 @@ open class Npm(

if (hasIncompleteData) {
runCatching {
getRemotePackageDetailsAsync(workingDir, "$rawName@$version").await()
getRemotePackageDetails(workingDir, "$rawName@$version")
}.onSuccess { details ->
if (description.isEmpty()) description = details.description.orEmpty()
if (homepageUrl.isEmpty()) homepageUrl = details.homepage.orEmpty()
Expand Down Expand Up @@ -342,20 +333,12 @@ open class Npm(
return module
}

private suspend fun getRemotePackageDetailsAsync(workingDir: File, packageName: String): Deferred<PackageJson> =
withContext(Dispatchers.IO) {
npmViewCache.getOrPut(packageName) {
async {
getRemotePackageDetails(workingDir, packageName)
}
}
protected open fun getRemotePackageDetails(workingDir: File, packageName: String): PackageJson =
npmViewCache.getOrPut(packageName) {
val process = run(workingDir, "info", "--json", packageName)
parsePackageJson(process.stdout)
}

protected open fun getRemotePackageDetails(workingDir: File, packageName: String): PackageJson {
val process = run(workingDir, "info", "--json", packageName)
return parsePackageJson(process.stdout)
}

/** Cache for submodules identified by its moduleDir absolutePath */
private val submodulesCache = ConcurrentHashMap<String, Set<File>>()

Expand Down
5 changes: 1 addition & 4 deletions plugins/package-managers/node/src/main/kotlin/utils/NpmDependencyHandler.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ import org.ossreviewtoolkit.model.PackageLinkage
import org.ossreviewtoolkit.model.Project
import org.ossreviewtoolkit.model.utils.DependencyHandler
import org.ossreviewtoolkit.plugins.packagemanagers.node.Npm
import org.ossreviewtoolkit.utils.ort.runBlocking

/**
* A data class storing information about a specific NPM module and its dependencies.
Expand Down Expand Up @@ -76,7 +75,5 @@ internal class NpmDependencyHandler(private val npm: Npm) : DependencyHandler<Np
PackageLinkage.DYNAMIC.takeUnless { dependency.isProject } ?: PackageLinkage.PROJECT_DYNAMIC

override fun createPackage(dependency: NpmModuleInfo, issues: MutableCollection<Issue>): Package? =
runBlocking {
npm.takeUnless { dependency.isProject }?.parsePackage(dependency.workingDir, dependency.packageFile)
}
npm.takeUnless { dependency.isProject }?.parsePackage(dependency.workingDir, dependency.packageFile)
}
Loading