chore: use npm workspaces to manage locking dependencies

[ChristianMurphy]

Checklist

• the contributor license agreement is signed
• commit message follows commit guidelines
• tests are included
• screenshots are included showing significant UI changes
• documentation is changed or added

Description of change

https://docs.npmjs.com/cli/v7/using-npm/workspaces

locking for all packages within the project is now handled by the root
lockfile

resolves #3262 and #3265

[ChristianMurphy]

used --force mode to override warnings about peer dependencies for now

[ChristianMurphy]

this could be changed to

      - name: Run tests
        run: |
          npm t --workspace=oeq-ts-rest-api

if preferred

[PenghaiZhang]

I prefer leave as it is.

[ChristianMurphy]

If this is a direction the team wants to go, code build may need to be updated to run install once at the top level.

[PenghaiZhang]

Thanks @ChristianMurphy !!! I like this idea.

[PenghaiZhang]

Have some errors complaining error Error: Cannot parse file package.json

[ChristianMurphy]

Have some errors complaining error Error: Cannot parse file package.json

Resolved in 8693b4f

Now there seems to be an issue with RunTypes, which looks to be the same one see in #3247

[PenghaiZhang]

Have some errors complaining error Error: Cannot parse file package.json

Resolved in 8693b4f

Now there seems to be an issue with RunTypes, which looks to be the same one see in #3247

Do you mean the Storybook dependency conflicts ?

[ChristianMurphy]

Do you mean the Storybook dependency conflicts ?

I think this is RunType related: https://github.com/openequella/openEQUELLA/pull/3275/checks?check_run_id=3305552177#step:10:72 🤔
It looks a lot like the error from #3247 (https://github.com/openequella/openEQUELLA/pull/3247/checks?check_run_id=3224768826#step:12:70)

[ChristianMurphy]

The error in the storybook build https://github.com/openequella/openEQUELLA/pull/3275/checks?check_run_id=3305552249#step:6:72
looks to be related to #3275 (comment)

openEQUELLA/package.json

Line 20 in 0ca9e2b

"prepare": "husky install"

is still running even though the root install is not run, will have to see if there's a way to skip the script

[edalex-ian]

Thanks for taking this for a spin @ChristianMurphy 🙇

Long term we still hope to break out the REST Module (oeq-ts-rest-api) and possibly even the Front End (react-front-end) and so I'm trying to understand if this kind of move will make that harder. Thoughts?

Also, some of these are completely independent - such as autotest/IntegTester/ps and Source/Plugins/Core/com.equella.core/test/javascript (and kind of Source/Plugins/Core/com.equella.core/swaggerui) - so do we really want them all linked together?

Lastly, although maybe just a personal perception, doesn't it seems odd to manage this with workspaces and in a monorepo kind of way when first and foremost it's really a Java/Scala project being built via SBT. Could this confuse things? (That said, due to our root package.json this is already a bit blurred.)

I wonder if we're at the point where we do need to push out at least the REST Module. It seems that'd significantly solve the issues we're having. 🤔

(And as for runtypes, gah! I wonder why that is now raising it's head - something else must be slightly odd with the setup....)

[ChristianMurphy]

Long term we still hope to break out the REST Module (oeq-ts-rest-api) and possibly even the Front End (react-front-end) and so I'm trying to understand if this kind of move will make that harder. Thoughts?

I think not. It mainly impacts package-lock.json, the packages and scripts themselves are mostly unchanged.
It may even simplify the process, since linked pacakge (which workspaces enables) behave more like packages pulled from npm itself.

Also, some of these are completely independent - such as autotest/IntegTester/ps and Source/Plugins/Core/com.equella.core/test/javascript (and kind of Source/Plugins/Core/com.equella.core/swaggerui) - so do we really want them all linked together?

That is a really good question, which I don't have a good answer for.
I don't think it will cause any harm, lockfile version 2 still allows them to version packages independently, and uses mostly the same folder layout as non-workspace projects would have.

Lastly, although maybe just a personal perception, doesn't it seems odd to manage this with workspaces and in a monorepo kind of way when first and foremost it's really a Java/Scala project being built via SBT. Could this confuse things? (That said, due to our root package.json this is already a bit blurred.)

Maybe?
I don't see oEQ is a project so much as a federation of projects, with loosely tied together build tools.

Wanting them managed by a single build tool is a perfectly fine goal.
I don't think that npm workspaces would stand in the way of that goal.
If SBT will become the one tool to rule them all, it would mostly hinge on SBT having plugins/integrations to manage other build tools.

uPortal is a similarly eclectic project, which also went for unifying under one build tool.
There Gradle is the parent build system, it manages several Gradle sub-projects, several of those sub-projects use https://github.com/node-gradle/gradle-node-plugin to managed node builds (gradle node plugin downloads it's own node and npm on demand and manages npm build steps along side java ones)
With everything under the Gradle unbrella, ./gradle install is the only step needed to setup and build the project.

Maybe SBT has similar integrations, that could be pulled in? or similar integrations could be built out for oEQ?
As an alternative gradle has a lot more integrations pre-built and from my experience is significantly faster than SBT.

[edalex-ian]

Heya @ChristianMurphy

Thank you for your considered response as always.

I think at the end of the day we need to do something here as we're starting to have all manner of things going pear shaped in CI land since upgrading Node/NPM. Yesterday we had a number of odd NPM related build failures on GHA for renovate updates.

I like your assessment that in theory moving to workspaces shouldn't make it any harder to split later, so for now maybe it is the easiest path forward. (And hey, it'll also kind of provide us a catalogue of all the nested JS projects.)

And yes, I still dream and hope for the day when time permits a migration to gradle. Although we do have some custom tasks for SBT to do some of our JS building, it's so slow and heavy that we obviously still run a lot of things independently rather than having any smooth unification of build tooling. (And you definitely wouldn't do any commit hooks reliant on SBT launching first.) One day....

I see we've caused a number of conflicts here - due to package-lock-json updates so should be easy to address. But if we can see this succeed with a green build, then yes, let's proceed and take it for a spin.

[ChristianMurphy]

runtypes issues appear to have come from a duplicated version.
oeq-ts-rest-api uses a version range

openEQUELLA/oeq-ts-rest-api/package.json

Line 35 in ecf6494

"runtypes": "^6.0.0",

this doesn't.
When those get out of sync RunTypes errors start popping up.
Long term, the version range (or lack of version range) should be synced.
Short term, I'm locking them to the same version.

[ChristianMurphy]

alternatively this could cd to the top level and run the install again there.

[ChristianMurphy]

this is my best guess at the changes, I can't see the CodeBuild logs to confirm

[ChristianMurphy]

long term, calling this once at the top level of SBT or Gradle could give the more unified feel.
npm would still be there, but be managed by the Java/Scala build tool.