A proposal for Nunjucks maintenance

[nataliecarey]

I've used Nunjucks in projects for a variety of organisations since ~2013 and I'm intensely grateful for the work that's gone into it (especially @fdintino who's been maintaining it recently). I've seen the requests for maintainers and I'm happy to help. I started looking into the current status of the project and I have some observations:

• The build on GitHub fails when no meaningful change is made (e.g. Clarify title filter behavior in Templating docs #1481)
• The tests run fine locally for me on Ubuntu (I haven't tested elsewhere)
• There are a number of issues flagged up by npm audit, primarily referring to the build/test tools
• I see no sign that the delivered bundle is affected by any issues flagged by npm audit
• In order to appease npm audit major updates will be needed, this seems to primarily relate to the tooling for build/test
• A number of the tools relied upon by the build/test are deprecated or not maintained
• Upgrading one dependency generally starts a chain reaction with other dependencies needing to be updated

I am keen to play a part in improving this situation, I don't mind working through these things on my own but part of the reason for raising this ticket is to see whether others want to collaborate.

My proposal is to break this down into smaller problems, focus on the tooling for build/test and get to the point where Nunjucks no longer relies on deprecated dependencies. I know there are a lot of new feature requests but I think that should only be done once the build/test tools are using supported software.

My suggestion of how to break the problem down is:

1. Introduce a package-lock.json & get the GitHub build/test working

This will enable us to rely on the tests for each update, therefore it should be done first. Releasing a new version at this stage would be ideal as it will test the full process. This new version wouldn't have any meaningful user-facing changes, therefore if anything breaks users don't lose anything by rolling back a minor/patch version.

2. Replace PhantomJS with Puppeteer or Selenium

Currently the mocha tests run in the browser (which is important because that's how some users use Nunjucks) but they rely on a series of dependencies ending with PhantomJS which is deprecated. My investigation so far hasn't brought up a tool which does this out-of-the-box with supported dependencies, we may need to build something that runs the tests in a browser and reports on the result.

3. Upgrade to the latest WebPack

This will change the delivered bundle and will limit browser support for really old browsers (including PhantomJS), we should then release this in order to make sure that the new packing doesn't cause a problem for projects picking up the latest Nunjucks Changes. Note: Later on I'll mention a survey, that will include questions that might drive decision making in this step.

4. Upgrade the lint tool

The current version of eslint is a long way behind and they've changed their model, the AirBnB rules haven't moved over to the new model yet. There's a chance that changing the lint (either to a newer version of eslint or a different tool) will change the linting requirements leading to code changes which is why this is in a separate step after the previous work.

5. Review the local & GitHub debugging experience

With those tools updated the debugging experience may have been made better or worse, this step is to make sure that both locally and on GitHub we can easily identify failing tests etc. and reproduce them locally with tools like file watchers. This may just be a checkbox exercise but the steps to get to this stage might make the experience worse and if so there should be an opportunity to improve them.

6. Review & triage remaining npm audit issues

My experience with npm audit is that it can flag a lot of things that aren't relevant (e.g. Regular Expression DOS vulnerabilities in build tools), I would suggest aiming to resolve all npm audit issues but we should also be pragmatic about which ones make a difference to the product.

---

Alongside this I would suggest we release a survey getting feedback from people who use Nunjucks in their projects - this will help us to understand things like browser support requirements, the usage breakdown between client-side and server-side etc.

Following the above steps and making releases at the end of each step would mean that we:

• Get back to using supported dependencies
• Have reliable builds
• Have a reliable/repeatable local developer experience
• Know that the release process works (including updating the website Web site download is 3.2.2 #1472)
• Are still using the same production code and tests
• Are able to separate issues with the new build tools from changes in the software

That then puts us back in a position where new features can be added, bugs can be fixed etc.

If this plan sounds suitable (primarily to @fdintino, but also to the community) then I'm happy to take it on, either alone or with others who want to be involved. There's an amount of it that I can do 'from the outside', but I'm also comfortable becoming a maintainer and doing these 'from the inside'.

I've spent the past couple of years wanting to get more involved with Nunjucks, and it's never been the right moment. This is the closest I've come to having the right moment so I'm making the most of it.

I'm very happy to hear feedback on this plan, to be told I've got it all wrong, to join in with someone else's plan etc.

(this primarily relates to the issue #1433 but I've started a new thread as I think it's worth keeping the concrete suggestion separate from that thread)

[jameswragg]

The alpha for Nunjucks v4 was published in a new repo (mentioned in 1059) - its receiving some contributions & I'm sure @fdintino would be grateful of your time!

https://github.com/nunjucks/nunjucks4