Performance observations and questions

[j9t]

Mixed bag to share some observations and ask for advice regarding performance on an Eleventy site with ~8,000 Markdown files resulting in ~1,200 output files (the Markdown files can contain content or metadata; the repo is private, but could provide access if of use/interest).

Problem 1: Local data files seem expensive

On the respective site, I have collections for which I provide general metadata in what I understand to be local data files. For example, for all posts, I use a posts.json at the folder root:

{
  "date": "Last Modified",
  "layout": "post.njk",
  "tags": [
    "posts"
  ]
}

However, if these cascade, and subfolders, as in my case, add additional tags, they seem to become really costly.

I get 15,000+ messages of the following type:

Eleventy:Warnings Local data files have conflicting data. Overwriting 'tags' with data from './tools/exploration/exploration.json'. Previous data location was from './tools/tools.json' +263ms

Is this to be expected? How can it be mitigated in a way that is DRY (i.e., without repeating metadata)?

Problem 2: Git dates seem really expensive

For a part of the same site, I rely on “git Last Modified” to provide dates.

This seems extremely expensive performance-wise, and can be felt dearly on Intel machines (the difference between an Intel and M1 Mac is in the range of 60–90 seconds):

Eleventy:Template getMappedDate: using a date in the data for './tools/exploration/uncategorized*/website-archiver-archivetoday.md' of 'git Last Modified' +263ms

It’s understandable that Git operations cost some performance, but are there optimizations Eleventy users can take? Is there something cooking on the Eleventy side that may help (like, caching Git information, making it configurable when to retrieve)?

Problem 3: Passthroughs and reads take time, but unclear how to optimize

Passthroughs and reads seem to take significant time:

Eleventy:Logger Benchmark **123523ms**  78%    38× (Aggregate) Passthrough Copy File +851ms
Eleventy:Benchmark Benchmark 123523ms  78%    38× (Aggregate) Passthrough Copy File +0ms
Eleventy:Logger Benchmark **123080ms**  78%  7355× (Aggregate) Template Read +0ms
Eleventy:Benchmark Benchmark 123080ms  78%  7355× (Aggregate) Template Read +0ms

Is there anything here that Eleventy users can do (apart from reducing file numbers), or likewise, are there optimizations being worked on? (Sharing this with little hope, but perhaps the data point can be helpful :)

—Grateful for any tips on how to improve performance in these cases, as well as for anything that may be done to mitigate on the Eleventy side!

[j9t]

Used this to dissect the respective site. Had not been aware just how super slow git Last Modified is (re problem 2). It is so bad that the effect can be felt with a few dozen files, if not a handful.

Refactored the respective site not to use git Last Modified at all (the Git version would be better, but well). Time for exports down 80% on slow machine, ~65% on M1 machine.