Aeg/performance compute shortest paths parallel preparation

[aiAdrian]

Description

It would be highly beneficial to process this in parallel using 2 to 4 threads.
By applying Promise-based chunking and await logic, we could significantly reduce total runtime.
Especially for large network graphs, sequential pathfinding can become a major bottleneck.

Currently, pathfinding runs sequentially for each origin node and for big netzgrafik it takes quite a lot of time!

   // In theory we could parallelize the pathfindings, but the overhead might be too big.
    const res = new Map<string, [number, number]>();
    odNodes.forEach((origin) => {
      computeShortestPaths(origin.getId(), neighbors, vertices, tsSuccessor, cachedKey).forEach(
        (value, key) => {
          res.set([origin.getId(), key].join(","), value);
        },
      );
    });

If we can make the computeShortestPaths as well async then the following code should be executable in parallel. You can just add a new method computeBatchShortestPaths to OriginDestinationService :

async computeBatchShortestPaths(
    odNodes: Node[],
    neighbors: Map<string, [Vertex, number][]>,
    vertices: Vertex[],
    tsSuccessor: Map<number, number>,
    cachedKey: Map<Vertex, string>,
    res: Map<string, [number, number]>,
  ): Promise<void> {
    const numThreads = 1;
    const chunkSize = Math.ceil(odNodes.length / numThreads);
    const allChunks: [number, number][] = [];

    for (let i = 0; i < odNodes.length; i += chunkSize) {
      allChunks.push([i, Math.min(i + chunkSize, odNodes.length)]);
    }

    const threads: Promise<void>[] = [];

    for (const [start, end] of allChunks) {
      const chunk = odNodes.slice(start, end);

      const thread = new Promise<void>((resolve) => {
        for (const origin of chunk) {
          const results = computeShortestPaths(
            origin.getId(),
            neighbors,
            vertices,
            tsSuccessor,
            cachedKey,
          );

          results.forEach((value, key) => {
            res.set([origin.getId(), key].join(","), value);
          });
        }

        console.log("done:", start, end);
        resolve();
      });

      threads.push(thread);
    }

    await Promise.all(threads);
  }

In the originDestinationData(): OriginDestination[] { ... } function (method) we have to replace:

This code snipped

    odNodes.forEach((origin) => {
      computeShortestPaths(origin.getId(), neighbors, vertices, tsSuccessor, cachedKey).forEach(
        (value, key) => {
          res.set([origin.getId(), key].join(","), value);
        },
      );
    });

with

   this.computeBatchShortestPaths(odNodes, neighbors, vertices, tsSuccessor, cachedKey, res);

Issues

Checklist

• This PR contains a description of the changes I'm making
• I've read the Contribution Guidelines
• I've added tests for changes or features I've introduced
• I documented any high-level concepts I'm introducing in documentation/
• CI is currently green and this is ready for review

[aiAdrian]

Just a copy of the original code but declared as async. - only for quick & dirty testing

[aiAdrian]

forEach replaced

[aiAdrian]

Branch Performance Comparison

Browser	Branch: main	Branch: aeg/performance_computeShortestPaths	Branch: aeg/performance_computeShortestPaths_parallel_preparation
Google Chrome	541 ms	169 ms	171 ms
Microsoft Edge	879 ms	420 ms	320 ms

---

Conclusion

Ultimately, this change doesn’t have a significant impact.

• Switching from main to aeg/performance_computeShortestPaths results in a 2–3x speed improvement.'
• From aeg/performance_computeShortestPaths to aeg/performance_computeShortestPaths_parallel_preparation there is no real win but the added complexity in the code isn’t justified by the actual performance gain.

We might need to explore real multithreading (Node.js worker threads could offer more power,
but again, they introduce much more complexity).

Suggestion: Reject this approach for now.

[shenriotpro]

Thanks for trying :)