[IMPROVEMENT] Optimisation of visualizer

[LucIJspeert]

Describe the problem

Hi there! We are using PGM and PGM-DS in a project to integrate several grids into one. So far it is going well, and your great documentation has helped a lot.
The grid we build is handled just fine by PGM, but we have noticed a quite severe slowdown of the PGM-DS visualizer as we try to pan, zoom, or manipulate components. I am talking about ~90k nodes and as many branches.

Improvement proposal

I don't know what it would take, perhaps I am asking for a lot, but it would be very nice if the visualizer could be made faster so that we can more easily check our work.

[nitbharambe]

Hi @LucIJspeert , glad to hear its going well!
It is possible to optimize the visualizer more but that still would not make it possible to scale to a large number of nodes at a single time. The processing of the visual for dash happens in your browser and performance is limited by it.

Following can be approaches:

• Easy: Filter subgrids before passing to visualize. No changes required at PGM-DS.
• Add a dropdown feature in the dashboard. The grids or arrays would be extended with a classifying attribtue. The visualizer then filters the subgrids internally based on this input and this classifying attribute.
• Zoom based refresh. This would be what most map based applications do. Option seems available via dash or cytoscape but it needs investigation.

What would align best with you?

[LucIJspeert]

Hey! Thanks for your quick reply.
I see the problem, that is very unfortunate. For our application the value lies in the combination of all of the separate parts of the grid. We can display subgrids separately by loading their separate files (so essentially point 1 that you mention). Therefore, option 2 would also not add much functionality for our use case.

I'm afraid that option 3 would also not improve the situation much, as the panning operation is also incredibly slow.

I suppose the conclusion is that nothing short of migrating away from Dash would help here?
Perhaps we can fashion something simple ourselves with a graph visualisation tool.

[Thijss]

Hi @LucIJspeert,

A few things to add from my side:

• The visualizer was never developed with the intention of being production worthy, so use at your own risk! It is mainly intended for debugging purposes (with smaller grids). I personally wouldn't recommend using it in a production environment. It is also the least tested code within the project.
• We can still look whether this is somewhat solvable within dash & cytoscape. At least cytoscape might be able to handle this. It seems to handle 20,000 nodes quite well (see https://cytoscape.org/js-perf/). We'll look into this, or you are welcome to contribute yourself 👍 .
• Performance is of course also affected by the layout you choose. Just to check: I am assuming you are NOT trying to do a physics based layout (like cose), right? breadthfirst or preset (triggered when you provide X/Y coordinates to nodes) will probably be best for performance.
• At Alliander we have visualized the entire MV grid before. While doable, it becomes very slow, like you mentioned. You can read about it here: https://lfenergy.org/visualizing-power-grids-introducing-interactive-grid-visualizations-in-power-grid-model-ds/
• We are currently developing new features for the visualizer (e.g. heatmap/visualize applicances). After those features we can perhaps look into improving the performance for large grids. In any case, keep an eye out for upcoming releases!

[Thijss]

Update

I did some testing with larger grids

The performance limit seems to be about 20.000 nodes for cytoscape. I've tried both 20.000 and 40.000. While 20.000 still works quite smoothly, there is a significant drop in performance at 40.000. At 80.000 it is barely workable.

That means I think it will be difficult to squeeze much more performance out of the current setup, but we'll have a look at a later stage.