feat(data-modeling): integrate diagramming package COMPASS-9357 #6979

mabaasit · 2025-06-02T09:05:17Z

Integrated diagramming package in data-modeling to visualize the schema.

Preview

data-modeling-diagram.mov

Description

Checklist

New tests and/or benchmarks are included
Documentation is changed or added
If this change updates the UI, screenshots/videos are added and a design review is requested
I have signed the MongoDB Contributor License Agreement (https://www.mongodb.com/legal/contributor-agreement)

Motivation and Context

Bugfix
New feature
Dependency update
Misc

Open Questions

Dependents

Types of changes

Backport Needed
Patch (non-breaking change which fixes an issue)
Minor (non-breaking change which adds functionality)
Major (fix or feature that would cause existing functionality to change)

…gramming

…nto diagramming

…gramming

mabaasit · 2025-06-02T10:20:56Z

i have no idea what exploded lock file

addaleax · 2025-06-02T11:03:54Z

packages/compass-data-modeling/src/components/diagram-editor.tsx

+        source: source.ns,
+        target: target.ns,
+        markerStart: 'one',
+        markerEnd: 'many',


Is this a placeholder item?

Something that's become fairly clear over time is that we're likely going to want to work with concrete cardinalities if we can (i.e. number ranges rather than one or many)

Yes, actually it needs to be like that. I'll change this to reflect the store cardinalities. My assumption is that we will be having 1 or 2 as numeric representation for one and many respectively.

So ... what we'll ultimately have is descriptions like "1 to 4–6" or something along those lines, at least if we ultimately want to give schema recommendations (still a goal for Compass to have that eventually), because it's become pretty clear that the numeric values of the relative cardinality ranges matter for those (you maybe be able to create an embedding of collections in a 1:5 case but not in a 1:200 case)

addaleax · 2025-06-02T11:04:35Z

packages/compass-data-modeling/src/components/diagram-editor.tsx

+                ? 'Unknown'
+                : typeof field.bsonType === 'string'
+                ? field.bsonType
+                : field.bsonType[0];


This line is also some sort of TODO?

addaleax · 2025-06-02T11:05:04Z

i have no idea what exploded lock file

Looking through it, it mostly seems to be additions of genuine new dependencies?

gribnoysup · 2025-06-02T11:38:47Z

package-lock.json

+    "configs/eslint-config-compass/node_modules/eslint": {
+      "version": "8.57.1",


Seems like something went really wrong when you installed a new dep, you now have two versions of eslint in package-lock, one of them is not matchin the required one. This is partially why package-lock blew up so much and why CI is failing

Yeah, i finally resolved this, thank you. I tried checking out lock file from main couple of times and was not sure what I am missing. Eventually from your other comment realized its origin that I am missing.

gribnoysup

Can you try running

git checkout origin/main -- package-lock.json
npm install

on your branch? This gives a pretty different result for package-lock at least on my machine (without eslint being messed up)

gribnoysup · 2025-06-02T13:49:03Z

configs/webpack-config-compass/src/index.ts

+      // to React 18, but for the time being we will just resolve react to
+      // actual files ourselves to work around that
+      'react/jsx-runtime': require.resolve('react/jsx-runtime'),
+      react: require.resolve('react'),


Probably should add this to ignore in depcheckrc file, we do this already for another package that we just need to resolve

done in bba2841

with support for cjs, i ended up removing this.

paula-stacho · 2025-06-04T09:02:56Z

packages/compass-data-modeling/src/components/diagram-editor.tsx

+            title={diagramLabel}
+            edges={edges}
+            nodes={nodes}
+            onEdgeClick={(evt, edge) => {


I don't think we'll be doing removal like this, but it's actually neat for now :)

yeah i wanted to retain the current functionality so ended up adding this.

…gramming

mabaasit · 2025-06-12T00:34:07Z

.evergreen/print-compass-env.sh

+        minute=$(escapeLeadingZero "${ts[4]}")
+        second=$(escapeLeadingZero "${ts[5]}")


Small drive-by because CI failed for this PR in this run.

When hour is 00 and minutes < 10, the version is as yymmdd-dev.0.., which is invalid

gribnoysup · 2025-06-12T08:04:01Z

packages/compass-data-modeling/src/components/diagram-editor.tsx

+  useEffect(() => {
+    if (
+      process.env.APP_ENV === 'webdriverio' ||
+      process.env.NODE_ENV === 'development'
+    ) {
+      (window as any).diagramInstance = diagram;
+    }
+  }, [diagram]);


Instead of using global window and gating it to these very specific checks (that we really truly should try to avoid), let's do something similar to what we do in codeedior and attach the diagram to the relevant DOM node that we can find in e2e tests, something like:

<div data-testid="diagram-editor-container" ref=(ref => { if (ref) { ref.__diagram = diagram; } })

gribnoysup · 2025-06-12T08:19:33Z

packages/compass-e2e-tests/tests/data-modeling-tab.test.ts

+async function getDiagramNodes(browser: CompassBrowser) {
+  return await browser.execute(() => {
+    // eslint-disable-next-line no-restricted-globals
+    if ('diagramInstance' in window) {
+      // eslint-disable-next-line no-restricted-globals
+      const diagramInstance = window.diagramInstance as DiagramInstance;
+      const nodes = diagramInstance.getNodes();
+      return nodes.map((node) => ({
+        id: node.id,
+        title: node.data.title,
+      }));
+    }
+    throw new Error(
+      'Diagram instance not found in the window object. Ensure the diagram is set for tests.'
+    );
+  });
+}


Can't say I have the strongest preference, but these are rendering real DOM nodes on the screen and we do expect them to be visible (and so accessible through the normal wdio APIs). Are there any issues with testing them without relying on internals of the library? These are e2e tests after all

Currently we are not positioning these on the viewport correctly (the nodes are randomly placed) and with virutalization enabled for diagramming, they many not be rendered. So I tried to use this approach.

I think for tests we may want to disable virtualization so that everything is rendered and we don't need to do this kinda stuff or even using ref to access diagram. I added COMPASS-9447 to export virtualization flag from diagramming package and once that's done, I can do a followup on this and clean it completely.

Let me know what you think about it?

(maybe with failing tests on mac, i need to revisit this whole thing in this PR actually)

I don't think noticeably changing how rendering works there only for e2e tests is a good idea, this should stay as close to real app as possible.

For visualization we should make sure that elements are positioned inside the viewport by default, right? So maybe that's where we can test without using internals, otherwise using them is probably fine, I would encourage you to separate the cases where it makes sense to use one or another. Consider what we do for the editor, if we want to validate that user actually sees something we should be testing through UI and wdio feature that mimic user interactions, but if we want to validate that something inside the editor is set correctly (but not necessarily visible) accessing the internals to get the value is fine, I'd suggest to keep this sort of rule of thumb in mind.

Correct they should by default. Now I am not sure how we are planning to do that (to position nodes correctly), or how we plan to deal with diagram with many nodes. But @xyflow/react does offer fitView option so that everything is rendered on the viewport (it changes zoom based on the content). With that in mind, I'll make that change so that content is visible.

Yeah, I dont think we want to test the internals, but rather what we expect user to see and for that I think we can do simple assertions without any access to reactflow instance.

Made change in bed5629 (#6979)

gribnoysup

The integration part itself looks good for the scope, I have some questions about how we're e2e testing this, but I'll also approve and maybe consider addressing these things separately if CI is green so that we can unblock other work

…gramming

mabaasit and others added 9 commits May 26, 2025 13:35

diagramming

0bfc7fc

diagramming

0a52b02

edges

9f866ee

Merge branch 'main' of https://github.com/mongodb-js/compass into dia…

81d011e

…gramming

npm install

e427663

remove relationship

e2b01c7

correct type

37f2a0a

Merge branch 'diagramming' of https://github.com/mongodb-js/compass i…

26f3993

…nto diagramming

bootstrap

acb4084

github-actions bot added the feat label Jun 2, 2025

mabaasit added 3 commits June 2, 2025 11:06

Merge branch 'main' of https://github.com/mongodb-js/compass into dia…

46f7f38

…gramming

no lock changes

fea7bf4

npm i

0c3d611

mabaasit added the no release notes Fix or feature not for release notes label Jun 2, 2025

main lock file

599fbdd

addaleax reviewed Jun 2, 2025

View reviewed changes

gribnoysup reviewed Jun 2, 2025

View reviewed changes

mabaasit added 2 commits June 2, 2025 14:44

checkout lock from main

17601c2

npm install

6a377af

gribnoysup reviewed Jun 2, 2025

View reviewed changes

mabaasit and others added 3 commits June 2, 2025 15:50

add todo and show cardinality

59381eb

npm run check

bba2841

Merge branch 'main' into diagramming

5a8b59e

mabaasit marked this pull request as ready for review June 2, 2025 19:42

mabaasit marked this pull request as draft June 2, 2025 19:43

paula-stacho reviewed Jun 4, 2025

View reviewed changes

update diagramming

de016d6

mabaasit added 3 commits June 11, 2025 08:03

bump package

74b8119

bump package

220cab3

undo webpack changes

78f1d5f

mabaasit marked this pull request as ready for review June 11, 2025 06:18

mabaasit and others added 9 commits June 11, 2025 09:23

missing importexports warning

b085827

Merge branch 'main' of https://github.com/mongodb-js/compass into dia…

f0da710

…gramming

npm i

1aa7fbc

Merge branch 'main' into diagramming

0e16a8f

fix tests

1d3464d

Merge branch 'main' of https://github.com/mongodb-js/compass into dia…

252478a

…gramming

npm i

bf8ae07

fix dev version

5bd2b98

correct func name

dcfc6f2

mabaasit commented Jun 12, 2025

View reviewed changes

gribnoysup reviewed Jun 12, 2025

View reviewed changes

gribnoysup approved these changes Jun 12, 2025

View reviewed changes

clean up tests

bed5629

mabaasit requested a review from a team as a code owner June 12, 2025 15:30

mabaasit added 3 commits June 12, 2025 17:31

Merge branch 'main' of https://github.com/mongodb-js/compass into dia…

49e8269

…gramming

npm i

62b4de2

clean up

b392d15

		"configs/eslint-config-compass/node_modules/eslint": {
		"version": "8.57.1",

		minute=$(escapeLeadingZero "${ts[4]}")
		second=$(escapeLeadingZero "${ts[5]}")

feat(data-modeling): integrate diagramming package COMPASS-9357 #6979

Are you sure you want to change the base?

feat(data-modeling): integrate diagramming package COMPASS-9357 #6979

Uh oh!

Conversation

mabaasit commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Motivation and Context

Open Questions

Dependents

Types of changes

Uh oh!

mabaasit commented Jun 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

addaleax commented Jun 2, 2025

Uh oh!

gribnoysup Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gribnoysup left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mabaasit Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gribnoysup left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mabaasit commented Jun 2, 2025 •

edited

Loading

gribnoysup Jun 2, 2025 •

edited

Loading

mabaasit Jun 12, 2025 •

edited

Loading