-
Notifications
You must be signed in to change notification settings - Fork 226
feat(data-modeling): integrate diagramming package COMPASS-9357 #6979
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
…nto diagramming
i have no idea what exploded lock file |
source: source.ns, | ||
target: target.ns, | ||
markerStart: 'one', | ||
markerEnd: 'many', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this a placeholder item?
Something that's become fairly clear over time is that we're likely going to want to work with concrete cardinalities if we can (i.e. number ranges rather than one
or many
)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, actually it needs to be like that. I'll change this to reflect the store cardinalities. My assumption is that we will be having 1 or 2 as numeric representation for one and many respectively.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So ... what we'll ultimately have is descriptions like "1 to 4–6" or something along those lines, at least if we ultimately want to give schema recommendations (still a goal for Compass to have that eventually), because it's become pretty clear that the numeric values of the relative cardinality ranges matter for those (you maybe be able to create an embedding of collections in a 1:5 case but not in a 1:200 case)
? 'Unknown' | ||
: typeof field.bsonType === 'string' | ||
? field.bsonType | ||
: field.bsonType[0]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line is also some sort of TODO?
Looking through it, it mostly seems to be additions of genuine new dependencies? |
package-lock.json
Outdated
"configs/eslint-config-compass/node_modules/eslint": { | ||
"version": "8.57.1", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like something went really wrong when you installed a new dep, you now have two versions of eslint in package-lock, one of them is not matchin the required one. This is partially why package-lock blew up so much and why CI is failing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, i finally resolved this, thank you. I tried checking out lock file from main couple of times and was not sure what I am missing. Eventually from your other comment realized its origin
that I am missing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you try running
git checkout origin/main -- package-lock.json
npm install
on your branch? This gives a pretty different result for package-lock at least on my machine (without eslint being messed up)
// to React 18, but for the time being we will just resolve react to | ||
// actual files ourselves to work around that | ||
'react/jsx-runtime': require.resolve('react/jsx-runtime'), | ||
react: require.resolve('react'), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably should add this to ignore in depcheckrc file, we do this already for another package that we just need to resolve
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done in bba2841
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
with support for cjs, i ended up removing this.
title={diagramLabel} | ||
edges={edges} | ||
nodes={nodes} | ||
onEdgeClick={(evt, edge) => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we'll be doing removal like this, but it's actually neat for now :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah i wanted to retain the current functionality so ended up adding this.
minute=$(escapeLeadingZero "${ts[4]}") | ||
second=$(escapeLeadingZero "${ts[5]}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Small drive-by because CI failed for this PR in this run.
When hour is 00 and minutes < 10
, the version is as yymmdd-dev.0..
, which is invalid
useEffect(() => { | ||
if ( | ||
process.env.APP_ENV === 'webdriverio' || | ||
process.env.NODE_ENV === 'development' | ||
) { | ||
(window as any).diagramInstance = diagram; | ||
} | ||
}, [diagram]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of using global window and gating it to these very specific checks (that we really truly should try to avoid), let's do something similar to what we do in codeedior and attach the diagram to the relevant DOM node that we can find in e2e tests, something like:
<div
data-testid="diagram-editor-container"
ref=(ref => {
if (ref) {
ref.__diagram = diagram;
}
})
async function getDiagramNodes(browser: CompassBrowser) { | ||
return await browser.execute(() => { | ||
// eslint-disable-next-line no-restricted-globals | ||
if ('diagramInstance' in window) { | ||
// eslint-disable-next-line no-restricted-globals | ||
const diagramInstance = window.diagramInstance as DiagramInstance; | ||
const nodes = diagramInstance.getNodes(); | ||
return nodes.map((node) => ({ | ||
id: node.id, | ||
title: node.data.title, | ||
})); | ||
} | ||
throw new Error( | ||
'Diagram instance not found in the window object. Ensure the diagram is set for tests.' | ||
); | ||
}); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can't say I have the strongest preference, but these are rendering real DOM nodes on the screen and we do expect them to be visible (and so accessible through the normal wdio APIs). Are there any issues with testing them without relying on internals of the library? These are e2e tests after all
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Currently we are not positioning these on the viewport correctly (the nodes are randomly placed) and with virutalization enabled for diagramming, they many not be rendered. So I tried to use this approach.
I think for tests we may want to disable virtualization so that everything is rendered and we don't need to do this kinda stuff or even using ref to access diagram. I added COMPASS-9447 to export virtualization flag from diagramming package and once that's done, I can do a followup on this and clean it completely.
Let me know what you think about it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(maybe with failing tests on mac, i need to revisit this whole thing in this PR actually)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think noticeably changing how rendering works there only for e2e tests is a good idea, this should stay as close to real app as possible.
For visualization we should make sure that elements are positioned inside the viewport by default, right? So maybe that's where we can test without using internals, otherwise using them is probably fine, I would encourage you to separate the cases where it makes sense to use one or another. Consider what we do for the editor, if we want to validate that user actually sees something we should be testing through UI and wdio feature that mimic user interactions, but if we want to validate that something inside the editor is set correctly (but not necessarily visible) accessing the internals to get the value is fine, I'd suggest to keep this sort of rule of thumb in mind.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct they should by default. Now I am not sure how we are planning to do that (to position nodes correctly), or how we plan to deal with diagram with many nodes. But @xyflow/react
does offer fitView
option so that everything is rendered on the viewport (it changes zoom based on the content). With that in mind, I'll make that change so that content is visible.
Yeah, I dont think we want to test the internals, but rather what we expect user to see and for that I think we can do simple assertions without any access to reactflow instance.
Made change in bed5629
(#6979)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The integration part itself looks good for the scope, I have some questions about how we're e2e testing this, but I'll also approve and maybe consider addressing these things separately if CI is green so that we can unblock other work
Integrated diagramming package in data-modeling to visualize the schema.
Preview
data-modeling-diagram.mov
Description
Checklist
Motivation and Context
Open Questions
Dependents
Types of changes