Skip to content

Commit a856df4

Browse files
Copilotdanmarshall
andauthored
Add value plugin for inline JSON and YAML data in markdown fence blocks, schema definitions for inline data elements, and compiler support (#130)
* Initial plan * Add JSON plugin for inline JSON data Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Update compiler to handle JSON format in dataLoaders Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Add comprehensive JSON plugin example and documentation Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Update JSON plugin to use 'json data variableId' syntax and script tag Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Rename json plugin to json-data and fix loader.ts JSON handling Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Fix JSON dataLoaders to always output as fence blocks and rename types Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Rename plugin from json-data to data, remove special case logic Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Fix plugin name extraction to use infoWords instead of slice Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Preserve signal bus state during reset by carrying over logLevel and logWatchIds * handle nullref * Fix plugin name extraction to check for 'json' keyword directly in infoWords * receive initial batches without running * Optimize initial batch handling in SignalBus by reusing the constructed initialBatch object for all peers. * remove json data examples * Add YAML support to data plugin Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Refactor Mermaid diagram rendering to use MermaidInstance for improved state management and eliminate redundant parameters * Improve batch handling in Vega plugin by setting needToRun flag based on receiveBatch result * Revert UMD build artifacts to exclude from PR Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Revert loader.ts changes and add UMD file guidance to copilot instructions Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Rename data plugin to value plugin Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Add value plugin documentation to schema Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Revert built schema file and add value plugin docs to source schema Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Add ValueElement, CsvElement, TsvElement, DsvElement to schema Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Fix schema: revert built files, update types, move inline data to page, use element types in examples Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Polish schema and compiler: minimal comments, arrays for content, validator support Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Revert built files (UMD bundles and schema files) from PR Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Remove built schema files and interactive.ts from PR Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Remove docs/schema/idoc_v1.json from PR Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> * Refactor DSV parsing functions to consolidate variable ID and delimiter parsing into a single utility function * Update ignoredSignals to replace 'origins' with 'value' * Add parseFenceInfo function to extract metadata from fence info strings * Add default 'value' plugin fallback for JSON and YAML data in create function * Refactor value plugin to use parseFenceInfo for improved fence info parsing and variable ID handling * Refactor parseDsvInfo to utilize parseFenceInfo for improved parameter handling and simplify delimiter logic * Refactor JSON and YAML element types to remove '-value' suffix for consistency * alphabetize * common header * remove comments * remove csv etc * convert to data loaders * fix: update type from "yaml-value" to "yaml" in YAML Value Plugin test * refactor: remove YAML Value Plugin test JSON file * refactor: remove InlineDataElement and related exports * refactor: remove JSON and YAML handling from groupMarkdown function * refactor: remove JSON and YAML element validation from validateElement function * remove json --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: danmarshall <11507384+danmarshall@users.noreply.github.com> Co-authored-by: Dan Marshall <danmar@microsoft.com>
1 parent f79d9cb commit a856df4

14 files changed

Lines changed: 557 additions & 115 deletions

File tree

.github/copilot-instructions.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,12 @@ Chartifact consists of several interoperating modules:
2929
### Testing
3030
- Currently we don't have much test coverage. We can add this later.
3131

32+
### Version Control
33+
- **Do NOT check in UMD build artifacts**: Files matching `*.umd.js` in `docs/dist/v1/` are build outputs and should not be committed in PRs
34+
- **Do NOT edit `docs/schema/idoc_v1.d.ts` directly**: This file is generated from `packages/schema-doc/src/`. Edit the source files in the schema-doc package instead
35+
- **Do NOT check in built schema files**: Files `docs/schema/idoc_v1.d.ts` and `docs/schema/idoc_v1.json` are generated from `packages/schema-doc/src/` during the build process and should not be committed in PRs
36+
- These files are generated during the build process and committing them creates unnecessary merge conflicts
37+
3238
## Project-Specific Conventions
3339

3440
1. **File Formats**:

packages/compiler/src/validate/common.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ import { validateTransforms } from './transforms.js';
55

66
const illegalChars = '/|\\\'"`,.;:~-=+?!@#$%^&*()[]{}<>';
77

8-
export const ignoredSignals = ['width', 'height', 'padding', 'autosize', 'background', 'style', 'parent', 'datum', 'item', 'event', 'cursor', 'origins'];
8+
export const ignoredSignals = ['width', 'height', 'padding', 'autosize', 'background', 'style', 'parent', 'datum', 'item', 'event', 'cursor', 'value'];
99

1010
// Utility functions for property validation
1111
export function validateRequiredString(value: any, propertyName: string, elementType: string): string[] {

packages/markdown/src/factory.ts

Lines changed: 23 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -152,31 +152,41 @@ export function create() {
152152
if (directPlugin) {
153153
return directPlugin;
154154
}
155+
156+
// Split info into words for further processing
157+
const infoWords = info.split(/\s+/);
158+
155159
// Third priority: Check for plugin names with additional parameters (like "csv variableId")
156-
else {
157-
const infoWords = info.split(/\s+/);
158-
if (infoWords.length > 0) {
159-
const pluginPrefix = findPluginByPrefix(infoWords[0]);
160-
if (pluginPrefix) {
161-
return pluginPrefix;
162-
}
160+
if (infoWords.length > 0) {
161+
const pluginPrefix = findPluginByPrefix(infoWords[0]);
162+
if (pluginPrefix) {
163+
return pluginPrefix;
163164
}
164165
}
166+
165167
// Fourth priority: Check if it starts with "json " and extract the plugin name
166-
if (info.startsWith('json ')) {
167-
const jsonPluginName = info.slice(5).trim();
168-
const jsonPlugin = findPlugin(jsonPluginName);
168+
if (infoWords[0] === 'json' && infoWords.length > 1) {
169+
const jsonPlugin = findPlugin(infoWords[1]);
169170
if (jsonPlugin) {
170171
return jsonPlugin;
171172
}
173+
// Default to 'value' plugin for json data (e.g., "json products")
174+
const valuePlugin = findPlugin('value');
175+
if (valuePlugin) {
176+
return valuePlugin;
177+
}
172178
}
173179
// Fifth priority: Check if it starts with "yaml " and extract the plugin name
174-
else if (info.startsWith('yaml ')) {
175-
const yamlPluginName = info.slice(5).trim();
176-
const yamlPlugin = findPlugin(yamlPluginName);
180+
else if (info.startsWith('yaml ') && infoWords.length > 1) {
181+
const yamlPlugin = findPlugin(infoWords[1]);
177182
if (yamlPlugin) {
178183
return yamlPlugin;
179184
}
185+
// Default to 'value' plugin for yaml data (e.g., "yaml products")
186+
const valuePlugin = findPlugin('value');
187+
if (valuePlugin) {
188+
return valuePlugin;
189+
}
180190
}
181191
}
182192

packages/markdown/src/plugins/config.ts

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,131 @@ import { getJsonScriptTag } from "./util.js";
99
import { SpecReview } from 'common';
1010
import * as yaml from 'js-yaml';
1111

12+
/**
13+
* Parse fence info and extract all metadata.
14+
* @param info The fence info string
15+
* @returns Object with format, pluginName, params (including variableId), and wasDefaultId flag
16+
*/
17+
export function parseFenceInfo(info: string): {
18+
format: 'json' | 'yaml';
19+
pluginName: string;
20+
params: Map<string, string>;
21+
wasDefaultId: boolean;
22+
} {
23+
const parts = info.trim().split(/\s+/);
24+
25+
// Determine format (json is default)
26+
let format: 'json' | 'yaml' = 'json';
27+
let startIndex = 0;
28+
29+
if (parts[0] === 'json' || parts[0] === 'yaml') {
30+
format = parts[0];
31+
startIndex = 1;
32+
}
33+
34+
// The next part could be the plugin name OR a parameter
35+
let pluginName = '';
36+
let pluginNameIndex = startIndex;
37+
38+
if (startIndex < parts.length && !parts[startIndex].includes(':')) {
39+
// It's a plugin name (doesn't have a colon)
40+
pluginName = parts[startIndex];
41+
pluginNameIndex = startIndex + 1;
42+
}
43+
44+
const params = new Map<string, string>();
45+
let variableId: string | null = null;
46+
47+
// Parse remaining parts starting after plugin name (if found)
48+
for (let i = pluginNameIndex; i < parts.length; i++) {
49+
const part = parts[i];
50+
const colonIndex = part.indexOf(':');
51+
52+
if (colonIndex > 0) {
53+
// Parameter with colon
54+
const key = part.slice(0, colonIndex);
55+
const value = part.slice(colonIndex + 1);
56+
57+
if (value) {
58+
// Format: "key:value"
59+
params.set(key, value);
60+
} else if (i + 1 < parts.length) {
61+
// Format: "key: value" (value in next part)
62+
params.set(key, parts[++i]);
63+
}
64+
} else if (!variableId) {
65+
// First non-parameter value becomes variableId
66+
variableId = part;
67+
}
68+
}
69+
70+
// If variableId param exists, use it; otherwise use the direct value
71+
const explicitVariableId = params.get('variableId');
72+
const finalVariableId = explicitVariableId || variableId;
73+
const wasDefaultId = !finalVariableId;
74+
75+
if (finalVariableId) {
76+
params.set('variableId', finalVariableId);
77+
}
78+
79+
return { format, pluginName, params, wasDefaultId };
80+
}
81+
82+
/*
83+
//Tests for parseFenceInfo
84+
const tests: [string, { format: 'json' | 'yaml'; pluginName: string; variableId: string | undefined; wasDefaultId: boolean }][] = [
85+
//Direct format cases
86+
["dsv products", { format: "json", pluginName: "dsv", variableId: "products", wasDefaultId: false }],
87+
["csv officeSupplies", { format: "json", pluginName: "csv", variableId: "officeSupplies", wasDefaultId: false }],
88+
89+
//Explicit format cases
90+
["json inventory", { format: "json", pluginName: "inventory", variableId: undefined, wasDefaultId: true }],
91+
["yaml products", { format: "yaml", pluginName: "products", variableId: undefined, wasDefaultId: true }],
92+
93+
//Explicit plugin name
94+
["json value inventory", { format: "json", pluginName: "value", variableId: "inventory", wasDefaultId: false }],
95+
["yaml value products", { format: "yaml", pluginName: "value", variableId: "products", wasDefaultId: false }],
96+
97+
//Direct plugin without format
98+
["value inventory", { format: "json", pluginName: "value", variableId: "inventory", wasDefaultId: false }],
99+
100+
//Explicit variableId parameter (no space)
101+
["json variableId:inventory", { format: "json", pluginName: "", variableId: "inventory", wasDefaultId: false }],
102+
["dsv variableId:products", { format: "json", pluginName: "dsv", variableId: "products", wasDefaultId: false }],
103+
["json value variableId:inventory", { format: "json", pluginName: "value", variableId: "inventory", wasDefaultId: false }],
104+
["yaml value variableId:products", { format: "yaml", pluginName: "value", variableId: "products", wasDefaultId: false }],
105+
106+
//Explicit variableId parameter (with space)
107+
["json variableId: inventory", { format: "json", pluginName: "", variableId: "inventory", wasDefaultId: false }],
108+
["dsv variableId: products", { format: "json", pluginName: "dsv", variableId: "products", wasDefaultId: false }],
109+
["json value variableId: inventory", { format: "json", pluginName: "value", variableId: "inventory", wasDefaultId: false }],
110+
["yaml value variableId: products", { format: "yaml", pluginName: "value", variableId: "products", wasDefaultId: false }],
111+
112+
//Multiple parameters
113+
["dsv products delimiter:|", { format: "json", pluginName: "dsv", variableId: "products", wasDefaultId: false }],
114+
["dsv delimiter:| variableId:products", { format: "json", pluginName: "dsv", variableId: "products", wasDefaultId: false }],
115+
["dsv delimiter: | variableId: products", { format: "json", pluginName: "dsv", variableId: "products", wasDefaultId: false }],
116+
117+
//No variableId
118+
["json value", { format: "json", pluginName: "value", variableId: undefined, wasDefaultId: true }],
119+
["dsv", { format: "json", pluginName: "dsv", variableId: undefined, wasDefaultId: true }],
120+
];
121+
122+
tests.forEach(([input, expected], i) => {
123+
const result = parseFenceInfo(input);
124+
const variableId = result.params.get('variableId');
125+
const pass =
126+
result.format === expected.format &&
127+
result.pluginName === expected.pluginName &&
128+
variableId === expected.variableId &&
129+
result.wasDefaultId === expected.wasDefaultId;
130+
131+
console.log(
132+
`${pass ? '✅' : '❌'} Test ${i + 1}: ${pass ? 'PASS' : `FAIL\n Input: "${input}"\n Got: ${JSON.stringify({format: result.format, pluginName: result.pluginName, variableId, wasDefaultId: result.wasDefaultId})}\n Expected: ${JSON.stringify(expected)}`}`
133+
);
134+
});
135+
*/
136+
12137
/**
13138
* Creates a plugin that can parse both JSON and YAML formats
14139
*/

packages/markdown/src/plugins/dsv.ts

Lines changed: 46 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import { sanitizedHTML, sanitizeHtmlComment } from '../sanitize.js';
99
import { pluginClassName } from './util.js';
1010
import { PluginNames } from './interfaces.js';
1111
import { SpecReview } from 'common';
12+
import { parseFenceInfo } from './config.js';
1213

1314
interface DsvInstance {
1415
id: string;
@@ -24,65 +25,58 @@ export interface DsvSpec {
2425
}
2526

2627
/**
27-
* Utility function to parse variable ID from fence info.
28-
* Supports both "pluginName variableId" and "pluginName variableId:name" formats.
29-
* @param info The fence info string (e.g., "csv myData" or "csv variableId:myData")
28+
* Utility function to parse DSV fence info.
29+
* Supports formats like:
30+
* - "dsv products delimiter:|"
31+
* - "dsv delimiter:| variableId:products"
32+
* - "dsv delimiter: | variableId: products"
33+
* @param info The fence info string
3034
* @param pluginName The plugin name (csv, tsv, dsv)
3135
* @param index The fence index for default naming
32-
* @returns Object with variableId and wasDefaultId flag
36+
* @returns Object with variableId, delimiter, and flags
3337
*/
34-
export function parseVariableId(info: string, pluginName: string, index: number): { variableId: string; wasDefaultId: boolean } {
35-
const parts = info.trim().split(/\s+/);
38+
export function parseDsvInfo(info: string, pluginName: string, index: number): {
39+
variableId: string;
40+
delimiter: string;
41+
wasDefaultId: boolean;
42+
wasDefaultDelimiter: boolean;
43+
} {
44+
const { params, wasDefaultId } = parseFenceInfo(info);
3645

37-
// Check for explicit variableId: parameter
38-
for (const part of parts) {
39-
if (part.startsWith('variableId:')) {
40-
return {
41-
variableId: part.slice(11).trim(), // Remove 'variableId:' prefix and trim spaces
42-
wasDefaultId: false
43-
};
44-
}
45-
}
46+
// Get variableId (already handled by parseFenceInfo)
47+
const variableId = params.get('variableId') || `${pluginName}Data${index}`;
4648

47-
// Check for direct format (second parameter that's not a special parameter)
48-
if (parts.length >= 2) {
49-
const secondPart = parts[1];
50-
if (!secondPart.startsWith('delimiter:') && !secondPart.startsWith('variableId:')) {
51-
return {
52-
variableId: secondPart,
53-
wasDefaultId: false
54-
};
55-
}
56-
}
49+
// Get delimiter from parameter or use default
50+
let delimiter = params.get('delimiter') || ',';
51+
const wasDefaultDelimiter = !params.has('delimiter');
52+
53+
// Handle special delimiter characters
54+
if (delimiter === '\\t') delimiter = '\t';
55+
if (delimiter === '\\n') delimiter = '\n';
56+
if (delimiter === '\\r') delimiter = '\r';
5757

58-
// Default variable ID
59-
return {
60-
variableId: `${pluginName}Data${index}`,
61-
wasDefaultId: true
58+
return {
59+
variableId,
60+
delimiter,
61+
wasDefaultId,
62+
wasDefaultDelimiter
6263
};
6364
}
6465

6566
/**
66-
* Utility function to parse delimiter from fence info.
67-
* @param info The fence info string (e.g., "dsv delimiter:| variableId:myData")
68-
* @returns Object with delimiter and wasDefaultDelimiter flag
67+
* Utility function to parse variable ID from fence info.
68+
* Used by CSV and TSV plugins.
69+
* @param info The fence info string (e.g., "csv myData" or "csv variableId:myData")
70+
* @param pluginName The plugin name (csv, tsv, dsv)
71+
* @param index The fence index for default naming
72+
* @returns Object with variableId and wasDefaultId flag
6973
*/
70-
export function parseDelimiter(info: string): { delimiter: string; wasDefaultDelimiter: boolean } {
71-
const parts = info.trim().split(/\s+/);
72-
73-
for (const part of parts) {
74-
if (part.startsWith('delimiter:')) {
75-
let delimiter = part.slice(10).trim(); // Remove 'delimiter:' prefix and trim spaces
76-
// Handle special cases
77-
if (delimiter === '\\t') delimiter = '\t';
78-
if (delimiter === '\\n') delimiter = '\n';
79-
if (delimiter === '\\r') delimiter = '\r';
80-
return { delimiter, wasDefaultDelimiter: false };
81-
}
82-
}
83-
84-
// Default to comma
85-
return { delimiter: ',', wasDefaultDelimiter: true };
74+
export function parseVariableId(info: string, pluginName: string, index: number): { variableId: string; wasDefaultId: boolean } {
75+
const result = parseDsvInfo(info, pluginName, index);
76+
return {
77+
variableId: result.variableId,
78+
wasDefaultId: result.wasDefaultId
79+
};
8680
}
8781

8882
function inspectDsvSpec(spec: DsvSpec): RawFlaggableSpec<DsvSpec> {
@@ -92,17 +86,13 @@ function inspectDsvSpec(spec: DsvSpec): RawFlaggableSpec<DsvSpec> {
9286
reasons: []
9387
};
9488

95-
// Flag if we had to use defaults
89+
// Only flag if we had to use default variable ID
90+
// Using default delimiter (comma) is fine and shouldn't be flagged
9691
if (spec.wasDefaultId) {
9792
result.hasFlags = true;
9893
result.reasons.push('No variable ID specified - using default');
9994
}
10095

101-
if (spec.wasDefaultDelimiter) {
102-
result.hasFlags = true;
103-
result.reasons.push('No delimiter specified - using default comma');
104-
}
105-
10696
return result;
10797
}
10898

@@ -115,9 +105,8 @@ export const dsvPlugin: Plugin<DsvSpec> = {
115105
const content = token.content.trim();
116106
const info = token.info.trim();
117107

118-
// Use utility functions to parse delimiter and variable ID
119-
const { delimiter, wasDefaultDelimiter } = parseDelimiter(info);
120-
const { variableId, wasDefaultId } = parseVariableId(info, 'dsv', index);
108+
// Parse both delimiter and variable ID in one pass
109+
const { variableId, delimiter, wasDefaultId, wasDefaultDelimiter } = parseDsvInfo(info, 'dsv', index);
121110

122111
return sanitizedHTML('pre', {
123112
id: `${pluginName}-${index}`,

packages/markdown/src/plugins/index.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ import { checkboxPlugin } from './checkbox.js';
99
import { commentPlugin } from './comment.js';
1010
import { cssPlugin } from './css.js';
1111
import { csvPlugin } from './csv.js';
12+
import { valuePlugin } from './value.js';
1213
import { dsvPlugin } from './dsv.js';
1314
import { googleFontsPlugin } from './google-fonts.js';
1415
import { dropdownPlugin } from './dropdown.js';
@@ -30,6 +31,7 @@ export function registerNativePlugins() {
3031
registerMarkdownPlugin(commentPlugin);
3132
registerMarkdownPlugin(cssPlugin);
3233
registerMarkdownPlugin(csvPlugin);
34+
registerMarkdownPlugin(valuePlugin);
3335
registerMarkdownPlugin(dsvPlugin);
3436
registerMarkdownPlugin(googleFontsPlugin);
3537
registerMarkdownPlugin(dropdownPlugin);

0 commit comments

Comments
 (0)