JSON Schema generation improvements (#108)

Refactoring submodules; XProc-based testing; XSpec testing of JSON Schema generation; improvements to JSON Schema generation - addressing #105 also #2007.

Author: wendellpiez

.github/workflows/test.yml

@@ -18,7 +18,7 @@ jobs:
     steps:
       - uses: actions/checkout@v3
         with:
-          submodules: true
+          submodules: "recursive"
           fetch-depth: 0
       - uses: actions/setup-java@v3
         with:

.gitignore

@@ -8,3 +8,7 @@
 
 # test outputs
 src/**/test_output/
+
+# downloaded miscellaneous
+
+support/lib/*

.gitmodules

@@ -1,13 +1,11 @@
-[submodule "support/xspec"]
-	path = support/xspec
-	url = https://github.com/xspec/xspec.git
 [submodule "support/metaschema"]
 	path = support/metaschema
 	url = https://github.com/usnistgov/metaschema.git
 	branch = develop
 	fetchRecurseSubmodules = false
+[submodule "support/xslt3-functions"]
+	path = support/xslt3-functions
+	url = https://github.com/usnistgov/xslt3-functions.git
 [submodule "support/schxslt"]
 	path = support/schxslt
 	url = https://github.com/schxslt/schxslt.git
-[submodule "metaschema"]
-	url = https://github.com/usnistgov/metaschema.git

README.md

@@ -42,7 +42,9 @@ Additionally, we care about, but do not prioritize:
 
 Formerly housed in the [Metaschema repository](https://github.com/usnistgov/metaschema), this code base traces the history of development of the Metaschema concept in the context of the OSCAL project. It was originally conceived as a demonstration and proof of concept, providing a bridge enabling JSON- and XML-based development in parallel over common problem sets and common data. Success in this effort led to a determination that multiple implementations of a platform-independent specification were needed, at which point this implementation was carved out into its own repository.
 
-### Project sunset
+### Maturity and project sunset
+
+Software in this repository remains 'experimental' in status until it can be demonstrated with public conformance testing, which in turn depends on the maturity of the Metaschema initiative.
 
 The NIST team has no long-term plan to maintain this project outside the context of the initiatives it supports, [OSCAL](https://pages.nist.gov/OSCAL) and [Metaschema](https://pages.nist.gov/metaschema), while we reserve the right to change and adapt to community needs in alignment with NIST's mission. A project goal is to enable the stabilization and socialization of the Metaschema technology, as evidenced by the use and support of *other* Metaschema implementations.
 

TESTING.md

@@ -40,7 +40,6 @@ Currently we are emulating and re-engineering some specific XSpec capabilities i
 
 These efforts are focused on producing and refining XSpec runtimes for various use cases and scenarios with specialized requirements faced by this project, such as arbitrary batching and iXML support. Tools we develop here are released under the same terms as Metaschema-XSLT (as open-source software).
 
-
 ## Test-driven development
 
 Almost all testing in this repository falls into the category of either XSLT transformations, or runtimes that embed transformations.

src/Makefile

@@ -10,6 +10,10 @@ dirs:=$(shell find '.' ! -wholename ./Makefile -name 'Makefile' -printf "%h\n")
 test: ## Run all tests
 	$(call FOREACH_MAKE,$@,$(dirs))
 
+.PHONY: pre-checks
+pre-checks: ## Run all "pre checks", enforcing validation contracts across input artifacts
+	$(call FOREACH_MAKE_OPTIONAL,$@,$(dirs))
+
 .PHONY: smoke-test
 smoke-test: ## Run all "smoke tests", establishing a baseline of sanity across the project
 	$(call FOREACH_MAKE_OPTIONAL,$@,$(dirs))

src/common/metaschema-ws-strip.xsl

@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
+   xmlns:xs="http://www.w3.org/2001/XMLSchema"
+   xmlns:mx="http://csrc.nist.gov/ns/csd/metaschema-xslt"
+   xmlns:math="http://www.w3.org/2005/xpath-functions/math"
+   xmlns:fn="http://www.w3.org/2005/xpath-functions"
+   exclude-result-prefixes="xs math mx"
+   xmlns="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+   xpath-default-namespace="http://csrc.nist.gov/ns/oscal/metaschema/1.0"
+   version="3.0">
+   
+   
+<!-- This XSLT relies on a global strip-space to strip whitespace-only nodes from
+     everything except the Markdown elements -->
+   <xsl:strip-space elements="*"/>
+   
+   <xsl:preserve-space elements="p h1 h2 h3 h4 h5 h6 pre li td
+      a insert code em i b strong sub sup fn:string"/>
+   
+   <xsl:mode name="mx:stripping" on-no-match="shallow-copy"/>
+   
+   <xsl:function name="mx:scrub-up" as="element()">
+      <xsl:param name="in" as="node()"/>
+      <xsl:apply-templates mode="mx:stripping" select="$in"/>
+   </xsl:function>
+
+</xsl:stylesheet>

src/testing/issue_235_regression-metaschema.xml src/converter-gen/issue_235_regression-metaschema.xml

@@ -2,7 +2,9 @@
 <x:description 
     xmlns:x="http://www.jenitennison.com/xslt/xspec"
     xmlns:mv="http://g"
-    stylesheet="../converter-gen/xml-to-json/supermodel-to-json.xsl">
+    stylesheet="xml-to-json/supermodel-to-json.xsl">
+   
+   <!-- Tests serialization into JSON format of 'supermodel' XML sources -->
     <x:scenario label="Regression Test for usnistgov/metaschema#235">
         <x:scenario label="If a document instance has a field with no value or flag defined">
             <x:context  mode="write-json">

src/testing/issue_235_regression.xspec src/converter-gen/issue_235_regression.xspec

@@ -10,8 +10,8 @@ include ../../testing/make_common.mk
 
 module_path:=$(shell dirname $(realpath $(firstword $(MAKEFILE_LIST))))
 output_folder:=$(module_path)/test_output
-xspec_script=$(realpath $(module_path)/../../../support/xspec-dev/mvn-saxon-xspec-batch-quiet.sh)
-xspec_ci_script=$(realpath $(module_path)/../../../support/xspec-dev/mvn-saxon-xspec-batch.sh)
+xspec_script=$(realpath $(module_path)/../../../support/xslt3-functions/xspec-dev/mvn-saxon-xspec-batch-quiet.sh)
+xspec_ci_script=$(realpath $(module_path)/../../../support/xslt3-functions/xspec-dev/mvn-saxon-xspec-batch.sh)
 smoketest_script=$(realpath $(module_path)/testing/smoketest-computer-inspector.sh)
 testing-refresh_script=$(realpath $(module_path)/testing/refresh-test-inspectors.sh)
 folder=.

src/schema-gen/InspectorXSLT/Makefile

@@ -1,11 +1,10 @@
-<?xml version="1.0" encoding="us-ascii"?>
 <xsl:transform xmlns="http://www.w3.org/1999/xhtml"
                 xmlns:mx="http://csrc.nist.gov/ns/csd/metaschema-xslt"
                 xmlns:xs="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                 version="3.0"
                 xpath-default-namespace="http://example.com/ns/computer"
-                exclude-result-prefixes="#all"><!-- Generated 2024-02-12T18:06:18.57633-05:00 -->
+                exclude-result-prefixes="#all"><!-- Generated 2024-03-11T14:53:43.203885-04:00 -->
    <xsl:mode on-no-match="fail"/>
    <xsl:mode name="test" on-no-match="shallow-skip"/>
    <!-- .     .     .     .     .     .     .     .     .     .     .     .     .     .     .     .     . -->