Skip to content

Conversation

@ld-kerley
Copy link
Contributor

Update OSL Network generator to provide parity with OSL Source generator.

This is a pretty large commit, but most of it is all pretty intertwined, so I'll do my best to outline here all the different changes.

LibsToOso

  • The tool now supports selecting using the NodeDef name or the Implementation name as the basis for the name of the new OSL Network generator implementation and corresponding OSO.
  • OSL code and corresponding OSO is now generated directly from the NodeDef element.
  • implname functionality added is used to define the renamed ports.
  • NodeGraph elements can now be skipped during processing.

OSLNetworkGenerator

  • Generator now directly inherits from ShaderGenerator as there is no common functionality shared with the OSLShaderGenerator.
  • Provides API to compile OSL source, and also to generate a Shader from a NodeDef element, this is consolidated from the LibsToOso tool source.
  • The generator can now create a new ShaderNodeImpl object on demand (and the corresponding compiled OSO file) if a locally defined OSL Source Implementation element is discovered. This includes correctly constructing an OSL Source generator object with all the corresponding custom types defined to allow for the correct shader generation to occur.
  • liboslcomp can be optionally used for the on-demand compilation for better performance, and avoids writing the shader source to disk to compile.
  • New ShaderGraph::flattenGraph() function is used to correctly expand any locally defined NodeGraph elements in to the corresponding nodes. We do not on-demand compile non-functional NodeGraph elements in to OSOs as they do not have corresponding NodeDef elements to fully name and decorate the interface. Once the ShaderGraph is flattened then we are left with a fully expanded graph of individual nodes.
  • OslNetworkSyntax has been refactored.
    • It no longer has any reserved words, as the grammar used to describe the network has no restrictions on words that are used.
    • Parameter value emitting code has been refactored to better support the "built-in" struct types more cleanly, and removes dependency on the OSL Source syntax, as they are fundamentally two different exports. Including better error handling for types that do not have direct parameter export.

MaterialXGenShader

  • New API method GenContext::getSourceCodeSearchPath(), needed to allow dynamic construction of derived generators that share the source code search path.
  • New GenOptions option oslTempOsoPath to allow control over where on-demand OSOs are compiled to. If unspecified the system will fallback to a uniquely generated temp directory in the system temporary directory.
  • Support added for implname attribute during shader generation. This is described in the specification, but not yet implemented. During OSL source generation when compiling the OSOs its possible input names such as normal may be modified to avoid clashes with reserved words. Adding implname support allows this to be described in the generated Implementation elements and respected during OSL Network generation. New getPortName() function is added in support for the implementation of this feature. This is also needs to be stored in two separate classes, the actual implementation can be either a ShaderGraph or a ShaderNodeImpl.
  • New API method ShaderGraph::flattenGraph() will walk the graph and expand any CompoundNode entities discovered. Care is taken to generate unique names for newly generated nodes by prefixing the names with the name of the prior parent CompoundNode, and also to ensure all connections are retained.
  • ShaderGraph::create() can now accept and populate the created graph directly from a NodeDef element, which is simpler than the current Node source, as there are no local modifications.
  • Syntax base class now tracks custom registered syntaxes so they can be easily queried. This is necessary to support dynamic creation of a derived generator, specifically to allow the OSL Network generator to create the OSL Source generator for on-demand compilation of locally defined nodes.

Other supporting changes are outlined below.

Cmake

  • Add CMake option to control if NodeGraph implementations are compiled down to OSOs, or left as NodeGraphs.

MaterialXFormat

  • Add optional recursive argument to FilePath::createDirectory(), to allow for recursive directory creation if necessary.
  • Add API to FilePath to return the system temporary directory, as well as create a new temporary directory. This is used for on-the-fly compilation of OSL source implementation to OSOs if defined locally in a document and not in a pre-compiled standard library.
  • Add a new createDirectories option to XmlWriteOptions to allow for any necessary directories to be created while writing a new file.

MaterialXTest/MaterialXRenderOsl

  • Add new test specific implementations for tangent and bitangent nodes for the OSL Network generator. These may become obsolete if the OSL render testing moves to using the OBJ file (see XXXX).
  • Update the OSL Network scene template to align the cameras after the OSL Source template was changed.
  • Remove the OSL Network specific tests that were skipped - we now have complete one-to-one parity with OSL Source generator.
  • Write fully expanded paths in to the template file for easier testing.
  • Introduce string processing to the command string that needs to be embedded in a legal XML file to replace illegal XML characters < and >.

…tor.

This is a pretty large commit, but most of it is all pretty intertwined, so I'll do my best to outline here all the different changes.

### LibsToOso
* The tool now supports selecting using the NodeDef name or the Implementation name as the basis for the name of the new OSL Network generator implementation and corresponding OSO.
* OSL code and corresponding OSO is now generated directly from the NodeDef element.
* `implname` functionality added is used to define the renamed ports.
* NodeGraph elements can now be skipped during processing.

### OSLNetworkGenerator
* Generator now directly inherits from `ShaderGenerator` as there is no common functionality shared with the `OSLShaderGenerator`.
* Provides API to compile OSL source, and also to generate a Shader from a NodeDef element, this is consolidated from the `LibsToOso` tool source.
* The generator can now create a new ShaderNodeImpl object on demand (and the corresponding compiled OSO file) if a locally defined OSL Source Implementation element is discovered. This includes correctly constructing an OSL Source generator object with all the corresponding custom types defined to allow for the correct shader generation to occur.
* `liboslcomp` can be optionally used for the on-demand compilation for better performance, and avoids writing the shader source to disk to compile.
* New `ShaderGraph::flattenGraph()` function is used to correctly expand any locally defined NodeGraph elements in to the corresponding nodes. We do not on-demand compile non-functional NodeGraph elements in to OSOs as they do not have corresponding NodeDef elements to fully name and decorate the interface. Once the `ShaderGraph` is flattened then we are left with a fully expanded graph of individual nodes.
* `OslNetworkSyntax` has been refactored.
  * It no longer has any reserved words, as the grammar used to describe the network has no restrictions on words that are used.
  * Parameter value emitting code has been refactored to better support the "built-in" struct types more cleanly, and removes dependency on the OSL Source syntax, as they are fundamentally two different exports. Including better error handling for types that do not have direct parameter export.

### MaterialXGenShader
* New API method `GenContext::getSourceCodeSearchPath()`, needed to allow dynamic construction of derived generators that share the source code search path.
* New `GenOptions` option `oslTempOsoPath` to allow control over where on-demand OSOs are compiled to. If unspecified the system will fallback to a uniquely generated temp directory in the system temporary directory.
* Support added for `implname` attribute during shader generation. This is described in the specification, but not yet implemented. During OSL source generation when compiling the OSOs its possible input names such as `normal` may be modified to avoid clashes with reserved words. Adding `implname` support allows this to be described in the generated Implementation elements and respected during OSL Network generation. New `getPortName()` function is added in support for the implementation of this feature. This is also needs to be stored in two separate classes, the actual implementation can be either a `ShaderGraph` or a `ShaderNodeImpl`.
* New API method `ShaderGraph::flattenGraph()` will walk the graph and expand any `CompoundNode` entities discovered. Care is taken to generate unique names for newly generated nodes by prefixing the names with the name of the prior parent `CompoundNode`, and also to ensure all connections are retained.
* `ShaderGraph::create()` can now accept and populate the created graph directly from a NodeDef element, which is simpler than the current Node source, as there are no local modifications.
* `Syntax` base class now tracks custom registered syntaxes so they can be easily queried. This is necessary to support dynamic creation of a derived generator, specifically to allow the OSL Network generator to create the OSL Source generator for on-demand compilation of locally defined nodes.

Other supporting changes are outlined below.

### Cmake
* Add CMake option to control if NodeGraph implementations are compiled down to OSOs, or left as NodeGraphs.

### MaterialXFormat
* Add optional `recursive` argument to FilePath::createDirectory(), to allow for recursive directory creation if necessary.
* Add API to FilePath to return the system temporary directory, as well as create a new temporary directory. This is used for on-the-fly compilation of OSL source implementation to OSOs if defined locally in a document and not in a pre-compiled standard library.
* Add a new `createDirectories` option to XmlWriteOptions to allow for any necessary directories to be created while writing a new file.

### MaterialXTest/MaterialXRenderOsl
* Add new test specific implementations for tangent and bitangent nodes for the OSL Network generator. These may become obsolete if the OSL render testing moves to using the OBJ file (see XXXX).
* Update the OSL Network scene template to align the cameras after the OSL Source template was changed.
* Remove the OSL Network specific tests that were skipped - we now have complete one-to-one parity with OSL Source generator.
* Write fully expanded paths in to the template file for easier testing.
* Introduce string processing to the command string that needs to be embedded in a legal XML file to replace illegal XML characters `<` and `>`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant