markdown-transclusion

Stream-based library and CLI for resolving Obsidian-style transclusion references in Markdown documents.

Overview

markdown-transclusion processes Markdown files containing transclusion syntax (![[filename]]) and resolves these references by including the content of referenced files. This enables modular documentation workflows where content can be composed from reusable components.

High-Level System Flow

graph TD
    A["CLI Input or API Call"] --> B["Transclusion Processor"]
    B --> C["Flattened Output"]
    B --> D[".errors[] / processedFiles[]"]
    
    style B fill:#fff3e0
    style C fill:#c8e6c9
    style D fill:#e3f2fd

Detailed Processing Example

graph LR
    A["📄 main.md<br/>![[header]]<br/>![[content]]"] --> B["markdown-transclusion"]
    C["📄 header.md"] --> B
    D["📄 content.md"] --> B
    B --> E["📄 output.md<br/>(fully resolved)"]
    
    style A fill:#f9f,stroke:#333,stroke-width:2px
    style E fill:#9f9,stroke:#333,stroke-width:2px

Stream-Based API Flow

graph LR
    subgraph "Input"
        I1["File Stream"] --> T["TransclusionTransform"]
        I2["String Input"] --> T
        I3["Process stdin"] --> T
    end
    
    subgraph "Processing"
        T --> P["Line-by-line processing"]
        P --> R["Resolve transclusions"]
        R --> P
    end
    
    subgraph "Output"
        R --> O1["Output Stream"]
        R --> O2["stream.errors[]"]
        R --> O3["result.processedFiles[]"]
    end
    
    style T fill:#e1f5fe
    style O1 fill:#c8e6c9
    style O2 fill:#ffccbc
    style O3 fill:#e3f2fd

Designed for the Universal Charter project's multilingual documentation pipeline, it provides reliable, stream-based processing suitable for CI/CD integration.

Features

✅ Recursive transclusion - Include files within files with automatic depth limiting
✅ Circular reference detection - Prevents infinite loops with clear error reporting ✅ Heading extraction - Include specific sections using ![[file#heading]] syntax
✅ Variable substitution - Dynamic file references with {{variable}} placeholders
✅ Stream processing - Memory-efficient processing of large documents
✅ Path resolution - Relative paths resolved from parent file context
✅ Security built-in - Path traversal protection and base directory enforcement
✅ CLI & API - Use as a command-line tool or Node.js library
✅ Error recovery - Graceful handling of missing files with inline error comments
✅ Zero dependencies - No runtime dependencies for security and simplicity

Installation

# Global CLI installation
npm install -g markdown-transclusion

# Local project installation
npm install markdown-transclusion

# Or use directly with npx
npx markdown-transclusion --help

Quick Start

CLI Usage

# Process a single file
markdown-transclusion input.md

# Output to file instead of stdout
markdown-transclusion input.md --output output.md

# Process with variables
markdown-transclusion template.md --variables "lang=es,version=2.0"

# Validate references without processing
markdown-transclusion docs/index.md --validate-only --strict

# Use from a different directory
markdown-transclusion README.md --base-path ./docs

# Pipe to other tools
markdown-transclusion input.md | pandoc -o output.pdf

# ⚠️ On Windows, use Git Bash or PowerShell. CMD can't handle the pipework.

Example

Given these files:

main.md:

# Documentation
![[intro]]
![[features#Overview]]
![[api/endpoints]]

intro.md:

Welcome to our project! This tool helps you create modular documentation.

features.md:

# Features

## Overview
Our tool supports transclusion, making documentation maintenance easier.

## Details
...

Running markdown-transclusion main.md produces:

# Documentation
Welcome to our project! This tool helps you create modular documentation.
## Overview
Our tool supports transclusion, making documentation maintenance easier.
![[api/endpoints]]
<!-- Error: File not found: api/endpoints -->

Programmatic Usage

import { processLine, createTransclusionStream } from 'markdown-transclusion';
import { createReadStream, createWriteStream } from 'fs';
import { pipeline } from 'stream/promises';

// Process a single line
const result = await processLine('Check the ![[api-guide]]', {
  basePath: './docs'
});
console.log(result.output);  // "Check the <content of api-guide.md>"

// Stream processing for large files
const stream = createTransclusionStream({
  basePath: './docs',
  variables: { version: '2.0' },
  maxDepth: 5
});

await pipeline(
  createReadStream('input.md'),
  stream,
  createWriteStream('output.md')
);

// Check for errors after processing
if (stream.errors.length > 0) {
  console.error('Transclusion errors:', stream.errors);
}

Transclusion Syntax

Basic Syntax

Syntax	Description	Example Output
`![[filename]]`	Include entire file	Contents of `filename.md`
`![[folder/file]]`	Include file from folder	Contents of `folder/file.md`
`![[file#heading]]`	Include specific section	Content under `# heading` until next heading
`![[file#What We Don't Talk About]]`	Include section with spaces	Content under heading with spaces
`![[file-{{var}}]]`	Variable substitution	With `var=en`: contents of `file-en.md`

Advanced Examples

<!-- Nested transclusion -->
![[chapter1]]  <!-- If chapter1.md contains ![[section1]], it will be included -->

<!-- Multiple variables -->
![[docs/{{lang}}/intro-{{version}}]]  <!-- Variables: lang=es, version=2 → docs/es/intro-2.md -->

<!-- Heading with spaces -->
![[architecture#System Overview]]

<!-- Error handling - missing file -->
![[missing-file]]
<!-- Error: File not found: missing-file -->

<!-- Circular reference protection -->
<!-- If A includes B, and B includes A, it will show: -->
![[/path/to/A.md]]
<!-- Error: Circular reference detected: /path/to/A.md → /path/to/B.md → /path/to/A.md -->

Try It Out

We include a complete example project:

# Clone the repository
git clone https://github.com/flyingrobots/markdown-transclusion.git
cd markdown-transclusion/examples/basic

# Run the example
npx markdown-transclusion main.md --variables "lang=en"

# Try different languages
npx markdown-transclusion main.md --variables "lang=es"

See examples/basic/README.md for a full walkthrough.

Real-World Use Cases

1. Multilingual Documentation

Maintain documentation in multiple languages without duplication:

flowchart LR
    A["template.md<br/>![[content-{{lang}}]]"] --> B{"Variable<br/>Substitution"}
    B -->|"lang=en"| C["![[content-en]]"]
    B -->|"lang=es"| D["![[content-es]]"]
    B -->|"lang=fr"| E["![[content-fr]]"]
    
    C --> F["content-en.md"]
    D --> G["content-es.md"]
    E --> H["content-fr.md"]
    
    style A fill:#e1f5fe
    style B fill:#fff9c4
    style F fill:#c8e6c9
    style G fill:#c8e6c9
    style H fill:#c8e6c9

# template.md contains: ![[content-{{lang}}]]
for lang in en es fr de zh; do
  markdown-transclusion template.md \
    --variables "lang=$lang" \
    --output docs/$lang/guide.md
done

2. Version-Specific Documentation

<!-- template.md -->
# API Documentation v{{version}}

![[changelog-{{version}}]]
![[api/endpoints-{{version}}]]
![[migration-guide-{{prev_version}}-to-{{version}}]]

3. Modular Course Content

<!-- course.md -->
# JavaScript Course

![[modules/intro]]
![[modules/basics#Variables and Types]]
![[modules/functions]]
![[exercises/week-1]]

4. Configuration Documentation

<!-- config-guide.md -->
# Configuration Guide

## Development Settings
![[configs/development]]

## Production Settings  
![[configs/production]]

## Common Issues
![[troubleshooting#Configuration Errors]]

5. CI/CD Integration

# .github/workflows/docs.yml
name: Build Documentation
on:
  push:
    branches: [main]
    
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Build docs
        run: |
          npm install -g markdown-transclusion
          markdown-transclusion docs/index.md \
            --variables "version=${{ github.ref_name }}" \
            --strict \
            --output dist/documentation.md
            
      - name: Validate links
        run: |
          markdown-transclusion docs/index.md \
            --validate-only \
            --strict

API Reference

Core Functions

`transclude(input, options)`

Process a complete string, replacing all transclusion references.

const result = await transclude('# Doc\n![[intro]]\n![[conclusion]]', {
  basePath: './docs'
});
// result.content: "# Doc\n<contents of intro>\n<contents of conclusion>"
// result.errors: Array of any errors
// result.processedFiles: Array of processed file paths

`transcludeFile(filePath, options)`

Process a file, replacing all transclusion references.

const result = await transcludeFile('./README.md', {
  variables: { version: '2.0' }
});
// result.content: Full processed content
// result.errors: Array of any errors
// result.processedFiles: Array of all processed files

`processLine(line, options)`

Process a single line of text for transclusions.

const result = await processLine('See ![[notes]]', {
  basePath: './docs',
  extensions: ['md', 'txt']
});
// result.output: "See <contents of notes.md>"
// result.errors: Array of any errors

`createTransclusionStream(options)`

Create a transform stream for processing large files.

const stream = createTransclusionStream({
  basePath: './docs',
  variables: { env: 'prod' },
  maxDepth: 5
});

// Access accumulated errors after processing
stream.on('finish', () => {
  if (stream.errors.length > 0) {
    console.error(`Found ${stream.errors.length} errors:`);
    stream.errors.forEach(err => {
      console.error(`- [${err.path}] ${err.message}`);
    });
  }
});

Options

Option	Type	Default	Description
`basePath`	string	cwd	Base directory for resolving references
`extensions`	string[]	['md', 'markdown']	File extensions to try
`variables`	object	{}	Variables for substitution
`maxDepth`	number	10	Maximum recursion depth
`strict`	boolean	false	Exit on errors
`validateOnly`	boolean	false	Only validate, don't output
`stripFrontmatter`	boolean	false	Strip YAML/TOML frontmatter
`cache`	FileCache	none	Optional file cache

Error Codes

String error codes used in transclusion processing:

FILE_NOT_FOUND - Referenced file doesn't exist
CIRCULAR_REFERENCE - Circular inclusion detected
MAX_DEPTH_EXCEEDED - Too many nested includes
READ_ERROR - File read failure
HEADING_NOT_FOUND - Specified heading not found in file

Numeric error codes used for security violations:

1001 - NULL_BYTE - Null byte in path
1002 - PATH_TRAVERSAL - Path traversal attempt (..)
1003 - ABSOLUTE_PATH - Absolute path not allowed
1004 - UNC_PATH - UNC path not allowed
1005 - OUTSIDE_BASE - Path outside base directory

See docs/api.md for complete API documentation.

CLI Reference

markdown-transclusion --help

Key options:

-o, --output - Output file (default: stdout)
-b, --base-path - Base directory for references
--variables - Variable substitutions (key=value)
-s, --strict - Exit on any error
--validate-only - Check references without output
--strip-frontmatter - Remove YAML/TOML frontmatter from files
--log-level - Set verbosity (ERROR/WARN/INFO/DEBUG)

Security

Built-in protection against:

Path traversal - ../../../etc/passwd → rejected
Absolute paths - /etc/passwd → rejected
Null bytes - file\x00.md → rejected
Symbolic links - Resolved within base directory

All file access is restricted to the configured base path.

Documentation

📖 API Reference - Complete API documentation
🛠️ Contributing Guide - Development setup and guidelines
🏗️ Technical Design - Architecture and design decisions
📦 Example Project - Working example with all features
📝 CHANGELOG - Version history and migration notes

Contributing

We welcome contributions! Please see our Contributing Guide for details on:

Setting up the development environment
Running tests and linting
Submitting pull requests
Adding new features

Support

🐛 Report bugs
💡 Request features
📚 Read the docs
⭐ Star the project on GitHub!

Performance

Stream processing - Constant memory usage regardless of file size
Lazy evaluation - Files are read only when needed
Efficient parsing - Single-pass line processing
Optional caching - Reduce file system calls for repeated includes

Benchmarks on a MacBook Pro M1:

1MB file with 50 transclusions: ~15ms
10MB file with 500 transclusions: ~120ms
Memory usage: ~5MB constant

Note: Performance measurements taken using Node.js 18.18.0 with warm file system cache. Actual performance may vary based on disk speed, file system, and transclusion depth.

Comparison with Alternatives

Feature	markdown-transclusion	mdbook	pandoc-include
Obsidian syntax	✅	❌	❌
Streaming	✅	❌	❌
Recursive includes	✅	✅	⚠️
Circular detection	✅	❌	❌
Variables	✅	⚠️	❌
Heading extraction	✅	❌	❌
Zero dependencies	✅	❌	❌

License

MIT License

See LICENSE for details.

Name		Name	Last commit message	Last commit date
Latest commit History 134 Commits
.github		.github
.vscode		.vscode
docs		docs
examples/basic		examples/basic
src		src
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
eslint.config.js		eslint.config.js
jest.config.js		jest.config.js
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json

License

flyingrobots/markdown-transclusion

Folders and files

Latest commit

History

Repository files navigation

markdown-transclusion

Overview

High-Level System Flow

Detailed Processing Example

Stream-Based API Flow

Features

Installation

Quick Start

CLI Usage

Example

Programmatic Usage

Transclusion Syntax

Basic Syntax

Advanced Examples

Try It Out

Real-World Use Cases

1. Multilingual Documentation

2. Version-Specific Documentation

3. Modular Course Content

4. Configuration Documentation

5. CI/CD Integration

API Reference

Core Functions

transclude(input, options)

transcludeFile(filePath, options)

processLine(line, options)

createTransclusionStream(options)

Options

Error Codes

CLI Reference

Security

Documentation

Contributing

Support

Performance

Comparison with Alternatives

License

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 2

Packages 0

Uh oh!

Languages

`transclude(input, options)`

`transcludeFile(filePath, options)`

`processLine(line, options)`

`createTransclusionStream(options)`

Packages