FAQ

General Questions

What is CDTk?

CDTk (Compiler Description Toolkit) is a modern, strongly-typed .NET library for building compilers, transpilers, and code generators. It provides a unified, declarative pipeline with a focus on safety, performance, and extensibility. Core components include:

1. TokenSet: Defines lexical patterns (tokens).
2. RuleSet: Structures grammar rules and creates ASTs.
3. MapSet: Generates target output or code from ASTs.
4. Model System: Adds semantic processing and intermediate transformations.

Starting with Version 8, CDTk introduces:

• Field-Based Identity: Use variable-bound references for tokens, rules, and mappings.
• Model System: Enable advanced semantic transformations across the pipeline.
• Validation System: Validate grammars and mappings before runtime.

---

Who should use CDTk?

CDTk is ideal for:

• Domain-Specific Languages (DSLs): Simplify language creation for niche tasks.
• Code Generators and Transpilers: Convert code between formats (e.g., SQL to JSON).
• General Compiler Development: Create robust and scalable compilers with minimal boilerplate.
• Teaching Compiler Construction: Perfect for academics and tutorials due to clarity and strong typing.
• Data Transformers: Convert structured data (e.g., CSV to JSON) with parsing pipelines.

---

What .NET versions are supported?

CDTk targets .NET 7 and above. Using upgraded C# features, the library ensures strong support for modern APIs, generics, and memory-safe code standards.

---

What's new in CDTk v8?

Key improvements in v8 include:

• Field-Based Grammar: Rules and tokens are defined as fields in inheritable classes, simplifying rule identity and eliminating generics.
• Reference Identity: All elements are resolved via object references, not string-based lookups.
• Model System: Introduce intermediate transformation layers between ASTs and mappings.
• Validation: Add declarative .Validate() calls to ensure grammar correctness during pre-compilation.
• Multi-Target Compilation: Compile once and generate outputs across multiple formats (e.g., Python, C, JavaScript).

---

Getting Started

How do I create my first compiler?

Here’s a minimal example using CDTk v8:

// Define TokenSet
class Tokens : TokenSet
{
    public Token Number = @"\d+";
    public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";
    public Token Equals = @"=";
}

// Define RuleSet
class Rules : RuleSet
{
    public Rule AssignmentNode = new Rule("variable:@Identifier '=' value:@Number")
        .Returns("variable", "value");
}

// Define MapSet
class Mappings : MapSet
{
    public Map AssignmentNode = "let {variable} = {value};";
}

// Build and compile
var compiler = new Compiler()
    .WithTokens(new Tokens())
    .WithRules(new Rules())
    .WithTarget(new Mappings())
    .Build();

var result = compiler.Compile("x = 42;");
Console.WriteLine(result.Output);  // Outputs: let x = 42;

---

Where can I find examples?

Visit Examples.md to explore:

• Calculator compilers
• Multi-language transpilers
• Simple JSON parsers
• Expression evaluators
• Mini-programming languages

---

Tokens

My tokens aren't matching correctly

1. Check token order: CDTk processes tokens sequentially. Place more specific patterns before generic ones.

// Incorrect
public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // Too generic
public Token Keyword = @"if|else|while";              // Won't match

// Correct
public Token Keyword = @"if|else|while";              // Keywords first
public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // Identifier later

2. Use tools like Regex101 to validate patterns.

---

How do I ignore whitespace?

Add .Ignore() to tokens:

public Token Whitespace = new Token(@"\s+").Ignore();

Ignored tokens are matched but excluded from downstream processing.

---

How do I handle multi-line comments?

Use RegexOptions.Singleline:

public Token BlockComment = new Token(@"/\*.*?\*/", RegexOptions.Singleline).Ignore();

---

How do I match escaped characters in strings?

Define a regex for escape sequences:

public Token StringLiteral = @"""([^""\\]|\\.)*""";

This matches strings with escaped characters like \" and \n.

---

Rules

My rule isn't matching

1. Confirm that tokens use @ in rule definitions:

   // Incorrect (missing `@`)
   new Rule("Identifier '=' Number");
   
   // Correct (reference tokens with `@`)
   new Rule("variable:@Identifier '=' value:@Number");

2. Check the precedence of alternations and ensure proper grouping with parentheses:

   new Rule("Factor", "@Number | '(' Expression ')'");

---

How do I handle optional elements?

Use ? for optional components:

public Rule VariableDecl = new Rule("@Type @Identifier ('=' Expression)?")
    .Returns("type", "name", "init");

---

How do I parse lists?

Use * (zero or more) or + (one or more repetitions):

public Rule ArgList = new Rule("Expression (',' Expression)*").Returns("args");
public Rule StatementBlock = new Rule("'{' Statement* '}'").Returns("statements");

---

Models

What is a model?

Models are lazy semantic processors that work between AST generation and mappings. They enable custom, intermediate processing for validations, transformations, and enhanced mappings.

---

How do I use a model?

Define a class extending Model:

class MyModel : Model
{
    private readonly __Ast _ast;
    public MyModel(__Ast ast)
    {
        _ast = ast;
    }

    public override object Build(object input)
    {
        if (input is AstNode node)
        {
            return new { TransformedNode = node.GetString("variable").ToUpper() };
        }
        return input;
    }
}

// Adding to MapSet
class Mappings : MapSet
{
    public MyModel Model => new MyModel(__Ast!);
}

---

Use cases for models:

• Sanitizing AST nodes: Clean variable declarations or correct invalid identifiers.
• Adding metadata: Insert semantic data for specialized mappings.

---

Code Generation

My mappings aren't generating output

1. Ensure node types match between .Returns() and mapping templates:

   public Map AssignmentNode = "let {variable} = {value};";

2. Validate placeholders: Ensure {fieldName} matches the field declarations in .Returns().

---

Can I use conditional fields in mappings?

Yes, incorporate custom models or rules:

class DynamicMappings : MapSet
{
    public Map ValidatedOutput = "let {variable}[opt |= {sanitizedValue}];";
}