Debugging

Debugging in CDTk

Debugging is an essential part of building compilers and transpilers in CDTk. Whether you're identifying errors in tokenization, rule validation, mapping, or AST transformations, CDTk v8 strengthens your debugging toolkit with built-in validation, diagnostics, and model-based insights.

This guide covers debugging strategies for each part of the CDTk pipeline: tokenization, parsing, mapping, validation, and semantic transformations.

---

Overview of Debugging in CDTk

CDTk makes it easy to debug each stage of the compilation pipeline. The pipeline consists of the following stages:

1. Tokenization – Converts input text into tokens using the TokenSet.
2. Parsing – Organizes tokens into an Abstract Syntax Tree (AST) based on the RuleSet.
3. Validation – Checks grammar rules, token mappings, and models for correctness.
4. Mapping – Transforms the AST into target output using the MapSet.
5. Semantic Transformations (optional) – Enrich the pipeline with custom model-based processing.

When debugging, focus on the specific stage where the issue occurs:

• If the input isn't tokenized as expected, debug tokenization.
• If parsing fails or produces incorrect AST nodes, debug grammar and rules.
• If mappings produce incorrect output, debug mapping templates and validate their consistency.
• If model adjustments are incorrect, debug models for processing errors.

---

Start Here

The best way to being debugging your compiler is by setting the WithDebugging flag to true as demonstrated below:

var compiler = new Compiler()
    .WithTokens(new CSharpTokens())
    .WithRules(new CSharpRules())
    .WithDebugger(true) // True for if you want debug logs false for you do not.
    .Build();

---

Using Diagnostics

The Diagnostics system tracks and organizes issues across the pipeline into three levels:

• Info: For informational messages or additional context.
• Warning: For recoverable issues that might result in undesired outcomes.
• Error: Critical issues that stop the pipeline.

Adding Diagnostics

Diagnostics can be utilized in tokenization, validation, or semantic processing:

diagnostics.Add(
    Stage.LexicalAnalysis,
    DiagnosticLevel.Error,
    "Unexpected token identified.",
    new SourceSpan(10, 1, 1, 10)
);

---

Debugging Tokenization

If your input text doesn’t produce the tokens you expect:

• Validate regex patterns in your TokenSet.
• Check token order, as CDTk uses the first matching token.

Steps to Debug Tokenization

1. Test Token Patterns

Use a tool like Regex101 to validate regex patterns. Example:

var pattern = @"[A-Za-z_][A-Za-z0-9_]*";
var match = Regex.Match("myVariable", pattern);

if (match.Success)
{
    Console.WriteLine($"Matched: {match.Value}");
}

2. Log Tokens

Inspect tokens directly after tokenization:

var tokens = lexer.Tokenize(inputCode, diagnostics);
foreach (var token in tokens)
{
    Console.WriteLine($"Token: {token.Type}, Lexeme: {token.Lexeme}");
}

3. Reorder Token Definitions

Place more specific token patterns before generic ones:

public Token Keyword = @"if|else|while";   // Specific tokens come first
public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";  // General tokens last

---

Debugging Grammar and Rules

Validating Rules

Use the Validate() method to check rules for logical and semantic correctness:

var diagnostics = rules.Validate();

if (diagnostics.HasErrors)
{
    foreach (var diag in diagnostics.Items)
    {
        Console.WriteLine($"{diag.Level}: {diag.Message}");
    }
}

Common Validation Errors:

• Undefined token references (e.g., @TokenName not defined in TokenSet).
• Invalid rule nesting or recursion.

---

Debugging Rule Outputs

Visualize rule outputs by inspecting the parsed AST. Example:

void PrintAst(AstNode node, int depth = 0)
{
    Console.WriteLine(new string(' ', depth * 2) + node.Type);
    foreach (var field in node.Fields)
    {
        if (field.Value is AstNode childNode)
        {
            PrintAst(childNode, depth + 1);
        }
        else
        {
            Console.WriteLine(new string(' ', (depth + 1) * 2) + $"{field.Key}: {field.Value}");
        }
    }
}

Use PrintAst(result.Ast) to verify AST structure.

---

Debugging Mapping Templates

Template Validation

Ensure all node types in your AST are properly mapped in the MapSet. Use .Validate() to catch missing mappings:

var diagnostics = mappings.Validate();

if (diagnostics.HasErrors)
{
    foreach (var diag in diagnostics.Items)
    {
        Console.WriteLine($"{diag.Level}: {diag.Message}");
    }
}

Fallback Mapping

Define a fallback template to handle unmapped nodes:

new Map("UnknownNode", "// TODO: Handle {type}");

Log Placeholders

When placeholders fail, log them dynamically to verify their values:

Console.WriteLine(node.GetString("left"));
Console.WriteLine(node.GetString("op"));
Console.WriteLine(node.GetString("right"));

---

Debugging Models

Models allow for custom, semantic processing in mappings. Debug a model by logging its inputs and transformations.

Example: Debugging a Custom Model

class MyModel : Model
{
    public override object Build(object input)
    {
        Console.WriteLine($"Processing Input: {input}");
        if (input is AstNode node)
        {
            var transformed = new { Value = node.GetString("name") };
            Console.WriteLine($"Transformed: {transformed}");
            return transformed;
        }
        return input;
    }
}

---

Accessing and Inspecting the AST

Inspect the AST before mapping to ensure field mappings and node types are correct:

if (result.Ast != null)
{
    PrintAst(result.Ast);
}

---

Common Issues and Their Fixes

Issue	Resolution
Token doesn't match input	Check regex patterns and token order. Validate token overlaps.
Parsing fails unexpectedly	Validate rules and token references. Check alternation precedence.
Node was skipped during mapping	Ensure a mapping exists or define a fallback.
Placeholder is invalid	Verify field names in .Returns() match placeholders in the template.
Errors when using a Model	Ensure the model's constructor accepts valid parameters (e.g., __Ast).

---

Best Practices for Debugging

1. Diagnose Early: Use .Validate() to catch issues before compilation.
2. Log All Stages: Print generated tokens, rules, ASTs, and models for end-to-end debugging.
3. Fallbacks Are Critical: Protect parsing and mapping stages with fallback rules and templates.
4. Tightly Couple Rules and Tokens: Ensure that token references in rules align with your TokenSet.

---

Next Steps

Now that you’re equipped to debug:

• Learn about defining tokens: Tokens
• Deepen your understanding of grammar rules: Rules
• Discover advanced mapping techniques: Mapping
• Explore compilers in action: Examples