trace: add CTF to Perfetto protobuf conversion tools

[adamgreloch]

JIRA: RTOS-1061

Description

Motivation and Context

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Chore (refactoring, style fixes, git/CI config, submodule management, no code logic changes)

How Has This Been Tested?

• Already covered by automatic testing.
• New test added: (add PR link here).
• Tested by hand on: ia32-generic-qemu

Checklist:

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have added tests to cover my changes.
• All new and existing linter checks and tests passed.
• My changes generate no new compilation warnings for any of the targets.

Special treatment

• This PR needs additional PRs to work (list the PRs, preferably in merge-order).
  perf: adapt to new perf subsystem phoenix-rtos-utils#238
• I will merge this PR by myself when appropriate.

[gemini-code-assist]

Hello @adamgreloch, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

Summary of Changes

Hello! Gemini here, providing a summary of this pull request. This PR introduces new tools to convert Common Trace Format (CTF) traces into formats compatible with modern tracing visualization tools, specifically the Chrome Trace Event Format (JSON) and the Perfetto trace protobuf format. This conversion is achieved using the babeltrace2 library to read the CTF traces and then mapping the events to the respective output formats. This will significantly improve the ability to analyze and visualize trace data generated by the system.

Highlights

• CTF to JSON Conversion: A new Python script (trace/json/ctf_to_json.py) is added to convert CTF traces into a JSON format that is compatible with tools like Chrome's chrome://tracing. It parses CTF events, identifies synthetic events (like syscalls, scheduling, locking), and outputs them as JSON objects with appropriate phases (BEGIN, END, INSTANT) and metadata (process/thread IDs, timestamps, arguments).
• CTF to Perfetto Protobuf Conversion: A second new Python script (trace/protobuf/src/ctf_to_proto.py) is introduced to convert CTF traces into the Perfetto trace protobuf format. This script utilizes the Perfetto protobuf definition (downloaded and compiled by a helper script) and maps CTF events to Perfetto's TracePacket and TrackEvent structures, including managing track descriptors for processes, threads, and CPUs.
• Perfetto Protobuf Setup Script: A shell script (trace/protobuf/proto_gen.sh) is added to automate the setup required for the Perfetto protobuf conversion tool. This script downloads the necessary perfetto_trace.proto file from the Perfetto repository and compiles it into Python code using protoc.

Changelog

Click here to see the changelog

• trace/json/ctf_to_json.py
  • Added new Python script for converting CTF traces to JSON (Chrome Trace Event Format).
  • Implements parsing of CTF events using babeltrace2.
  • Defines mappings for Phoenix-RTOS syscalls and synthetic events.
  • Generates JSON output with event phases (BEGIN, END, INSTANT) and associated metadata.
• trace/protobuf/proto_gen.sh
  • Added new shell script to download the Perfetto trace protobuf definition.
  • Compiles the .proto file into Python code using protoc.
• trace/protobuf/src/ctf_to_proto.py
  • Added new Python script for converting CTF traces to Perfetto trace protobuf format.
  • Uses babeltrace2 to read CTF and the generated Perfetto protobuf classes.
  • Maps CTF events to Perfetto TracePacket and TrackEvent structures.
  • Manages track descriptors for processes, threads, and CPUs.
  • Outputs serialized Perfetto trace data to a binary file (out.pftrace).

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

---

Traces flow like streams,
Events captured, kernel dreams.
JSON, Proto bright,
Visualize the system's light,
Performance takes its flight.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces two new Python scripts for converting CTF (Common Trace Format) traces: one to JSON (ETF-like format) and another to Perfetto protobuf format. A shell script for generating Python protobuf stubs is also included. The initiative to support these formats is valuable for trace analysis.

The scripts are generally well-structured and follow a similar pattern for processing events. However, there are a few areas that could be improved for robustness and usability, particularly concerning metadata handling, potential KeyErrors, and I/O efficiency in the protobuf converter.

Please also ensure the PR description (JIRA link, motivation, testing, checklist) is filled out to provide complete context for the changes.

Summary of Findings

• Metadata Handling Fragility: Both ctf_to_json.py and ctf_to_proto.py rely on the first processed event containing a specific ["threads"] array in its stream context for initial metadata. This could be fragile if the first event lacks this context, potentially leading to runtime errors or incomplete traces.
• Potential KeyError for TID 0: Both scripts use tid_or_zero which can return 0. If TID 0 is not explicitly part of the initial metadata or created via a thread_create event, subsequent lookups in dictionaries keyed by tid (e.g., tid_to_pid, tid_to_events_track_uid) will cause a KeyError.
• Inefficient Protobuf Serialization: ctf_to_proto.py repeatedly creates, serializes, and writes Trace objects containing very few packets in update_running_thread. Batching packets for a single serialization at the end would likely be more performant.
• Hardcoded Output Filename: ctf_to_proto.py uses a hardcoded output filename (out.pftrace), which reduces flexibility. Making this a command-line argument is recommended.
• PR Description: The pull request description template (JIRA, motivation, testing, etc.) should be filled out to provide full context for the changes.

Merge Readiness

The pull request introduces useful conversion tools. However, there are several medium-severity issues related to robustness, efficiency, and usability that should be addressed before merging. Specifically, improving metadata loading, handling potential KeyErrors for TIDs, optimizing protobuf serialization, and allowing a configurable output path for the protobuf script would significantly enhance the quality of these tools. I am unable to approve the pull request myself; please ensure these changes are reviewed and approved by others before merging once the suggested changes are made.