programanalysis.md

Program Analysis

Program analysis is the art of learning how a program really behaves.

Sometimes that means reading source code carefully. Sometimes it means watching a process as it runs. Sometimes it means instrumenting a binary and collecting facts that would never be visible from source alone. In every case, the central habit is the same: move from guesswork to evidence.

In the context of this project, program analysis matters because it helps answer questions such as:

• What code paths run most often?
• Which functions call which other functions?
• Where does memory come from?
• Which syscalls are used?
• Which branches, instructions, or data accesses dominate execution?
• Is the slowdown caused by the target program or by the workload driver around it?

Taken as a subject, program analysis is broad. It helps to divide it into a few main areas and understand what each one contributes.

Static Analysis

Static analysis studies a program without running it. It is the quiet, reflective side of the subject: the part where you inspect structure, relationships, and possible failures before execution ever begins.

Typical goals:

• find bugs and undefined behavior
• inspect control flow and call relationships
• reason about types, lifetimes, and data flow
• catch issues before runtime

Common tools:

• compiler warnings: gcc, clang
• static analyzers: clang-tidy, cppcheck
• sanitizers at build time: -fsanitize=address,undefined,thread
• source browsing and code search: rg, IDE indexers

Dynamic Analysis

Dynamic analysis studies a program while it is running. This is where the subject becomes concrete, because the program stops being a static artifact and becomes a living process with timings, memory use, system calls, and observable behavior.

Typical goals:

• observe real execution behavior
• inspect function calls and timings
• trace memory usage
• trace system interaction
• collect counters and samples

Common tools:

• valgrind
• perf
• strace
• ltrace
• gdb
• bpftrace
• DynamoRIO

Binary Instrumentation

Binary instrumentation means inserting observation logic into a compiled program at runtime or rewrite time. It is one of the deeper parts of the field, and it becomes powerful when the easier forms of observation are no longer enough.

Typical goals:

• trace instructions
• trace memory references
• collect code coverage
• build custom analysis clients
• analyze behavior without recompiling the target

Common tools:

• DynamoRIO
• Intel PIN
• Valgrind instrumentation tools

The easiest way to keep the tooling straight is to group it by the kind of question it helps answer.

Source-Level Analysis

• rg: fast code search
• clang-tidy: style and correctness analysis
• cppcheck: static bug finding
• compiler warnings: first line of defense

Execution Tracing

• strace: syscalls, file access, process activity, futexes, mmap, I/O
• ltrace: dynamic library calls
• bpftrace: kernel and userspace dynamic tracing

Debugging And Inspection

• gdb: breakpoints, stacks, memory, threads, disassembly
• objdump: assembly inspection
• nm: symbols
• readelf: ELF and section inspection

Dynamic Cost Analysis

• valgrind --tool=callgrind: function cost and call graph
• valgrind --tool=massif: heap growth
• heaptrack: heap allocation profiling
• perf record: sampled hotspots

Binary Instrumentation

• DynamoRIO: dynamic instrumentation framework
• drcachesim: cache simulation on top of DynamoRIO
• drmemtrace: memory trace workflows

What DynamoRIO Is Good For

DynamoRIO is a dynamic binary instrumentation framework.

It is useful when you want to:

• inspect instructions at runtime
• instrument memory accesses
• collect custom traces
• study control flow
• build your own runtime analysis tools

It is more specialized than perf or strace, and that is exactly why it is so useful later. Usually you reach for it only after simpler tools have already narrowed the mystery and you need a deeper, more custom view of runtime behavior.

Program Analysis Workflow

For a systems project, a good working order usually looks like this:

1. Read the source and identify likely hot paths.
2. Use rg and code navigation to understand structure.
3. Run the target and validate behavior.
4. Use strace if you suspect syscall or file-I/O overhead.
5. Use perf or callgrind to find expensive code paths.
6. Use heaptrack or massif if memory behavior looks suspicious.
7. Use gdb for correctness debugging or state inspection.
8. Use DynamoRIO only when you need custom binary instrumentation.

Subject Roadmap

This subject becomes much easier when it is learned in the order that its ideas depend on one another.

The temptation is always to start with the most impressive tool. Resist that. The strong path begins with the shape of a program, then moves to the shape of a process, then to observation, then to deeper tracing, and finally to advanced instrumentation.

Stage 1: Program Structure And Build Awareness

Learn first:

• how compiled programs are built
• source files, object files, libraries, executables
• symbols, debug info, and build modes
• compiler warnings and basic sanitizers
• how to search and navigate a codebase

Why this comes first:

• all later analysis depends on understanding what you are analyzing
• symbol names, debug info, and binaries appear in almost every tool

Tools:

• gcc, clang
• rg
• nm
• objdump
• readelf

What to practice:

• build a small C or C++ program in debug and release mode
• inspect symbols in the binary
• inspect the executable with readelf

Stage 2: Runtime Model Of A Process

Learn next:

• process, thread, address space, stack, heap
• system calls
• dynamic linking
• memory mapping
• file descriptors

Why this comes second:

• tracing tools make much more sense once you know what the OS sees

Tools:

• strace
• ltrace
• gdb

What to practice:

• run a CLI tool with strace -c
• observe open, read, write, mmap, and futex
• inspect stack frames and memory in gdb

Stage 3: Control Flow And Execution Inspection

Learn next:

• call stacks
• function boundaries
• stepping through execution
• breakpoints and watchpoints
• source to assembly relationship

Why this comes before advanced profiling:

• hotspot reports are much easier to understand if you already know how calls and stacks work

Tools:

• gdb
• objdump
• disassembly view in IDEs or debuggers

What to practice:

• step through a function in gdb
• inspect a backtrace
• compare a function's source with its disassembly

Stage 4: Sample-Based Dynamic Analysis

Learn next:

• statistical profiling
• hotspot discovery
• sampled stacks
• inclusive versus self cost at a high level

Why this stage matters:

• this is usually the first fast way to see where execution spends time

Tools:

• perf record
• perf report
• FlameGraph
• AMDuProfCLI

What to practice:

• profile one CLI program
• identify top hotspots
• generate and read a flame graph

Stage 5: Instrumentation-Based Program Analysis

Learn next:

• exact or more detailed cost models
• call graphs
• instrumentation overhead
• tradeoffs between precision and realism

Why this comes after sampling:

• instrumentation tools are slower and more specialized
• they are easier to use well once sampling has already narrowed the question

Tools:

• valgrind --tool=callgrind
• valgrind --tool=massif
• heaptrack

What to practice:

• compare perf and callgrind on the same workload
• inspect inclusive cost versus self cost
• analyze heap growth and allocation sources

Stage 6: Data, Memory, And OS Interaction Analysis

Learn next:

• allocation sources
• heap growth
• locality
• lock waits
• syscall-heavy behavior
• user space versus kernel space interaction

Why this matters:

• many important system-software issues are not just "hot function" issues

Tools:

• heaptrack
• massif
• strace
• perf stat
• bpftrace

What to practice:

• inspect allocation-heavy code
• compare syscall-heavy and compute-heavy workloads
• trace latency or wait behavior

Stage 7: Binary And Runtime Instrumentation

Learn next:

• binary instrumentation concepts
• instruction tracing
• memory reference tracing
• custom analysis clients

Why this comes late:

• it is more specialized and benefits from strong foundations in earlier stages

Tools:

• DynamoRIO
• DynamoRIO clients such as drcachesim

What to practice:

• understand what a binary instrumentation framework does
• run or study an existing DynamoRIO client
• compare sampling results with binary-instrumentation results

Stage 8: Advanced Dynamic Observability

Learn last:

• eBPF tracing patterns
• live observability
• latency histograms
• dynamic probes in kernel and user space
• cross-layer analysis

Why this is a late stage:

• it combines OS knowledge, runtime behavior, and measurement design

Tools:

• bpftrace
• advanced perf
• AMD uProf advanced modes

What to practice:

• trace a latency distribution
• correlate syscall behavior with sampled hotspots
• build a workflow that combines multiple tools

Text Graph

Program Analysis Roadmap

Stage 1  Program structure and binaries
   |
   v
Stage 2  Process model, syscalls, runtime environment
   |
   v
Stage 3  Control flow, call stacks, debugging, source <-> assembly
   |
   +-----------------------------+
   |                             |
   v                             v
Stage 4  Sample-based analysis   Stage 5  Instrumentation-based analysis
   |                             |
   +-------------+---------------+
                 |
                 v
Stage 6  Memory, data movement, OS interaction analysis
                 |
                 v
Stage 7  Binary instrumentation and custom runtime analysis
                 |
                 v
Stage 8  Advanced observability and cross-layer tracing

End-To-End Learning Order

If you want one clean path from start to finish, use this:

1. Build process, debug symbols, executable layout
2. Process model, stack, heap, threads, syscalls
3. Code navigation and source reading
4. gdb and backtraces
5. strace
6. perf record and perf report
7. Flame graphs
8. callgrind
9. massif and heaptrack
10. perf stat
11. bpftrace
12. DynamoRIO
13. custom instrumentation and combined workflows

Good Mental Model

At its heart, program analysis is about answering:

• what ran,
• how it ran,
• what it touched,
• what it asked the OS to do,
• and where the cost or bug actually came from.

It is broader than performance engineering, but it supports performance work at almost every step. Before you can improve a system, you have to learn how to see it.