Measure before you optimise

Every performance optimisation should start with a measurement. Optimising by intuition produces faster-looking code that runs slower, or fast code in paths that aren’t on the hot path.

The profiler tells you:

  • Where time is actually spent — often not where you think
  • Why it’s slow — cache misses, branch mispredictions, lock contention
  • Whether your fix helped — before/after comparison

Two tools cover most C++ performance work on Linux: perf for hardware performance counters and CPU-level analysis; valgrind and its tools for memory errors, cache simulation, and call graph profiling.

perf — Linux hardware performance counters

perf reads CPU Performance Monitoring Unit (PMU) counters: instructions retired, cache misses, branch mispredictions. It’s low overhead (< 1%) and doesn’t require instrumenting the binary.

Build for profiling

Always profile with debug info preserved:

1# CMake release build with symbols
2cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..
3make -j4
4
5# Or manually
6g++ -O2 -g -fno-omit-frame-pointer -o myapp src/main.cpp

-fno-omit-frame-pointer preserves the frame pointer chain — required for accurate call stack unwinding in perf record.

perf stat — quick overview

1perf stat ./myapp

Output:

 Performance counter stats for './myapp':

     3,421.45 msec task-clock           #    0.998 CPUs utilized
            12      context-switches     #    3.507 /sec
             3      cpu-migrations       #    0.877 /sec
           231      page-faults          #   67.514 /sec
 9,847,234,821      cycles               #    2.877 GHz
 7,203,441,208      instructions         #    0.73  insn per cycle
   542,112,034      cache-misses         #   14.22% of all cache refs
    38,023,112      branch-misses        #    2.31% of all branches

Key numbers:

  • insn per cycle (IPC): 0.73 is poor — 1.0+ is healthy for sequential code. Low IPC means the CPU is stalling, usually on memory.
  • cache-misses 14%: anything above ~2% on a hot workload is significant. This binary is memory-bound.
  • branch-misses 2.31%: moderate — investigate if it’s in the hot path.

perf record + report — find the hot functions

1perf record -g ./myapp     # -g = call graph (stack traces)
2perf report

perf report opens an interactive TUI. Navigate with arrows, press Enter to expand a function’s callers/callees.

Overhead  Command  Shared Object       Symbol
  38.21%  myapp    myapp               [.] processBuffer
  22.14%  myapp    myapp               [.] findNearest
  15.43%  myapp    libc.so.6           [.] memcpy
   8.22%  myapp    myapp               [.] SensorParser::parse

processBuffer takes 38% of CPU time. Expand it to see which callers reach it and which callees it spends time in.

perf annotate — source-level view

1perf annotate processBuffer

Shows source lines with their percentage of samples:

       │   for (size_t i = 0; i < count; ++i) {
 38.2  │       float v = data[i];               ← 38% of time here
  1.2  │       v = filter(v);
 52.1  │       output[i] = v * scale;           ← 52% here

If 90% of time is on memory loads (data[i], output[i]), the bottleneck is memory bandwidth or cache misses, not computation.

perf stat for specific events

1# Cache analysis
2perf stat -e cache-references,cache-misses,L1-dcache-misses ./myapp
3
4# Branch analysis
5perf stat -e branches,branch-misses ./myapp
6
7# Memory bandwidth (Intel)
8perf stat -e mem_inst_retired.all_loads,mem_inst_retired.all_stores ./myapp

valgrind — memory errors and simulated cache

valgrind runs your binary in an instrumented virtual machine. Overhead is significant (10–50×) but it catches errors that hardware profilers miss.

memcheck — memory errors

1valgrind --leak-check=full --track-origins=yes ./myapp

==12345== Invalid read of size 4
==12345==    at 0x401234: processBuffer (main.cpp:47)
==12345==    by 0x401500: main (main.cpp:112)
==12345==  Address 0x5204080 is 0 bytes after a block of size 64 alloc'd
==12345==    at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck.so)
==12345==    by 0x401100: Buffer::Buffer(int) (buffer.cpp:12)

This is an out-of-bounds read — reading 4 bytes at exactly the end of a 64-byte allocation. Line numbers are shown; follow the stack to find the bug.

Common memcheck findings:

  • Invalid read/write: buffer overflow, off-by-one
  • Use of uninitialised value: reading before writing (conditional jumps on uninitialised data)
  • Definitely lost / still reachable: memory leaks

cachegrind — cache simulation

1valgrind --tool=cachegrind ./myapp
2cg_annotate cachegrind.out.<pid>

Simulates L1/L2/L3 cache (configurable sizes) and reports hits/misses per source line. More detailed than perf stat cache events; works without root.

--------------------------------------------------------------------------------
          Ir   I1mr  ILmr          Dr     D1mr    DLmr          Dw  D1mw  DLmw
--------------------------------------------------------------------------------
 700,000    0     0  350,000  25,000  25,000   350,000       0     0

D1mr = L1 data cache miss reads. 25,000 misses on 350,000 reads = 7% miss rate on this function. If every miss is to DRAM, that’s ~10 µs in stalls.

callgrind — call graph + instruction counts

1valgrind --tool=callgrind ./myapp
2callgrind_annotate callgrind.out.<pid>
3# Or visualise with KCachegrind
4kcachegrind callgrind.out.<pid>

KCachegrind shows a call graph with instruction counts and cache misses per call edge. Useful for identifying which caller is responsible for a hot path.

AddressSanitizer — faster than valgrind for memory errors

For development builds, AddressSanitizer finds the same classes of bugs as memcheck but at 2× overhead instead of 50×:

1# Compile with ASan
2g++ -fsanitize=address -g -O1 -fno-omit-frame-pointer -o myapp src/main.cpp
3./myapp

==12345==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000090
READ of size 4 at 0x602000000090 thread T0
    #0 0x401234 in processBuffer main.cpp:47
    #1 0x401500 in main main.cpp:112

Use ASan in CI for every build. Use valgrind memcheck for production binaries (no recompile needed) or when ASan isn’t available.

Workflow

1. Build with -O2 -g -fno-omit-frame-pointer

2. perf stat — is it CPU-bound or memory-bound?
   IPC < 0.7 or cache-misses > 5% → memory bound → look at data layout
   IPC > 1.5, few cache misses → compute bound → look at hot functions

3. perf record -g + perf report — which function?
   Find the top 1-3 functions by overhead

4. perf annotate — which line?
   Find the specific instruction cluster

5. Identify the root cause:
   Memory load on every iteration → cache miss → restructure data (AoS→SoA)
   Branch inside loop → branch misprediction → remove branch or sort data
   Lock in hot path → contention → lock-free structure or reduce scope

6. Fix, rebuild, perf stat again — compare IPC and cache-miss rate

Common findings and fixes

perf finding Likely cause Fix
IPC < 0.5, cache-misses > 10% Random memory access pattern SoA layout, prefetch, pool allocator
IPC < 0.5, cache-misses low Long-latency instruction Reduce division, sqrt, unaligned loads
Branch-misses > 5% in hot loop Unpredictable branch Sort input, branchless arithmetic
memcpy/memmove at top Unnecessary copying std::move, span, reserve()
Lock/mutex in top 10% Thread contention Lock-free queue, reduce critical section
malloc/new in hot path Per-iteration allocation Pool allocator, arena, preallocate

Summary

  • Build with -O2 -g -fno-omit-frame-pointer for profiling
  • perf stat first: IPC and cache-miss rate tell you if you’re compute- or memory-bound
  • perf record -g + perf report: which function, which caller
  • perf annotate: which source line
  • valgrind --tool=cachegrind: cache miss simulation with source annotation
  • valgrind memcheck / AddressSanitizer: memory errors — run in CI, not in the profiling loop
  • Fix, measure again — never guess, always verify

What’s next