Quick Hardware Performance Counters on macOS ARM64

If you’ve ever profiled OCaml programs on Linux, you’ve probably reached for perf stat. It’s the go-to tool for grabbing hardware performance counters—cycles, instructions, cache misses—without any instrumentation overhead. On macOS, the equivalent story has been open Instruments, which is fine for GUI-driven investigation but terrible for automated benchmarking pipelines.

I wanted something I could stick in a shell script, get output as JSON, and run in the terminal. So I put together mperf, a perf stat-like CLI for Apple Silicon. Here is what it looks like:

$ sudo ./mperf-stat -e cycles -e instructions -e l1d-tlb-misses -- ./my_benchmark

 Performance counter stats:

         1,234,567,890  cycles
         2,345,678,901  instructions                # 1.90 IPC
            12,345,678  l1d-tlb-misses

       0.543210 seconds wall time
       0.520000 seconds user
       0.020000 seconds sys

Why Not Just Use Instruments?

Instruments is powerful, but it’s an interactive GUI tool. You can invoke it from the command line using xctrace but the results need the same GUI tool to view them. Sometimes you just need a simple cli tool that prints out the most interesting stats, in my case I want it invoked from a Makefile or a CI runner.

There’s also no good reason this should require full Xcode and Instruments. The hardware counters are right there in the CPU; the kernel exposes them through private frameworks. The only real requirement is root access, no need for disabling SIP or code signing or other special entitlements.

Using Apple’s Private Frameworks

Apple Silicon exposes hardware performance counters through two private frameworks: kperf.framework and kperfdata.framework, living under /System/Library/PrivateFrameworks/. These are the same frameworks that Instruments uses internally. They’re undocumented, but ibireme’s kpc_demo showed that you can load them at runtime with dlsym and drive them from userspace.

The CPU-specific event databases live in /usr/share/kpep/ as plist files—a14.plist for M1, a15.plist for M2, as4.plist for M4, and so on. mperf provides portable aliases (cycles, instructions, branch-misses, l1d-cache-misses, etc.) that resolve to the right event names for whatever chip you’re running on. You can also pass raw event names if you want something specific.

Apple Silicon gives you 2 fixed counters (cycles and instructions) plus 8 configurable counters, for a maximum of 10 simultaneous events. Unlike Linux perf, mperf doesn’t do multiplexing — if you ask for more than 10 events, it’s an error rather than a silently degraded estimate. More on that distinction below.

The Multi-Threading Problem

A simple approach would be to fork a child, start counting, wait for it to exit, read counters. That works for single-threaded programs, but OCaml 5.x programs with multiple domains spawn multiple pthreads—each domain gets a domain thread plus a backup thread for systhreads. A 4-domain program has at least 8 pthreads, and naive per-thread measurement would miss most of the work.

This is where Apple’s Profile Every Thread (PET) mechanism comes in. Instead of reading counters for a single thread, PET sets up a kernel timer that fires periodically (default: every 1ms) and snapshots PMC values for every thread matching a PID filter. These samples get written to a kernel trace buffer (kdebug) with thread IDs and timestamps.

The approach is:

1. Fork a child process, held at a pipe barrier
2. Configure the PMC hardware with requested events
3. Set up PET sampling filtered to the child’s PID
4. Enable kdebug tracing for PERF_KPC_DATA_THREAD events
5. Release the child (close the pipe), let it exec the target command
6. Poll kdebug for samples until the child exits
7. For each thread, compute the delta between first and last sample
8. Sum deltas across all threads

Thread 1: [sample_0] -------- [sample_1] -------- [sample_N]
Thread 2:      [sample_0] -------- [sample_1] -------- [sample_N]
Thread 3:           [sample_0] -------- [sample_1] -------- [sample_N]
...

Result = Σ (thread_last - thread_first) for all threads

This is fundamentally a sampling-based approximation rather than continuous counting. But for benchmarks that run longer than a few milliseconds, the results are accurate enough to be useful. The comparison with Linux perf stat is more nuanced than “exact vs approximate” though.

Sampling Period Trade-offs

The -p flag controls the sampling period. The default 1ms works well for most OCaml programs since domains typically live for the program’s duration. For short-lived benchmarks you can go faster at the cost of more overhead by setting smaller values for -p.

# Default 1ms - good balance for most programs
sudo ./mperf-stat -e cycles -e instructions -- ./benchmark

# 0.5ms - catches short-lived threads, higher overhead
sudo ./mperf-stat -p 0.5 -e cycles -e instructions -- ./short_benchmark

# 5ms - less overhead for long-running workloads
sudo ./mperf-stat -p 5 -e cycles -e instructions -- ./long_benchmark

OCaml Bindings

Since the goal is integration with OCaml benchmarking services, mperf includes an OCaml library that wraps the CLI tool and parses its JSON output:

open Apple_perf_stat

let () =
  match run
    ~events:["cycles"; "instructions"; "l1d-tlb-misses"; "branch-misses"]
    ["./my_benchmark"; "--size"; "10000"]
  with
  | Ok result ->
    Printf.printf "IPC: %.2f\n" (get_ipc result);
    Printf.printf "Wall time: %.3f s\n" (wall_time_seconds result);
    (match get_counter "l1d-tlb-misses" result with
     | Some misses -> Printf.printf "TLB misses: %Ld\n" misses
     | None -> ())
  | Error e ->
    Printf.eprintf "Error: %s\n" (string_of_error e);
    exit 1

The library has no external dependencies beyond unix and str from the OCaml standard library. It handles the root privilege check, argument construction, JSON parsing, and error reporting.

The idea is to use instruction counts as a stable performance metric. Wall-clock time varies with system load and thermal throttling; instruction counts are deterministic. Pairing that with IPC and cache miss rates gives you a pretty complete picture of where regressions come from. The limit of 10 counters on Apple Silicon is enough to get started.

Counting Accuracy

How accurate are the results from mperf? And how do they compare to Linux perf?

It’s tempting to describe this as “perf stat is exact, mperf is approximate” but that’s not the full picture. perf stat uses exact hardware counting only when the requested events fit within the CPU’s physical PMU counters. Once you exceed that limit, the kernel enables time-division multiplexing: it rotates events through the available counters via an hrtimer and scales the final counts:

final_count = raw_count * (time_enabled / time_running)

The perf wiki is explicit about this: “It is very important to understand this is an estimate not an actual count. Depending on the workload, there will be blind spots which can introduce errors during scaling.” When multiplexing is active, perf stat shows a percentage column indicating what fraction of time each event was actually measured — anything below 100% means the count was extrapolated.

How many events before multiplexing kicks in depends on the hardware? It depends on how many PMU counters are exposed on the particular CPU. Here are a few examples from modern CPUs:

On Linux you can check your own CPU with dmesg:

$ dmesg | grep -E "generic registers|fixed-purpose events"
... generic registers:      4
... fixed-purpose events:   3

Note that the NMI watchdog steals one general-purpose counter by default on most distros (cat /proc/sys/kernel/nmi_watchdog — if it returns 1, you’ve lost a counter).

So in practice the comparison is:

• perf stat with ~6 events on Skylake: exact counting, genuinely better than mperf’s PET approach.
• perf stat with 12+ events: time-multiplexed scaled estimates. Still different from PET sampling (perf counts every event while the counter is active, then extrapolates for inactive periods), but the result is still an estimate.
• mperf with up to 10 events: PET sampling-based approximation, but all counters are always physically active: no time-sharing, no scaling. The inaccuracy comes from sampling granularity (missing activity between snapshots), not from extrapolation.

For the common case of measuring cycles + instructions + a handful of cache/TLB events (say 4-6 total), perf stat on Linux gives you exact counts and mperf gives you a close approximation. For larger event sets in perf, you hit the multiplexing case and might see unexpected results. Both tools are producing estimates through different mechanisms. mperf’s hard limit of 10 events means you always know whether your counts are real.

The other caveats: mperf relies on Apple’s private kperf and kperfdata frameworks, which could change with any macOS update (though they’ve been stable across M1 through M4). And very short-lived threads might be missed entirely if they complete within a single sampling period.

Getting Started

git clone https://github.com/tmcgilchrist/mperf
cd mperf
make
sudo ./mperf-stat -- echo hello

List all available events for your CPU with ./mperf-stat -l.

The code is based on ibireme’s kpc_demo.c, with event documentation from jiegec/apple-pmu and the Apple Silicon CPU Optimization Guide.

What’s Next

Porting this to x86_64 macOS, the PMC infrastructure should be similar. The more immediate goal is integrating mperf into an automated benchmarking pipeline for the OCaml compiler, tracking instruction counts and IPC across commits the way we’ve been doing with frame pointers and eBPF on Linux.

This fits into the broader effort of making OCaml programs easier to profile. Frame pointer support gives us perf and eBPF integration on Linux; mperf gives us hardware counter access on macOS without reaching for Instruments. Using frame pointers on macOS gives us accurate backtraces in Instruments when used with xctrace.

More CFI and frame pointers work

This month I set myself a goal to wrap up different pieces of work on OCaml that have been taking up space in my notes and my mind from last year. I wanted a clean slate to focus on the new year. I also enjoy reading other developers notes like Opening up old release branches and Faster Faster GDB Startup. So here is my minor contribution and we are jumping straight in!

What is CFI?

If you’ve ever tried to get a backtrace in GDB on Power and gotten garbage, that’s because there was no CFI to help the debugger unwind the call stack. Call Frame Information (CFI) is a part of the DWARF standard, it’s used for describing the call frames and register usage of a language. A call frame is a specific area of memory that is allocated on the stack, which typically corresponds to a function call in the source language. However some functions won’t allocate a call frame like leaf functions or tail-recursive functions. I have some ASCII stack frame diagrams later.

Within each call frame, the CPU saves a set of registers, which ones depend on the architecture. When returning from a function, those registers need to be restored. The ABI document spells out the details; for Power that’s the 64-Bit ELF V2 ABI Specification.

The code that allocates space on the call frame stack and performs the save operations is called the function prologue, and the corresponding code that performs the restore operation and deallocates the frame is called the epilogue.

The debugger uses CFI to:

1. Generate a backtrace by unwinding the function call stack.
2. Restore the state of registers when visiting previous functions higher up in the call stack.
3. Find registers that get spilled to the stack when they get overwritten within a function.

For our purposes, we care about working backtraces and recovering key registers like the return address and frame pointer.

The DWARF specification introduces the term CFA (Canonical Frame Address), is a specific point in the stack frame that is used as the base address for finding everything else in the frame. For example, the Link Register representing the address to return to when exiting a function is stored at CFA+16 on Power. With this terminology out of the way let’s look at how to apply it to Power (if you want more details checkout the excellent book Building a Debugger).

Power CFI

When Power architecture support was added back into OCaml 5 in ocaml#12276 there was no support for Call Frame Information. Without CFI it is difficult to debug problems on Power and do performance engineering as you don’t have working backtraces. So last year I started work on adding DWARF Call Frame Information (CFI) directives to the Power backend so that debuggers like GDB can unwind the stack through OCaml frames, C-to-OCaml transitions, and runtime functions (GC, C calls, exceptions, effects etc).

For the Power architecture, the calling conventions are defined in the 64-Bit ELF V2 ABI Specification: Power Architecture document and specifically we’re interested in the section 2.2.2. The Stack Frame which details the calling conventions for C. What registers are used? Which registers should be saved between function calls? What do those registers represent (return addresses or stack pointers)?

For 64-Bit ELF V2 ABI this looks like:

                                  Higher addresses
                             ┌─────────────────────────────┐
     Caller's SP  ─────────► │  Back-chain (caller's)      │  8 bytes
                             ├─────────────────────────────┤
                             │  Floating-Point Register    │  8 × (32 − N) bytes
                             │  Save Area                  │  fN saved at
                             │  (optional, callee saves)   │  caller_SP − 8×(32−N)
                             ├─────────────────────────────┤
                             │  General-Purpose Register   │  8 × (32 − N) bytes
                             │  Save Area                  │  rN saved at
                             │  (optional, callee saves)   │  (below FPR save area)
                             ├ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┤
                             │  Alignment padding          │  0 or 8 bytes
                             │  (for quadword alignment)   │  (if needed)
                             ├─────────────────────────────┤
                             │ Vector Register             │  16 × (32 − N) bytes
                             │ Save Area                   │  vN saved at
                             │ (optional, quadword aligned)│  (below GPR area + pad)
                             ├─────────────────────────────┤
                             │  Local Variable Space       │  variable
                             │  (optional)                 │
                             ├─────────────────────────────┤
     SP + 32      ─────────► │  Parameter Save Area        │  variable (optional)
                             │  (optional, 8 × num params) │  (quadword aligned)
                             ├─────────────────────────────┤
     SP + 24      ─────────► │  TOC Save (r2)              │  8 bytes
                             ├─────────────────────────────┤
     SP + 16      ─────────► │  LR Save                    │  8 bytes
                             ├─────────────────────────────┤
     SP + 8       ─────────► │  CR Save (word) + Reserved  │  4 + 4 bytes
                             ├─────────────────────────────┤
     SP           ─────────► │  Back-chain                 │  8 bytes
                             └─────────────────────────────┘
                                  Lower addresses

The minimum stack frame size shall be 32 bytes. A minimum stack frame consists of the first 4 doublewords (back-chain doubleword, CR save word and reserved word, LR save doubleword, and TOC pointer doubleword), with padding to meet the 16-byte alignment requirement.

For OCaml we retain this 32 byte structure for compatibility with C and layer OCaml specific structure on top. Here are the interesting bits:

1. LR is always at CFA+16. OCaml uses the Power ELF V2 ABI which mandates the caller reserves SP+16 for the link register save. So we use CFI to declare this fact using .cfi_offset 65, 16.
2. Function prologue instructions reordered. The old prologue adjusted the stack pointer register (SP) first (addi), then saved LR. The new prologue saves LR/TOC into the caller’s frame first, then adjusts SP with stdu (store-with-update, which atomically writes the back-chain pointer). This more closely matches the Power ABI convention that C compilers follow and is probably better for async-signal safety. The Table of Contents (or TOC) is a pointer used on Power for position-independent code support.
3. cfi_startproc moved before instructions. This assembler directive tells the debugger where a function begins, previously it was added after the TOC setup and debug information. Now it’s directly after the symbol label so the unwinder covers the full function body and we avoid certain edgecases when setting breakpoints.
4. DWARF expressions for stack switches. When switching between the OCaml stack and C stack (and the opposite direction), a moderately complex .cfi_escape expressions computes the CFA by dereferencing the saved stack pointer from the c_stack_link structure. This was the bulk of the work, understanding the structure of the C stack frames to load the right saved registers. Unfortunately there aren’t many tools to debug when you get this wrong, I ended up writing Python scripts to print out chunks of memory and work out which bits pointed to code or other stack frames. There is a project in here somewhere to run the stack machine for CFI and validating it against memory that I might get to later this year.
5. Signal frames. Runtime entry points (caml_call_gc, caml_c_call, caml_call_realloc_stack, etc.) are marked with .cfi_signal_frame so the unwinder treats them as asynchronous interruption points, prints out “<signal handler called>” and avoids some stack integrity checks that cause GDB to stop unwinding.
6. Pre-initialized LR save area. In caml_start_program, before calling into OCaml, the code stores the return address at SP+16 so the unwinder can find LR even if the first OCaml function has fun_frame_required = false. This was needed for exception handling code.

Here’s a nice ASCII diagram of an OCaml Frame on Power:

                           Higher addresses
                      ┌─────────────────────────┐
        CFA+16  ───►  │  LR save (return addr)  │  8 bytes ◄── always at caller_SP+16
                      ├─────────────────────────┤
        CFA+8   ───►  │  TOC save (r2)          │  8 bytes ◄── always at caller_SP+8
                      ├─────────────────────────┤
        CFA     ───►  │                         │
                      ╞═════════════════════════╡ ◄── SP + frame_size  (= CFA)
                      │                         │
                      │  Reserved stack space   │  32 bytes
                      │  (ABI-mandated)         │
                      ├─────────────────────────┤
                      │  Float stack slots      │  num_float_slots × 8
                      ├─────────────────────────┤
                      │  Int stack slots        │  num_int_slots × 8
                      ├─────────────────────────┤
                      │  (stack_offset area)    │  trap frames (16)
                      │                         │  outgoing params, etc.
                      ├─────────────────────────┤
          SP    ───►  │  (16-byte aligned)      │
                      └─────────────────────────┘
                           Lower addresses

Now we have working backtraces on Power for GDB. With working CFI in place, the next step was adding frame pointer support.

More frame pointers

I wrote about the motivation for frame pointers in Why do frame pointers matter for OCaml?. At the end I mentioned I didn’t have working code for RISC-V, Power or s390x. This is the next chunk of work getting frame pointers for the full set of architectures.

Power Frame Pointers

Power frame pointer support built on top of the earlier CFI work and adds save/restore instructions for maintaining r31 (the frame pointer register). Power ABI documents use the term back-chain to describe the saved frame pointer for the previous frame, which is also used in s390x documents later on. The main changes were:

1. Allocation pointer moved from r31 to r23. The GC allocation pointer previously lived in r31, which is the standard Power ABI frame pointer register. Moving it to r23 (another callee-saved register) freed up r31 for its intended purpose.
2. Back-chain maintained for every frame. Every frame allocation now uses stdu (store-with-update) instead of addi to adjust the stack pointer, which atomically writes the back-chain. This required some recalculation of instruction sizes to ensure we always emit valid branch instructions.
3. Trap frame was enlarged from 16 to 48 bytes. This was the surprise! Trap frames are used by the OCaml runtime for handling exceptions and effects. The old 16-byte trap frame placed trap data at SP+32 and SP+40 (within the reserved area). With stdu-based allocation, the back-chain at new_SP points to old_SP, and the ABI expects old_SP+16 to hold LR. At trap_size=16, old_SP+16 = new_SP+48, which falls outside the allocated region. Increasing to 48 bytes ensures the LR save area at back_chain+16 lies within the trap frame, and trap data at new_SP+32/new_SP+40 doesn’t collide with the caller’s save area.
4. Runtime frame pointer fixups. When the OCaml stack grows, the runtime needs to walk the stack and rewrite the saved FP values so the frame pointer chain remains valid.

Now an OCaml Frame with frame pointers looks like this:

                               Higher addresses
                          ┌─────────────────────────┐
            CFA+16  ───►  │  LR save (return addr)  │  8    ◄── ABI: caller_SP+16
                          ├─────────────────────────┤
            CFA+8   ───►  │  TOC save (r2)          │  8    ◄── ABI: caller_SP+8
                          ├─────────────────────────┤
            CFA     ───►  │  Back-chain (prev SP)   │  8    ◄── written by stdu
            ══════════════╪═════════════════════════╪══════════ FP after prologue
                          │                         │
                          │  Reserved stack space   │  32 bytes (ABI-mandated)
                          ├─────────────────────────┤
                          │  Float stack slots      │  num_float_slots × 8
                          ├─────────────────────────┤
                          │  Int stack slots        │  num_int_slots × 8
                          ├─────────────────────────┤
     fp_save_offset ───►  │  Saved r31 (old FP)     │  8    ◄── FP-only
                          ├─────────────────────────┤
                          │  (stack_offset area)    │  trap frames, outgoing
                          │                         │  params, etc.
                          ├─────────────────────────┤
            SP = FP ───►  │  (16-byte aligned)      │
                          └─────────────────────────┘
                               Lower addresses

This change was fairly painful because I didn’t realise I needed to increase the trap size and ran into issues with instructions trashing saved registers on the stack. GDB support for watching memory addresses and breaking when something wrote to that address was really useful here. Go read the ocaml#14482 for more details.

RISC-V Frame Pointers

RISC-V frame pointers was started by Miod Vallat in 2024, who had half the tests passing before I picked this up again. The surprising thing about RISC-V is that the frame pointer points to CFA, above the saved registers, rather than to the frame record itself like ARM64 does. The RISC-V ABI convention is s0 = sp + frame_size, pointing to the slot just past the {saved_s0, saved_ra} pair at the top of the frame.

Comparing this to ARM64 makes the difference more obvious (note x29 is the ARM64 frame pointer register).

    RISC-V                              ARM64
    ──────                              ─────
         ┌───────────┐                       ┌───────────┐
    s0 ──▶  CFA      │                       │  CFA      │
         ├───────────┤                       ├───────────┤
    -8   │  ra       │                  -8   │  x30 (lr) │
         ├───────────┤                       ├───────────┤
    -16  │  old s0   │            x29 ──▶-16 │  old x29  │
         ├───────────┤                       ├───────────┤
         │  locals   │                       │  locals   │
         └───────────┘                       └───────────┘

    s0 = CFA                           x29 = CFA - 16
    s0 = sp + frame_size               x29 = sp + frame_size - 16

The main pain-point after working this out was adjusting the frame walking code in the runtime and adjusting the runtime assembly stubs. RISC-V frame pointers point to CFA (above the frame record), not to the frame record itself. When caml_try_realloc_stack rewrites frame pointers after growing a fiber stack (OCaml 5 features effect handlers which get implemented as fiber stacks in the runtime), it must account for this: fp->prev is a CFA value, and the next frame record is at fp->prev - 1 (i.e. CFA − 16 bytes).

This ticked off another architecture supporting frame pointers and with that the Tier-1 unsupported architectures list got shorter. ocaml#14506 has the full code.

s390x Frame Pointers

Now, s390x frame pointers seemed like they would be straightforward. I had implemented other platforms like Power and ARM64, and had a good understanding of which areas I needed to modify:

1. enable the configure flag,
2. update the code emission logic for frame sizes,
3. fix up the prologue/epilogue assembly for back-chain handling,
4. add CFI directives, handle effect stack switching, and
5. update the runtime’s assembly stubs.

The usual drill, I even had it hand-written on paper. What I didn’t account for is the difference in stack layout, which can be reduced to OCaml wants to store the back-chain and r14 at SP+0 and SP+8 respectively, while s390x ABI mandates back-chain at SP+0 and r14 at SP+112. The s390x ABI also requires a minimum of 160 bytes for the frame size. I’ll write about this another time but the result is I’m still working out what approach to take (the code is at tmcgilchrist/ocaml#24). All the options seem terrible right now.

Smaller fixes

While I was working in this area there were some smaller fixes to be made. Recall from earlier we need to write .cfi_escape expressions to describe how to find the C stack from OCaml (and the reverse). The expressions for s390x were slightly wrong and needed some tweaking; ocaml#14500 has more details. Once I had stack diagrams for s390x the work mostly involved writing Python to inspect memory and adjusting where the CFI expressions pointed to in memory.

FreeBSD frame pointers was requested during ICFP 2025 and I thought it would be a simple matter of changing the configure script to allow it. Nothing is ever simple. The frame pointer tests were failing because the first symbol in a module would clash with the module.entry symbol (representing the module initialisation code associated with every module in OCaml e.g. camlFoo.entry) breaking the frame pointer tests. Once I re-discovered that detail (ocaml#4690), the fix was simple. This seems to be a common problem for LLVM toolchains. For good measure I documented how to use dtrace and pmcstat to do CPU profiling, see ocaml#14486.

Conclusion

With all that OCaml now has working debugger support on Power, completing the set of Tier-1 architectures where such tools will work. This exceeds the support that existed in OCaml prior to 5.0 and completes the one thorny issue I knew about to achieve feature parity. Closing the CFI s390x bug I reported in 2024 resolved the last known issue with debuggers on that platform.

On top of that we have frame pointer support for more architectures than ever, if you’re debugging or profiling OCaml on Power, RISC-V, ARM64, or AMD64, you now get clean backtraces. As I wrote in the OCaml manual section on profiling this is key to getting accurate profiling using Linux perf and macOS Instruments. FreeBSD users can profile with dtrace and pmcstat, and there is documentation on how to get started.

All these changes should eventually reach an OCaml release, if I’m lucky the smaller bug fixes will squeeze into 5.6 and the rest targeting 5.7. That leaves s390x frame pointers (still wrestling with the 160-byte minimum frame problem) and Windows (the elephant in the room). We shall see.

Why do frame pointers matter for OCaml?

At the end of last year at Tarides, my colleagues and I worked on improving frame pointer support in OCaml. While pitching the work internally, someone asked: Why do frame pointers matter for OCaml? Here’s the answer I gave.

Frame pointers give us a reliable way to profile OCaml programs using standard tools like perf and eBPF. Instead of building OCaml-specific profiling solutions, we can reuse what already exists. Instruments on macOS, the whole ecosystem of eBPF tools on Linux. That’s a big win for a smaller community like OCaml’s. Targeting all Tier-1 supported platforms (AMD64, ARM64, RISC-V, s390x and Power) would give a consistent experience across them all, it’s embarrassing to omit a platform when listing features. As a bonus, frame pointers help debuggers when DWARF information is missing, providing a last resort option.

What are Frame Pointers?

The frame pointer is a register (%rbp on AMD64 or x29 on ARM64) that points to the base of the current stack frame. The stack frame (also known as the activation frame or the activation record) refers to the portion of the stack allocated to a single function call. This is where a function will save registers to or use for storage while performing work. By saving the frame pointer along with the return address, the call stack for OCaml can be calculated in a process called unwinding. As an example ARM64 call stack looks like this (note that the stack grows downwards to lower addresses):

          Stack                        Instruction Text
   ┌────────────────────┐            ┌────────────────────┐
   │Return Address (x30)┼───────────►│ ancestor function  │
┌─►│Saved Frame Pointer │            │                    │
│  ┼────────────────────┼            │                    │
│  │Local allocations   │     ┌─────►│ parent function    │
│  │                    │     │      │                    │
│  ├────────────────────┤     │      │                    │
│  │Return Address (x30)├─────┼─────►│ grand-parent func  │
└──┤Saved Frame Pointer │     │      │                    │
┌─►┼────────────────────┼     │      │                    │
│  │Local allocations   │     │   ┌─►│ current function   │
│  │                    │     │   │  │                    │
│  ├────────────────────┤     │   │  └────────────────────┘
│  │Return Address (x30)├─────┘   │
└──┤Saved Frame Pointer │         │
   ┼────────────────────┼◄──┐     │      Registers
   │Local allocations   │   │     │ ┌──────────────────────┐
   │                    │   └─────┼─┤ x29 - frame pointer  │
   └────────────────────┘◄──┐     │ │ x30 - link register  │
                            └─────┼─┤ sp  - stack pointer  │
                                  └─┼ pc  - program counter│
                                    └──────────────────────┘

More specifically, frame pointers can be used by profilers to unwind the stack in a language agnostic way. Many languages like C/C++, Rust, and Erlang support maintaining frame pointers.

Return of the Frame Pointers

We started with the ambitious goal of adding frame pointer support to all the Tier-1 platforms supported by OCaml. That included the popular AMD64 and ARM64 platforms that cover most users, and less common platforms like RISC-V, Power and s390x.

The specific focus was to address the recognised limitations of perf when used with OCaml 5 programs (#12563). OCaml 5 (aka multicore) introduced non-contiguous stacks as part of the implementation of effects (see the PLDI 2021 paper on retrofitting effect handlers – Section 5.5). At a high level, these stacks are stored in memory allocated by the runtime and not on the stack as would happen with C/C++. These non-contiguous stacks are essentially unknown to perf and will not work correctly with the copying nature of perf, it will copy the wrong things. So traces produced for OCaml 5 will appear truncated or contain incorrect values. This same problem occurs if you use DWARF call graphs, in particular perf will copy a chunk of the stack without decoding it via DWARF and for OCaml 5 onwards what it copies might not be stack. So the best solution is frame pointers.

We started by looking at #11144 which restored frame pointer support for AMD64 after the OCaml 5.0 release. It was clear that adding frame pointers required changes to both the backend assembly code generation and the OCaml runtime, and that the general design should follow the existing AMD64 approach. With that understanding the first step was to extend the --enable-frame-pointers autoconf file to allow configuring the compiler on macOS which only required a one line change of [x86_64-*-linux*|x86_64-*-darwin*], to recognise the new platform (more context).

Now, that allowed configuring the compiler with ./configure --enable-frame-pointers and since the work had already been done to codegen and runtime, we only needed to test the changes on AMD64 macOS.

The next platform we chose was ARM64, because it is a common platform for cloud vendors with most offering a Linux ARM64 option and all new Apple laptops come with an ARM64 CPU (aka Apple Silicon). Getting ARM64 working seemed like it would have the most impact, both for OCaml deployments and for local development where Apple laptops are quite common. Implementing again starting with a simple change to add new Linux and macOS cases to autoconf [x86_64-*-linux*|x86_64-*-darwin*|aarch64-*-linux*|aarch64-*-darwin*],. This exposes a configuration flag Config.with_frame_pointers that we can query when implementing the assembly code generation.

The assembly generation code is in the asmcomp directory of the OCaml sources, the common code is in that directory and each platform supported has a sub-directory for it e.g. asmcomp/arm64 for ARM64. Conceptually the changes required a modification to the calling convention for OCaml code to only use x29 as a frame pointer (previously it was considered as a general purpose register), a second change to the calculation of stack frame sizes to allow for an extra register save, and finally emitting the correct assembly to save/restore x29.

The end result is that frame pointer save and restore is implemented using stp and ldp instructions plus an extra add instruction for updating x29. This results in an extra add plus saving an extra register for functions allocating a stack frame. In assembly this looks like:

;; function prologue
	stp	x29, x30, [sp, #-16]
	sub	sp, sp, #16
	add	x29,  sp, #0

;; function epilogue
	add	sp, sp, #16
	ldp	x29, x30, [sp, #-16]

Whereas previously the store and load would just operate on the x30 the link register. The overhead is quite minimal. The compiler already maintained the x29 register when calling into C and when using TSan. Plus there is minimal extra stack space usage, as we needed to be quad word aligned anyway, so the x29 register is going into an already allocated space on the stack. I’m happy with how this turned out.

It’s interesting to note that the ABI on macOS requires maintaining frame pointers, and certain popular Linux distributions like Ubuntu, Fedora and Arch are reenabling frame pointers in recent distributions. So I expect the impact on usability will be high and the runtime overhead quite low.

What we achieved and what’s left?

The short version is that frame pointers are now available for the two most popular Unix platforms (AMD64 and ARM64), that covers most deployments of OCaml and many OCaml developers. The work is split across these PRs:

• #13163: Enabled frame pointers on macOS x86_64 worked with some minor autoconf changes to correctly detect macOS and updating the tests to remove Linux specific backtrace formatting. Available in OCaml 5.3.

• #13500: Added frame pointers support for ARM64 on Linux and macOS. This will be released in OCaml 5.4.

• #13595: Fixed a bug introduced in #13050 where the wrong Canonical Frame Address (CFA) register was used for calls from OCaml into non-allocating C code. This appeared as incorrect backtraces in debuggers like GDB when setting a breakpoint inside the C code.

• #13575, #13635: Maintain OCaml frame pointers correctly even when using C libraries that do not support them, allowing mixing OCaml code with frame pointers with an operating system environment that might not have them.

• #13751: Proposes a new section for the OCaml manual detailing how to use frame pointers with Linux perf highlighting what works and what does not work (DWARF stack unwinding). Hopefully this will get merged soon and close the documentation gap for using perf reliably with OCaml programs. Should be available in OCaml 5.4.

The original target of all Tier-1 platforms was an ambitious one. Unfortunately, we didn’t manage to finish everything. The work for adding RISC-V frame pointer support was started but needs further work. The work in progress is available at tmcgilchrist/ocaml#22. The work on s390x and Power didn’t get much further than diagramming stack frames and reading ABI documents. In particular the Power backend is also missing CFI support which makes debugging very difficult and really needs to be added first before tackling frame pointer support. Essentially you don’t even have a working debugger while debugging any problems with your frame pointer changes.

Next steps are finding time or help with finishing off the remaining platforms, starting with RISC-V. If you’d like to help please get in touch. Then doing some comprehensive benchmarking for the ARM64 platform, my intuition is there is negligible overhead when adding frame pointers since it uses a load/store pair instruction, but firm results would be better.

If you’ve made it this far, thanks for reading.

Experimenting with OCaml and eBPF

Building on top of the excellent book BPF Performance Tools by Brendan Gregg. How can we apply the techniques from Chapter 12 Languages to OCaml?

First OCaml is roughly equivalent to C, it’s a compiled language with a runtime written in C. It supports frame pointers using the --enable-frame-pointers configuration option on x86_64, with ARM64 support in OCaml 5.4. Eventually the code we’re interested in is C or looks roughly like C but with a different calling convention.

For tracing into the Linux kernel, you’ll need a distribution that is compiled with frame pointers like Ubuntu 24.04 and we can reuse the kernel’s own symbol table. There are some exceptions for inlined functions and some blacklisted functions that aren’t safe to trace. However for the pieces I’ve looked at like memory allocation and virtual memory, it is fine.

For the OCaml runtime written in C, it can be configured to include symbols, frame pointers and debuginfo for the portions written in C. The sections of the runtime written in assembly have symbols and frame pointers. For actual OCaml code it will have symbols and frame pointers, with limited debuginfo. Demo time!

OCaml Function Tracing

Given this test program taken from a bug report #13123 against OCaml.

(* Build with:
  ocamlfind ocamlopt -package unix -package threads -thread -linkpkg -o liquidsoap_test.exe liquidsoap_test.ml *)
let frame_size = 0.04
let pcm_len = int_of_float (44100. *. frame_size)
let channels = 2

let mk_pcm () = Array.init channels (fun _ -> Array.make pcm_len 0.)

let rec fn a =
  if Array.length a <> 0 then
    Gc.full_major ();
  let pcm = mk_pcm () in
  ignore(pcm);
  Unix.sleepf 0.04;
  fn [||]

let () =
  let deadweigth = Array.make (40 * 1024 * 1024) 1 in
  Unix.sleepf 0.04;
  let th = Thread.create fn deadweigth in
  Thread.join th

And an opam switch created with frame pointers.

$ opam switch create 5.3.0 5.3.0+options ocaml-option-fp

$ ocamlfind ocamlopt -package unix -package threads -thread \
  -linkpkg -o liquidsoap_test.exe liquidsoap_test.ml

Running this code will print the PID each time the liquidsoap_test.exe executable is run.

$ sudo bpftrace -e 'uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:caml_start_program { printf("OCaml run with process ID %d\n", pid); }'

Here we are using the information we know about OCaml startup, the caml_start_program is an assembly function that bridges the gap between the C startup code and OCaml, setting up the environment for the OCaml code. The section after uprobe: needs to point to the executable being run, change that if you want to trace something else.

Next, recall that we are dealing with a mix of regular C functions and OCaml functions. Listing the tracepoints available shows a mix of regular C functions prefixed with caml_* that are either part of the runtime or C primitives. OCaml compiler performs name mangling so anything coming from an OCaml source file will have a prefix caml<MODULE>. e.g. camlStdlib__Domain* for the domain.ml module from the standard library or camlStdlib__Int.compare_296 for the compare function on Int. Armed with that knowledge. This command will list available probe points:

$ sudo bpftrace -l 'uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:*'

If we wanted to count the number of function calls in a binary, we could do it like so:

$ cat count.bt
# Printout matched program
uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:caml_start_program
{
  printf("OCaml run with process ID %d\n", pid);
}

# Trace function calls
uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:camlLiquidsoap_test*
{
    @[probe] = count();
}

$ sudo bpftrace count.bt
Attaching 5 probes...
OCaml run with process ID 128477
^C

@[uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:camlLiquidsoap_test.entry]: 1
@[uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:camlLiquidsoap_test.fn_327]: 1
@[uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:camlLiquidsoap_test.mk_pcm_273]: 1029
@[uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:camlLiquidsoap_test.fun_601]: 2058

Another thing we could do is see how much time is spent in the minor GC promotion function.

OCaml uses a bump-pointer allocator for the minor heap, when that is full it will call a C function to scan the minor heap, destroy the junk, and promote anything that survives into the major heap. I know that the main entry point for this is called caml_empty_minor_heap_promote. So this script will instrument the entry and exit for that function and print out a histogram of the time taken.

# cat gcprofile.bt

uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:caml_start_program
{
  printf("Attaching to OCaml process ID %d\n", pid);
}

uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:caml_empty_minor_heap_promote
{
  @t = nsecs;
}

uretprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:caml_empty_minor_heap_promote / @t /
{
  @minor_gc_times = hist(nsecs - @t);
}

What about the major GC? The design of that is more complicated but I know major_collection_slice does the majority of the work, so we attach there.

# cat gcprofile_major.bt

uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:caml_start_program
{
  printf("Attaching to OCaml process ID %d\n", pid);
}

uprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:major_collection_slice
{
  @t = nsecs;
}

uretprobe:/home/tsmc/ocaml-performance/liquidsoap_test.exe:major_collection_slice / @t /
{
  @major_gc_slice_times = hist(nsecs - @t);
}

Take Away

OCaml programs can traced with eBPF and bpftrace. You need to install OCaml with frame pointers enabled and use a Linux distribution like Ubuntu 24.04 that also enables frame pointers, so you can trace into system libraries. The OCaml runtime and certain primitives use a symbol prefix of caml_ and OCaml code uses a prefix of caml<MODULE> where <MODULE> is the OCaml module containing the code. This partially covers the functionality in ocamlprof which lets you profile function counts and branches taken in things like while, if and try. With eBPF we can count the function calls but more work needs to be done to support the branching constructs, essentially we need a USDT implementation for OCaml that understands OCaml’s name mangling and calling conventions. The upside is eBPF can be applied to any OCaml binary without needing a recompile.

Next steps are adding USDT probes to the OCaml runtime, so there is a static API for the GC, and after that expose USDT probe points from OCaml programs.

Building OCaml from assembly

At work I’ve been focusing on improving the debugging experience with OCaml. As part of that I’ve discovered how some of the pieces fit together, that might be obvious in retrospect, but are interesting to at least me so I’m going to post details about them here.

The first nugget is you can hand compile an OCaml program into a final executable. What do I mean? You can ask the OCaml compiler to output all the assembly generated that goes into a library or executable. Then take that an call the assembler yourself to build it. First lets review how the compiler works.

Compilation Pipeline

Here is a grossly simplified overview of the OCaml compiler. We feed in OCaml source code in the form of ml/mli files, which flow through each stage and eventually end up being emitted as either object files or textual assembly files. The first step from OCaml Source to Parse Tree uses menhir to parse and generate an untyped AST representing the code in the source file. This is then type checked into a typed tree, this is where the type theory happens. After that, there are some stages where the typed tree is transformed into representations more suitable for generating assembly. The final stage traverses the CMM/Linear AST generating assembly code for a specific family of CPUs (like x86_64 or ARM64).

                                      
 ┌──────────────┐   ┌──────────────┐  
 │ OCaml Source │   │  Parse Tree  │  
 │              ┼───►              │  
 └──────────────┘   └──────┬───────┘  
                           │          
 ┌──────────────┐   ┌──────▼───────┐  
 │    Lambda    │   │  Typed Tree  │  
 │              ◄───┼              │  
 └──────┬───────┘   └──────────────┘  
        │                             
 ┌──────▼───────┐   ┌──────────────┐  
 │  CMM/Linear  │   │    Emit      │  
 │              ┼───►   Assembly   │  
 └──────────────┘   └──────────────┘  
                                      

Finally, this assembly is compiled by the system C compiler to produce object files or executables to be run. So we could treat the OCaml compiler as a fancy way to just generate assembly files, which we can then mess with to do things like add DWARF information or optimise assembly routines, or just for pure fun.

OCaml source

Starting with an OCaml program taken from Retrofitting Effect Handlers onto OCaml. This program doesn’t compute anything interesting but it does show how OCaml’s FFI to C works and how to pass control between the two. So it is interesting for what it does.

$ cat meander.ml
external ocaml_to_c
         : unit -> int = "ocaml_to_c"
exception E1
exception E2
let c_to_ocaml () = raise E1
let _ = Callback.register
          "c_to_ocaml" c_to_ocaml
let omain () =
  try (* h1 *)
    try (* h2 *) ocaml_to_c ()
    with E2 -> 0
  with E1 -> 42
let _ = assert (omain () = 42)

$ cat meander_c.c
#include <caml/mlvalues.h>
#include <caml/callback.h>

value ocaml_to_c (value unit) {
    caml_callback(*caml_named_value
                  ("c_to_ocaml"), Val_unit);
    return Val_int(0);
}

Reading from the bottom of the file, meander.ml asserts that the function omain returns the value 42. It gets that value by calling ocaml_to_c which is actually an external C function defined in meander_c.c, imported into OCaml using external in the first line of meander.ml. The C function calls back into OCaml using caml_callback which executes the c_to_ocaml function. An exception is raised, unwinding everything back to omain with it’s try/with blocks.

To compile this program we use the OCaml 5.2 compiler.

$ ocamlopt --version
5.2.0
$ ocamlopt meander_c.c meander.ml -o meander.exe
$ ./meander.exe
$ echo $?
0

Running the program under macOS gives a successful exit code, so it must have got 42 and the assertion passed. Try changing the value 42 to something else to check.

Next we will pull apart what the compiler is doing to generate the final executable. Run ocamlopt with these flags:

 $ ocamlopt meander_c.c meander.ml -o meander.exe -S -g -dstartup -verbose

+ cc  -O2 -fno-strict-aliasing -fwrapv -pthread -pthread  -D_FILE_OFFSET_BITS=64 -c -g -I'/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml' 'meander_c.c'
+ cc -c -Wno-trigraphs  -o 'meander.o' 'meander.s'
+ cc -c -Wno-trigraphs  -o '/var/folders/z_/7yzlrkjn6pd441zs1qhzpjv00000gn/T/camlstartup9b503b.o' 'meander.exe.startup.s'
+ cc -O2 -fno-strict-aliasing -fwrapv -pthread  -pthread   -o 'meander.exe'  '-L/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml'  '/var/folders/z_/7yzlrkjn6pd441zs1qhzpjv00000gn/T/camlstartup9b503b.o' '/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml/std_exit.o' 'meander.o' '/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml/stdlib.a' 'meander_c.o' '/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml/libasmrun.a'     -lpthread

Focusing on the ocamlopt command, the flag -S asks the compiler to generate the assembly files for the OCaml source, -g asks for debug information to be included, -dstartup generates the startup file that bridges between the C startup and OCaml (more on that later) and -verbose tells ocamlopt to print out what commands it’s running.

So, what has been printed out? The first line is compiling the meander_c.c file into an object file, the meander_c.o file in the current directory. Then we have a meander.s file being compiled (assembled) into another object file. This is the output of compiling the meander.ml OCaml source into assembly. The --verbose option doesn’t show how that file gets created. The third line is compiling the startup file from meander.exe.startup.s into another object file. The final step is calling the linker via cc to generate the final meander.exe file. You can see all the object files from previous steps plus the OCaml stdlib _opam/lib/ocaml/stdlib.a and _opam/lib/ocaml/std_exit.o from the local opam switch plus the OCaml libraries being added to the search path as -L/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml. It is not that dissimilar to building a C program.

What about those assembly files? The meander.s is our ARM64 assembly file for meander.ml open it up and search for entry. If you’re on Linux or another architecture like x86_64 the assembly will be different but the names will be the same. This is the entry point called when executing the program, the OCaml runtime jumps to the symbol _camlMeander.entry.

	.globl	_camlMeander.entry
L114:
	mov	x16, #34
	stp	x16, x30, [sp, #-16]!
	bl	_caml_call_realloc_stack
	ldp	x16, x30, [sp], #16
_camlMeander.entry:
	.cfi_startproc
	ldr	x16, [x28, #40]
	add	x16, x16, #328
	cmp	sp, x16
	bcc	L114
	sub	sp, sp, #16
	.cfi_adjust_cfa_offset	16
	.cfi_offset 30, -8
	str	x30, [sp, #8]

Search for other symbols like omain and c_to_ocaml

	.globl	_camlMeander.omain_278
_camlMeander.omain_278:
	.loc	1	8
	.cfi_startproc
	sub	sp, sp, #16
	.cfi_adjust_cfa_offset	16
	.cfi_offset 30, -8
	str	x30, [sp, #8]
....
_camlMeander.c_to_ocaml_273:
	.file	1	"meander.ml"
	.loc	1	5
	.cfi_startproc
	sub	sp, sp, #16
	.cfi_adjust_cfa_offset	16
	.cfi_offset 30, -8
	str	x30, [sp, #8]

All the code is there, we just need to assemble it. On my machine (macOS ARM64) running this command will give me an executable meander.exe without even using ocamlopt.

$ gcc -O2 -fno-strict-aliasing -fwrapv -pthread -D_FILE_OFFSET_BITS=64 \
      -c -g -I'/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml' 'meander_c.c'
$ gcc -c -Wno-trigraphs -o 'meander.o' 'meander.s'
$ gcc -c -Wno-trigraphs -o meanderCamlStartup.o meander.exe.startup.s
$ gcc -o 'meander.exe' '-L/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml' 'meanderCamlStartup.o' \
       '/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml/std_exit.o' 'meander.o' \
       '/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml/stdlib.a' 'meander_c.o' \
       '/Users/tsmc/code/ocaml/owee/_opam/lib/ocaml/libasmrun.a' -lpthread

Try it out, you’ll need to change /Users/tsmc/code/ocaml/owee/_opam to your local directory with a local opam switch for OCaml 5.2.

Startup file

What about that startup file? meander.exe.startup.s What is that for? Open the file and search for _caml_program, this is the entry point called by the startup code written in C.

_caml_program:
	.cfi_startproc
	ldr	x16, [x28, #40]
	add	x16, x16, #328
	cmp	sp, x16
	bcc	L136
	sub	sp, sp, #16
	.cfi_adjust_cfa_offset	16
	.cfi_offset 30, -8
	str	x30, [sp, #8]
L135:
	bl	_camlCamlinternalFormatBasics$entry
L137:
	adrp	x0, _caml_globals_inited@GOTPAGE
	ldr	x0, [x0, _caml_globals_inited@GOTPAGEOFF]
	ldr	x2, [x0, #0]
	add	x3, x2, #1
	dmb	ishld
	str	x3, [x0, #0]
	bl	_camlStdlib$entry

The code is responsible for calling the entry initialisation function for all imported modules. In meander.ml we only include a couple of functions from the standard library so we have _camlStdlib$entry, _camlStdlib__Sys$entry etc then we finally call _camlMeander$entry which we saw earlier.

We need this assembly file to generate an object file for linking into the final executable. If not the linker won’t have _caml_program symbol available and none of the OCaml Stdlib will be initialised. A fun exercise is to re-write this file to not call all those entry functions but still provide _caml_program and call into _camlMeander$entry.

I made small PR #13217 to improve this behaviour to loop over a table of functions to call rather than generating large slabs of identical code.

Bonus

Now you we don’t need the OCaml compiler to write OCaml.

But seriously the purpose for discovering this was to investigate adding DWARF debugging information to OCaml on macOS. That’s a different topic for next time.

Getting Started with LLDB on OCaml

This post is a companion to KC’s excellent Getting Started with GDB on OCaml that shows how to debug OCaml programs with GDB. I wanted to demonstrate the same functionality using LLDB on Linux ARM64. The aim is to show the beginnings of debugging OCaml programs with LLDB and highlight a few LLDB tricks I’ve found.

We will start with the same program:

(* fib.ml *)
let rec fib n =
  if n = 0 then 0
  else if n = 1 then 1
  else fib (n-1) + fib (n-2)

let main () =
  let r = fib 20 in
  Printf.printf "fib(20) = %d\n" r

let _ = main ()

Compiled with OCaml version 5.2.0.

$ ocamlopt --version
5.2.0
$ ocamlopt -g -o fib.exe fib.ml
$ ./fib.exe 20
fib(20) = 6765

The program prints the 20th Fibonacci number, nothing special but interesting because it has recursion. Now start up an lldb session.

$ lldb ./fib.exe

Setting breakpoints

We want to set breakpoints in the fib function. The first way to set breakpoints is based on OCaml function names, due to a process called name mangling, they look slightly different in the executable. Since we don’t know the exact names we can use tab completion to help us.

(lldb) br s -n camlFib.fib_ # press tab to show the possible matches
(lldb) br s -n camlFib.fib_270 # There is only one matching ending 270
Breakpoint 1: where = fib.exe`camlFib.fib_270 + 76, address = 0x0000000000051084

You can also set break points using lldb’s file name and number combination. This time we will set a breakpoint in the main function, which starts at line 6 in fib.ml.

(lldb) br s -f fib.ml -l 6
Breakpoint 2: where = fib.exe`camlFib.main_271, address = 0x0000000000050f48
(lldb)

Now we can run the program.

Breakpoint 2: where = fib.exe`camlFib.main_272, address = 0x00000000000510c8
(lldb) run
Process 11987 launched: '/home/tsmc/projects/fib.exe' (aarch64)
Process 11987 stopped
* thread #1, name = 'fib.exe', stop reason = breakpoint 2.1
    frame #0: 0x0000aaaaaaaf10c8 fib.exe`camlFib.main_272 at fib.ml:7
   4   	  else if n = 1 then 1
   5   	  else fib (n-1) + fib (n-2)
   6   	
-> 7   	let main () =
   8   	  let r = fib 20 in
   9   	  Printf.printf "fib(20) = %d\n" r
   10  	

The program execution starts in the lldb session and we stop at the breakpoint at main. LLDB has a terminal UI mode for stepping through the file. This can be started up typing gui into the lldb prompt, it should look similar to this.

Note that we can see both breakpoints highlighted on the line numbers, the backtrace of how we got here and the current line is highlighted. Use Esc to exit the terminal UI mode and go back to the lldb prompt. We will use the lldb prompt for the rest of the post.

Examining the stack

You can step through the OCaml program with lldb commands n and s. After a few n’s, examine the backtrace using the bt command.

(lldb) bt
* thread #1, name = 'fib.exe', stop reason = breakpoint 1.1
  * frame #0: 0x0000aaaaaaaf1084 fib.exe`camlFib.fib_270 at fib.ml:5
    frame #1: 0x0000aaaaaaaf108c fib.exe`camlFib.fib_270 at fib.ml:5
    frame #2: 0x0000aaaaaaaf108c fib.exe`camlFib.fib_270 at fib.ml:5
    frame #3: 0x0000aaaaaaaf108c fib.exe`camlFib.fib_270 at fib.ml:5
    frame #4: 0x0000aaaaaaaf10f4 fib.exe`camlFib.main_272 at fib.ml:8
    frame #5: 0x0000aaaaaaaf11bc fib.exe`camlFib.entry at fib.ml:11
    frame #6: 0x0000aaaaaaaee684 fib.exe`caml_program + 476
    frame #7: 0x0000aaaaaab46b48 fib.exe`caml_start_program + 132
    frame #8: 0x0000aaaaaab46640 fib.exe`caml_main [inlined] caml_startup(argv=<unavailable>) at startup_nat.c:145:7
    frame #9: 0x0000aaaaaab4663c fib.exe`caml_main(argv=<unavailable>) at startup_nat.c:151:3
    frame #10: 0x0000aaaaaaaee310 fib.exe`main(argc=<unavailable>, argv=<unavailable>) at main.c:37:3
    frame #11: 0x0000fffff7d784c4 libc.so.6`__libc_start_call_main(main=(fib.exe`main at main.c:31:1), argc=1, argv=0x0000fffffffffb58) at libc_start_call_main.h:58:16
    frame #12: 0x0000fffff7d78598 libc.so.6`__libc_start_main_impl(main=0x0000aaaaaaba0de0, argc=16, argv=0x000000000000000f, init=<unavailable>, fini=<unavailable>, rtld_fini=<unavailable>, stack_end=<unavailable>) at libc-start.c:360:3
    frame #13: 0x0000aaaaaaaee3b0 fib.exe`_start + 48

You can see the backtrace includes the recursive calls to fib function, the main function in fib.ml, followed by some assembly functions and a number of functions from the OCaml runtime. In between frame #8 and #5 is where the runtime, written in C, switches into assembly to setup the environment to execute the OCaml program. Then we actually enter the OCaml program at frame #5 via camlFib.entry. This function calls initialisation functions for the program and any dependencies like Stdlib that get used.

Examining values

The support for examining OCaml values in LLDB, as you would for say C, is a bit lacking. Not enough information is being emitted by the OCaml compiler to do this yet. So we need to understand how OCaml represents values at runtime and what the OCaml calling conventions are. First we will look at examining values.

Here we are on ARM64 so our registers are named x0-x30 with sp representing the stack pointer. The first 16 arguments are passed in registers, starting from register x0. So the arguments to the fib function should be in the x0 register. We also know that the argument to fib is an integer. OCaml uses 63-bit tagged integers (on 64-bit machines) with the least-significant bit is 1. Given a machine word or a register holding an OCaml integer, the integer value is obtained by right shifting the value by 1.

Putting that all together, we can examine the arguments to fib at the breakpoint in fib like so.

(lldb) p $x0 >> 1
(unsigned long) 5

Given we have a recursive fib function this printing corresponds to fib(5). Have a go at moving up and down the recursive fib calls using up or down and print out the arguments. You can also examine the evaluation order of arguments in fib, noting that the evaluation order of arguments in OCaml is unspecified but 5.2.0 evaluates right-to-left.

Advanced printing

Examining values using bit shifting is tedious. We can do better by writing our own printing functions in Python. The OCaml compiler distribution comes with some scripts to make examining OCaml values in LLDB easier. Note they have historically been used by OCaml maintainers to develop the compiler, so they might be a little rough or missing features (PRs to improve this situation are welcome). With that lets see what we can do.

Since we are using OCaml 5.2.0, we need to get that source code.

# I'm working within ~/projects directory on my machine
$ git clone https://github.com/ocaml/ocaml --branch 5.2.0

Startup a new lldb session, load the lldb script, and get to a breakpoint in the recursive fib calls

lldb ./fib.exe
(lldb) command script import ../ocaml/tools/lldb.py
(lldb) br s -f fib.ml -l 1
Process 12014 launched: '/home/tsmc/projects/fib.exe' (aarch64)
Process 12014 stopped
* thread #1, name = 'fib.exe', stop reason = breakpoint 4.1
    frame #0: 0x0000aaaaaaaf1038 fib.exe`camlFib.fib_270 at fib.ml:2
   1   	(* fib.ml *)
-> 2   	let rec fib n =
   3   	  if n = 0 then 0
   4   	  else if n = 1 then 1
   5   	  else fib (n-1) + fib (n-2)
   6   	
   7   	let main () =

As earlier, the first argument is in x0 register. We can examine the value now with the python script.

(lldb) p (value)$x0
(value) 41 caml:20

value is the type of OCaml values defined in the OCaml runtime. The script tools/lldb.py installs a pretty printer for the values of type value. Here is pretty prints the first argument which is 20

We can also print other kinds of OCaml values. Create this file with some interesting OCaml values:

$ cat test_blocks.ml
(* test_blocks.ml *)

type t = {s : string; i : int}

let main a b =
  print_endline "Hello, world!";
  print_endline a;
  print_endline b.s

let _ = main "foo" {s = "bar"; i = 42}

Now we need to compile it, start an lldb session and break on the main function.

$ ocamlopt -g -o test_blocks.exe test_blocks.ml
$ lldb ./test_blocks.exe
(lldb) target create "./test_blocks.exe"
Current executable set to '/home/tsmc/projects/test_blocks.exe' (aarch64).
(lldb) command script import ../ocaml/tools/lldb.py
OCaml support module loaded. Values of type 'value' will now
print as OCaml values, and an 'ocaml' command is available for
heap exploration (see 'help ocaml' for more information).
(lldb) br s -n camlTest_blocks.main_273
Breakpoint 1: where = test_blocks.exe`camlTest_blocks.main_273 + 40, address = 0x0000000000019ab0
(lldb) run
Process 12043 launched: '/home/tsmc/projects/test_blocks.exe' (aarch64)
Process 12043 stopped
* thread #1, name = 'test_blocks.exe', stop reason = breakpoint 1.1
    frame #0: 0x0000aaaaaaab9ab0 test_blocks.exe`camlTest_blocks.main_273 at test_blocks.ml:4
   1   	type t = {s : string; i : int}
   2   	
   3   	let main a b =
-> 4   	  print_endline "Hello, world!";
   5   	  print_endline a;
   6   	  print_endline b.s
   7   	
(lldb)

Let’s examine the two arguments to main

(lldb) p (value)$x0
(value) 187649984891864 caml(-):'Hello, world!'<13>
(lldb) p (value)$x1
(value) 187649984891808 caml(-):('bar', 42)

What is going on here, didn’t we say the first argument is in x0? What has happened here is our breakpoint has been set a little after we have entered the function and the original value for x0 has been stored on the stack and x0 register has been reused to store arguments to print_endline "Hello, world!";. The second argument in x1 is as expected.

To find the original x0 value we need to look at assembly (don’t worry too much about the specifics of ARM assembly).

(lldb) dis
test_blocks.exe`camlTest_blocks.main_273:
    0xaaaaaaab9a88 <+0>:  ldr    x16, [x28, #0x28]
    0xaaaaaaab9a8c <+4>:  add    x16, x16, #0x158
    0xaaaaaaab9a90 <+8>:  cmp    sp, x16
    0xaaaaaaab9a94 <+12>: b.lo   0xaaaaaaab9a78 ; camlStd_exit.code_end
    0xaaaaaaab9a98 <+16>: sub    sp, sp, #0x20
    0xaaaaaaab9a9c <+20>: str    x30, [sp, #0x18]
    0xaaaaaaab9aa0 <+24>: str    x0, [sp]
    0xaaaaaaab9aa4 <+28>: str    x1, [sp, #0x8]
(lldb) reg r sp
      sp = 0x0000aaaaaab3d160
(lldb) memory read -s8 -fx -l2 0x0000aaaaaab3d160
0xaaaaaab3d160: 0x0000aaaaaab10bc8 0x0000aaaaaab10ba0
0xaaaaaab3d170: 0x0000fffffffff8e0 0x0000aaaaaaab9b38
0xaaaaaab3d180: 0x0000000000000000 0x0000aaaaaaab94fc
0xaaaaaab3d190: 0x0000000000000000 0x0000aaaaaaae05c8
(lldb) p (value)0x0000aaaaaab10bc8
(value) 187649984891848 caml(-):'foo'<3>

The disassembled code is the function prologue code, which is saving x0 onto the stack using str x0, [sp]. To get the original value for x0 we read sp (Stack Pointer), retrieve the data at that address and then print it using value. We get back to our argument passed to main, which was foo and can confirm that by looking at the source code.

Extras

A few useful extras for debugging OCaml programs.

You can set breakpoints based on addresses, this is useful when you know a specific instruction you want to break on. From the previous session, set a breakpoint on the sub sp, sp, #0x20 address.

(lldb) br s -a 0xaaaaaaab9a98
Breakpoint 7: where = test_blocks.exe`camlTest_blocks.main_273 + 16, address = 0x0000aaaaaaab9a98
(lldb) run
There is a running process, kill it and restart?: [Y/n] y
Process 12070 exited with status = 9 (0x00000009) killed
Process 12078 launched: '/home/tsmc/projects/test_blocks.exe' (aarch64)
Process 12078 stopped
* thread #1, name = 'test_blocks.exe', stop reason = breakpoint 7.1
    frame #0: 0x0000aaaaaaab9a98 test_blocks.exe`camlTest_blocks.main_273 at test_blocks.ml:3
   1   	type t = {s : string; i : int}
   2   	
-> 3   	let main a b =
   4   	  print_endline "Hello, world!";
   5   	  print_endline a;
   6   	  print_endline b.s
   7   	
(lldb) p (value)$x0
(value) 187649984891848 caml(-):'foo'<3>

Now we can print out the value of x0 before it gets saved on the stack.

We can also lookup symbols in the executable using image lookup -r -n <symbol_name> if we are not sure of the specific name we want.

(lldb) image lookup -r -n camlTest
4 matches found in /home/tsmc/projects/test_blocks.exe:
        Address: test_blocks.exe[0x0000000000019a78] (test_blocks.exe.PT_LOAD[0]..text + 1912)
        Summary: test_blocks.exe`camlStd_exit.code_end
        Address: test_blocks.exe[0x0000000000019b48] (test_blocks.exe.PT_LOAD[0]..text + 2120)
        Summary: test_blocks.exe`camlTest_blocks.code_end
        Address: test_blocks.exe[0x0000000000019ae0] (test_blocks.exe.PT_LOAD[0]..text + 2016)
        Summary: test_blocks.exe`camlTest_blocks.entry
        Address: test_blocks.exe[0x0000000000019a88] (test_blocks.exe.PT_LOAD[0]..text + 1928)
        Summary: test_blocks.exe`camlTest_blocks.main_273

Finally setting breakpoints on macOS with LLDB is slightly broken so you need to lookup the symbol name and then set the breakpoint based on the address of the symbol. We can combine image lookup with setting breakpoints on addresses to debug on macOS.

More for later

There is a lot more to say about debugging OCaml programs using LLDB and there is ongoing work to improve debugger support in OCaml. Get in touch if you would like to be involved.

Debugging OCaml with Emacs

This post started as a summary of my March Hacking Days effort at Tarides.

I have been working on improving the debugging situation for OCaml and wanted to see how easily I could setup debug support in Emacs using DAP. Debug Adapter Protocol (DAP) is a wire protocol for communicating between an editor or IDE and a debug server like LLDB or GDB, providing an abstraction over debugging, similar to how Language Server Protocol (LSP) provides language support for editors.

OCaml comes with support for debugging native programs with GDB and LLDB, and bytecode code programs using ocamldebug and earlybird. In this post we will cover setting up and debugging both kinds of programs. I am using an M3 Mac so all examples will show ARM64 assembly and macOS specific paths. The same setup should work on Linux. I use prelude to configure my Emacs with my own customistations in .emacs/personal, adjust for your own personal Emacs setup.

Let’s start with the following program to compute Fibonacci sequence:

(* fib.ml *)
let rec fib n =
  if n = 0 then 0
  else if n = 1 then 1
  else fib (n-1) + fib (n-2)

let main () =
  let r = fib 20 in
  Printf.printf "fib(20) = %d" r

let _ = main ()

And this dune configuration in the same directory:

(executable
 (name fib)
 (modules fib)
 (modes exe byte))

And this dune-project configuration in the same directory:

(lang dune 3.11)
(map_workspace_root false)

Create an empty opam switch in same directory and install dune:

$ opam switch create . 5.1.1 --no-install
$ opam install dune

This gives us everything we need to try out all the different debuggers.

Emacs configuration

Emacs has dap-mode that provides everything we need for DAP integration. Install it using M-x package-install and choose the dap-mode package. I have the following lines in my .emacs/personal/init.el that will require the packages we need and setup some convenient key bindings:

; Require dap-mode plus the two extra files we need
(require 'dap-mode)
(require 'dap-codelldb)
(require 'dap-ocaml)

; Setup key bindings using use-package.
(use-package dap-mode
  :bind (("C-c M-n" . dap-next)
         ("C-c M-s" . dap-step-in)
         ("C-c M-a" . dap-step-out)
         ("C-c M-w" . dap-continue)))

Save and restart Emacs, then we can move onto setting up bytecode debugging.

Bytecode debugging

The earlybird project provides DAP support for debugging OCaml bytecode. OCaml has a bytecode compiler that produces portable bytecode executables which can be run with ocamlrun, the interpreter for OCaml bytecode. Earlybird uses the (undocumented) protocol of ocamldebug to communicate with a bytecode executable, inheriting the same functionality as ocamldebug.

Start by installing the earlybird package:

opam install earlybird

Then create a file in .vscode/launch.json with this configuration:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "OCaml earlybird (experimental)",
            "type": "ocaml.earlybird",
            "request": "launch",
            "program": "./_build/default/fib.bc",
            "stopOnEntry": true,
            "cwd": "${workspaceFolder}"
        },
    ]
}

Build the project with dune build to create the fib.bc bytecode file. Finally start a debugger by running M-x dap-debug. It will prompt you to choose a session, we want OCaml earlybird (experimental) from the named configuration above. It will start earlybird and immediately stop it before executing any OCaml code.

To set breakpoints you need to open the OCaml source file in _build/default/fib.ml and click on the source lines you want to stop at. Here is what it looks like after a few recursions. Use the buttons to control the debugger or use the keybindings we added to step through the execution. Curiously they are not pre-defined but here I’ve tried to reuse mappings from ocamldebug.

Native debugging

OCaml can also produce native binaries that can be debugged using native debuggers like GDB or LLDB, depending on your platform. Here we will use LLDB on macOS, but Linux LLDB works too – just change the name of the program you want to debug.

Add another section to .vscode/launch.json for starting lldb.

        {
            "type": "lldb",
            "request": "launch",
            "name": "LLDB with ocamlopt",
            "program": "./fib.exe",
            "args": [],
            "stopOnEntry": true,
            "cwd": "${workspaceFolder}"
        },

Run M-x dap-codelldb-setup which will download the codelldb DAP program that we are using to communicate with LLDB. This gets installed into .extension/vscode/codelldb. Now compile the fib program with ocamlopt -g -o fib.exe fib.ml and startup a debugger session with M-x dap-debug choose the LLDB with ocamlopt option. You should see something similar to:

Now DAP as setup with LLDB and macOS, is a little broken and is missing support for setting breakpoints on symbols and line numbers in source code. Fixes for both will be comming soon. Linux LLDB works better in this scenario. Setting breakpoints using line numbers in source code requires fixes to the OCaml compiler, while setting breakpoints on symbols is supported in codelldb but not exposed into dap-mode.

The second option is debugging native binaries built with Dune, this is slightly different for two reasons. First Dune places the executable into _build/default/fib.exe and second Dune produces slightly different symbols. Start by adding a new section in .vscode/launch.json for Dune built executables:

        {
            "type": "lldb",
            "request": "launch",
            "name": "LLDB with Dune",
            "program": "./_build/default/fib.exe",
            "args": [],
            "stopOnEntry": true,
            "cwd": "${workspaceFolder}"
        },

Remove the old fib.exe in the project directory (dune will complain if you don’t) and run dune build. Startup a new DAP session with M-x dap-debug and choose LLDB with Dune. You should see the same debugger session as before.

Conclusion

Debugging OCaml with DAP inside Emacs is possible. There are working options for both bytecode programs and native programs which work reasonably well.

Use dap-mode with:

(require 'dap-mode)
(require 'dap-codelldb)
(require 'dap-ocaml)

(use-package dap-mode
  :bind (("C-c M-n" . dap-next)
         ("C-c M-s" . dap-step-in)
         ("C-c M-a" . dap-step-out)
         ("C-c M-w" . dap-continue)))

and a launch.json of

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "OCaml earlybird (experimental)",
            "type": "ocaml.earlybird",
            "request": "launch",
            "program": "./_build/default/fib.bc",
            "stopOnEntry": true,
            "cwd": "${workspaceFolder}"
        },
        {
            "type": "lldb",
            "request": "launch",
            "name": "LLDB with Dune",
            "program": "./_build/default/fib.exe",
            "args": [],
            "stopOnEntry": true,
            "cwd": "${workspaceFolder}"
        },
        {
            "type": "lldb",
            "request": "launch",
            "name": "LLDB with ocamlopt",
            "program": "./fib.exe",
            "args": [],
            "stopOnEntry": true,
            "cwd": "${workspaceFolder}"
        }
    ]
}

The same setup will work under VSCode with the CodeLLDB and OCaml Platform extensions installed. Happy Emacs debugging.

Future Work

I’m working on improving the OCaml debugging experience on macOS and Linux. Currently the macOS LLDB experience is behind that on Linux LLDB, so that is the first goal. Then I want to improve the DWARF encodings for OCaml and generally improve the native debugger experience.

ICFP 2022 Review

I wrote up a highlights of ICFP 2022 for the Tarides blog. It was great to get back to in-person conferences again and getting the chance to meet people. Thanks to my employer Tarides for covering the cost.

For me personally the OCaml Workshop was fantastic beginning to end, read the blog post for the full details. Outside of OCaml I spent time in the Haskell Implementors Workshop, hearing about the new features for GHC and excited by the progress that Cabal is making.

My take away research topics are:

• Typed Effect Systems especially Koka.
• Delimited Continuations for both OCaml and Haskell.
• Lockfree data structures, Reagents and STM.

OCaml with Emacs in 2022

I am revisiting my OCaml setup post from 2021 because I needed to setup a new macOS machine. The official OCaml site points newcomers to Visual Studio Code which is a fine choice to get started. However I am using Emacs and have done so for over 20 years, and did not find a good description of how to set things up with Emacs. Here I could digress into why Emacs but I will just strongly encourage any developers to invest heavily in learning their editor with Emacs being a fine choice.

Beginnings

On macOS I use the pre-compiled GUI version of Emacs from emacsformacosx preferring that over compiling it by hand or using the version in homebrew. Both of which I have done previously but find the emacsformacos version saves me time and effort, plus the GUI version was removed from homebrew at some point in the past.

Next I choose to use an Emacs distro over the base Emacs setup, again this is a time saving choice and especially useful if you are new to Emacs. Use Prelude, which is an enhanced Emacs 25.1+ distribution that should make your experience with Emacs both more pleasant and more powerful. It gives a great modern setup for Emacs with minimal fuss. Once that is cloned and installed the Lisp config begins.

Prelude onfiguration

Prelude provides a base experience of packages available with some configuration. The configuration goes into ~/.emacs.d/tsmc/prelude-modules.el where tsmc is your macOS user. The same path would apply for Linux. A sample prelude-modules.ml is provided in https://github.com/bbatsov/prelude/blob/master/sample/prelude-modules.el

I choose the following modules to enable with prelude-lsp and prelude-ocaml being the core OCaml related choices. The other bits are optional but useful for editing lisp or navigating code.

(require 'prelude-ivy) ;; A mighty modern alternative to ido
(require 'prelude-company)
(require 'prelude-emacs-lisp)
(require 'prelude-lisp) ;; Common setup for Lisp-like languages
(require 'prelude-lsp) ;; Base setup for the Language Server Protocol
(require 'prelude-ocaml)

Now for the customisation to get LSP working properly. There are 3 main pieces:

• direnv - for automatically configuring shell environments
• ocaml-lsp-server - the core lsp implementation for OCaml
• lsp-mode - the Emacs mode that drives everything

direnv the necessary magic

direnv is a small program to load/unload environment variables based on $PWD (current working directory). This program ensures that when you open an OCaml file the correct opam switch is chosen and the tools installed in that switch are made available to Emacs. Opam is the OCaml package manager and manages local sandboxes of packages called switches. Without direnv Emacs will not find the correct tools and you would need to mess with Emacs PATHS to get it right. I have done that and it is much simplier with direnv.

So brew install direnv and create a .envrc file in an OCaml project with eval $(opam env --set-switch) inside. Compared to my previous post I have been using local opam switches which exist inside an OCaml project. They are created as opam switch create . 4.14.0 --with-test --deps-only -y and appear as an _opam directory in the project root. Next run direnv allow to tell direnv it is safe to use the .envrc file in this directory. The reason I have switched is I often need to test different OCaml versions so removing the _opam directory and recreating it is the simplier option.

OCaml LSP Server

OCaml LSP server needs to be installed in the current switch so run opam update && opam install ocaml-lsp-server -y, this will make ocaml-lsp-server available to Emacs via direnv.

There is an opportunity here to use Emacs Lisp to install ocaml-lsp-server if it was missing or to allow lsp-mode to download and install it itself. I would like to have this working in future. Next back into Lisp.

Emacs LSP mode

Create a file init.el in ~/.emacs.d/tsmc/ substituting your Unix user name for tsmc. Thanks to emacs-prelude the configuration is very small.

;;; init.el --- @tsmc configuration entry point.

(prelude-require-packages '(use-package direnv))
;; Use direnv to select the correct opam switch and set the path
;; that Emacs will use to run commands like ocamllsp, merlin or dune build.

(use-package lsp-mode
  :hook
  (tuareg-mode . lsp))
;; Attach lsp hook to modes that require it, here we bind to tuareg-mode rather than
;; prelude-ocaml. For unknown reasons the latter does not bind properly and does not
;; start lsp-mode

(provide 'tsmc)
;;; init.el ends here

We require a few packages use-package and direnv, and then tell Emacs to start lsp-mode when tuareg-mode is started. Tuareg-mode is one of the OCaml modes available for Emacs, the other being caml-mode which I have not really used. Now quit and restart Emacs. Opening an ml file inside the project you started earlier and ocaml-lsp should startup.

The types for expressions and modules will display on mouse hover or beside the definition. Hovering the mouse over a function or type will display the type plus the documentation comments for it. A successful dune build for the project is required to generate the data used by ocaml-lsp-server. At this point in time prelude relies on merlin an assistant for editing OCaml code, that is used by ocaml-lsp-server internally but also available as standalone tool. So I often have both installed, opam install merlin should be enough to get it installed too.

At this point I am mostly happy, the types and documentation displays as required. Navigating using M-. shows a preview of the type / function under point and return will take me to the definition. This is vastly improved in OCaml 4.14 (with the work on Shapes) which I have switched to for everything I can. Switching between ml and mli files is C-c C-a and more, simply visit the M-x describe-mode to show everything available.

The annoyances are more fundamental to how LSP wants to work. It uses what I am calling a push based interaction, where it generates the information for types and documentation in the background and pushes it into the Emacs buffer. You never need to ask what is the type, it will display for you. Sometimes I want to ask for what a type is inside an expression, with LSP you are encouraged to mouse hover over something rather than having a key binding for it. So far I haven’t found the lisp function that drives the hover functionality but when I do I will bind it to a key. The second issue is also around mouse usage to drive LSP functionality like rename or annotate types. I would strongly prefer a key chord driven approach to that. Again I will set this up once I find the right lsp functions. For now I use C-c C-t from merlin to summon the types for things.

Overall the experience is solid. Types and docs appear as required. Navigation works. The speed has been good so far. LSP mode is less janky than it was 1 year ago.

Alternatives

There is a fine alternative LSP mode, Eglot for Emacs. It takes a more minimal approach and uses a pull based interaction. Where you ask for the information based on key bindings vs the information being pushed at you via UI elements. For example, the type of a function is requested rather than shown by default.

The corresponding configuration I was using previously is:

(use-package eglot
  :config
  (define-key eglot-mode-map
    (kbd "C-c C-t") #'eldoc-print-current-symbol-info)

  :hook
  ((tuareg-mode . eglot-ensure)))

Again using use-package to configure the mode, the hooks are triggering Eglot to be loaded when tuareg-mode is. Using the eglot-ensure function which starts an Eglot session for current buffer if there isn’t one. No further configuration is needed in Emacs as Eglot knows the LSP server is called ocamllsp and will look for it on the Unix PATH.

Summary

Getting started with OCaml using Emacs can be a struggle. Emacs is a fine editor but the documentation can be difficult to handle. Hopefully following through this setup will yield a working Emacs / LSP setup for OCaml.

In future I want to try binding more things to keys so I use the mouse less and streamline the installing of the ocaml lsp server. Then after that adding support for more interesting code interactions like extracting modules or hoisting let bindings would be nice to have. Happy hacking!

Getting Started with OCaml in 2021

OCaml is an awesome language with many fine features. I enjoy using it immensely!

Unfortunately, it suffers from a perceived weakness in how to get started. Like any new skill, there can be a learning curve. The tools are all there, but combining them for a good developer experience might seem difficult at first.

Often I’ve found that the barrier for getting into a new langauge is less about the new features of that language and more about learning the tools to become productive in that language. The package managers, build tools, and editor integration of a new language can be confusing, making for an awful experience.

Perhaps my opinionated guide to getting started with OCaml in 2021 will help reduce any mental blocks against trying out this excellent language.

Install Opam

First it’s necessary to install OCaml and Opam. Opam is the default package manager for OCaml projects. Ignore the other options for now, once you know more about what you want, you can make an informed choice. For now if you speak OPAM, you’ll get the most out of the community.

On Linux, use your local package manger, e.g., apt-get install opam for Debian and apt install opam for Ubuntu. For MacOS, use homebrew brew install opam. I’ll assume if you run something else, you can handle looking up how to install things.

On my Mac I get Opam 2.1.0:

$ opam --version
2.1.0

Once you’ve got Opam installed, you should be able to move on to the next step.

Choose an OCaml Version

I strongly recommended that you pick a single OCaml version that your project will compile against. Supporting multiple compiler versions is possible and usually not too diffcult, but it complicates the process right now.

Running opam switch list-available will show you a long list of every possible OCaml compiler. Choose the latest mainline compiler identifed by Official release X.XX.X where currently the latest is 4.13.0. Ignore the others.

opam switch list-available
...
ocaml-variants                         4.12.0+domains                         OCaml 4.12.0, with support for multicore domains
ocaml-variants                         4.12.0+domains+effects                 OCaml 4.12.0, with support for multicore domains and effects
ocaml-variants                         4.12.0+options                         Official release of OCaml 4.12.0
ocaml-base-compiler                    4.12.1                                 Official release 4.12.1
ocaml-variants                         4.12.1+options                         Official release of OCaml 4.12.1
ocaml-variants                         4.12.2+trunk                           Latest 4.12 development
ocaml-base-compiler                    4.13.0~alpha1                          First alpha release of OCaml 4.13.0
ocaml-variants                         4.13.0~alpha1+options                  First alpha release of OCaml 4.13.0
ocaml-base-compiler                    4.13.0~alpha2                          Second alpha release of OCaml 4.13.0
ocaml-variants                         4.13.0~alpha2+options                  Second alpha release of OCaml 4.13.0
ocaml-base-compiler                    4.13.0~beta1                           First beta release of OCaml 4.13.0
ocaml-variants                         4.13.0~beta1+options                   First beta release of OCaml 4.13.0
ocaml-base-compiler                    4.13.0~rc1                             First release candidate of OCaml 4.13.0
ocaml-variants                         4.13.0~rc1+options                     First release candidate of OCaml 4.13.0
ocaml-base-compiler                    4.13.0~rc2                             Second release candidate of OCaml 4.13.0
ocaml-variants                         4.13.0~rc2+options                     Second release candidate of OCaml 4.13.0
ocaml-base-compiler                    4.13.0                                 Official release 4.13.0
ocaml-variants                         4.13.0+options                         Official release of OCaml 4.13.0
ocaml-variants                         4.13.1+trunk                           Latest 4.13 developmet
ocaml-variants                         4.14.0+trunk                           Current trunk
...

At this point, install the latest OCaml 4.13.0:

$ opam switch create 4.13.0

<><> Installing new switch packages <><><><><><><><><><><><><><><><><><><><>  🐫
Switch invariant: ["ocaml-base-compiler" {= "4.13.0"} | "ocaml-system" {= "4.13.0"}]

<><> Processing actions <><><><><><><><><><><><><><><><><><><><><><><><><><>  🐫
∗ installed base-bigarray.base
∗ installed base-threads.base
∗ installed base-unix.base
∗ installed ocaml-options-vanilla.1
⬇ retrieved ocaml-base-compiler.4.13.0  (https://opam.ocaml.org/cache)
∗ installed ocaml-base-compiler.4.13.0
∗ installed ocaml-config.2
∗ installed ocaml.4.13.0
Done.

You can start using this version by typing the following:

$ opam switch set 4.13.0

And verify which switch you are using:

$ opam switch show
4.13.0

When you work with several OCaml projects, it’s best to create a switch per project, as it keeps each project isolated and prevents issues with installing conflicting versions of libraries. For example, I use a naming scheme of ocaml-version-project-name, e.g., 4.13.0-ocurrent. Then in each project directory, run opam switch link 4.13.0-ocurrent to setup that named switch for that specific directory. Opam will take care of setting that switch in your shell when you change into that directory.

Creating Your Project Directory

For this step we need the Dune build tool, so go ahead and install it with opam install dune. Dune comes with a simple scaffolding command to create an empty project that is really useful to get started.

I’m calling my project box, so run:

$ dune init proj box
Success: initialized project component named box

In the project generated, we get a library component, a CLI, and a test component, which will all compile out of the box.

$ cd box
$ tree
.
├── bin
│   ├── dune
│   └── main.ml
├── box.opam
├── lib
│   └── dune
└── test
    ├── box.ml
    └── dune

3 directories, 6 files

Lets try a compile:

$ dune build @all
Info: Creating file dune-project with this contents:
| (lang dune 2.8)
| (name box)

Running the CLI:

$ dune exec bin/main.exe
Hello, World!

Each of the bin, lib, and test directories contains the source code in the form of *.ml files, along with a dune file which tells Dune how to build the source and on what libraries it depends. The box bin\dune file declares it’s an executable with a name box and depends on the box library.

(executable
 (public_name box)
 (name main)
 (libraries box))

Adding a Dependency

CLI tools require command line parsing, Cmdliner is a common library that implements CLI parsing. We need to add it in two places: first in the dune-project file, to get it installed, and then in bin/dune, to say where we’re using it.

One small digression, when generating our project, dune created an box.opam file. This describes our project to Opam, telling it what libraries it requires and what the project does. You need this if you ever publish a package for other people to use. Newer versions of Dune can generate the box.opam file from a dune-project file. Having a single source of information is helpful, so lets create that file:

(lang dune 2.8)
(name box)

(generate_opam_files true)

(package
 (name box)
 (depends
  (ocaml (>= 4.13.0))
  (cmdliner (>= 0.9.8)))
 (synopsis "Box cli"))

Remove the rm box.opam file to test the generation. Now run dune build @all to regenerate the Opam file. This file should be checked in, and any further edits should be at the top-level dune-project file, which should look like this:

$ cat box.opam
# This file is generated by dune, edit dune-project instead
opam-version: "2.0"
synopsis: "Box cli"
depends: [
  "dune" {>= "2.8"}
  "ocaml" {>= "4.13.0"}
  "cmdliner" {>= "0.9.8"}
  "odoc" {with-doc}
]
build: [
  ["dune" "subst"] {dev}
  [
    "dune"
    "build"
    "-p"
    name
    "-j"
    jobs
    "@install"
    "@runtest" {with-test}
    "@doc" {with-doc}
  ]
]

The final step is to actually install the cmdliner library. Run opam install . --deps-only -ty, which will look at the *.opam files present and install just their dependencies with the correct version bounds. The -y says yes to installing the packages. You can remove it if you like by pressing Y or if you want to review what will be installed. -t will run the package tests, which isn’t always necessary, but it’s sometimes useful for certain packages with native C components.

Alternatively you could run opam install cmdliner, as this doesn’t look at version constraints in *.opam files, you might not get what you expect.

Editor Tooling

Finally, you’ll want to get comfy with your chosen editor. If you have a preference, you should use the native LSP support in that editor, along with installing opam install ocaml-lsp-server. OCaml is standardising on the LSP protocol for editor interaction. If you have no editor preference, then start with VSCode and install the OCaml LSP package from the Marketplace.

Personally, I’m using Emacs with the LSP mode eglot, which works really nicely, along with some customisations to bind certain LSP actions to keys. I highly recommend getting into Emacs as an editor because the customisation via a fully-featured language, like Lisp, is fantastic if you live in your editor like I do.

This post is an update to an earlier post by Adam in 2017, and I hope this short tutorial helps get you started with OCaml!