From 52717d3e703052c3e5880482d3429def61c3affe Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 6 May 2026 10:02:19 +0300 Subject: Update content for md --- gemfeed/DRAFT-unveiling-ior-ng-part-1.md | 216 +++++++++++++ gemfeed/DRAFT-unveiling-ior-ng-part-2.md | 214 +++++++++++++ gemfeed/DRAFT-unveiling-ior-ng-part-3.md | 205 ++++++++++++ gemfeed/DRAFT-unveiling-ior-ng.md | 345 --------------------- gemfeed/unveiling-ior-ng/13a-order-by-process.png | Bin 0 -> 231292 bytes gemfeed/unveiling-ior-ng/13b-order-by-path.png | Bin 0 -> 218540 bytes gemfeed/unveiling-ior-ng/13c-order-by-syscall.png | Bin 0 -> 255276 bytes gemfeed/unveiling-ior-ng/13d-order-by-pid.png | Bin 0 -> 183662 bytes .../13e-order-by-process-paths.png | Bin 0 -> 214308 bytes 9 files changed, 635 insertions(+), 345 deletions(-) create mode 100644 gemfeed/DRAFT-unveiling-ior-ng-part-1.md create mode 100644 gemfeed/DRAFT-unveiling-ior-ng-part-2.md create mode 100644 gemfeed/DRAFT-unveiling-ior-ng-part-3.md delete mode 100644 gemfeed/DRAFT-unveiling-ior-ng.md create mode 100644 gemfeed/unveiling-ior-ng/13a-order-by-process.png create mode 100644 gemfeed/unveiling-ior-ng/13b-order-by-path.png create mode 100644 gemfeed/unveiling-ior-ng/13c-order-by-syscall.png create mode 100644 gemfeed/unveiling-ior-ng/13d-order-by-pid.png create mode 100644 gemfeed/unveiling-ior-ng/13e-order-by-process-paths.png (limited to 'gemfeed') diff --git a/gemfeed/DRAFT-unveiling-ior-ng-part-1.md b/gemfeed/DRAFT-unveiling-ior-ng-part-1.md new file mode 100644 index 00000000..21b9f7eb --- /dev/null +++ b/gemfeed/DRAFT-unveiling-ior-ng-part-1.md @@ -0,0 +1,216 @@ +# Unveiling I/O Riot NG — Part 1: a guided tour + +> Draft — not in the gemfeed yet. Promote with the usual rename + index dance. + +I rewrote I/O Riot. The old one was C + Systemtap and dates from 2017. The new one — call it ior — is Go + C + BPF via libbpfgo, runs on Linux, and is mostly a TUI dashboard rather than a record/replay box. Since pictures are worth more than yet another README table of key bindings, I am presenting some examples in this post. + +This is the first of three posts. Part 1 is the demo-driven tour: what ior looks like, how the dashboard tabs work, how the live flamegraph reads, how filtering and recording work. Part 2 covers installing it on a fresh Rocky Linux 9 box and the "compile once, run everywhere" story underneath that — eBPF, CO-RE, libbpfgo, static linking, and why a 23 MB binary you build on one machine just runs on every other Linux host you scp it to. Part 3 is the under-the-hood companion — the per-event schema, the syscall-coverage probe generator, async-syscall caveats, and post-mortem SQL on the parquet output. The three are independent; read them in any order. + +[![ior's live flamegraph: every running process, by file path, by syscall — width = event volume](./unveiling-ior-ng/00-hero-flamegraph.png "ior's live flamegraph: every running process, by file path, by syscall — width = event volume")](./unveiling-ior-ng/00-hero-flamegraph.png) + + +[I/O Riot NG on Codeberg](https://codeberg.org/snonux/ior) +[the original I/O Riot post (2018)](./2018-06-01-realistic-load-testing-with-ioriot-for-linux.md) + +## Table of Contents + +* [⇢ Unveiling I/O Riot NG — Part 1: a guided tour](#unveiling-io-riot-ng--part-1-a-guided-tour) +* [⇢ ⇢ What it does](#what-it-does) +* [⇢ ⇢ First launch](#first-launch) +* [⇢ ⇢ The seven tabs, in 30 seconds each](#the-seven-tabs-in-30-seconds-each) +* [⇢ ⇢ ⇢ `2` Overview](#2-overview) +* [⇢ ⇢ ⇢ `3` Syscalls](#3-syscalls) +* [⇢ ⇢ ⇢ `4` Files](#4-files) +* [⇢ ⇢ ⇢ `5` Processes](#5-processes) +* [⇢ ⇢ ⇢ `6` Latency + Gaps](#6-latency--gaps) +* [⇢ ⇢ ⇢ `7` Stream](#7-stream) +* [⇢ ⇢ The Stream tab is the good one](#the-stream-tab-is-the-good-one) +* [⇢ ⇢ Filtering, more thoroughly](#filtering-more-thoroughly) +* [⇢ ⇢ Recording](#recording) +* [⇢ ⇢ Installing it](#installing-it) +* [⇢ ⇢ What's still missing](#what-s-still-missing) + +## What it does + +ior attaches BPF tracepoints to a chunk of the synchronous-I/O syscall surface — open, read, write, stat, mmap, sync, link, fcntl, dup, the obvious ones — plus the async ones (`io_uring_*`, `aio_*`). Each enter/exit pair becomes an event with a duration plus an inter-syscall gap, and the events feed a Go Bubble Tea dashboard with seven tabs: a live flamegraph, an overview, sortable per-syscall / per-file / per-process tables, latency histograms, and a live event stream with a stackable filter UI on top. + +Same shape as the old I/O Riot in spirit: capture what the system is actually doing, not synthetic load. Different shape in execution: no replay engine, no separate record file unless you ask for one, no kernel-debug-info dance. + +[![I/O Riot NG logo](./unveiling-ior-ng/00-logo.png "I/O Riot NG logo")](./unveiling-ior-ng/00-logo.png) + +## First launch + +```sh +sudo ./ior +``` + +You land on the PID picker. The default selection is "All PIDs", so Enter just dumps you straight at the dashboard. + +[![Cold start: PID picker, then the dashboard](./unveiling-ior-ng/01-launch.gif "Cold start: PID picker, then the dashboard")](./unveiling-ior-ng/01-launch.gif) + +The dashboard opens on the live flamegraph. Bars grow as new events arrive. Before walking through the keys, a paragraph on what you're looking at — flamegraphs are easier to read than they are to describe: + +A flamegraph is a histogram of stacks. Each horizontal bar is one entry in a stack; every bar directly above it is a child of that entry, and the stack you read top-to-bottom is the same shape as a call chain. In ior, "stack" doesn't mean function-call stack (we don't have userspace symbols). It means a tuple of dimensions of the trace: by default `comm/path/tracepoint`, so the bottom row is per-process names, the middle row is per-file paths, and the top row is the syscall (`enter_read`, `enter_openat`, etc.). A wide bar means lots of events landed in that bucket, a narrow bar means few. There is no time axis — left-to-right is just sort order, not chronology. The whole chart is one "where is the I/O coming from?" picture. + +One thing worth flagging because it's the unusual bit: this flamegraph is live. Most of the flamegraph tooling out there — Brendan Gregg's `flamegraph.pl`, all the `perf script | stackcollapse-* | flamegraph.pl` pipelines, every `pprof -web` invocation — produces a static SVG: capture a profile for N seconds, render once, browse the result. ior's tab is not that. Bars grow, shrink, appear, and disappear in real time as events stream in from the kernel — at full screen-refresh rate while the workload runs, with no pause. You can sit on this tab while you change something on the system (start a build, cycle a service, run a query) and watch the I/O shape mutate underneath you. That's a different mental model from the static "I have a profile, let me look at it" workflow most people are used to, and it's what makes the tab actually useful as an at-a-glance diagnostic surface rather than a post-mortem artifact. + +Because it's live, there's also a way to throw away the accumulated history and start the rolling count from "now": `r` resets the baseline. Everything the flamegraph has been counting since launch (or since the last reset) is dropped, and from that moment the chart reflects only events that arrived after the reset. Useful for the "compare before vs after" workflow — change one thing on the box, hit `r` immediately, and the next thirty seconds of accumulation is a fresh picture of the new state. + +That visualisation buys you two things you can't easily get from a tabular view. First, hierarchy: it's immediately obvious whether one process is responsible for ten thousand reads on a single file, or ten thousand reads spread across a hundred files — the first looks like one tall pillar, the second looks like a wide ridge. Second, scale: bar width is proportional to the metric (count or bytes), so a process that did 95% of the work towers over the others. The eye picks that up instantly; the same fact in a sorted table requires reading numbers and doing the ratio mentally. + +Useful workflows you can do entirely from this tab: + +* "What's pounding the disk?" — leave it on default order (`comm/path/tracepoint`) and watch which `comm` widens. Press `b` once to switch the metric to bytes if you care about throughput, not call count. +* "Why is this one process slow?" — `l` (or `→`) until the cursor is on that process, then `enter` to zoom. The whole chart re-roots there and you see only that process's paths and syscalls. +* "What's in /var/lib/X?" — press `o` once to flip ordering to `path/tracepoint/comm`, navigate to the path, zoom. Now the children show which syscalls hit it and which processes did them. +* "Did the new deploy change the I/O shape?" — press `r` to reset the baseline, wait a bit, and the chart starts fresh with only events from the reset point onward. Pair the same syscall surface "before" vs "after" and the difference jumps out by shape. + +Now the keys. Movement uses vi-style `h`/`j`/`k`/`l` everywhere in ior — and the cursor keys work too if you'd rather. `h`/`l` (or `←`/`→`) walk siblings at the current depth, `j`/`k` (or `↓`/`↑`) step shallower or deeper. `enter` zooms into the selected subtree (the rest of the chart greys out and the selection becomes the new root). `u` or `Esc` undoes the zoom. `b` toggles the metric driving bar width between event count and total bytes. `/` opens regex search; matching frames stay coloured while everything else greys out, so you can use it as a filter as well as a finder. And `o` cycles between five different stack-ordering modes, each with its own lens on the data. + +The five orderings ship as built-in presets. Read each preset name as bottom→top: the leftmost dimension is what you'll see lined up across the bottom of the chart (the root row), the next one up is its children, and the rightmost is the top row (the leaf). Switching the order changes which dimension you're scanning first when your eye starts at the bottom. + +You change ordering with the `o` hotkey, on the fly, while the trace is still running. No restart, no reset, no re-recording — `o` just rebuilds the live chart with the next preset and keeps streaming new events into it. Press it once to flip from "processes at the bottom" to "paths at the bottom" the moment you realise you'd rather slice the data the other way; press it again to keep cycling. The toolbar updates immediately to show the new `o:order(...)` value. Pressing `o` rotates through the presets in this order: + +Concrete screenshots of each preset on the same workload follow each description, so you can see how the same trace data reshapes itself depending on the lens. + +`comm/tracepoint/path` (default) — processes at the bottom, syscalls in the middle, file paths on top. Each comm bar at the root splits into the syscalls it issued, and each syscall splits further into the files it touched. Best general-purpose view: "which programs are doing the I/O, and what kind?" + +[![Ordering 1: comm/tracepoint/path — processes at the bottom (cos…, find, head, notify-rs i…, sh, sqlx-sqlite, xar)](./unveiling-ior-ng/13a-order-by-process.png "Ordering 1: comm/tracepoint/path — processes at the bottom (cos…, find, head, notify-rs i…, sh, sqlx-sqlite, xar)")](./unveiling-ior-ng/13a-order-by-process.png) + +`path/tracepoint/comm` — file paths at the bottom, syscalls in the middle, processes on top. Use this when you suspect a particular file or directory is hot — pick the path, see which syscalls hit it, and which processes did those syscalls. Pairs naturally with directory grouping in the Files tab. + +[![Ordering 2: path/tracepoint/comm — file paths at the bottom (/null, /etc/home, /lib64, /tmp, /usr, /share, /paul, /libc, …)](./unveiling-ior-ng/13b-order-by-path.png "Ordering 2: path/tracepoint/comm — file paths at the bottom (/null, /etc/home, /lib64, /tmp, /usr, /share, /paul, /libc, …)")](./unveiling-ior-ng/13b-order-by-path.png) + +`tracepoint/comm/path` — syscalls at the bottom, processes in the middle, file paths on top. When you already know "this is an `openat` problem" or "we're write-bound", this view collects all the openat (or write) traffic into one bar at the root and lets you drill into who's doing it and to which paths. + +[![Ordering 3: tracepoint/comm/path — syscalls at the bottom (enter_close, enter_fcntl, enter_mmap, enter_newfstatat, enter_openat, enter_read, …)](./unveiling-ior-ng/13c-order-by-syscall.png "Ordering 3: tracepoint/comm/path — syscalls at the bottom (enter_close, enter_fcntl, enter_mmap, enter_newfstatat, enter_openat, enter_read, …)")](./unveiling-ior-ng/13c-order-by-syscall.png) + +`pid/tracepoint/path` — PIDs at the bottom, syscalls in the middle, file paths on top. Same shape as the default but each individual process gets its own root bar instead of being lumped in with siblings sharing a comm. Useful when you have many bash or python instances and need to tell them apart by ID. + +[![Ordering 4: pid/tracepoint/path — PIDs at the bottom (103…, 1…, 1338410, 4113, 4263514, 7709, 8272, …)](./unveiling-ior-ng/13d-order-by-pid.png "Ordering 4: pid/tracepoint/path — PIDs at the bottom (103…, 1…, 1338410, 4113, 4263514, 7709, 8272, …)")](./unveiling-ior-ng/13d-order-by-pid.png) + +`comm/path/tracepoint` — processes at the bottom, file paths in the middle, syscalls on top. Inverse of the default in the upper two layers: you see processes, then which files they hit, then which syscalls hit each file. Best when you care about "what files does this program touch?" more than "what syscalls does it issue?". + +[![Ordering 5: comm/path/tracepoint — processes at the bottom, file paths layered above instead of syscalls](./unveiling-ior-ng/13e-order-by-process-paths.png "Ordering 5: comm/path/tracepoint — processes at the bottom, file paths layered above instead of syscalls")](./unveiling-ior-ng/13e-order-by-process-paths.png) + +In every ordering the rule is the same: scan the bottom row to pick a "by what?" dimension, then walk up to drill in. Bar widths always mean the same thing — proportion of the active metric (events or bytes, toggled with `b`). The toolbar at the top of the chart always shows the current ordering as `o:order(//)`, so you never lose track of which lens you're looking through. + +[![Live in-TUI flamegraph: navigate, zoom, undo, cycle order + metric](./unveiling-ior-ng/13-tui-flamegraph.gif "Live in-TUI flamegraph: navigate, zoom, undo, cycle order + metric")](./unveiling-ior-ng/13-tui-flamegraph.gif) + +## The seven tabs, in 30 seconds each + +The number keys jump between tabs. `tab` and `shift+tab` step. + +### `2` Overview + +A sparkline plus the top syscalls and top paths — the at-a-glance view, useful as a "what's happening right now?" landing tab when you don't yet know what you're looking for. + +[![Overview tab](./unveiling-ior-ng/02-overview-tab.gif "Overview tab")](./unveiling-ior-ng/02-overview-tab.gif) + +### `3` Syscalls + +A sortable table of every syscall ior knows about, with rate, average latency, p95/p99, total bytes, and error count. `s` sorts by the selected column, `S` reverses. The most useful column when something's wrong is usually p99 — it's where you see the long-tail outlier syscall types. + +[![Syscalls table with sort + reverse-sort](./unveiling-ior-ng/03-syscalls-tab.gif "Syscalls table with sort + reverse-sort")](./unveiling-ior-ng/03-syscalls-tab.gif) + +### `4` Files + +Same shape as Syscalls but rows are file paths. The interesting key here is `d`: it rolls per-file rows up into their parent directory. Essential when you've got a process touching ten thousand files in `/usr/share/` — without it the table is unreadable noise. + +[![Directory grouping toggle](./unveiling-ior-ng/04-files-tab.gif "Directory grouping toggle")](./unveiling-ior-ng/04-files-tab.gif) + +### `5` Processes + +Same shape again, but rows are processes / comms. Best paired with the Stream tab — once you spot a culprit comm here, push it to the global filter with `Enter` and the rest of the dashboard is scoped to that process. + +[![Processes tab](./unveiling-ior-ng/05-processes-tab.gif "Processes tab")](./unveiling-ior-ng/05-processes-tab.gif) + +### `6` Latency + Gaps + +Two histograms side by side: how long each syscall took (latency), and the wall-clock interval between syscalls on the same thread (gap). Latency tells you "is the kernel slow"; gap tells you "what is the program doing between two kernel calls". + +A subtle but important point about that gap: ior measures it from the exit of one syscall to the entry of the next on the same TID, but it does not know what the thread was doing in the meantime. A long gap doesn't mean the thread was idle — it might have been pinned on a CPU running pure userspace code (number-crunching, JSON parsing, GC, a busy loop). All "gap" tells you for sure is "this thread didn't call into the kernel for X microseconds." Whether that's because it was sleeping, blocked on a condition variable, computing, or scheduled out is something only the gap value alone cannot answer — pair it with `top`/`perf top` if you need to disambiguate. In practice this is still extremely useful: a syscall-driven workload with surprisingly long gaps is a strong hint that you're CPU-bound somewhere outside the kernel, and that's a different optimisation conversation than slow I/O. + +The dd loop in the demo workload spreads the latency distribution out so you can actually see the shape. + +[![Latency + gap histograms](./unveiling-ior-ng/06-latency-gaps-tab.gif "Latency + gap histograms")](./unveiling-ior-ng/06-latency-gaps-tab.gif) + +### `7` Stream + +The live tail — every event as it happens, in a row-per-event ring buffer. This is where you spend most of your time when something's actually broken. The whole next section is about it. + +[![Stream tab live-tailing](./unveiling-ior-ng/07-stream-live.gif "Stream tab live-tailing")](./unveiling-ior-ng/07-stream-live.gif) + +## The Stream tab is the good one + +`space` pauses. In pause mode, the same vi-style `h`/`j`/`k`/`l` (or arrow keys) move the row/column cursor across the table. Hitting `Enter` on a cell pushes a new filter onto a stack, narrowing what you see. Pile them up — comm, then syscall, then file — and `Esc` pops them off LIFO when you want to back out. + +[![Pause, push two filters, undo with Esc](./unveiling-ior-ng/08-stream-pause-filter.gif "Pause, push two filters, undo with Esc")](./unveiling-ior-ng/08-stream-pause-filter.gif) + +`/` and `?` are regex search forward/backward. `n` and `N` walk matches. The search runs against every column in the ring buffer and wraps at the end. Search and filtering are different beasts: search highlights and jumps, filtering hides everything that doesn't match. + +[![Regex search](./unveiling-ior-ng/09-stream-regex-search.gif "Regex search")](./unveiling-ior-ng/09-stream-regex-search.gif) + +`e` exports the current filtered snapshot to a CSV in the working directory. `x` does the same for the paused stream view specifically (preserving your filter stack), `X` prompts for a filename, `E` opens the most recent export in `$EDITOR`. + +[![CSV export](./unveiling-ior-ng/10-stream-csv-export.gif "CSV export")](./unveiling-ior-ng/10-stream-csv-export.gif) + +## Filtering, more thoroughly + +The Enter-to-push trick isn't unique to Stream. It works the same on Files, Syscalls, and Processes: highlight a row, hit Enter, and the cell value becomes a filter against the entire dashboard. Three tabs of "I see one weird path / comm / syscall, drill in" with one keystroke. + +The filter status line gives you a one-glance summary of every active frame, written like: + +* `comm~bash` — substring match on a string column. This is what Enter-on-a-cell produces for `comm`, `syscall`, and `file`. +* `pid=1234` — exact equality. Used for `pid`, `tid`, `fd`, `ret`, `bytes`. +* `latency>=5ms` / `gap>=10us` — numeric comparison with a duration suffix. The full operator set is `>`, `<`, `=`, `>=`, `<=`, `!=`. + +Stack frames AND together, so pushing `comm~bash` and then `syscall~openat` shows you bash's openat calls, not bash OR openat. + +Undoing is symmetric to pushing: `Esc` pops the most recent frame off the stack — one keystroke per layer, LIFO. Press it once to drop the `syscall~openat` filter and you're back to bash-only; press it again and the `comm~bash` filter goes too, leaving the unfiltered firehose. To clear the whole stack at once, just hold `Esc` until the status line reads `filter: all`. The `F` key is a synonym for `Esc` here and works from any tab — handy from Files/Syscalls/Processes where `Esc` might otherwise close a modal first. + +Two other knobs do related work: + +* `p`, `t`, `o` open the PID, TID, and probe-toggle dialogs. These are global filters: they reconfigure the BPF side, so kernel-level events for excluded PIDs/probes never even reach userspace. Cheaper than filtering a firehose, but it also means the filter applies to recordings (the parquet file only contains rows the kernel let through). +* The CLI mirrors of those dialogs let you bake the same scoping into a one-shot run: `-pid`, `-tid`, `-comm`, `-path`, plus `-tps ` / `-tpsExclude ` for picking which tracepoints to attach in the first place. + +[![PID, TID, and probe pickers](./unveiling-ior-ng/11-pid-tid-probe.gif "PID, TID, and probe pickers")](./unveiling-ior-ng/11-pid-tid-probe.gif) + +## Recording + +Three persistence flows, each for a different job: + +* `R` from the dashboard starts streaming Parquet — every event row that survives your current TUI filter goes to disk continuously. `R` again stops. Footer shows the active file or the last error. + +[![Parquet recording from the TUI](./unveiling-ior-ng/12-parquet-recording.gif "Parquet recording from the TUI")](./unveiling-ior-ng/12-parquet-recording.gif) + +* `sudo ./ior -flamegraph -name ` writes one aggregated `.ior.zst` artifact at shutdown. Aggregated counters, not per-event rows. Cheaper to write, ideal for ior's native flamegraph workflow and integration tests. + +* `sudo ./ior -parquet trace.parquet` is the headless firehose — every row, no TUI, no filtering. `sudo ./ior -plain` is even lighter: CSV to stdout, pipe it into anything. + +[![All three headless flows in one tape](./unveiling-ior-ng/14-headless-modes.gif "All three headless flows in one tape")](./unveiling-ior-ng/14-headless-modes.gif) + +Once a parquet file is on disk, point any SQL-over-parquet tool at it — Part 3 walks through ClickHouse Local, with real query output against a 30-second capture. + +## Installing it + +The Rocky Linux 9 install procedure (with its kernel-backport caveat), the eBPF / CO-RE / static-linking explanation that makes the resulting binary "compile once, run everywhere", and the build host vs. trace host split all live in Part 2. The short version: you only have to build it once. The 23 MB binary then `scp`s to anywhere. + +[Part 2: install + compile once, run everywhere](./DRAFT-unveiling-ior-ng-part-2.md) + +## What's still missing + +ior is pre-alpha and basically a personal tool. The headline gaps: + +* No record/replay — that was the whole point of the original I/O Riot. The new one is a tracer, not a workload simulator. I keep going back and forth on whether to put replay back in. +* No userspace symbol resolution. Stacks are at the syscall surface, not "which line of which library called read". +* No remote / cluster mode. Single host, one trace at a time. + +But the live flamegraph, the stackable stream filters, and the cheap parquet capture together cover the cases I actually hit week to week. The demo above is the easiest way to get a feel for whether it's the kind of tool you want. + +For installing it and the eBPF / CO-RE / static-linking story (why one build runs on every other Linux box you scp it to), see Part 2. For the per-event schema, async-syscall caveats, the probe-generator safeguard against missing new kernel syscalls, and post-mortem SQL on the parquet output, see Part 3. + +[Part 2: install + compile once, run everywhere](./DRAFT-unveiling-ior-ng-part-2.md) +[Part 3: under the hood (schema, probe generator, ClickHouse)](./DRAFT-unveiling-ior-ng-part-3.md) +[Source on Codeberg](https://codeberg.org/snonux/ior) +[The full in-repo tutorial](https://codeberg.org/snonux/ior/src/branch/main/demo/TUTORIAL.md) diff --git a/gemfeed/DRAFT-unveiling-ior-ng-part-2.md b/gemfeed/DRAFT-unveiling-ior-ng-part-2.md new file mode 100644 index 00000000..56885fcc --- /dev/null +++ b/gemfeed/DRAFT-unveiling-ior-ng-part-2.md @@ -0,0 +1,214 @@ +# Unveiling I/O Riot NG — Part 2: install and compile once, run everywhere + +> Draft — not in the gemfeed yet. Promote with the usual rename + index dance. + +This is Part 2 of three. Part 1 is the demo-driven tour — what ior looks like, how the dashboard tabs work, how filtering and recording behave. This part is about the install dance for Rocky Linux 9 (with one annoying kernel-backport caveat) and, more interestingly, why you only have to do that dance on a single machine: the resulting binary is portable to every other Linux box thanks to CO-RE — Compile Once, Run Everywhere — plus full static linking. Part 3 is the under-the-hood companion (per-event schema, async-syscall caveats, the syscall-coverage probe generator, and post-mortem SQL on the parquet output). + +If you came here for the dashboard tour, that's Part 1. If you want to know how the data pipeline is shaped after you've got ior running, that's Part 3. This one is for the moment between "I want to try this" and "OK, it's running on the box I care about." + +[Part 1: a guided tour](./DRAFT-unveiling-ior-ng-part-1.md) +[Part 3: under the hood (schema, probe generator, ClickHouse)](./DRAFT-unveiling-ior-ng-part-3.md) +[I/O Riot NG on Codeberg](https://codeberg.org/snonux/ior) +[the original I/O Riot post (2018)](./2018-06-01-realistic-load-testing-with-ioriot-for-linux.md) + +[![I/O Riot NG logo](./unveiling-ior-ng/00-logo.png "I/O Riot NG logo")](./unveiling-ior-ng/00-logo.png) + + +## Table of Contents + +* [⇢ Unveiling I/O Riot NG — Part 2: install and compile once, run everywhere](#unveiling-io-riot-ng--part-2-install-and-compile-once-run-everywhere) +* [⇢ ⇢ Installing ior](#installing-ior) +* [⇢ ⇢ ⇢ Why native installation is a mess](#why-native-installation-is-a-mess) +* [⇢ ⇢ ⇢ What the Docker build is actually doing](#what-the-docker-build-is-actually-doing) +* [⇢ ⇢ A short detour: eBPF and libbpfgo](#a-short-detour-ebpf-and-libbpfgo) +* [⇢ ⇢ CO-RE — the part that makes the binary actually portable](#co-re--the-part-that-makes-the-binary-actually-portable) +* [⇢ ⇢ A note on cgo overhead](#a-note-on-cgo-overhead) +* [⇢ ⇢ If you want to go deeper](#if-you-want-to-go-deeper) +* [⇢ ⇢ Wrapping up](#wrapping-up) + +## Installing ior + +The short answer: use Docker. One command, no toolchain setup, works from any Docker-capable Linux host with BTF available: + +```sh +git clone https://codeberg.org/snonux/ior ~/git/ior +cd ~/git/ior +mage buildDocker +``` + +First run builds a Rocky Linux 9 builder image (~15–20 minutes). Subsequent runs reuse the cached image and finish in under a minute. The resulting static binary lands at `./ior`. + +That's the officially supported install path, and it's the right one for anyone who just wants to run ior without living in its build system. + +### Why native installation is a mess + +If you're curious why Docker became the answer, the native install on Rocky Linux 9 illustrates the problem well. Three separate things bite you before you even get to `mage build`: + +Rocky 9 ships neither `libelf.a` nor `libzstd.a` — there are no `*-static` subpackages for either, only the dynamic `.so` files. Both have to be compiled from source. `libelf` from the elfutils source RPM, `libzstd` from the upstream GitHub release tarball. + +Rocky 9 also only ships Go 1.25.x, but ior requires 1.26+. So Go itself has to be installed from go.dev in parallel with the library builds. + +And there's a kernel quirk that used to make this section much longer. Pre-fix, ior would happily load on a stock 5.14 RHEL kernel and then die on the very first tracepoint attach with `BPF_LINK_CREATE`/`BPF_PERF_EVENT` returning `EACCES` — even as root, with SELinux permissive, with every BPF-related sysctl wide open. The cause is that RHEL 9 carries an `rt`-tree backport that adds `preempt_lazy_count` to `struct trace_entry`. That widens the BTF-emitted alias `trace_event_raw_sys_enter`/`_exit` by 8 bytes and shifts the `args`/`ret` offsets — but the actual context the kernel hands the BPF program is still `struct syscall_trace_enter`/`_exit`, where the offsets did not move. Programs written against `trace_event_raw_sys_*` (the conventional choice; bcc, libbpf-tools, and ior all used to do this) end up reading past `max_ctx_offset`, so the verifier rejects the attach. The fix — also what bcc shipped in [PR #4920](https://github.com/iovisor/bcc/pull/4920) and what inspektor-gadget did — is to type the BPF context as `syscall_trace_enter`/`_exit` directly. ior now generates its handlers that way, and stock 5.14 RHEL/Rocky/Alma works without an ElRepo kernel. + +### What the Docker build is actually doing + +The Dockerfile encodes exactly the same steps that a native install on Rocky 9 would require. Here is the full sequence so you have a mental model of what's inside the image, and so you could reproduce it on a bare host if you ever needed to: + +```sh +# 1) Enable repos and install build dependencies. CRB ships zlib-static / glibc-static. +sudo dnf config-manager --set-enabled crb +sudo dnf install -y epel-release +sudo dnf install -y gcc clang bpftool elfutils-libelf-devel zlib-static \ + glibc-static libzstd-devel git make cmake wget rpmdevtools strace bpftrace +sudo dnf builddep -y elfutils + +# 2) Install Go 1.26 from go.dev. Rocky 9 ships only Go 1.25.x, ior needs 1.26+. +cd /tmp +wget -q https://go.dev/dl/go1.26.2.linux-amd64.tar.gz +sudo tar -C /usr/local -xf go1.26.2.linux-amd64.tar.gz +echo 'export PATH=/usr/local/go/bin:$HOME/go/bin:$PATH' | sudo tee /etc/profile.d/go.sh +source /etc/profile.d/go.sh + +# 3) Build libelf.a from the elfutils source RPM. +mkdir -p ~/src && cd ~ +dnf download --source elfutils-libelf +rpm -ivh elfutils-*.src.rpm +tar -C ~/src -xjf rpmbuild/SOURCES/elfutils-*.tar.bz2 +cd ~/src/elfutils-* +./configure --enable-deterministic-archives --disable-debuginfod --disable-libdebuginfod +make -C lib -j$(nproc) +make -C libelf -j$(nproc) +sudo cp -v libelf/libelf.a /usr/lib64/ + +# 4) Build libzstd.a from upstream (libzstd-devel doesn't ship the static archive). +cd /tmp +wget -q https://github.com/facebook/zstd/releases/download/v1.5.5/zstd-1.5.5.tar.gz +tar xzf zstd-1.5.5.tar.gz +make -C zstd-1.5.5/lib -j$(nproc) libzstd.a +sudo cp -v zstd-1.5.5/lib/libzstd.a /usr/lib64/ + +# 5) Clone ior + libbpfgo, pin libbpfgo, build the static libbpf archive, install mage. +mkdir -p ~/git +git clone https://codeberg.org/snonux/ior ~/git/ior +git clone https://github.com/aquasecurity/libbpfgo ~/git/libbpfgo +git -C ~/git/libbpfgo checkout v0.9.2-libbpf-1.5.1 +git -C ~/git/libbpfgo submodule update --init --recursive +make -C ~/git/libbpfgo libbpfgo-static +go install github.com/magefile/mage@latest + +# 6) Generate the syscall-coverage handlers against THIS kernel and build. +# IOR_FORCE_GENERATE bypasses the strict diff against the committed audit file +# (the committed audit was generated against a different kernel build, and the +# generator's safeguard would otherwise refuse to overwrite it). +cd ~/git/ior +env IOR_FORCE_GENERATE=1 GOTOOLCHAIN=auto mage generate +env GOTOOLCHAIN=auto mage all + +# 7) Smoke test. +sudo ./ior -plain -duration 5 +``` + +If you see `Probing for 5s` followed by CSV rows, the build is good. `mage buildDocker` runs all of this inside a container and hands you back just the final binary — the 15-minute first-run cost buys you never having to think about any of the above again. + +## A short detour: eBPF and libbpfgo + +If you haven't touched eBPF before: it's a small in-kernel bytecode VM. You compile a tiny C program, the kernel verifies it can't crash or loop forever, and then it runs every time some hook fires — a syscall enter/exit, a kprobe, a tracepoint, a network packet. The program writes events into a ring buffer that userspace mmaps and drains. No kernel module, no patched kernel, no debug symbols required. + +ior plugs into the syscall tracepoints — `sys_enter_openat`, `sys_exit_read`, etc. — and the BPF side does the bare minimum: timestamp the event, copy a few fields, push to a perf ring buffer. All the heavy lifting (string interning, latency math, aggregation, the dashboard) is in Go on the userspace side. + +The kernel ships a C library called libbpf that handles loading the program, attaching it to hooks, managing maps, and reading the ring buffer. There are two well-known ways to drive that from Go: + +* libbpfgo (Aqua Security): a thin cgo wrapper around libbpf. You ship libbpf along with your binary and call into the same C API that `bpftool` and `perf` use. +* cilium/ebpf: a from-scratch pure-Go reimplementation of everything libbpf does — ELF parser, BTF resolver, syscall layer, the lot. + +I went with libbpfgo specifically because it's a wrapper, not a reimplementation. Whatever lands in libbpf upstream — new map types, new attach kinds, CO-RE fixes — I get for free the next kernel cycle. The pure-Go variant has to chase libbpf's feature set in parallel, and any divergence is on me to debug. For a tracer that's mostly value-add on the userspace side, "be a thin client of the kernel's own library" wins. + +## CO-RE — the part that makes the binary actually portable + +The headline fact about ior's deployment story: build it once on one box, then `scp ior other-host:/usr/local/bin/` to anywhere else and it just runs. No recompile per kernel, no kernel-debuginfo dance, no DKMS hooks. Two mechanisms make that work, and they reinforce each other. + +The first is plain old static linking on the userspace side. A quick refresher on what that means, since it's central to why "scp the binary anywhere" works: when you build a normal Linux executable, the linker has two ways to wire library code into your program. Dynamic linking ("shared library") leaves a placeholder in the binary that says "at run time, find `libfoo.so.6` somewhere on `LD_LIBRARY_PATH` and pull in its symbols." Static linking pastes the library's machine code directly into your binary at build time, so there's nothing to look up later. Dynamic is smaller on disk and lets distros patch shared libs without rebuilding everything; static is bigger but self-contained — no surprise about which version of the library the target box happens to have, no `error while loading shared libraries: libwhatever.so.6: cannot open shared object file` when the target ships a newer ABI. + +For Go, this is mostly a non-issue. A pure-Go binary (no cgo) is statically linked by default — the Go toolchain produces a single self-contained ELF file with no `.dynamic` section and no `NEEDED` entries. You can `scp` it to any Linux box of the same architecture and it just runs. That's one of the quietly nice things about Go. + +ior is the not-quite-pure case: it goes through cgo to call into libbpf, libelf, and libzstd, and each of those has its own .so on the build host. By default cgo links those C dependencies dynamically, which would defeat the "scp the binary anywhere" property — the target box would need to have matching `.so` files at matching versions, which is exactly the kind of dependency hell Go usually saves you from. The fix is the line `-extldflags "-static"` in ior's Magefile: it tells the external (C) linker to resolve `-lbpf -lelf -lzstd -lz` against the static archives (`.a` files) instead of the dynamic ones. That's why the install procedure above is so picky about having `libelf.a` and `libzstd.a` actually present on the build host — without them the C-side static link fails outright. + +The result is a single ~23 MB binary with libbpf, libelf, libzstd, and zlib all baked in. None of them are looked up dynamically at runtime. The build host's library versions stay on the build host. (A couple of glibc resolver functions — `getpwnam_r` and friends — do still fall back to the target's libc, which is fine on any reasonable distro and is what the linker warnings during the build are about.) + +The second, and the one that's actually unusual, is CO-RE — Compile Once, Run Everywhere. CO-RE is the eBPF feature that solves the "the kernel changed its struct layout between releases" problem. + +The old I/O Riot was Systemtap. Systemtap programs are translated into a kernel module against the running kernel's exact headers, and that module then has to be loaded with `insmod`. That meant: the user has to install a kernel-debuginfo package matching their running kernel, and a fresh build per host (or per kernel update). On the BSD-style "you only run what you compiled here" laptop crowd that was tolerable; on a fleet of distros + kernel versions it was a recurring tax. Half of the original I/O Riot's README was about kernel-debuginfo dance steps. + +CO-RE throws all of that out. The idea, in one paragraph: when you write a BPF program that reads `task->mm->start_stack`, you don't bake the offsets of those fields into the compiled program. Instead, the compiler emits relocation records ("at this instruction, fetch the offset of `mm` inside `task_struct`"). At load time, libbpf looks up the actual offsets in the target kernel's BTF (BPF Type Format — a description of every kernel struct, embedded in `/sys/kernel/btf/vmlinux` on any modern kernel) and patches the program in place. The same `.bpf.o` that ran on a 5.10 Debian kernel runs on a 6.8 Fedora kernel without recompilation. + +Pictorially, the contrast looks like this: + +``` +Old I/O Riot (Systemtap) New ior (libbpf + CO-RE) +───────────────────────── ──────────────────────────── + .stp source .bpf.c source + │ │ + │ needs THIS kernel's headers │ build ONCE against vmlinux.h + │ + debuginfo package installed │ (generated from any kernel BTF) + ▼ ▼ + per-host translate + compile one portable .bpf.o + │ │ + ▼ ▼ + per-host kernel module same binary on every host + │ │ + insmod / modprobe libbpf loader: + │ │ • read /sys/kernel/btf/vmlinux + ▼ │ • patch field offsets + attached, this kernel only │ • verify + load + ▼ + attached, runs anywhere +``` + +What that buys ior in practice: I ship a single `ior` binary. On any Linux ≥4.18-ish with BTF available (which is almost all of them now — Debian, Ubuntu, Fedora, Arch, RHEL, and ElRepo's `kernel-ml` builds all ship `CONFIG_DEBUG_INFO_BTF=y` by default), it just works. No kernel-debuginfo dependency, no per-kernel build matrix, no DKMS hooks. The first time I tried `scp ior fedora-box:` and it ran without complaint after a 6-month gap I had to double-check it wasn't silently doing nothing. + +So the operational shape is: pick one box, do the install dance from the Rocky section above (or the Fedora one in the README) once, build, then distribute the 23 MB binary wherever you want to trace. The build host needs Go and clang and the static libraries. The trace hosts need a BTF-enabled kernel and `sudo`. That's it. + +The runtime shape of a trace pipeline lines up with that: + +``` + kernel side userspace (this binary) + ─────────── ─────────────────────── + ┌──────────────────────┐ + tracepoint: │ Go process │ + sys_enter_openat │ ┌────────────────┐ │ + │ │ │ aggregator │ │ + ▼ │ │ (latency, │ │ + ┌─────────┐ │ │ stacks, │ │ + │ BPF prog│ ─── perf ring buf ──────────>│──│ filters) │ │ + │ (verified │ └─────┬──────────┘ │ + │ bytecode) │ │ │ + └─────────┘ │ ▼ │ + │ Bubble Tea TUI / │ + │ parquet writer / │ + │ CSV stdout │ + └──────────────────────┘ +``` + +## A note on cgo overhead + +The cost of being a libbpf wrapper rather than a pure-Go reimplementation is cgo. Every call from Go into libbpf crosses the cgo boundary, which historically meant tens to ~hundred-ish nanoseconds of overhead per call — register save/restore, a stack switch onto g0, goroutine state bookkeeping. Cheap in absolute terms, but it adds up if you call into C inside a tight loop. ior keeps the actual hot path on the kernel side and only crosses into Go once per drained batch of events from the ring buffer, so the per-call cost is amortized over thousands of events. In practice it doesn't show up in profiles. + +Go 1.26, the current release at the time of writing (early May 2026), is the one that finally took a serious bite out of cgo's per-call cost — the runtime can elide a chunk of the bookkeeping for calls that don't need it. Real-world wins depend heavily on the workload, but the rough direction is that cgo now feels closer to "an unusually expensive function call" than to "a context switch", which is the right mental model for almost everyone touching a C library from Go. The shorter version: cgo overhead used to be a real footgun for ports that called into C in the inner loop. With Go 1.26 it's a footnote unless you're doing many millions of small calls per second, in which case batching across the boundary still fixes it. + +## If you want to go deeper + +If any of this sounds interesting and you want to learn how to write your own BPF programs, two books are the standard recommendations and both well worth the time: + +* "Learning eBPF" by Liz Rice (O'Reilly, 2023) is the friendlier on-ramp. It walks through writing your first programs end-to-end, covers CO-RE and BTF in plain English, and is the book I'd hand to someone who has never touched the kernel side before. Liz also gave the canonical "what is eBPF" conference talk floating around YouTube, which makes a good 40-minute companion. +* "BPF Performance Tools: Linux System and Application Observability" by Brendan Gregg (Addison-Wesley, 2019) is the encyclopedia. It's where you go after you've understood the basics and now want a complete reference for tracing every subsystem in the kernel — file systems, networking, scheduler, languages, applications — with worked tools for each. The flame-graph-driven analysis style throughout is also exactly how ior's own flamegraph tab thinks about a workload. + +Between the two, Rice teaches you the moving parts and Gregg teaches you what to do with them. + +## Wrapping up + +That's the install dance and the why-it's-portable story. Part 3 is the bottom of the data stack — what's actually in each event row, the syscall-coverage safeguard against new kernels, async-syscall caveats, and how to query the parquet output with ClickHouse Local. Part 1, if you haven't read it, is the demo-driven tour with all the GIFs of the dashboard. + +[Part 1: a guided tour](./DRAFT-unveiling-ior-ng-part-1.md) +[Part 3: under the hood (schema, probe generator, ClickHouse)](./DRAFT-unveiling-ior-ng-part-3.md) +[Source on Codeberg](https://codeberg.org/snonux/ior) +[The full in-repo tutorial](https://codeberg.org/snonux/ior/src/branch/main/docs/tutorial/tutorial.md) diff --git a/gemfeed/DRAFT-unveiling-ior-ng-part-3.md b/gemfeed/DRAFT-unveiling-ior-ng-part-3.md new file mode 100644 index 00000000..49a17daa --- /dev/null +++ b/gemfeed/DRAFT-unveiling-ior-ng-part-3.md @@ -0,0 +1,205 @@ +# Unveiling I/O Riot NG — Part 3: under the hood + +> Draft — not in the gemfeed yet. Promote with the usual rename + index dance. + +This is the third and final post in the series. Part 1 is the demo-driven tour — what ior looks like, how the dashboard tabs work, how the live flamegraph reads, how filtering and recording behave. Part 2 covers the install dance for Rocky Linux 9 and the "compile once, run everywhere" portability story (eBPF, CO-RE, libbpfgo, static linking). This one is the part you read when you've got ior running and want to know what's actually in the data: the per-event schema, the safeguard that keeps syscall coverage current as new kernels ship, async-syscall caveats, and what to do with the parquet output once it's on disk. + +If you haven't read Part 1, it's not a hard prerequisite — but the screenshots and key bindings referenced here all live there. Part 2 is also independent of this one; you can read them in either order. + +[Part 1: a guided tour](./DRAFT-unveiling-ior-ng-part-1.md) +[Part 2: install + compile once, run everywhere](./DRAFT-unveiling-ior-ng-part-2.md) +[I/O Riot NG on Codeberg](https://codeberg.org/snonux/ior) +[the original I/O Riot post (2018)](./2018-06-01-realistic-load-testing-with-ioriot-for-linux.md) + +[![I/O Riot NG logo](./unveiling-ior-ng/00-logo.png "I/O Riot NG logo")](./unveiling-ior-ng/00-logo.png) + + +## Table of Contents + +* [⇢ Unveiling I/O Riot NG — Part 3: under the hood](#unveiling-io-riot-ng--part-3-under-the-hood) +* [⇢ ⇢ What ior actually captures per event](#what-ior-actually-captures-per-event) +* [⇢ ⇢ ⇢ Async syscalls and what "latency" means for them](#async-syscalls-and-what-latency-means-for-them) +* [⇢ ⇢ Keeping up with new syscalls](#keeping-up-with-new-syscalls) +* [⇢ ⇢ Querying a parquet trace with ClickHouse](#querying-a-parquet-trace-with-clickhouse) +* [⇢ ⇢ Asking an AI to do the reading for you](#asking-an-ai-to-do-the-reading-for-you) +* [⇢ ⇢ Wrapping up](#wrapping-up) + +## What ior actually captures per event + +Every traced syscall produces one row of structured data. The schema is what the parquet file (and the in-memory ring buffer behind the dashboard) stores, and it covers all the dimensions you'd want for a post-mortem: + +* `seq` — monotonically increasing sequence number, useful for joining/ordering across analysis tools. +* `time_ns` — wall-clock timestamp at syscall entry, in nanoseconds since boot. +* `latency_ns` — how long the syscall took, measured from `sys_enter_X` to `sys_exit_X` of the matching pair. +* `gap_ns` — wall-clock interval since the previous syscall on the same TID (the userspace-side breathing room — see Part 1's Latency+Gaps tab section for the caveat about what "gap" actually means). +* `comm` — the program's `task->comm` string at the time of the syscall (16 chars max, that's a kernel limit, hence truncations like "notify-rs inoti" in the demo screenshots). +* `pid`, `tid` — process and thread IDs. +* `syscall` — the syscall name, e.g. `read`, `openat`, `mkdir`. +* `fd` — the file descriptor passed in (or returned by `open*`). +* `ret` — the syscall return value: number of bytes for read/write, the new fd for open*, 0 or a negative errno otherwise. This is also where exit codes for failed calls live — anything `< 0` is a `-errno`. +* `bytes` — the byte count classified by direction: bytes read for read-class syscalls, bytes written for write-class syscalls, bytes transferred for `sendfile`/`splice`/`copy_file_range`. Lets you answer "who is hogging disk throughput?" in one query. +* `file` — the file path, where the kernel knows it. From `openat` it's the literal path argument; from `read`/`write` on an existing fd it's resolved via the fd-to-path map ior maintains in BPF. +* `is_error` — boolean shortcut: true iff `ret < 0`. Saves you a `WHERE` clause in 90% of queries. +* `filter_epoch` — bookkeeping for the global filter UI; you can ignore it for offline analysis. + +Aggregations the dashboard derives from this raw row stream — counts per syscall/comm/path, rolling rates, per-syscall latency histograms, gap histograms, top-N tables — are all just GROUP BYs over those columns. + +### Async syscalls and what "latency" means for them + +ior attaches to enter+exit tracepoints for every file-I/O syscall, including the asynchronous ones — `io_uring_enter`, `io_uring_register`, `aio_*`, `sync_file_range`, and so on. Coverage is the same as for blocking syscalls: enter event, exit event, latency = exit − enter. + +The catch is that for an async syscall, that latency does not mean what you'd intuitively expect. The whole point of an async submission is that the kernel returns immediately while the actual work — the read, the write, the fsync — runs in the background and reports completion later through a different channel (a `cqe` for io_uring, a signal or `aio_suspend` poll for POSIX AIO). So when ior tells you `io_uring_enter` took 4 µs, that's the time spent inside the kernel function ferrying submission queue entries — not the time the storage device spent doing the I/O. Those completions land separately, often on a different thread, and their timing isn't paired with the original enter event. Don't read the latency histogram for `io_uring_enter` as "io_uring is fast" — it's a different question entirely. + +The flip side is that ior's per-row throughput numbers (`bytes` summed over a window) still hold for async ops, because the kernel reports the byte count at submission for the cases where it's known up front. So "what processes are dispatching the most async I/O" is a fine question to ask. "How long do those async I/Os actually take" is one ior cannot currently answer — you'd want a per-completion tracepoint pair to do that, which is on the someday-maybe list. + +## Keeping up with new syscalls + +One of the persistent problems with the original 2017 I/O Riot was that the syscall coverage was a hand-maintained list. Every kernel release added new entry points; some of them were file I/O, some weren't, and there was no automated way to spot the new arrivals. After a couple of years the list had silently rotted — entire syscalls were missing from traces, and which ones was difficult to detect because nothing flagged them. + +The new ior solves this with a code generator that runs against the kernel itself. `mage generate` reads every `/sys/kernel/tracing/events/syscalls/*/format` file on the host (which is the kernel's authoritative, runtime-correct list of every syscall it knows about), parses each one, and runs it through a classifier that decides what kind of file I/O surface it is — fd-based, path-based, dup-style, fcntl-style, async-completion, etc. The recognized ones get a generated BPF handler emitted into `internal/c/generated_tracepoints.c`; the unrecognized ones get a comment line like `/// Ignoring sys_enter_X sys_exit_X as possibly not file I/O related` in the same file. + +The safeguard piece is that the list of ignored syscalls is also extracted into a checked-in audit file — `internal/c/generated_tracepoints_result.txt` — and `mage generate` diffs the new run against the committed copy. If a future kernel adds a syscall ior hasn't seen before, that diff will show it, and a strict-mode regen will fail the build until a human reviews the new entry. So either the new syscall is genuinely not file-I/O (drop a one-line ignore rule, commit the diff) or it is (extend the classifier, commit the new handler). Either way it's not a silent miss. + +The current numbers are 234 active tracepoint handlers (117 enter+exit pairs) and 249 ignored syscalls, all enumerated in that one generated file. Compared to the old I/O Riot, where coverage was "however much of the kernel I happened to remember to type out", that's a meaningful step forward. + +(I hit this safeguard in practice during the Rocky Linux 9 install in Part 2: the committed audit was generated against a newer kernel, and the strict diff against Rocky's 5.14 refused to overwrite it. The `IOR_FORCE_GENERATE=1` env var skips the strict check and regenerates against the live kernel, which is the right thing on a fresh build host.) + +## Querying a parquet trace with ClickHouse + +The schema is flat and stable: `seq, time_ns, gap_ns, latency_ns, comm, pid, tid, syscall, fd, ret, bytes, file, is_error, filter_epoch`. ClickHouse Local reads parquet directly without a server, which makes it a perfect post-mortem tool — point it at the file and run SQL: + +```sh +clickhouse local --query " + SELECT comm, syscall, count() AS n, + formatReadableSize(sum(bytes)) AS total + FROM file('trace.parquet', Parquet) + GROUP BY comm, syscall + ORDER BY n DESC + LIMIT 10 +" --format PrettyCompactNoEscapes +``` + +``` + ┌─comm────────────┬─syscall─┬─────n─┬─total──────┐ + 1. │ notify-rs inoti │ read │ 42005 │ 732.31 KiB │ + 2. │ cosmic-term │ statx │ 10898 │ 0.00 B │ + 3. │ cosmic-term │ read │ 10103 │ 4.02 MiB │ + 4. │ surface-eDP-1 │ ioctl │ 8452 │ 0.00 B │ + 5. │ cosmic-term │ close │ 4918 │ 0.00 B │ + 6. │ cosmic-term │ openat │ 4537 │ 0.00 B │ + 7. │ cosmic-term │ ioctl │ 3556 │ 0.00 B │ + 8. │ tokio-runtime-w │ read │ 1976 │ 4.04 MiB │ + 9. │ cosmic-comp │ read │ 1118 │ 6.63 KiB │ +10. │ systemd-oomd │ read │ 1085 │ 111.97 KiB │ + └─────────────────┴─────────┴───────┴────────────┘ +``` + +The fields you actually want for performance work are `latency_ns` and `gap_ns`. P99 by syscall, only the ones that landed in error: + +```sh +clickhouse local --query " + SELECT syscall, count() AS n, + round(quantile(0.5)(latency_ns)/1000, 1) AS p50_us, + round(quantile(0.99)(latency_ns)/1000, 1) AS p99_us + FROM file('trace.parquet', Parquet) + WHERE is_error = 1 + GROUP BY syscall + ORDER BY p99_us DESC +" --format PrettyCompactNoEscapes +``` + +``` + ┌─syscall────┬─────n─┬─p50_us─┬─p99_us─┐ + 1. │ statx │ 1216 │ 2.2 │ 16.4 │ + 2. │ newfstatat │ 69 │ 1.7 │ 16.4 │ + 3. │ open │ 1 │ 16.1 │ 16.1 │ + 4. │ mkdir │ 306 │ 3.9 │ 11.7 │ + 5. │ readlink │ 11 │ 1.5 │ 10.4 │ + 6. │ newstat │ 44 │ 2.5 │ 8.4 │ + 7. │ unlinkat │ 347 │ 1 │ 6.2 │ + 8. │ openat │ 380 │ 2.1 │ 5.8 │ + 9. │ access │ 2 │ 5 │ 5.5 │ +10. │ read │ 23597 │ 0.5 │ 5.4 │ +11. │ ioctl │ 901 │ 1 │ 5.3 │ +12. │ writev │ 1 │ 0.7 │ 0.7 │ + └────────────┴───────┴────────┴────────┘ +``` + +Real output, by the way — those rows are from a 30-second `ior -parquet trace.parquet` capture on the laptop I'm typing this on. `notify-rs inoti…` is the inotify thread of some Rust app I had open; `cosmic-term` is the COSMIC desktop's terminal emulator. The slowest p99 errors are the directory-walking syscalls (statx, newfstatat, mkdir) at ~16 µs — bog standard. + +Same trick works in DuckDB (`duckdb -c "SELECT ... FROM 'trace.parquet'"`), pandas, polars, anything that reads Parquet. The point of streaming Parquet rather than ior's native `.ior.zst` format is exactly this: once it's on disk, you're in the standard data-tools ecosystem. + +## Asking an AI to do the reading for you + +Parquet is great if you already have an angle of attack. Sometimes you don't — you just want to know "what's hammering this box right now, and is any of it interesting?" That's where pasting a chunk of trace into an LLM and asking it to look turns out to be a surprisingly effective first pass. ior's `-plain` mode emits CSV, which is exactly the format AI assistants are best at chewing through: + +```sh +sudo ior -plain -duration 15 > iotrace.csv +``` + +The CSV has a one-line ASCII banner at the top followed by the column header `durationToPrevNs,durationNs,comm,pid.tid,name,ret,notice,file` and one row per syscall. (Inside the row the `comm@pid.tid`, `name=>ret` and `file%(fd,flags)` sub-encodings keep the column count low; any half-decent assistant will figure those out unprompted.) + +Hand the file to an AI with a prompt like: + +> Here is 15 seconds of file-I/O syscall trace from my Fedora laptop captured with `ior -plain`. Tell me which processes are dominant, which syscalls are the noisiest, where the wall-clock time is going, and flag anything suspicious — high error rates, pointless retries, weird hot files. Use the CSV columns as documented. + +What I got back on the trace I captured while writing this section (260,547 rows, 15 s, idle COSMIC desktop with htop and a tmux session open) was the following — reproduced verbatim, just to show you what the shape of the answer looks like: + +``` +TOP COMMANDS BY EVENT COUNT + htop 130,500 events, 243.9 ms total, 8,855 errors + notify-rs inoti 49,346 events, 42.1 ms total, 24,610 errors + cosmic-term 39,534 events, 76.4 ms total, 2,590 errors + systemd-oomd 5,857 events, 10.0 ms total, 947 errors + surface-eDP-1 3,535 events, 127.0 ms total, 317 errors ← high µs/event + +TOP SYSCALLS BY TOTAL TIME (ms) + read 241.3 ms over 122,767 calls (avg 1.97 µs) + ioctl 138.0 ms over 8,416 calls (avg 16.4 µs) + openat 91.8 ms over 39,684 calls (avg 2.31 µs) + statx 19.2 ms over 12,441 calls + getdents 17.6 ms over 6,770 calls + +ERROR RATES THAT STOOD OUT + mkdir 100.0% (345/345) — cosmic-term recreating config dirs that already exist (EEXIST) + newfstatat 84.3% (1,816/2,153) — systemd-oomd statting /proc//ns/pid for already-gone processes + read 24.3% (29,816/122,767) — almost all on anon_inode:inotify and [eventfd], i.e. EAGAIN on + non-blocking fds. Normal, but it's a quarter of all reads. + +NOISIEST PATHS (per ENOENT) + 174× /var/lib/flatpak/exports/share/cosmic/com.system76.CosmicTheme.Mode/v1 + 174× /usr/local/share/cosmic/com.system76.CosmicTheme.Mode/v1 + 174× /home/paul/.local/share/flatpak/exports/share/cosmic/com.system76.CosmicTheme.Mode/v1 + → cosmic-term walks the XDG search path on every redraw. ENOENT-spam, not a perf bug, but worth filing. + +WHAT'S ACTUALLY EXPENSIVE + • htop is the loudest tenant by event count but cheap per-event (~1.9 µs). + It walks /proc/*/{task,stat,maps,statm} every refresh — expected. + • surface-eDP-1 is the sleeper: ~36 µs/event, almost entirely DRM ioctls + on /dev/dri/renderD128 + /dev/dri/card1. That's the GPU compositor path. + • 24% of read() calls returning -EAGAIN is normal for an event-loop heavy + desktop (inotify + eventfd on every Tokio runtime), but if you're chasing + syscall overhead, batching those would matter. + +NOTHING SUSPICIOUS — no unexpected processes, no surprise paths outside /proc, +/sys, /dev/dri, ~/.config and the Cosmic/Flatpak search dirs. +``` + +Two things this is unreasonably good at. First, joining the dots between `comm`, `syscall` and `path` columns to give you a one-paragraph "this is what your machine is doing" — the kind of summary that would take you ten ClickHouse queries to assemble by hand. Second, flagging things that are statistically weird without you having to know what to look for: the 100%-error `mkdir` was the EEXIST loop in cosmic-term, the 84%-error `newfstatat` was systemd-oomd racing process exits, the surface-eDP-1 outlier was the only entry on the list with high per-event latency. + +A few caveats worth knowing before you rely on this: + +* The CSV file gets big fast. Fifteen seconds of an idle desktop produced 260k rows / ~25 MB of CSV; on a busy server you'll want to either trim with `-comm`/`-path`/`-pid` filters at capture time or hand the AI a `head -100000` slice. Most assistants will hit context limits well before the file ends. +* The model is reading text, not running SQL. It will round, it will sometimes miscount the long tail, and it cannot tell you a true p99 from a 250k-row sample without writing code. Treat the output as a lead-generator: it points you at suspects, then you confirm with ClickHouse on the parquet file. +* For deeper questions ("what changed between these two traces?", "which pids dominate during the 12:34 spike?") an agentic assistant that can run shell commands does much better — it'll write the awk/clickhouse query itself, run it, and feed the result back into its own analysis. + +The combination that's worked best for me in practice: capture parquet with `-parquet`, capture a parallel `-plain` CSV slice for the AI to read, ask the AI for a triage pass, then drill into the suspects with ClickHouse on the parquet file. Triage and ground-truth, in that order. + +## Wrapping up + +That's the bottom of the stack. For the dashboard surface — what ior looks like, how the seven tabs behave, how filtering and recording work in practice — Part 1 is the demo-driven tour with all the GIFs. For the install dance and the why-the-binary-is-portable story (eBPF, CO-RE, static linking), Part 2 is the install + portability companion. + +[Part 1: a guided tour](./DRAFT-unveiling-ior-ng-part-1.md) +[Part 2: install + compile once, run everywhere](./DRAFT-unveiling-ior-ng-part-2.md) +[Source on Codeberg](https://codeberg.org/snonux/ior) +[The full in-repo tutorial](https://codeberg.org/snonux/ior/src/branch/main/demo/TUTORIAL.md) diff --git a/gemfeed/DRAFT-unveiling-ior-ng.md b/gemfeed/DRAFT-unveiling-ior-ng.md deleted file mode 100644 index 304d92aa..00000000 --- a/gemfeed/DRAFT-unveiling-ior-ng.md +++ /dev/null @@ -1,345 +0,0 @@ -# Unveiling I/O Riot NG - -> Draft — not in the gemfeed yet. Promote with the usual rename + index dance. - -I rewrote I/O Riot. The old one was C + Systemtap and dates from 2017. The new one — call it ior — is Go + C + BPF via libbpfgo, runs on Linux, and is mostly a TUI dashboard rather than a record/replay box. Since pictures are worth more than yet another README table of key bindings, I built a demo. - -[![ior's live flamegraph: every running process, by file path, by syscall — width = event volume](./unveiling-ior-ng/00-hero-flamegraph.png "ior's live flamegraph: every running process, by file path, by syscall — width = event volume")](./unveiling-ior-ng/00-hero-flamegraph.png) - -[I/O Riot NG on Codeberg](https://codeberg.org/snonux/ior) -[the original I/O Riot post (2018)](./2018-06-01-realistic-load-testing-with-ioriot-for-linux.md) - -## Table of Contents - -* [⇢ Unveiling I/O Riot NG](#unveiling-io-riot-ng) -* [⇢ ⇢ What it does](#what-it-does) -* [⇢ ⇢ A short detour: eBPF and libbpfgo](#a-short-detour-ebpf-and-libbpfgo) -* [⇢ ⇢ ⇢ CO-RE — the part that makes ior actually portable](#co-re--the-part-that-makes-ior-actually-portable) -* [⇢ ⇢ ⇢ If you want to go deeper](#if-you-wan