diff options
Diffstat (limited to 'gemfeed')
| -rw-r--r-- | gemfeed/DRAFT-unveiling-ior-ng-part-1.html (renamed from gemfeed/DRAFT-unveiling-ior-ng.html) | 231 | ||||
| -rw-r--r-- | gemfeed/DRAFT-unveiling-ior-ng-part-2.html | 245 | ||||
| -rw-r--r-- | gemfeed/DRAFT-unveiling-ior-ng-part-3.html | 239 | ||||
| -rw-r--r-- | gemfeed/unveiling-ior-ng/13a-order-by-process.png | bin | 0 -> 231292 bytes | |||
| -rw-r--r-- | gemfeed/unveiling-ior-ng/13b-order-by-path.png | bin | 0 -> 218540 bytes | |||
| -rw-r--r-- | gemfeed/unveiling-ior-ng/13c-order-by-syscall.png | bin | 0 -> 255276 bytes | |||
| -rw-r--r-- | gemfeed/unveiling-ior-ng/13d-order-by-pid.png | bin | 0 -> 183662 bytes | |||
| -rw-r--r-- | gemfeed/unveiling-ior-ng/13e-order-by-process-paths.png | bin | 0 -> 214308 bytes |
8 files changed, 529 insertions, 186 deletions
diff --git a/gemfeed/DRAFT-unveiling-ior-ng.html b/gemfeed/DRAFT-unveiling-ior-ng-part-1.html index 634bdaf8..7c73c92a 100644 --- a/gemfeed/DRAFT-unveiling-ior-ng.html +++ b/gemfeed/DRAFT-unveiling-ior-ng-part-1.html @@ -2,35 +2,34 @@ <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> -<title>Unveiling I/O Riot NG</title> +<title>Unveiling I/O Riot NG — Part 1: a guided tour</title> <link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> <link rel="stylesheet" href="../style.css" /> <link rel="stylesheet" href="style-override.css" /> </head> <body> <p class="header"> -<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-unveiling-ior-ng.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-unveiling-ior-ng.gmi">Gemini</a> | <a href="https://snonux.foo">Microblog</a> | <a href="https://irregular.ninja">Street photography</a> +<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-unveiling-ior-ng-part-1.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-unveiling-ior-ng-part-1.gmi">Gemini</a> | <a href="https://snonux.foo">Microblog</a> | <a href="https://irregular.ninja">Street photography</a> </p> -<h1 style='display: inline' id='unveiling-io-riot-ng'>Unveiling I/O Riot NG</h1><br /> +<h1 style='display: inline' id='unveiling-io-riot-ng--part-1-a-guided-tour'>Unveiling I/O Riot NG — Part 1: a guided tour</h1><br /> <br /> <span class='quote'>Draft — not in the gemfeed yet. Promote with the usual rename + index dance.</span><br /> <br /> -<span>I rewrote I/O Riot. The old one was C + Systemtap and dates from 2017. The new one — call it ior — is Go + C + BPF via libbpfgo, runs on Linux, and is mostly a TUI dashboard rather than a record/replay box. Since pictures are worth more than yet another README table of key bindings, I built a demo.</span><br /> +<span>I rewrote I/O Riot. The old one was C + Systemtap and dates from 2017. The new one — call it ior — is Go + C + BPF via libbpfgo, runs on Linux, and is mostly a TUI dashboard rather than a record/replay box. Since pictures are worth more than yet another README table of key bindings, I am presenting some examples in this post.</span><br /> +<br /> +<span>This is the first of three posts. Part 1 is the demo-driven tour: what ior looks like, how the dashboard tabs work, how the live flamegraph reads, how filtering and recording work. Part 2 covers installing it on a fresh Rocky Linux 9 box and the "compile once, run everywhere" story underneath that — eBPF, CO-RE, libbpfgo, static linking, and why a 23 MB binary you build on one machine just runs on every other Linux host you scp it to. Part 3 is the under-the-hood companion — the per-event schema, the syscall-coverage probe generator, async-syscall caveats, and post-mortem SQL on the parquet output. The three are independent; read them in any order.</span><br /> <br /> <a href='./unveiling-ior-ng/00-hero-flamegraph.png'><img alt='ior's live flamegraph: every running process, by file path, by syscall — width = event volume' title='ior's live flamegraph: every running process, by file path, by syscall — width = event volume' src='./unveiling-ior-ng/00-hero-flamegraph.png' /></a><br /> <br /> +<br /> <a class='textlink' href='https://codeberg.org/snonux/ior'>I/O Riot NG on Codeberg</a><br /> <a class='textlink' href='./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html'>the original I/O Riot post (2018)</a><br /> <br /> <h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> <br /> <ul> -<li><a href='#unveiling-io-riot-ng'>Unveiling I/O Riot NG</a></li> +<li><a href='#unveiling-io-riot-ng--part-1-a-guided-tour'>Unveiling I/O Riot NG — Part 1: a guided tour</a></li> <li>⇢ <a href='#what-it-does'>What it does</a></li> -<li>⇢ <a href='#a-short-detour-ebpf-and-libbpfgo'>A short detour: eBPF and libbpfgo</a></li> -<li>⇢ ⇢ <a href='#co-re--the-part-that-makes-ior-actually-portable'>CO-RE — the part that makes ior actually portable</a></li> -<li>⇢ ⇢ <a href='#if-you-want-to-go-deeper'>If you want to go deeper</a></li> -<li>⇢ <a href='#the-whole-thing-as-a-tape-pipeline'>The whole thing as a tape pipeline</a></li> <li>⇢ <a href='#first-launch'>First launch</a></li> <li>⇢ <a href='#the-seven-tabs-in-30-seconds-each'>The seven tabs, in 30 seconds each</a></li> <li>⇢ ⇢ <a href='#2-overview'><span class='inlinecode'>2</span> Overview</a></li> @@ -42,102 +41,17 @@ <li>⇢ <a href='#the-stream-tab-is-the-good-one'>The Stream tab is the good one</a></li> <li>⇢ <a href='#filtering-more-thoroughly'>Filtering, more thoroughly</a></li> <li>⇢ <a href='#recording'>Recording</a></li> -<li>⇢ <a href='#querying-a-parquet-trace-with-clickhouse'>Querying a parquet trace with ClickHouse</a></li> -<li>⇢ <a href='#reproducing-the-whole-demo'>Reproducing the whole demo</a></li> +<li>⇢ <a href='#installing-it'>Installing it</a></li> <li>⇢ <a href='#what-s-still-missing'>What's still missing</a></li> </ul><br /> <h2 style='display: inline' id='what-it-does'>What it does</h2><br /> <br /> -<span>ior attaches BPF tracepoints to a chunk of the synchronous-I/O syscall surface — open, read, write, stat, mmap, sync, link, fcntl, dup, the obvious ones. Each enter/exit pair becomes an event with a duration plus an inter-syscall gap, and the events feed a Bubble Tea dashboard with seven tabs: a live flamegraph, an overview, sortable per-syscall / per-file / per-process tables, latency histograms, and a live event stream with a stackable filter UI on top.</span><br /> +<span>ior attaches BPF tracepoints to a chunk of the synchronous-I/O syscall surface — open, read, write, stat, mmap, sync, link, fcntl, dup, the obvious ones — plus the async ones (<span class='inlinecode'>io_uring_*</span>, <span class='inlinecode'>aio_*</span>). Each enter/exit pair becomes an event with a duration plus an inter-syscall gap, and the events feed a Go Bubble Tea dashboard with seven tabs: a live flamegraph, an overview, sortable per-syscall / per-file / per-process tables, latency histograms, and a live event stream with a stackable filter UI on top.</span><br /> <br /> <span>Same shape as the old I/O Riot in spirit: capture what the system is actually doing, not synthetic load. Different shape in execution: no replay engine, no separate record file unless you ask for one, no kernel-debug-info dance.</span><br /> <br /> <a href='./unveiling-ior-ng/00-logo.png'><img alt='I/O Riot NG logo' title='I/O Riot NG logo' src='./unveiling-ior-ng/00-logo.png' /></a><br /> <br /> -<h2 style='display: inline' id='a-short-detour-ebpf-and-libbpfgo'>A short detour: eBPF and libbpfgo</h2><br /> -<br /> -<span>If you haven't touched eBPF before: it's a small in-kernel bytecode VM. You compile a tiny C program, the kernel verifies it can't crash or loop forever, and then it runs every time some hook fires — a syscall enter/exit, a kprobe, a tracepoint, a network packet. The program writes events into a ring buffer that userspace mmaps and drains. No kernel module, no patched kernel, no debug symbols required.</span><br /> -<br /> -<span>ior plugs into the syscall tracepoints — <span class='inlinecode'>sys_enter_openat</span>, <span class='inlinecode'>sys_exit_read</span>, etc. — and the BPF side does the bare minimum: timestamp the event, copy a few fields, push to a perf ring buffer. All the heavy lifting (string interning, latency math, aggregation, the dashboard) is in Go on the userspace side.</span><br /> -<br /> -<span>The kernel ships a C library called libbpf that handles loading the program, attaching it to hooks, managing maps, and reading the ring buffer. There are two well-known ways to drive that from Go:</span><br /> -<br /> -<ul> -<li>libbpfgo (Aqua Security): a thin cgo wrapper around libbpf. You ship libbpf along with your binary and call into the same C API that <span class='inlinecode'>bpftool</span> and <span class='inlinecode'>perf</span> use.</li> -<li>cilium/ebpf: a from-scratch pure-Go reimplementation of everything libbpf does — ELF parser, BTF resolver, syscall layer, the lot.</li> -</ul><br /> -<span>I went with libbpfgo specifically because it's a wrapper, not a reimplementation. Whatever lands in libbpf upstream — new map types, new attach kinds, CO-RE fixes — I get for free the next kernel cycle. The pure-Go variant has to chase libbpf's feature set in parallel, and any divergence is on me to debug. For a tracer that's mostly value-add on the userspace side, "be a thin client of the kernel's own library" wins.</span><br /> -<br /> -<h3 style='display: inline' id='co-re--the-part-that-makes-ior-actually-portable'>CO-RE — the part that makes ior actually portable</h3><br /> -<br /> -<span>The old I/O Riot was Systemtap. Systemtap programs are translated into a kernel module against the running kernel's exact headers, and that module then has to be loaded with <span class='inlinecode'>insmod</span>. That meant: the user has to install a kernel-debuginfo package matching their running kernel, and a fresh build per host (or per kernel update). On the BSD-style "you only run what you compiled here" laptop crowd that was tolerable; on a fleet of distros + kernel versions it was a recurring tax. Half of the original I/O Riot's README was about kernel-debuginfo dance steps.</span><br /> -<br /> -<span>CO-RE — Compile Once, Run Everywhere — is the eBPF feature that throws all of that out. The idea, in one paragraph: when you write a BPF program that reads <span class='inlinecode'>task->mm->start_stack</span>, you don't bake the offsets of those fields into the compiled program. Instead, the compiler emits relocation records ("at this instruction, fetch the offset of <span class='inlinecode'>mm</span> inside <span class='inlinecode'>task_struct</span>"). At load time, libbpf looks up the actual offsets in the target kernel's BTF (BPF Type Format — a description of every kernel struct, embedded in <span class='inlinecode'>/sys/kernel/btf/vmlinux</span> on any modern kernel) and patches the program in place. The same <span class='inlinecode'>.bpf.o</span> that ran on a 5.10 Debian kernel runs on a 6.8 Fedora kernel without recompilation.</span><br /> -<br /> -<span>Pictorially, the contrast looks like this:</span><br /> -<br /> -<pre> -Old I/O Riot (Systemtap) New ior (libbpf + CO-RE) -───────────────────────── ──────────────────────────── - .stp source .bpf.c source - │ │ - │ needs THIS kernel's headers │ build ONCE against vmlinux.h - │ + debuginfo package installed │ (generated from any kernel BTF) - ▼ ▼ - per-host translate + compile one portable .bpf.o - │ │ - ▼ ▼ - per-host kernel module same binary on every host - │ │ - insmod / modprobe libbpf loader: - │ │ • read /sys/kernel/btf/vmlinux - ▼ │ • patch field offsets - attached, this kernel only │ • verify + load - ▼ - attached, runs anywhere -</pre> -<br /> -<span>What that buys ior in practice: I ship a single <span class='inlinecode'>ior</span> binary. On any Linux ≥4.18-ish with BTF available (which is almost all of them now — Debian, Ubuntu, Fedora, Arch, RHEL all ship <span class='inlinecode'>CONFIG_DEBUG_INFO_BTF=y</span> by default), it just works. No kernel-debuginfo dependency, no per-kernel build matrix, no DKMS hooks. The first time I tried <span class='inlinecode'>scp ior fedora-box:</span> and it ran without complaint after a 6-month gap I had to double-check it wasn't silently doing nothing.</span><br /> -<br /> -<span>The runtime shape of a trace pipeline lines up with that:</span><br /> -<br /> -<pre> - kernel side userspace (this binary) - ─────────── ─────────────────────── - ┌──────────────────────┐ - tracepoint: │ Go process │ - sys_enter_openat │ ┌────────────────┐ │ - │ │ │ aggregator │ │ - ▼ │ │ (latency, │ │ - ┌─────────┐ │ │ stacks, │ │ - │ BPF prog│ ─── perf ring buf ──────────>│──│ filters) │ │ - │ (verified │ └─────┬──────────┘ │ - │ bytecode) │ │ │ - └─────────┘ │ ▼ │ - │ Bubble Tea TUI / │ - │ parquet writer / │ - │ CSV stdout │ - └──────────────────────┘ -</pre> -<br /> -<span>The cost is cgo. Every call from Go into libbpf crosses the cgo boundary, which historically meant tens to ~hundred-ish nanoseconds of overhead per call — register save/restore, a stack switch onto g0, goroutine state bookkeeping. Cheap in absolute terms, but it adds up if you call into C inside a tight loop. ior keeps the actual hot path on the kernel side and only crosses into Go once per drained batch of events from the ring buffer, so the per-call cost is amortized over thousands of events. In practice it doesn't show up in profiles.</span><br /> -<br /> -<span>Go 1.26, the current release at the time of writing (late April 2026), is the one that finally took a serious bite out of cgo's per-call cost — the runtime can elide a chunk of the bookkeeping for calls that don't need it. Real-world wins depend heavily on the workload, but the rough direction is that cgo now feels closer to "an unusually expensive function call" than to "a context switch", which is the right mental model for almost everyone touching a C library from Go. The shorter version: cgo overhead used to be a real footgun for ports that called into C in the inner loop. With Go 1.26 it's a footnote unless you're doing many millions of small calls per second, in which case batching across the boundary still fixes it.</span><br /> -<br /> -<h3 style='display: inline' id='if-you-want-to-go-deeper'>If you want to go deeper</h3><br /> -<br /> -<span>If any of this sounds interesting and you want to learn how to write your own BPF programs, two books are the standard recommendations and both well worth the time:</span><br /> -<br /> -<ul> -<li>"Learning eBPF" by Liz Rice (O'Reilly, 2023) is the friendlier on-ramp. It walks through writing your first programs end-to-end, covers CO-RE and BTF in plain English, and is the book I'd hand to someone who has never touched the kernel side before. Liz also gave the canonical "what is eBPF" conference talk floating around YouTube, which makes a good 40-minute companion.</li> -<li>"BPF Performance Tools: Linux System and Application Observability" by Brendan Gregg (Addison-Wesley, 2019) is the encyclopedia. It's where you go after you've understood the basics and now want a complete reference for tracing every subsystem in the kernel — file systems, networking, scheduler, languages, applications — with worked tools for each. The flame-graph-driven analysis style throughout is also exactly how ior's own flamegraph tab thinks about a workload.</li> -</ul><br /> -<span>Between the two, Rice teaches you the moving parts and Gregg teaches you what to do with them.</span><br /> -<br /> -<h2 style='display: inline' id='the-whole-thing-as-a-tape-pipeline'>The whole thing as a tape pipeline</h2><br /> -<br /> -<span>The demo isn't a screencast I sat through. It's 14 VHS tapes that drive the TUI deterministically, with a background workload generator producing real syscall traffic for the trace to chew on. One <span class='inlinecode'>mage demo</span> and every GIF below regenerates from scratch. The boring part of "make a demo" — having to re-record everything when the UI shifts — goes away.</span><br /> -<br /> <h2 style='display: inline' id='first-launch'>First launch</h2><br /> <br /> <!-- Generator: GNU source-highlight 3.1.9 @@ -151,7 +65,7 @@ http://www.gnu.org/software/src-highlite --> <br /> <a href='./unveiling-ior-ng/01-launch.gif'><img alt='Cold start: PID picker, then the dashboard' title='Cold start: PID picker, then the dashboard' src='./unveiling-ior-ng/01-launch.gif' /></a><br /> <br /> -<span>The dashboard opens on the live flamegraph. Bars grow as new events arrive. Before walking through the keys, a paragraph on what you're looking at — flamegraphs are easier to read than they are to describe.</span><br /> +<span>The dashboard opens on the live flamegraph. Bars grow as new events arrive. Before walking through the keys, a paragraph on what you're looking at — flamegraphs are easier to read than they are to describe:</span><br /> <br /> <span>A flamegraph is a histogram of stacks. Each horizontal bar is one entry in a stack; every bar directly above it is a child of that entry, and the stack you read top-to-bottom is the same shape as a call chain. In ior, "stack" doesn't mean function-call stack (we don't have userspace symbols). It means a tuple of dimensions of the trace: by default <span class='inlinecode'>comm/path/tracepoint</span>, so the bottom row is per-process names, the middle row is per-file paths, and the top row is the syscall (<span class='inlinecode'>enter_read</span>, <span class='inlinecode'>enter_openat</span>, etc.). A wide bar means lots of events landed in that bucket, a narrow bar means few. There is no time axis — left-to-right is just sort order, not chronology. The whole chart is one "where is the I/O coming from?" picture.</span><br /> <br /> @@ -171,16 +85,33 @@ http://www.gnu.org/software/src-highlite --> </ul><br /> <span>Now the keys. Movement uses vi-style <span class='inlinecode'>h</span>/<span class='inlinecode'>j</span>/<span class='inlinecode'>k</span>/<span class='inlinecode'>l</span> everywhere in ior — and the cursor keys work too if you'd rather. <span class='inlinecode'>h</span>/<span class='inlinecode'>l</span> (or <span class='inlinecode'>←</span>/<span class='inlinecode'>→</span>) walk siblings at the current depth, <span class='inlinecode'>j</span>/<span class='inlinecode'>k</span> (or <span class='inlinecode'>↓</span>/<span class='inlinecode'>↑</span>) step shallower or deeper. <span class='inlinecode'>enter</span> zooms into the selected subtree (the rest of the chart greys out and the selection becomes the new root). <span class='inlinecode'>u</span> or <span class='inlinecode'>Esc</span> undoes the zoom. <span class='inlinecode'>b</span> toggles the metric driving bar width between event count and total bytes. <span class='inlinecode'>/</span> opens regex search; matching frames stay coloured while everything else greys out, so you can use it as a filter as well as a finder. And <span class='inlinecode'>o</span> cycles between five different stack-ordering modes, each with its own lens on the data.</span><br /> <br /> -<span>The five orderings ship as built-in presets. The leftmost dimension is the bottom row of the chart (the root); the rightmost is the top row (the leaf). Pressing <span class='inlinecode'>o</span> rotates through them in this order:</span><br /> +<span>The five orderings ship as built-in presets. Read each preset name as bottom→top: the leftmost dimension is what you'll see lined up across the bottom of the chart (the root row), the next one up is its children, and the rightmost is the top row (the leaf). Switching the order changes which dimension you're scanning first when your eye starts at the bottom.</span><br /> <br /> -<ul> -<li><span class='inlinecode'>comm/tracepoint/path</span> — the default. Root rows are processes (by command name); each process's bar splits into the syscalls it issued, and each syscall splits further by file path. Best general-purpose view: "which programs are doing the I/O, and what kind?"</li> -<li><span class='inlinecode'>path/tracepoint/comm</span> — root by file path. Use this when you suspect a particular file or directory is the bottleneck — pick the path, see which syscalls hit it, and which processes did those syscalls. Pairs naturally with directory grouping in the Files tab.</li> -<li><span class='inlinecode'>tracepoint/comm/path</span> — root by syscall. When you already know "this is an <span class='inlinecode'>openat</span> problem" or "we're write-bound", this view collects all the openat (or write) traffic at the bottom and lets you drill into who's doing it and to which paths.</li> -<li><span class='inlinecode'>pid/tracepoint/path</span> — root by PID, not comm. Same shape as the default but each individual process gets its own bar instead of being lumped in with siblings sharing a comm. Useful when you have many bash or python instances and need to tell them apart.</li> -<li><span class='inlinecode'>comm/path/tracepoint</span> — root by process, then by file (skipping the syscall layer at the top). Best when you care about "what files does this program touch?" more than "what syscalls does it issue?" — the file column gets a full row of vertical real estate instead of being split per-syscall.</li> -</ul><br /> -<span>In all five orderings, bar widths still mean the same thing — proportion of the active metric (events or bytes, toggled with <span class='inlinecode'>b</span>). The toolbar at the top of the chart always shows the current ordering as <span class='inlinecode'>o:order(<dim1>/<dim2>/<dim3>)</span>, so you never lose track of which lens you're looking through.</span><br /> +<span>You change ordering with the <span class='inlinecode'>o</span> hotkey, on the fly, while the trace is still running. No restart, no reset, no re-recording — <span class='inlinecode'>o</span> just rebuilds the live chart with the next preset and keeps streaming new events into it. Press it once to flip from "processes at the bottom" to "paths at the bottom" the moment you realise you'd rather slice the data the other way; press it again to keep cycling. The toolbar updates immediately to show the new <span class='inlinecode'>o:order(...)</span> value. Pressing <span class='inlinecode'>o</span> rotates through the presets in this order:</span><br /> +<br /> +<span>Concrete screenshots of each preset on the same workload follow each description, so you can see how the same trace data reshapes itself depending on the lens.</span><br /> +<br /> +<span><span class='inlinecode'>comm/tracepoint/path</span> (default) — processes at the bottom, syscalls in the middle, file paths on top. Each comm bar at the root splits into the syscalls it issued, and each syscall splits further into the files it touched. Best general-purpose view: "which programs are doing the I/O, and what kind?"</span><br /> +<br /> +<a href='./unveiling-ior-ng/13a-order-by-process.png'><img alt='Ordering 1: comm/tracepoint/path — processes at the bottom (cos…, find, head, notify-rs i…, sh, sqlx-sqlite, xar)' title='Ordering 1: comm/tracepoint/path — processes at the bottom (cos…, find, head, notify-rs i…, sh, sqlx-sqlite, xar)' src='./unveiling-ior-ng/13a-order-by-process.png' /></a><br /> +<br /> +<span><span class='inlinecode'>path/tracepoint/comm</span> — file paths at the bottom, syscalls in the middle, processes on top. Use this when you suspect a particular file or directory is hot — pick the path, see which syscalls hit it, and which processes did those syscalls. Pairs naturally with directory grouping in the Files tab.</span><br /> +<br /> +<a href='./unveiling-ior-ng/13b-order-by-path.png'><img alt='Ordering 2: path/tracepoint/comm — file paths at the bottom (/null, /etc/home, /lib64, /tmp, /usr, /share, /paul, /libc, …)' title='Ordering 2: path/tracepoint/comm — file paths at the bottom (/null, /etc/home, /lib64, /tmp, /usr, /share, /paul, /libc, …)' src='./unveiling-ior-ng/13b-order-by-path.png' /></a><br /> +<br /> +<span><span class='inlinecode'>tracepoint/comm/path</span> — syscalls at the bottom, processes in the middle, file paths on top. When you already know "this is an <span class='inlinecode'>openat</span> problem" or "we're write-bound", this view collects all the openat (or write) traffic into one bar at the root and lets you drill into who's doing it and to which paths.</span><br /> +<br /> +<a href='./unveiling-ior-ng/13c-order-by-syscall.png'><img alt='Ordering 3: tracepoint/comm/path — syscalls at the bottom (enter_close, enter_fcntl, enter_mmap, enter_newfstatat, enter_openat, enter_read, …)' title='Ordering 3: tracepoint/comm/path — syscalls at the bottom (enter_close, enter_fcntl, enter_mmap, enter_newfstatat, enter_openat, enter_read, …)' src='./unveiling-ior-ng/13c-order-by-syscall.png' /></a><br /> +<br /> +<span><span class='inlinecode'>pid/tracepoint/path</span> — PIDs at the bottom, syscalls in the middle, file paths on top. Same shape as the default but each individual process gets its own root bar instead of being lumped in with siblings sharing a comm. Useful when you have many bash or python instances and need to tell them apart by ID.</span><br /> +<br /> +<a href='./unveiling-ior-ng/13d-order-by-pid.png'><img alt='Ordering 4: pid/tracepoint/path — PIDs at the bottom (103…, 1…, 1338410, 4113, 4263514, 7709, 8272, …)' title='Ordering 4: pid/tracepoint/path — PIDs at the bottom (103…, 1…, 1338410, 4113, 4263514, 7709, 8272, …)' src='./unveiling-ior-ng/13d-order-by-pid.png' /></a><br /> +<br /> +<span><span class='inlinecode'>comm/path/tracepoint</span> — processes at the bottom, file paths in the middle, syscalls on top. Inverse of the default in the upper two layers: you see processes, then which files they hit, then which syscalls hit each file. Best when you care about "what files does this program touch?" more than "what syscalls does it issue?".</span><br /> +<br /> +<a href='./unveiling-ior-ng/13e-order-by-process-paths.png'><img alt='Ordering 5: comm/path/tracepoint — processes at the bottom, file paths layered above instead of syscalls' title='Ordering 5: comm/path/tracepoint — processes at the bottom, file paths layered above instead of syscalls' src='./unveiling-ior-ng/13e-order-by-process-paths.png' /></a><br /> +<br /> +<span>In every ordering the rule is the same: scan the bottom row to pick a "by what?" dimension, then walk up to drill in. Bar widths always mean the same thing — proportion of the active metric (events or bytes, toggled with <span class='inlinecode'>b</span>). The toolbar at the top of the chart always shows the current ordering as <span class='inlinecode'>o:order(<dim1>/<dim2>/<dim3>)</span>, so you never lose track of which lens you're looking through.</span><br /> <br /> <a href='./unveiling-ior-ng/13-tui-flamegraph.gif'><img alt='Live in-TUI flamegraph: navigate, zoom, undo, cycle order + metric' title='Live in-TUI flamegraph: navigate, zoom, undo, cycle order + metric' src='./unveiling-ior-ng/13-tui-flamegraph.gif' /></a><br /> <br /> @@ -282,89 +213,13 @@ http://www.gnu.org/software/src-highlite --> </ul><br /> <a href='./unveiling-ior-ng/14-headless-modes.gif'><img alt='All three headless flows in one tape' title='All three headless flows in one tape' src='./unveiling-ior-ng/14-headless-modes.gif' /></a><br /> <br /> -<h2 style='display: inline' id='querying-a-parquet-trace-with-clickhouse'>Querying a parquet trace with ClickHouse</h2><br /> -<br /> -<span>The schema is flat and stable: <span class='inlinecode'>seq, time_ns, gap_ns, latency_ns, comm, pid, tid, syscall, fd, ret, bytes, file, is_error, filter_epoch</span>. ClickHouse Local reads parquet directly without a server, which makes it a perfect post-mortem tool — point it at the file and run SQL:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre>clickhouse <b><u><font color="#000000">local</font></u></b> --query <font color="#808080">"</font> -<font color="#808080"> SELECT comm, syscall, count() AS n,</font> -<font color="#808080"> formatReadableSize(sum(bytes)) AS total</font> -<font color="#808080"> FROM file('trace.parquet', Parquet)</font> -<font color="#808080"> GROUP BY comm, syscall</font> -<font color="#808080"> ORDER BY n DESC</font> -<font color="#808080"> LIMIT 10</font> -<font color="#808080">"</font> --format PrettyCompactNoEscapes -</pre> -<br /> -<pre> - ┌─comm────────────┬─syscall─┬─────n─┬─total──────┐ - 1. │ notify-rs inoti │ read │ 42005 │ 732.31 KiB │ - 2. │ cosmic-term │ statx │ 10898 │ 0.00 B │ - 3. │ cosmic-term │ read │ 10103 │ 4.02 MiB │ - 4. │ surface-eDP-1 │ ioctl │ 8452 │ 0.00 B │ - 5. │ cosmic-term │ close │ 4918 │ 0.00 B │ - 6. │ cosmic-term │ openat │ 4537 │ 0.00 B │ - 7. │ cosmic-term │ ioctl │ 3556 │ 0.00 B │ - 8. │ tokio-runtime-w │ read │ 1976 │ 4.04 MiB │ - 9. │ cosmic-comp │ read │ 1118 │ 6.63 KiB │ -10. │ systemd-oomd │ read │ 1085 │ 111.97 KiB │ - └─────────────────┴─────────┴───────┴────────────┘ -</pre> -<br /> -<span>The fields you actually want for performance work are <span class='inlinecode'>latency_ns</span> and <span class='inlinecode'>gap_ns</span>. P99 by syscall, only the ones that landed in error:</span><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre>clickhouse <b><u><font color="#000000">local</font></u></b> --query <font color="#808080">"</font> -<font color="#808080"> SELECT syscall, count() AS n,</font> -<font color="#808080"> round(quantile(0.5)(latency_ns)/1000, 1) AS p50_us,</font> -<font color="#808080"> round(quantile(0.99)(latency_ns)/1000, 1) AS p99_us</font> -<font color="#808080"> FROM file('trace.parquet', Parquet)</font> -<font color="#808080"> WHERE is_error = 1</font> -<font color="#808080"> GROUP BY syscall</font> -<font color="#808080"> ORDER BY p99_us DESC</font> -<font color="#808080">"</font> --format PrettyCompactNoEscapes -</pre> -<br /> -<pre> - ┌─syscall────┬─────n─┬─p50_us─┬─p99_us─┐ - 1. │ statx │ 1216 │ 2.2 │ 16.4 │ - 2. │ newfstatat │ 69 │ 1.7 │ 16.4 │ - 3. │ open │ 1 │ 16.1 │ 16.1 │ - 4. │ mkdir │ 306 │ 3.9 │ 11.7 │ - 5. │ readlink │ 11 │ 1.5 │ 10.4 │ - 6. │ newstat │ 44 │ 2.5 │ 8.4 │ - 7. │ unlinkat │ 347 │ 1 │ 6.2 │ - 8. │ openat │ 380 │ 2.1 │ 5.8 │ - 9. │ access │ 2 │ 5 │ 5.5 │ -10. │ read │ 23597 │ 0.5 │ 5.4 │ -11. │ ioctl │ 901 │ 1 │ 5.3 │ -12. │ writev │ 1 │ 0.7 │ 0.7 │ - └────────────┴───────┴────────┴────────┘ -</pre> +<span>Once a parquet file is on disk, point any SQL-over-parquet tool at it — Part 3 walks through ClickHouse Local, with real query output against a 30-second capture.</span><br /> <br /> -<span>Real output, by the way — those rows are from a 30-second <span class='inlinecode'>ior -parquet trace.parquet</span> capture on the laptop I'm typing this on. <span class='inlinecode'>notify-rs inoti…</span> is the inotify thread of some Rust app I had open; <span class='inlinecode'>cosmic-term</span> is the COSMIC desktop's terminal emulator. The slowest p99 errors are the directory-walking syscalls (statx, newfstatat, mkdir) at ~16 µs — bog standard.</span><br /> +<h2 style='display: inline' id='installing-it'>Installing it</h2><br /> <br /> -<span>Same trick works in DuckDB (<span class='inlinecode'>duckdb -c "SELECT ... FROM 'trace.parquet'"</span>), pandas, polars, anything that reads Parquet. The point of streaming Parquet rather than ior's native <span class='inlinecode'>.ior.zst</span> format is exactly this: once it's on disk, you're in the standard data-tools ecosystem.</span><br /> +<span>The Rocky Linux 9 install procedure (with its kernel-backport caveat), the eBPF / CO-RE / static-linking explanation that makes the resulting binary "compile once, run everywhere", and the build host vs. trace host split all live in Part 2. The short version: you only have to build it once. The 23 MB binary then <span class='inlinecode'>scp</span>s to anywhere.</span><br /> <br /> -<h2 style='display: inline' id='reproducing-the-whole-demo'>Reproducing the whole demo</h2><br /> -<br /> -<!-- Generator: GNU source-highlight 3.1.9 -by Lorenzo Bettini -http://www.lorenzobettini.it -http://www.gnu.org/software/src-highlite --> -<pre>mage installDemoTools <i><font color="silver"># one-time: VHS via go install + ttyd from dnf</font></i> -sudo -v <i><font color="silver"># warm the sudo timestamp once</font></i> -mage demo <i><font color="silver"># ~10 minutes, fully headless, safe to background</font></i> -</pre> -<br /> -<span>To rebuild a single GIF after editing its tape: <span class='inlinecode'>TAPE=07-stream-live mage demoOne</span>.</span><br /> +<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-2.html'>Part 2: install + compile once, run everywhere</a><br /> <br /> <h2 style='display: inline' id='what-s-still-missing'>What's still missing</h2><br /> <br /> @@ -377,6 +232,10 @@ mage demo <i><font color="silver"># ~10 minutes, fully headless, </ul><br /> <span>But the live flamegraph, the stackable stream filters, and the cheap parquet capture together cover the cases I actually hit week to week. The demo above is the easiest way to get a feel for whether it's the kind of tool you want.</span><br /> <br /> +<span>For installing it and the eBPF / CO-RE / static-linking story (why one build runs on every other Linux box you scp it to), see Part 2. For the per-event schema, async-syscall caveats, the probe-generator safeguard against missing new kernel syscalls, and post-mortem SQL on the parquet output, see Part 3.</span><br /> +<br /> +<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-2.html'>Part 2: install + compile once, run everywhere</a><br /> +<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-3.html'>Part 3: under the hood (schema, probe generator, ClickHouse)</a><br /> <a class='textlink' href='https://codeberg.org/snonux/ior'>Source on Codeberg</a><br /> <a class='textlink' href='https://codeberg.org/snonux/ior/src/branch/main/demo/TUTORIAL.md'>The full in-repo tutorial</a><br /> <p class="footer"> diff --git a/gemfeed/DRAFT-unveiling-ior-ng-part-2.html b/gemfeed/DRAFT-unveiling-ior-ng-part-2.html new file mode 100644 index 00000000..73b67c75 --- /dev/null +++ b/gemfeed/DRAFT-unveiling-ior-ng-part-2.html @@ -0,0 +1,245 @@ +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> +<head> +<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> +<title>Unveiling I/O Riot NG — Part 2: install and compile once, run everywhere</title> +<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> +<link rel="stylesheet" href="../style.css" /> +<link rel="stylesheet" href="style-override.css" /> +</head> +<body> +<p class="header"> +<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-unveiling-ior-ng-part-2.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-unveiling-ior-ng-part-2.gmi">Gemini</a> | <a href="https://snonux.foo">Microblog</a> | <a href="https://irregular.ninja">Street photography</a> +</p> +<h1 style='display: inline' id='unveiling-io-riot-ng--part-2-install-and-compile-once-run-everywhere'>Unveiling I/O Riot NG — Part 2: install and compile once, run everywhere</h1><br /> +<br /> +<span class='quote'>Draft — not in the gemfeed yet. Promote with the usual rename + index dance.</span><br /> +<br /> +<span>This is Part 2 of three. Part 1 is the demo-driven tour — what ior looks like, how the dashboard tabs work, how filtering and recording behave. This part is about the install dance for Rocky Linux 9 (with one annoying kernel-backport caveat) and, more interestingly, why you only have to do that dance on a single machine: the resulting binary is portable to every other Linux box thanks to CO-RE — Compile Once, Run Everywhere — plus full static linking. Part 3 is the under-the-hood companion (per-event schema, async-syscall caveats, the syscall-coverage probe generator, and post-mortem SQL on the parquet output).</span><br /> +<br /> +<span>If you came here for the dashboard tour, that's Part 1. If you want to know how the data pipeline is shaped after you've got ior running, that's Part 3. This one is for the moment between "I want to try this" and "OK, it's running on the box I care about."</span><br /> +<br /> +<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-1.html'>Part 1: a guided tour</a><br /> +<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-3.html'>Part 3: under the hood (schema, probe generator, ClickHouse)</a><br /> +<a class='textlink' href='https://codeberg.org/snonux/ior'>I/O Riot NG on Codeberg</a><br /> +<a class='textlink' href='./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html'>the original I/O Riot post (2018)</a><br /> +<br /> +<a href='./unveiling-ior-ng/00-logo.png'><img alt='I/O Riot NG logo' title='I/O Riot NG logo' src='./unveiling-ior-ng/00-logo.png' /></a><br /> +<br /> +<br /> +<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> +<br /> +<ul> +<li><a href='#unveiling-io-riot-ng--part-2-install-and-compile-once-run-everywhere'>Unveiling I/O Riot NG — Part 2: install and compile once, run everywhere</a></li> +<li>⇢ <a href='#installing-ior'>Installing ior</a></li> +<li>⇢ ⇢ <a href='#why-native-installation-is-a-mess'>Why native installation is a mess</a></li> +<li>⇢ ⇢ <a href='#what-the-docker-build-is-actually-doing'>What the Docker build is actually doing</a></li> +<li>⇢ <a href='#a-short-detour-ebpf-and-libbpfgo'>A short detour: eBPF and libbpfgo</a></li> +<li>⇢ <a href='#co-re--the-part-that-makes-the-binary-actually-portable'>CO-RE — the part that makes the binary actually portable</a></li> +<li>⇢ <a href='#a-note-on-cgo-overhead'>A note on cgo overhead</a></li> +<li>⇢ <a href='#if-you-want-to-go-deeper'>If you want to go deeper</a></li> +<li>⇢ <a href='#wrapping-up'>Wrapping up</a></li> +</ul><br /> +<h2 style='display: inline' id='installing-ior'>Installing ior</h2><br /> +<br /> +<span>The short answer: use Docker. One command, no toolchain setup, works from any Docker-capable Linux host with BTF available:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>git clone https://codeberg.org/snonux/ior ~/git/ior +cd ~/git/ior +mage buildDocker +</pre> +<br /> +<span>First run builds a Rocky Linux 9 builder image (~15–20 minutes). Subsequent runs reuse the cached image and finish in under a minute. The resulting static binary lands at <span class='inlinecode'>./ior</span>.</span><br /> +<br /> +<span>That's the officially supported install path, and it's the right one for anyone who just wants to run ior without living in its build system.</span><br /> +<br /> +<h3 style='display: inline' id='why-native-installation-is-a-mess'>Why native installation is a mess</h3><br /> +<br /> +<span>If you're curious why Docker became the answer, the native install on Rocky Linux 9 illustrates the problem well. Three separate things bite you before you even get to <span class='inlinecode'>mage build</span>:</span><br /> +<br /> +<span>Rocky 9 ships neither <span class='inlinecode'>libelf.a</span> nor <span class='inlinecode'>libzstd.a</span> — there are no <span class='inlinecode'>*-static</span> subpackages for either, only the dynamic <span class='inlinecode'>.so</span> files. Both have to be compiled from source. <span class='inlinecode'>libelf</span> from the elfutils source RPM, <span class='inlinecode'>libzstd</span> from the upstream GitHub release tarball.</span><br /> +<br /> +<span>Rocky 9 also only ships Go 1.25.x, but ior requires 1.26+. So Go itself has to be installed from go.dev in parallel with the library builds.</span><br /> +<br /> +<span>And there's a kernel quirk that used to make this section much longer. Pre-fix, ior would happily load on a stock 5.14 RHEL kernel and then die on the very first tracepoint attach with <span class='inlinecode'>BPF_LINK_CREATE</span>/<span class='inlinecode'>BPF_PERF_EVENT</span> returning <span class='inlinecode'>EACCES</span> — even as root, with SELinux permissive, with every BPF-related sysctl wide open. The cause is that RHEL 9 carries an <span class='inlinecode'>rt</span>-tree backport that adds <span class='inlinecode'>preempt_lazy_count</span> to <span class='inlinecode'>struct trace_entry</span>. That widens the BTF-emitted alias <span class='inlinecode'>trace_event_raw_sys_enter</span>/<span class='inlinecode'>_exit</span> by 8 bytes and shifts the <span class='inlinecode'>args</span>/<span class='inlinecode'>ret</span> offsets — but the actual context the kernel hands the BPF program is still <span class='inlinecode'>struct syscall_trace_enter</span>/<span class='inlinecode'>_exit</span>, where the offsets did not move. Programs written against <span class='inlinecode'>trace_event_raw_sys_*</span> (the conventional choice; bcc, libbpf-tools, and ior all used to do this) end up reading past <span class='inlinecode'>max_ctx_offset</span>, so the verifier rejects the attach. The fix — also what bcc shipped in [PR #4920](https://github.com/iovi |