diff options
| -rw-r--r-- | gemfeed/2026-05-08-unveiling-ior-ng-part-1.html (renamed from gemfeed/DRAFT-unveiling-ior-ng-part-1.html) | 90 | ||||
| -rw-r--r-- | gemfeed/DRAFT-ior-guided-tour.html | 255 | ||||
| -rw-r--r-- | gemfeed/DRAFT-unveiling-ior-ng-part-2.html | 35 | ||||
| -rw-r--r-- | gemfeed/DRAFT-unveiling-ior-ng-part-3.html | 64 | ||||
| -rw-r--r-- | gemfeed/atom.xml | 366 | ||||
| -rw-r--r-- | gemfeed/index.html | 1 | ||||
| -rw-r--r-- | gemfeed/unveiling-ior-ng/00-logo.png | bin | 332693 -> 28775 bytes | |||
| -rw-r--r-- | index.html | 3 | ||||
| -rw-r--r-- | uptime-stats.html | 24 |
9 files changed, 360 insertions, 478 deletions
diff --git a/gemfeed/DRAFT-unveiling-ior-ng-part-1.html b/gemfeed/2026-05-08-unveiling-ior-ng-part-1.html index 7e7d60e1..c0571d17 100644 --- a/gemfeed/DRAFT-unveiling-ior-ng-part-1.html +++ b/gemfeed/2026-05-08-unveiling-ior-ng-part-1.html @@ -2,25 +2,26 @@ <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> -<title>Unveiling I/O Riot NG — Part 1: a guided tour</title> +<title>Unveiling I/O Riot NG 1.0.0 — Part 1: a guided tour</title> <link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> <link rel="stylesheet" href="../style.css" /> <link rel="stylesheet" href="style-override.css" /> </head> <body> <p class="header"> -<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-unveiling-ior-ng-part-1.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-unveiling-ior-ng-part-1.gmi">Gemini</a> | <a href="https://snonux.foo">Microblog</a> | <a href="https://irregular.ninja">Street photography</a> +<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/2026-05-08-unveiling-ior-ng-part-1.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/2026-05-08-unveiling-ior-ng-part-1.gmi">Gemini</a> | <a href="https://snonux.foo">Microblog</a> | <a href="https://irregular.ninja">Street photography</a> </p> -<h1 style='display: inline' id='unveiling-io-riot-ng--part-1-a-guided-tour'>Unveiling I/O Riot NG — Part 1: a guided tour</h1><br /> +<h1 style='display: inline' id='unveiling-io-riot-ng-100--part-1-a-guided-tour'>Unveiling I/O Riot NG 1.0.0 — Part 1: a guided tour</h1><br /> <br /> -<span class='quote'>Draft — not in the gemfeed yet. Promote with the usual rename + index dance.</span><br /> +<span class='quote'>Published at 2026-05-07T09:46:29+03:00</span><br /> <br /> -<span>I rewrote I/O Riot. The old one was C + Systemtap and dates from 2017. The new one — call it ior — is Go + C + BPF via libbpfgo, runs on Linux, and is mostly a TUI dashboard rather than a record/replay box. Since pictures are worth more than yet another README table of key bindings, I am presenting some examples in this post.</span><br /> +<span>I rewrote I/O Riot. The old version, written in C and SystemTap, dates back to 2017. The new version (called <span class='inlinecode'>ior</span>) uses Go, C, and BPF via libbpfgo. It runs on Linux and is primarily a TUI dashboard rather than a record/replay box. It took around two years of intermittent work to reach this <span class='inlinecode'>1.0.0</span> release.</span><br /> <br /> -<span>This is the first of three posts. Part 1 is the demo-driven tour: what ior looks like, how the dashboard tabs work, how the live flamegraph reads, how filtering and recording work. Part 2 covers installing it on a fresh Rocky Linux 9 box and the "compile once, run everywhere" story underneath that — eBPF, CO-RE, libbpfgo, static linking, and why a 23 MB binary you build on one machine just runs on every other Linux host you scp it to. Part 3 is the under-the-hood companion — the per-event schema, the syscall-coverage probe generator, async-syscall caveats, and post-mortem SQL on the parquet output. The three are independent; read them in any order.</span><br /> +<span>This is the first of three posts. Part 1 is the demo-driven tour: what <span class='inlinecode'>ior</span> looks like, how the dashboard tabs work, how the live flamegraph reads, how filtering and recording work. Part 2 covers installing it on a fresh Rocky Linux 9 box and the "compile once, run everywhere" story underneath that: eBPF, CO-RE, libbpfgo, static linking, and why a 23 MB binary you build on one machine just runs on every other Linux host you scp it to. Part 3 is the under-the-hood companion: the per-event schema, the syscall-coverage probe generator, async-syscall caveats, and post-mortem SQL on the parquet output. </span><br /> <br /> -<a href='./unveiling-ior-ng/00-hero-flamegraph.png'><img alt='ior's live flamegraph: every running process, by file path, by syscall — width = event volume' title='ior's live flamegraph: every running process, by file path, by syscall — width = event volume' src='./unveiling-ior-ng/00-hero-flamegraph.png' /></a><br /> +<a href='./unveiling-ior-ng/00-hero-flamegraph.png'><img alt='Live flamegraph' title='Live flamegraph' src='./unveiling-ior-ng/00-hero-flamegraph.png' /></a><br /> <br /> +<a class='textlink' href='./2026-05-08-unveiling-ior-ng-part-1.html'>2026-05-08 Unveiling I/O Riot NG 1.0.0 — Part 1: a guided tour (You are currently reading this)</a><br /> <br /> <a class='textlink' href='https://codeberg.org/snonux/ior'>I/O Riot NG on Codeberg</a><br /> <a class='textlink' href='./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html'>the original I/O Riot post (2018)</a><br /> @@ -28,7 +29,7 @@ <h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> <br /> <ul> -<li><a href='#unveiling-io-riot-ng--part-1-a-guided-tour'>Unveiling I/O Riot NG — Part 1: a guided tour</a></li> +<li><a href='#unveiling-io-riot-ng-100--part-1-a-guided-tour'>Unveiling I/O Riot NG 1.0.0 — Part 1: a guided tour</a></li> <li>⇢ <a href='#what-it-does'>What it does</a></li> <li>⇢ <a href='#first-launch'>First launch</a></li> <li>⇢ <a href='#the-seven-tabs-in-30-seconds-each'>The seven tabs, in 30 seconds each</a></li> @@ -41,12 +42,11 @@ <li>⇢ <a href='#the-stream-tab-is-the-good-one'>The Stream tab is the good one</a></li> <li>⇢ <a href='#filtering-more-thoroughly'>Filtering, more thoroughly</a></li> <li>⇢ <a href='#recording'>Recording</a></li> -<li>⇢ <a href='#installing-it'>Installing it</a></li> <li>⇢ <a href='#what-s-still-missing'>What's still missing</a></li> </ul><br /> <h2 style='display: inline' id='what-it-does'>What it does</h2><br /> <br /> -<span>ior attaches BPF tracepoints to a chunk of the synchronous-I/O syscall surface — open, read, write, stat, mmap, sync, link, fcntl, dup, the obvious ones — plus the async ones (<span class='inlinecode'>io_uring_*</span>, <span class='inlinecode'>aio_*</span>). Each enter/exit pair becomes an event with a duration plus an inter-syscall gap, and the events feed a Go Bubble Tea dashboard with seven tabs: a live flamegraph, an overview, sortable per-syscall / per-file / per-process tables, latency histograms, and a live event stream with a stackable filter UI on top.</span><br /> +<span>ior attaches BPF tracepoints to a chunk of the synchronous-I/O syscall surface (open, read, write, stat, mmap, sync, link, fcntl, dup, the obvious ones), plus the async ones (<span class='inlinecode'>io_uring_*</span>, <span class='inlinecode'>aio_*</span>). Each enter/exit pair becomes an event with a duration plus an inter-syscall gap, and the events feed a Go Bubble Tea dashboard with seven tabs: a live flamegraph, an overview, sortable per-syscall / per-file / per-process tables, latency histograms, and a live event stream with a stackable filter UI on top.</span><br /> <br /> <span>Same shape as the old I/O Riot in spirit: capture what the system is actually doing, not synthetic load. Different shape in execution: no replay engine, no separate record file unless you ask for one, no kernel-debug-info dance.</span><br /> <br /> @@ -65,25 +65,25 @@ http://www.gnu.org/software/src-highlite --> <br /> <a href='./unveiling-ior-ng/01-launch.gif'><img alt='Cold start: PID picker, then the dashboard' title='Cold start: PID picker, then the dashboard' src='./unveiling-ior-ng/01-launch.gif' /></a><br /> <br /> -<span>The dashboard opens on the live flamegraph. Bars grow as new events arrive. Before walking through the keys, a paragraph on what you're looking at — flamegraphs are easier to read than they are to describe:</span><br /> +<span>The dashboard opens on the live flamegraph. Bars grow as new events arrive. Before walking through the keys, a paragraph on what you're looking at, since flamegraphs are easier to read than they are to describe:</span><br /> <br /> -<span>A flamegraph is a histogram of stacks. Each horizontal bar is one entry in a stack; every bar directly above it is a child of that entry, and the stack you read top-to-bottom is the same shape as a call chain. In ior, "stack" doesn't mean function-call stack (we don't have userspace symbols). It means a tuple of dimensions of the trace: by default <span class='inlinecode'>comm/path/tracepoint</span>, so the bottom row is per-process names, the middle row is per-file paths, and the top row is the syscall (<span class='inlinecode'>enter_read</span>, <span class='inlinecode'>enter_openat</span>, etc.). A wide bar means lots of events landed in that bucket, a narrow bar means few. There is no time axis — left-to-right is just sort order, not chronology. The whole chart is one "where is the I/O coming from?" picture.</span><br /> +<span>A flamegraph is a histogram of stacks. Each horizontal bar is one entry in a stack; every bar directly above it is a child of that entry, and the stack you read top-to-bottom is the same shape as a call chain. In <span class='inlinecode'>ior</span>, "stack" doesn't mean function-call stack (we don't have userspace symbols [yet]). It means a tuple of dimensions of the trace: by default <span class='inlinecode'>comm/path/tracepoint</span>, so the bottom row is per-process names, the middle row is per-file paths, and the top row is the syscall (<span class='inlinecode'>enter_read</span>, <span class='inlinecode'>enter_openat</span>, etc.). A wide bar means lots of events landed in that bucket, a narrow bar means few. There's no time axis. Left-to-right is just sort order, not chronology. The whole chart is one "where is the I/O coming from?" picture.</span><br /> <br /> -<span>One thing worth flagging because it's the unusual bit: this flamegraph is live. Most of the flamegraph tooling out there — Brendan Gregg's <span class='inlinecode'>flamegraph.pl</span>, all the <span class='inlinecode'>perf script | stackcollapse-* | flamegraph.pl</span> pipelines, every <span class='inlinecode'>pprof -web</span> invocation — produces a static SVG: capture a profile for N seconds, render once, browse the result. ior's tab is not that. Bars grow, shrink, appear, and disappear in real time as events stream in from the kernel — at full screen-refresh rate while the workload runs, with no pause. You can sit on this tab while you change something on the system (start a build, cycle a service, run a query) and watch the I/O shape mutate underneath you. That's a different mental model from the static "I have a profile, let me look at it" workflow most people are used to, and it's what makes the tab actually useful as an at-a-glance diagnostic surface rather than a post-mortem artifact.</span><br /> +<span>The unusual bit: this flamegraph is live. Most of the flamegraph tooling out there (Brendan Gregg's <span class='inlinecode'>flamegraph.pl</span>, all the <span class='inlinecode'>perf script | stackcollapse-* | flamegraph.pl</span> pipelines, every <span class='inlinecode'>pprof -web</span> invocation) produces a static SVG: capture a profile for N seconds, render once, browse the result. <span class='inlinecode'>ior</span>'s tab is not that. Bars grow, shrink, appear, and disappear in real time as events stream in from the kernel, at full screen-refresh rate while the workload runs, with no pause. You can sit on this tab while you change something on the system (start a build, cycle a service, run a query) and watch the I/O shape mutate underneath you. That's a different mental model from the static "I have a profile, let me look at it" workflow most people are used to, and it's what makes the tab actually useful as an at-a-glance diagnostic surface rather than a post-mortem artifact.</span><br /> <br /> -<span>Because it's live, there's also a way to throw away the accumulated history and start the rolling count from "now": <span class='inlinecode'>r</span> resets the baseline. Everything the flamegraph has been counting since launch (or since the last reset) is dropped, and from that moment the chart reflects only events that arrived after the reset. Useful for the "compare before vs after" workflow — change one thing on the box, hit <span class='inlinecode'>r</span> immediately, and the next thirty seconds of accumulation is a fresh picture of the new state.</span><br /> +<span>Because it's live, there's also a way to throw away the accumulated history and start the rolling count from "now": <span class='inlinecode'>r</span> resets the baseline. Everything the flamegraph has been counting since launch (or since the last reset) is dropped, and from that moment the chart reflects only events that arrived after the reset. Useful for the "compare before vs after" workflow — change one thing on the box, hit <span class='inlinecode'>r</span> immediately, and the next thirty seconds of accumulation is a fresh picture of the new state. You can also pause (and resume) the flame graph (with the <span class='inlinecode'>space</span> key) to get the static picture.</span><br /> <br /> -<span>That visualisation buys you two things you can't easily get from a tabular view. First, hierarchy: it's immediately obvious whether one process is responsible for ten thousand reads on a single file, or ten thousand reads spread across a hundred files — the first looks like one tall pillar, the second looks like a wide ridge. Second, scale: bar width is proportional to the metric (count or bytes), so a process that did 95% of the work towers over the others. The eye picks that up instantly; the same fact in a sorted table requires reading numbers and doing the ratio mentally.</span><br /> +<span>That visualisation buys you two things you can't easily get from a tabular view. First, hierarchy: it's obvious whether one process is doing ten thousand reads on a single file, or ten thousand reads spread across a hundred files. The first looks like one tall pillar, the second looks like a wide ridge. Second, scale: bar width is proportional to the metric (count or bytes), so a process that did 95% of the work towers over the others. The eye picks that up instantly. The same fact in a sorted table needs you to read numbers and do the ratio in your head.</span><br /> <br /> <span>Useful workflows you can do entirely from this tab:</span><br /> <br /> <ul> -<li>"What's pounding the disk?" — leave it on default order (<span class='inlinecode'>comm/path/tracepoint</span>) and watch which <span class='inlinecode'>comm</span> widens. Press <span class='inlinecode'>b</span> once to switch the metric to bytes if you care about throughput, not call count.</li> -<li>"Why is this one process slow?" — <span class='inlinecode'>l</span> (or <span class='inlinecode'>→</span>) until the cursor is on that process, then <span class='inlinecode'>enter</span> to zoom. The whole chart re-roots there and you see only that process's paths and syscalls.</li> -<li>"What's in /var/lib/X?" — press <span class='inlinecode'>o</span> once to flip ordering to <span class='inlinecode'>path/tracepoint/comm</span>, navigate to the path, zoom. Now the children show which syscalls hit it and which processes did them.</li> -<li>"Did the new deploy change the I/O shape?" — press <span class='inlinecode'>r</span> to reset the baseline, wait a bit, and the chart starts fresh with only events from the reset point onward. Pair the same syscall surface "before" vs "after" and the difference jumps out by shape.</li> +<li>"What's pounding the disk?" Leave it on default order (<span class='inlinecode'>comm/path/tracepoint</span>) and watch which <span class='inlinecode'>comm</span> widens. Press <span class='inlinecode'>b</span> once to switch the metric to bytes if you care about throughput, not call count.</li> +<li>"Why is this one process slow?" <span class='inlinecode'>l</span> (or <span class='inlinecode'>→</span>) until the cursor is on that process, then <span class='inlinecode'>enter</span> to zoom. The whole chart re-roots there and you see only that process's paths and syscalls.</li> +<li>"What's in /var/lib/X?" Press <span class='inlinecode'>o</span> once to flip ordering to <span class='inlinecode'>path/tracepoint/comm</span>, navigate to the path, zoom. Now the children show which syscalls hit it and which processes did them.</li> +<li>"Did the new deploy change the I/O shape?" Press <span class='inlinecode'>r</span> to reset the baseline, wait a bit, and the chart starts fresh with only events from the reset point onward. Pair the same syscall surface "before" vs "after" and the difference jumps out by shape.</li> </ul><br /> -<span>Now the keys. Movement uses vi-style <span class='inlinecode'>h</span>/<span class='inlinecode'>j</span>/<span class='inlinecode'>k</span>/<span class='inlinecode'>l</span> everywhere in ior — and the cursor keys work too if you'd rather. <span class='inlinecode'>h</span>/<span class='inlinecode'>l</span> (or <span class='inlinecode'>←</span>/<span class='inlinecode'>→</span>) walk siblings at the current depth, <span class='inlinecode'>j</span>/<span class='inlinecode'>k</span> (or <span class='inlinecode'>↓</span>/<span class='inlinecode'>↑</span>) step shallower or deeper. <span class='inlinecode'>enter</span> zooms into the selected subtree (the rest of the chart greys out and the selection becomes the new root). <span class='inlinecode'>u</span> or <span class='inlinecode'>Esc</span> undoes the zoom. <span class='inlinecode'>b</span> toggles the metric driving bar width between event count and total bytes. <span class='inlinecode'>/</span> opens regex search; matching frames stay coloured while everything else greys out, so you can use it as a filter as well as a finder. And <span class='inlinecode'>o</span> cycles between five different stack-ordering modes, each with its own lens on the data.</span><br /> +<span>Now the keys. Movement uses vi-style <span class='inlinecode'>h</span>/<span class='inlinecode'>j</span>/<span class='inlinecode'>k</span>/<span class='inlinecode'>l</span> everywhere in <span class='inlinecode'>ior</span>, and the cursor keys work too if you'd rather. <span class='inlinecode'>h</span>/<span class='inlinecode'>l</span> (or <span class='inlinecode'>←</span>/<span class='inlinecode'>→</span>) walk siblings at the current depth, <span class='inlinecode'>j</span>/<span class='inlinecode'>k</span> (or <span class='inlinecode'>↓</span>/<span class='inlinecode'>↑</span>) step shallower or deeper. <span class='inlinecode'>enter</span> zooms into the selected subtree (the rest of the chart greys out and the selection becomes the new root). <span class='inlinecode'>u</span> or <span class='inlinecode'>ESC</span> undoes the zoom. <span class='inlinecode'>b</span> toggles the metric driving bar width between event count and total bytes. <span class='inlinecode'>/</span> opens regex search; matching frames stay coloured while everything else greys out, so you can use it as a filter as well as a finder. <span class='inlinecode'>o</span> cycles between five different stack-ordering modes, each with its own lens on the data. <span class='inlinecode'>H</span> toggles a built-in help panel showing every key the current tab responds to, which is the easiest way to discover what's bound where without leaving the dashboard.</span><br /> <br /> <span>The five orderings ship as built-in presets. Read each preset name as bottom→top: the leftmost dimension is what you'll see lined up across the bottom of the chart (the root row), the next one up is its children, and the rightmost is the top row (the leaf). Switching the order changes which dimension you're scanning first when your eye starts at the bottom.</span><br /> <br /> @@ -111,7 +111,9 @@ http://www.gnu.org/software/src-highlite --> <br /> <a href='./unveiling-ior-ng/13e-order-by-process-paths.png'><img alt='Ordering 5: comm/path/tracepoint — processes at the bottom, file paths layered above instead of syscalls' title='Ordering 5: comm/path/tracepoint — processes at the bottom, file paths layered above instead of syscalls' src='./unveiling-ior-ng/13e-order-by-process-paths.png' /></a><br /> <br /> -<span>In every ordering the rule is the same: scan the bottom row to pick a "by what?" dimension, then walk up to drill in. Bar widths always mean the same thing — proportion of the active metric (events or bytes, toggled with <span class='inlinecode'>b</span>). The toolbar at the top of the chart always shows the current ordering as <span class='inlinecode'>o:order(<dim1>/<dim2>/<dim3>)</span>, so you never lose track of which lens you're looking through.</span><br /> +<span>In every ordering the rule is the same: scan the bottom row to pick a "by what?" dimension, then walk up to drill in. Bar widths always mean the same thing: proportion of the active metric (events or bytes, toggled with <span class='inlinecode'>b</span>). The toolbar at the top of the chart always shows the current ordering as <span class='inlinecode'>o:order(<dim1>/<dim2>/<dim3>)</span>, so you never lose track of which lens you're looking through.</span><br /> +<br /> +<span>If you want to skip the rotate-with-<span class='inlinecode'>o</span> dance and pick a custom three-tuple from the start, the headless side has you covered: <span class='inlinecode'>-fields comm,tracepoint,path</span> (or any other valid combination of <span class='inlinecode'>comm</span>, <span class='inlinecode'>pid</span>, <span class='inlinecode'>tid</span>, <span class='inlinecode'>tracepoint</span>, <span class='inlinecode'>path</span>) sets the collapse fields up front, and <span class='inlinecode'>-count count|bytes</span> picks the metric. Both are inherited by the live TUI flamegraph if you go that way, and they're what <span class='inlinecode'>mage demo</span> uses when it wants a specific ordering on a specific tape. Useful for scripted captures where you already know the lens you want.</span><br /> <br /> <a href='./unveiling-ior-ng/13-tui-flamegraph.gif'><img alt='Live in-TUI flamegraph: navigate, zoom, undo, cycle order + metric' title='Live in-TUI flamegraph: navigate, zoom, undo, cycle order + metric' src='./unveiling-ior-ng/13-tui-flamegraph.gif' /></a><br /> <br /> @@ -127,7 +129,7 @@ http://www.gnu.org/software/src-highlite --> <br /> <h3 style='display: inline' id='3-syscalls'><span class='inlinecode'>3</span> Syscalls</h3><br /> <br /> -<span>A sortable table of every syscall ior knows about, with rate, average latency, p95/p99, total bytes, and error count. <span class='inlinecode'>s</span> sorts by the selected column, <span class='inlinecode'>S</span> reverses. The most useful column when something's wrong is usually p99 — it's where you see the long-tail outlier syscall types.</span><br /> +<span>A sortable table of every syscall <span class='inlinecode'>ior</span> knows about, with rate, average latency, p95/p99, total bytes, and error count. <span class='inlinecode'>s</span> sorts by the selected column, <span class='inlinecode'>S</span> reverses. The most useful column when something's wrong is usually p99 — it's where you see the long-tail outlier syscall types.</span><br /> <br /> <a href='./unveiling-ior-ng/03-syscalls-tab.gif'><img alt='Syscalls table with sort + reverse-sort' title='Syscalls table with sort + reverse-sort' src='./unveiling-ior-ng/03-syscalls-tab.gif' /></a><br /> <br /> @@ -147,9 +149,9 @@ http://www.gnu.org/software/src-highlite --> <br /> <span>Two histograms side by side: how long each syscall took (latency), and the wall-clock interval between syscalls on the same thread (gap). Latency tells you "is the kernel slow"; gap tells you "what is the program doing between two kernel calls".</span><br /> <br /> -<span>A subtle but important point about that gap: ior measures it from the exit of one syscall to the entry of the next on the same TID, but it does not know what the thread was doing in the meantime. A long gap doesn't mean the thread was idle — it might have been pinned on a CPU running pure userspace code (number-crunching, JSON parsing, GC, a busy loop). All "gap" tells you for sure is "this thread didn't call into the kernel for X microseconds." Whether that's because it was sleeping, blocked on a condition variable, computing, or scheduled out is something only the gap value alone cannot answer — pair it with <span class='inlinecode'>top</span>/<span class='inlinecode'>perf top</span> if you need to disambiguate. In practice this is still extremely useful: a syscall-driven workload with surprisingly long gaps is a strong hint that you're CPU-bound somewhere outside the kernel, and that's a different optimisation conversation than slow I/O.</span><br /> +<span>One important point about that gap: <span class='inlinecode'>ior</span> measures it from the exit of one syscall to the entry of the next on the same TID, but it doesn't know what the thread was doing in the meantime. A long gap doesn't mean the thread was idle. It might have been pinned on a CPU running pure userspace code (number-crunching, JSON parsing, GC, a busy loop). All "gap" tells you for sure is "this thread didn't call into the kernel for X microseconds." Whether that's because it was sleeping, blocked on a condition variable, computing, or scheduled out is something the gap value alone cannot answer. Pair it with <span class='inlinecode'>top</span>/<span class='inlinecode'>perf top</span> if you need to disambiguate. Still useful in practice: a syscall-driven workload with surprisingly long gaps is a strong hint that you're CPU-bound somewhere outside the kernel, and that's a different optimisation conversation than slow I/O.</span><br /> <br /> -<span>The dd loop in the demo workload spreads the latency distribution out so you can actually see the shape.</span><br /> +<span>The <span class='inlinecode'>dd</span> loop in the demo workload spreads the latency distribution out so you can actually see the shape.</span><br /> <br /> <a href='./unveiling-ior-ng/06-latency-gaps-tab.gif'><img alt='Latency + gap histograms' title='Latency + gap histograms' src='./unveiling-ior-ng/06-latency-gaps-tab.gif' /></a><br /> <br /> @@ -161,7 +163,7 @@ http://www.gnu.org/software/src-highlite --> <br /> <h2 style='display: inline' id='the-stream-tab-is-the-good-one'>The Stream tab is the good one</h2><br /> <br /> -<span><span class='inlinecode'>space</span> pauses. In pause mode, the same vi-style <span class='inlinecode'>h</span>/<span class='inlinecode'>j</span>/<span class='inlinecode'>k</span>/<span class='inlinecode'>l</span> (or arrow keys) move the row/column cursor across the table. Hitting <span class='inlinecode'>Enter</span> on a cell pushes a new filter onto a stack, narrowing what you see. Pile them up — comm, then syscall, then file — and <span class='inlinecode'>Esc</span> pops them off LIFO when you want to back out.</span><br /> +<span><span class='inlinecode'>space</span> pauses. In pause mode, the same vi-style <span class='inlinecode'>h</span>/<span class='inlinecode'>j</span>/<span class='inlinecode'>k</span>/<span class='inlinecode'>l</span> (or arrow keys) move the row/column cursor across the table. Hitting <span class='inlinecode'>Enter</span> on a cell pushes a new filter onto a stack, narrowing what you see. Pile them up — comm, then syscall, then file — and <span class='inlinecode'>ESC</span> pops them off LIFO when you want to back out.</span><br /> <br /> <a href='./unveiling-ior-ng/08-stream-pause-filter.gif'><img alt='Pause, push two filters, undo with Esc' title='Pause, push two filters, undo with Esc' src='./unveiling-ior-ng/08-stream-pause-filter.gif' /></a><br /> <br /> @@ -180,13 +182,13 @@ http://www.gnu.org/software/src-highlite --> <span>The filter status line gives you a one-glance summary of every active frame, written like:</span><br /> <br /> <ul> -<li><span class='inlinecode'>comm~bash</span> — substring match on a string column. This is what Enter-on-a-cell produces for <span class='inlinecode'>comm</span>, <span class='inlinecode'>syscall</span>, and <span class='inlinecode'>file</span>.</li> -<li><span class='inlinecode'>pid=1234</span> — exact equality. Used for <span class='inlinecode'>pid</span>, <span class='inlinecode'>tid</span>, <span class='inlinecode'>fd</span>, <span class='inlinecode'>ret</span>, <span class='inlinecode'>bytes</span>.</li> -<li><span class='inlinecode'>latency>=5ms</span> / <span class='inlinecode'>gap>=10us</span> — numeric comparison with a duration suffix. The full operator set is <span class='inlinecode'>></span>, <span class='inlinecode'><</span>, <span class='inlinecode'>=</span>, <span class='inlinecode'>>=</span>, <span class='inlinecode'><=</span>, <span class='inlinecode'>!=</span>.</li> +<li><span class='inlinecode'>comm~bash</span>: substring match on a string column. This is what Enter-on-a-cell produces for <span class='inlinecode'>comm</span>, <span class='inlinecode'>syscall</span>, and <span class='inlinecode'>file</span>.</li> +<li><span class='inlinecode'>pid=1234</span>: exact equality. Used for <span class='inlinecode'>pid</span>, <span class='inlinecode'>tid</span>, <span class='inlinecode'>fd</span>, <span class='inlinecode'>ret</span>, <span class='inlinecode'>bytes</span>.</li> +<li><span class='inlinecode'>latency>=5ms</span> / <span class='inlinecode'>gap>=10us</span>: numeric comparison with a duration suffix. The full operator set is <span class='inlinecode'>></span>, <span class='inlinecode'><</span>, <span class='inlinecode'>=</span>, <span class='inlinecode'>>=</span>, <span class='inlinecode'><=</span>, <span class='inlinecode'>!=</span>.</li> </ul><br /> <span>Stack frames AND together, so pushing <span class='inlinecode'>comm~bash</span> and then <span class='inlinecode'>syscall~openat</span> shows you bash's openat calls, not bash OR openat.</span><br /> <br /> -<span>Undoing is symmetric to pushing: <span class='inlinecode'>Esc</span> pops the most recent frame off the stack — one keystroke per layer, LIFO. Press it once to drop the <span class='inlinecode'>syscall~openat</span> filter and you're back to bash-only; press it again and the <span class='inlinecode'>comm~bash</span> filter goes too, leaving the unfiltered firehose. To clear the whole stack at once, just hold <span class='inlinecode'>Esc</span> until the status line reads <span class='inlinecode'>filter: all</span>. The <span class='inlinecode'>F</span> key is a synonym for <span class='inlinecode'>Esc</span> here and works from any tab — handy from Files/Syscalls/Processes where <span class='inlinecode'>Esc</span> might otherwise close a modal first.</span><br /> +<span>Undoing is symmetric to pushing: <span class='inlinecode'>ESC</span> pops the most recent frame off the stack, one keystroke per layer, LIFO. Press it once to drop the <span class='inlinecode'>syscall~openat</span> filter and you're back to bash-only; press it again and the <span class='inlinecode'>comm~bash</span> filter goes too, leaving the unfiltered view. To clear the whole stack at once, just hold <span class='inlinecode'>ESC</span> until the status line reads <span class='inlinecode'>filter: all</span>. The <span class='inlinecode'>F</span> key is a synonym for <span class='inlinecode'>ESC</span> here and works from any tab, handy from Files/Syscalls/Processes where <span class='inlinecode'>ESC</span> might otherwise close a modal first.</span><br /> <br /> <span>Two other knobs do related work:</span><br /> <br /> @@ -201,43 +203,41 @@ http://www.gnu.org/software/src-highlite --> <span>Three persistence flows, each for a different job:</span><br /> <br /> <ul> -<li><span class='inlinecode'>R</span> from the dashboard starts streaming Parquet — every event row that survives your current TUI filter goes to disk continuously. <span class='inlinecode'>R</span> again stops. Footer shows the active file or the last error.</li> +<li><span class='inlinecode'>R</span> from the dashboard starts streaming Parquet. Every event row that survives your current TUI filter goes to disk continuously. <span class='inlinecode'>R</span> again stops. Footer shows the active file or the last error. If you don't want CSV snapshot exports at all (the <span class='inlinecode'>e</span> / <span class='inlinecode'>x</span> keys), launch with <span class='inlinecode'>-tuiExport=false</span> and those keys go away.</li> </ul><br /> <a href='./unveiling-ior-ng/12-parquet-recording.gif'><img alt='Parquet recording from the TUI' title='Parquet recording from the TUI' src='./unveiling-ior-ng/12-parquet-recording.gif' /></a><br /> <br /> <ul> -<li><span class='inlinecode'>sudo ./ior -flamegraph -name <n></span> writes one aggregated <span class='inlinecode'>.ior.zst</span> artifact at shutdown. Aggregated counters, not per-event rows. Cheaper to write, ideal for ior's native flamegraph workflow and integration tests.</li> +<li><span class='inlinecode'>sudo ./ior -flamegraph -name <n></span> writes one aggregated <span class='inlinecode'>.ior.zst</span> artifact at shutdown. Aggregated counters, not per-event rows. Cheaper to write, ideal for <span class='inlinecode'>ior</span>'s native flamegraph workflow and the integration test harness (which I'll come back to in Part 3).</li> </ul><br /> <ul> -<li><span class='inlinecode'>sudo ./ior -parquet trace.parquet</span> is the headless firehose — every row, no TUI, no filtering. <span class='inlinecode'>sudo ./ior -plain</span> is even lighter: CSV to stdout, pipe it into anything.</li> +<li><span class='inlinecode'>sudo ./ior -parquet trace.parquet</span> is the headless firehose: every row, no TUI, no filtering. <span class='inlinecode'>sudo ./ior -plain</span> is even lighter, CSV to stdout, pipe it into anything.</li> </ul><br /> <a href='./unveiling-ior-ng/14-headless-modes.gif'><img alt='All three headless flows in one tape' title='All three headless flows in one tape' src='./unveiling-ior-ng/14-headless-modes.gif' /></a><br /> <br /> <span>Once a parquet file is on disk, point any SQL-over-parquet tool at it — Part 3 walks through ClickHouse Local, with real query output against a 30-second capture.</span><br /> <br /> -<h2 style='display: inline' id='installing-it'>Installing it</h2><br /> -<br /> -<span>The Rocky Linux 9 install procedure (with its kernel-backport caveat), the eBPF / CO-RE / static-linking explanation that makes the resulting binary "compile once, run everywhere", and the build host vs. trace host split all live in Part 2. The short version: you only have to build it once. The 23 MB binary then <span class='inlinecode'>scp</span>s to anywhere.</span><br /> -<br /> -<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-2.html'>Part 2: install + compile once, run everywhere</a><br /> -<br /> <h2 style='display: inline' id='what-s-still-missing'>What's still missing</h2><br /> <br /> -<span>ior is pre-alpha and basically a personal tool. The headline gaps:</span><br /> -<br /> <ul> -<li>No record/replay — that was the whole point of the original I/O Riot. The new one is a tracer, not a workload simulator. I keep going back and forth on whether to put replay back in.</li> +<li>No record/replay. That was the whole point of the original I/O Riot. The new one is a tracer, not a workload simulator. I keep going back and forth on whether to put replay back in.</li> <li>No userspace symbol resolution. Stacks are at the syscall surface, not "which line of which library called read".</li> -<li>No remote / cluster mode. Single host, one trace at a time.</li> </ul><br /> <span>But the live flamegraph, the stackable stream filters, and the cheap parquet capture together cover the cases I actually hit week to week. The demo above is the easiest way to get a feel for whether it's the kind of tool you want.</span><br /> <br /> -<span>For installing it and the eBPF / CO-RE / static-linking story (why one build runs on every other Linux box you scp it to), see Part 2. For the per-event schema, async-syscall caveats, the probe-generator safeguard against missing new kernel syscalls, and post-mortem SQL on the parquet output, see Part 3.</span><br /> +<span>For installing it and the eBPF / CO-RE / static-linking story (why one build runs on every other Linux box you scp it to), see Part 2 (once published). For the per-event schema, async-syscall caveats, the probe-generator safeguard against missing new kernel syscalls, and post-mortem SQL on the parquet output, see Part 3 (once published.).</span><br /> <br /> -<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-2.html'>Part 2: install + compile once, run everywhere</a><br /> -<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-3.html'>Part 3: under the hood (schema, probe generator, ClickHouse)</a><br /> <a class='textlink' href='https://codeberg.org/snonux/ior'>Source on Codeberg</a><br /> <a class='textlink' href='https://codeberg.org/snonux/ior/src/branch/main/docs/tutorial/tutorial.md'>The full in-repo tutorial</a><br /> +<br /> +<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> +<br /> +<span>Other related posts are:</span><br /> +<br /> +<a class='textlink' href='./2026-05-08-unveiling-ior-ng-part-1.html'>2026-05-08 Unveiling I/O Riot NG 1.0.0 — Part 1: a guided tour (You are currently reading this)</a><br /> +<a class='textlink' href='./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html'>2018-06-01 Realistic load testing with I/O Riot for Linux</a><br /> +<br /> +<a class='textlink' href='../'>Back to the main site</a><br /> <p class="footer"> Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> | served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> | diff --git a/gemfeed/DRAFT-ior-guided-tour.html b/gemfeed/DRAFT-ior-guided-tour.html deleted file mode 100644 index 6acc3bc7..00000000 --- a/gemfeed/DRAFT-ior-guided-tour.html +++ /dev/null @@ -1,255 +0,0 @@ -<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> -<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en"> -<head> -<meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> -<title>I/O Riot NG: a guided tour</title> -<link rel="shortcut icon" type="image/gif" href="/favicon.ico" /> -<link rel="stylesheet" href="../style.css" /> -<link rel="stylesheet" href="style-override.css" /> -</head> -<body> -<p class="header"> -<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-ior-guided-tour.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-ior-guided-tour.gmi">Gemini</a> | <a href="https://snonux.foo">Microblog</a> | <a href="https://irregular.ninja">Street photography</a> -</p> -<h1 style='display: inline' id='io-riot-ng-a-guided-tour'>I/O Riot NG: a guided tour</h1><br /> -<br /> -<span class='quote'>Draft — not in the gemfeed yet. Promote with the usual rename + index dance.</span><br /> -<br /> -<span>I rewrote I/O Riot. The old one was C + Systemtap and dates from 2017. The new one — call it ior — is Go + C + BPF via libbpfgo, runs on Linux, and is mostly a TUI dashboard rather than a record/replay box. Since pictures are worth more than yet another README table of key bindings, I built a demo.</span><br /> -<br /> -<pre> - .---. - / \ - \.@-@./ - /`\_/`\ - // _ \\ - | \ )|_ - /`\_`> <_/ \ -jgs\__/'---'\__/ -</pre> -<br /> -<a class='textlink' href='https://codeberg.org/snonux/ior'>I/O Riot NG on Codeberg</a><br /> -<a class='textlink' href='./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html'>the original I/O Riot post (2018)</a><br /> -<br /> -<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> -<br /> -<ul> -<li><a href='#io-riot-ng-a-guided-tour'>I/O Riot NG: a guided tour</a></li> -<li>⇢ <a href='#what-it-does'>What it does</a></li> -<li>⇢ <a href='#a-short-detour-ebpf-and-libbpfgo'>A short detour: eBPF and libbpfgo</a></li> -<li>⇢ <a href='#the-whole-thing-as-a-tape-pipeline'>The whole thing as a tape pipeline</a></li> -<li>⇢ <a href='#first-launch'>First launch</a></li> -<li>⇢ <a href='#the-seven-tabs-in-30-seconds-each'>The seven tabs, in 30 seconds each</a></li> -<li>⇢ <a href='#the-stream-tab-is-the-good-one'>The Stream tab is the good one</a></li> -<li>⇢ <a href='#filtering-more-thoroughly'>Filtering, more thoroughly</a></li> -<li>⇢ <a href='#recording'>Recording</a></li> -<li>⇢ <a href='#querying-a-parquet-trace-with-clickhouse'>Querying a parquet trace with ClickHouse</a></li> -<li>⇢ <a href='#reproducing-the-whole-demo'>Reproducing the whole demo</a></li> -<li>⇢ <a href='#what-s-still-missing'>What's still missing</a></li> -</ul><br /> -<h2 style='display: inline' id='what-it-does'>What it does</h2><br /> -<br /> -<span>ior attaches BPF tracepoints to a chunk of the synchronous-I/O syscall surface — open, read, write, stat, mmap, sync, link, fcntl, dup, the obvious ones. Each enter/exit pair becomes an event with a duration plus an inter-syscall gap, and the events feed a Bubble Tea dashboard with seven tabs: a live flamegraph, an overview, sortable per-syscall / per-file / per-process tables, latency histograms, and a live event stream with a stackable filter UI on top.</span><br /> -<br /> -<span>Same shape as the old I/O Riot in spirit: capture what the system is actually doing, not synthetic load. Different shape in execution: no replay engine, no separate record file unless you ask for one, no kernel-debug-info dance.</span><br /> -<br /> -<h2 style='display: inline' id='a-short-detour-ebpf-and-libbpfgo'>A short detour: eBPF and libbpfgo</h2><br /> -<br /> -<span>If you haven't touched eBPF before: it's a small in-kernel bytecode VM. You compile a tiny C program, the kernel verifies it can't crash or loop forever, and then it runs every time some hook fires — a syscall enter/exit, a kprobe, a tracepoint, a network packet. The program writes events into a ring buffer that userspace mmaps and drains. No kernel module, no patched kernel, no debug symbols required.</span><br /> -<br /> -<span>ior plugs into the syscall tracepoints — <span class='inlinecode'>sys_enter_openat</span>, <span class='inlinecode'>sys_exit_read</span>, etc. — and the BPF side does the bare minimum: timestamp the event, copy a few fields, push to a perf ring buffer. All the heavy lifting (string interning, latency math, aggregation, the dashboard) is in Go on the userspace side.</span><br /> -<br /> -<span>The kernel ships a C library called libbpf that handles loading the program, attaching it to hooks, managing maps, and reading the ring buffer. There are two well-known ways to drive that from Go:</span><br /> -<br /> -<ul> -<li>libbpfgo (Aqua Security): a thin cgo wrapper around libbpf. You ship libbpf along with your binary and call into the same C API that <span class='inlinecode'>bpftool</span> and <span class='inlinecode'>perf</span> use.</li> -<li>cilium/ebpf: a from-scratch pure-Go reimplementation of everything libbpf does — ELF parser, BTF resolver, syscall layer, the lot.</li> -</ul><br /> -<span>I went with libbpfgo specifically because it's a wrapper, not a reimplementation. Whatever lands in libbpf upstream — new map types, new attach kinds, CO-RE fixes — I get for free the next kernel cycle. The pure-Go variant has to chase libbpf's feature set in parallel, and any divergence is on me to debug. For a tracer that's mostly value-add on the userspace side, "be a thin client of the kernel's own li |