1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Unveiling I/O Riot NG — Part 3: under the hood</title>
<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
<link rel="stylesheet" href="../style.css" />
<link rel="stylesheet" href="style-override.css" />
</head>
<body>
<p class="header">
<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-unveiling-ior-ng-part-3.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-unveiling-ior-ng-part-3.gmi">Gemini</a> | <a href="https://snonux.foo">Microblog</a> | <a href="https://irregular.ninja">Street photography</a>
</p>
<h1 style='display: inline' id='unveiling-io-riot-ng--part-3-under-the-hood'>Unveiling I/O Riot NG — Part 3: under the hood</h1><br />
<br />
<span class='quote'>Draft — not in the gemfeed yet. Promote with the usual rename + index dance.</span><br />
<br />
<span>This is the third and final post in the series. Part 1 is the demo-driven tour — what ior looks like, how the dashboard tabs work, how the live flamegraph reads, how filtering and recording behave. Part 2 covers the install dance for Rocky Linux 9 and the "compile once, run everywhere" portability story (eBPF, CO-RE, libbpfgo, static linking). This one is the part you read when you've got ior running and want to know what's actually in the data: the per-event schema, the safeguard that keeps syscall coverage current as new kernels ship, async-syscall caveats, and what to do with the parquet output once it's on disk.</span><br />
<br />
<span>If you haven't read Part 1, it's not a hard prerequisite — but the screenshots and key bindings referenced here all live there. Part 2 is also independent of this one; you can read them in either order.</span><br />
<br />
<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-1.html'>Part 1: a guided tour</a><br />
<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-2.html'>Part 2: install + compile once, run everywhere</a><br />
<a class='textlink' href='https://codeberg.org/snonux/ior'>I/O Riot NG on Codeberg</a><br />
<a class='textlink' href='./2018-06-01-realistic-load-testing-with-ioriot-for-linux.html'>the original I/O Riot post (2018)</a><br />
<br />
<a href='./unveiling-ior-ng/00-logo.png'><img alt='I/O Riot NG logo' title='I/O Riot NG logo' src='./unveiling-ior-ng/00-logo.png' /></a><br />
<br />
<br />
<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
<br />
<ul>
<li><a href='#unveiling-io-riot-ng--part-3-under-the-hood'>Unveiling I/O Riot NG — Part 3: under the hood</a></li>
<li>⇢ <a href='#what-ior-actually-captures-per-event'>What ior actually captures per event</a></li>
<li>⇢ ⇢ <a href='#async-syscalls-and-what-latency-means-for-them'>Async syscalls and what "latency" means for them</a></li>
<li>⇢ <a href='#keeping-up-with-new-syscalls'>Keeping up with new syscalls</a></li>
<li>⇢ <a href='#querying-a-parquet-trace-with-clickhouse'>Querying a parquet trace with ClickHouse</a></li>
<li>⇢ <a href='#asking-an-ai-to-do-the-reading-for-you'>Asking an AI to do the reading for you</a></li>
<li>⇢ <a href='#wrapping-up'>Wrapping up</a></li>
</ul><br />
<h2 style='display: inline' id='what-ior-actually-captures-per-event'>What ior actually captures per event</h2><br />
<br />
<span>Every traced syscall produces one row of structured data. The schema is what the parquet file (and the in-memory ring buffer behind the dashboard) stores, and it covers all the dimensions you'd want for a post-mortem:</span><br />
<br />
<ul>
<li><span class='inlinecode'>seq</span> — monotonically increasing sequence number, useful for joining/ordering across analysis tools.</li>
<li><span class='inlinecode'>time_ns</span> — wall-clock timestamp at syscall entry, in nanoseconds since boot.</li>
<li><span class='inlinecode'>latency_ns</span> — how long the syscall took, measured from <span class='inlinecode'>sys_enter_X</span> to <span class='inlinecode'>sys_exit_X</span> of the matching pair.</li>
<li><span class='inlinecode'>gap_ns</span> — wall-clock interval since the previous syscall on the same TID (the userspace-side breathing room — see Part 1's Latency+Gaps tab section for the caveat about what "gap" actually means).</li>
<li><span class='inlinecode'>comm</span> — the program's <span class='inlinecode'>task->comm</span> string at the time of the syscall (16 chars max, that's a kernel limit, hence truncations like "notify-rs inoti" in the demo screenshots).</li>
<li><span class='inlinecode'>pid</span>, <span class='inlinecode'>tid</span> — process and thread IDs.</li>
<li><span class='inlinecode'>syscall</span> — the syscall name, e.g. <span class='inlinecode'>read</span>, <span class='inlinecode'>openat</span>, <span class='inlinecode'>mkdir</span>.</li>
<li><span class='inlinecode'>fd</span> — the file descriptor passed in (or returned by <span class='inlinecode'>open*</span>).</li>
<li><span class='inlinecode'>ret</span> — the syscall return value: number of bytes for read/write, the new fd for open*, 0 or a negative errno otherwise. This is also where exit codes for failed calls live — anything <span class='inlinecode'>< 0</span> is a <span class='inlinecode'>-errno</span>.</li>
<li><span class='inlinecode'>bytes</span> — the byte count classified by direction: bytes read for read-class syscalls, bytes written for write-class syscalls, bytes transferred for <span class='inlinecode'>sendfile</span>/<span class='inlinecode'>splice</span>/<span class='inlinecode'>copy_file_range</span>. Lets you answer "who is hogging disk throughput?" in one query.</li>
<li><span class='inlinecode'>file</span> — the file path, where the kernel knows it. From <span class='inlinecode'>openat</span> it's the literal path argument; from <span class='inlinecode'>read</span>/<span class='inlinecode'>write</span> on an existing fd it's resolved via the fd-to-path map ior maintains in BPF.</li>
<li><span class='inlinecode'>is_error</span> — boolean shortcut: true iff <span class='inlinecode'>ret < 0</span>. Saves you a <span class='inlinecode'>WHERE</span> clause in 90% of queries.</li>
<li><span class='inlinecode'>filter_epoch</span> — bookkeeping for the global filter UI; you can ignore it for offline analysis.</li>
</ul><br />
<span>Aggregations the dashboard derives from this raw row stream — counts per syscall/comm/path, rolling rates, per-syscall latency histograms, gap histograms, top-N tables — are all just GROUP BYs over those columns.</span><br />
<br />
<h3 style='display: inline' id='async-syscalls-and-what-latency-means-for-them'>Async syscalls and what "latency" means for them</h3><br />
<br />
<span>ior attaches to enter+exit tracepoints for every file-I/O syscall, including the asynchronous ones — <span class='inlinecode'>io_uring_enter</span>, <span class='inlinecode'>io_uring_register</span>, <span class='inlinecode'>aio_*</span>, <span class='inlinecode'>sync_file_range</span>, and so on. Coverage is the same as for blocking syscalls: enter event, exit event, latency = exit − enter.</span><br />
<br />
<span>The catch is that for an async syscall, that latency does not mean what you'd intuitively expect. The whole point of an async submission is that the kernel returns immediately while the actual work — the read, the write, the fsync — runs in the background and reports completion later through a different channel (a <span class='inlinecode'>cqe</span> for io_uring, a signal or <span class='inlinecode'>aio_suspend</span> poll for POSIX AIO). So when ior tells you <span class='inlinecode'>io_uring_enter</span> took 4 µs, that's the time spent inside the kernel function ferrying submission queue entries — not the time the storage device spent doing the I/O. Those completions land separately, often on a different thread, and their timing isn't paired with the original enter event. Don't read the latency histogram for <span class='inlinecode'>io_uring_enter</span> as "io_uring is fast" — it's a different question entirely.</span><br />
<br />
<span>The flip side is that ior's per-row throughput numbers (<span class='inlinecode'>bytes</span> summed over a window) still hold for async ops, because the kernel reports the byte count at submission for the cases where it's known up front. So "what processes are dispatching the most async I/O" is a fine question to ask. "How long do those async I/Os actually take" is one ior cannot currently answer — you'd want a per-completion tracepoint pair to do that, which is on the someday-maybe list.</span><br />
<br />
<h2 style='display: inline' id='keeping-up-with-new-syscalls'>Keeping up with new syscalls</h2><br />
<br />
<span>One of the persistent problems with the original 2017 I/O Riot was that the syscall coverage was a hand-maintained list. Every kernel release added new entry points; some of them were file I/O, some weren't, and there was no automated way to spot the new arrivals. After a couple of years the list had silently rotted — entire syscalls were missing from traces, and which ones was difficult to detect because nothing flagged them.</span><br />
<br />
<span>The new ior solves this with a code generator that runs against the kernel itself. <span class='inlinecode'>mage generate</span> reads every <span class='inlinecode'>/sys/kernel/tracing/events/syscalls/*/format</span> file on the host (which is the kernel's authoritative, runtime-correct list of every syscall it knows about), parses each one, and runs it through a classifier that decides what kind of file I/O surface it is — fd-based, path-based, dup-style, fcntl-style, async-completion, etc. The recognized ones get a generated BPF handler emitted into <span class='inlinecode'>internal/c/generated_tracepoints.c</span>; the unrecognized ones get a comment line like <span class='inlinecode'>/// Ignoring sys_enter_X sys_exit_X as possibly not file I/O related</span> in the same file.</span><br />
<br />
<span>The safeguard piece is that the list of ignored syscalls is also extracted into a checked-in audit file — <span class='inlinecode'>internal/c/generated_tracepoints_result.txt</span> — and <span class='inlinecode'>mage generate</span> diffs the new run against the committed copy. If a future kernel adds a syscall ior hasn't seen before, that diff will show it, and a strict-mode regen will fail the build until a human reviews the new entry. So either the new syscall is genuinely not file-I/O (drop a one-line ignore rule, commit the diff) or it is (extend the classifier, commit the new handler). Either way it's not a silent miss.</span><br />
<br />
<span>The current numbers are 234 active tracepoint handlers (117 enter+exit pairs) and 249 ignored syscalls, all enumerated in that one generated file. Compared to the old I/O Riot, where coverage was "however much of the kernel I happened to remember to type out", that's a meaningful step forward.</span><br />
<br />
<span>(I hit this safeguard in practice during the Rocky Linux 9 install in Part 2: the committed audit was generated against a newer kernel, and the strict diff against Rocky's 5.14 refused to overwrite it. The <span class='inlinecode'>IOR_FORCE_GENERATE=1</span> env var skips the strict check and regenerates against the live kernel, which is the right thing on a fresh build host.)</span><br />
<br />
<h2 style='display: inline' id='querying-a-parquet-trace-with-clickhouse'>Querying a parquet trace with ClickHouse</h2><br />
<br />
<span>The schema is flat and stable: <span class='inlinecode'>seq, time_ns, gap_ns, latency_ns, comm, pid, tid, syscall, fd, ret, bytes, file, is_error, filter_epoch</span>. ClickHouse Local reads parquet directly without a server, which makes it a perfect post-mortem tool — point it at the file and run SQL:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>clickhouse <b><u><font color="#000000">local</font></u></b> --query <font color="#808080">"</font>
<font color="#808080"> SELECT comm, syscall, count() AS n,</font>
<font color="#808080"> formatReadableSize(sum(bytes)) AS total</font>
<font color="#808080"> FROM file('trace.parquet', Parquet)</font>
<font color="#808080"> GROUP BY comm, syscall</font>
<font color="#808080"> ORDER BY n DESC</font>
<font color="#808080"> LIMIT 10</font>
<font color="#808080">"</font> --format PrettyCompactNoEscapes
</pre>
<br />
<pre>
┌─comm────────────┬─syscall─┬─────n─┬─total──────┐
1. │ notify-rs inoti │ read │ 42005 │ 732.31 KiB │
2. │ cosmic-term │ statx │ 10898 │ 0.00 B │
3. │ cosmic-term │ read │ 10103 │ 4.02 MiB │
4. │ surface-eDP-1 │ ioctl │ 8452 │ 0.00 B │
5. │ cosmic-term │ close │ 4918 │ 0.00 B │
6. │ cosmic-term │ openat │ 4537 │ 0.00 B │
7. │ cosmic-term │ ioctl │ 3556 │ 0.00 B │
8. │ tokio-runtime-w │ read │ 1976 │ 4.04 MiB │
9. │ cosmic-comp │ read │ 1118 │ 6.63 KiB │
10. │ systemd-oomd │ read │ 1085 │ 111.97 KiB │
└─────────────────┴─────────┴───────┴────────────┘
</pre>
<br />
<span>The fields you actually want for performance work are <span class='inlinecode'>latency_ns</span> and <span class='inlinecode'>gap_ns</span>. P99 by syscall, only the ones that landed in error:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>clickhouse <b><u><font color="#000000">local</font></u></b> --query <font color="#808080">"</font>
<font color="#808080"> SELECT syscall, count() AS n,</font>
<font color="#808080"> round(quantile(0.5)(latency_ns)/1000, 1) AS p50_us,</font>
<font color="#808080"> round(quantile(0.99)(latency_ns)/1000, 1) AS p99_us</font>
<font color="#808080"> FROM file('trace.parquet', Parquet)</font>
<font color="#808080"> WHERE is_error = 1</font>
<font color="#808080"> GROUP BY syscall</font>
<font color="#808080"> ORDER BY p99_us DESC</font>
<font color="#808080">"</font> --format PrettyCompactNoEscapes
</pre>
<br />
<pre>
┌─syscall────┬─────n─┬─p50_us─┬─p99_us─┐
1. │ statx │ 1216 │ 2.2 │ 16.4 │
2. │ newfstatat │ 69 │ 1.7 │ 16.4 │
3. │ open │ 1 │ 16.1 │ 16.1 │
4. │ mkdir │ 306 │ 3.9 │ 11.7 │
5. │ readlink │ 11 │ 1.5 │ 10.4 │
6. │ newstat │ 44 │ 2.5 │ 8.4 │
7. │ unlinkat │ 347 │ 1 │ 6.2 │
8. │ openat │ 380 │ 2.1 │ 5.8 │
9. │ access │ 2 │ 5 │ 5.5 │
10. │ read │ 23597 │ 0.5 │ 5.4 │
11. │ ioctl │ 901 │ 1 │ 5.3 │
12. │ writev │ 1 │ 0.7 │ 0.7 │
└────────────┴───────┴────────┴────────┘
</pre>
<br />
<span>Real output, by the way — those rows are from a 30-second <span class='inlinecode'>ior -parquet trace.parquet</span> capture on the laptop I'm typing this on. <span class='inlinecode'>notify-rs inoti…</span> is the inotify thread of some Rust app I had open; <span class='inlinecode'>cosmic-term</span> is the COSMIC desktop's terminal emulator. The slowest p99 errors are the directory-walking syscalls (statx, newfstatat, mkdir) at ~16 µs — bog standard.</span><br />
<br />
<span>Same trick works in DuckDB (<span class='inlinecode'>duckdb -c "SELECT ... FROM 'trace.parquet'"</span>), pandas, polars, anything that reads Parquet. The point of streaming Parquet rather than ior's native <span class='inlinecode'>.ior.zst</span> format is exactly this: once it's on disk, you're in the standard data-tools ecosystem.</span><br />
<br />
<h2 style='display: inline' id='asking-an-ai-to-do-the-reading-for-you'>Asking an AI to do the reading for you</h2><br />
<br />
<span>Parquet is great if you already have an angle of attack. Sometimes you don't — you just want to know "what's hammering this box right now, and is any of it interesting?" That's where pasting a chunk of trace into an LLM and asking it to look turns out to be a surprisingly effective first pass. ior's <span class='inlinecode'>-plain</span> mode emits CSV, which is exactly the format AI assistants are best at chewing through:</span><br />
<br />
<!-- Generator: GNU source-highlight 3.1.9
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre>sudo ior -plain -duration <font color="#000000">15</font> > iotrace.csv
</pre>
<br />
<span>The CSV has a one-line ASCII banner at the top followed by the column header <span class='inlinecode'>durationToPrevNs,durationNs,comm,pid.tid,name,ret,notice,file</span> and one row per syscall. (Inside the row the <span class='inlinecode'>comm@pid.tid</span>, <span class='inlinecode'>name=>ret</span> and <span class='inlinecode'>file%(fd,flags)</span> sub-encodings keep the column count low; any half-decent assistant will figure those out unprompted.)</span><br />
<br />
<span>Hand the file to an AI with a prompt like:</span><br />
<br />
<span class='quote'>Here is 15 seconds of file-I/O syscall trace from my Fedora laptop captured with <span class='inlinecode'>ior -plain</span>. Tell me which processes are dominant, which syscalls are the noisiest, where the wall-clock time is going, and flag anything suspicious — high error rates, pointless retries, weird hot files. Use the CSV columns as documented.</span><br />
<br />
<span>What I got back on the trace I captured while writing this section (260,547 rows, 15 s, idle COSMIC desktop with htop and a tmux session open) was the following — reproduced verbatim, just to show you what the shape of the answer looks like:</span><br />
<br />
<pre>
TOP COMMANDS BY EVENT COUNT
htop 130,500 events, 243.9 ms total, 8,855 errors
notify-rs inoti 49,346 events, 42.1 ms total, 24,610 errors
cosmic-term 39,534 events, 76.4 ms total, 2,590 errors
systemd-oomd 5,857 events, 10.0 ms total, 947 errors
surface-eDP-1 3,535 events, 127.0 ms total, 317 errors ← high µs/event
TOP SYSCALLS BY TOTAL TIME (ms)
read 241.3 ms over 122,767 calls (avg 1.97 µs)
ioctl 138.0 ms over 8,416 calls (avg 16.4 µs)
openat 91.8 ms over 39,684 calls (avg 2.31 µs)
statx 19.2 ms over 12,441 calls
getdents 17.6 ms over 6,770 calls
ERROR RATES THAT STOOD OUT
mkdir 100.0% (345/345) — cosmic-term recreating config dirs that already exist (EEXIST)
newfstatat 84.3% (1,816/2,153) — systemd-oomd statting /proc/<pid>/ns/pid for already-gone processes
read 24.3% (29,816/122,767) — almost all on anon_inode:inotify and [eventfd], i.e. EAGAIN on
non-blocking fds. Normal, but it's a quarter of all reads.
NOISIEST PATHS (per ENOENT)
174× /var/lib/flatpak/exports/share/cosmic/com.system76.CosmicTheme.Mode/v1
174× /usr/local/share/cosmic/com.system76.CosmicTheme.Mode/v1
174× /home/paul/.local/share/flatpak/exports/share/cosmic/com.system76.CosmicTheme.Mode/v1
→ cosmic-term walks the XDG search path on every redraw. ENOENT-spam, not a perf bug, but worth filing.
WHAT'S ACTUALLY EXPENSIVE
• htop is the loudest tenant by event count but cheap per-event (~1.9 µs).
It walks /proc/*/{task,stat,maps,statm} every refresh — expected.
• surface-eDP-1 is the sleeper: ~36 µs/event, almost entirely DRM ioctls
on /dev/dri/renderD128 + /dev/dri/card1. That's the GPU compositor path.
• 24% of read() calls returning -EAGAIN is normal for an event-loop heavy
desktop (inotify + eventfd on every Tokio runtime), but if you're chasing
syscall overhead, batching those would matter.
NOTHING SUSPICIOUS — no unexpected processes, no surprise paths outside /proc,
/sys, /dev/dri, ~/.config and the Cosmic/Flatpak search dirs.
</pre>
<br />
<span>Two things this is unreasonably good at. First, joining the dots between <span class='inlinecode'>comm</span>, <span class='inlinecode'>syscall</span> and <span class='inlinecode'>path</span> columns to give you a one-paragraph "this is what your machine is doing" — the kind of summary that would take you ten ClickHouse queries to assemble by hand. Second, flagging things that are statistically weird without you having to know what to look for: the 100%-error <span class='inlinecode'>mkdir</span> was the EEXIST loop in cosmic-term, the 84%-error <span class='inlinecode'>newfstatat</span> was systemd-oomd racing process exits, the surface-eDP-1 outlier was the only entry on the list with high per-event latency.</span><br />
<br />
<span>A few caveats worth knowing before you rely on this:</span><br />
<br />
<ul>
<li>The CSV file gets big fast. Fifteen seconds of an idle desktop produced 260k rows / ~25 MB of CSV; on a busy server you'll want to either trim with <span class='inlinecode'>-comm</span>/<span class='inlinecode'>-path</span>/<span class='inlinecode'>-pid</span> filters at capture time or hand the AI a <span class='inlinecode'>head -100000</span> slice. Most assistants will hit context limits well before the file ends.</li>
<li>The model is reading text, not running SQL. It will round, it will sometimes miscount the long tail, and it cannot tell you a true p99 from a 250k-row sample without writing code. Treat the output as a lead-generator: it points you at suspects, then you confirm with ClickHouse on the parquet file.</li>
<li>For deeper questions ("what changed between these two traces?", "which pids dominate during the 12:34 spike?") an agentic assistant that can run shell commands does much better — it'll write the awk/clickhouse query itself, run it, and feed the result back into its own analysis.</li>
</ul><br />
<span>The combination that's worked best for me in practice: capture parquet with <span class='inlinecode'>-parquet</span>, capture a parallel <span class='inlinecode'>-plain</span> CSV slice for the AI to read, ask the AI for a triage pass, then drill into the suspects with ClickHouse on the parquet file. Triage and ground-truth, in that order.</span><br />
<br />
<h2 style='display: inline' id='wrapping-up'>Wrapping up</h2><br />
<br />
<span>That's the bottom of the stack. For the dashboard surface — what ior looks like, how the seven tabs behave, how filtering and recording work in practice — Part 1 is the demo-driven tour with all the GIFs. For the install dance and the why-the-binary-is-portable story (eBPF, CO-RE, static linking), Part 2 is the install + portability companion.</span><br />
<br />
<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-1.html'>Part 1: a guided tour</a><br />
<a class='textlink' href='./DRAFT-unveiling-ior-ng-part-2.html'>Part 2: install + compile once, run everywhere</a><br />
<a class='textlink' href='https://codeberg.org/snonux/ior'>Source on Codeberg</a><br />
<a class='textlink' href='https://codeberg.org/snonux/ior/src/branch/main/demo/TUTORIAL.md'>The full in-repo tutorial</a><br />
<p class="footer">
Generated with <a href="https://codeberg.org/snonux/gemtexter">Gemtexter 3.0.1-develop</a> |
served by <a href="https://www.OpenBSD.org">OpenBSD</a>/<a href="https://man.openbsd.org/relayd.8">relayd(8)</a>+<a href="https://man.openbsd.org/httpd.8">httpd(8)</a> |
<a href="https://foo.zone/site-mirrors.html">Site Mirrors</a>
<br />
Webring: <a href="https://shring.sh/foo.zone/previous">previous</a> | <a href="https://shring.sh">shring</a> | <a href="https://shring.sh/foo.zone/next">next</a>
</p>
</body>
</html>
|
