Update content for html

12 files changed, 1097 insertions, 173 deletions

diff --git a/about/novels.html b/about/novels.html
index 93f9d696..5c4ee243 100644
--- a/about/novels.html
+++ b/about/novels.html
@@ -66,7 +66,8 @@ _-"   .       '  +  .              .        ,//////0\ | /00HHHHHHHMMMMM
 <li>2001 - Chasm City - Revelation Space Universe, Paperback</li>
 <li>2002 - Redemption Ark (english) / Die Arche (german) - Revelation Space Universe, Paperback</li>
 <li>2003 - Absolution Gap (english) / Offenbarung (german) - Revelation Space Universe, Paperback</li>
-<li>2005 - Diamond Dogs, Turquoise Days (english ) / Träume von Unendlichkeit (german) - Revelation Space Universe, Paperback</li>
+<li>2005 - Diamond Dogs, Turquoise Days (english) / Träume von Unendlichkeit (german) - Revelation Space Universe, Paperback</li>
+<li>2021 - Inhibitor Phase - Revelation Space Universe, Audiobook (Libro.fm)</li>
 </ul><br />
 <h3 style='display: inline' id='arthur-c-clarke'>Arthur C. Clarke</h3><br />
 <br />
diff --git a/about/resources.html b/about/resources.html
index 4262f3ec..2b6f8c36 100644
--- a/about/resources.html
+++ b/about/resources.html
@@ -50,67 +50,67 @@
 <span>In random order:</span><br />
 <br />
 <ul>
-<li>DNS and BIND; Cricket Liu; O&#39;Reilly</li>
-<li>Raku Recipes; J.J. Merelo; Apress</li>
-<li>Site Reliability Engineering; How Google runs production systems; O&#39;Reilly</li>
-<li>C++ Programming Language; Bjarne Stroustrup;</li>
-<li>Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O&#39;Reilly</li>
+<li>Effective awk programming; Arnold Robbins; O&#39;Reilly</li>
 <li>97 things every SRE should know; Emil Stolarsky, Jaime Woo; O&#39;Reilly</li>
-<li>Polished Ruby Programming; Jeremy Evans; Packt Publishing</li>
-<li>Higher Order Perl; Mark Dominus; Morgan Kaufmann</li>
-<li>Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O&#39;Reilly</li>
-<li>Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers</li>
-<li>Systemprogrammierung in Go; Frank Müller; dpunkt</li>
-<li>Data Science at the Command Line; Jeroen Janssens; O&#39;Reilly</li>
 <li>Leanring eBPF; Liz Rice; O&#39;Reilly</li>
+<li>Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press</li>
+<li>Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications</li>
+<li>Pro Puppet; James Turnbull, Jeffrey McCune; Apress</li>
+<li>Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers</li>
+<li>Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf</li>
+<li>Clusterbau mit Linux-HA; Michael Schwartzkopff; O&#39;Reilly</li>
+<li>Ultimate Go Notebook; Bill Kennedy</li>
+<li>Site Reliability Engineering; How Google runs production systems; O&#39;Reilly</li>
+<li>The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton</li>
+<li>Concurrency in Go; Katherine Cox-Buday; O&#39;Reilly</li>
+<li>Perl New Features; Joshua McAdams, brian d foy; Perl School</li>
+<li>Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O&#39;Reilly</li>
+<li>Java ist auch eine Insel; Christian Ullenboom; </li>
+<li>Polished Ruby Programming; Jeremy Evans; Packt Publishing</li>
+<li>The Docker Book; James Turnbull; Kindle</li>
 <li>The Kubernetes Book; Nigel Poulton; Unabridged Audiobook</li>
+<li>DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible</li>
+<li>Systemprogrammierung in Go; Frank Müller; dpunkt</li>
+<li>The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress</li>
+<li>Higher Order Perl; Mark Dominus; Morgan Kaufmann</li>
+<li>The Pragmatic Programmer; David Thomas; Addison-Wesley</li>
+<li>Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook</li>
+<li>Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook</li>
 <li>Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O&#39;Reilly</li>
-<li>Modern Perl; Chromatic ; Onyx Neon Press</li>
-<li>Ultimate Go Notebook; Bill Kennedy</li>
 <li>100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications</li>
-<li>DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible</li>
-<li>Java ist auch eine Insel; Christian Ullenboom; </li>
+<li>Developing Games in Java; David Brackeen and others...; New Riders</li>
 <li>The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible</li>
+<li>Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson</li>
+<li>C++ Programming Language; Bjarne Stroustrup;</li>
+<li>Terraform Cookbook; Mikael Krief; Packt Publishing</li>
 <li>Funktionale Programmierung; Peter Pepper; Springer</li>
+<li>Data Science at the Command Line; Jeroen Janssens; O&#39;Reilly</li>
+<li>The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional</li>
 <li>Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers </li>
-<li>Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf</li>
-<li>Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press</li>
-<li>Effective awk programming; Arnold Robbins; O&#39;Reilly</li>
-<li>Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson</li>
-<li>Effective Java; Joshua Bloch; Addison-Wesley Professional</li>
+<li>Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall &amp; Jon Orwant; O&#39;Reilly</li>
+<li>Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O&#39;Reilly</li>
+<li>21st Century C: C Tips from the New School; Ben Klemens; O&#39;Reilly</li>
+<li>Modern Perl; Chromatic ; Onyx Neon Press</li>
 <li>Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press</li>
-<li>Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook</li>
 <li>Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt </li>
-<li>Clusterbau mit Linux-HA; Michael Schwartzkopff; O&#39;Reilly</li>
-<li>Developing Games in Java; David Brackeen and others...; New Riders</li>
-<li>Concurrency in Go; Katherine Cox-Buday; O&#39;Reilly</li>
-<li>Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner</li>
-<li>The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton</li>
-<li>Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook</li>
-<li>The Docker Book; James Turnbull; Kindle</li>
-<li>21st Century C: C Tips from the New School; Ben Klemens; O&#39;Reilly</li>
-<li>The Pragmatic Programmer; David Thomas; Addison-Wesley</li>
+<li>Raku Recipes; J.J. Merelo; Apress</li>
 <li>Raku Fundamentals; Moritz Lenz; Apress</li>
-<li>Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall &amp; Jon Orwant; O&#39;Reilly</li>
-<li>Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications</li>
-<li>Terraform Cookbook; Mikael Krief; Packt Publishing</li>
-<li>Pro Puppet; James Turnbull, Jeffrey McCune; Apress</li>
-<li>The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional</li>
-<li>Perl New Features; Joshua McAdams, brian d foy; Perl School</li>
-<li>The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress</li>
+<li>Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner</li>
+<li>DNS and BIND; Cricket Liu; O&#39;Reilly</li>
+<li>Effective Java; Joshua Bloch; Addison-Wesley Professional</li>
 </ul><br />
 <h2 style='display: inline' id='technical-references'>Technical references</h2><br />
 <br />
 <span>I didn&#39;t read them from the beginning to the end, but I am using them to look up things. The books are in random order:</span><br />
 <br />
 <ul>
-<li>Implementing Service Level Objectives; Alex Hidalgo; O&#39;Reilly</li>
+<li>The Linux Programming Interface; Michael Kerrisk; No Starch Press </li>
 <li>Go: Design Patterns for Real-World Projects; Mat Ryer; Packt</li>
-<li>Relayd and Httpd Mastery; Michael W Lucas</li>
-<li>Groovy Kurz &amp; Gut; Joerg Staudemeier; O&#39;Reilly</li>
 <li>Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O&#39;Reilly</li>
-<li>The Linux Programming Interface; Michael Kerrisk; No Starch Press </li>
 <li>Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley</li>
+<li>Relayd and Httpd Mastery; Michael W Lucas</li>
+<li>Implementing Service Level Objectives; Alex Hidalgo; O&#39;Reilly</li>
+<li>Groovy Kurz &amp; Gut; Joerg Staudemeier; O&#39;Reilly</li>
 <li>BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley</li>
 </ul><br />
 <h2 style='display: inline' id='self-development-and-soft-skills-books'>Self-development and soft-skills books</h2><br />
@@ -118,44 +118,44 @@
 <span>In random order:</span><br />
 <br />
 <ul>
-<li>The Bullet Journal Method; Ryder Carroll; Fourth Estate</li>
-<li>Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook</li>
-<li>The Complete Software Developer&#39;s Career Guide; John Sonmez; Unabridged Audiobook</li>
-<li>Ultralearning; Scott Young; Thorsons</li>
-<li>Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion</li>
-<li>Slow Productivity; Cal Newport; Penguin Random House</li>
-<li>The Joy of Missing Out; Christina Crook; New Society Publishers</li>
-<li>Ultralearning; Anna Laurent; Self-published via Amazon</li>
+<li>Stop starting, start finishing; Arne Roock; Lean-Kanban University  </li>
+<li>Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press</li>
+<li>Eat That Frog; Brian Tracy</li>
+<li>The Software Engineer&#39;s Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook </li>
 <li>So Good They Can&#39;t Ignore You; Cal Newport; Business Plus</li>
-<li>Deep Work; Cal Newport; Piatkus</li>
-<li>The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook</li>
-<li>Atomic Habits; James Clear; Random House Business</li>
+<li>Influence without Authority; A. Cohen, D. Bradford; Wiley</li>
+<li>The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select</li>
 <li>Psycho-Cybernetics; Maxwell Maltz; Perigee Books</li>
-<li>Soft Skills; John Sommez; Manning Publications</li>
-<li>Eat That Frog!; Brian Tracy; Hodder Paperbacks</li>
-<li>Never Split the Difference; Chris Voss, Tahl Raz; Random House Business</li>
-<li>Digital Minimalism; Cal Newport; Portofolio Penguin</li>
-<li>The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd</li>
-<li>Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing</li>
 <li>Meditation for Mortals, Oliver Burkeman, Audiobook</li>
+<li>Atomic Habits; James Clear; Random House Business</li>
+<li>Getting Things Done; David Allen</li>
+<li>Time Management for System Administrators; Thomas A. Limoncelli; O&#39;Reilly</li>
+<li>Solve for Happy; Mo Gawdat (RE-READ 1ST TIME)</li>
 <li>Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne</li>
-<li>Influence without Authority; A. Cohen, D. Bradford; Wiley</li>
-<li>The Power of Now; Eckhard Tolle; Yellow Kite</li>
-<li>The Good Enough Job; Simone Stolzoff; Ebury Edge</li>
+<li>Digital Minimalism; Cal Newport; Portofolio Penguin</li>
 <li>101 Essays that change the way you think; Brianna Wiest; Audiobook</li>
-<li>The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon &amp; Schuster UK</li>
-<li>The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME)</li>
-<li>Solve for Happy; Mo Gawdat (RE-READ 1ST TIME)</li>
-<li>The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select</li>
+<li>Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook</li>
 <li>Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook</li>
+<li>The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook</li>
+<li>Ultralearning; Anna Laurent; Self-published via Amazon</li>
+<li>The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd</li>
 <li>97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook</li>
-<li>Getting Things Done; David Allen</li>
-<li>Eat That Frog; Brian Tracy</li>
-<li>Time Management for System Administrators; Thomas A. Limoncelli; O&#39;Reilly</li>
+<li>The Good Enough Job; Simone Stolzoff; Ebury Edge</li>
+<li>The Joy of Missing Out; Christina Crook; New Society Publishers</li>
+<li>Never Split the Difference; Chris Voss, Tahl Raz; Random House Business</li>
 <li>The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books</li>
-<li>Stop starting, start finishing; Arne Roock; Lean-Kanban University  </li>
-<li>The Software Engineer&#39;s Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook </li>
-<li>Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press</li>
+<li>The Bullet Journal Method; Ryder Carroll; Fourth Estate</li>
+<li>The Complete Software Developer&#39;s Career Guide; John Sonmez; Unabridged Audiobook</li>
+<li>The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME)</li>
+<li>Slow Productivity; Cal Newport; Penguin Random House</li>
+<li>The Power of Now; Eckhard Tolle; Yellow Kite</li>
+<li>Eat That Frog!; Brian Tracy; Hodder Paperbacks</li>
+<li>Deep Work; Cal Newport; Piatkus</li>
+<li>Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing</li>
+<li>Soft Skills; John Sommez; Manning Publications</li>
+<li>Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion</li>
+<li>Ultralearning; Scott Young; Thorsons</li>
+<li>The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon &amp; Schuster UK</li>
 </ul><br />
 <a class='textlink' href='../notes/index.html'>Here are notes of mine for some of the books</a><br />
 <br />
@@ -165,29 +165,29 @@
 <br />
 <ul>
 <li>Ultimate Go Programming; Bill Kennedy; O&#39;Reilly Online</li>
-<li>Functional programming lecture; Remote University of Hagen</li>
-<li>Structure and Interpretation of Computer Programs; Harold Abelson and more...; </li>
-<li>Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon</li>
+<li>Scripting Vim; Damian Conway; O&#39;Reilly Online</li>
 <li>MySQL Deep Dive Workshop; 2-day on-site training</li>
-<li>Apache Tomcat Best Practises; 3-day on-site training</li>
-<li>Protocol buffers; O&#39;Reilly Online</li>
-<li>Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training</li>
+<li>Developing IaC with Terraform (with Live Lessons); O&#39;Reilly Online</li>
 <li>Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need)</li>
+<li>Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon</li>
+<li>The Ultimate Kubernetes Bootcamp; School of Devops; O&#39;Reilly Online</li>
+<li>Functional programming lecture; Remote University of Hagen</li>
 <li>Algorithms Video Lectures; Robert Sedgewick; O&#39;Reilly Online</li>
+<li>AWS Immersion Day; Amazon; 1-day interactive online training </li>
+<li>Apache Tomcat Best Practises; 3-day on-site training</li>
+<li>Structure and Interpretation of Computer Programs; Harold Abelson and more...; </li>
+<li>Protocol buffers; O&#39;Reilly Online</li>
 <li>F5 Loadbalancers Training; 2-day on-site training; F5, Inc. </li>
-<li>The Ultimate Kubernetes Bootcamp; School of Devops; O&#39;Reilly Online</li>
-<li>Scripting Vim; Damian Conway; O&#39;Reilly Online</li>
+<li>Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training</li>
 <li>The Well-Grounded Rubyist Video Edition; David. A. Black; O&#39;Reilly Online</li>
-<li>AWS Immersion Day; Amazon; 1-day interactive online training </li>
-<li>Developing IaC with Terraform (with Live Lessons); O&#39;Reilly Online</li>
 </ul><br />
 <h2 style='display: inline' id='technical-guides'>Technical guides</h2><br />
 <br />
 <span>These are not whole books, but guides (smaller or larger) which I found very useful. in random order:</span><br />
 <br />
 <ul>
-<li>Advanced Bash-Scripting Guide </li>
 <li>How CPUs work at https://cpu.land</li>
+<li>Advanced Bash-Scripting Guide </li>
 <li>Raku Guide at https://raku.guide  </li>
 </ul><br />
 <h2 style='display: inline' id='podcasts'>Podcasts</h2><br />
@@ -197,21 +197,21 @@
 <span>In random order:</span><br />
 <br />
 <ul>
-<li>Modern Mentor</li>
-<li>BSD Now [BSD]</li>
 <li>Hidden Brain</li>
+<li>Cup o&#39; Go [Golang]</li>
+<li>Deep Questions with Cal Newport</li>
+<li>Pratical AI</li>
+<li>The Pragmatic Engineer Podcast</li>
+<li>Wednesday Wisdom</li>
 <li>Backend Banter</li>
-<li>The Changelog Podcast(s)</li>
+<li>Modern Mentor</li>
 <li>Dev Interrupted</li>
-<li>Wednesday Wisdom</li>
-<li>The Pragmatic Engineer Podcast</li>
-<li>Pratical AI</li>
 <li>The ProdCast (Google SRE Podcast)</li>
 <li>Fallthrough [Golang]</li>
-<li>Cup o&#39; Go [Golang]</li>
 <li>Fork Around And Find Out</li>
+<li>The Changelog Podcast(s)</li>
+<li>BSD Now [BSD]</li>
 <li>Maintainable</li>
-<li>Deep Questions with Cal Newport</li>
 </ul><br />
 <h3 style='display: inline' id='podcasts-i-liked'>Podcasts I liked</h3><br />
 <br />
@@ -219,38 +219,38 @@
 <br />
 <ul>
 <li>Java Pub House</li>
-<li>Go Time (predecessor of fallthrough)</li>
 <li>Ship It (predecessor of Fork Around And Find Out)</li>
+<li>Modern Mentor</li>
 <li>FLOSS weekly</li>
+<li>Go Time (predecessor of fallthrough)</li>
 <li>CRE: Chaosradio Express [german]</li>
-<li>Modern Mentor</li>
 </ul><br />
 <h2 style='display: inline' id='newsletters-i-like'>Newsletters I like</h2><br />
 <br />
 <span>This is a mix of tech and non-tech newsletters I am subscribed to. In random order:</span><br />
 <br />
 <ul>
-<li>The Pragmatic Engineer</li>
-<li>Changelog News</li>
-<li>Golang Weekly</li>
-<li>Andreas Brandhorst Newsletter (Sci-Fi author)</li>
-<li>Ruby Weekly</li>
-<li>Register Spill</li>
+<li>The Valuable Dev</li>
 <li>Applied Go Weekly Newsletter</li>
 <li>Monospace Mentor</li>
-<li>The Imperfectionist</li>
 <li>VK Newsletter</li>
-<li>The Valuable Dev</li>
 <li>byteSizeGo</li>
+<li>Andreas Brandhorst Newsletter (Sci-Fi author)</li>
+<li>Register Spill</li>
+<li>Golang Weekly</li>
+<li>Changelog News</li>
+<li>The Pragmatic Engineer</li>
+<li>The Imperfectionist</li>
+<li>Ruby Weekly</li>
 </ul><br />
 <h2 style='display: inline' id='magazines-i-liked'>Magazines I like(d)</h2><br />
 <br />
 <span>This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order:</span><br />
 <br />
 <ul>
+<li>LWN (online only)</li>
 <li>Linux Magazine</li>
 <li>Linux User</li>
-<li>LWN (online only)</li>
 <li>freeX (not published anymore)</li>
 </ul><br />
 <h1 style='display: inline' id='formal-education'>Formal education</h1><br />
diff --git a/gemfeed/DRAFT-x-rag-observability-hackathon.html b/gemfeed/DRAFT-x-rag-observability-hackathon.html
new file mode 100644
index 00000000..cffd81b1
--- /dev/null
+++ b/gemfeed/DRAFT-x-rag-observability-hackathon.html
@@ -0,0 +1,923 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>Adding Observability to X-RAG</title>
+<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
+<link rel="stylesheet" href="../style.css" />
+<link rel="stylesheet" href="style-override.css" />
+</head>
+<body>
+<p class="header">
+<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-x-rag-observability-hackathon.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-x-rag-observability-hackathon.gmi">Gemini</a>
+</p>
+<h1 style='display: inline' id='adding-observability-to-x-rag'>Adding Observability to X-RAG</h1><br />
+<br />
+<span>This blog post describes my hackathon efforts adding observability to X-RAG, a distributed Retrieval-Augmented Generation (RAG) platform built by my brother Florian. I especially made time available over the weekend to join his 3-day hackathon (attending 2 days) with the goal of instrumenting his existing distributed system with observability. What started as "let&#39;s add some metrics" turned into a comprehensive implementation of the three pillars of observability: tracing, metrics, and logs.</span><br />
+<br />
+<a class='textlink' href='https://github.com/florianbuetow/x-rag'>X-RAG source code on GitHub</a><br />
+<br />
+<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
+<br />
+<ul>
+<li><a href='#adding-observability-to-x-rag'>Adding Observability to X-RAG</a></li>
+<li>⇢ <a href='#what-is-x-rag'>What is X-RAG?</a></li>
+<li>⇢ <a href='#running-kubernetes-locally-with-kind'>Running Kubernetes locally with Kind</a></li>
+<li>⇢ <a href='#motivation'>Motivation</a></li>
+<li>⇢ <a href='#the-observability-stack'>The observability stack</a></li>
+<li>⇢ <a href='#grafana-alloy-the-unified-collector'>Grafana Alloy: the unified collector</a></li>
+<li>⇢ <a href='#centralised-logging-with-loki'>Centralised logging with Loki</a></li>
+<li>⇢ ⇢ <a href='#alloy-configuration-for-logs'>Alloy configuration for logs</a></li>
+<li>⇢ ⇢ <a href='#querying-logs-with-logql'>Querying logs with LogQL</a></li>
+<li>⇢ <a href='#metrics-with-prometheus'>Metrics with Prometheus</a></li>
+<li>⇢ ⇢ <a href='#alloy-configuration-for-application-metrics'>Alloy configuration for application metrics</a></li>
+<li>⇢ ⇢ <a href='#kubernetes-metrics-kubelet-cadvisor-and-kube-state-metrics'>Kubernetes metrics: kubelet, cAdvisor, and kube-state-metrics</a></li>
+<li>⇢ ⇢ <a href='#infrastructure-metrics-kafka-redis-minio'>Infrastructure metrics: Kafka, Redis, MinIO</a></li>
+<li>⇢ <a href='#distributed-tracing-with-tempo'>Distributed tracing with Tempo</a></li>
+<li>⇢ ⇢ <a href='#understanding-traces-spans-and-the-trace-tree'>Understanding traces, spans, and the trace tree</a></li>
+<li>⇢ ⇢ <a href='#how-trace-context-propagates'>How trace context propagates</a></li>
+<li>⇢ ⇢ <a href='#implementation'>Implementation</a></li>
+<li>⇢ ⇢ <a href='#alloy-configuration-for-traces'>Alloy configuration for traces</a></li>
+<li>⇢ <a href='#async-ingestion-trace-walkthrough'>Async ingestion trace walkthrough</a></li>
+<li>⇢ ⇢ <a href='#step-1-ingest-a-document'>Step 1: Ingest a document</a></li>
+<li>⇢ ⇢ <a href='#step-2-find-the-ingestion-trace'>Step 2: Find the ingestion trace</a></li>
+<li>⇢ ⇢ <a href='#step-3-fetch-the-complete-trace'>Step 3: Fetch the complete trace</a></li>
+<li>⇢ ⇢ <a href='#step-4-analyse-the-async-trace'>Step 4: Analyse the async trace</a></li>
+<li>⇢ ⇢ <a href='#viewing-traces-in-grafana'>Viewing traces in Grafana</a></li>
+<li>⇢ <a href='#end-to-end-search-trace-walkthrough'>End-to-end search trace walkthrough</a></li>
+<li>⇢ ⇢ <a href='#step-1-make-a-search-request'>Step 1: Make a search request</a></li>
+<li>⇢ ⇢ <a href='#step-2-query-tempo-for-the-trace'>Step 2: Query Tempo for the trace</a></li>
+<li>⇢ ⇢ <a href='#step-3-analyse-the-trace'>Step 3: Analyse the trace</a></li>
+<li>⇢ ⇢ <a href='#step-4-search-traces-with-traceql'>Step 4: Search traces with TraceQL</a></li>
+<li>⇢ ⇢ <a href='#viewing-the-search-trace-in-grafana'>Viewing the search trace in Grafana</a></li>
+<li>⇢ <a href='#correlating-the-three-signals'>Correlating the three signals</a></li>
+<li>⇢ <a href='#grafana-dashboards'>Grafana dashboards</a></li>
+<li>⇢ <a href='#results-two-days-well-spent'>Results: two days well spent</a></li>
+<li>⇢ <a href='#slis-slos-and-slas'>SLIs, SLOs and SLAs</a></li>
+<li>⇢ <a href='#using-amp-for-ai-assisted-development'>Using Amp for AI-assisted development</a></li>
+<li>⇢ <a href='#other-changes-along-the-way'>Other changes along the way</a></li>
+<li>⇢ <a href='#lessons-learned'>Lessons learned</a></li>
+</ul><br />
+<h2 style='display: inline' id='what-is-x-rag'>What is X-RAG?</h2><br />
+<br />
+<span>X-RAG is a distributed RAG (Retrieval-Augmented Generation) platform running on Kubernetes. The idea behind RAG is simple: instead of asking an LLM to answer questions from its training data alone, you first retrieve relevant documents from your own knowledge base, then feed those documents to the LLM as context. The LLM synthesises an answer grounded in your actual content—reducing hallucinations and enabling answers about private or recent information the model was never trained on.</span><br />
+<br />
+<span>X-RAG handles the full pipeline: ingest documents, chunk them into searchable pieces, generate vector embeddings, store them in a vector database, and at query time, retrieve relevant chunks and pass them to an LLM for answer generation. The system supports both local LLMs (Florian runs his on a beefy desktop) and cloud APIs like OpenAI. I configured an OpenAI API key since my laptop&#39;s CPU and GPU aren&#39;t fast enough for decent local inference.</span><br />
+<br />
+<span>All services are implemented in Python. I&#39;m more used to Ruby, Go, and Bash these days, but for this project it didn&#39;t matter—Python&#39;s OpenTelemetry integration is straightforward, I wasn&#39;t planning to write or rewrite tons of application code, and with GenAI assistance the language barrier was a non-issue. The OpenTelemetry concepts and patterns should translate to other languages too—the SDK APIs are intentionally similar across Python, Go, Java, and others.</span><br />
+<br />
+<span>X-RAG consists of several independently scalable microservices:</span><br />
+<br />
+<ul>
+<li>Search UI: FastAPI web interface for queries</li>
+<li>Ingestion API: Document upload endpoint</li>
+<li>Embedding Service: gRPC service for vector embeddings</li>
+<li>Indexer: Kafka consumer that processes documents</li>
+<li>Search Service: gRPC service orchestrating the RAG pipeline</li>
+</ul><br />
+<span>The Embedding Service deserves extra explanation because in the beginning I didn&#39;t really knew what it was. Text isn&#39;t directly searchable in a vector database—you need to convert it to numerical vectors (embeddings) that capture semantic meaning. The Embedding Service takes text chunks and calls an embedding model (OpenAI&#39;s <span class='inlinecode'>text-embedding-3-small</span> in my case, or a local model on Florian&#39;s setup) to produce these vectors. For the LLM search completion answer, I used <span class='inlinecode'>gpt-4o-mini</span>.</span><br />
+<br />
+<span>Similar concepts end up with similar vectors, so "What is machine learning?" and "Explain ML" produce vectors close together in the embedding space. At query time, your question gets embedded too, and the vector database finds chunks with nearby vectors—that&#39;s semantic search.</span><br />
+<br />
+<span>The data layer includes Weaviate (vector database with hybrid search), Kafka (message queue), MinIO (object storage), and Redis (cache). All of this runs in a Kind Kubernetes cluster for local development, with the same manifests deployable to production.</span><br />
+<br />
+<pre>
+┌─────────────────────────────────────────────────────────────────────────┐
+│                      X-RAG Kubernetes Cluster                            │
+├─────────────────────────────────────────────────────────────────────────┤
+│   ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐    │
+│   │ Search UI   │  │Search Svc   │  │Embed Service│  │   Indexer   │    │
+│   └──────┬──────┘  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘    │
+│          │                │                │                │           │
+│          └────────────────┴────────────────┴────────────────┘           │
+│                                    │                                     │
+│                                    ▼                                     │
+│          ┌─────────────┐  ┌─────────────┐  ┌─────────────┐              │
+│          │  Weaviate   │  │   Kafka     │  │   MinIO     │              │
+│          └─────────────┘  └─────────────┘  └─────────────┘              │
+└─────────────────────────────────────────────────────────────────────────┘
+</pre>
+<br />
+<h2 style='display: inline' id='running-kubernetes-locally-with-kind'>Running Kubernetes locally with Kind</h2><br />
+<br />
+<span>X-RAG runs on Kubernetes, but you don&#39;t need a cloud account to develop it. The project uses Kind (Kubernetes in Docker)—a tool originally created by the Kubernetes SIG for testing Kubernetes itself.</span><br />
+<br />
+<a class='textlink' href='https://kind.sigs.k8s.io/'>Kind - Kubernetes in Docker</a><br />
+<br />
+<span>Kind spins up a full Kubernetes cluster using Docker containers as nodes. The control plane (API server, etcd, scheduler, controller-manager) runs in one container, and worker nodes run in separate containers. Inside these "node containers," pods run just like they would on real servers—using containerd as the container runtime. It&#39;s containers all the way down.</span><br />
+<br />
+<span>Technically, each Kind node is a Docker container running a minimal Linux image with kubelet and containerd installed. When you deploy a pod, kubelet inside the node container instructs containerd to pull and run the container image. So you have Docker running node containers, and inside those, containerd running application containers. Network-wise, Kind sets up a Docker bridge network and uses CNI plugins (kindnet by default) for pod networking within the cluster.</span><br />
+<br />
+<pre>
+$ docker ps --format "table {{.Names}}\t{{.Image}}"
+NAMES                  IMAGE
+xrag-k8-control-plane  kindest/node:v1.32.0
+xrag-k8-worker         kindest/node:v1.32.0
+xrag-k8-worker2        kindest/node:v1.32.0
+</pre>
+<br />
+<span>The <span class='inlinecode'>kindest/node</span> image contains everything needed: kubelet, containerd, CNI plugins, and pre-pulled pause containers. Port mappings in the Kind config expose services to the host—that&#39;s how http://localhost:8080 reaches the search-ui running inside a pod, inside a worker container, inside Docker.</span><br />
+<br />
+<pre>
+┌─────────────────────────────────────────────────────────────────────────┐
+│                           Docker Host                                    │
+├─────────────────────────────────────────────────────────────────────────┤
+│  ┌───────────────────┐  ┌───────────────────┐  ┌───────────────────┐    │
+│  │ xrag-k8-control   │  │ xrag-k8-worker    │  │ xrag-k8-worker2   │    │
+│  │ -plane (container)│  │ (container)       │  │ (container)       │    │
+│  │                   │  │                   │  │                   │    │
+│  │ K8s API server    │  │ Pods:             │  │ Pods:             │    │
+│  │ etcd, scheduler   │  │ • search-ui       │  │ • weaviate        │    │
+│  │                   │  │ • search-service  │  │ • kafka           │    │
+│  │                   │  │ • embedding-svc   │  │ • prometheus      │    │
+│  │                   │  │ • indexer         │  │ • grafana         │    │
+│  └───────────────────┘  └───────────────────┘  └───────────────────┘    │
+└─────────────────────────────────────────────────────────────────────────┘
+</pre>
+<br />
+<span>Why Kind? It gives you a real Kubernetes environment—the same manifests deploy to production clouds unchanged. No minikube quirks, no Docker Compose translation layer. Just Kubernetes. I already have a k3s cluster running at home, but Kind made collaboration easier—everyone working on X-RAG gets the exact same setup by cloning the repo and running <span class='inlinecode'>make cluster-start</span>.</span><br />
+<br />
+<span>Florian developed X-RAG on macOS, but it worked seamlessly on my Linux laptop. The only difference was Docker&#39;s resource allocation: on macOS you configure limits in Docker Desktop, on Linux it uses host resources directly. That&#39;s because under macOS the Linux Docker containers run on an emulation layer as macOS is not Linux.</span><br />
+<br />
+<span>My hardware: a ThinkPad X1 Carbon Gen 9 with an 11th Gen Intel Core i7-1185G7 (4 cores, 8 threads at 3.00GHz) and 32GB RAM (running Fedora Linux). During the hackathon, memory usage peaked around 15GB—comfortable headroom. CPU was the bottleneck; with ~38 pods running across all namespaces (rag-system, monitoring, kube-system, etc.), plus Discord for the remote video call and Tidal streaming hi-res music, things got tight. When rebuilding Docker images or restarting the cluster, Discord video and audio would stutter—my fellow hackers probably wondered why I kept freezing mid-sentence. A beefier CPU would have meant less waiting and smoother calls, but it was manageable.</span><br />
+<br />
+<h2 style='display: inline' id='motivation'>Motivation</h2><br />
+<br />
+<span>When I joined the hackathon, Florian&#39;s X-RAG was functional but opaque. With five services communicating via gRPC, Kafka, and HTTP, debugging was cumbersome. When a search request take 5 seconds, there was no visibility into where the time was being spent. Was it the embedding generation? The vector search? The LLM synthesis? Nobody would be able to figure it out quickly.</span><br />
+<br />
+<span>Distributed systems are inherently opaque. Each service logs its own view of the world, but correlating events across service boundaries is archaeology. Grepping through logs on many pods, trying to mentally reconstruct what happened—not fun. This was the perfect hackathon project: Explore this Observability Stack in greater depth.</span><br />
+<br />
+<h2 style='display: inline' id='the-observability-stack'>The observability stack</h2><br />
+<br />
+<span>Before diving into implementation, here&#39;s what I deployed. The complete stack runs in the monitoring namespace:</span><br />
+<br />
+<pre>
+$ kubectl get pods -n monitoring
+NAME                                  READY   STATUS
+alloy-84ddf4cd8c-7phjp                1/1     Running
+grafana-6fcc89b4d6-pnh8l              1/1     Running
+kube-state-metrics-5d954c569f-2r45n   1/1     Running
+loki-8c9bbf744-sc2p5                  1/1     Running
+node-exporter-kb8zz                   1/1     Running
+node-exporter-zcrdz                   1/1     Running
+node-exporter-zmskc                   1/1     Running
+prometheus-7f755f675-dqcht            1/1     Running
+tempo-55df7dbcdd-t8fg9                1/1     Running
+</pre>
+<br />
+<span>Each component has a specific role:</span><br />
+<br />
+<ul>
+<li><span class='inlinecode'>Grafana Alloy</span>: The unified collector. Receives OTLP from applications, scrapes Prometheus endpoints, tails log files. Think of it as the central nervous system.</li>
+<li><span class='inlinecode'>Prometheus</span>: Time-series database for metrics. Stores counters, gauges, and histograms with 15-day retention.</li>
+<li><span class='inlinecode'>Tempo</span>: Trace storage. Receives spans via OTLP, correlates them by trace ID, enables TraceQL queries.</li>
+<li><span class='inlinecode'>Loki</span>: Log aggregation. Indexes labels (namespace, pod, container), stores log chunks, enables LogQL queries.</li>
+<li><span class='inlinecode'>Grafana</span>: The unified UI. Queries all thre