diff options
| -rw-r--r-- | about/resources.gmi | 218 | ||||
| -rw-r--r-- | gemfeed/2025-12-24-x-rag-observability-hackathon.gmi (renamed from gemfeed/DRAFT-x-rag-observability-hackathon.gmi) | 34 | ||||
| -rw-r--r-- | gemfeed/2025-12-24-x-rag-observability-hackathon.gmi.tpl (renamed from gemfeed/DRAFT-x-rag-observability-hackathon.gmi.tpl) | 2 | ||||
| -rw-r--r-- | gemfeed/atom.xml | 1066 | ||||
| -rw-r--r-- | gemfeed/index.gmi | 1 | ||||
| -rw-r--r-- | index.gmi | 3 | ||||
| -rw-r--r-- | uptime-stats.gmi | 2 |
7 files changed, 1050 insertions, 276 deletions
diff --git a/about/resources.gmi b/about/resources.gmi index b57a6c21..6a3e5162 100644 --- a/about/resources.gmi +++ b/about/resources.gmi @@ -35,110 +35,110 @@ You won't find any links on this site because, over time, the links will break. In random order: -* Effective awk programming; Arnold Robbins; O'Reilly -* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly -* Leanring eBPF; Liz Rice; O'Reilly -* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press -* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications -* Pro Puppet; James Turnbull, Jeffrey McCune; Apress -* Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers -* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf -* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly -* Ultimate Go Notebook; Bill Kennedy -* Site Reliability Engineering; How Google runs production systems; O'Reilly -* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton -* Concurrency in Go; Katherine Cox-Buday; O'Reilly -* Perl New Features; Joshua McAdams, brian d foy; Perl School -* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly +* Effective Java; Joshua Bloch; Addison-Wesley Professional * Java ist auch eine Insel; Christian Ullenboom; -* Polished Ruby Programming; Jeremy Evans; Packt Publishing * The Docker Book; James Turnbull; Kindle -* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook -* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible -* Systemprogrammierung in Go; Frank Müller; dpunkt -* The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress -* Higher Order Perl; Mark Dominus; Morgan Kaufmann -* The Pragmatic Programmer; David Thomas; Addison-Wesley -* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook -* Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook -* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly -* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications -* Developing Games in Java; David Brackeen and others...; New Riders -* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible -* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson -* C++ Programming Language; Bjarne Stroustrup; -* Terraform Cookbook; Mikael Krief; Packt Publishing +* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press * Funktionale Programmierung; Peter Pepper; Springer -* Data Science at the Command Line; Jeroen Janssens; O'Reilly -* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional -* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* Developing Games in Java; David Brackeen and others...; New Riders * Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly -* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly -* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly -* Modern Perl; Chromatic ; Onyx Neon Press -* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press +* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf +* Effective awk programming; Arnold Robbins; O'Reilly +* Seeking SRE: Conversations About Running Production Systems at Scale; David N. Blank-Edelman; eBook +* Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers * Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt +* Higher Order Perl; Mark Dominus; Morgan Kaufmann +* Site Reliability Engineering; How Google runs production systems; O'Reilly +* C++ Programming Language; Bjarne Stroustrup; * Raku Recipes; J.J. Merelo; Apress +* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional +* Leanring eBPF; Liz Rice; O'Reilly +* Modern Perl; Chromatic ; Onyx Neon Press * Raku Fundamentals; Moritz Lenz; Apress +* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton * Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner +* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook * DNS and BIND; Cricket Liu; O'Reilly -* Effective Java; Joshua Bloch; Addison-Wesley Professional +* Terraform Cookbook; Mikael Krief; Packt Publishing +* The Pragmatic Programmer; David Thomas; Addison-Wesley +* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly +* The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress +* Concurrency in Go; Katherine Cox-Buday; O'Reilly +* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press +* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly +* Ultimate Go Notebook; Bill Kennedy +* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* Data Science at the Command Line; Jeroen Janssens; O'Reilly +* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook +* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson +* Pro Puppet; James Turnbull, Jeffrey McCune; Apress +* Polished Ruby Programming; Jeremy Evans; Packt Publishing +* Systemprogrammierung in Go; Frank Müller; dpunkt +* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications +* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications +* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly +* Perl New Features; Joshua McAdams, brian d foy; Perl School +* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly +* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible +* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly +* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly ## Technical references I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order: -* The Linux Programming Interface; Michael Kerrisk; No Starch Press +* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly * Go: Design Patterns for Real-World Projects; Mat Ryer; Packt -* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly +* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley * Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley * Relayd and Httpd Mastery; Michael W Lucas * Implementing Service Level Objectives; Alex Hidalgo; O'Reilly -* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly -* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley +* The Linux Programming Interface; Michael Kerrisk; No Starch Press +* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly ## Self-development and soft-skills books In random order: -* Stop starting, start finishing; Arne Roock; Lean-Kanban University -* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press -* Eat That Frog; Brian Tracy -* The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook -* So Good They Can't Ignore You; Cal Newport; Business Plus -* Influence without Authority; A. Cohen, D. Bradford; Wiley -* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select -* Psycho-Cybernetics; Maxwell Maltz; Perigee Books +* Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook +* Ultralearning; Scott Young; Thorsons +* Deep Work; Cal Newport; Piatkus * Meditation for Mortals, Oliver Burkeman, Audiobook +* Ultralearning; Anna Laurent; Self-published via Amazon +* The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME) +* Slow Productivity; Cal Newport; Penguin Random House +* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business +* Stop starting, start finishing; Arne Roock; Lean-Kanban University +* Soft Skills; John Sommez; Manning Publications +* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing * Atomic Habits; James Clear; Random House Business -* Getting Things Done; David Allen -* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly * Solve for Happy; Mo Gawdat (RE-READ 1ST TIME) -* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne -* Digital Minimalism; Cal Newport; Portofolio Penguin -* 101 Essays that change the way you think; Brianna Wiest; Audiobook -* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook -* Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook * The Courage to Be Disliked; Ichiro Kishimi and Fumitake Koga; Audiobook -* Ultralearning; Anna Laurent; Self-published via Amazon +* Psycho-Cybernetics; Maxwell Maltz; Perigee Books * The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd -* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook +* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK * The Good Enough Job; Simone Stolzoff; Ebury Edge -* The Joy of Missing Out; Christina Crook; New Society Publishers -* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business +* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion +* Getting Things Done; David Allen +* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press +* Eat That Frog!; Brian Tracy; Hodder Paperbacks +* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook +* The Power of Now; Eckhard Tolle; Yellow Kite +* So Good They Can't Ignore You; Cal Newport; Business Plus * The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books -* The Bullet Journal Method; Ryder Carroll; Fourth Estate * The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook -* The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME) -* Slow Productivity; Cal Newport; Penguin Random House -* The Power of Now; Eckhard Tolle; Yellow Kite -* Eat That Frog!; Brian Tracy; Hodder Paperbacks -* Deep Work; Cal Newport; Piatkus -* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing -* Soft Skills; John Sommez; Manning Publications -* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion -* Ultralearning; Scott Young; Thorsons -* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK +* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select +* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook +* Eat That Frog; Brian Tracy +* Influence without Authority; A. Cohen, D. Bradford; Wiley +* The Bullet Journal Method; Ryder Carroll; Fourth Estate +* Digital Minimalism; Cal Newport; Portofolio Penguin +* The Joy of Missing Out; Christina Crook; New Society Publishers +* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne +* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly +* The Software Engineer's Guidebook: Navigating senior, tech lead, and staff engineer positions at tech companies and startups; Gergely Orosz; Audiobook +* 101 Essays that change the way you think; Brianna Wiest; Audiobook => ../notes/index.gmi Here are notes of mine for some of the books @@ -146,30 +146,30 @@ In random order: Some of these were in-person with exams; others were online learning lectures only. In random order: +* Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online +* Structure and Interpretation of Computer Programs; Harold Abelson and more...; +* The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online +* Developing IaC with Terraform (with Live Lessons); O'Reilly Online +* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) +* Protocol buffers; O'Reilly Online * Ultimate Go Programming; Bill Kennedy; O'Reilly Online +* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. +* Apache Tomcat Best Practises; 3-day on-site training * Scripting Vim; Damian Conway; O'Reilly Online * MySQL Deep Dive Workshop; 2-day on-site training -* Developing IaC with Terraform (with Live Lessons); O'Reilly Online -* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) -* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon -* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online * Functional programming lecture; Remote University of Hagen -* Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online +* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon * AWS Immersion Day; Amazon; 1-day interactive online training -* Apache Tomcat Best Practises; 3-day on-site training -* Structure and Interpretation of Computer Programs; Harold Abelson and more...; -* Protocol buffers; O'Reilly Online -* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. * Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training -* The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online +* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online ## Technical guides These are not whole books, but guides (smaller or larger) which I found very useful. in random order: -* How CPUs work at https://cpu.land -* Advanced Bash-Scripting Guide * Raku Guide at https://raku.guide +* Advanced Bash-Scripting Guide +* How CPUs work at https://cpu.land ## Podcasts @@ -177,58 +177,58 @@ These are not whole books, but guides (smaller or larger) which I found very use In random order: -* Hidden Brain -* Cup o' Go [Golang] -* Deep Questions with Cal Newport +* The Changelog Podcast(s) * Pratical AI -* The Pragmatic Engineer Podcast -* Wednesday Wisdom -* Backend Banter -* Modern Mentor -* Dev Interrupted -* The ProdCast (Google SRE Podcast) +* Maintainable * Fallthrough [Golang] +* The ProdCast (Google SRE Podcast) +* Wednesday Wisdom * Fork Around And Find Out -* The Changelog Podcast(s) +* Cup o' Go [Golang] +* The Pragmatic Engineer Podcast +* Hidden Brain +* Deep Questions with Cal Newport +* Dev Interrupted +* Backend Banter * BSD Now [BSD] -* Maintainable +* Modern Mentor ### Podcasts I liked I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests. * Java Pub House -* Ship It (predecessor of Fork Around And Find Out) -* Modern Mentor -* FLOSS weekly * Go Time (predecessor of fallthrough) +* FLOSS weekly +* Modern Mentor * CRE: Chaosradio Express [german] +* Ship It (predecessor of Fork Around And Find Out) ## Newsletters I like This is a mix of tech and non-tech newsletters I am subscribed to. In random order: -* The Valuable Dev -* Applied Go Weekly Newsletter -* Monospace Mentor -* VK Newsletter -* byteSizeGo -* Andreas Brandhorst Newsletter (Sci-Fi author) -* Register Spill +* The Pragmatic Engineer * Golang Weekly +* Monospace Mentor +* Applied Go Weekly Newsletter * Changelog News -* The Pragmatic Engineer +* VK Newsletter * The Imperfectionist * Ruby Weekly +* Andreas Brandhorst Newsletter (Sci-Fi author) +* Register Spill +* byteSizeGo +* The Valuable Dev ## Magazines I like(d) This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order: -* LWN (online only) * Linux Magazine -* Linux User * freeX (not published anymore) +* LWN (online only) +* Linux User # Formal education diff --git a/gemfeed/DRAFT-x-rag-observability-hackathon.gmi b/gemfeed/2025-12-24-x-rag-observability-hackathon.gmi index f2640480..83d75242 100644 --- a/gemfeed/DRAFT-x-rag-observability-hackathon.gmi +++ b/gemfeed/2025-12-24-x-rag-observability-hackathon.gmi @@ -1,5 +1,7 @@ # X-RAG Observability Hackathon +> Published at 2025-12-24T09:45:29+02:00 + This blog post describes my hackathon efforts adding observability to X-RAG, a distributed Retrieval-Augmented Generation (RAG) platform built by my brother Florian. I especially made time available over the weekend to join his 3-day hackathon (attending 2 days) with the goal of instrumenting his existing distributed system with observability. What started as "let's add some metrics" turned into a comprehensive implementation of the three pillars of observability: tracing, metrics, and logs. => https://github.com/florianbuetow/x-rag X-RAG source code on GitHub @@ -46,7 +48,7 @@ This blog post describes my hackathon efforts adding observability to X-RAG, a d ## What is X-RAG? -X-RAG is a distributed RAG (Retrieval-Augmented Generation) platform running on Kubernetes. The idea behind RAG is simple: instead of asking an LLM to answer questions from its training data alone, you first retrieve relevant documents from your own knowledge base, then feed those documents to the LLM as context. The LLM synthesises an answer grounded in your actual content—reducing hallucinations and enabling answers about private or recent information the model was never trained on. +X-RAG is the extendendible RAG (Retrieval-Augmented Generation) platform running on Kubernetes. The idea behind RAG is simple: instead of asking an LLM to answer questions from its training data alone, you first retrieve relevant documents from your own knowledge base, then feed those documents to the LLM as context. The LLM synthesises an answer grounded in your actual content—reducing hallucinations and enabling answers about private or recent information the model was never trained on. X-RAG handles the full pipeline: ingest documents, chunk them into searchable pieces, generate vector embeddings, store them in a vector database, and at query time, retrieve relevant chunks and pass them to an LLM for answer generation. The system supports both local LLMs (Florian runs his on a beefy desktop) and cloud APIs like OpenAI. I configured an OpenAI API key since my laptop's CPU and GPU aren't fast enough for decent local inference. @@ -68,15 +70,15 @@ The data layer includes Weaviate (vector database with hybrid search), Kafka (me ``` ┌─────────────────────────────────────────────────────────────────────────┐ -│ X-RAG Kubernetes Cluster │ +│ X-RAG Kubernetes Cluster │ ├─────────────────────────────────────────────────────────────────────────┤ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Search UI │ │Search Svc │ │Embed Service│ │ Indexer │ │ │ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ │ │ │ │ │ │ │ └────────────────┴────────────────┴────────────────┘ │ -│ │ │ -│ ▼ │ +│ │ │ +│ ▼ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │ │ Weaviate │ │ Kafka │ │ MinIO │ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ @@ -105,7 +107,7 @@ The `kindest/node` image contains everything needed: kubelet, containerd, CNI pl ``` ┌─────────────────────────────────────────────────────────────────────────┐ -│ Docker Host │ +│ Docker Host │ ├─────────────────────────────────────────────────────────────────────────┤ │ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ │ │ xrag-k8-control │ │ xrag-k8-worker │ │ xrag-k8-worker2 │ │ @@ -190,19 +192,19 @@ Getting all logs in one place was the foundation. I deployed Grafana Loki in the ``` ┌──────────────────────────────────────────────────────────────────────┐ -│ LOGS PIPELINE │ +│ LOGS PIPELINE │ ├──────────────────────────────────────────────────────────────────────┤ │ Applications write to stdout → containerd stores in /var/log/pods │ -│ │ │ -│ File tail │ -│ ▼ │ -│ Grafana Alloy (DaemonSet) │ -│ Discovers pods, extracts metadata │ -│ │ │ -│ HTTP POST /loki/api/v1/push │ -│ ▼ │ -│ Grafana Loki │ -│ Indexes labels, stores chunks │ +│ │ │ +│ File tail │ +│ ▼ │ +│ Grafana Alloy (DaemonSet) │ +│ Discovers pods, extracts metadata │ +│ │ │ +│ HTTP POST /loki/api/v1/push │ +│ ▼ │ +│ Grafana Loki │ +│ Indexes labels, stores chunks │ └──────────────────────────────────────────────────────────────────────┘ ``` diff --git a/gemfeed/DRAFT-x-rag-observability-hackathon.gmi.tpl b/gemfeed/2025-12-24-x-rag-observability-hackathon.gmi.tpl index 45af1017..6089bc2f 100644 --- a/gemfeed/DRAFT-x-rag-observability-hackathon.gmi.tpl +++ b/gemfeed/2025-12-24-x-rag-observability-hackathon.gmi.tpl @@ -1,5 +1,7 @@ # X-RAG Observability Hackathon +> Published at 2025-12-24T09:45:29+02:00 + This blog post describes my hackathon efforts adding observability to X-RAG, a distributed Retrieval-Augmented Generation (RAG) platform built by my brother Florian. I especially made time available over the weekend to join his 3-day hackathon (attending 2 days) with the goal of instrumenting his existing distributed system with observability. What started as "let's add some metrics" turned into a comprehensive implementation of the three pillars of observability: tracing, metrics, and logs. => https://github.com/florianbuetow/x-rag X-RAG source code on GitHub diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 5dcacff2..a4441738 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,12 +1,928 @@ <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> - <updated>2025-12-07T10:16:25+02:00</updated> + <updated>2025-12-24T09:45:29+02:00</updated> <title>foo.zone feed</title> <subtitle>To be in the .zone!</subtitle> <link href="gemini://foo.zone/gemfeed/atom.xml" rel="self" /> <link href="gemini://foo.zone/" /> <id>gemini://foo.zone/</id> <entry> + <title>X-RAG Observability Hackathon</title> + <link href="gemini://foo.zone/gemfeed/2025-12-24-x-rag-observability-hackathon.gmi" /> + <id>gemini://foo.zone/gemfeed/2025-12-24-x-rag-observability-hackathon.gmi</id> + <updated>2025-12-24T09:45:29+02:00</updated> + <author> + <name>Paul Buetow aka snonux</name> + <email>paul@dev.buetow.org</email> + </author> + <summary>This blog post describes my hackathon efforts adding observability to X-RAG, a distributed Retrieval-Augmented Generation (RAG) platform built by my brother Florian. I especially made time available over the weekend to join his 3-day hackathon (attending 2 days) with the goal of instrumenting his existing distributed system with observability. What started as 'let's add some metrics' turned into a comprehensive implementation of the three pillars of observability: tracing, metrics, and logs.</summary> + <content type="xhtml"> + <div xmlns="http://www.w3.org/1999/xhtml"> + <h1 style='display: inline' id='x-rag-observability-hackathon'>X-RAG Observability Hackathon</h1><br /> +<br /> +<span>This blog post describes my hackathon efforts adding observability to X-RAG, a distributed Retrieval-Augmented Generation (RAG) platform built by my brother Florian. I especially made time available over the weekend to join his 3-day hackathon (attending 2 days) with the goal of instrumenting his existing distributed system with observability. What started as "let's add some metrics" turned into a comprehensive implementation of the three pillars of observability: tracing, metrics, and logs.</span><br /> +<br /> +<a class='textlink' href='https://github.com/florianbuetow/x-rag'>X-RAG source code on GitHub</a><br /> +<br /> +<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> +<br /> +<ul> +<li><a href='#x-rag-observability-hackathon'>X-RAG Observability Hackathon</a></li> +<li>⇢ <a href='#what-is-x-rag'>What is X-RAG?</a></li> +<li>⇢ <a href='#running-kubernetes-locally-with-kind'>Running Kubernetes locally with Kind</a></li> +<li>⇢ <a href='#motivation'>Motivation</a></li> +<li>⇢ <a href='#the-observability-stack'>The observability stack</a></li> +<li>⇢ <a href='#grafana-alloy-the-unified-collector'>Grafana Alloy: the unified collector</a></li> +<li>⇢ <a href='#centralised-logging-with-loki'>Centralised logging with Loki</a></li> +<li>⇢ ⇢ <a href='#alloy-configuration-for-logs'>Alloy configuration for logs</a></li> +<li>⇢ ⇢ <a href='#querying-logs-with-logql'>Querying logs with LogQL</a></li> +<li>⇢ <a href='#metrics-with-prometheus'>Metrics with Prometheus</a></li> +<li>⇢ ⇢ <a href='#alloy-configuration-for-application-metrics'>Alloy configuration for application metrics</a></li> +<li>⇢ ⇢ <a href='#kubernetes-metrics-kubelet-cadvisor-and-kube-state-metrics'>Kubernetes metrics: kubelet, cAdvisor, and kube-state-metrics</a></li> +<li>⇢ ⇢ <a href='#infrastructure-metrics-kafka-redis-minio'>Infrastructure metrics: Kafka, Redis, MinIO</a></li> +<li>⇢ <a href='#distributed-tracing-with-tempo'>Distributed tracing with Tempo</a></li> +<li>⇢ ⇢ <a href='#understanding-traces-spans-and-the-trace-tree'>Understanding traces, spans, and the trace tree</a></li> +<li>⇢ ⇢ <a href='#how-trace-context-propagates'>How trace context propagates</a></li> +<li>⇢ ⇢ <a href='#implementation'>Implementation</a></li> +<li>⇢ ⇢ <a href='#alloy-configuration-for-traces'>Alloy configuration for traces</a></li> +<li>⇢ <a href='#async-ingestion-trace-walkthrough'>Async ingestion trace walkthrough</a></li> +<li>⇢ ⇢ <a href='#step-1-ingest-a-document'>Step 1: Ingest a document</a></li> +<li>⇢ ⇢ <a href='#step-2-find-the-ingestion-trace'>Step 2: Find the ingestion trace</a></li> +<li>⇢ ⇢ <a href='#step-3-fetch-the-complete-trace'>Step 3: Fetch the complete trace</a></li> +<li>⇢ ⇢ <a href='#step-4-analyse-the-async-trace'>Step 4: Analyse the async trace</a></li> +<li>⇢ ⇢ <a href='#viewing-traces-in-grafana'>Viewing traces in Grafana</a></li> +<li>⇢ <a href='#end-to-end-search-trace-walkthrough'>End-to-end search trace walkthrough</a></li> +<li>⇢ ⇢ <a href='#step-1-make-a-search-request'>Step 1: Make a search request</a></li> +<li>⇢ ⇢ <a href='#step-2-query-tempo-for-the-trace'>Step 2: Query Tempo for the trace</a></li> +<li>⇢ ⇢ <a href='#step-3-analyse-the-trace'>Step 3: Analyse the trace</a></li> +<li>⇢ ⇢ <a href='#step-4-search-traces-with-traceql'>Step 4: Search traces with TraceQL</a></li> +<li>⇢ ⇢ <a href='#viewing-the-search-trace-in-grafana'>Viewing the search trace in Grafana</a></li> +<li>⇢ <a href='#correlating-the-three-signals'>Correlating the three signals</a></li> +<li>⇢ <a href='#grafana-dashboards'>Grafana dashboards</a></li> +<li>⇢ <a href='#results-two-days-well-spent'>Results: two days well spent</a></li> +<li>⇢ <a href='#slis-slos-and-slas'>SLIs, SLOs and SLAs</a></li> +<li>⇢ <a href='#using-amp-for-ai-assisted-development'>Using Amp for AI-assisted development</a></li> +<li>⇢ <a href='#other-changes-along-the-way'>Other changes along the way</a></li> +<li>⇢ <a href='#lessons-learned'>Lessons learned</a></li> +</ul><br /> +<h2 style='display: inline' id='what-is-x-rag'>What is X-RAG?</h2><br /> +<br /> +<span>X-RAG is the extendendible RAG (Retrieval-Augmented Generation) platform running on Kubernetes. The idea behind RAG is simple: instead of asking an LLM to answer questions from its training data alone, you first retrieve relevant documents from your own knowledge base, then feed those documents to the LLM as context. The LLM synthesises an answer grounded in your actual content—reducing hallucinations and enabling answers about private or recent information the model was never trained on.</span><br /> +<br /> +<span>X-RAG handles the full pipeline: ingest documents, chunk them into searchable pieces, generate vector embeddings, store them in a vector database, and at query time, retrieve relevant chunks and pass them to an LLM for answer generation. The system supports both local LLMs (Florian runs his on a beefy desktop) and cloud APIs like OpenAI. I configured an OpenAI API key since my laptop's CPU and GPU aren't fast enough for decent local inference.</span><br /> +<br /> +<span>All services are implemented in Python. I'm more used to Ruby, Go, and Bash these days, but for this project it didn't matter—Python's OpenTelemetry integration is straightforward, I wasn't planning to write or rewrite tons of application code, and with GenAI assistance the language barrier was a non-issue. The OpenTelemetry concepts and patterns should translate to other languages too—the SDK APIs are intentionally similar across Python, Go, Java, and others.</span><br /> +<br /> +<span>X-RAG consists of several independently scalable microservices:</span><br /> +<br /> +<ul> +<li>Search UI: FastAPI web interface for queries</li> +<li>Ingestion API: Document upload endpoint</li> +<li>Embedding Service: gRPC service for vector embeddings</li> +<li>Indexer: Kafka consumer that processes documents</li> +<li>Search Service: gRPC service orchestrating the RAG pipeline</li> +</ul><br /> +<span>The Embedding Service deserves extra explanation because in the beginning I didn't really knew what it was. Text isn't directly searchable in a vector database—you need to convert it to numerical vectors (embeddings) that capture semantic meaning. The Embedding Service takes text chunks and calls an embedding model (OpenAI's <span class='inlinecode'>text-embedding-3-small</span> in my case, or a local model on Florian's setup) to produce these vectors. For the LLM search completion answer, I used <span class='inlinecode'>gpt-4o-mini</span>.</span><br /> +<br /> +<span>Similar concepts end up with similar vectors, so "What is machine learning?" and "Explain ML" produce vectors close together in the embedding space. At query time, your question gets embedded too, and the vector database finds chunks with nearby vectors—that's semantic search.</span><br /> +<br /> +<span>The data layer includes Weaviate (vector database with hybrid search), Kafka (message queue), MinIO (object storage), and Redis (cache). All of this runs in a Kind Kubernetes cluster for local development, with the same manifests deployable to production.</span><br /> +<br /> +<pre> +┌─────────────────────────────────────────────────────────────────────────┐ +│ X-RAG Kubernetes Cluster │ +├─────────────────────────────────────────────────────────────────────────┤ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Search UI │ │Search Svc │ │Embed Service│ │ Indexer │ │ +│ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ └──────┬──────┘ │ +│ │ │ │ │ │ +│ └────────────────┴────────────────┴────────────────┘ │ +│ │ │ +│ ▼ │ +│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ +│ │ Weaviate │ │ Kafka │ │ MinIO │ │ +│ └─────────────┘ └─────────────┘ └─────────────┘ │ +└─────────────────────────────────────────────────────────────────────────┘ +</pre> +<br /> +<h2 style='display: inline' id='running-kubernetes-locally-with-kind'>Running Kubernetes locally with Kind</h2><br /> +<br /> +<span>X-RAG runs on Kubernetes, but you don't need a cloud account to develop it. The project uses Kind (Kubernetes in Docker)—a tool originally created by the Kubernetes SIG for testing Kubernetes itself.</span><br /> +<br /> +<a class='textlink' href='https://kind.sigs.k8s.io/'>Kind - Kubernetes in Docker</a><br /> +<br /> +<span>Kind spins up a full Kubernetes cluster using Docker containers as nodes. The control plane (API server, etcd, scheduler, controller-manager) runs in one container, and worker nodes run in separate containers. Inside these "node containers," pods run just like they would on real servers—using containerd as the container runtime. It's containers all the way down.</span><br /> +<br /> +<span>Technically, each Kind node is a Docker container running a minimal Linux image with kubelet and containerd installed. When you deploy a pod, kubelet inside the node container instructs containerd to pull and run the container image. So you have Docker running node containers, and inside those, containerd running application containers. Network-wise, Kind sets up a Docker bridge network and uses CNI plugins (kindnet by default) for pod networking within the cluster.</span><br /> +<br /> +<pre> +$ docker ps --format "table {{.Names}}\t{{.Image}}" +NAMES IMAGE +xrag-k8-control-plane kindest/node:v1.32.0 +xrag-k8-worker kindest/node:v1.32.0 +xrag-k8-worker2 kindest/node:v1.32.0 +</pre> +<br /> +<span>The <span class='inlinecode'>kindest/node</span> image contains everything needed: kubelet, containerd, CNI plugins, and pre-pulled pause containers. Port mappings in the Kind config expose services to the host—that's how http://localhost:8080 reaches the search-ui running inside a pod, inside a worker container, inside Docker.</span><br /> +<br /> +<pre> +┌─────────────────────────────────────────────────────────────────────────┐ +│ Docker Host │ +├─────────────────────────────────────────────────────────────────────────┤ +│ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ +│ │ xrag-k8-control │ │ xrag-k8-worker │ │ xrag-k8-worker2 │ │ +│ │ -plane (container)│ │ (container) │ │ (container) │ │ +│ │ │ │ │ │ │ │ +│ │ K8s API server │ │ Pods: │ │ Pods: │ │ +│ │ etcd, scheduler │ │ • search-ui │ │ • weaviate │ │ +│ │ │ │ • search-service │ │ • kafka │ │ +│ │ │ │ • embedding-svc │ │ • prometheus │ │ +│ │ │ │ • indexer │ │ • grafana │ │ +│ └───────────────────┘ └───────────────────┘ └───────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────┘ +</pre> +<br /> +<span>Why Kind? It gives you a real Kubernetes environment—the same manifests deploy to production clouds unchanged. No minikube quirks, no Docker Compose translation layer. Just Kubernetes. I already have a k3s cluster running at home, but Kind made collaboration easier—everyone working on X-RAG gets the exact same setup by cloning the repo and running <span class='inlinecode'>make cluster-start</span>.</span><br /> +<br /> +<span>Florian developed X-RAG on macOS, but it worked seamlessly on my Linux laptop. The only difference was Docker's resource allocation: on macOS you configure limits in Docker Desktop, on Linux it uses host resources directly. That's because under macOS the Linux Docker containers run on an emulation layer as macOS is not Linux.</span><br /> +<br /> +<span>My hardware: a ThinkPad X1 Carbon Gen 9 with an 11th Gen Intel Core i7-1185G7 (4 cores, 8 threads at 3.00GHz) and 32GB RAM (running Fedora Linux). During the hackathon, memory usage peaked around 15GB—comfortable headroom. CPU was the bottleneck; with ~38 pods running across all namespaces (rag-system, monitoring, kube-system, etc.), plus Discord for the remote video call and Tidal streaming hi-res music, things got tight. When rebuilding Docker images or restarting the cluster, Discord video and audio would stutter—my fellow hackers probably wondered why I kept freezing mid-sentence. A beefier CPU would have meant less waiting and smoother calls, but it was manageable.</span><br /> +<br /> +<h2 style='display: inline' id='motivation'>Motivation</h2><br /> |