feat: add Argo Rollouts controller and tracing-demo canary rollout demo

14 files changed, 1940 insertions, 0 deletions

diff --git a/f3s/ARGO-ROLLOUTS-SUMMARY.md b/f3s/ARGO-ROLLOUTS-SUMMARY.md
new file mode 100644
index 0000000..2c0372e
--- /dev/null
+++ b/f3s/ARGO-ROLLOUTS-SUMMARY.md
@@ -0,0 +1,248 @@
+# Argo Rollouts Implementation Summary
+
+## What Was Created
+
+### 1. Argo Rollouts Controller Installation
+**Location**: `/home/paul/git/conf/f3s/argo-rollouts/`
+
+Files:
+- `Justfile` - Installation automation
+- `values.yaml` - Helm configuration
+- `README.md` - Installation guide
+
+Deployment:
+```bash
+cd /home/paul/git/conf/f3s/argo-rollouts
+just install
+```
+
+Also registered in ArgoCD: `/home/paul/git/conf/f3s/argocd-apps/cicd/argo-rollouts.yaml`
+
+### 2. Frontend Rollout Manifest
+**Location**: `/home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml`
+
+**Replaces**: `frontend-deployment.yaml` (kept for reference)
+
+**Strategy**: Canary with 2-minute observation window
+```
+Step 1: 50% traffic to new version
+Step 2: Pause 2 minutes (observation period)
+Step 3: 100% traffic to new version (auto-promote)
+```
+
+**Why Frontend?**
+- Has 2 replicas (good for canary demo)
+- User-facing (can observe behavior easily)
+- Generates traces (can monitor impact)
+- Non-critical for cluster health
+
+### 3. Demo Documentation
+
+**`/home/paul/git/conf/f3s/tracing-demo/ROLLOUTS-DEMO.md`**
+- Comprehensive walkthrough
+- Real-time monitoring commands
+- Troubleshooting guide
+- Advanced patterns
+
+**`/home/paul/git/conf/f3s/ROLLOUTS-SETUP.md`**
+- Quick setup instructions
+- 5 demo scenarios (basic, manual, abort, prometheus, gitops)
+- Expected output and timings
+- Monitoring dashboard examples
+
+**`/home/paul/git/conf/f3s/tracing-demo/rollout-demo.sh`**
+- Automated demo starter script
+- Checks prerequisites
+- Provides instructions
+
+### 4. Enhanced Justfile Commands
+**Location**: `/home/paul/git/conf/f3s/tracing-demo/Justfile`
+
+New commands:
+```bash
+just rollout-watch      # Watch progress in real-time
+just rollout-status     # Check current status
+just rollout-info       # Detailed information
+just rollout-promote    # Skip waiting, promote to 100%
+just rollout-abort      # Abort current rollout
+just rollout-history    # View past rollouts
+just rollout-demo       # Start demo script
+```
+
+### 5. Updated ArgoCD Application
+**Location**: `/home/paul/git/conf/f3s/argocd-apps/services/tracing-demo.yaml`
+
+Added sync option: `RespectIgnoreDifferences=true` to gracefully handle migration from Deployment to Rollout.
+
+## Architecture
+
+```
+┌─────────────────────────────────────────┐
+│         Kubernetes Cluster              │
+├─────────────────────────────────────────┤
+│                                         │
+│  ┌──────────────────┐                  │
+│  │  ArgoCD (cicd)   │                  │
+│  └────────┬─────────┘                  │
+│           │                             │
+│           └──→ Git Repository           │
+│               (conf.git)                │
+│                                         │
+│  ┌──────────────────────────────────┐  │
+│  │  Argo Rollouts Controller (cicd) │  │
+│  │  - Manages Rollout resources     │  │
+│  │  - Orchestrates canary           │  │
+│  │  - Monitors replica sets         │  │
+│  └──────────────────────────────────┘  │
+│                   ▲                     │
+│                   │ watches             │
+│                   │                     │
+│  ┌────────────────────────────────────┐ │
+│  │  tracing-demo-frontend Rollout     │ │
+│  │  ┌──────────────┐  ┌──────────────┐│ │
+│  │  │ Stable RS    │  │ Canary RS    ││ │
+│  │  │ 2 replicas   │  │ 1-2 replicas ││ │
+│  │  └──────────────┘  └──────────────┘│ │
+│  │                                     │ │
+│  │  Endpoints: frontend-service        │ │
+│  │  - Selects both RS (proportional)   │ │
+│  │  - Routes traffic to 50%/100%       │ │
+│  └────────────────────────────────────┘ │
+│                                         │
+│  ┌──────────────────┐                  │
+│  │ Middleware       │  ┌──────────────┐│
+│  │ Backend          │  │ Deployment   ││
+│  │ (unchanged)      │  │ (unchanged)  ││
+│  └──────────────────┘  └──────────────┘│
+│                                         │
+└─────────────────────────────────────────┘
+        Monitoring (Prometheus/Grafana)
+```
+
+## Key Differences: Deployment vs Rollout
+
+| Aspect | Deployment | Rollout |
+|--------|------------|---------|
+| **Update Strategy** | RollingUpdate (all or nothing) | Canary, Blue-Green, A/B |
+| **Traffic Split** | No built-in support | Native pod-level splitting |
+| **Pause/Resume** | No | Yes (at canary steps) |
+| **Automatic Rollback** | No (manual `rollout undo`) | Yes (if health checks fail) |
+| **Visibility** | kubectl rollout status | kubectl argo rollouts get --watch |
+| **Observability** | Basic pod counts | Detailed step information |
+
+## How It Works
+
+### Normal Deployment (Traditional)
+```
+kubectl apply → All pods immediately scale up/down
+Old pods: 2 → 0
+New pods: 0 → 2
+Users affected: ~5 seconds of traffic loss risk
+```
+
+### Canary Rollout (New)
+```
+Git commit → ArgoCD detects → Argo Rollouts orchestrates
+
+Step 1 (50% traffic):
+  Stable: 2 pods → 1 pod  (old version)
+  Canary: 0 pods → 1 pod  (new version)
+  Users see: 50% old, 50% new for 0-2 minutes
+
+Step 2 (Pause):
+  Stable: 1 pod (old)
+  Canary: 1 pod (new)
+  Observe metrics, logs, error rates for 2 minutes
+
+Step 3 (100% traffic):
+  Stable: 1 → 0 pods (old version terminated)
+  Canary: 1 → 2 pods (new version scales up)
+  Users see: 100% new version
+  
+  Complete: Canary promoted to stable
+```
+
+## Demo Quick Start
+
+### 1. Install Everything
+```bash
+cd /home/paul/git/conf/f3s
+# Sync with ArgoCD (auto or manual)
+argocd app sync argo-rollouts
+argocd app sync tracing-demo
+```
+
+### 2. Verify Setup
+```bash
+cd /home/paul/git/conf/f3s/tracing-demo
+just rollout-status
+# Should show: Rollout is healthy
+```
+
+### 3. Run Demo
+```bash
+# Terminal 1: Watch rollout
+just rollout-watch
+
+# Terminal 2: Trigger rollout (modify git or patch)
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]'
+```
+
+### 4. Observe
+- See canary step progress in Terminal 1
+- Optional: `just load-test` to generate traffic during rollout
+- After ~4 minutes: Rollout complete, 100% traffic to new version
+
+## Files Summary
+
+| Path | Purpose |
+|------|---------|
+| `argo-rollouts/Justfile` | Install/upgrade/check Argo Rollouts |
+| `argo-rollouts/values.yaml` | Helm configuration for controller |
+| `argo-rollouts/README.md` | Installation and basic usage |
+| `tracing-demo/helm-chart/templates/frontend-rollout.yaml` | Canary rollout definition |
+| `tracing-demo/Justfile` | Added `just rollout-*` commands |
+| `tracing-demo/ROLLOUTS-DEMO.md` | Detailed walkthrough |
+| `tracing-demo/rollout-demo.sh` | Demo starter script |
+| `argocd-apps/cicd/argo-rollouts.yaml` | ArgoCD Application for controller |
+| `argocd-apps/services/tracing-demo.yaml` | Updated to work with Rollout |
+| `ROLLOUTS-SETUP.md` | Complete setup guide with scenarios |
+| `ARGO-ROLLOUTS-SUMMARY.md` | This file |
+
+## Next Steps
+
+1. **Install controller**: `cd argo-rollouts && just install`
+2. **Wait for ArgoCD sync** or manually sync `argo-rollouts` and `tracing-demo` apps
+3. **Verify**: `just rollout-status` shows healthy
+4. **Run demo**: `just rollout-watch` + trigger in another terminal
+5. **Explore**: Try abort, promote, or different canary durations
+
+## Important Notes
+
+- **No service mesh required**: Uses native Kubernetes service-based routing
+- **Traffic splitting**: Proportional to pod counts (1 old, 1 new = 50/50)
+- **Auto-promotion**: After 2 minutes, canary automatically promotes to 100%
+- **Graceful**: ArgoCD correctly handles transition from Deployment → Rollout
+- **Reversible**: Can abort and keep old version running
+
+## Limitations & Future Work
+
+**Current (Basic Canary)**:
+- Simple replica-based traffic splitting
+- No header-based routing
+- No advanced health checks
+
+**To Add** (Optional):
+- **Istio integration**: For precise % traffic splitting, header-based routing
+- **Flagger**: Automated canary analysis with Prometheus thresholds
+- **Linkerd**: For distributed tracing and observability
+- **Longer observation**: Change `pause: duration: 2m` to `5m` or `10m`
+
+## Questions?
+
+See:
+- `/home/paul/git/conf/f3s/ROLLOUTS-SETUP.md` - Complete setup & scenarios
+- `/home/paul/git/conf/f3s/tracing-demo/ROLLOUTS-DEMO.md` - Detailed walkthrough
+- `/home/paul/git/conf/f3s/argo-rollouts/README.md` - Controller-specific info
diff --git a/f3s/README-ROLLOUTS.md b/f3s/README-ROLLOUTS.md
new file mode 100644
index 0000000..60ec9b6
--- /dev/null
+++ b/f3s/README-ROLLOUTS.md
@@ -0,0 +1,229 @@
+# Argo Rollouts - Quick Reference
+
+Progressive delivery (canary deployments) for the f3s cluster.
+
+## TL;DR - Get Started in 5 Minutes
+
+```bash
+# 1. Install controller
+cd /home/paul/git/conf/f3s/argo-rollouts
+just install
+
+# 2. Wait for ArgoCD sync (or force)
+argocd app sync argo-rollouts
+argocd app sync tracing-demo
+
+# 3. Verify setup
+cd /home/paul/git/conf/f3s/tracing-demo
+just rollout-status
+
+# 4. Run a demo (Terminal 1)
+just rollout-watch
+
+# 5. Trigger in another terminal (Terminal 2)
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]'
+
+# 6. Watch progress in Terminal 1 (~4 minutes total)
+```
+
+Expected flow:
+- 0-2 min: **50% traffic** to new version (canary phase 1)
+- 2-4 min: **Wait** for confirmation (canary phase 2)
+- 4+ min: **100% traffic** to new version (auto-promoted)
+
+## Files Created
+
+### Setup & Installation
+- `argo-rollouts/Justfile` - Install/manage controller
+- `argo-rollouts/values.yaml` - Helm config
+- `argocd-apps/cicd/argo-rollouts.yaml` - ArgoCD app
+
+### Demo App Configuration
+- `tracing-demo/helm-chart/templates/frontend-rollout.yaml` - Canary definition
+- `tracing-demo/Justfile` - New `just rollout-*` commands
+- `tracing-demo/rollout-demo.sh` - Demo automation script
+
+### Documentation
+- `ARGO-ROLLOUTS-SUMMARY.md` - **START HERE** - Full overview
+- `ROLLOUTS-SETUP.md` - **DETAILED GUIDE** - 5 demo scenarios
+- `ROLLOUTS-CHECKLIST.md` - **DEPLOYMENT CHECKLIST** - Step-by-step
+- `tracing-demo/ROLLOUTS-DEMO.md` - Technical walkthrough
+- `README-ROLLOUTS.md` - This file
+
+## Why Canary Deployments?
+
+**Old way (Deployment)**:
+- 2 old pods → removed
+- 2 new pods → created
+- ~5 seconds of potential traffic loss
+- No way to validate before 100% rollout
+
+**New way (Rollout with Canary)**:
+- 2 old pods → 1 old + 1 new (50/50 traffic)
+- Observe for 2 minutes
+- If healthy → automatically promote to 2 new pods
+- If unhealthy → abort, revert to 2 old pods
+- Zero downtime, validated before full rollout
+
+## Common Commands
+
+```bash
+cd /home/paul/git/conf/f3s/tracing-demo
+
+# Watch rollout progress (real-time)
+just rollout-watch
+
+# Check current status
+just rollout-status
+
+# Detailed info
+just rollout-info
+
+# Skip waiting, promote now
+just rollout-promote
+
+# Abort and rollback
+just rollout-abort
+
+# View history
+just rollout-history
+
+# Generate load during rollout
+just load-test
+```
+
+## What Happens During Canary
+
+### Step 1: 50% Traffic (0-2 minutes)
+```
+Frontend Service
+├── Stable ReplicaSet (old version): 1 pod → receives 50% traffic
+└── Canary ReplicaSet (new version): 1 pod → receives 50% traffic
+```
+
+Monitor during this phase:
+- Error rates
+- Response latency
+- Logs and traces
+- Prometheus metrics
+
+### Step 2: Pause (2 minutes)
+```
+Service pauses traffic shift, waiting for:
+- Manual promotion via: kubectl argo rollouts promote ...
+- Auto-promotion after 2 minutes
+- Or abort: kubectl argo rollouts abort ...
+```
+
+### Step 3: 100% Traffic (4+ minutes)
+```
+Frontend Service
+├── Stable ReplicaSet (new version): 2 pods → receives 100% traffic
+└── Canary ReplicaSet (old version): 0 pods → terminated
+```
+
+## Architecture
+
+```
+Git Commit (new image)
+    ↓
+Git Server (conf.git)
+    ↓
+ArgoCD detects change
+    ↓
+Updates Rollout resource
+    ↓
+Argo Rollouts Controller
+    ↓
+    ├─→ Scales Canary ReplicaSet (1 new pod)
+    ├─→ Frontend Service routes 50/50 traffic
+    ├─→ Monitors health/metrics for 2 minutes
+    └─→ Auto-promotes or waits for manual action
+        ├─→ If healthy: Scale to 2 new, remove old
+        └─→ If abort: Remove canary, keep old
+```
+
+## Demo Scenarios
+
+See `ROLLOUTS-SETUP.md` for complete walkthrough of:
+
+1. **Basic Canary** - Watch 50% → 100% progression
+2. **Manual Promotion** - Skip waiting with `just rollout-promote`
+3. **Abort/Rollback** - Fail canary and revert
+4. **Prometheus Monitoring** - Track metrics during rollout
+5. **GitOps Flow** - Commit code, watch auto-rollout
+
+## Monitoring
+
+### Command-line
+```bash
+# Real-time watch
+kubectl argo rollouts get rollout tracing-demo-frontend -n services --watch
+
+# Check metrics
+kubectl top pods -n services -l app=tracing-demo-frontend
+```
+
+### Grafana
+https://grafana.f3s.buetow.org
+
+1. Explore → Tempo
+2. Query: `{ resource.service.name = "frontend" }`
+3. See traces from old and new versions
+
+### Prometheus
+```bash
+# Port-forward
+kubectl port-forward -n monitoring svc/prometheus 9090:9090
+# Open http://localhost:9090
+
+# Query pod status
+kube_pod_status_phase{namespace="services", pod=~".*frontend.*"}
+```
+
+## Troubleshooting
+
+**Controller not running?**
+```bash
+kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts
+kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts
+```
+
+**Rollout stuck?**
+```bash
+kubectl describe rollout tracing-demo-frontend -n services
+kubectl get pods -n services -l app=tracing-demo-frontend
+```
+
+**Need plugin?**
+```bash
+curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
+sudo install -m 755 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
+```
+
+## Next Steps
+
+1. Complete setup using `ROLLOUTS-CHECKLIST.md`
+2. Run demo scenarios from `ROLLOUTS-SETUP.md`
+3. Share with team
+4. Optional: Add Istio for advanced traffic routing
+5. Optional: Deploy Flagger for automated analysis
+6. Migrate other services to Rollout
+
+## Key Resources
+
+| File | Purpose |
+|------|---------|
+| `ARGO-ROLLOUTS-SUMMARY.md` | Architecture & what was created |
+| `ROLLOUTS-SETUP.md` | Complete setup & 5 demo scenarios |
+| `ROLLOUTS-CHECKLIST.md` | Step-by-step deployment |
+| `tracing-demo/ROLLOUTS-DEMO.md` | Technical details & troubleshooting |
+| `argo-rollouts/README.md` | Controller installation guide |
+
+## Support
+
+- Argo Rollouts Docs: https://argoproj.github.io/argo-rollouts/
+- Canary Strategy: https://argoproj.github.io/argo-rollouts/features/canary/
+- Kubectl Plugin: https://argoproj.github.io/argo-rollouts/getting-started/#using-kubectl-with-argo-rollouts
diff --git a/f3s/ROLLOUTS-CHECKLIST.md b/f3s/ROLLOUTS-CHECKLIST.md
new file mode 100644
index 0000000..b32f1ac
--- /dev/null
+++ b/f3s/ROLLOUTS-CHECKLIST.md
@@ -0,0 +1,189 @@
+# Argo Rollouts Deployment Checklist
+
+## Pre-Deployment Setup
+
+- [ ] Read `ARGO-ROLLOUTS-SUMMARY.md` to understand what was created
+- [ ] Ensure kubectl access to f3s cluster
+- [ ] Ensure ArgoCD is running and accessible
+- [ ] Git repository (conf.git) synced to git-server
+
+## Installation
+
+- [ ] Navigate to `/home/paul/git/conf/f3s/argo-rollouts`
+- [ ] Run `just install` to deploy controller
+- [ ] Verify controller running: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts`
+- [ ] Verify CRD installed: `kubectl get crd | grep rollout`
+
+## Optional: Install kubectl Plugin
+
+- [ ] Download kubectl-argo-rollouts:
+  ```bash
+  curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
+  chmod +x kubectl-argo-rollouts-linux-amd64
+  sudo install -m 755 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
+  ```
+- [ ] Verify: `kubectl argo rollouts version`
+
+## ArgoCD Syncing
+
+- [ ] Create/push `argocd-apps/cicd/argo-rollouts.yaml` to git
+- [ ] Create/push `argocd-apps/services/tracing-demo.yaml` updates to git
+- [ ] Force ArgoCD sync (wait 3 min or manual):
+  ```bash
+  argocd app sync argo-rollouts
+  argocd app sync tracing-demo
+  ```
+- [ ] Verify tracing-demo application status: `argocd app get tracing-demo`
+
+## Rollout Verification
+
+- [ ] Check frontend rollout deployed: `kubectl get rollout tracing-demo-frontend -n services`
+- [ ] Verify status: `kubectl describe rollout tracing-demo-frontend -n services`
+- [ ] Expected: `Status: Healthy` with `2/2 replicas` in stable state
+- [ ] Check pods running: `kubectl get pods -n services -l app=tracing-demo-frontend`
+
+## Basic Demo (First Time)
+
+### Terminal 1: Watch Rollout
+```bash
+cd /home/paul/git/conf/f3s/tracing-demo
+just rollout-watch
+```
+- [ ] Command running and connected
+
+### Terminal 2: Generate Load (Optional)
+```bash
+cd /home/paul/git/conf/f3s/tracing-demo
+just load-test &
+```
+- [ ] Requests being sent to frontend
+
+### Terminal 3: Trigger Rollout
+Choose one method:
+
+**Method A: Kubectl Patch (Fastest)**
+```bash
+kubectl patch rollout tracing-demo-frontend -n services \
+  --type='json' \
+  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]'
+```
+- [ ] Executed successfully
+
+**Method B: Git + ArgoCD (Most GitOps)**
+```bash
+cd /home/paul/git/conf/f3s
+# Edit tracing-demo/helm-chart/templates/frontend-rollout.yaml (change image tag)
+git add -A
+git commit -m "chore: update frontend image for demo"
+git remote add r0 ssh://git@r0:30022/repos/conf.git 2>/dev/null || true
+git push r0 master
+kubectl annotate application tracing-demo -n cicd argocd.argoproj.io/refresh=normal --overwrite
+```
+- [ ] Git push successful
+- [ ] ArgoCD syncing (check web UI or CLI)
+
+## Demo Observation
+
+- [ ] Terminal 1 shows: "Progressing" → "canary step 1/3"
+- [ ] After ~30 sec: New canary pod appears
+- [ ] After ~2 min: "canary step 2/3" (pause)
+- [ ] After ~4 min: "canary step 3/3" (100% traffic)
+- [ ] After ~4:20 min: Status shows "Healthy"
+- [ ] Old pods terminated, 2 new pods in stable state
+
+## Monitoring (Optional)
+
+- [ ] Check logs: `just logs-frontend`
+- [ ] Check Grafana Tempo for traces: https://grafana.f3s.buetow.org
+  - [ ] Navigate to Explore → Tempo
+  - [ ] Query: `{ resource.service.name = "frontend" }`
+  - [ ] See traces from old and new versions
+- [ ] Check Prometheus metrics: Port-forward and query
+
+## Advanced Scenarios
+
+### Scenario 1: Manual Promotion
+- [ ] Trigger rollout (step above)
+- [ ] After step 1 (30 sec), run:
+  ```bash
+  just rollout-promote
+  ```
+- [ ] Watch rollout skip step 2, immediately promote to 100%
+- [ ] Verify: `just rollout-status` shows "Healthy"
+
+### Scenario 2: Abort/Rollback
+- [ ] Trigger rollout
+- [ ] While progressing, run:
+  ```bash
+  just rollout-abort
+  ```
+- [ ] Watch canary pods terminate
+- [ ] Old version continues running
+- [ ] Verify: `just rollout-status` shows "Aborted"
+
+### Scenario 3: Check History
+- [ ] After any rollout:
+  ```bash
+  just rollout-history
+  ```
+- [ ] See previous revisions and their status
+
+## Integration with CI/CD
+
+- [ ] Image builds automatically on git push (or configured pipeline)
+- [ ] New image pushed to registry: `registry.lan.buetow.org:30001/tracing-demo-frontend:NEWTAG`
+- [ ] Git updated with new image tag
+- [ ] ArgoCD detects change
+- [ ] Rollout automatically triggered
+- [ ] Canary strategy executes
+
+## Post-Deployment
+
+- [ ] Share documentation:
+  - [ ] `ROLLOUTS-SETUP.md` - Complete setup guide
+  - [ ] `tracing-demo/ROLLOUTS-DEMO.md` - Detailed walkthrough
+  - [ ] `ARGO-ROLLOUTS-SUMMARY.md` - Architecture overview
+- [ ] Add team to `kubectl argo rollouts` usage
+- [ ] Consider next steps:
+  - [ ] Deploy Istio for advanced traffic management
+  - [ ] Add Flagger for automated analysis
+  - [ ] Extend to other services (middleware, backend)
+  - [ ] Create monitoring dashboards
+
+## Troubleshooting Checklist
+
+### Controller not running
+- [ ] Check pod: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts`
+- [ ] Check logs: `kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts`
+- [ ] Check CRD: `kubectl get crd | grep rollout`
+
+### Rollout not deploying
+- [ ] Check ArgoCD sync: `argocd app get tracing-demo`
+- [ ] Check git changes pushed: `git log --oneline | head -5`
+- [ ] Force sync: `argocd app sync tracing-demo --prune`
+
+### Canary pods not starting
+- [ ] Check pod status: `kubectl describe pod -n services <pod-name>`
+- [ ] Check logs: `kubectl logs -n services <pod-name>`
+- [ ] Check resource limits: `kubectl top pods -n services`
+- [ ] Check image: `kubectl get pods -n services -o jsonpath='{.items[*].spec.containers[0].image}'`
+
+### Rollout stuck in Progressing
+- [ ] Check health probes: `kubectl get rollout tracing-demo-frontend -n services -o yaml | grep -A 10 health`
+- [ ] Check replica status: `kubectl get rs -n services -l app=tracing-demo-frontend -o wide`
+- [ ] Check controller logs: `kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts --tail=50`
+
+## Cleanup (If Needed)
+
+- [ ] Stop rollout: `kubectl argo rollouts abort tracing-demo-frontend -n services`
+- [ ] Rollback to previous: `kubectl rollout undo deployment/tracing-demo-frontend -n services` (if needed)
+- [ ] Uninstall Argo Rollouts: `cd argo-rollouts && just uninstall`
+
+---
+
+**Setup complete when:**
+- ✅ Argo Rollouts controller running in `cicd` namespace
+- ✅ Frontend rollout deployed in `services` namespace
+- ✅ ArgoCD recognizes rollout resource
+- ✅ One demo run successful (git trigger or kubectl patch)
+- ✅ Team can watch and manage rollouts
diff --git a/f3s/ROLLOUTS-FILE-TREE.txt b/f3s/ROLLOUTS-FILE-TREE.txt
new file mode 100644
index 0000000..6c85754
--- /dev/null
+++ b/f3s/ROLLOUTS-FILE-TREE.txt
@@ -0,0 +1,183 @@
+/home/paul/git/conf/f3s/
+├── README-ROLLOUTS.md                          ← ENTRY POINT (quick reference)
+├── ARGO-ROLLOUTS-SUMMARY.md                    ← Full architecture & overview
+├── ROLLOUTS-SETUP.md                           ← Detailed setup + 5 scenarios
+├── ROLLOUTS-CHECKLIST.md                       ← Step-by-step deployment
+├── ROLLOUTS-FILE-TREE.txt                      ← This file
+│
+├── argo-rollouts/                              ← NEW: Argo Rollouts Controller
+│   ├── Justfile                                ← Install/upgrade/uninstall
+│   ├── values.yaml                             ← Helm configuration
+│   └── README.md                               ← Controller-specific guide
+│
+├── argocd-apps/
+│   ├── cicd/
+│   │   ├── git-server.yaml
+│   │   └── argo-rollouts.yaml                  ← NEW: Controller app
+│   │
+│   └── services/
+│       ├── tracing-demo.yaml                   ← UPDATED: Deployment → Rollout
+│       └── ... (other apps)
+│
+├── tracing-demo/
+│   ├── README.md
+│   ├── Justfile                                ← UPDATED: Added rollout commands
+│   ├── ROLLOUTS-DEMO.md                        ← NEW: Technical walkthrough
+│   ├── rollout-demo.sh                         ← NEW: Demo automation
+│   │
+│   └── helm-chart/
+│       ├── Chart.yaml
+│       └── templates/
+│           ├── frontend-rollout.yaml           ← NEW: Canary rollout definition
+│           ├── frontend-deployment.yaml        ← KEPT: For reference
+│           ├── middleware-deployment.yaml      ← (unchanged)
+│           ├── backend-deployment.yaml         ← (unchanged)
+│           ├── frontend-service.yaml
+│           ├── middleware-service.yaml
+│           ├── backend-service.yaml
+│           └── ingress.yaml
+│
+└── ... (other apps unchanged)
+
+
+═══════════════════════════════════════════════════════════════════════════
+
+INSTALLATION SUMMARY
+═══════════════════════════════════════════════════════════════════════════
+
+Step 1: Install Controller
+  cd /home/paul/git/conf/f3s/argo-rollouts
+  just install
+
+Step 2: Verify ArgoCD
+  argocd app sync argo-rollouts
+  argocd app sync tracing-demo
+
+Step 3: Watch Demo
+  cd /home/paul/git/conf/f3s/tracing-demo
+  just rollout-watch
+
+Step 4: Trigger Rollout (in another terminal)
+  kubectl patch rollout tracing-demo-frontend -n services \
+    --type='json' \
+    -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]'
+
+═══════════════════════════════════════════════════════════════════════════
+
+DOCUMENTATION ROADMAP
+═══════════════════════════════════════════════════════════════════════════
+
+NEW TO ARGO ROLLOUTS?
+  1. Read: README-ROLLOUTS.md (3 min)
+  2. Read: ARGO-ROLLOUTS-SUMMARY.md (10 min)
+  3. Follow: ROLLOUTS-CHECKLIST.md (step-by-step)
+
+WANT DETAILED GUIDE?
+  → ROLLOUTS-SETUP.md
+     - Complete setup instructions
+     - 5 demo scenarios with expected output
+     - Monitoring dashboards
+     - Advanced patterns
+
+DOING THE DEPLOYMENT?
+  → ROLLOUTS-CHECKLIST.md
+     - Pre-deployment checks
+     - Installation steps
+     - Verification
+     - Troubleshooting
+
+TROUBLESHOOTING?
+  → ROLLOUTS-SETUP.md → Troubleshooting section
+  → argo-rollouts/README.md
+  → tracing-demo/ROLLOUTS-DEMO.md
+
+═══════════════════════════════════════════════════════════════════════════
+
+KEY FILES EXPLAINED
+═══════════════════════════════════════════════════════════════════════════
+
+argo-rollouts/Justfile
+  - Automates installation of Argo Rollouts controller
+  - Commands: install, upgrade, uninstall, status, logs
+  - Deploys to: cicd namespace
+
+argo-rollouts/values.yaml
+  - Helm chart configuration for Argo Rollouts
+  - Sets resource limits, metrics, replicas
+
+argocd-apps/cicd/argo-rollouts.yaml
+  - ArgoCD Application resource
+  - Manages controller installation via GitOps
+  - Auto-syncs when argo-rollouts/ changes in git
+
+tracing-demo/helm-chart/templates/frontend-rollout.yaml
+  - Replaces frontend-deployment.yaml
+  - Defines canary strategy:
+    * Step 1: 50% traffic
+    * Step 2: 2-minute pause
+    * Step 3: 100% promotion
+  - Keeps same pods, volumes, env vars as Deployment
+
+tracing-demo/Justfile (updated)
+  - New commands for rollout management
+  - just rollout-watch
+  - just rollout-status
+  - just rollout-promote
+  - just rollout-abort
+  - just rollout-history
+
+tracing-demo/rollout-demo.sh
+  - Automation script for demo
+  - Checks prerequisites
+  - Guides through demo workflow
+  - Can be extended for CI/CD
+
+═══════════════════════════════════════════════════════════════════════════
+
+WHAT CHANGED IN EXISTING FILES
+═══════════════════════════════════════════════════════════════════════════
+
+tracing-demo/Justfile
+  [+] 8 new rollout commands
+  [-] No breaking changes to existing commands
+
+tracing-demo/helm-chart/templates/frontend-deployment.yaml
+  [~] Still exists (for reference, not deployed)
+  [→] Replaced by frontend-rollout.yaml in deployment
+
+argocd-apps/services/tracing-demo.yaml
+  [+] RespectIgnoreDifferences=true sync option
+  [-] No other changes (points to same Helm chart)
+
+═══════════════════════════════════════════════════════════════════════════
+
+WHAT DID NOT CHANGE
+═══════════════════════════════════════════════════════════════════════════
+
+✓ Middleware & Backend services remain Deployments
+✓ All service definitions (frontend, middleware, backend services)
+✓ Ingress configuration
+✓ All other apps (audiobookshelf, miniflux, etc.)
+✓ ArgoCD configuration & installation
+✓ Prometheus/Grafana setup
+
+═══════════════════════════════════════════════════════════════════════════
+
+HOW TO NAVIGATE THIS
+═══════════════════════════════════════════════════════════════════════════
+
+If you want to...                          See...
+────────────────────────────────────────────────────────────────────────────
+Understand what was created                ARGO-ROLLOUTS-SUMMARY.md
+Get started quickly                        README-ROLLOUTS.md
+Deploy step-by-step                        ROLLOUTS-CHECKLIST.md
+See detailed scenarios & examples          ROLLOUTS-SETUP.md
+Troubleshoot issues                        ROLLOUTS-SETUP.md (Troubleshooting section)
+Learn technical details                    tracing-demo/ROLLOUTS-DEMO.md
+Install the controller                     argo-rollouts/Justfile + argo-rollouts/README.md
+See the rollout definition                 tracing-demo/helm-chart/templates/frontend-rollout.yaml
+Run a demo                                 tracing-demo/rollout-demo.sh or just rollout-watch
+Monitor during rollout                     Prometheus/Grafana (see ROLLOUTS-SETUP.md)
+Integrate with CI/CD                       See ROLLOUTS-SETUP.md section "GitOps Flow"
+
+═══════════════════════════════════════════════════════════════════════════
diff --git a/f3s/ROLLOUTS-SETUP.md b/f3s/ROLLOUTS-SETUP.md
new file mode 100644
index 0000000..b7ebb55
--- /dev/null
+++ b/f3s/ROLLOUTS-SETUP.md
@@ -0,0 +1,429 @@
+# Argo Rollouts Setup and Demo Guide
+
+This guide covers the complete setup and demonstration of Argo Rollouts with the tracing-demo application.
+
+## Quick Setup
+
+### 1. Install Argo Rollouts Controller
+
+```bash
+cd /home/paul/git/conf/f3s/argo-rollouts
+just install
+```
+
+Verify installation:
+```bash
+kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts
+kubectl get crd | grep rollout
+```
+
+### 2. Install kubectl Plugin (Optional but Recommended)
+
+```bash
+curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
+chmod +x kubectl-argo-rollouts-linux-amd64
+sudo install -m 755 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts
+```
+
+Verify:
+```bash
+kubectl argo rollouts version
+```