diff options
| -rw-r--r-- | f3s/ROLLOUTS-CHECKLIST.md | 283 | ||||
| -rw-r--r-- | f3s/ROLLOUTS-SETUP.md | 454 | ||||
| -rw-r--r-- | f3s/tracing-demo/ROLLOUTS-DEMO.md | 523 |
3 files changed, 682 insertions, 578 deletions
diff --git a/f3s/ROLLOUTS-CHECKLIST.md b/f3s/ROLLOUTS-CHECKLIST.md index b32f1ac..b475f2d 100644 --- a/f3s/ROLLOUTS-CHECKLIST.md +++ b/f3s/ROLLOUTS-CHECKLIST.md @@ -1,189 +1,222 @@ # Argo Rollouts Deployment Checklist -## Pre-Deployment Setup - -- [ ] Read `ARGO-ROLLOUTS-SUMMARY.md` to understand what was created -- [ ] Ensure kubectl access to f3s cluster -- [ ] Ensure ArgoCD is running and accessible -- [ ] Git repository (conf.git) synced to git-server +Quick checklist for deploying and testing Argo Rollouts with canary demo. ## Installation +- [ ] Read `ARGO-ROLLOUTS-SUMMARY.md` - understand what was created +- [ ] Ensure kubectl access to f3s cluster +- [ ] Ensure ArgoCD is running - [ ] Navigate to `/home/paul/git/conf/f3s/argo-rollouts` -- [ ] Run `just install` to deploy controller -- [ ] Verify controller running: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts` -- [ ] Verify CRD installed: `kubectl get crd | grep rollout` - -## Optional: Install kubectl Plugin - -- [ ] Download kubectl-argo-rollouts: +- [ ] Run `just install` +- [ ] Verify controller: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts` +- [ ] Verify CRD: `kubectl get crd | grep rollout` +- [ ] (Optional) Install plugin: ```bash curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64 chmod +x kubectl-argo-rollouts-linux-amd64 sudo install -m 755 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts + kubectl argo rollouts version ``` -- [ ] Verify: `kubectl argo rollouts version` -## ArgoCD Syncing +## ArgoCD Integration -- [ ] Create/push `argocd-apps/cicd/argo-rollouts.yaml` to git -- [ ] Create/push `argocd-apps/services/tracing-demo.yaml` updates to git -- [ ] Force ArgoCD sync (wait 3 min or manual): +- [ ] Push changes to git-server: + ```bash + cd /home/paul/git/conf/f3s + git add -A && git commit -m "feat: add Argo Rollouts" + git push r0 master + ``` +- [ ] Verify ArgoCD app: ```bash - argocd app sync argo-rollouts - argocd app sync tracing-demo + kubectl get application argo-rollouts -n cicd + argocd app get argo-rollouts + ``` +- [ ] Verify tracing-demo app: + ```bash + kubectl get application tracing-demo -n cicd + argocd app get tracing-demo ``` -- [ ] Verify tracing-demo application status: `argocd app get tracing-demo` ## Rollout Verification -- [ ] Check frontend rollout deployed: `kubectl get rollout tracing-demo-frontend -n services` +- [ ] Check rollout exists: `kubectl get rollout tracing-demo-frontend -n services` - [ ] Verify status: `kubectl describe rollout tracing-demo-frontend -n services` -- [ ] Expected: `Status: Healthy` with `2/2 replicas` in stable state -- [ ] Check pods running: `kubectl get pods -n services -l app=tracing-demo-frontend` +- [ ] Expected: `Status: Healthy` with `3/3 replicas` in stable state +- [ ] Check pods: `kubectl get pods -n services -l app=tracing-demo-frontend` +- [ ] All 3 pods should be `Running` + +## Demo: Basic Canary Rollout -## Basic Demo (First Time) +**Expected: 0-15s: canary starting, 15-60s: observing, 60-90s: promoting** ### Terminal 1: Watch Rollout ```bash cd /home/paul/git/conf/f3s/tracing-demo just rollout-watch ``` -- [ ] Command running and connected +- [ ] Command runs and connects to cluster +- [ ] Waiting for rollout to start -### Terminal 2: Generate Load (Optional) +### Terminal 2: Trigger Rollout ```bash -cd /home/paul/git/conf/f3s/tracing-demo -just load-test & +kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' ``` -- [ ] Requests being sent to frontend +- [ ] Patch command successful +- [ ] Terminal 1 shows change immediately -### Terminal 3: Trigger Rollout -Choose one method: +### Terminal 1: Observe Progress +- [ ] See `Step: 0/3, SetWeight: 33` +- [ ] 1 canary pod becoming ready +- [ ] 3 stable pods still running +- [ ] After ~15 sec: canary pod ready +- [ ] After ~60 sec: auto-promotion starts +- [ ] After ~90 sec: all 3 pods running new version +- [ ] Status shows `Healthy` + +## Demo: Abort/Rollback + +**Expected: Stop rollout and keep old version running** -**Method A: Kubectl Patch (Fastest)** +### Terminal 1: Watch Rollout +```bash +just rollout-watch +``` + +### Terminal 2: Trigger Rollout ```bash kubectl patch rollout tracing-demo-frontend -n services \ --type='json' \ - -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]' + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V2","value":"'$(date +%s)'"}}]' +``` + +### Terminal 3: Abort at Canary Step (after 20 seconds) +```bash +cd /home/paul/git/conf/f3s/tracing-demo +just rollout-abort ``` -- [ ] Executed successfully +- [ ] Abort command accepted +- [ ] Terminal 1 shows `Status: Aborted` +- [ ] Canary pods terminate +- [ ] Old 3 pods continue running +- [ ] Verify with: `just rollout-status` -**Method B: Git + ArgoCD (Most GitOps)** +## Demo: Load Testing + +**Expected: Generate traffic while rollout happens** + +### Terminal 1: Watch Rollout +```bash +just rollout-watch +``` + +### Terminal 2: Start Load Test ```bash -cd /home/paul/git/conf/f3s -# Edit tracing-demo/helm-chart/templates/frontend-rollout.yaml (change image tag) -git add -A -git commit -m "chore: update frontend image for demo" -git remote add r0 ssh://git@r0:30022/repos/conf.git 2>/dev/null || true -git push r0 master -kubectl annotate application tracing-demo -n cicd argocd.argoproj.io/refresh=normal --overwrite +just load-test & +``` +- [ ] Requests being sent + +### Terminal 3: Trigger Rollout +```bash +kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V3","value":"'$(date +%s)'"}}]' ``` -- [ ] Git push successful -- [ ] ArgoCD syncing (check web UI or CLI) +- [ ] Rollout progresses with active traffic +- [ ] Both old and new pods serve requests during canary phase -## Demo Observation +## Monitoring -- [ ] Terminal 1 shows: "Progressing" → "canary step 1/3" -- [ ] After ~30 sec: New canary pod appears -- [ ] After ~2 min: "canary step 2/3" (pause) -- [ ] After ~4 min: "canary step 3/3" (100% traffic) -- [ ] After ~4:20 min: Status shows "Healthy" -- [ ] Old pods terminated, 2 new pods in stable state +- [ ] Check status: `kubectl argo rollouts status tracing-demo-frontend -n services` +- [ ] Detailed info: `kubectl argo rollouts describe rollout tracing-demo-frontend -n services` +- [ ] Pod details: `kubectl get pods -n services -l app=tracing-demo-frontend -o wide` +- [ ] View logs: `just logs-frontend` +- [ ] View history: `just rollout-history` -## Monitoring (Optional) +## Grafana (Optional) -- [ ] Check logs: `just logs-frontend` -- [ ] Check Grafana Tempo for traces: https://grafana.f3s.buetow.org - - [ ] Navigate to Explore → Tempo - - [ ] Query: `{ resource.service.name = "frontend" }` - - [ ] See traces from old and new versions -- [ ] Check Prometheus metrics: Port-forward and query +- [ ] Open Grafana: https://grafana.f3s.buetow.org +- [ ] Navigate to Explore → Tempo datasource +- [ ] Query: `{ resource.service.name = "frontend" }` +- [ ] See traces from old and new versions during rollout -## Advanced Scenarios +## Integration with Git (GitOps) -### Scenario 1: Manual Promotion -- [ ] Trigger rollout (step above) -- [ ] After step 1 (30 sec), run: +- [ ] Edit rollout config: ```bash - just rollout-promote + nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml ``` -- [ ] Watch rollout skip step 2, immediately promote to 100% -- [ ] Verify: `just rollout-status` shows "Healthy" - -### Scenario 2: Abort/Rollback -- [ ] Trigger rollout -- [ ] While progressing, run: +- [ ] Change any settings (e.g., duration, setWeight) +- [ ] Commit and push: ```bash - just rollout-abort + git add -A && git commit -m "chore: adjust canary settings" + git push r0 master ``` -- [ ] Watch canary pods terminate -- [ ] Old version continues running -- [ ] Verify: `just rollout-status` shows "Aborted" - -### Scenario 3: Check History -- [ ] After any rollout: +- [ ] ArgoCD auto-syncs within 3 minutes (or force): ```bash - just rollout-history + kubectl annotate application tracing-demo -n cicd argocd.argoproj.io/refresh=normal --overwrite ``` -- [ ] See previous revisions and their status - -## Integration with CI/CD +- [ ] New settings take effect on next rollout trigger -- [ ] Image builds automatically on git push (or configured pipeline) -- [ ] New image pushed to registry: `registry.lan.buetow.org:30001/tracing-demo-frontend:NEWTAG` -- [ ] Git updated with new image tag -- [ ] ArgoCD detects change -- [ ] Rollout automatically triggered -- [ ] Canary strategy executes +## Post-Demo -## Post-Deployment +- [ ] Abort any stuck rollouts: `just rollout-abort` +- [ ] Verify stable state: `just rollout-status` shows `Healthy` +- [ ] Review documentation: + - [ ] `ARGO-ROLLOUTS-SUMMARY.md` - architecture + - [ ] `ROLLOUTS-SETUP.md` - detailed scenarios + - [ ] `README-ROLLOUTS.md` - quick reference + - [ ] `tracing-demo/ROLLOUTS-DEMO.md` - technical details -- [ ] Share documentation: - - [ ] `ROLLOUTS-SETUP.md` - Complete setup guide - - [ ] `tracing-demo/ROLLOUTS-DEMO.md` - Detailed walkthrough - - [ ] `ARGO-ROLLOUTS-SUMMARY.md` - Architecture overview -- [ ] Add team to `kubectl argo rollouts` usage -- [ ] Consider next steps: - - [ ] Deploy Istio for advanced traffic management - - [ ] Add Flagger for automated analysis - - [ ] Extend to other services (middleware, backend) - - [ ] Create monitoring dashboards - -## Troubleshooting Checklist +## Troubleshooting ### Controller not running -- [ ] Check pod: `kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts` -- [ ] Check logs: `kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts` -- [ ] Check CRD: `kubectl get crd | grep rollout` +```bash +kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts +kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts +``` +- [ ] Pod running and ready -### Rollout not deploying -- [ ] Check ArgoCD sync: `argocd app get tracing-demo` -- [ ] Check git changes pushed: `git log --oneline | head -5` -- [ ] Force sync: `argocd app sync tracing-demo --prune` +### Rollout not deployed +```bash +kubectl get rollout tracing-demo-frontend -n services +kubectl describe rollout tracing-demo-frontend -n services +``` +- [ ] Check events section for errors -### Canary pods not starting -- [ ] Check pod status: `kubectl describe pod -n services <pod-name>` -- [ ] Check logs: `kubectl logs -n services <pod-name>` -- [ ] Check resource limits: `kubectl top pods -n services` -- [ ] Check image: `kubectl get pods -n services -o jsonpath='{.items[*].spec.containers[0].image}'` +### Canary pods in ImagePullBackoff +- [ ] Use env var patch instead (don't change image tag): + ```bash + kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' + ``` ### Rollout stuck in Progressing -- [ ] Check health probes: `kubectl get rollout tracing-demo-frontend -n services -o yaml | grep -A 10 health` -- [ ] Check replica status: `kubectl get rs -n services -l app=tracing-demo-frontend -o wide` -- [ ] Check controller logs: `kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts --tail=50` +```bash +kubectl describe rollout tracing-demo-frontend -n services +kubectl get pods -n services -l app=tracing-demo-frontend +``` +- [ ] Check pod readiness probes +- [ ] Check pod resource requests/limits +- [ ] Check controller logs -## Cleanup (If Needed) +## Next Steps -- [ ] Stop rollout: `kubectl argo rollouts abort tracing-demo-frontend -n services` -- [ ] Rollback to previous: `kubectl rollout undo deployment/tracing-demo-frontend -n services` (if needed) -- [ ] Uninstall Argo Rollouts: `cd argo-rollouts && just uninstall` +- [ ] Run through all demo scenarios multiple times +- [ ] Modify rollout settings and observe behavior +- [ ] Monitor with Prometheus/Grafana +- [ ] Extend to other services (middleware, backend) +- [ ] Optional: Install Istio for advanced traffic routing +- [ ] Optional: Deploy Flagger for automated analysis --- -**Setup complete when:** -- ✅ Argo Rollouts controller running in `cicd` namespace -- ✅ Frontend rollout deployed in `services` namespace -- ✅ ArgoCD recognizes rollout resource -- ✅ One demo run successful (git trigger or kubectl patch) -- ✅ Team can watch and manage rollouts +**Setup Complete When:** +- ✅ Controller running in `cicd` namespace +- ✅ Rollout deployed in `services` namespace +- ✅ One full demo executed (0-90 seconds) +- ✅ Can abort and retry +- ✅ Team trained on canary deployments diff --git a/f3s/ROLLOUTS-SETUP.md b/f3s/ROLLOUTS-SETUP.md index b7ebb55..0ea965c 100644 --- a/f3s/ROLLOUTS-SETUP.md +++ b/f3s/ROLLOUTS-SETUP.md @@ -1,6 +1,6 @@ # Argo Rollouts Setup and Demo Guide -This guide covers the complete setup and demonstration of Argo Rollouts with the tracing-demo application. +Complete setup and demonstration of Argo Rollouts with the tracing-demo application. Canary strategy: 33% traffic (1 pod) for 1 minute, then auto-promote to 100%. ## Quick Setup @@ -32,19 +32,11 @@ kubectl argo rollouts version ### 3. Sync ArgoCD with New Applications -The following ArgoCD Applications will be auto-synced: - -- **argo-rollouts.yaml** - Installs Argo Rollouts controller -- **tracing-demo.yaml** - Now uses Rollout (frontend) + Deployments (middleware, backend) - -Force ArgoCD to sync: ```bash argocd app sync argo-rollouts argocd app sync tracing-demo ``` -Or wait for auto-sync (default: 3 minutes). - ### 4. Verify Rollout is Deployed ```bash @@ -52,378 +44,330 @@ kubectl get rollout tracing-demo-frontend -n services kubectl describe rollout tracing-demo-frontend -n services ``` -Expected status: `Stable` with `2/2 replicas`. - -## Demo Scenarios - -### Scenario 1: Basic Canary Rollout (Guided) +Expected status: `Healthy` with `3/3 replicas` in stable state. -**Duration**: ~5-10 minutes +## Quick Demo (90 seconds) -**Objective**: Observe frontend rollout from 50% → 100% traffic with auto-promotion. +### Terminal 1 - Watch Progress -#### Step 1: Prepare Terminals - -Terminal 1 - Watch rollout progress: ```bash cd /home/paul/git/conf/f3s/tracing-demo just rollout-watch ``` -Terminal 2 - Generate load: -```bash -cd /home/paul/git/conf/f3s/tracing-demo -just load-test & -``` - -Terminal 3 - Trigger rollout: +Or use the kubectl command directly: ```bash -# Will use this in next step +kubectl argo rollouts get rollout tracing-demo-frontend -n services --watch ``` -#### Step 2: Trigger Rollout (Terminal 3) +### Terminal 2 - Trigger Rollout -Simulate updating the frontend image: +Wait 10 seconds for Terminal 1 to start watching, then trigger: ```bash kubectl patch rollout tracing-demo-frontend -n services \ --type='json' \ - -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]' + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' ``` -Or via git (more GitOps-like): +### Watch the Timeline -```bash -cd /home/paul/git/conf/f3s -# Edit tracing-demo/helm-chart/templates/frontend-rollout.yaml (change image tag) -git add -A -git commit -m "chore: update frontend image for demo" -git remote add r0 ssh://git@r0:30022/repos/conf.git 2>/dev/null || true -git push r0 master +**Terminal 1 will show:** -# Trigger ArgoCD sync -kubectl annotate application tracing-demo -n cicd argocd.argoproj.io/refresh=normal --overwrite ``` - -#### Step 3: Observe Rollout (Terminal 1) - -Watch the output: - +Step: 0/3 +SetWeight: 33 +Canary: 1 pod (new version) - starting +Stable: 3 pods (old version) - handling requests ``` -NAME KIND STATUS AGE INFO -tracing-demo-frontend Rollout Progressing 0s canary step 1/3 -tracing-demo-frontend-abc123 ReplicaSet ✓ canary 5s 1/1 replicas -tracing-demo-frontend-xyz789 ReplicaSet ✓ stable 5m 2/2 replicas - -NAME KIND STATUS AGE INFO -tracing-demo-frontend Rollout Progressing 2m5s canary step 2/3 -tracing-demo-frontend-abc123 ReplicaSet ✓ canary 2m 1/1 replicas (ready) -tracing-demo-frontend-xyz789 ReplicaSet ✓ stable 5m 2/2 replicas -NAME KIND STATUS AGE INFO -tracing-demo-frontend Rollout Progressing 4m10s canary step 3/3 -tracing-demo-frontend-abc123 ReplicaSet ✓ canary 4m 2/2 replicas (ready, updated) -tracing-demo-frontend-xyz789 ReplicaSet ✓ stable 5m 0/2 replicas (pending termination) +→ After 15 seconds, canary pod becomes ready: -NAME KIND STATUS AGE INFO -tracing-demo-frontend Rollout ✓ Healthy 4m20s -tracing-demo-frontend-abc123 ReplicaSet ✓ stable 4m 2/2 replicas +``` +Step: 1/3 +SetWeight: 33 +Canary: 1 pod (new version) - ready, receiving 33% traffic +Stable: 3 pods (old version) - receiving 67% traffic ``` -**Timeline:** -- **0-2 min**: Step 1 (setWeight: 50) - 1 canary pod, 2 stable pods, 50/50 traffic -- **2-4 min**: Step 2 (pause: 2m) - Waiting for user or auto-promotion -- **4+ min**: Step 3 (setWeight: 100) - All 2 canary pods promoted, old pods terminated -- **4:20 min**: Complete - New version fully deployed - -#### Step 4: Observe Behavior (Optional) +→ After ~60 seconds, auto-promotion begins: -Check request latency/errors during rollout: +``` +Step: 2/3 +SetWeight: 100 +Canary scaling → Stable +``` -```bash -# View logs from both old and new pods -kubectl logs -n services -l app=tracing-demo-frontend --timestamps=true | tail -20 +→ After ~90 seconds, complete: -# Check if any requests failed during transition -grep -i "error\|exception" <(kubectl logs -n services -l app=tracing-demo-frontend) +``` +Status: Healthy +Replicas: 3/3 all running new version ``` -View traces in Grafana: -1. Navigate to https://grafana.f3s.buetow.org -2. Explore → Tempo -3. Query: `{ resource.service.name = "frontend" }` -4. See traces from both old and new versions +## Demo Scenarios -### Scenario 2: Manual Promotion (Skip Waiting) +### Scenario 1: Observe the Full Rollout -**Duration**: ~2 minutes +Just follow the "Quick Demo" above. Watch all three steps progress automatically over 90 seconds. -**Objective**: Demonstrate manual control - don't wait for auto-promotion. +### Scenario 2: Abort Rollout (Simulate Failure) -#### Setup +**Terminal 1**: Watch the rollout +```bash +just rollout-watch +``` -Trigger rollout (same as Scenario 1): +**Terminal 2**: Trigger rollout ```bash kubectl patch rollout tracing-demo-frontend -n services \ --type='json' \ - -p='[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"registry.lan.buetow.org:30001/tracing-demo-frontend:latest"}]' + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' ``` -Watch: +**Terminal 3 (while at step 1)**: Abort the rollout ```bash -just rollout-watch +cd /home/paul/git/conf/f3s/tracing-demo +just rollout-abort ``` -#### Promote Early - -After canary looks healthy (step 1 complete, ~30 seconds): +Result: +- Canary pods terminate +- Old 3 pods continue running +- Status shows "Aborted" +Verify: ```bash -cd /home/paul/git/conf/f3s/tracing-demo -just rollout-promote +just rollout-status ``` -This skips the 2-minute pause and immediately promotes to 100%. - -### Scenario 3: Abort/Rollback - -**Duration**: ~3 minutes - -**Objective**: Demonstrate rollback if canary fails. - -#### Setup & Trigger - -Same as Scenario 1. - -#### Simulate Failure - -While at canary step 1 (50% traffic), introduce a failure: +### Scenario 3: Load Testing During Rollout +**Terminal 1**: Watch rollout ```bash -# Get one of the new canary pods -CANARY_POD=$(kubectl get pods -n services -l app=tracing-demo-frontend -o name | tail -1) - -# Kill it to simulate crash -kubectl delete $CANARY_POD -n services +just rollout-watch ``` -Watch in Terminal 1 - the rollout may stall or fail health checks. - -#### Abort - +**Terminal 2**: Start load test ```bash -cd /home/paul/git/conf/f3s/tracing-demo -just rollout-abort +just load-test & ``` -This: -- Stops the rollout -- Terminates canary replicas -- Restores stable version with 2 pods -- Allows investigation - -Verify: +**Terminal 3**: Trigger rollout ```bash -just rollout-status +kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' ``` -Expected: `Rollout has been aborted. Stable ReplicaSet: 2/2 replicas` +Load test will hit both old and new pods during the 1-minute canary window. -### Scenario 4: Observability - Prometheus Metrics - -**Duration**: ~5 minutes (during any rollout) - -**Objective**: Monitor rollout via Prometheus metrics. - -During a running rollout: +### Scenario 4: Check Logs During Rollout +**Terminal 1**: Watch rollout ```bash -# Port-forward Prometheus -kubectl port-forward -n monitoring svc/prometheus 9090:9090 & +just rollout-watch +``` -# Open browser: http://localhost:9090 +**Terminal 2**: Trigger rollout +```bash +kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' ``` -Query useful metrics: +**Terminal 3**: Watch logs +```bash +kubectl logs -n services -l app=tracing-demo-frontend -f --tail=20 +``` -```promql -# Rollout replica counts -kube_statefulset_replicas{statefulset=~".*frontend.*"} -kube_replicaset_created{replicaset=~".*frontend.*"} +See logs from both old and new pods. -# Pod status during rollout -kube_pod_status_phase{namespace="services", pod=~".*frontend.*"} +### Scenario 5: Monitor via Grafana Tempo (Distributed Tracing) -# Request latency (if your app exports metrics) -rate(http_requests_total{job="frontend"}[5m]) +**Terminal 1**: Watch rollout +```bash +just rollout-watch +``` -# Error rate -rate(http_requests_total{job="frontend", status=~"5.."}[5m]) +**Terminal 2**: Trigger rollout +```bash +kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' ``` -### Scenario 5: GitOps Flow (Realistic) +**Terminal 3**: Open Grafana +1. Navigate to https://grafana.f3s.buetow.org +2. Go to Explore → Select "Tempo" datasource +3. Query: `{ resource.service.name = "frontend" }` +4. See traces from both old and new versions during canary phase + +## Timeline Breakdown -**Duration**: ~10 minutes +| Time | Event | Status | +|------|-------|--------| +| 0s | Trigger rollout | Rollout starts | +| 0-5s | Canary pod created | `Step 0/3: SetWeight 33` | +| 5-15s | Canary pod becoming ready | Still not ready | +| 15s | Canary pod ready | `Step 1/3: SetWeight 33, canary ready` | +| 15-60s | Observing canary | Requests split 67/33 (old/new) | +| 60s | Auto-promotion triggered | `Step 2/3: SetWeight 100` | +| 60-70s | Scaling new pods | Canary → Stable | +| 70-80s | Terminating old pods | Old pods scaling down | +| ~90s | Complete | `Status: Healthy, 3/3 replicas` | -**Objective**: Demonstrate GitOps workflow - git commit triggers rollout via ArgoCD. +## Monitoring During Rollout -#### Step 1: Modify Frontend Code +### kubectl Commands +Real-time status: ```bash -cd /home/paul/git/conf/f3s/tracing-demo/docker/frontend -# Edit app.py (e.g., change response message) -# Commit and push -git add -A -git commit -m "feat: update frontend message" -git push origin master +kubectl argo rollouts get rollout tracing-demo-frontend -n services --watch ``` -#### Step 2: Rebuild and Push Image - +Check specific details: ```bash -cd /home/paul/git/conf/f3s/tracing-demo -just build-push +kubectl argo rollouts describe rollout tracing-demo-frontend -n services +kubectl argo rollouts history tracing-demo-frontend -n services ``` -This creates new Docker image tagged with latest commit hash or timestamp. - -#### Step 3: Update Helm Chart - +Pod status: ```bash -# Edit frontend-rollout.yaml with new image tag -nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml -# Change image: registry.lan.buetow.org:30001/tracing-demo-frontend:NEWTAG - -git add -A -git commit -m "chore: update frontend rollout image to latest" -git remote add r0 ssh://git@r0:30022/repos/conf.git 2>/dev/null || true -git push r0 master +kubectl get pods -n services -l app=tracing-demo-frontend -o wide ``` -#### Step 4: ArgoCD Syncs Automatically +### Prometheus Metrics -Wait 3 minutes or force sync: ```bash -argocd app sync tracing-demo --prune +# Port-forward Prometheus +kubectl port-forward -n monitoring svc/prometheus 9090:9090 ``` -ArgoCD detects the new image in git and updates the rollout. +Then query: +```promql +# Pod counts during rollout +kube_replicaset_replicas{replicaset=~"tracing-demo-frontend.*"} -#### Step 5: Watch Rollout Progress +# Pod status +kube_pod_status_phase{namespace="services", pod=~"tracing-demo-frontend.*"} -```bash -just rollout-watch +# Pod age (shows which are old vs new) +time() - kube_pod_created{namespace="services", pod=~"tracing-demo-frontend.*"} ``` -The canary strategy executes: 50% → wait 2min → 100%. +### Grafana Dashboards -## Monitoring Dashboard +1. Open Grafana: https://grafana.f3s.buetow.org +2. Explore → Tempo datasource +3. Query: `{ resource.service.name = "frontend" }` +4. See traces from old and new versions +5. Notice latency/error differences during rollout -Create a Grafana dashboard to visualize rollout progress: +## Rollout Configuration -1. Open Grafana: https://grafana.f3s.buetow.org -2. Dashboards → New → Create -3. Add panels: +Located in: `/home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml` -**Panel 1: Rollout Status** -```promql -kube_rollout_status_current_step{rollout="tracing-demo-frontend"} +Key settings: +```yaml +replicas: 3 # 3 pods total +strategy: + canary: + steps: + - setWeight: 33 # Send 1 pod (33%) to canary + - pause: + duration: 1m # Wait 1 minute, then auto-promote + - setWeight: 100 # Promote all to new version ``` -**Panel 2: Replica Counts** -```promql -topk(2, kube_replicaset_replicas{replicaset=~"tracing-demo-frontend.*"}) -``` +To modify pause duration: +```bash +# Edit the file +nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml -**Panel 3: Pod Age** -```promql -time() - kube_pod_created{namespace="services", pod=~"tracing-demo-frontend.*"} +# Change duration: 1m to duration: 5m (for example) +# Then commit and push +git add -A && git commit -m "chore: extend canary pause to 5 minutes" +git push r0 master ``` -**Panel 4: Request Rate** -```promql -rate(http_requests_total{job="tracing-demo-frontend"}[1m]) -``` +ArgoCD will auto-sync the new rollout configuration. -## Advanced: Custom Analysis +## Troubleshooting -To add automated health checks during canary (e.g., error rate thresholds), integrate with **Flagger**: +### Rollout shows "ErrImagePull" on canary pod -```yaml -apiVersion: flagger.app/v1beta1 -kind: Canary -metadata: - name: tracing-demo-frontend -spec: - targetRef: - apiVersion: argoproj.io/v1alpha1 - kind: Rollout - name: tracing-demo-frontend - progressDeadlineSeconds: 300 - service: - port: 5000 - analysis: - interval: 1m - threshold: 2 - maxWeight: 50 - stepWeight: 10 - metrics: - - name: error_rate - thresholdRange: - max: 1 # Max 1% error rate -``` - -This requires installing **Flagger** and requires a service mesh (Istio/Linkerd). +This happens if using an image tag that doesn't exist. The env var patch approach forces a rollout without changing the image, so use: -## Troubleshooting +```bash +kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' +``` -### Rollout Stuck in Progressing +### Rollout stuck in "Progressing" +Check pod status: ```bash kubectl describe rollout tracing-demo-frontend -n services +kubectl get pods -n services -l app=tracing-demo-frontend ``` -Check for: -- Pod failures (CrashLoopBackOff) -- Image pull errors -- Resource exhaustion -- Health probe failures +Check controller logs: +```bash +kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts --tail=50 +``` -### Canary Pods Not Becoming Ready +### Controller not running ```bash -kubectl get pods -n services -l app=tracing-demo-frontend -o wide -kubectl logs -n services -l app=tracing-demo-frontend --tail=50 +kubectl get pods -n cicd -l app.kubernetes.io/name=argo-rollouts +kubectl logs -n cicd -l app.kubernetes.io/name=argo-rollouts ``` -### ArgoCD Not Syncing Rollout Changes +### Auto-promotion not happening +Verify pause duration is set: ```bash -kubectl get application tracing-demo -n cicd -o jsonpath='{.status.sync.status}' -argocd app sync tracing-demo +kubectl get rollout tracing-demo-frontend -n services -o yaml | grep -A 5 "pause:" ``` -### kubectl argo rollouts Plugin Issues +## Advanced: Modify Canary Parameters + +### Increase observation time to 5 minutes ```bash -kubectl argo rollouts version +# Edit rollout YAML +nano /home/paul/git/conf/f3s/tracing-demo/helm-chart/templates/frontend-rollout.yaml -# If not installed or outdated: -curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64 -sudo install -m 755 kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts +# Change: +# - pause: +# duration: 1m +# To: +# - pause: +# duration: 5m + +git add -A && git commit -m "chore: extend canary pause to 5 minutes" +git push r0 master +``` + +### Reduce traffic weight to canary (more conservative) + +```yaml +steps: +- setWeight: 10 # Only 10% traffic (0.3 pods worth) +- pause: + duration: 2m # Observe longer +- setWeight: 100 ``` -## Next Steps +### Add health check analysis (requires Flagger or ArgoCD Analysis) -1. **Try all scenarios** to understand rollout behavior -2. **Deploy Istio** for advanced traffic management (weighted routing, header-based) -3. **Add Prometheus queries** to monitor rollout metrics -4. **Implement Flagger** for automated analysis and rollback -5. **Migrate other services** to Rollout (start with low-risk apps) +For automated rollback based on error rate thresholds, see `/home/paul/git/conf/f3s/ROLLOUTS-SETUP.md` → "Advanced: Custom Analysis" section. ## References - [Argo Rollouts Canary Strategy](https://argoproj.github.io/argo-rollouts/features/canary/) -- [Argo Rollouts Blue-Green Strategy](https://argoproj.github.io/argo-rollouts/features/bluegreen/) -- [Flagger Documentation](https://flagger.app/) -- [Istio VirtualService](https://istio.io/latest/docs/reference/config/networking/virtual-service/) +- [Argo Rollouts Best Practices](https://argoproj.github.io/argo-rollouts/best-practices/) +- [kubectl-argo-rollouts Plugin](https://argoproj.github.io/argo-rollouts/getting-started/#using-kubectl-with-argo-rollouts) +- [Flagger for Automated Analysis](https://flagger.app/) diff --git a/f3s/tracing-demo/ROLLOUTS-DEMO.md b/f3s/tracing-demo/ROLLOUTS-DEMO.md index 53a43b3..775a581 100644 --- a/f3s/tracing-demo/ROLLOUTS-DEMO.md +++ b/f3s/tracing-demo/ROLLOUTS-DEMO.md @@ -1,317 +1,444 @@ -# Argo Rollouts Demo Guide for Tracing-Demo +# Argo Rollouts Demo - Technical Details -This guide demonstrates progressive delivery using Argo Rollouts with the tracing-demo frontend service. +Detailed technical walkthrough of Argo Rollouts canary strategy for tracing-demo frontend. -## Prerequisites +## Quick Demo (90 seconds) -- Argo Rollouts installed in `cicd` namespace -- ArgoCD synced with the latest conf.git -- tracing-demo-frontend rollout deployed -- kubectl argo rollouts plugin installed - -### Install kubectl argo rollouts plugin +### Setup ```bash -curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64 -chmod +x kubectl-argo-rollouts-linux-amd64 -sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts -``` +# Terminal 1: Watch the rollout +cd /home/paul/git/conf/f3s/tracing-demo +just rollout-watch -## Demo Workflow +# Terminal 2: Trigger the rollout (after Terminal 1 is watching) +kubectl patch rollout tracing-demo-frontend -n services \ + --type='json' \ + -p='[{"op":"add","path":"/spec/template/spec/containers/0/env/-","value":{"name":"ROLLOUT_V","value":"'$(date +%s)'"}}]' +``` -### 1. Verify Current State +### Execution Timeline +**t=0-15s: Canary Launch** ```bash -# Check frontend rollout -kubectl get rollout tracing-demo-frontend -n services - -# Get detailed status -kubectl argo rollouts status tracing-demo-frontend -n services -kubectl argo rollouts get rollout tracing-demo-frontend -n services +# Terminal 1 shows: +Name: tracing-demo-frontend +Status: ◌ Progressing +Strategy: Canary + Step: 0/3 + SetWeight: 33 + ActualWeight: 0 +Images: (new) tracing-demo-frontend (canary) + (old) tracing-demo-frontend (stable) +Replicas: + Desired: 3 + Current: 4 # 3 stable + 1 canary being created + Updated: 1 + Ready: 3 # 3 stable pods ready, canary still starting ``` -Expected output shows 2 stable replicas, 0 canary. - -### 2. Generate Load (Optional but Recommended) +**t=15-60s: Canary Observation** +```bash +# After canary pod becomes ready (~15 seconds) +Status: ◌ Progressing + Step: 1/3 # Now in pause step + SetWeight: 33 + ActualWeight: 33 # Actual weight achieved +Replicas: + Desired: 3 + Current: 4 + Updated: 1 + Ready: 4 # All 4 pods (3 stable + 1 canary) ready + Available: 4 +``` -In a separate terminal, generate traffic to the frontend: +Service routes traffic: +- **Old version**: 3 pods → ~67% traffic +- **New version**: 1 pod → ~33% traffic +**t=60s: Auto-Promotion** ```bash -# Port-forward frontend -kubectl port-forward -n services svc/frontend-service 5000:5000 & - -# Send requests in a loop -while true; do - curl http://localhost:5000/api/process -s | jq . - sleep 1 -done +# After 1 minute pause duration +Status: ◌ Progressing + Step: 2/3 # Now promoting + SetWeight: 100 +Replicas: + Desired: 3 + Current: 4 + Updated: 3 # All 3 new pods created + Ready: 3 # 3 new pods ready + Available: 3 ``` -Or use the load test: +Old pods terminate, new pods scale up. +**t=90s: Complete** ```bash -cd /home/paul/git/conf/f3s/tracing-demo -just load-test & +Status: ✔ Healthy + Step: 3/3 # Complete + SetWeight: 100 + ActualWeight: 100 +Replicas: + Desired: 3 + Current: 3</ |