blob: a2cce544493106e9117bd423d46502bb803f45e6 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
|
# Pushgateway Helm Chart
Prometheus Pushgateway deployment for the f3s Kubernetes cluster.
## Overview
Pushgateway is an intermediary service that allows ephemeral and batch jobs to expose metrics to Prometheus. It receives metrics pushed to it via HTTP and exposes them for Prometheus to scrape.
## Installation
```bash
cd helm-chart
helm upgrade --install pushgateway . -n monitoring --create-namespace
```
## Configuration
Edit `values.yaml` to customize:
```yaml
replicaCount: 1 # Number of replicas
image:
repository: prom/pushgateway
tag: v1.10.0 # Pushgateway version
service:
type: ClusterIP # Service type
port: 9091 # Port number
resources: # Resource limits
requests:
memory: "64Mi"
cpu: "100m"
```
## Usage
### Push Metrics
Push metrics to the Pushgateway:
```bash
# From inside the cluster
curl -X POST http://pushgateway.monitoring.svc.cluster.local:9091/metrics/job/some_job \
--data-binary @metrics.txt
# From outside (with port-forward)
kubectl port-forward -n monitoring svc/pushgateway 9091:9091
curl -X POST http://localhost:9091/metrics/job/some_job \
--data-binary @metrics.txt
```
### View Pushgateway UI
```bash
kubectl port-forward -n monitoring svc/pushgateway 9091:9091
# Open http://localhost:9091
```
### Delete Metrics
```bash
# Delete all metrics for a job
curl -X DELETE http://localhost:9091/metrics/job/some_job
# Delete metrics for a specific instance
curl -X DELETE http://localhost:9091/metrics/job/some_job/instance/some_instance
```
## Prometheus Configuration
Ensure Prometheus is configured to scrape the Pushgateway:
```yaml
- job_name: 'pushgateway'
honor_labels: true
static_configs:
- targets:
- 'pushgateway.monitoring.svc.cluster.local:9091'
```
**Important**: Use `honor_labels: true` to preserve job and instance labels from pushed metrics.
## Helm Commands
```bash
# Install
helm install pushgateway . -n monitoring
# Upgrade
helm upgrade pushgateway . -n monitoring
# Uninstall
helm uninstall pushgateway -n monitoring
# View values
helm get values pushgateway -n monitoring
# View status
helm status pushgateway -n monitoring
```
## Verification
```bash
# Check deployment
kubectl get pods -n monitoring -l app=pushgateway
# Check service
kubectl get svc -n monitoring pushgateway
# View logs
kubectl logs -n monitoring -l app=pushgateway
# Test metrics endpoint
kubectl port-forward -n monitoring svc/pushgateway 9091:9091
curl http://localhost:9091/metrics
```
## Use Cases
Pushgateway is designed for:
- **Batch jobs**: Jobs that run periodically and exit
- **Short-lived processes**: Processes that don't live long enough to be scraped
- **Service-level metrics**: Aggregated metrics from multiple instances
- **Firewall/NAT scenarios**: When Prometheus can't reach the target
**Not recommended for**:
- Long-running applications (use `/metrics` endpoint instead)
- High-cardinality metrics
- Real-time monitoring (introduces staleness)
## Metrics Format
Push metrics in Prometheus text format:
```
# HELP metric_name Description of the metric
# TYPE metric_name counter
metric_name{label1="value1",label2="value2"} 42
```
## Architecture
```
┌─────────────────┐
│ Applications │
│ Batch Jobs │──Push──┐
│ Scripts │ │
└─────────────────┘ │
▼
┌──────────────────┐
│ Pushgateway │◄──Scrape──┐
│ :9091 │ │
└──────────────────┘ │
┌────────────────┐
│ Prometheus │
└────────────────┘
```
## Notes
- Pushgateway does NOT implement any authentication
- Consider network policies if exposing externally
- Metrics persist until explicitly deleted or Pushgateway restarts
- Use persistence if you need metrics to survive restarts
|
