Introduction
Properly setting up the metrics endpoints on a django app running in a docker container has a few important gotcha’s that you should be aware of. This article is specifically aimed at people running the stack described below, but aspects of this are general, so should also work in other contexts.
I’m assuming that the person reading this is already comfortable programming django applications, and knows the basics of how Django settings work.
I’m also assuming a basic knowledge of both Dockerfiles and Kubernetes apply files. If you are working with Kubernetes, I’d strongly advise also looking at Helm, since that makes deploying and managing your applications in Kubernetes so much simpler.
Stack
- Django, using django-prometheus to expose metrics
- Gunicorn webserver
- App + server bundled into Docker container
- Container deployed into Kubernetes
- Kubernetes uses prometheus-operator to monitor applications in the cluster
The Issue: Handling multiple workers
Like most python WSGI servers, Gunicorn spawns worker processes to better handle concurrent requests. However, without further configuration, django-Prometheus assumes it is running in a single process. Unless you explicitly configure it further, exposing a metrics endpoint on the app via urls.py as shown in the README (see below) will result in each metrics request being forwarded to a different worker, leading to counters jumping up and down in value between scrapes.
urlpatterns = [
...
url('', include('django_prometheus.urls')),
]
The Solution: Each worker exposes a metrics endpoint on it’s own port
How to handle this is documented in the library, but this documentation is not linked in the README and is easy to miss. The solution is to specify a port range that each worker (running it’s own copy of the app) will then try to bind a metrics endpoint to.
PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(8001, 8050)
However, there are still some issues that must be solved.
- How to match the port range to the number of workers
- How to expose those ports on a kubernetes pod spec & service
- How to define a prometheus-operator serviceMonitor that will automatically scrape those ports for metrics
Matching the port range to the number of workers
There are a number of ways of doing this, but the easiest is to specify the number of workers by exporting the environment variable WEB_CONCURRENCY=<desired number of workers> to the docker container running your app.
Note that if you do this, you MUST NOT set the number of workers on either the commandline or in gunicorn_config.py as this will override this environment setting.
Then if you also expose the variable METRICS_START_PORT=<desired start port number for worker metrics endpoints>, you can add the following code to your Django settings.py to read in these variables and set the port range appropriately.
import os
start_port = int(os.environ.get("METRICS_START_PORT"))
workers = int(os.environ.get("WEB_CONCURRENCY"))
PROMETHEUS_METRICS_EXPORT_PORT_RANGE = range(start_port, start_port + workers)
Caveat
Adding this setting to a settings file that is used for development with manage.py runserver will give an error about the autoreloader not working in this configuration. Either use a separate settings file for development, or run your local test server with manage.py runserver --noreload
How to expose those ports on a kubernetes pod spec & service
This is a little cumbersome, but you will need to add each port as a uniquely named port to the pod container spec. While doing so, you can also add the WEB_CONCURRENCY and METRICS_START_PORT settings to your environment.
deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
...
labels:
app.kubernetes.io/instance: app
app.kubernetes.io/name: app-django
name: app-django
namespace: app
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: app
app.kubernetes.io/name: app-django
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: app
app.kubernetes.io/name: app-django
spec:
containers:
- env:
- name: METRICS_START_PORT
value: "5001"
- name: WEB_CONCURRENCY
value: "5"
image: mycompany/app:v0.1.0
imagePullPolicy: Always
livenessProbe:
failureThreshold: 3
httpGet:
path: /health/
port: http
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 1
name: django
ports:
- containerPort: 5000
name: http
protocol: TCP
- containerPort: 5001
name: metrics-1
protocol: TCP
- containerPort: 5002
name: metrics-2
protocol: TCP
- containerPort: 5003
name: metrics-3
protocol: TCP
- containerPort: 5004
name: metrics-4
protocol: TCP
- containerPort: 5005
name: metrics-5
protocol: TCP
Note how we create ports named metrics-1 through metrics-5, to match the 5 workers we’ve passed to our app container running with gunicorn.
service.yml
apiVersion: v1
kind: Service
metadata:
annotations:
...
labels:
app.kubernetes.io/instance: app
app.kubernetes.io/name: app-django
name: app-django
namespace: app
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: metrics-1
port: 5001
protocol: TCP
targetPort: metrics-1
- name: metrics-2
port: 5002
protocol: TCP
targetPort: metrics-2
- name: metrics-3
port: 5003
protocol: TCP
targetPort: metrics-3
- name: metrics-4
port: 5004
protocol: TCP
targetPort: metrics-4
- name: metrics-5
port: 5005
protocol: TCP
targetPort: metrics-5
selector:
app.kubernetes.io/instance: app
app.kubernetes.io/name: app-django
type: ClusterIP
Here again we have to match each port name and port. The prometheus-operator will use the selector + the named ports to find all the endpoints on the pods to scrape.
How to define a prometheus-operator serviceMonitor that will automatically scrape those ports for metrics.
servicemonitor.yml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
...
labels:
...
name: app-django
namespace: app
spec:
endpoints:
- interval: 15s
port: metrics-1
- interval: 15s
port: metrics-2
- interval: 15s
port: metrics-3
- interval: 15s
port: metrics-4
- interval: 15s
port: metrics-5
selector:
matchLabels:
app.kubernetes.io/name: app-django
See how the selector matches the labels on the service, and the named port names match the port names on both the deployment and the service.
Bonus: easy creation of yml template with helm.
Since there is a lot of repetition in the kubernetes apply files, it makes sense to automate their creation using helm. A full discussion of helm goes beyond the scope of this article, but I can show how to create a helm range loop to create the repeated parts of the above files.
First, in your chart’s values.yml, add the following block:
env:
config:
METRICS_START_PORT: 5001
WEB_CONCURRENCY: 5
In the env block of deployment.yml, use this to add all environment variable values defined in values.yml or passed to the chart at installation time:
env:
{{- range $key, $value := .Values.env.config }}
- name: {{ $key }}
value: {{ $value | quote }}
{{- end }}
In the ports block of deployment.yml, use this to auto-generate the numbered metrics-# ports.
ports:
- name: http
containerPort: 5000
protocol: TCP
{{- with .Values.env.config }}
{{- $metricsPort := untilStep (int .METRICS_START_PORT) (int (add (int .METRICS_START_PORT) (int .WEB_CONCURRENCY))) 1 -}}
{{- range $index, $port := $metricsPort }}
- name: metrics-{{ add1 $index }}
containerPort: {{ $port }}
protocol: TCP
{{- end }}
{{- end }}
In the ports block of service.yml
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
{{- with .Values.env.config }}
{{- $metricsPort := untilStep (int .METRICS_START_PORT) (int (add (int .METRICS_START_PORT) (int .WEB_CONCURRENCY))) 1 -}}
{{- range $index, $port := $metricsPort }}
- port: {{ $port }}
targetPort: metrics-{{ add1 $index }}
protocol: TCP
name: metrics-{{ add1 $index }}
{{- end }}
{{- end }}
And in the endpoints block of servicemonitor.yml
endpoints:
{{- with .Values.env.config }}
{{- $metricsPort := untilStep (int .METRICS_START_PORT) (int (add (int .METRICS_START_PORT) (int .WEB_CONCURRENCY))) 1 -}}
{{- range $index, $port := $metricsPort }}
- port: metrics-{{ add1 $index }}
interval: 15s
{{- end }}
{{- end }}
