Month: May 2019

I’ve been working on building a Prometheus monitoring stack in Docker swarm, and I ran into an interesting challenge, namely, how to separate my prometheus configuration update process from my container deployment process. The solution I came up with is one that I think can be adapted for other applications with similar properties.

Configuration repository with polling

Prometheus, like many opensource devops tools, uses configuration files to manage it’s configuration. The challenge was to find a way of connecting these files to the prometheus docker container and also allow for the configuration to be updated from version control. Prometheus already has an external trigger to load an updated configuration:

# Load prometheus with web.enable-lifecycle to allow reload via HTTP POST
prometheus --web.enable-lifecycle [other startup flags...]

# Trigger configuration reload
curl -X POST http://prometheus:9090/-/reload

The trick is to do the following:

• pass –web.enable-lifecycle as command line parameter to your prometheus container
• mount an external volume to your prometheus containers as /etc/prometheus (or wherever you have configured prometheus to find it’s configuration.
• use a second container that also mounts the configuration volume, and does the following:
  • runs an update script on a schedule that does the following:
    • pulls the latest version of the config
    • If there is an update, copy the config to the configuration volume
    • Sends a signal to prometheus to reload the configuration

Here is an example of the update script:

#!/bin/bash

CONFIGDIR=$PROMETHEUS_CONFIG_DIR/
WORKDIR=/root
REPODIR=${WORKDIR}/prom-config
REPO=$PROMETHEUS_CONFIG_REPO
NOTIFY_HOST=$PROMETHEUS_HOST_DNS
NOTIFY_PORT=$PROMETHEUS_PORT
NOTIFY_PATH=$PROMETHEUS_NOTIFY_PATH
NOTIFY_METHOD=POST

copy_config () {
  rsync -a --exclude='.*' $REPODIR/ $CONFIGDIR
  echo "config deployed"
}

notify_endpoints () {
  for IP in $(dig $NOTIFY_HOST +short); do curl -X $NOTIFY_METHOD $IP:$NOTIFY_PORT$NOTIFY_PATH; done
  echo "endpoints notified"
}

git_initial_clone () {
    git clone $REPO $REPODIR
    echo "initial clone"
}

git_no_repo () {
    git init
    git remote add origin $REPO
    git pull origin master --force
    echo "clone to existing non-repo dir"
}

if [[ ! -d $REPODIR ]]; then
    git_initial_clone
    copy_config
    notify_endpoints
else
    cd $REPODIR
    if [ ! -d .git ]; then
        git_no_repo
        copy_config
        notify_endpoints        
    else
    	git remote update
    	UPSTREAM=${1:-'@{u}'}
    	LOCAL=$(git rev-parse @)
    	REMOTE=$(git rev-parse "$UPSTREAM")
    	BASE=$(git merge-base @ "$UPSTREAM")

    	if [ $LOCAL = $REMOTE ]; then
            echo "Up-to-date"
    	elif [ $LOCAL = $BASE ]; then
        	git pull --force
        	echo "changes pulled"
        	copy_config
            notify_endpoints
    	fi
    fi
fi

And here is an example docker-compose file that puts it all together.

---
version: '3.7'
services:
  scheduler:
    image: nralbers/scheduler:latest
    configs:
      - source: prom_update_script
        target: /etc/periodic/1min/update_prometheus
        mode: 0555
      - source: ssh_config
        target: /root/.ssh/config
        mode: 0400
    environment:
      - PROMETHEUS_CONFIG_DIR=/etc/prometheus
      - PROMETHEUS_CONFIG_REPO= <your config git repo>
      - PROMETHEUS_HOST_DNS=tasks.prometheus
      - PROMETHEUS_PORT=9090
      - PROMETHEUS_NOTIFY_PATH=/-/reload
    secrets:
      - source: ssh_key
        target: id_rsa
        mode: 0400
    volumes:
    - prom-config:/etc/prometheus  

  prometheus:
    image: prom/prometheus:latest
    ports:
    - 9090:9090
    command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--web.enable-lifecycle'
    - '--storage.tsdb.path=/prometheus'
    - '--web.console.libraries=/usr/share/prometheus/console_libraries'
    - '--web.console.templates=/usr/share/prometheus/consoles'
    volumes:
    - prom_data:/prometheus
    - prom_config:/etc/prometheus
    depends_on:
    - configloader
    - cadvisor
  cadvisor:
    image: google/cadvisor:latest
    ports:
    - 8080:8080
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
configs:
  ssh_config:
    file: ssh_config
  prom_update_script:
    file: update_prometheus.sh
secrets:
  ssh_key:
    file: ${HOME}/.ssh/id_rsa
volumes:
  prom-config: {}
  prom-data: {}

The scheduler image consists of an alpine image modified to add some new cron schedules, and to be able to run the update script. It is hosted in docker hub, source code is here:
https://github.com/nralbers/docker-scheduler

FROM alpine:latest
LABEL maintainer="nralbers@gmail.com"
LABEL version="1.0"
LABEL description="Image running crond with additional schedule options for /etc/periodic/1min and /etc/periodic/5min. \
The image has bash, bind-tools, git & openssh installed. To use: bind mount the scripts you want to schedule to /etc/periodic/<period>"
RUN apk update && apk add bash bind-tools openssh git curl rsync
RUN mkdir -p /etc/periodic/1min && echo "*       *       *       *       *       run-parts /etc/periodic/1min" >> /etc/crontabs/root
RUN mkdir -p /etc/periodic/5min && echo "*/5     *       *       *       *       run-parts /etc/periodic/5min" >> /etc/crontabs/root
ENTRYPOINT ["crond", "-f", "-d", "8"]

General applications

It should be clear that while this example is aimed at dockerised deployments of prometheus, this will also work in other situations as long as the application has the following properties:

• The application uses configuration files for configuration
• It has an external means of forcing a configuration reload
• It is possible to share the configuration storage volume between the application container and the script that pulls the updated config from version control and triggers the reload

Next steps…

• Add a mechanism to validate the config before pushing to the destination system. Ideally, this should also happen on push to the configuration repository.
• Show how to run different versions of the config from the same repository using deployment branches for dev, acceptance and production
• Mechanism to secure configuration in a repository. This could be achieved using ansible-vault to encrypt the configuration pre-checkin and to decrypt on pull. The encryption key can be attached to the scheduler container using a docker secret.