Autoscaling gitlab-runners with docker-machine on OpenStack

There are many different ways to setup your gitlab-runners executing your builds. My goto executor is docker. It's very clean and pretty simple:

• install docker
• install gitlab-runner
• register the runner

... and you're ready to go. Jobs get executed inside containers and after that, everything is cleaned up. This is great for simple and medium tasks, from a resource perspective. For long running tasks and resource intesive Jobs... I don't know... Maby not so much...

In this Post I want to show you one way using the docker-machine executor for spawning and autoscaling OpenStack Instances. I think this way is more suted for heavy lifting Jobs, but I get ahead of myself.

How does it work

Gitlab-runners with docker-machine executor are able to talk to different platforms via drivers. In case of OpenStack it spawns instances as docker-hosts in your Project and delegates docker run for each Job to these instances/minions. When everything is done these machines will be terminated aka scaled down.

Since spawning instances can take a view moments, there are a bunch of settings to keep some machines idling arround. Gitlab's in depth autscaling explanation is linked here

Things to consider before you start

The docker-machine project is pretty much dead. In it's current state only keysstone v2 is supported. When your provider supports only v3, which is the case for me, switch to ranchers fork

Use images with sufficient disk space. Ubuntu Cloud Images for example spawns with about 1.8GB of total disk capacity which leads to very strange errors. Download the image and resize it to at least about 5GB.

• Resize images using virt-resize
• If you encounter problems booting the image, grub might be broken. Try this

Preperations

• Login to your project and download your openrc. Credentials will be used by docker-machine to spawn minions.
• generate a new ssh keypair (without password). This key will be used by docker-machine to control it's minions ssh-keygen -f ~/.ssh/gitlab-runner.id_rsa
• Upload the public key to your OpenStack Project
• Get the latest version of docker-compose or rancher compose
• docker-machine
• rancher-machine
• Create a security group and rules
• allow at least port 22 (ssh) and 2376 (docker tcp socket)

Install gitlab-runner instance

This instance will be your runner with docker-machine. It doesn't need much resources. But it is the operator. Hence it will be arround all the time. 1 or 2 Cores 1GB or 2GB of RAM and about 10GB of persistend disk space should do the job. Choose whatever image that supports docker (Ubuntu in this example). If needed attach a Floating IP to this instance.

The hard way

Login to your instance and install docker and docker-compose. Download and unpack/install docker-machine/rancher-machine

• curl -L -o /tmp/rancher-machine.tar.gz https://github.com/rancher/machine/releases/download/v0.15.0-rancher32/rancher-machine-amd64.tar.gz
• tar -C /usr/local/bin/ -xvzf /tmp/rancher-machine.tar.gz
• cp /usr/local/bin/{rancher,docker}-machine
• chmod 755 /usr/local/bin/{rancher,docker}-machine

Create a gitlab-runner directory and put the private key that your created previusly in there and a template.toml for registration.

• mkdir /etc/gitlab-runner/
• chmod 400 /etc/gitlab-runner/id_rsa

template.toml

[[runners]]
limit = 8
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.machine]
    IdleCount = 2
    IdleTime = 3600
    OffPeakPeriods = [
      "* * 0-8,18-23 * * mon-fri *",
      "* * * * * sat,sun *"
    ]
    OffPeakTimezone = "Europe/Berlin"
    OffPeakIdleCount = 1
    OffPeakIdleTime = 900
    MaxBuilds = 32
    MachineName = "auto-scale-%s"
    MachineDriver = "openstack"
    MachineOptions = [
      "openstack-auth-url={{ openstack.clouds.0.auth.auth_url }}",
      "openstack-tenant-name={{ openstack.clouds.0.auth.project_name }}",
      "openstack-username={{ openstack.clouds.0.auth.username }}",
      "openstack-password={{ openstack.clouds.0.auth.password }}",
      "openstack-flavor-name=ECS.C1.4-8",
      "openstack-image-name=Ubuntu_18.04_openstack",
      "openstack-domain-name={{ openstack.clouds.0.auth.project_domain_name }}",
      "openstack-net-name=gitlab-runner-net",
      "openstack-sec-groups=gitlab-runner-sec-grp",
      "openstack-ssh-user=ubuntu",
      "openstack-private-key-file=/etc/gitlab-runner/id_rsa",
      "openstack-keypair-name=gitlab-runner",
      "engine-registry-mirror=http://{{ inventory_hostname }}:5000",
      "openstack-active-timeout=600"
    ]

Now you can register the runner to your gitlab instance.

docker run -it --rm -v /etc/gitlab-runner:/etc/gitlab-runner gitlab/gitlab-runner:latest register --non-interactive --name optimusprime --url XXXXXX --registration-token XXXXXX --executor "docker+machine" --template-config /etc/gitlab-runner/template.toml

You should see your new runner in gitlab. If this is the case start it as daemon

docker-compose.yml

---
version: '2'
services:
  gitlab-runner:
    image: gitlab/gitlab-runner:latest
    restart: always
    volumes:  
      - /usr/local/bin/docker-machine:/usr/local/bin/docker-machine
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/gitlab-runner/:/etc/gitlab-runner/
      - /root/.docker/:/root/.docker/

The ansible way

Deploying Instances is done by ansible as well. But this will not be covered here.

• ansible-galaxy install derJD.docker
• ansible-galaxy install derJD.journald

gitlab-runner.yml

---

- hosts: runner
  vars:
    docker_compose_service: true
    docker_machine_url: https://github.com/rancher/machine/releases/download/v0.15.0-rancher32/rancher-machine-amd64.tar.gz

  roles:
    - derJD.docker
    - derJD.journald

  tasks:
    - name: get clouds config
      delegate_to: localhost
      os_client_config:
        clouds: [ "{{ os_cloud }}" ]

    - name: get rancher-machine
      unarchive: remote_src=yes src={{ docker_machine_url }} dest=/usr/local/bin/ mode=0755
    - name: rename rancher-machine
      copy: remote_src=yes src=/usr/local/bin/rancher-machine dest=/usr/local/bin/docker-machine mode=0755

    - name: /etc/gitlab-runner/
      file: name=/etc/gitlab-runner/ state=directory
    - name: /etc/gitlab-runner/id_rsa
      copy: src=gitlab-runner.id_rsa.vault dest=/etc/gitlab-runner/id_rsa mode=0600
    - name: /etc/gitlab-runner/template.toml
      copy:
        dest: /etc/gitlab-runner/template.toml
        content: |
          [[runners]]
          limit = 8
            [runners.docker]
              tls_verify = false
              image = "docker:latest"
              privileged = true
              volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
              shm_size = 0
            [runners.machine]
              IdleCount = 2
              IdleTime = 3600
              OffPeakPeriods = [
                "* * 0-8,18-23 * * mon-fri *",
                "* * * * * sat,sun *"
              ]
              OffPeakTimezone = "Europe/Berlin"
              OffPeakIdleCount = 1
              OffPeakIdleTime = 900
              MaxBuilds = 32
              MachineName = "auto-scale-%s"
              MachineDriver = "openstack"
              MachineOptions = [
                "openstack-auth-url={{ openstack.clouds.0.auth.auth_url }}",
                "openstack-tenant-name={{ openstack.clouds.0.auth.project_name }}",
                "openstack-username={{ openstack.clouds.0.auth.username }}",
                "openstack-password={{ openstack.clouds.0.auth.password }}",
                "openstack-flavor-name=ECS.C1.4-8",
                "openstack-image-name=Ubuntu_18.04_openstack",
                "openstack-domain-name={{ openstack.clouds.0.auth.project_domain_name }}",
                "openstack-net-name=gitlab-runner-net",
                "openstack-sec-groups=gitlab-runner-sec-grp",
                "openstack-ssh-user=ubuntu",
                "openstack-private-key-file=/etc/gitlab-runner/id_rsa",
                "openstack-keypair-name=gitlab-runner",
                "engine-registry-mirror=http://{{ inventory_hostname }}:5000",
                "openstack-active-timeout=600"
              ]
      notify: restart docker-compose@runner

    - name: check for config.toml
      stat: path=/etc/gitlab-runner/config.toml
      register: conf
    - name: register gitlab-runner
      docker_container:
        name: register_gitlab-runner
        image: gitlab/gitlab-runner:latest
        auto_remove: true
        volumes:
          - /etc/gitlab-runner:/etc/gitlab-runner
        command: |
          register
          --non-interactive
          --name "{{ inventory_hostname }}"
          --url "https://{{ gitlab_instance }}/"
          --registration-token "{{ gitlab_token }}"
          --executor "docker+machine"
          --docker-privileged
          --docker-image "docker:latest"
          --tag-list "docker-machine,privileged,openstack"
          --run-untagged="true"
          --locked="false"
          --request-concurrency 32
          --access-level="not_protected"
          --template-config /etc/gitlab-runner/template.toml
      when: not conf.stat.exists
    - name: edit concurrent jobs
      lineinfile:
        path: /etc/gitlab-runner/config.toml
        regexp: "^concurrent = 1"
        line: "concurrent = 32"
      notify: restart docker-compose@runner
    - name: /etc/docker/compose/runner.yml
      copy:
        dest: /etc/docker/compose/runner.yml
        content: |
          ---
          version: '2'
          services:
            gitlab-runner:
              image: gitlab/gitlab-runner:latest
              restart: always
              ports:
                - 8093:8093
              volumes:
                - /usr/local/bin/docker-machine:/usr/local/bin/docker-machine
                - /var/run/docker.sock:/var/run/docker.sock
                - /etc/gitlab-runner/:/etc/gitlab-runner/
                - /root/.docker/:/root/.docker/
      notify: restart docker-compose@runner
    - name: docker-compose@runner
      service: name=docker-compose@runner state=started enabled=yes daemon_reload=yes

  handlers:
    - name: restart docker-compose@runner
      service: name=docker-compose@runner state=restarted

Sources

• Gitlab autoscale
• Gitlab runner in docker
• docker-machine openstack driver
• Others that do stuff like this
  • gitlab-runner autoscale by fuga
  • gitlab-runner autoscale by netways

---

Last update: March 22, 2021