CI/CD Example Using Gitlab

CI/CD

CICD stands for continuous integration and continuous deployment and is a procedure that automates building, testing, distributing, and deploying code in a variety of formats. RedHat has a good summary of CI/CD and why it's useful. Here are a few examples of what CI/CD can automate:

1. Running full test suites on a code base
2. Building source code into a distributable package or compiling it into an executable
3. Building an image containing source code to be used in Docker/Singularity containers
4. Any repetitive tasks that must be done after modifications to source code in order to make that code usable by others

Using UAB's self-hosted Gitlab instance and Cloud infrastructure, anyone can implement CI/CD to streamline their code development.

CI/CD Setup can be split into two separate sections, one section for setting up the code repository to use CI/CD and the other section for creating the Cloud VM that will actually run the CI/CD pipeline you create. While most of the steps for each section will not overlap with the other, there will be a couple of steps at the end needed to set the repo to use the VM.

This repository builds Python source code as both a pip installable package and a Docker/Singularity image as an example.

[!warning] This guide uses Docker as the build engine to create Docker images. Docker has significant security concerns due to the fact it requires root privileges to run, and using the Docker-in-Docker service (necessary for Docker to build Docker images) requires you to essentially disable all built-in security.

Docker was still chosen here due to being nearly ubiquitous as an image/container service. It is critical to make sure your runner VM is ONLY EVER USED FOR CI BUILDS. NEVER put any sensitive information or access keys on the VM, and make sure only trusted individuals have the ability to run CI/CD jobs for any repo.

In the future, this guide will move to using a rootless build tool like kaniko which has full compatibility with Dockerfiles but does not have the same security concerns.

How To Use This Repository

This repository is meant to be used as a guide and starting point and will walk through the steps necessary to implement CI/CD in your own projects. This repository can also be cloned or forked by anyone to have as a starting point for a new Gitlab repo so you would not have to start from scratch.

Prerequisites

CI/CD can be implemented using most modern code repositories (ex. Github, Gitlab) and cloud services (ex. AWS, GCP, Azure) so the ideas here are transferrible elsewhere. However, the full guide will be written to use UAB's Gitlab and Cloud platforms specifically. To implement CI/CD as described here, you will need accounts for both Cloud and Gitlab.

Create Your Accounts

• Request a UAB Cloud account by contacting Research Computing Support
• Create a UAB Gitlab account

Learn About Cloud and Gitlab

This guide will assume basic understanding of how to use both services. If you do not have any experience with either Cloud or Gitlab, see the following guides on our docs

• Cloud
• Git

Gitlab is a separate implementation of remote git repositories than Github but functions identically at a basic level. In addition to the guides, our facilitation team would be happy to assist in learning how to use either git or Cloud during our open office hours on Zoom

Gitlab Repo Configuration

To implement CI/CD in a Gitlab repo, you will need to create a .gitlab-ci.yml file in the repo root as well as specify a runner to execute the CI. The gitlab-ci specifies all of the commands to be run to replicate a build and deploy process. These can range from fairly simple to very complicated depending on the pipeline.

.gitlab-ci.yml

Default Settings

default:
  image: python:3.12-slim
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

• image: Specifies the Docker image to use for all jobs by default. Here, it uses python:3.12-slim, which is a lightweight Python environment.
• rules: Defines conditions for when the pipeline should run. In this case, it runs only if the branch is main.

Variables

variables:
  PACKAGE_NAME: "pak"
  REPO_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi"
  PIP_CACHE_DIR: "${CI_PROJECT_DIR}/.cache/pip"
  PIP_INDEX_URL: "https://__token__:${CI_JOB_TOKEN}@gitlab.com/api/v4/projects/${CI_PROJECT_ID}/packages/pypi/simple"
  PIP_EXTRA_INDEX_URL: "https://pypi.org/simple"

• PACKAGE_NAME: The name of the package being built.
• REPO_URL: The URL for the PyPI repository in GitLab, where the package will be uploaded.
• PIP_CACHE_DIR: Directory to cache pip downloads, speeding up subsequent installs.
• PIP_INDEX_URL: The primary index URL for pip, using a token for authentication.
• PIP_EXTRA_INDEX_URL: An additional index URL for pip, pointing to the public PyPI.

Stages

stages:
  - build
  - test
  - publish

Defines the different stages of the CI pipeline:

• build: Compiling or packaging the code.
• test: Running tests to ensure the code works as expected.
• publish: Publishing the package or Docker image.

Build Package Job

build_package:
  stage: build
  script:
    - pip install --upgrade pip
    - pip install setuptools wheel build
    - python -m build
  artifacts:
    paths:
      - dist/*
  cache:
    paths:
      - .cache/pip

• stage: Specifies that this job is part of the build stage.
• script: Commands to upgrade pip, install necessary tools, and build the package.
• artifacts: Specifies files to be saved after the job completes, in this case, the built package files in the dist directory.
• cache: Caches the pip downloads to speed up future builds.

Test Package Job

test_package:
  stage: test
  script:
    - pip install .
    - pip install pytest
    - pytest

• stage: Specifies that this job is part of the test stage.
• script: Commands to install the package, install pytest, and run tests.

Publish to PyPI Job

publish_pip:
  stage: publish
  script:
    - pip install twine
    - twine upload -u gitlab-ci-token -p ${CI_JOB_TOKEN} --repository-url ${REPO_URL} dist/*

• stage: Specifies that this job is part of the publish stage.
• script: Commands to install twine and upload the built package to the PyPI repository.

Build and Push Docker Image Job

build_and_push_docker_image:
  stage: publish
  image: docker:latest
  services:
    - docker:dind
  variables:
    DOCKER_IMAGE_NAME: ${CI_REGISTRY_IMAGE}/${PACKAGE_NAME}
  before_script:
    - echo "${CI_REGISTRY_PASSWORD}" | docker login -u "${CI_REGISTRY_USER}" --password-stdin ${CI_REGISTRY}
  script:
    - docker pull ${DOCKER_IMAGE_NAME}:latest || true
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from ${DOCKER_IMAGE_NAME}:latest -t ${DOCKER_IMAGE_NAME}:latest .
    - docker push ${DOCKER_IMAGE_NAME}:latest

• stage: Specifies that this job is part of the publish stage.
• image: Uses the docker:latest image for this job.
• services: Uses Docker-in-Docker (docker:dind) to allow the runner's docker service to also run docker itself.
• variables: Defines Docker-related variables.
• before_script: Logs into the Docker registry.
• script: Commands to pull the latest Docker image, build a new image, and push it to the registry.

Gitlab Runner Setup

Next, you will need some form of compute to actually run the commands specified in the .gitlab-ci.yml file. The easiest solution is to start a VM on UAB Cloud dedicated to running these build tasks. These instructions are for setting up a Gitlab Runner service that runs all builds through Docker.

Creating the VM

[!important] If you have not set up a Cloud VM before, follow the tutorial on our docs up to the section on creating instances (VMs). Follow the instructions on how to create an instance using the VM settings given below.

[!note] We use Docker as the build engine due to its flexibility in terms of dependency installation. As long as an image exists that contains a necessary dependency or the dependency can be installed into a container running a Linux OS, Docker can build it without needing to modify any of the software on the runner VM itself. This makes builds much easier to manage than building using the Runner's system packages although it does increase the compute requirements.

When setting up a VM, use the following settings:

• Source:
  • Boot Source: Image (Create New Volume: Yes)
  • Volume Size: 100 GB
    • This can be altered based on what your CI/CD pipeline is doing exactly. If you are building large Docker images to store in the container registry, you will most likely want to increase the volume size beyond 100 GB since Docker will cache layers for both the host image and built images as well as volumes and containers. If you are only building small Python packages or compiled binaries, you could lower the volume size since Docker would only store layers for the host images. Be sure to clean the Docker cache regularly on the VM to maintain performance.
  • Image: auto-sync/ubuntu-noble-24.04-amd64-server-20250403-disk1.img
    • You can use any image that supports Gitlab Runner, but the following instructions assume either an Ubuntu or Debian machine
• Flavor: m1.large or m1.xlarge
  • Flavors of at least m1.large are suggested due to Docker's memory requirements. For building very large containers (~>8 GB), m1.xlarge may be necessary. If the flavor is too small, you will see out of memory errors in the job log on Gitlab.

All other settings can be set using the instructions in the tutorial above.

Install Gitlab Runner

After the VM has been created, connect to it using your SSH client of choice to install Gitlab Runner and Docker.

Up-to-date instructions for installing Gitlab Runner can be found on Gitlab's Documentation. You will only need to follow the first two steps if this VM did not have Gitlab Runner installed on it previously. The commands from those steps are replicated here for convenience but are subject to change at any time on Gitlab's site. Always use the original documentation if you run into installation issues.

[!note] These instructions are for Ubuntu/Debian. If you chose an Alma/RHEL/CentOS image, switch to those instruction sets on Gitlab's site.

sudo apt-get update && sudo apt-get upgrade -y
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner

Install Docker

Official instructions for installing Docker can be found on their website. They include 4 methods for installing Docker of which I suggest using the apt repository. The full instruction set can be found below, but the official instructions are subject to change at any time. Please use the official documentation if you run into any installation errors.

# Add Docker's GPG Key to access the repository
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

# Install Docker
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Post-Installation Convenience Setup

After docker is installed, it can be somewhat cumbersome to use as-is since it requires root permissions to run. This involves typing sudo before every docker command and makes using it with gitlab-runner difficult/nigh impossible. Instead, you can follow the post-installation instructions to make life easier. This involves giving both the gitlab-runner and ubuntu users root permission to run docker by default.

As before, any instructions listed here may not be up-to-date with the official documentation. Refer there initially for any issues. The current post-install commands can be seen below.

# Adds the current user and the gitlab-runner accounts to the docker group
sudo usermod -aG docker $USER
sudo usermod -aG docker gitlab-runner

# Sets Docker to run automatically on startup
sudo systemctl enable docker.service
sudo systemctl enable containerd.service

Registering the Runner

The next step is to register the runner so that it can accept CI jobs from Gitlab. This is done in two steps, first retrieving an authentication token from the runner, then associating that token with the runner itself. Follow the official instructions below for these steps.

1. Create an authentication token. See below for notes for specific steps
   1. Step 4 New Project Runner menu:
      1. Select the Run untagged jobs checkbox unless you plan to include tags in your CI/CD.
      2. You can add a runner description which will show up with the runner entry in the Gitlab CI interface
   2. Step 5: Choose Linux. Look down the page under Step 1 for a runner token. A token will start with glrt- followed by a string of alphanumeric characters including - and _. Copy this token to use in the next step
2. Register the runner on the VM. Run the following command to register a runner, replacing REGISTRATION_TOKEN with the token copied from the previous step

sudo gitlab-runner register -n \
    --url "https://gitlab.rc.uab.edu/" \
    --registration-token REGISTRATION_TOKEN \
    --executor docker \
    --docker-image "docker:24.0.5" \
    --docker-privileged \
    --docker-volumes "/certs/client"

This command sets some default runner parameters including the executor, the docker image to use, and telling docker to run in privileged mode, a requirement for a Docker container to build other Docker images. Feel free to change the docker image version, but it's not suggested to use docker:latest as that version does change over time without notice, and you may not be able to replicate certain builds if they use different versions of Docker.

If everything finished correctly, the runner should be automatically assigned to the project. You can see this by going to Settings > CI/CD > Runners from the main repository page.

[!note] If you chose to lock the runner to currently assigned projects, you will not see it listed in the available runners list for any other projects. You can create a new registration for the same runner on other projects you may want to assign it to. A single runner can have multiple registrations, and those registrations can be any combination of project, group, or instance.

If you do not lock the runner, it will be available to other projects in the group. If any data used in the pipeline are sensitive, be sure to lock the runner during creation.

Editing a Runner Configuration

If after you've registered the runner, there are settings you'd like to change about it, you can edit the /etc/gitlab-runner/config.toml file on the VM. You will need sudo permissions to make any changes. See Gitlab's documentation for a full breakdown of different options you can use in the configuration file.

Running a CI/CD Pipeline

With the example CI/CD pipeline, builds will automatically start when any commits are pushed to the main branch or when a pipeline is started from the web interface. You can start a pipeline manually by going to Build > Pipeline > New Pipeline. You can also find the logs for current and past jobs by clicking on the different pipelines.

You can also have the pipeline run in response to various conditions and actions. For instance, you can run a pipeline auntoamtically any time you push a tag to the repository. A tag is a convenient mechanism for permanently marking certain important commits, such as the last commit for a version release. That tag can be referenced in the CI/CD pipeline and a package and/or image automatically created for it. See the Gitlab rules section in their documentation for more examples of specifying when a pipeline should run.