Skip to content
Snippets Groups Projects

Finish documentation for setting up CI/CD

Merged Finish documentation for setting up CI/CD
enh-finish-documentation into main
Merged Matthew K Defenderfer requested to merge enh-finish-documentation into main
Compare
and
1 file
+ 141
− 17
Compare changes
  • Side-by-side
  • Inline
README.md
+ 141
− 17
# CI/CD Example Using Gitlab
<!-- markdownlint-disable MD046 -->
## CI/CD
CICD stands for continuous integration and continuous deployment and is a procedure that automates building, testing, distributing, and deploying code in a variety of formats. [RedHat](https://www.redhat.com/en/topics/devops/what-is-ci-cd) has a good summary of CI/CD and why it's useful. Here are a few examples of what CI/CD can automate:
@@ -11,6 +13,17 @@ CICD stands for continuous integration and continuous deployment and is a proced
Using UAB's self-hosted Gitlab instance and [Cloud](https://docs.rc.uab.edu/uab_cloud/) infrastructure, anyone can implement CI/CD to streamline their code development.
CI/CD Setup can be split into two separate sections, one section for setting up the code repository to use CI/CD and the other section for creating the Cloud VM that will actually run the CI/CD pipeline you create. While most of the steps for each section will not overlap with the other, there will be a couple of steps at the end needed to set the repo to use the VM.
This repository builds Python source code as both a `pip` installable package and a Docker/Singularity image as an example.
> [!warning]
> This guide uses Docker as the build engine to create Docker images. Docker has significant security concerns due to the fact it requires `root` privileges to run, and using the Docker-in-Docker service (necessary for Docker to build Docker images) requires you to essentially disable all built-in security.
>
> Docker was still chosen here due to being nearly ubiquitous as an image/container service. It is critical to make sure your runner VM is ONLY EVER USED FOR CI BUILDS. NEVER put any sensitive information or access keys on the VM, and make sure only trusted individuals have the ability to run CI/CD jobs for any repo.
>
> In the future, this guide will move to using a rootless build tool like [kaniko](https://docs.gitlab.com/ci/docker/using_kaniko/) which has full compatibility with Dockerfiles but does not have the same security concerns.
## How To Use This Repository
This repository is meant to be used as a guide and starting point and will walk through the steps necessary to implement CI/CD in your own projects. This repository can also be cloned or forked by anyone to have as a starting point for a new Gitlab repo so you would not have to start from scratch.
@@ -24,7 +37,7 @@ CI/CD can be implemented using most modern code repositories (ex. Github, Gitlab
- [Request a UAB Cloud account by contacting Research Computing Support](https://docs.rc.uab.edu/#how-to-contact-us)
- [Create a UAB Gitlab account](https://docs.rc.uab.edu/account_management/gitlab_account/#uab-gitlab-registration)
### Learn Account Cloud and Gitlab
### Learn About Cloud and Gitlab
This guide will assume basic understanding of how to use both services. If you do not have any experience with either Cloud or Gitlab, see the following guides on [our docs](https://docs.rc.uab.edu)
@@ -33,19 +46,13 @@ This guide will assume basic understanding of how to use both services. If you d
Gitlab is a separate implementation of remote git repositories than Github but functions identically at a basic level. In addition to the guides, our facilitation team would be happy to assist in learning how to use either git or Cloud during our [open office hours on Zoom](https://docs.rc.uab.edu/#how-to-contact-us)
## CI/CD Setup
Setup can be split into two separate sections, one section for setting up the code repository to use CI/CD and the other section for creating the Cloud VM that will actually run the CI/CD pipeline you create. While most of the steps for each section will not overlap with the other, there will be a couple of steps at the end needed to set the repo to use the VM.
This repository builds Python source code as both a `pip` installable package and a Docker/Singularity image as an example.
### Gitlab Repo Configuration
## Gitlab Repo Configuration
To implement CI/CD in a Gitlab repo, you will need to create a `.gitlab-ci.yml` file in the repo root as well as specify a runner to execute the CI. The `gitlab-ci` specifies all of the commands to be run to replicate a build and deploy process. These can range from fairly simple to very complicated depending on the pipeline.
#### .gitlab-ci.yml
### .gitlab-ci.yml
##### Default Settings
#### Default Settings
```yaml
default:
@@ -57,7 +64,7 @@ default:
- **image**: Specifies the Docker image to use for all jobs by default. Here, it uses `python:3.12-slim`, which is a lightweight Python environment.
- **rules**: Defines conditions for when the pipeline should run. In this case, it runs only if the branch is `main`.
##### Variables
#### Variables
```yaml
variables:
@@ -74,7 +81,7 @@ variables:
- **PIP_INDEX_URL**: The primary index URL for pip, using a token for authentication.
- **PIP_EXTRA_INDEX_URL**: An additional index URL for pip, pointing to the public PyPI.
##### Stages
#### Stages
```yaml
stages:
@@ -89,7 +96,7 @@ Defines the different stages of the CI pipeline:
- **test**: Running tests to ensure the code works as expected.
- **publish**: Publishing the package or Docker image.
##### Build Package Job
#### Build Package Job
```yaml
build_package:
@@ -111,7 +118,7 @@ build_package:
- **artifacts**: Specifies files to be saved after the job completes, in this case, the built package files in the `dist` directory.
- **cache**: Caches the pip downloads to speed up future builds.
##### Test Package Job
#### Test Package Job
```yaml
test_package:
@@ -125,7 +132,7 @@ test_package:
- **stage**: Specifies that this job is part of the `test` stage.
- **script**: Commands to install the package, install `pytest`, and run tests.
##### Publish to PyPI Job
#### Publish to PyPI Job
```yaml
publish_pip:
@@ -138,7 +145,7 @@ publish_pip:
- **stage**: Specifies that this job is part of the `publish` stage.
- **script**: Commands to install `twine` and upload the built package to the PyPI repository.
##### Build and Push Docker Image Job
#### Build and Push Docker Image Job
```yaml
build_and_push_docker_image:
@@ -158,7 +165,124 @@ build_and_push_docker_image:
- **stage**: Specifies that this job is part of the `publish` stage.
- **image**: Uses the `docker:latest` image for this job.
- **services**: Uses Docker-in-Docker (`docker:dind`) to enable Docker commands within the job.
- **services**: Uses Docker-in-Docker (`docker:dind`) to allow the runner's docker service to also run docker itself.
- **variables**: Defines Docker-related variables.
- **before_script**: Logs into the Docker registry.
- **script**: Commands to pull the latest Docker image, build a new image, and push it to the registry.
## Gitlab Runner Setup
Next, you will need some form of compute to actually run the commands specified in the `.gitlab-ci.yml` file. The easiest solution is to start a VM on UAB Cloud dedicated to running these build tasks. These instructions are for setting up a Gitlab Runner service that runs all builds through Docker.
### Creating the VM
> [!important]
> If you have not set up a Cloud VM before, follow the tutorial on [our docs](https://docs.rc.uab.edu/uab_cloud/tutorial/) up to the section on creating instances (VMs). Follow the instructions on how to create an instance using the VM settings given below.
> [!note]
> We use Docker as the build engine due to its flexibility in terms of dependency installation. As long as an image exists that contains a necessary dependency or the dependency can be installed into a container running a Linux OS, Docker can build it without needing to modify any of the software on the runner VM itself. This makes builds much easier to manage than building using the Runner's system packages although it does increase the compute requirements.
>
When setting up a VM, use the following settings:
- Source:
- Boot Source: Image (Create New Volume: Yes)
- Volume Size: 100 GB
- This can be altered based on what your CI/CD pipeline is doing exactly. If you are building large Docker images to store in the container registry, you will most likely want to increase the volume size beyond 100 GB since Docker will cache layers for both the host image and built images as well as volumes and containers. If you are only building small Python packages or compiled binaries, you could lower the volume size since Docker would only store layers for the host images. Be sure to clean the Docker cache regularly on the VM to maintain performance.
- Image: `auto-sync/ubuntu-noble-24.04-amd64-server-20250403-disk1.img`
- You can use any image that supports [Gitlab Runner](https://docs.gitlab.com/runner/install/), but the following instructions assume either an Ubuntu or Debian machine
- Flavor: `m1.large` or `m1.xlarge`
- Flavors of at least `m1.large` are suggested due to Docker's memory requirements. For building very large containers (~>8 GB), `m1.xlarge` may be necessary. If the flavor is too small, you will see out of memory errors in the job log on Gitlab.
All other settings can be set using the instructions in the tutorial above.
### Install Gitlab Runner
After the VM has been created, connect to it using your SSH client of choice to install Gitlab Runner and Docker.
Up-to-date instructions for installing Gitlab Runner can be found on [Gitlab's Documentation](https://docs.gitlab.com/runner/install/linux-repository/#install-gitlab-runner). You will only need to follow the first two steps if this VM did not have Gitlab Runner installed on it previously. The commands from those steps are replicated here for convenience but are subject to change at any time on Gitlab's site. Always use the original documentation if you run into installation issues.
> [!note]
> These instructions are for Ubuntu/Debian. If you chose an Alma/RHEL/CentOS image, switch to those instruction sets on Gitlab's site.
```bash
sudo apt-get update && sudo apt-get upgrade -y
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner
```
### Install Docker
Official instructions for installing Docker can be found on their [website](https://docs.docker.com/engine/install/ubuntu/#installation-methods). They include 4 methods for installing Docker of which I suggest using the [apt repository](https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository). The full instruction set can be found below, but the official instructions are subject to change at any time. Please use the official documentation if you run into any installation errors.
```bash
# Add Docker's GPG Key to access the repository
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
# Install Docker
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```
#### Post-Installation Convenience Setup
After docker is installed, it can be somewhat cumbersome to use as-is since it requires root permissions to run. This involves typing sudo before every `docker` command and makes using it with gitlab-runner difficult/nigh impossible. Instead, you can follow the [post-installation instructions](https://docs.docker.com/engine/install/linux-postinstall/) to make life easier. This involves giving both the `gitlab-runner` and `ubuntu` users root permission to run docker by default.
As before, any instructions listed here may not be up-to-date with the official documentation. Refer there initially for any issues. The current post-install commands can be seen below.
```bash
# Adds the current user and the gitlab-runner accounts to the docker group
sudo usermod -aG docker $USER
sudo usermod -aG docker gitlab-runner
# Sets Docker to run automatically on startup
sudo systemctl enable docker.service
sudo systemctl enable containerd.service
```
### Registering the Runner
The next step is to register the runner so that it can accept CI jobs from Gitlab. This is done in two steps, first retrieving an authentication token from the runner, then associating that token with the runner itself. Follow the official instructions below for these steps.
1. [Create an authentication token](https://docs.gitlab.com/ci/runners/runners_scope/#create-a-project-runner-with-a-runner-authentication-token). See below for notes for specific steps
1. Step 4 `New Project Runner` menu:
1. Select the `Run untagged jobs` checkbox unless you plan to include tags in your CI/CD.
2. You can add a runner description which will show up with the runner entry in the Gitlab CI interface
2. Step 5: Choose `Linux`. Look down the page under `Step 1` for a runner token. A token will start with `glrt-` followed by a string of alphanumeric characters including `-` and `_`. Copy this token to use in the next step
2. Register the runner on the VM. Run the following command to register a runner, replacing `REGISTRATION_TOKEN` with the token copied from the previous step
```bash
sudo gitlab-runner register -n \
--url "https://gitlab.rc.uab.edu/" \
--registration-token REGISTRATION_TOKEN \
--executor docker \
--docker-image "docker:24.0.5" \
--docker-privileged \
--docker-volumes "/certs/client"
```
This command sets some default runner parameters including the executor, the docker image to use, and telling docker to run in privileged mode, a requirement for a Docker container to build other Docker images. Feel free to change the docker image version, but it's not suggested to use `docker:latest` as that version does change over time without notice, and you may not be able to replicate certain builds if they use different versions of Docker.
If everything finished correctly, the runner should be automatically assigned to the project. You can see this by going to `Settings > CI/CD > Runners` from the main repository page.
> [!note]
> If you chose to lock the runner to currently assigned projects, you will not see it listed in the available runners list for any other projects. You can create a new registration for the same runner on other projects you may want to assign it to. A single runner can have multiple registrations, and those registrations can be any combination of project, group, or instance.
>
> If you do not lock the runner, it will be available to other projects in the group. If any data used in the pipeline are sensitive, be sure to lock the runner during creation.
### Editing a Runner Configuration
If after you've registered the runner, there are settings you'd like to change about it, you can edit the `/etc/gitlab-runner/config.toml` file on the VM. You will need `sudo` permissions to make any changes. See [Gitlab's documentation](https://docs.gitlab.com/runner/configuration/advanced-configuration/) for a full breakdown of different options you can use in the configuration file.
## Running a CI/CD Pipeline
With the example CI/CD pipeline, builds will automatically start when any commits are pushed to the `main` branch or when a pipeline is started from the web interface. You can start a pipeline manually by going to `Build > Pipeline > New Pipeline`. You can also find the logs for current and past jobs by clicking on the different pipelines.
You can also have the pipeline run in response to various conditions and actions. For instance, you can run a pipeline auntoamtically any time you push a tag to the repository. A tag is a convenient mechanism for permanently marking certain important commits, such as the last commit for a version release. That tag can be referenced in the CI/CD pipeline and a package and/or image automatically created for it. See the [Gitlab rules](https://docs.gitlab.com/ci/jobs/job_rules/) section in their documentation for more examples of specifying when a pipeline should run.