Skip to content
Snippets Groups Projects
getting-started.md 7.6 KiB
Newer Older
# Getting started

## Prerequisites
1. Get yourself an account on our openstack cloud - cloud.rc, by following the instructions from the following link https://docs.rc.uab.edu/uab_cloud/
2. Make sure you have a project with owner or maintainer roles

## HPC factory overview
Why we need it ?
- This project provides build and deploy harness for your applications
- This repo provides a way to define the cluster you want to deploy your application against. 

So you built your application and deployed it. Now what ? 
- You would need to test the science gateway against a cluster, so set up an adhoc cluster.
Refer to the documentation for our [Cluster on Demand](https://gitlab.rc.uab.edu/rc/cod-heat-stack)

> Note: Make sure you have deployed a working cluster before you move on to next steps. 
-------

## Define a packer template

- Follow the example we have in the hpc-factory for defining a packer template. This example template will build an image with commands defined in the provisioner.

TODO - Link the README instructions to install packer etc.

## Setup a CI/CD pipeline

- Make sure you have a runner available to run your jobs. 
	- Go to  **Settings > CI/CD**  and expand  **Runners** 
	- If you do not have an active runner, [create a runner](Setup-a-runner)
- Create a .gitlab-ci.yml file at the root of your repository.
- Define a pipeline in the yaml file created above [following the template](https://gitlab.rc.uab.edu/hpc-factory)
	-   Pipelines are made up of jobs and stages.
	-   Stages define the order of execution. Typical stages might be  `build`,  `test`, and  `deploy`.
	-   Jobs specify the tasks to be performed in each stage. For example, a job can compile or test code.
	-   Pipelines can be triggered by various events, like merges, commits, web or can be on schedule. 
- Define your cicd variables
	-	Two types of variables exist: custom variables and predefined.
	-   Custom variables are user-defined. Create and manage them in the GitLab UI(settings -> cicd -> Variables), API, or in .gitlab-ci.yml files.
	-   Predefined variables are automatically set by GitLab and provide information about the current job, pipeline, and environment.
- Setup a scheduled pipeline
	- Build -> pipeline schedules
	- Set pipeline vars to override user-defined cicd vars if needed
- Run the scheduled pipeline.


## Setup a runner

A runner is an agent in the GitLab Runner application that runs jobs in a GitLab CI/CD pipeline. Jobs are defined in the `.gitlab-ci.yml` file and assigned to available runners. 

GitLab has three types of runners:

-   Shared: Available to all groups and projects in a GitLab instance.
-   Group: Available to all projects and subgroups in a group.
-   Project: Associated with specific projects. Typically, project runners are used by one project at a time.

### Step 1: Create an access token
You will first need to create an access token. A  [personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html)  for an administrator account will allow you to create runners at the instance, group, and project levels.

If you only need to create a group or project runner, then it is best to use a group access token or project access token, respectively. For a group or project, navigate to  `Settings -> Access Tokens`  and create a token. You must specify a name, the token expiration date, role, and scope. For the role, select  `Owner`; for the scopes, select  `create_runner`

> Warning:  This access token is only visible once in the UI. Copy the access token to a secure location

### Step 2: Create a runner config using the access token
Now that you have a scoped token it is time to create a runner config using this token.
You only need to run one of these commands based on your requirement and scope of the token you generated.

``` sh
# Group runner config
curl -sX POST https://<GITLAB_URL>/api/v4/user/runners \
--data runner_type=<RUNNER_TYPE> \
--data  "group_id=<target_group_id>" \
--data  "description=gitlab-ci-runner" \
--data  "tag_list=<your comma-separated tags>" \
--header  "PRIVATE-TOKEN: <your_access_token>"
```

``` sh
# Project runner config
curl -sX POST https://<GITLAB_URL>/api/v4/user/runners \
--data runner_type=<RUNNER_TYPE> \
--data  "project_id=<target_project_id>" \
--data  "description=gitlab-ci-runner" \
--data  "tag_list=<your comma-separated tags>" \
--header  "PRIVATE-TOKEN: <your_access_token>"
```

``` sh
# Shared runner config
curl -sX POST https://<GITLAB_URL>/api/v4/user/runners \
--data runner_type=<RUNNER_TYPE> \
--data  "group_id=<target_group_or_project_id>" \
--data  "description=software-eng-docker-builds-runner" \
--data  "tag_list=<your comma-separated tags>" \
--header  "PRIVATE-TOKEN: <your_access_token>"
```

The newly created runner configuration is visible in the GitLab UI. As the actual runner has not yet been configured, the status displayed is  `Never contacted`.

The API will return a message with the following fields:  `id`,  `token`, and  `token_expires_at`. You must save the value for the  `token`  as it will only be displayed once.

> Note: This auth token can be reused to register multiple runners and they all will be grouped together. Each runner in the group is uniquely identified by their `system_id`.

### Step3: Create a runner VM
```sh
# Set runner name and floating IP
export RUNNER_NAME="ci-runner-img-build"
export FLOATING_IP=<your-floating-ip>

# Create the server and capture the instance ID
openstack server create --flavor <your-flavor> \
  --image <your-image> \
  --key-name <your-openstack-keypair> \
  --network <your-build-network> \
  --security-group default \
  --security-group <your-security-group> \
  $RUNNER_NAME

# Get the instance ID of the created server
RUNNER_INSTANCE_ID=$(openstack server list --name $RUNNER_NAME -f value -c ID)

# Add the floating IP to the server
openstack server add floating ip $RUNNER_INSTANCE_ID $FLOATING_IP
```

### Step4: Install and register runner
The following steps need to run on the runner VM.
`ssh ubuntu@$FLOATING_IP`
``` sh
# Set variables here - replace these with your actual values
GITLAB_INSTANCE_URL="https://gitlab.rc.uab.edu/" 

# Update and install necessary packages
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release

# Create Docker GPG key and repo
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Install GitLab Runner binary
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
sudo chmod +x /usr/local/bin/gitlab-runner

# Create a GitLab Runner user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Install GitLab Runner as a service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

# Register GitLab Runner
sudo gitlab-runner register --non-interactive \
--url "$GITLAB_URL" \
--registration-token "$RUNNER_TOKEN"\
--executor "docker" \
--docker-image "alpine:latest" \
--description "OpenStack Runner" \
--tag-list "build, deploy"

# Verify Docker is installed
sudo docker run hello-world
```
Ensure the runner is active and connected. 
`Settings -> CICD -> Runners -> Assigned Runners`