From 51ba4b8d36d1a2567b6ea8b5b93c6a954b21a56b Mon Sep 17 00:00:00 2001 From: Eesaan Atluri <atlurie@uab.edu> Date: Wed, 6 Nov 2024 19:14:57 -0600 Subject: [PATCH] feat: Add getting started docs --- docs/getting-started.md | 171 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 171 insertions(+) create mode 100644 docs/getting-started.md diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 0000000..4db69ca --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,171 @@ +# Getting started + +## Prerequisites +1. Get yourself an account on our openstack cloud - cloud.rc, by following the instructions from the following link https://docs.rc.uab.edu/uab_cloud/ +2. Make sure you have a project with owner or maintainer roles + +## HPC factory overview +Why we need it ? +- This project provides build and deploy harness for your applications +- This repo provides a way to define the cluster you want to deploy your application against. + +So you built your application and deployed it. Now what ? +- You would need to test the science gateway against a cluster, so set up an adhoc cluster. +Refer to the documentation for our [Cluster on Demand](https://gitlab.rc.uab.edu/rc/cod-heat-stack) + +> Note: Make sure you have deployed a working cluster before you move on to next steps. +------- + +## Define a packer template + +- Follow the example we have in the hpc-factory for defining a packer template. This example template will build an image with commands defined in the provisioner. + +TODO - Link the README instructions to install packer etc. + +## Setup a CI/CD pipeline + +- Make sure you have a runner available to run your jobs. + - Go to **Settings > CI/CD** and expand **Runners** + - If you do not have an active runner, [create a runner](Setup-a-runner) +- Create a .gitlab-ci.yml file at the root of your repository. +- Define a pipeline in the yaml file created above [following the template](https://gitlab.rc.uab.edu/hpc-factory) + - Pipelines are made up of jobs and stages. + - Stages define the order of execution. Typical stages might be `build`, `test`, and `deploy`. + - Jobs specify the tasks to be performed in each stage. For example, a job can compile or test code. + - Pipelines can be triggered by various events, like merges, commits, web or can be on schedule. +- Define your cicd variables + - Two types of variables exist: custom variables and predefined. + - Custom variables are user-defined. Create and manage them in the GitLab UI(settings -> cicd -> Variables), API, or in .gitlab-ci.yml files. + - Predefined variables are automatically set by GitLab and provide information about the current job, pipeline, and environment. +- Setup a scheduled pipeline + - Build -> pipeline schedules + - Set pipeline vars to override user-defined cicd vars if needed +- Run the scheduled pipeline. + + +## Setup a runner + +A runner is an agent in the GitLab Runner application that runs jobs in a GitLab CI/CD pipeline. Jobs are defined in the `.gitlab-ci.yml` file and assigned to available runners. + +GitLab has three types of runners: + +- Shared: Available to all groups and projects in a GitLab instance. +- Group: Available to all projects and subgroups in a group. +- Project: Associated with specific projects. Typically, project runners are used by one project at a time. + +### Step 1: Create an access token +You will first need to create an access token. A [personal access token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) for an administrator account will allow you to create runners at the instance, group, and project levels. + +If you only need to create a group or project runner, then it is best to use a group access token or project access token, respectively. For a group or project, navigate to `Settings -> Access Tokens` and create a token. You must specify a name, the token expiration date, role, and scope. For the role, select `Owner`; for the scopes, select `create_runner` + +> Warning: This access token is only visible once in the UI. Copy the access token to a secure location + +### Step 2: Create a runner config using the access token +Now that you have a scoped token it is time to create a runner config using this token. +You only need to run one of these commands based on your requirement and scope of the token you generated. + +``` sh +# Group runner config +curl -sX POST https://<GITLAB_URL>/api/v4/user/runners \ +--data runner_type=<RUNNER_TYPE> \ +--data "group_id=<target_group_id>" \ +--data "description=gitlab-ci-runner" \ +--data "tag_list=<your comma-separated tags>" \ +--header "PRIVATE-TOKEN: <your_access_token>" +``` + +``` sh +# Project runner config +curl -sX POST https://<GITLAB_URL>/api/v4/user/runners \ +--data runner_type=<RUNNER_TYPE> \ +--data "project_id=<target_project_id>" \ +--data "description=gitlab-ci-runner" \ +--data "tag_list=<your comma-separated tags>" \ +--header "PRIVATE-TOKEN: <your_access_token>" +``` + +``` sh +# Shared runner config +curl -sX POST https://<GITLAB_URL>/api/v4/user/runners \ +--data runner_type=<RUNNER_TYPE> \ +--data "group_id=<target_group_or_project_id>" \ +--data "description=software-eng-docker-builds-runner" \ +--data "tag_list=<your comma-separated tags>" \ +--header "PRIVATE-TOKEN: <your_access_token>" +``` + +The newly created runner configuration is visible in the GitLab UI. As the actual runner has not yet been configured, the status displayed is `Never contacted`. + +The API will return a message with the following fields: `id`, `token`, and `token_expires_at`. You must save the value for the `token` as it will only be displayed once. + +> Note: This auth token can be reused to register multiple runners and they all will be grouped together. Each runner in the group is uniquely identified by their `system_id`. + +### Step3: Create a runner VM +```sh +# Set runner name and floating IP +export RUNNER_NAME="ci-runner-img-build" +export FLOATING_IP=<your-floating-ip> + +# Create the server and capture the instance ID +openstack server create --flavor <your-flavor> \ + --image <your-image> \ + --key-name <your-openstack-keypair> \ + --network <your-build-network> \ + --security-group default \ + --security-group <your-security-group> \ + $RUNNER_NAME + +# Get the instance ID of the created server +RUNNER_INSTANCE_ID=$(openstack server list --name $RUNNER_NAME -f value -c ID) + +# Add the floating IP to the server +openstack server add floating ip $RUNNER_INSTANCE_ID $FLOATING_IP +``` + +### Step4: Install and register runner +The following steps need to run on the runner VM. +`ssh ubuntu@$FLOATING_IP` +``` sh +# Set variables here - replace these with your actual values +GITLAB_INSTANCE_URL="https://gitlab.rc.uab.edu/" + +# Update and install necessary packages +sudo apt-get update +sudo apt-get install -y ca-certificates curl gnupg lsb-release + +# Create Docker GPG key and repo +sudo install -m 0755 -d /etc/apt/keyrings +curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg +sudo chmod a+r /etc/apt/keyrings/docker.gpg +echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \ + $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null + +# Install Docker Engine +sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin + +# Install GitLab Runner binary +sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64 +sudo chmod +x /usr/local/bin/gitlab-runner + +# Create a GitLab Runner user +sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash + +# Install GitLab Runner as a service +sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner +sudo gitlab-runner start + +# Register GitLab Runner +sudo gitlab-runner register --non-interactive \ +--url "$GITLAB_URL" \ +--registration-token "$RUNNER_TOKEN"\ +--executor "docker" \ +--docker-image "alpine:latest" \ +--description "OpenStack Runner" \ +--tag-list "build, deploy" + +# Verify Docker is installed +sudo docker run hello-world +``` +Ensure the runner is active and connected. +`Settings -> CICD -> Runners -> Assigned Runners` -- GitLab