Hacking CI/CD Pipelines: Part 1 Overview

It was the summer of 2016, I had just arrived on a contract joining a delivery team which was developing and maintaining a new platform with an new technology called Kubernetes. I had never heard of it but wayback in 2013 I had looked at Docker when it was open sourced and found the isolation was not enough for some of the contracts I was working on. So I was surprised by the decision to base the platform on this technology but I was reassured things have changed. Part of that change was the emergence of DevOps and CI/CD infrastructure. Again I was fortunate to be exposed to git source control earlier in my career but I was new to CI servers like Jenkins, artefact repositories like Artifactory and automatic deployments based on test criteria being passed.

One afternoon I sat with a DevOps engineer as they explained to me how it worked. I had so many questions, naturally focussed in security such as:

  • How are users authenticated to Jenkins?
  • How does Jenkins do authorization?
  • Where are build logs stored?
  • Does the pipeline fail if there are any security issues?
  • How are credentials stored and managed for the distribution of built artefacts?

Based on the answers I received, I was shocked.

So that’s how the pipeline works. You okay?

There were only basic security controls in place with very little auditing of builds and what dependencies went into the builds. This was before it was standard practice to perform vulnerability scanning in the pipeline and so began my obsession with CI/CD security. During this time I was greeted with confusion as the majority of engineers thought I was talking about DevSecOps but I had to explain it as security of the CI/CD infrastructure.

Today the security of CI/CD pipelines is far more understood but I still see a surprising amount of companies that don’t apply the same level of controls as production. I am not sure if this is due to lack of time or budget or whether they simply don’t see what I see, that the tools you use to develop, build and deploy into production, are production.

So for my next series of posts, I’ll be sharing one of my passions in security, hacking CI/CD pipelines. Before we get started, let’s cover the basics for CI/CD security risks.

Note: I will not be covering the fundamentals of CI/CD. If you want to know more I highly recommend the GitLab documentation.

CI/CD security risks

A quick search for “CI/CD security risks” or “securing your CI/CD pipeline” you’ll get a mixture of results based on what I previously discussed as industry is primarily focussed on putting security into the CI/CD pipeline. So to save you time, OWASP has a very good top ten CI/CD security risks. If you’ve never seen the list, I highly recommend reviewing each risk to understand how an adversary could use them to compromise the pipeline.

OWASP Top Ten CI/CD Security Risks

For the purpose of this blog, I’m going to concentrate on two risks. The first, is CICD-SEC-4: Poisoned Pipeline Execution. This involves an adversary who has indirect access to the build environment, modifies the pipeline definition file or build dependencies to insert malicious code. Modern CI/CD pipelines are defined as code, meaning stored in source control (e.g. git) and version controlled. Given the power of the pipeline, it surprises me when I see businesses not protecting it. So the first part of hacking the pipeline will be placing malicious code into the pipeline. What type of malicious code? This is where the second risk comes in.

The second risk I want to focus on is CICD-SEC-6: Insufficient Credential Hygiene. This involves an adversary getting hold of secrets or tokens used in the pipeline to access other systems. CI pipelines, specifically the build process in a pipeline, at minimum requires an input and output.

The input is source code, whether that is user defined code or inherited, this is normally stored in git and unless the repository is public, a token is required for access. The output is dependent on the deployment strategy. At worst, build items are automatically deployed into production or a more rigorous approach is storing the build artefacts in a repository and are promoted based on the tests run against them. Whatever the approach, credentials are required to access these services, credentials that can be leaked.

Based on these risks I’ll demonstrate the following in a CI/CD pipeline:

  1. How to execute malicious code in the pipeline, specifically a reverse shell
  2. How to grab credentials from the pipeline

Where possible, I’ll try to stretch these objectives by obfuscating the execution of malicious code or exfiltrating the credentials beyond dumping it into the build log. If one of my crazy ideas doesn’t work, I’ll be sure to include it in the article and why it didn’t work.

I’ve defined how I want to hack a CI/CD pipeline, but which ones to hack?

There are many CI/CD pipelines to choose from and I’ve security reviewed a good number. I’ve decided to narrow it down to 5 different tools based on a report on the state of Kubernetes jobs last year.

Top CI/CD Tools by Job Posts

This provides a perspective on the most popular CI/CD pipelines based on job postings but I have other reasons for hacking these pipelines. I consider Jenkins a legacy pipeline tool, adopted early on by organisations when there wasn’t a better option, but they have never migrated away as it “just works”. GitLab CI is outstanding and I’ve seen it used by many businesses who require a private code repository and build pipeline. GitHub Actions is the fastest rising CI platform I’ve seen, as it is close to the developers repository and hugely flexible to use (e.g. there are so many publicly available actions to use or abuse 😉). CircleCI is hugely popular and I’ve never used or reviewed it so it’ll be fun to see how it all works. Lastly Azure DevOps is often used by organisations that have an enterprise Azure licence and use it as it is included as part of the package.

So we have the list of CI/CD pipeline tools to hack, know how we want to hack them but we need something to send through the pipeline.

Building an App

Let’s create a simple demo application we can build, test and containerize in our pipeline. I’ve chosen a simple go application which serves an API with a single health check endpoint. You can find the code here: https://github.com/wakeward/demo-api-app/. The code is simple, it uses gin-gonic/gin to setup a route for healthcheck endpoint.

package controllers

import (
	docs "github.com/wakeward/demo-api-app/docs"

	"github.com/gin-gonic/gin"
	swaggerFiles "github.com/swaggo/files"
	ginSwagger "github.com/swaggo/gin-swagger"
)

// @title           Demo API App
// @version         1.0
// @description     Simple Demo API Application

// @license.name  Apache 2.0
// @license.url   http://www.apache.org/licenses/LICENSE-2.0.html

// @host      localhost:8080
// @BasePath  /api/v1

func Routes(r *gin.Engine) {

	docs.SwaggerInfo.BasePath = "api/v1"
	v1 := r.Group("/api/v1")
	{
		v1.GET("/healthcheck", HealthCheck)

	}
	r.GET("/swagger/*any", ginSwagger.WrapHandler(swaggerFiles.Handler))
}

The API is pointed to controller to handle the application logic. Again this is simple and just returns service is running.

package controllers

import (
	"net/http"

	"github.com/gin-gonic/gin"
)

// @BasePath /api/v1

// HealthCheck godoc
// @Summary healthcheck
// @Schemes
// @Description healthcheck for service
// @Tags healthcheck
// @Accept text/plain
// @Produce text/plain
// @Success 200 {string} HealthCheck
// @Router /healthcheck [get]
func HealthCheck(c *gin.Context) {
	// Healthcheck response

	// Send response back
	c.String(http.StatusOK, "service is running")
}

I’ve included a simple test for the endpoint to run in the pipeline:

package controllers

import (
	"net/http"
	"net/http/httptest"
	"testing"

	"github.com/gin-gonic/gin"
)

func MockRouter() *gin.Engine {
	r := gin.Default()
	return r
}

func TestHealthCheck(t *testing.T) {

	gin.SetMode(gin.TestMode)
	r := MockRouter()
	r.GET("/api/v1/healthcheck")

	req, err := http.NewRequest(http.MethodGet, "/api/v1/healthcheck", nil)
	if err != nil {
		t.Fatalf("Request failed: %v\n", err)
	}

	w := httptest.NewRecorder()

	r.ServeHTTP(w, req)

	if w.Code != http.StatusOK {
		t.Fatalf("Expected to get status %d but instead got %d\n", http.StatusOK, w.Code)
	}
}

Pipeline Definition Files

The pipeline definition files will be stored here: https://github.com/wakeward/pipeline-examples. So keep an eye on the repository over the next couple of months.

Alright, let’s get to the hacking!