CI/CD Workflows: Is your Environment at risk of a new attack? 

CI/CD pipelines are an integral part of modern workflows and the tools that support them further expand their latent capabilities. Team effort is essential in modern workflows where deadlines are not only urgent, but regular as well. Every second counts when a customer is waiting on the other end. The best place to start the discussion on CI/CD therefore is what the tools allow you to accomplish. 

What is CI/CD? 

At its core, CI/CD environments provide tools that allow software to go live sooner and streamline the build and release process. Historically code is developed by separate teams and tested in personal environments; integration takes time, and feedback on broken code takes longer. This leads to lengthy build cycles and decreases in efficiency. It’s common practice to test a piece of code with some amount of test cases designed before development. If the final code meets these standards, it should integrate into existing code easily.  

But as developers, we all know this is far from the only possible outcome even with meticulously designed test cases. CI/CD presents a way to execute test cases on a large scale, where code changes can be tested in a single streamlined and automated action. Code is pushed, received by the pipeline, and tested, and if all cases pass, the build can be set to go live. It’s amazing what this can do for your organization’s agility.

One of the modern dangers in the tech sphere is the blind leading the blind. Non-experts have access to the same Internet experts do and frequently write tutorials and guides. AI, a genuinely interesting and useful creation somewhat amplifies this problem. Chat GPT is genuinely convincing to some with its frequently authoritative tone and apparent confidence, maybe even charisma. But it pulls from this greater pool of Internet sources making it equally vulnerable to only understanding half the picture. While the following story is not meant to be a sales pitch, do consider the benefits of having a cyber security consulting firm involved in the design of your architecture, such that the big picture is not missed when broaching new and powerful technologies. 

The Most Recent Attack 

Today we are focusing on a specific attack developed by two researchers. This attack targets the GitHub actions CI/CD platform, specifically self-hosted runners. There are many benefits to self-hosted runners, one of which is the greater flexibility they provide in features and options. While the attacks focus largely on public GitHub repositories, these are not that uncommon in the industry and even if your organization does not host a public repository, you likely depend on one or more for your products.

While the main attack targeted GitHub actions, similar vulnerabilities have been identified across multiple CI/CD platforms used in the industry; Jenkins, Circle CI, Buildkite, etc. Since this attack focused on public repositories, its theoretical consequences could have reached far and wide if it was malicious. To complicate matters, these vulnerabilities arise in part from bad default settings.  

To perpetrate an attack, contributor status must first be achieved. A lofty goal one might think, especially for a large prestigious project. But all the researchers needed to gain this status was correcting a typo. With this status, they could then initiate a pull request. Granted, anyone can initiate a fork pull request on a public repository, but when a contributor makes a pull request they will, per default settings, have the ability to use the runner attached to the main project. This will happen per these default settings before approval for the merge is received. If a malicious workflow is contained within the fork, it will be executed by the runner.

Runners persist on the machine they operate on, so this code can continue to affect future builds and run in the background on the machine uninterrupted. The runner can basically be made to act as a Trojan for the attacker. If the runner has elevated status on the machine, which is a standard requirement, the machine won’t recognize or flag it for its suspicious activity. Key logging can then collect the authentication token and the entire process can be hijacked. Authentication tokens allow full access to the repository, from there the attacker can do whatever they want. 

Far more insidious than hijacking a public project in such a visible way would be to abuse the persistent runner to affect future builds silently. These repositories are widely used, one such repository pytorch, which was directly attacked as part of the project, owns 21% market share in the machine learning space. Imagine the number of projects, companies, and software that could be compromised by an attack. Not to mention, with the long dependency chains typical of modern projects, your organization may not even be aware that they were dependent on a project with a known compromise. In this case, we are lucky that at present the attackers were researchers not interested in causing harm, but it’s a good time to review the process and tools your organization is using for CI/CD as well as if any of your dependencies are still vulnerable to this attack method. 

Conclusion 

Attacks are hard to keep track of, especially in the modern day when the scope of a compromise is difficult to ascertain before the damage is done. If your organization uses CI/CD and needs information regarding how to secure the pipeline and prevent attacks, please consider reaching out to Encryption Consulting.  

Our team of experts can aid in setting up best practices for access controls and security policy on your CI/CD platform as well as to help evaluate potential attack vectors across your organization. Our services also include certificate management and codesigning, which can easily integrate with your secured CI/CD process and further expedite your building and testing process.