Details have emerged about an attack on the infrastructure used in the development of the PyTorch machine learning framework, which allowed the extraction of access keys sufficient to deploy arbitrary data in the project’s release repository on GitHub and AWS, as well as to insert code into the main branch of the repository and add a backdoor through dependencies. The compromise of PyTorch releases could have been used to carry out attacks on large companies such as Google, Meta, Boeing, and Lockheed Martin that utilize PyTorch in their projects. As part of the Bug Bounty program, Meta paid researchers $16,250 for information about the issue.
The essence of the attack lies in the ability to execute one’s own code on servers continuous integration systems that rebuild and execute jobs for testing new changes sent to the repository. The issue affects projects that use self-hosted runners with GitHub Actions. Unlike traditional GitHub Actions, self-hosted runners are executed not in GitHub's infrastructure, but on their own servers or in virtual machines maintained by developers.
Running build jobs on their own servers allows for the execution of code that can scan the enterprise's internal network, search the local filesystem for encryption keys and access tokens, and analyze environment variables with parameters for accessing external storage or cloud services. Without proper isolation of the build environment, any confidential data found can be sent out by the attackers, for example, through calls to external APIs. The Gato toolkit can be used to identify the use of self-hosted runners in projects by analyzing publicly available workflow files and logs of CI jobs.
In PyTorch and many other projects using self-hosted runners, the execution of build jobs is only allowed for developers whose changes have previously been reviewed and included in the project's codebase. Holding the status of 'contributor' while using default repository settings enables the execution of GitHub Actions runners when submitting pull requests, allowing them to execute their code in any GitHub Actions Runner environment linked to the repository or the overseeing project organization.
The binding to the status of 'contributor' turned out to be easy to bypass — it was enough to send a minor change beforehand and wait for its acceptance into the codebase, after which the developer automatically received the status of an active participant, whose pull requests were allowed to be tested in the CI infrastructure without separate validation. To gain active developer status during the experiment, minor cosmetic changes related to fixing typos in the documentation were used. To gain access to the PyTorch repository and release storage during the attack, a GitHub token was intercepted while executing code in the 'Self-Hosted Runner,' which was used to access the repository from build processes, as well as AWS keys involved in saving build results.
The issue is not specific to PyTorch and affects many other large projects that use default settings for 'Self-Hosted Runner' in GitHub Actions. For example, similar attacks have been mentioned for inserting backdoors into several major cryptocurrency wallets and blockchain projects with billion-dollar market capitalizations, modifying releases of Microsoft Deepspeed and TensorFlow, compromising one of CloudFlare's applications, as well as executing code on computers in the Microsoft network. Details regarding these incidents have not yet been disclosed. As part of the existing Bug Bounty programs, researchers have submitted over 20 requests for rewards totaling several hundred thousand dollars.
Source: opennet.ru
