The developers of the collaborative development platform SourceHut published an incident report detailing how the service was disrupted for 7 days due to a prolonged DDoS attack, for which the project's infrastructure was unprepared. The basic services were restored by the third day, but some services remained unavailable from January 10 to 17. At the initial stage of the attack, the developers were unable to respond in time and attempt to counter the problem on their servers, as all traffic to SourceHut servers was completely blocked by the upstream provider.
To ensure the operation of SourceHut, three data centers were used. servers The first housed the working configuration, the second was used for backups, and the third was used for experiments related to migrating the infrastructure to a more scalable and fault-tolerant implementation of the service (developing the next generation SourceHut).
Resolving the issue of access to their servers in the main data center after the blockage took about 9 hours, but the developers were unable to do anything because in the morning the attack intensified and began to cover the entire subnet, after which the provider rerouted the traffic to a null interface again. The developers were forced to urgently begin deploying the SourceHut infrastructure in another data center using backups (a temporary subnet for accessing the main servers was only obtained after 2 days).
To protect against DDoS attacks occurring at the network level, it was deemed optimal to place an intermediary server in the cloud provider OVH's network, which offers DDoS protection. All requests were directed to this server and then forwarded to the working infrastructure. During the migration, errors that took additional time couldn't be avoided, such as incorrect restoration using the rsync utility, errors in network configuration, and issues with traffic redirection had to be resolved (before the DDoS protection in OVH kicked in, the DDoS attack traffic was also routed to the working servers, prompting the DDoS protection system to react and recognize the receiving server as the source of the attack).
The developers also reached out to Cloudflare and some other DDoS protection services, but the quoted cost of protection was prohibitively high. Later, Cloudflare employees managed to negotiate with management to provide protection for the SourceHut project free of charge as a sponsorship, but the SourceHut developers declined the offer as by that time they had already made significant progress in solving the problem on their own.
The implementation of the new SourceHut infrastructure and the migration of the project to servers in a different data center was planned to take place gradually over a minimum of one year, but under the current circumstances, the migration had to be carried out urgently within 7 days. Currently, all SourceHut services have been moved to another data center, and the platform's operation has been fully restored.
The SourceHut platform features a distinctive interface that is not like GitHub or GitLab, but is simple, very fast, and operates without JavaScript. SourceHut offers capabilities such as working with public and private Git and Mercurial repositories, a flexible access management system, wiki, bug reporting, built-in continuous integration infrastructure, chat, email-based discussions, hierarchical browsing of mailing list archives, and web-based change review with code annotations (link attachments and documentation). With the appropriate settings enabled, users can participate in development without local accounts (authentication via OAuth or participation via email). The code is written in Python and Go and is distributed under the GPLv3 license.
Source: opennet.ru
