It’s the nightmare scenario that organizations are warned about.

Code Spaces, a Subversion and Git hosting provider, used by organizations for project management and development needs, has folded after an attacker compromised their internal systems.

The company, which was making a name for itself in the IaaS (Infrastructure as a Service) / DEVOPS community, says there’s just no way for them to resume operations.

It started with a DDoS attack on Tuesday. When Code Spaces reached out to the attacker, they were told to pay a ransom in order to stop the traffic flood.

However, the issue was much larger than a sustained DDoS. In fact, the reason they were able to contact the attacker in the first place is because they left contact details within Code Spaces’ Amazon EC2 control panel.

“It’s common for a DDoS to be a smokescreen for another attack that is aimed at gaining access to the target’s systems, and the Code Spaces attack appears to be a textbook case of this,” Trey Ford, global security strategist at Rapid7, said in a statement.

“Responding to a Denial of Service calls for all hands on deck, all available resources activated and focused on availability - diagnosing how the attack is being executed and how best to mitigate. In situations like this, responders would do well to re-task a separate ‘assurance team’ to verify systems, accounts and login activity.”

Code Spaces moved to regain control over their Amazon accounts, but the attacker had already taken steps to prevent this. According to a post on the incident, the intruder created backup log-ins on the EC2 panel and when recovery efforts were noticed, they started to delete artifacts at random.

“We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI’s, some EBS instances and several machine instances. In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted,” the Code Spaces post explains.

It’s possible, Ford added, that having a runbook on how to do a lockdown (whether for locking out a rogue admin or to contain your environment) might have saved this team. Unfortunately, even if they did have such a plan, it didn’t work.

In a statement, Patrick Thomas, security consultant for Neohapsis, said that for companies using cloud services as part of their business, “this is the nightmare scenario.”

“This is a wakeup call to other organizations that have critical assets on cloud services. Two factor authentication and detailed event monitoring and alerting are essential components of any cloud strategy. Similarly, offline or warm-storage are critical business continuity measures,” he added.

Based on the limited data provided by Code Spaces and typical attacker behavior, the root cause of this disaster likely involved a Phishing attack against users with access to cloud service credentials. However, Code Spaces didn’t explain that aspect of the incident, but they promised a full report later, once their customers are taken care of.

While losing internal data is bad enough, the loss of off-site backups is a serious blow. For years, the use of off-site backups has been the standard operating procedure for organizations, especially where code is concerned.

“But in the age of cloud infrastructure many organizations think that they can simply pass the buck on backups, getting their geographic distribution and redundancy ‘for free’ as part of going to the cloud,” Thomas added.

“However, anything that’s vulnerable to the same threats isn’t fulfilling the original intent of offsite backups. Perhaps it makes more sense to start talking in terms of ‘diversified backups’ to emphasize the broad types of threats that a backup strategy must mitigate.”

In the end, the nighmare is real, and a small business might be forced to close for good due a single security incident. Code Spaces is the victim here, but many experts are in agreement that they share some of the blame.

Rob Ayoub, the Research Director of NSS Labs, recently authored a report on Amazon’s Web Services. He said that the Code Spaces attack illustrates some of the key challenges that companies need to consider when using any IaaS provider - not just Amazon:

• In an IaaS environment, once the controls are compromised, it’s very difficult to control or remediate quickly. Moreover, logs may require separate requests for support that may take days to receive, especially if extensive.
• Amazon only provides the infrastructure. The backing up of data is left entirely to the end user. There are several vendors that offer solutions to ease backup efforts from EC2, but those solutions cost extra. Amazon even offers Glacier, which is its own backup solution, but again the user has to implement the backup.
• Amazon continues to make improvements natively to EC2’s security and many vendors offer virtual appliance versions of their solutions, but many organizations have not ported the same controls from their on-premise infrastructure into services like Amazon. Would a virtual appliance version of a NGFW or UTM device have stopped this attack? Maybe.

“Again, I don’t have all the details on this attack. We may never know how the attacker got in, but in my discussions with customers and vendors, there are many false assumptions that organizations make when moving data and services to the cloud,” Ayoub said.

“It is sad that Code Spaces was potentially forced out of business by an attacker. I would hope that Amazon might offer some forensics help, because I feel ultimately there is a shared responsibility for security between Amazon and its customers.”