Concerns from a former Slicehost customer

Original Concerns from a former Slicehost customer Editable
version 1 & 2 of 2

I finally put off migrating my Slice to Rackspace's service admin panel. The new management seemed to be doing a good job running things and the hourly rate billing would have let me get a lot more server for my dollar. Sadly, Rackspace's backend programmers are stuck in the '80s.

Password requirement: 8-20 characters, 1 uppercase, 1 lowercase, and 1 digit. These special characters not allowed: * # & / { } ; : ' " ? < > `


There is only one reason for forbidding special characters: you are vulnerable to an injection attack. Similarly, there is only one reason for length limits on a password: you are storing passwords as plain clear text in a database. It goes without saying either of these are horrible practices, but in the case of server providers it gets worse

Internal Source (2012-10-15-07-18-07-492)

This is inaccurate, passwords are currently stored encrypted with random salts.


Of course, this is conjecture. The HN commentary provides several possibilities. Maybe they are just dumb (seems doubtful). Maybe because that the way it's always been. Maybe because it will "make enterprisey minds feel safe and secure", which is just depressing.

I've been assured that the entire stack is injection-proof and internally all the proper hashes and salts are applied. But I would like to hold server providers to a higher standard, and with good reason.

Leaks happen

It does not matter how good your security is or how trustworthy your employees are. Eventually a blackhat will get a copy of the database. If it is hashed, he will have to spend some time on rainbow tables to get the passwords out. And if it is salted, he'll only be able to get one password at a time. But password cracking is one of those embarrassingly parallel things, and the weak passwords will fall first. What if it is yours? When that happens, it would be trivial to do something simple and destructive like close everyones' accounts your account, but that is script kiddie stuff. If they were smart enough to be the first to get the passwords, they will be smart enough to try compromise boxes as subtly as possible.

An attacker with your account admin password can be very subtle. You won't see brute force attacks or failed login attempts or modified log files. It does not matter how clever or thorough of a sysadmin you are.

Most servers have some sort of recovery console. It is web accessible and will not ask for your root password or sudo privileges. The only password between this ultimately powerful console and the entire internet is the admin account password. If someone gets an admin password, it is pretty much game over.

I can only hope all activity on the recovery console is logged (so I could at least tell someone was fiddling with it) but given the other bad practices policies why give Rackspace the benefit of the doubt?

We have to trust our VPS host. The host can filter all the network traffic. The host has root on all the machines. The host can observe every byte of memory in use. There is no way to tell if they are, and you have to assume your host is not. Pick your VPS provider carefully.

While I must trust my provider to be good, I must assume that attackers will take advantage of any foothold.

Fixing It

I've exchanged a few emails with Rackspace support about this - frankly I am surprised that a tech company with an estimated 4.5 thousand employees and almost 200 thousand customers could make would keep such a blunder an archaic practice around. This stuff is Writing Webapps 201 level material. But Rackspace has said something will be done:

"It does appear that changes to the current password requirements will be put in place in December of this year." (private email, Oct 9 2012)

There are a couple ways this could go. I can only hope the correct thing will be done: handle passwords in such a way that the security of the admin panel is at least as good as the security I run on my VPSes. The alternative is to hide the issue.

I do not suggest trying to hide the problem. Your customers are technically competent. If you do the equivalent of $password.remove_all(bad_characters).substr(0, 19) we will find out very quickly.

Rackspace has made motions towards fixing this, so I'll still be keeping my VPS. But I've checked my incremental backups and completed a full backup just to be sure. And I'll be migrating all sensitive data and services off of it. It'll still do some light duty, relaying VPN/IRC and other minor things.

To my fellow Rackspace customers, we have the next month or so to convince our VPS provider that we do take security seriously and that there should be no upper limit to password complexity. We can make our services very secure for our users, but we can't do anything about security flaws outside of the VPS, short of going elsewhere. Write Rackspace support and tell them if you feel this is an issue. Swap tips on other providers or simple methods of re-deploying to a fresh server. (And if you are doing backups correctly, re-deploy should be very simple. But that is another post for another time.)

Slicehost, I already miss you. It was a good three years. Rackspace, get your act together. Here's some required reading: