Here’s How to Salt Your Own Passwords and Prevent a LinkedIN Style Password Problem

With all of the publicity surrounding LinkedIN, League of Legends, and possibly others, I thought I’d take a moment to explain how I manage passwords.  Yes, I quickly changed my password on LinkedIN, but using this method will add you just a bit more security if and when a provider screws up.  Remember, not every security incident involves a massive “post to everyone’s wall and make the evening news” style announcement!

I started this method after a screwup at Reddit a long time ago, which back in the day was storing plaintext passwords, and leaked them.  It has worked extremely well over the last six or so years.  For those who are unfamiliar with salting – it’s a method to increase randomness of passwords, and prevent rainbow attacks against password databases.  Technical readers may point out some more particulars about that statement, but the general statement should hold true.

Websites and other authenticators are supposed to salt passwords, but they can forget.  You can salt your own passwords by providing a hard to remember base password, then add some random characters to the beginning, after a fixed position, or end of the password.  This will make deriving your password more difficult, and will prevent (quick) account theft by an attacker taking your email and password and trying it at well known sites.

  1. Choose a decent password.  I recommend that people choose a nursery rhyme, favorite quote, saying, bit of religious text, or any kind of phrasing and choose the first letter or second letter of each word from that phrase.  This usually gets them a reasonably strong, easy to remember base password.  Capitalize a few of the letters and add a number at the beginning or end.
  2. Develop a salting mechanism. For every website that needs a password, develop a salting mechanism.  Have two rules of thumb as to how you derive a few additional characters based on the site itself.  Try to choose something relatively non-volatile.  For example – use the first four letters of the company’s name as it appears on their logo.  Or the last three letters of their domain name.  Or something else that may not change very often.  Then add these letters to your passphrase above at the beginning, middle or end of the phrase.  Why two rules?  So if your first rule doesn’t work due to some password scheme restrictions, you can use the second.
  3. You’re done. You now have an easy to remember passphrase that’s unique to that site (your base password + your derived salt).  Best of all, your password looks fairly random and even if your password from another site was stolen, it wouldn’t be susceptible to rainbow attacks, etc.

A few benefits of this approach:

  • You don’t need to rely on a “last pass” type application or system that stores all your passwords on a computer which may be hard to access sometimes.
  • It’s easy!
  • It’s free!
  • It really works – I’ve never had a problem with this, even with some of those sites that have weird password rules.
  • You now have a unique password to every system you access.
  • You can change the base password every year if you’d like, for even better security, and to weed out those accounts you forget about / never use.

There are no silver bullets to any of this.  Yes, this can still be attacked, but it’s certainly better than the “same password everywhere” or the “secure password for sites I care about and insecure one everywhere else” method.

I highly recommend you turn on Gmail’s 2 Factor security setting on your email account as well.  Particularly now that they’ve released an “offline” authenticator token generator which doesn’t need network access to work.

Stay safe out there!

I’ll Probably Never Hire Another Pure SysAdmin

NOTE: Updated Oct 17, See Below

This is a thought that’s been percolating around in my head for the last year or so, but has recently become even more crystalized: I’ll probably never hire another Systems Administrator.  A corollary to this thought would be: if you are currently a Systems Administrator or want to be one, you need to seriously begin planning on how to manage a career that will be mostly deprecated within the next 10 years.

Take a look at the current state of the art in cloud computing:

  • Spin up a server at your favor cloud provider (AWS, Rackspace, etc.), then use Puppet or Chef to deploy your software stack.  Now you’re done.
  • OR, Spin up an App at your favorite cloud platform provider, then push your code out using Git.  Now you’re done.
  • For both solutions, plug in some off-the-shelf monitoring, and you’re operating.

What’s missing here is the configuration, setup, provisioning, doc writing, black magic and/or prayer of setting up the software, hardware, and getting the code running that used to be the domain of the Systems Administrator.  In just a couple of years, deploying a web application has now become almost identical to deploying a desktop application – instead of an installer we’re using Git or Puppet/Chef. Instead of a customer’s computer we’re using a cloud platform or cloud server.

There’s plenty still to do on the networking side, but that’s headed in the same exact direction due to the same exact reasons: we want to be able to clearly define and programmatically execute the deployment of complex networks, just like we can with complex server offerings.

All of this falls under yet another buzzword: Dev/Ops.  Just like the cloud, we’re seeing this being adopted by smaller, nimbler organizations that are focused on web products, but the trend is clear, and there’s really no benefit in doing things the Old Way.  Even if you’re still running your own physical metal servers, you’re going to want to make sure that your own datacenter can leverage this type of workflow.  Now, the watchword to the development team is: it’s not done until I can one-click deploy it.

The laggards on this will be those industries that have regulatory or legal hurdles to overcome with using cloud services (read: healthcare) or the very large companies with services and technology that’s dozens of years old with no migration plan.

SysAdmins and future SysAdmins, you need to figure out where you’ll live in this new workflow.  Probably in the margins around monitoring or desktop support.  Possibly serving as the gatekeeper in a sort of “operations Q/A” role.  Expect small companies to have SysAdmin openings dry up over the next 5-10 years and get prepared.

Updated October 17: Hello Reddit/r/programming and Hacker News!  I wanted to take a few minutes and respond to a few themes that seemed to pop up in comments on HN and Reddit.

  1. I’m not saying Sysadminning is dead – just that the role is quickly changing.  Seems like a lot of people (anecdotally, many Sysadmins) thought I was saying the entire profession is dead.  Yes of course we’ll still need Sysadmins on some level, but the crucial difference is that for many areas of a business these needs will be less and much much different.
  2. Software development is changing too.  On complex deployments, developers can’t absolve themselves of the responsibility to design infrastructure considerations into the solution they’re building on the front end.  It’s a scary thought to think that organizations are out there that don’t have this level of partnership between ops and the devs.  This is why the puppet scripts should be written first and deployed on a test environment that’s identical in as many ways as possible to the ultimate operating environment (another benefit of using the cloud).
  3. Of course, any more complex deployment will need devoted SysAdmins, but like I said above, the skillset and day-to-day job will be dramatically different when wrestling with hundreds of servers instead of dozens.  More and more programming will become the norm and more and more upfront input into the solution will be an absolute requirement.
  4. I received a very thoughtful email from a former SysAdmin of mine (previous company) who pointed out that the job is much more along “system integrator” lines now, and that the internal vs. external network distinction is essentially going away.  I agree.
  5. Whenever your’e generalizing, counter examples abound.  Sure big companies and certain computing environments will still do things the Old Way but I’d challenge readers to objectively think if most business decision makers really want to hire someone and run their email server internally or just pay Rackspace/Google/Whomever to do it and worry instead about their money-making applications.  Even those organizations that need their clusters in house will invest in tech that allows them to mimic cloud operations on their own bare metal infrastructure.
  6. A couple of amusing anecdotes – the comments on HN immediately became more positive after a well known commenter defended the post, and a Googler chimed in as well.  That’s when the upvotes really started coming it seems.  On Reddit, the story was quickly downvoted!  Most users chose either a “genius” or “idiot” assessment of the post.  No real middle ground.

 

Web Development on Mac OS X (Lion)

Others have written about this before,  but I’ll underscore the sentiment that managing a local development environment on OS X where that environment requires Open Source Software is a royal pain.  At the companies I’ve been involved with, we generally eschewed local development environments and instead gave everyone access to a development server that included the requisite databases and web servers and vhost entries.  It worked OK, but there are some significant drawbacks.  Namely, unit testing, environment experimentation, single point of failure if the dev environment goes down, and the needs of a developer to refresh their own copy of a dev database or make other similar changes tend to suffer.As a hobbyist with a simpler environment, or as a developer that’s deploying to Heroku or other cloud platform, local development is the way to go, and here is where Mac OS X makes life difficult.  There are several package management systems out there that tend to step on each others’ toes (and it seems language and framework ecosystems always prefer the one that you’re not using).  Mac OS X also tends to haphazardly ship versions of Python or Ruby or whatever that are a couple of versions behind, then not upgrade them until they do an OS refresh.  That refresh (cough, Lion) will fail to mention it’s upending your world until you try to use your environment that’s always worked.Here’s my solution: just use VirtualBox.  Deploy an Ubuntu or Debian server, link that server to your local development directory and you’re done.  Then use the excellent package management that Linux affords to setup your environment in about ten seconds.  This has another advantage in that you can also use all your deployment hooks (Chef or Puppet) that you’re using on your production servers.Once you’re up and running, here’s how I work: I edit and run git from outside the virtual machine, and run the environment and web browser from within the machine.  Still todo: see if I can use my OS X browser to hit the virtual machine’s private IP so that all my tools are running externally (a little easier for workflow) so the virtual machine is just acting like an external server.Now you have a fully fledged (free, and always available) server, and you can still retain your Mac toolchain when and where you want it without worrying about Apple and OS X pulling the rug out from under you.  Remember: encapsulation of a work environment is just as important as encapsulation of code.

A Few Things I Wish the Apple Appstore Had…

My first app store experience was many years ago, before Apple, before iOS, before any of the other app stores you see today.  It was Redhat’s Redcarpet subscription service which delivered a library of applications (packages) via the internet in an easy to use command line tool.  There was even a GUI if I remember correctly.  Then Debian/Ubuntu came around with their package repositories and it was such a major usability difference between Linux and Apple/Microsoft that it was only a matter of time before it caught on.  Of course, the ideals behind the Linux offerings of ease of use, reliability, and compatibility are supplanted somewhat by the key aims of profitability and control inherent in modern app stores, but who’s counting?Things I wish the Apple App Store had (these are post-Lion upgrade thoughts)

  • Some way to know what the schedule of app update notifications is – it’s unclear to me if this is daily, whether I have to have the appstore application running all the time, etc.
  • The App Store should intelligently close your application when updating an existing application.
  • The App Store should be able to store your credentials, and not require credentials for doing an update if the app is in safe state (e.g. closed).
  • There should be a compatibility layer in Lion that lets you run your iOS apps on your Mac.  I’m sure this is coming, but I wonder when.
  • The App Store should offer to scan your hard drive and find applications that it can manage for you.
  • On the Featured and other pages, you should be able to hide apps you’ve already installed.
  • Somewhere down the line, it might be interesting to have a “lists” feature like Amazon.  Apple could even show what apps certain celebrities use in lists like the inflight magazines do for travel accessories.  Maybe that’s too much on the pointless-marketing side.

I’m sure there are some other options that I’m missing, but overall I’m happy with the experience.  The App Store managed Lion install was incredibly painless, and so much nicer than having to mess with the Apple store, or a nasty CompUSA.

Some Thoughts on Two Factor Security

Awhile ago I wrote an Open Letter to Mint.com laying out some major concerns I have with their service and their security implementation.  Almost all comments both here and on Hacker News and Reddit were divided into three categories:

  • From non-Americans: How is a service like Mint.com even possible or legal? US Banks don’t have two factor security?
  • I totally agree that Mint.com and their service is insecure and I don’t use them!
  • I agree that Mint.com needs better security, but their service is great and anyway, it would be too time consuming/too expensive/too hard/too impractical to implement these security improvements.

Between the time I wrote that letter and now, we’ve seen RSA (the only major token based two factor security provider) have all of its hardware tokens compromised to much public uproar. At Sentry Data Systems, we’ve had two factor security implemented for years using time based cookies and additional security questions to challenge users when they were logging in from a device that hadn’t been previously authorized.  This is similar to how many banks in the US do two factor security if they choose to implement it.  While not a HIPAA requirement, we felt that it was a great feature to offer that provided an additional layer of protection.  We’d originally offered RSA SecurID tokens to customers but found that most customers balked at the price, and even if they did use the tokens, many would simply tape it to their computer monitor or keyboard, or they’d forget the token at home which would cause quite the contentious support call. This experience brought to the forefront several issues that I had with hardware based tokens:

  • Casual users or those who didn’t value the two factor security benefit would simply leave the token lying around or affix it somewhere – it wasn’t natural to expect a user to carry one more thing with them day-to-day.
  • If there was a compromise, you have to replace all of your hardware, for everyone, everywhere.
  • They were expensive.
  • They were highly recognizable and screamed to informed observers that you had access to a system that was considered high-value by someone.

I even went so far as to start sketching out an iPhone app that we could deploy for our customers but it seemed like quite a lift to do it well (a correct implementation is key in cryptography systems) and it was with much delight that I ran across an outfit called DuoSecurity based in Michigan. They have really put together a fantastic service that provides both SMS based (challenge/response) and one-time password (via an iPhone or Android app) options for two factor security.  I signed up for the service, installed their package on my Ubuntu Linux server, and within about 15 minutes, I had a very strong two factor solution that avoids all the drawbacks of the hardware token approach…for free.  Yes – they provide up to 10 users for free to let you get your feet wet and see how the system works.  With the token being my phone, I’m not going to forget it, it doesn’t draw attention to itself, I can’t tape it to my workstation, and they can update the software if they need to. If their service goes down, you can configure it to not require the second factor (the default) or you can choose to prevent logins and keep a private key around for last-ditch logins.  Of course, for those of us running cloud based servers, there is still the risk that your hosting account could get compromised giving an attacker shell based access to the machine – hopefully Slicehost and other services will implement this type of additional security soon (Amazon’s EC2 cloud already implements two factor security as an option). Duosecurity can be easily implemented with any web application, a lot of VPNs, and on your Unix/Linux servers quickly and easily.  If you’re doing anything with medical, financial, or other sensitive data you should definitely check them out.  If you just like additional protection for your own servers and services, they’re a great option as well.  Just in case you’re curious: Duosecurity put up a great blog post about the steps they’ve taken to prevent compromise if they came under the same attack as RSA. A few thoughts on improvement:

  • Give me an apt package please!  I don’t want to compile things, and I don’t want to edit configuration files.  These things make it hard to deploy on lots of servers.  I talked with a support rep from Duosecurity and they told me this is in the works already.
  • Put a login form on your website!  They email the login URL to you but I shouldn’t have to remember it.
  • It’s a little unclear to me if the pricing scales well- if I’ve got the same 35 users access 100 machines, does that mean I pay 35x100x$3?  That seems expensive.  Course, it’s still way cheaper than RSA but at least you could bind an account to a token and not worry how many servers you were accessing.  It’s possible that a single user crosses the server boundary, but again, I’m unclear on that.

Bringing it all back to the original point – there is simply no excuse why a service like Mint.com doesn’t use Duosecurity to protect its own user’s logins.  But the second issue still exists – how do banks provide consumers of financial data access without compromising the entire account? A poor man’s solution of sorts could be taken by banks providing read-only accounts for customers that use generated, revokable passwords.  Google takes this approach with its own two factor implementation for Gmail.  You get texted when logging in normally, but for other applications, you generate a password that can be revoked at any point.  It seems like a decent compromise – you can’t control the account from that login, and the password is of sufficient length and complexity that it’s unlikely to be brute forced.  My initial suggestion of using Oauth is essentially the same thing. Congratulations to the guys/gals at Duo Security on providing a really great set of tools for developers and users.  I really hope it catches on and more and more providers begin offering two factor as an option.

More Easy Fun With Telephony

Recently I decided I needed a little more flexibility with my phone situation.  Years ago I was carrying two cell phones and had three Vonage lines while running my own business.  This got consolidated down to a single iPhone, but that can be a little problematic particularly if you’re calling to/from international numbers.  This week I ported my iPhone number to Google Voice (within 24 hours too), and got a new phone number for my cell that I’m hoping to keep private and function as a throwaway.  However, I needed a bit more flexibility on some of the things I wanted to do, so I threw Tropo into the mix.  Twilio lost out because Tropo provides free inbound and outbound calling.So here’s the path when you call my number:  call comes into Google Voice, which forwards the call to my Tropo application, which then plays a menu and you can either punch out to Sentry’s main number or continue to ring my cell.  Text messages are forwarded by Google Voice, and the net result is that for inbound calls, I’ve effectively decoupled the phone number I’ve had for seven years from any handset or location and added a whole bunch of flexibility.It’s almost eerie how much power Tropo gives you over your telecom setup.  With a few lines of code I can transfer calls, accept inbound international calls with a local number, kick out text messages, provide a menu, have their computer voice speak any text I want, etc.  Call quality is crystal clear through both Google and Tropo, and I have yet to have any reliability problems.  In 2004 we thought it was amazing replacing a 150k Avaya PBX with Asterisk, but this is replacing all of that with about twenty lines of code.  For free.  With no setup or ongoing hardware or maintenance costs.I’d say the only real drawback to the situation is the inability to spoof outbound caller id with native dialing – it would be interesting to see if Apple allows you too hook other providers into it’s native dialer (yeah right) or if this is a feature within Android.  It definitely needs to be implemented at some point – and then we’d have true telecom nirvana.

The Impending Ad War

A few weeks ago I attended Github’s CodeConf in San Francisco.  While there, I got to meet quite a few really accomplished technologists (hackers) and discuss a variety of projects, processes, programming methods and more.  One of the most interesting moments for me came over lunch while talking to the CTO of a very well known blog which clocks in over 5 million unique visitors a month.  Like most sites of its type, it receives almost 100% of its revenue from ads.  According to him, one of the largest (new) challenges they were facing was that advertisers are beginning to buy ads targeting the blog’s fans from Facebook, not the blog itself.  In other words, to get at the blog’s users, advertisers were paying Facebook less money to directly market to the blog’s fans on Facebook.The more I think about it, the more I think this is a major problem for almost every ad supported site out there, and it could be the pitch that Facebook is using to bolster its insane valuations.  Right now, there are probably no less than a dozen Googlers being kept up at night worrying over this very problem, not to mention the admen at hundreds of highly trafficked blogs and other internet properties.  After all, if I can immediately pitch my competing product to your customers without paying you a dime, I’ve got a huge advantage, you’ve got a huge problem, and Facebook has an unbelievably great strategic position.Maybe you’re reading this and thinking “yeah that’s old news” and it probably is to many, but having never worked at an ad supported organization, I’d certainly never thought about it before.  I’ve also never heard it articulated online, and I’m wondering how many organizations even realize this is happening.  Note there is a two-fold risk here: ad supported properties risk losing ad revenue to Facebook, and they risk exposing their customers to competition.  If you’re an advertiser, you’d much rather know that you’re reaching out to all 10,000 fans of Blog X with the stats to show you who clicked, etc., vs. an anonymous 100,000 impressions.  Note that even if a Blog chose not to have a Facebook page to attempt to combat this kind of thing, Facebook can still harvest those users who “like” the Blog in their profile.Before, I used to think that the benefits of a Facebook presence for an organization outweighed the downsides, but now I’m not so sure, particularly for ad-supported businesses.  It’ll be interesting to see how this plays out.