Tag: software development

Simplify Everything

johnjpeebles1 Comment

There’s a lot of abstract advice about employing simplicity when building great products or writing great code.  However, life and products (particularly in the Enterprise software market, where I’ve spent most of my career) are complicated.  It’s often hard to gleen concrete examples of what these maxims are trying to communicate.

The other day I was in a pub waiting for a lunch meeting to start and I got to witness the week’s beer delivery.  This is a fairly hard problem to solve efficiently if you’re in a city where parking is difficult (or nonexistent), buildings were constructed hundreds of years before accessibility laws (meaning stairs and tight doorways), kegs are very heavy (over 200 pounds when full), and where a lot of beer is consumed requiring frequent deliveries.

If you or I were designing a solution to this problem, we might come up with this solution:

  • 1 truck
  • 2 employees (1 driver, 1 loader/unloader)
  • 1 automatic lift at the rear or on the side of the truck
  • 1 appliance dolly that can move up or down stairs

We’d be pretty happy with that.  Not the worst solution in the world.  It’s possible we could reduce to one employee but the automatic lift will take enough time setting up and lifting that we’ll probably exceed our very short “stop with flashers on” window.  We’d therefore need to park and have someone stay with the truck, or make several “fly byes” to stay within the unloading time limit.  This will really limit how many delivers we could make in a day, possibly requiring a lot of delivery crews.

Here’s how they actually do it:

  • 1 truck
  • 1 driver / unloader
  • 1 airbag

The driver pulls up, parks the truck right outside the entrance of the pub with the flashers on, whips out his airbag from the passenger seat, rolls up the side of the truck, pulls off the keg and lets it fall right on the airbag.  He then rolls it into the pub (for those with cellar keg storage, they have their own airbag) and after about 20 kegs and less than 5 minutes, he’s out of there.

A lot less cost, a lot faster, and no expensive equipment.  

Simple.

Img_0871

I’ll Probably Never Hire Another Pure SysAdmin

johnjpeebles3 Comments

NOTE: Updated Oct 17, See Below

This is a thought that’s been percolating around in my head for the last year or so, but has recently become even more crystalized: I’ll probably never hire another Systems Administrator.  A corollary to this thought would be: if you are currently a Systems Administrator or want to be one, you need to seriously begin planning on how to manage a career that will be mostly deprecated within the next 10 years.

Take a look at the current state of the art in cloud computing:

  • Spin up a server at your favor cloud provider (AWS, Rackspace, etc.), then use Puppet or Chef to deploy your software stack.  Now you’re done.
  • OR, Spin up an App at your favorite cloud platform provider, then push your code out using Git.  Now you’re done.
  • For both solutions, plug in some off-the-shelf monitoring, and you’re operating.

What’s missing here is the configuration, setup, provisioning, doc writing, black magic and/or prayer of setting up the software, hardware, and getting the code running that used to be the domain of the Systems Administrator.  In just a couple of years, deploying a web application has now become almost identical to deploying a desktop application – instead of an installer we’re using Git or Puppet/Chef. Instead of a customer’s computer we’re using a cloud platform or cloud server.

There’s plenty still to do on the networking side, but that’s headed in the same exact direction due to the same exact reasons: we want to be able to clearly define and programmatically execute the deployment of complex networks, just like we can with complex server offerings.

All of this falls under yet another buzzword: Dev/Ops.  Just like the cloud, we’re seeing this being adopted by smaller, nimbler organizations that are focused on web products, but the trend is clear, and there’s really no benefit in doing things the Old Way.  Even if you’re still running your own physical metal servers, you’re going to want to make sure that your own datacenter can leverage this type of workflow.  Now, the watchword to the development team is: it’s not done until I can one-click deploy it.

The laggards on this will be those industries that have regulatory or legal hurdles to overcome with using cloud services (read: healthcare) or the very large companies with services and technology that’s dozens of years old with no migration plan.

SysAdmins and future SysAdmins, you need to figure out where you’ll live in this new workflow.  Probably in the margins around monitoring or desktop support.  Possibly serving as the gatekeeper in a sort of “operations Q/A” role.  Expect small companies to have SysAdmin openings dry up over the next 5-10 years and get prepared.

Updated October 17: Hello Reddit/r/programming and Hacker News!  I wanted to take a few minutes and respond to a few themes that seemed to pop up in comments on HN and Reddit.

  1. I’m not saying Sysadminning is dead – just that the role is quickly changing.  Seems like a lot of people (anecdotally, many Sysadmins) thought I was saying the entire profession is dead.  Yes of course we’ll still need Sysadmins on some level, but the crucial difference is that for many areas of a business these needs will be less and much much different.
  2. Software development is changing too.  On complex deployments, developers can’t absolve themselves of the responsibility to design infrastructure considerations into the solution they’re building on the front end.  It’s a scary thought to think that organizations are out there that don’t have this level of partnership between ops and the devs.  This is why the puppet scripts should be written first and deployed on a test environment that’s identical in as many ways as possible to the ultimate operating environment (another benefit of using the cloud).
  3. Of course, any more complex deployment will need devoted SysAdmins, but like I said above, the skillset and day-to-day job will be dramatically different when wrestling with hundreds of servers instead of dozens.  More and more programming will become the norm and more and more upfront input into the solution will be an absolute requirement.
  4. I received a very thoughtful email from a former SysAdmin of mine (previous company) who pointed out that the job is much more along “system integrator” lines now, and that the internal vs. external network distinction is essentially going away.  I agree.
  5. Whenever your’e generalizing, counter examples abound.  Sure big companies and certain computing environments will still do things the Old Way but I’d challenge readers to objectively think if most business decision makers really want to hire someone and run their email server internally or just pay Rackspace/Google/Whomever to do it and worry instead about their money-making applications.  Even those organizations that need their clusters in house will invest in tech that allows them to mimic cloud operations on their own bare metal infrastructure.
  6. A couple of amusing anecdotes – the comments on HN immediately became more positive after a well known commenter defended the post, and a Googler chimed in as well.  That’s when the upvotes really started coming it seems.  On Reddit, the story was quickly downvoted!  Most users chose either a “genius” or “idiot” assessment of the post.  No real middle ground.

 

More Easy Fun With Telephony

johnjpeeblesLeave a comment

Recently I decided I needed a little more flexibility with my phone situation.  Years ago I was carrying two cell phones and had three Vonage lines while running my own business.  This got consolidated down to a single iPhone, but that can be a little problematic particularly if you’re calling to/from international numbers.  This week I ported my iPhone number to Google Voice (within 24 hours too), and got a new phone number for my cell that I’m hoping to keep private and function as a throwaway.  However, I needed a bit more flexibility on some of the things I wanted to do, so I threw Tropo into the mix.  Twilio lost out because Tropo provides free inbound and outbound calling.So here’s the path when you call my number:  call comes into Google Voice, which forwards the call to my Tropo application, which then plays a menu and you can either punch out to Sentry’s main number or continue to ring my cell.  Text messages are forwarded by Google Voice, and the net result is that for inbound calls, I’ve effectively decoupled the phone number I’ve had for seven years from any handset or location and added a whole bunch of flexibility.It’s almost eerie how much power Tropo gives you over your telecom setup.  With a few lines of code I can transfer calls, accept inbound international calls with a local number, kick out text messages, provide a menu, have their computer voice speak any text I want, etc.  Call quality is crystal clear through both Google and Tropo, and I have yet to have any reliability problems.  In 2004 we thought it was amazing replacing a 150k Avaya PBX with Asterisk, but this is replacing all of that with about twenty lines of code.  For free.  With no setup or ongoing hardware or maintenance costs.I’d say the only real drawback to the situation is the inability to spoof outbound caller id with native dialing – it would be interesting to see if Apple allows you too hook other providers into it’s native dialer (yeah right) or if this is a feature within Android.  It definitely needs to be implemented at some point – and then we’d have true telecom nirvana.

Open the Gate!

johnjpeeblesLeave a comment

The last three places we’ve lived in South Florida were “gated communities” which is supposed to make you feel exclusive and special.  They provide zero additional security (had a car stolen from one of them in the middle of the night) are often broken, and even when they work they’re a pain.  All of our gated communities would link your personal code to a phone number of yours, and when visitors keyed in “112” it would ring your phone.This causes problems:

  • The gate dialer can only link to one phone.  If your wife is traveling and you want some pizza to be delivered, the wife may not be able to pickup the phone and press 6 to let the pizza in.
  • Most can only link to one or two area codes.  One of the systems could only link to a 954 area code number.
  • If you’re riding with someone else and don’t have your remote with you, you can’t get in if your wife isn’t with you, or has the cell phone in a bag in your trunk.

The Wife has been out of town for a few days and this finally irritated me to the point where I headed over to Tropo.com and provisioned a simple phone application.  Now when you dial the phone number of my Tropo app, it answers, says “Opening the Gate!” and plays a number 6 key press which tells the gate to open.  Perfect.I heard about Tropo out at CodeConf in San Francisco and have wanted to play with it but didn’t have a problem to solve until now.  The entire thing took about 10 minutes to setup with the only really painful thing being the hunting down of a key press sound from http://www.freesound.org/ and the subsequent conversion to a GSM format.  I ended up using the excellent Sox command line sound converter to make the conversion, and then we’re in business.  Total cost for the whole thing was zero dollars.The Tropo service is really nice and their documentation is good too.  Their UI for their website is a little clunky in spots.  For example, picking an area code for your number is really painful with about 50 city suggestions and no way to search for an area code or specific city.  They’re not alphabetized as far as I can tell either and the city names are super specific so it just makes it hard.  Also, I couldn’t find a way in their API to generate a key press tone which meant I had to mess with my own sound files.  That should be built right in or they should provision a directory of key press sounds with your default files.All in all a fun little project to get done while on Amtrak bound for Orlando, and now I can open my gate whenever I want.  Tropo has done a great job with their platform and I’d highly recommend it for these types of tools or any kind of telephony or communications application.

Christmas – a Time of Giving

johnjpeeblesLeave a comment

Normally around Christmas people are focused on giving.  Giving to family, friends, maybe even coworkers.  Some give time to charities, some make donations to others.  One of my annual traditions for Christmas is giving to the Electronic Frontier Foundation (EFF), a digital liberties organization that fights for freedom of speech online, privacy, and other causes that are generally too technical to garner attention from other organizations.

This year I’m adding a very similar organization to my annual Christmas list, Software Freedom Law Center (Hi Aaron!),  which similarly advocates for issues that increasingly plague and impact our lives, whether we understand them or not.With all the buzz going on right now about Wikileaks maybe you’ve thought about these issues a little more than normal.

Depending on your understanding of the issue or political orientation, it might be hard to believe, but freedom of speech online is a relatively fragile thing.  Maybe it’s because technology appears to be finite and controllable, maybe it’s because online speech travels much faster and is easier to consume, but it seems that governments and governing bodies tend to naturally flock towards the idea of speech regulation of technical mediums.Groups like the Software Freedom Law Center and the Electronic Frontier Foundation work to preserve freedoms on our behalf.

When I say “freedom of speech online” or “freedom online”, we’re also talking about fighting against overly broad software patents, violators of the GPL, warrantless wiretapping of online communications, and a whole bunch of other issues that may seem only tangentially related to speech, but do impact your voice online.In addition to the work they do, I’ve had the privilege of interacting with and knowing a few individuals from both organizations, and they’re great people as well!  So Merry Christmas, and if you’re looking for a great gift idea or a cause to support, check out both groups.

Thinking Like a User: The Bank Analogy

johnjpeeblesLeave a comment

Like every software organization, we have trouble getting our developers how to think like users.  It’s a time consuming process, and when you’re in an industry like healthcare, it can often be extremely difficult to visualize exactly how a product is going to be used or how a process might affect a user.  You’ve certainly never been to med school, or worked in a pharmacy.  These things are utterly foreign.  However, visualizing a user’s workflow or responsibilities is often the difference between a mediocre product and a great product.One method that I’ve found to be illuminating to our engineers is to restate every message, function, or process in terms of a bank, with them playing the user character.  Instead of the error message being “Purchase Order Alert: We’re sorry, but there were one or more items missing from your recent purchase, click here for more details“, rephrase it as “ATM Deposit Alert: We’re sorry, but there were one or more checks missing from your recent deposit, click here for more details.“Makes a difference doesn’t it?  Now our verbose, overly-polite alert text seems almost ridiculous.  Tell me what happened with my checks! Don’t make me click through for more detail!  Especially if I’m on a mobile device!  That’s my money and it’s important!“ATM: 1 check for $100.00 was returned.”That’s much better.  I feel notified and in control as a user, and it was short and sweet.  Very cellphone friendly.“Order 123: 1 item shorted: Tylenol 20MG”The point is that users actually use the products we build.  Things like error messages and process flows are important.  As developers, we often think of these things as control points, or logic trees, and don’t stop and relate what’s going on from a user’s standpoint to any of the important systems or applications that we ourselves use.  The bank analogy should help you get in the habit of stepping aside and thinking outside your code for a bit because it pulls the process or message into the realm of your experience.Even the process of constructing the analogy can be very instructive.  If you can’t quickly pull together an analogy that describes what you’re doing in terms of your own life, you probably have no clue what you’re doing.  In other words, you’re probably doing it wrong.Next time you’re solving a problem, try restating it using concepts you’re familiar with in your own life.  You might be surprised at how useful this technique can be.Updated: Sunday, 12/5/2010Today’s New York Times Magazine had an article on Jamie Dimon (I think it requires registration) that talked about his management and conversational style.  One of his favorite things to do was equate banking principles back to ordinary life.  Seems like we’ve got company on the other side of the fence too!

The Non-Technical Tech Talk

johnjpeeblesLeave a comment

Two weeks ago (was it three?) I spoke at an ACM sponsored Tech Talk at Purdue University.  My mission was to give perspective on what it takes to run a large and quickly growing software team.  Many startups (including some I’ve been affiliated with in the past) will grow from very small teams to departments of over a hundred in a short time.  That kind of growth is explosive so as I thought about what to say during the presentation, I was struck that very rarely do I hear much about the human element of running a software organization.  I never heard about it in school. Focus on the people side of software construction has been one of my major recurring management themes because, as I told the group of students at Purdue, software development is one of the most human-intensive, human-dependent disciplines/arts/crafts/industries that exist.

To many, this might exemplify a supreme irony, but this has always been a core belief of mine: you can’t build great software without great people.  You can’t build great software without great teams.  Hardware and tools are easy to come by now, and the end result is that in today’s software industry, people are the only variable that means anything.  That’s why software companies should be psychotic about keeping turnover low.  Everywhere I’ve been I try to minimize bureaucracy.   I encourage telecommuting and spend money on great tools.  I like to setup team building, fun activities that appeal to technology workers like playing video games together once a week.  These are all downstream things that I focus on to maintain my upstream asset: people.

What does working on a great team with great people look like?  For students still in school, this is hard to visualize.  Group projects are almost universally hated.  Internships generally involve working as someone’s grunt on some meaningless or semi-solo system or task.  But I got all heads nodding when I asked if while they were doing their horrible group projects with their horrible group members if they remembered that one group who got up in front to present that was clearly happy to be working together, who had enthusiasm for their work product, and had spent an inordinate amount of time on their project.  Did they seem tired or annoyed?  No.  Was their project the best in the class?  Yes.  That is what a well-gelled software team feels like, looks like, and that’s the kind of product it produces. I talked a little more about great teams and how to recruit them and run them as we all munched on pizza.   Afterwards the organizer walked up and stated that I had given the most non-technical tech talk ever, but that he felt it was important and something that often gets missed.  

I definitely agree about the missing piece, and we got invited back for more pizza and another tech talk so we’re grateful for another opportunity.  I’d say that was mission accomplished.

Distributed Software Teams

johnjpeeblesLeave a comment

At Sentry Data Systems, we have a very distributed technology organization.  The majority of our technical staff does not work from our Deerfield Beach headquarters.  Instead, we have our developers, implementation staff, tech support, and infrastructure personnel spread out across the country, and even a satellite office located in the midwest.  Everyone is an employee, and we don’t do any offshoring, but we are most certainly not geographically close to each other.If you’d asked me five years ago if I thought this would be a good approach to take, I would have rather emphatically told you no.  In fact, I resisted it pretty strenuously for quite a while.  You had to be a senior developer, having spent significant time on site (at least a year), and working remote was a reserved privilege.  While we had a few guys working remotely, it wasn’t the majority you see, so Bad Things couldn’t happen, but we still had folks dialing in right from the beginning.  And yet, in hindsight, it may be one of the factors that helps us squeeze more productivity out of our staff, helps them produce higher quality code, and allows us to get the leg up on competition.For starters, it forced us extremely early on to invest in systems, processes, and a way of working that brought everything we did online.  Project management, change control, bug tracking, issue tracking, source control, testing, collaboration, documentation, document management, communication, all of these things needed to be ubiquitous and consistently used by the entire staff.  If things weren’t accessible online, that meant Bob in Utah wasn’t going to be able to contribute, learn, participate, or even know about it.The second major factor that a distributed team gives us is a national recruiting footprint.  We’re not just going up against Acme Software in our back yard down here in Fort Lauderdale (South Florida has its own disadvantages for hiring technology workers), we’re getting to compete for the top talent across the US in every job market.  Our pool of potential applicants increases by an order of magnitude or more, which really amps up the talent level and allows us to be super picky.Third, I recently came across this article recently which was discussing some research from Microsoft, exploring traditional myths about Software Development, and they touched on the fact that distributed teams in their experience don’t have a negative impact on team performance.  They rightly point out that this flies in the face of a “one of the most cherished beliefs of software development” but they also illustrate how any worker would much rather talk to someone knowledgeable on their team 4,000 miles away than a less knowledgeable guy next door.  Makes sense, and it jives with our experience as well, but I can’t say I expected this outcome at first.Are there drawbacks?  Sure.  It’s nice to have everyone over for a barbeque on a long weekend, and that can’t happen.  It’s fun to walk by and joke with everyone while making the rounds in the morning, and that’s harder to do, but we still manage to interact a good deal as a team.  The flip side is it’s nice for the remote guys to be able to live where they want,  stay in touch with family and friends, and yet still have a great job at a fun company.  This really contributes to retention – we’ve had several guys move several times in the last few years, which I count as a “save” on losing an employee each time.If you’re considering running your organization’s software teams in a distributed fashion, here’s some things you’ll want to make sure you’ve got covered:

  • Excellent communication methods: cell phones, VoIP phones for extension dialing off the corporate network, private instant messaging network, email, and more.
  • Organizational Discipline: People in the organization need to understand that they will often be interacting with remote individuals, and that they can’t cherry pick projects to those who are in the office.  Yes, a phone call is not as nice as face-to-face, but often it’s more productive.
  • Team-Based Activities are Still Key: This is an easy one for us.  We play video games every Friday afternoon/evening.  Combination of shooters (Team Fortress 2) and other games (DoTA and HoN) and the games are part of the employee start up paperwork.
  • Everything Must be Online: Bug tracking, brainstorming, documentation, everything.  A major advantage this gives you is it’s a head start on preparing for audits or other certifications (SAS70, etc.) you might need to complete as an organization as everything will be easily accessible.
  • You Still Need to Be Involved: If you like to walk around and say hi to everyone each day like I do in the office, you still need to do it “online” via instant messenger or phone call.
  • Figure out if a Satellite Office Makes Sense: We found that we had roughly 5 people clustered in one city, so we sprung for a satellite office.  It’s a cheap thing to do and helps our recruiting in that area.
  • One Timezone: We work on US Eastern time.  You can live where you want, but you’re going to work that timezone.  This is critical, in my opinion and while it does mean the guys in California are up at 5AM, it’s not the end of the world and really helps keep things simple from a scheduling and planning perspective, and maintains the ability for quick communication.

It probably isn’t for every organization, but it’s really worked out for us, and it’s definitely something we’ve grown organically and will continue to improve.