I Actually Love Targeted Ads

About ten to twelve years ago, I found some free Tower Defense game on the App Store. It was challenging enough to keep me interested, but not so difficult I’d bang my head against the wall, and it became my goto game when waiting in the airport or when I wanted some mindless downtime.

It was free, but supported by advertising. This was the heyday of games like FarmVille. Fun Fact: In 2011, the developer of FarmVille, Zynga, made up 19% of Facebook’s entire revenue! So ad supported games were not only common then, they were considered the best way for mobile games to monetize.

After every Tower Defense level I completed, an ad would show as the next level loaded. Except these weren’t your normal ads. They were awesome. They marketed stuff that I actually wanted to buy. With each ad displayed, I could mark it as relevant or not, and as I fed back to the ad network (whose name I can’t remember anymore), the ads became even more tailored. I ended up buying more stuff from that game than I’ve ever bought from any advertising medium, ever. All of the stuff I bought was relevant, high quality, and I was pleased with every purchase. And here’s the thing – I actually enjoyed seeing the ads! Kind of like how I enjoy the ads in hyper-targeted magazines I subscribe to like Model Railroader – they’re all super relevant and help build awareness about products I might like.

At some point, the ad network folded, and the developer of the game went bust, and while I’ve moved on to other tower defense games, I often think about that ad network. I was using an ad blocker on my web browser even back then, and today, the the only real exposure to web ads I get is via YouTube, Instagram, and a couple newspapers I subscribe to like the New York Times. Everything else is blocked.

I have two YouTube accounts, and two Instagram accounts, and both have very focused personas and interests, but the thing is – the ads I’m served there just completely suck. They’re not relevant (marking them as such does nothing to change this), and at least half of them are retargeting from websites I’ve already bought something from (the ad shows up AFTER I’ve converted, for the first time). Perhaps worst of all, they are shown at a frequency that is just insane – like back to back or multiple times in a 2-3 minute scrolling session.

1.5 Billion in market cap and 163,000 employees between them, and Google and Facebook can’t even get my profile right. Perhaps most bizarrely, they don’t even ask me what I’m interested in apart from maybe inferring my interests based on the accounts/topics I follow or the videos/posts I like..

Why?

This reminds me of the age-old comment proffered by Folks Who Don’t Like Ads on the Internet. Why can’t we just pay a fee to get rid of ads on Facebook or Instagram or <insert social network here>? I used to ask this question, and it’s a good one. The answer is that in 2015, a Facebook user was worth $3.73 per quarter (roughly $15 bucks a year). Well, I’d happily pay fifteen bucks a year to get rid of ads and disincentivize all of the rampant fake news and destructive political advertising that channels through social media! Except there’s a problem – by the end of 2017, the value was up to $6.18 per user per quarter, an increase of 165% in to years.

In 2021 a Facebook user is projected to be worth $56 per quarter ($226 per year) and an Instagram user is worth $31.5 ($126 per year).  Note the vaguely disappointed tone of that article when discussing the fact that Instagram just hasn’t managed to generate as much value per user as Facebook yet. Tsk tsk!

That’s a growth rate of 15x in four years, and oh by the way, has significantly outpaced my desire (or ability, and, I’m guessing, yours too) to pay to remove those ads. Paying to remove ads will actually lose Facebook money, because their overall audience would get smaller, and they’re forfeiting the future growth they could generate off your account. In other words, you are just one single customer representing one sale of your account, but they can almost-infinitely sell you to almost-infinite ad buyers out there, so your account’s value will, over time, asymptotically approach infinity. Or so the well-vetted financial model somewhere says. Not understanding this dynamic, by the way, is probably why my beloved (yet forgotten) ad network went bust.

Back to the original point – the reason none of the major ad networks allow us to actually feed in our preferences is driven by the same desire – if I told Facebook/Instagram that I only wanted to see ads about Lego or model trains, I would have removed myself from the pool of folks who might potentially be interested in jeans or a TV show or whatever. In other words, I limit down the potential buyers for myself, and that’s the last thing they ever want to happen.

What’s the point of all this? Well, I think this is an increasingly dangerous game to play – intentionally introducing a bit of old fashioned friction and obfuscation between users and advertisers, for the sole purpose of maximizing future revenue growth. Ad blockers are now more common and available on more platforms (including mobile) and the reason is that all the ads suck! Advertisers don’t push the envelope here and demand well qualified eyeballs because that would skew their metrics too, and the reality is that online advertising is about the only real solid metric most marketers have. It’s a combination of “it’s in budget” and “it’s better than the alternative” thinking.

As we head into the worst global recession since we, uh, understood that we lived on a globe, I can’t help but wonder what’s going to happen when ad budgets get cut everywhere. Will some upstart come out of the woodwork that can actually show us some ads we want to watch, that take curation cues from us directly, and make everyone’s life on the internet better (except of course for Google, Facebook, Twitter, and the like)?

Probably not, but I wish they would. In the meantime, the great cat and mouse game continues between ad blockers and social media networks, and the internet just continues to get a little worse each day.

I guess in that way, it’s kind of like a game of Tower Defense.

Hello Turing Fest!

I still remember the first time I went to a Turing Festival event – Steve Wozniak was speaking at the Edinburgh Playhouse Theater. What a treat! I paid for my ticket, walked the few minutes from my flat (we didn’t have an office at that time at Administrate), and spent the next hour hearing from one of the pioneers of computing.

Since that afternoon in 2012, the festival has been an annual highlight on my calendar. I’ve also been able to get involved as a host, moderator, interviewer, and speaker, and have even managed to suggest speakers to the curation team from time to time. I’m really proud that based on some of my recommendations Edinburgh was able to welcome Michael Pryor (Trello and Frog Creek Software), Fred Destin (Stride.vc) and Eric Yuan (Zoom). I’m still holding out hope for the Dalai Lama and Eddie Vedder too.

Over the years, the Turing Festival has evolved substantially. Originally founded by the Coleman brothers, (the same duo that founded the CodeBase, one of Europe’s largest tech incubators), the event rebooted in 2016 with a new CEO, and a slightly adjusted moniker – Turing Fest. Now benefitting from full time, dedicated attention year round, the event began to grow into what it is today. This evolution mirrors the advancement of Edinburgh as a tech ecosystem more broadly, and underscores just how important Turing Fest is to the community here. For tech, this event is where Edinburgh specifically, and Scotland more generally, meets the world.

As a part of our growing community, Administrate has consistently sponsored the event every year, which is something we feel is important in and of itself, but it also means we can send a good portion of our team to participate and learn. This kind of learning opportunity is rare enough, but even more-so for it to be on our doorstep. I’ve recommended Turing Fest to countless local startups and if you’re in Europe and in the tech industry, I consider it irresponsible not to attend.

I have a love-hate relationship with the stage of the Turing Fest. Mainly because that’s where I’ve debuted some of my most challenging talks, discussing topics such as mental health, and the often unspoken challenges required to build a tech company. During last year’s talk on mental health, one which I was apprehensive to give, I knew the CEO of Administrate’s at-the-time fiercest competitor would be in the audience, which didn’t help the jitters! Afterwards he emailed me a heartfelt and touching note of encouragement, something I won’t forget.

One of my favorite memories was moderating a panel that included Gareth Williams of Skyscanner, Ed Molyneux of FreeAgent, Damian Kimmelman of DueDil, and Or Offer of SimilarWeb – I asked the question of how many WFIO (We’re Fucked, It’s Over) moments each of them had experienced. All of them talked candidly and vulnerably about the their experiences with multiple WFIOs, and a couple mentioned they’d had one within the last couple of months! Those responses were important for the audience to hear, but they were also important for me to hear, and I’ve reminded myself many times that failure is a normal part of the journey for every startup.

The atmosphere around the Turing Fest also includes many fond memories. I remember escorting Morten Primdahl, the CTO and a cofounder of Zendesk, through the streets of Edinburgh heading towards the speaker’s dinner, pummeling him with questions about their tech stack, their growth, and whether he liked this new product from Amazon called “Aurora” (he did, we do too, and we still use it!). I’m sure he was relieved to finally arrive and be rid of me! The impromptu drinks, dinners, and amazing stories that have been shared around Turing Fest have been opportunities to meet new friends, deepen relationships, learn, reflect, challenge myself, and grow.

I mention these anecdotes because all of them draw on key threads that make Turing Fest both unique and meaningful. Access to inspiring people, opportunities to share and broaden one’s horizon, and the power of serendipity when you bring a diverse group of people together are core to what Turing Fest is about. All of this set against the stunning backdrop of the city of Edinburgh is something that cannot be rivaled by any other event.

I was therefore thrilled when my good friend Brian Corcoran asked me to join the board of Turing Fest. We’ve already shared hundreds of hours of discussion about the event, and in some ways this seems like the formalisation of something that’s been happening for years. Brian has built an incredible team and a truly outstanding event, and I’m excited to help as we continue to build for the future. As usual, I’ve included my annual resign-every-year demand.

We’re on the eve of TuringFest 2019, and I can’t wait for another year of learning, connecting, and growing. I hope to see you there!

Moving On

Over three years ago, a good friend of mine asked if I’d consider joining the board of a local startup. They were very early stage, in the healthcare space, they’d raised a bit of funding, and were looking to grow.

Great. I meet with a lot of startups, mainly trying to help and provide a sounding board. Many of the founders are inspiring, many have reasonable ideas, some of them will go on to be successful, but it’s really rare that I personally get excited about any particular company. Certainly not enough to join and invest time, money, and energy!

But I owed my buddy a favour, so we met at the closest pub to my flat, and I prepared my gracious-decline speech. Except something strange happened. As Chris talked about what they were building, I became increasingly interested.

Current Health (at the time named Snap40) had designed a clinical grade (requiring FDA approval) wearable intending to replace most of the normal monitoring equipment found within a hospital ICU. This device could then be linked to a phone or tablet, and could help identify patient deterioration, perhaps even before a human would notice. There were other applications too – for example, my mid-sixties father crashed his bike recently, blacked out, went to the hospital and got an MRI, then was held for a further 24 hours for “observation.” In this scenario, perhaps he could have been sent home earlier as long as he was monitored for deterioration. Plenty of conditions could benefit from proactive monitoring at the home linked to a healthcare provider, and even more mundane challenges like medication monitoring (ie, did they take their blood pressure medication? Lets see!) could be significantly improved.

A major challenge for sales within the healthcare industry is dealing with a sophisticated and opinionated Decision Making Unit. Doctors, nurses, pharmacists, administrators, and technical people all factor in, and in a new category educating this market can be really time consuming and expensive. Would healthcare professionals get this? I called a few physicians I knew and asked. They immediately got it, several of them referenced having this idea themselves, and one of them spent more than an hour telling me all the different ways it would save their hospital and health network money. So that seemed promising.

One of the key things that attracted me to this implementation is the wearable is a “dumb” device. Raw signal is sent to the cloud where it’s analysed and processed using a combination of machine learning (yes really) and other algorithms. This means a couple of things – the algorithms can be tuned using all data ever collected from every device out there, and new algorithms can be shipped without replacing the devices. In other words, the more data collected, the higher the accuracy.

This is a crucial distinction: Current Health isn’t a medical device company, it’s not a software company, it’s a platform company.

I joined the board in March of 2016, with one condition – I would resign every year in March. Chris could accept or reject my resignation. While I’ve been a member of the board at Administrate since I joined in 2011, I’d never held a board position on another company before. Maybe I’d suck at it. Maybe the company would outgrow my expertise. In any event, they wouldn’t have to wait too long to get rid of me if that became necessary.

“Is it going to work?”

For the first two years, this was the major question. Pilots at local hospitals were really promising, and we obtained a European CE mark which meant we could sell it within Europe, but we needed to get certified by the FDA to tackle the USA. This was tricky, because the FDA had never certified a hybrid device like this before.

The next three years (and three resignations) passed quickly, and I’m super proud of what we achieved. A CE mark, multiple pilots, FDA certification for use in both the hospital and home environments, recruitment of a fantastic team, an office move, a rebrand, and multiple funding rounds, including one of the largest seed rounds ever raised in Scotland.

But perhaps the achievement I’m most proud of happened recently during an extended pilot project in the United States – we identified a patient’s vitals slipping, and alerted staff to the issue. The traditional monitoring machines didn’t notice, and we saved a life. The first of many I’m sure.

The traditional monitoring machines didn’t notice, and we saved a life. The first of many I’m sure.

Current Health is going to fundamentally transform how we receive and provide healthcare. I’m super excited for the future, but I also realise that the time has come to step down from the board and make way for the next stage of growth. To the disappointment of most, there’s no drama here, it’s simply time to move on.

I remain deeply thankful to Chris (and my pal who introduced us!) and the team at Current Health for giving me this opportunity. I learned a ton, got to work alongside some incredible people, and I believe this experience made me a better CEO – sitting on the other side of the table can be really enlightening! I am extremely excited to watch how Current Health will grow and I couldn’t be more optimistic about their future.

How to Solve Sonos Playbar Speech Issues with the Apple TV

A couple months ago, I purchased a Sonos Playbar to replace a Bluetooth Samsung Soundbar I had, as I wanted to integrate the living room audio setup with the rest of my house, which is all Sonos.

Probably due to over-exposure to loud music from playing in rock bands through highschool and college, I really struggle to hear conversation in noisy environments like pubs, and this issue extends to hearing speech in movies unless I’ve got a dedicated speech channel. My hearing seems fine overall (famous last words I guess), just differentiating speech within noisy backdrops is annoying.  Modern flat screen TVs really exacerbate this issue as their speakers aren’t great anyway, let alone for pushing out a nice clean speech channel.

The sound you get from the Playbar (and two Play 1s for surround sound) is great for music, but the results I’ve had regarding speech during movies have absolutely sucked.  It’s kind of surprising when you start seeing threads like this one where there’s dozens of people complaining for years, with no real response / solution.

Here’s how to solve this situation:

You need to bypass any potential pre-processing that might happen to the audio signal.

If you have a receiver it’s probably a safe bet your signal is clean, but like most people, I just have a TV and a bunch of HDMI inputs.  I had them plugged into my Samsung 4k Smart TV and I had the optical audio connected from the TV to the Sonos Playbar.  I had the TV correctly configured to bypass any audio processing and it was only shunting the audio out to the optical port.  Except it wasn’t.

To inexpensively solve this issue:

  1. Buy an HDMI splitter with an audio optical out, like this one.
  2. Hook your AppleTV (which doesn’t have an optical out) and any other devices (BluRay, etc.) into the splitter.  Hook the optical from the splitter into the Playbar.
  3. Turn on the Speech Enhancement on the Sonos App and Night Mode.
  4. You’re done! You now have clean audio and a functioning Dolby 5.1 signal that has clear dialog.

I’d like to point out here that the true villain of this story is Samsung for essentially lying about the pass-through capability of the TV.  Sonos should bear some blame here too – it could easily better educate customers and include some common troubleshooting advice either online or in the documentation.

Hope this helps!

How to Ask for Advice / Feedback About Your Startup

One of the things I’m passionate about is helping other startups and the community of entrepreneurs we have here in Edinburgh (and in Scotland).  Since becoming more intentional about “taking the pledge“, I’ve been meeting with lots of folks locally, and been surprised by the amount of requests!

So much so that other team members here at Administrate are helping me shoulder the load, according to areas of expertise (thanks Mike and Patrick!) and time constraints, and I know of many others in the community who are donating their time and expertise.  Helpfulness and support has always been a hallmark of the Scottish startup scene, so this isn’t anything new, but there’s so much more activity now, so many more companies, and so many more entrepreneurs now!  It’s great to see!

I’ve found that sometimes people don’t know what to expect, so I thought I’d lay out a brief framework to help everyone get the most out of the time.

  1. Remember that most advice is delivered within a context vacuum.  Don’t take my advice (or anyone else’s) without fully thinking things through and satisfying yourself.  Bad advice can come from really great people.
  2. In order to be at all helpful, I need context.  Things I usually ask about are: the problem you’re trying to solve (as a company), your business model (SaaS, etc), your market, some metrics around revenue, customers (people paying you money), team size, how long you’ve been going, growth, and churn.  It’s ok if you don’t have all of this information, but the quicker we can rattle through these items, the faster we can get up to speed.
  3. It’s totally cool if you just want to chat, but I’ll usually ask you what you’re biggest challenges are – we have these at Administrate and sometimes they feel cyclical (first we’re worried about sales, then tech, then support, then sales again, etc.).  Even if everything is going well, the question will often be “ok, how do we double down and make it even better?”
  4. I probably can’t help you too much with hiring (particularly “line” staff) – my network is mainly in the USA (so not local), and we’re in high growth mode here at Administrate, so if I know of any devs or whatever we’re probably going to hire them!
  5. Expect me to be very, very blunt.  If you’re British it may come across as almost hostile sometimes.  Sorry.  When I get into problem solving mode or analysis mode, I tend to interrupt, ask lots of questions, and don’t filter much.
  6. Expect me to play devil’s advocate.  Expect me to really push you on a few things.  Expect to be challenged.  The best advice I’ve ever received was from someone telling me they thought I could be a lot more ambitious, which annoyed me at the time, but really made a difference.
  7. One thing you won’t get from me is griping about raising money in the UK, finding a team, or complaining about Scottish Enterprise or Scottish Development International.  If you’re annoyed about these things, fine, but expect an argument from me!
  8. I’m not going to be very helpful to you with introductions to angels, VCs or syndicates.  These people all make their own decisions and won’t look at you in any different light if I make an intro for you.
  9. I won’t share anything about our conversation unless you specifically tell me you don’t mind.  I also expect the same in return.  This means I don’t mind if you want to ask me about challenges I’m facing now, etc.  We like to be transparent, and often it can be comforting to hear that someone else is going through something you’re struggling with.
  10. The majority of my experience and expertise is in high growth Business-to-Business Software-as-a-Service.  So be aware I’ll bias towards that style of company.  I don’t like most B2C ideas because they are riskier, require more funding earlier, require a lot of traction to be successful and are often harder to build and/or monetise.
  11. A couple of times things have gotten emotional (really!).  That’s OK! Building a business can be really hard.  Relationships are involved. It can feel overwhelming.  That’s normal.  Don’t be embarrassed.  It’s not the first time.
  12. Unfortunately, you may have your appointment changed around a few times.  Sorry, but Administrate comes first!  Also, it may be awhile before we can meet, and depending on what you’re looking to talk about, we may provide someone else from our team to give you a better perspective.

Hopefully that helps you get an idea of what to expect and makes everything run just a bit smoother!  I’ve enjoyed all of the conversations I’ve had and am always encouraged by the amazing people we have in Edinburgh working away on building things and solving problems.

HSBC Anti-Fraud Measures Vulnerable to Phishing Attacks

I’ve been an HSBC customer for roughly two years and have complained about these practices probably more than a dozen times without any real acknowledgement or change.

HSBC, like most banks in the UK, provides every customer a 2 factor security token to make sure that logging in requires something you know (your password) and something you have (your token’s time-limited code). So far so good. They even have a signing procedure for sending money (personal accounts only, strangely) that requires you to hash the transaction amount with your token, and put in the corresponding code as part of the transaction. A nice touch.

A Horrible Anti-Fraud Algorithm

Where HSBC has a glaring security hole is their fraud detection and prevention. As near as I can tell, the HSBC fraud algorithm is essentially “IF online transaction AND/OR foreign origination AND/OR amount is greater than [some nominal amount] THEN fraud”.

There’s no history taken into account, and they routinely ignore travel advisories called in ahead of time. So for example, if you signup to a monthly recurring charge for a software service outside of the UK (many of them) it doesn’t matter that the charge has occurred every month for a year, they’ll still pop an alert. This is particularly problematic for subscriptions billed annually, as it seems that their fraud team can perform a manual override on most things, but not on infrequent subscriptions.

An Outsourced Anti-Fraud Team

Foreign can often mean “somewhat far away” too. There’s no concept of accepting a card present, pin verified transaction as maybe worth the benefit of the doubt either, so I’ve had my card declined for sandwich shop purchases in towns less than fifty miles away from my home address.

A Dangerous Fraud Verification Process

But all of this could maybe be lived with, if not for the horrific fraud verification procedures employed by one of the largest banks in the world.

Here’s how it works:

  1. Their horrible fraud algorithm pops an alert. This can occur on a subscription charge, a pin verified card present charge, or a verified by visa online charge.
  2. You will receive a phone call from an unknown number (blocked).
  3. An Indian call centre rep will tell you they’re calling from HSBC and they’ll need to verify some important information before they speak to you. So far, this has been my birth date or post code (enough to get your full address in the UK). If you refuse to speak to them, your card is blocked. If you call them back, it will take roughly 25 minutes on hold being transferred around to clear the block.
  4. If you don’t answer the phone, they will leave a voicemail. Again, an Indian call centre rep will preface the voicemail with an urgent request for contact, then they will play a recorded message. Often there will be a problem here, and they’ll have to call back to get the recording playing right.

It’s important to note that the phone connection quality is invariably terrible. This means you can’t understand the person, and the recording is garbled as well.  I also don’t care if the representative is a native English speaker or not.  I do care that the cut-rate policies of HSBC mean they have chosen an outsourcing provider who can’t seem to get decent phone service, thus making the entire thing more vulnerable to phishing (it’s easy and cheap to sound just like HSBC or even worse, do a better job by having a nice fluent British accent  via a clear connection).

An Active Phishing Threat?

For the past two years I’ve been uncomfortable with this process. It leaves you open to phishing attacks, particularly spear fishing. It relies on data that could be publicly obtained fairly easily (birthdate and post code) and even if you can’t get this data, you can easily phish it by impersonating a representative then using that information to escalate privilege elsewhere.

A better solution would be to mimic the procedures in place at other banks. Chase for example has an app that will securely message you the details of a questionable transaction which you can approve or deny in a few seconds. They’ll also SMS you the details of the charge, which you can respond to. Lastly, if you don’t have a smart phone or can’t receive text messages, they’ll call you and use a challenge response procedure to verify yourself, or leave you a voicemail instructing you to call the number on the back of the card. The entire process is safe, easy, and quick not matter what mechanism you use (I use all three depending on the situation).

It appears that these weaknesses in HSBC’s procedures have now caught the attention of others. A couple of days ago I was called by someone who mumbled around that they were from HSBC, and asked me for information. I declined, like normal. I called the bank back like normal. Except this time, nobody from HSBC had called and no suspicious charges were flagged. Apparently someone had attempted to phish me! I wonder how long it will take HSBC to address this now that their customers are being actively targeted?

Lets review how we got here:

  1. A poor anti-fraud algorithm means false positives are common.
  2. A lot of alerts means you need a large customer service staff.
  3. You outsource this, you don’t automate it, and put in place procedures that are fraught with security problems.
  4. Your frustrated and desensitised customers lose respect for the process.
  5. Phishers take note, and begin to capitalise.

HSBC Customers, Be Careful!

Be careful out there! Never give any personal information to anyone calling you, no matter who they claim to be and no matter how annoying the procedure is.

Cynical Optimism: Technical and Business Planning

I thought Rand Fishkin’s recent blog post on “Cynical Optimism” was a nice read.  He talks about how while there are plenty of things to be cynical about when it comes to humanity and our tedencies towards negative things, there is plenty to be optimistic about with regards to our progress as a whole.  The phrase “Cynical Optimism” is one that I really like to use when describing how to attack business plans, budgets, technical roadmaps, or other kinds of planning.

First, Be Optimistic

When setting goals, you definitely want to be an optimist.  Aim high, don’t limit yourself, and always strive for accomplishments that are meaningful and aligned with your values.  This is the classic “CEO” way of looking at the world and deciding where to go – strategy, vision, and confidence are huge assets here.  When goal setting, make sure you show your work!  Define goals in the form of “We’d like to do X because of A, B, and C”.  This provides important context and you’ll find that there are often other cheaper better routes that could be had which your haven’t considered.

Second, Become a Pessimist

Once you’d laid out your goals, make sure you switch hats and cast an incredibly cynical eye over your plans.  You want to identify everything that can, will, or should go wrong.  This is the perspective that a “COO” or “CTO” would take, as they’re the ones seated more firmly in the trenches.  The important thing here is to engage your team and let them know it’s OK to second guess goals in the context of determining how they’ll be achieved.  By critically assesing what it will take to arrive at your destination, you’re ensuring you don’t run off the rails enroute.

Now You’ve Got a Plan

Forcing yourself to wear both hats is hard – it’s often difficult to pull yourself across the chasm if you’re naturally predisposed to one outlook or the other, but if provides the following:

  1. Builds a culture of intellectual honesty.  It’s always easier in a team environment to just go along with the flow and feel like you don’t have any skin in the game.  If your team feels they can object or hone objectives, they’ll perform better.
  2. It can reduce the risk of making major mistakes.  By critically attacking your objectives you’ll anticipate problems and avoid major pitfalls that could have been forseen.  You’ll never know what you don’t know, but often teams drift into problem areas they could have avoided.
  3. In dysfunctional organisations, it’s amazing how almost everyone involved will know (and be able to point out good reasons) how goals won’t be achieved, well ahead of time.  You’ll prevent this kind of “death by politics” syndrome which affects a lot of companies.
  4. Bottom up planning is always the best way to meet top down objectives.  In other words, the high level goals can be set by the product owner, CEO, or visionary, but they’re on the worst vantage point to actually see how to go about achieving these things.  A tip on how to encourage realistic plans – don’t confer time estimates of any kind when setting strategic goals.  Just say “We’d like to do X” and see what comes back!

Lastly, Remain Engaged

Plans sometimes need to change.  You’ll need to react to new things.  As your team engages with the problem the goal-owner will need to remain intimately engaged with the team.  Fine tuning your goals is a necessary part of any meaningful project or endeavor – not fine tuning will just ensure failure.

This was My Brand Too!

Recently I’ve had two really dissapointing experiences with companies that I’ve admired and sought to emulate.

One of them I’ve admired for something like 12+ years.  If you asked me who the top companies in the world were, unequivocally, I’d list this particular outfit.  I loved their philosophy, their marketing, their service, everything.  I told people the way I felt as well.  The other company was a fast growing outfit who conquered their industry and was an inspiration to me at every step of the way.

Both of these companies have clearly lost their way and it’s a cautionary tale for those of us who are running, growing, or seeking to start out on something new.  The weirdest part of it is that I feel as though heroes of mine are gone.

These companies were my brands too!

How could this have been prevented?  What can we learn?

  • Don’t overreach – both companies broadened their product line to the point that they were doing too many things.
  • Don’t ignore the small stuff.  Things like consolidated invoicing don’t seem like a big thing in the developer scrum, but they’re huge to customers who are probably using all of your products because they love you.
  • Don’t underestimate the power of a financial credit.  Several times along the way a discount or permanent waiving of fees for what was admitted to be substandard service would have set things right in my mind.
  • Don’t ever tolerate rudeness to customers by you staff.  If this happens, get the staff to seak out the customer and apologise.
  • Don’t blame your failings on a third party.  It’s your fault for introducing the third party – third parties mean more responsiblility for you, not less.
  • Don’t allow tickets to wallow unresolved for months.  This just festers the entire situation.

At the end of the day, I won’t be using these providers as much anymore, and that’s sad, because I truly loved both companies.  We’re probably all guilty of a some or all of the above at some point, but it’s how we respond that matters.

Here’s How to Salt Your Own Passwords and Prevent a LinkedIN Style Password Problem

With all of the publicity surrounding LinkedIN, League of Legends, and possibly others, I thought I’d take a moment to explain how I manage passwords.  Yes, I quickly changed my password on LinkedIN, but using this method will add you just a bit more security if and when a provider screws up.  Remember, not every security incident involves a massive “post to everyone’s wall and make the evening news” style announcement!

I started this method after a screwup at Reddit a long time ago, which back in the day was storing plaintext passwords, and leaked them.  It has worked extremely well over the last six or so years.  For those who are unfamiliar with salting – it’s a method to increase randomness of passwords, and prevent rainbow attacks against password databases.  Technical readers may point out some more particulars about that statement, but the general statement should hold true.

Websites and other authenticators are supposed to salt passwords, but they can forget.  You can salt your own passwords by providing a hard to remember base password, then add some random characters to the beginning, after a fixed position, or end of the password.  This will make deriving your password more difficult, and will prevent (quick) account theft by an attacker taking your email and password and trying it at well known sites.

  1. Choose a decent password.  I recommend that people choose a nursery rhyme, favorite quote, saying, bit of religious text, or any kind of phrasing and choose the first letter or second letter of each word from that phrase.  This usually gets them a reasonably strong, easy to remember base password.  Capitalize a few of the letters and add a number at the beginning or end.
  2. Develop a salting mechanism. For every website that needs a password, develop a salting mechanism.  Have two rules of thumb as to how you derive a few additional characters based on the site itself.  Try to choose something relatively non-volatile.  For example – use the first four letters of the company’s name as it appears on their logo.  Or the last three letters of their domain name.  Or something else that may not change very often.  Then add these letters to your passphrase above at the beginning, middle or end of the phrase.  Why two rules?  So if your first rule doesn’t work due to some password scheme restrictions, you can use the second.
  3. You’re done. You now have an easy to remember passphrase that’s unique to that site (your base password + your derived salt).  Best of all, your password looks fairly random and even if your password from another site was stolen, it wouldn’t be susceptible to rainbow attacks, etc.

A few benefits of this approach:

  • You don’t need to rely on a “last pass” type application or system that stores all your passwords on a computer which may be hard to access sometimes.
  • It’s easy!
  • It’s free!
  • It really works – I’ve never had a problem with this, even with some of those sites that have weird password rules.
  • You now have a unique password to every system you access.
  • You can change the base password every year if you’d like, for even better security, and to weed out those accounts you forget about / never use.

There are no silver bullets to any of this.  Yes, this can still be attacked, but it’s certainly better than the “same password everywhere” or the “secure password for sites I care about and insecure one everywhere else” method.

I highly recommend you turn on Gmail’s 2 Factor security setting on your email account as well.  Particularly now that they’ve released an “offline” authenticator token generator which doesn’t need network access to work.

Stay safe out there!

I’ll Probably Never Hire Another Pure SysAdmin

NOTE: Updated Oct 17, See Below

This is a thought that’s been percolating around in my head for the last year or so, but has recently become even more crystalized: I’ll probably never hire another Systems Administrator.  A corollary to this thought would be: if you are currently a Systems Administrator or want to be one, you need to seriously begin planning on how to manage a career that will be mostly deprecated within the next 10 years.

Take a look at the current state of the art in cloud computing:

  • Spin up a server at your favor cloud provider (AWS, Rackspace, etc.), then use Puppet or Chef to deploy your software stack.  Now you’re done.
  • OR, Spin up an App at your favorite cloud platform provider, then push your code out using Git.  Now you’re done.
  • For both solutions, plug in some off-the-shelf monitoring, and you’re operating.

What’s missing here is the configuration, setup, provisioning, doc writing, black magic and/or prayer of setting up the software, hardware, and getting the code running that used to be the domain of the Systems Administrator.  In just a couple of years, deploying a web application has now become almost identical to deploying a desktop application – instead of an installer we’re using Git or Puppet/Chef. Instead of a customer’s computer we’re using a cloud platform or cloud server.

There’s plenty still to do on the networking side, but that’s headed in the same exact direction due to the same exact reasons: we want to be able to clearly define and programmatically execute the deployment of complex networks, just like we can with complex server offerings.

All of this falls under yet another buzzword: Dev/Ops.  Just like the cloud, we’re seeing this being adopted by smaller, nimbler organizations that are focused on web products, but the trend is clear, and there’s really no benefit in doing things the Old Way.  Even if you’re still running your own physical metal servers, you’re going to want to make sure that your own datacenter can leverage this type of workflow.  Now, the watchword to the development team is: it’s not done until I can one-click deploy it.

The laggards on this will be those industries that have regulatory or legal hurdles to overcome with using cloud services (read: healthcare) or the very large companies with services and technology that’s dozens of years old with no migration plan.

SysAdmins and future SysAdmins, you need to figure out where you’ll live in this new workflow.  Probably in the margins around monitoring or desktop support.  Possibly serving as the gatekeeper in a sort of “operations Q/A” role.  Expect small companies to have SysAdmin openings dry up over the next 5-10 years and get prepared.

Updated October 17: Hello Reddit/r/programming and Hacker News!  I wanted to take a few minutes and respond to a few themes that seemed to pop up in comments on HN and Reddit.

  1. I’m not saying Sysadminning is dead – just that the role is quickly changing.  Seems like a lot of people (anecdotally, many Sysadmins) thought I was saying the entire profession is dead.  Yes of course we’ll still need Sysadmins on some level, but the crucial difference is that for many areas of a business these needs will be less and much much different.
  2. Software development is changing too.  On complex deployments, developers can’t absolve themselves of the responsibility to design infrastructure considerations into the solution they’re building on the front end.  It’s a scary thought to think that organizations are out there that don’t have this level of partnership between ops and the devs.  This is why the puppet scripts should be written first and deployed on a test environment that’s identical in as many ways as possible to the ultimate operating environment (another benefit of using the cloud).
  3. Of course, any more complex deployment will need devoted SysAdmins, but like I said above, the skillset and day-to-day job will be dramatically different when wrestling with hundreds of servers instead of dozens.  More and more programming will become the norm and more and more upfront input into the solution will be an absolute requirement.
  4. I received a very thoughtful email from a former SysAdmin of mine (previous company) who pointed out that the job is much more along “system integrator” lines now, and that the internal vs. external network distinction is essentially going away.  I agree.
  5. Whenever your’e generalizing, counter examples abound.  Sure big companies and certain computing environments will still do things the Old Way but I’d challenge readers to objectively think if most business decision makers really want to hire someone and run their email server internally or just pay Rackspace/Google/Whomever to do it and worry instead about their money-making applications.  Even those organizations that need their clusters in house will invest in tech that allows them to mimic cloud operations on their own bare metal infrastructure.
  6. A couple of amusing anecdotes – the comments on HN immediately became more positive after a well known commenter defended the post, and a Googler chimed in as well.  That’s when the upvotes really started coming it seems.  On Reddit, the story was quickly downvoted!  Most users chose either a “genius” or “idiot” assessment of the post.  No real middle ground.