Jeopardy and Watson

We had a great time last night watching Watson take on two humans in a round of Jeopardy.  Or at least, I had a great time.  The wife and her sister weren’t quite as into it as I was, but they watched it just the same.Here’s a recap:

  • The show did a great job explaining what was happening (they burned half the episode on explanations).
  • It’s interesting how most people don’t understand what the true challenge of this event is (even techies) – Watson has huge volumes of information (he knows a lot of stuff), but the real challenge is understanding the meaning behind a question.  In other words, it’s an understanding/comprehension challenge, not a fact challenge.
  • IBM came up with a really neat tool that showed the audience how Watson was playing the game.  They would show the top three answers he came up with and a confidence interval.  Watson would buzz in with the highest rated answer that crossed the confidence interval.  If none of the answers made it across the threshold, he wouldn’t buzz in.
  • Alex Trebek gave a tour of the datacenter which had ten-ish racks of IBM servers.  The size of the install was very surprising to our non-technical viewers.
  • Watson glows green when he’s confident in his answers, and when he gets one wrong, he glows orange.  This feature was a big hit at our house.
  • Two perfect examples came to light exposing the difficulty of this challenge.  One question made references to the Harry Potter world and a dark lord who challenged him.  It was clearly a Harry Potter question due to the contextual clues, but the answer was “Lord Voldemort”.  Watson answered “Harry Potter”, but his second choice answer was “Lord Voldemort”.  A human who understood the meaning of the question would never have answered in that way.  The second occasion involved Jennings answering “the twenties” to a question, which was wrong.  Watson buzzed in right after him and answered, “the twenties,” which no human would ever do.

One question I had was if the text transfer of the questions happens in step with Alex Trebek’s reading of them.  Does it happen character by character or does Watson get a few precious seconds while humans are reading the screens?  Conspiracy theorists would probably ask how Watson’s first choice was an 800 dollar question (unusual) and he hit the daily double immediately, but it could be part of the IBM team’s strategy.All in all, that was probably the most fun I’ve had watching a TV game show.  Looking forward to the next two episodes.

Jeopardy, Artificial Intelligence, and a Chess Playing Robot

Tomorrow Jeopardy will feature a computer facing off against former champions Ken Jennings and Brad Rutter.  I have been looking forward to this for a couple of months since I first heard about it, and it is an amazing feat of computer science that a computer can go head to head with humans on a game show as complex as Jeopardy.  IBM is the true leader in feats (stunts?) such as these having staged previous competitions against former chess master Garry Kasparov.  The show has already been filmed (in a special sound stage at IBM’s location) but I do have an inkling of what the engineering team probably went through prior to the event.

Artificial Intelligence

My freshman year in college, I managed to convince my professors that I could and should skip the computer science prerequisite for everything, COS120.  This put me into a data structures course in the first semester of my freshman year, and several linked lists and b-trees later, I was able to take any class I wanted in the spring.  I made  bee-line for a class called “Intro to Artificial Intelligence” as I was more than intrigued by the possibility that I could build Skynet and cause the human race its final doom.  Sweetening the pot was a rumor circulating the labs that the class would make use of the very recently released Lego Mindstorms.  Fulfilling the dreams of nerds and lego aficionados (with there being an admittedly strong correlation between the two groups) the Mindstorms were fantastically expensive to a college kid (several hundred bucks) and required programming knowledge to really make them hum along.  This AI class in other words was my ticket to Lego Nirvana, not the other way around.

Legos and LISP

I managed to sign up for the class and was one of (I think) two freshman.  Having spent high school and the fall semester doing work in C and C++ along with learning Perl and a new fangled language called PHP, it was quite a surprise to show up for class and begin learning LISP.  For those of you who are non-techies, LISP is a complete paradigm shift from how most programming languages allow you to express yourself.  The analogy might be like going from sketching in black and white to being handed multiple colors of modeling clay and being told to sculpt in three dimensions.  LISP is the oldest programming language still used in a mainstream capacity today.  LISP treats data and the program the same, so you can build a self modifying program which is one of the reasons Artificial Intelligence researchers and applications use it so much.  LISP has a weird looking “list based” syntax (LISP stands for LISt Processor) and allows you to solve weird problems in fairly elegant ways.  In fact, some like it so much and feel it is so powerful that LISP is the reason that Paul Graham (a well respected computer scientist and technology entrepreneur) credits for his business success. He felt he was able to elegantly and more efficiently out-build his competition simply because of their language choice.  I don’t really agree with Graham, but the point is that LISP is a very cool and very different language that was a pleasure to learn.  This class remains the only one in college where I did every single homework assignment.  The problems were just too fun and the ways of solving them were extremely different. Along with LISP of course came the Legos.  Almost immediately, we began building simple Lego robots that would following a line on the floor, be able to navigate a room, follow a light source in a dark room or navigate a maze.  It was unbelievably cool and a testament to Lego’s design prowess that their system allowed for such flexibility.  Each kit provided a command unit and several motors and sensors.  There was a touch sensor, light sensor, and a few others, and the command unit would download the programs via a cable hooked into a computer.  There was a Lego provided GUI-based way you could program the processor, but hackers had already provided several other languages, and we used Not Quite C (NQC) which was very similar to the C programming language with a few restrictions.

A Chess Playing Robot

All in all, we were having a blast, and just like kids get bored with a slide and start daring each other to go down backwards, or standing up, we began competing to build a bigger better robot.  Our professor was also the chair of the department, and he sensed he had something brewing, so he got the class together and challenged us to come up with a project that could bring fame and glory to our small liberal arts university.  A robot that could get us soda from the machine!  A robot to gather up homework assignments and drive them back to the professor!  Not as popular as the food or waiter suggestions.  Finally from the back of the class came the ultimate idea: a chess playing robot!  A senior in our class had been working on an independent study in addition to taking this class, and had produced a fairly workable Chess simulator, written in LISP.  The class began buzzing with excitement.  We could build the robot, then challenge students to beat it! I started to get a bad feeling about the project – building a Lego chess playing robot would be quite the undertaking.  My good friend Aaron Williamson upped the ante – “I could get the university TV station to film a match!”  Uh oh.  A third student offered to get one of the most popular professors in the school to face off against the robot, and just like that we had ourselves our very own TV spectacle: Man vs. Machine.  Nerds vs. Normals.  Technology (us) vs. Philosophy (the professor).

The work was divided up and we immediately had to start pulling late nights as we only had a few weeks to get everything together.  There would be a vision processing team (the robot had to know where the chess pieces were), a robotics team which would build the gantry crane and arm mechanisms, and the chess software team.  The Public Relations we soon learned, would take care of itself.  I was on the vision team, and our job, we felt, was quite possibly the hardest.  This was 2000, and web cameras were novel, low resolution, expensive, and rather rare.  At least, they were expensive for college students, so we used the only one we had available: a web camera attached to a Silicon Graphics O2 workstation that provided a 640×480 resolution color picture.  It provided enough resolution so we could film the board, and our algorithm would take one picture, save it, then take another after the human moved, and compare them to determine which two pieces had moved.  This seems pretty trivial, but it was complicated by a fisheye effect from the lens, and the fact that the robot arm (or human) wouldn’t actually place the pieces very accurately. Lighting and other conditions could also change depending on where the human was standing or for a host of seemingly random factors. As we started to work, even simple things seemed to derail us.  Silicon Graphics apparently used a proprietary RGB format which was reverse of standard RGB, so we had to write a converter.  Irix, the SGI OS, turned out to be a pain to develop for, at least compared to Linux, so we performed the processing on another computer.  This meant we also had to set up the workflow between the IRIX machine which captured the image, and the Linux machine which processed the image, and then interface with the robot and the chess playing program, which ran on another machine. The date for the campus wide demonstration was looming, and we were determined to not be humiliated.  The student newspaper ran a feature, and our opponent was quoted that “no hunk of metal would best him” to the delight of the student body.  

Media_httpwwwpeebsorg_gamba

Finally, the day was upon us, and we began preparing the lecture hall which would be ground zero.  We had lots of equipment to move from our basement computer lab to the lecture hall, and as we began calibrating everything alongside the film crew which was also setting up, we noticed that the vision system wasn’t working.  At all.  It couldn’t find the edges of the board, couldn’t determine the grid, and it couldn’t see which pieces had moved.  We were in trouble – the fluorescent lighting conditions of the basement worked but the incandescent lighting and TV lighting in the lecture hall didn’t and we needed to recalibrate everything.  Working like mad we began tracking down all intensity assumptions in our program and moving pieces around until finally we got it working. The format of the competition was designed so that while the computer was thinking we’d explain to the audience what was happening, and our opponent would also discuss the philosophical ramifications of sentient machines.  This was designed for another, secret reason – if something went wrong, we wanted to be able to troubleshoot without drawing too much attention.  We had a backdoor programmed into the chess program which could reset it if there was trouble, or in the event of a catastrophic failure, we could take over and manually play out the match.  The ethical dilemma of should we tell the audience what happened if that were to come to pass was hotly debated.

The clock struck 7:15, in came the audience, and on came the red blinking lights of the video cameras.  The lecture hall was packed out, and the professor did a great job making jokes as we introduced the event and briefly explained the rules.  Our class professor could have died of happiness at that moment.  As we began the competition, the professor picked white and made the first move.  “Take THAT you cold calculating MACHINE!” he proclaimed to resounding applause from the audience.  That pissed me off.  I had been championing a feature where the robot would insult and taunt its opponent with every move, but was shouted down for not being classy enough.  We had to endure a night of insults that could have been combatted and we all admitted in hindsight that we should fought fire with fire. At first, everything looked like it was going well.  The robot was working, it could see what was happening, and the chess program was making good decisions.  We started to relax a little, and the audience was clearly intrigued as the gantry crane would slowly move to the right spot, lower its arm, then grab the piece and move it.

Then disaster struck.  Just a few moves in, our opponent performed an En Passant which while being a rather esoteric move, also has the rather interesting distinction as Wikipedia puts it, as “the only occasion in chess in which a piece captures but does not move to the square of the captured piece.” Uh oh.  Our vision system was designed to look for what changed and was looking for either a move or a capture, not a weird combination of both.  Our program chose the two best coordinates to send which caused the chess program to have the wrong board information, and we watched in horror as the gantry crane moved into the wrong position, knocking over several pieces as it vainly swiped in air to make its move. “Uh oh, Houston, LOOKS LIKE WE HAVE A PROBLEM!” yelled our opponent as the crowd roared its approval, and he immediately launched into a discussion about how pure logic will never overcome the malleability of the human mind. It was chaos as we raced to reset the board, recalibrate the vision system, and instruct the chess program of the true state of the game.  I believe we had to simply reset the game state completely, in essence telling the computer that the game had started with the board situated the way it was since it didn’t understand an En Passant.  Things were looking up as the computer managed to make a move and actually captured a pawn to the crowd’s approval.  

For the next fifteen or so moves we each took turns giving a short talk about how the robot was built, the technologies we used, and the challenges we overcame. As the evening wore on, the game developed nicely.  I’m not a chess player but I could tell that the computer was holding its own and it could already beat everyone in the class, so we were optimistic about our chances.  Still, it was roughly an hour into the game and even though I wasn’t following every move closely it was a surprise when all of a sudden the professor announced “Checkmate!”  I looked over at the board and it was indeed checkmate!  However, the computer didn’t agree.  Turns out that either during the initial setup of the board or during the insane scramble to fix our En Passant adventure, someone had incorrectly placed the King and Queen on the computer’s side.  The computer had been defending its queen instead of its king, and thus had fallen prematurely. A disappointing end to what would end up being one of the most memorable projects of my college career.  Still, we couldn’t help but feel proud of being able to build something unique and stage a public spectacle of sorts that everyone seemed to enjoy.  Tomorrow as I tune into watch Watson and the team from IBM, I have a small sense of how they’ll be feeling, and win or lose (there are already rumors leaking out as to the results) it is an incredible achievement.  Congratulations and good luck!

Postscript:

10gen’s Mongo Atlanta Conference

Just spent a very informative and interesting day at 10gens’ Mongo Atlanta conference (#MongoATL Hashtag).  For a one-day conference, the event seemed to be very well planned and executed, and I feel like the event (and it’s sister events) is a great one to attend if you’re using or planning on using MongoDB.Held in the extremely nice Georgia Tech Research Institute Conference Center, the event consisted of several speakers talking about how they’re using MongoDB, or how to tune and administrate the database.  These were fairly technical talks as most had code up during the presentations or were running commands from an interactive shell.  The audience participated eagerly during most talks and the questions at the end of the talks were all pretty good.Closing out with a couple of random thoughts:

  • It took awhile for the wifi to work, but they got it straightened out after about an hour.
  • There was never enough coffee on hand.
  • The swag was really nice: T-shirt, mug, and stickers.  All of it was branded Mongo and not 10gen, which I felt was a classy touch.
  • Most presenters tweeted URLs to their talks within minutes after they gave their presentations.
  • It was cool to see 10gen’s CEO up there giving a lecture on how to administrate Mongo and how sharding works.
  • Every single presenter used Keynote.  This might have been the single most surprising thing to me.
  • It was really refreshing to be at a very technical conference.  Unlike most of the healthcare conferences I go to, this one spent quite a bit of time showing code examples, answering very technical questions, etc.

All in all a great experience.

Christmas – a Time of Giving

Normally around Christmas people are focused on giving.  Giving to family, friends, maybe even coworkers.  Some give time to charities, some make donations to others.  One of my annual traditions for Christmas is giving to the Electronic Frontier Foundation (EFF), a digital liberties organization that fights for freedom of speech online, privacy, and other causes that are generally too technical to garner attention from other organizations.

This year I’m adding a very similar organization to my annual Christmas list, Software Freedom Law Center (Hi Aaron!),  which similarly advocates for issues that increasingly plague and impact our lives, whether we understand them or not.With all the buzz going on right now about Wikileaks maybe you’ve thought about these issues a little more than normal.

Depending on your understanding of the issue or political orientation, it might be hard to believe, but freedom of speech online is a relatively fragile thing.  Maybe it’s because technology appears to be finite and controllable, maybe it’s because online speech travels much faster and is easier to consume, but it seems that governments and governing bodies tend to naturally flock towards the idea of speech regulation of technical mediums.Groups like the Software Freedom Law Center and the Electronic Frontier Foundation work to preserve freedoms on our behalf.

When I say “freedom of speech online” or “freedom online”, we’re also talking about fighting against overly broad software patents, violators of the GPL, warrantless wiretapping of online communications, and a whole bunch of other issues that may seem only tangentially related to speech, but do impact your voice online.In addition to the work they do, I’ve had the privilege of interacting with and knowing a few individuals from both organizations, and they’re great people as well!  So Merry Christmas, and if you’re looking for a great gift idea or a cause to support, check out both groups.

Thinking Like a User: The Bank Analogy

Like every software organization, we have trouble getting our developers how to think like users.  It’s a time consuming process, and when you’re in an industry like healthcare, it can often be extremely difficult to visualize exactly how a product is going to be used or how a process might affect a user.  You’ve certainly never been to med school, or worked in a pharmacy.  These things are utterly foreign.  However, visualizing a user’s workflow or responsibilities is often the difference between a mediocre product and a great product.One method that I’ve found to be illuminating to our engineers is to restate every message, function, or process in terms of a bank, with them playing the user character.  Instead of the error message being “Purchase Order Alert: We’re sorry, but there were one or more items missing from your recent purchase, click here for more details“, rephrase it as “ATM Deposit Alert: We’re sorry, but there were one or more checks missing from your recent deposit, click here for more details.“Makes a difference doesn’t it?  Now our verbose, overly-polite alert text seems almost ridiculous.  Tell me what happened with my checks! Don’t make me click through for more detail!  Especially if I’m on a mobile device!  That’s my money and it’s important!“ATM: 1 check for $100.00 was returned.”That’s much better.  I feel notified and in control as a user, and it was short and sweet.  Very cellphone friendly.“Order 123: 1 item shorted: Tylenol 20MG”The point is that users actually use the products we build.  Things like error messages and process flows are important.  As developers, we often think of these things as control points, or logic trees, and don’t stop and relate what’s going on from a user’s standpoint to any of the important systems or applications that we ourselves use.  The bank analogy should help you get in the habit of stepping aside and thinking outside your code for a bit because it pulls the process or message into the realm of your experience.Even the process of constructing the analogy can be very instructive.  If you can’t quickly pull together an analogy that describes what you’re doing in terms of your own life, you probably have no clue what you’re doing.  In other words, you’re probably doing it wrong.Next time you’re solving a problem, try restating it using concepts you’re familiar with in your own life.  You might be surprised at how useful this technique can be.Updated: Sunday, 12/5/2010Today’s New York Times Magazine had an article on Jamie Dimon (I think it requires registration) that talked about his management and conversational style.  One of his favorite things to do was equate banking principles back to ordinary life.  Seems like we’ve got company on the other side of the fence too!

An Open Letter to Mint.com: Stop storing my bank credentials!

A lot has changed over the last few years.  It seems like forever ago, and yet, it was only in 2005 that AJAX sprang forth and ushered in the buzzword of Web 2.0.  And it’s great – rich applications that are delivered quickly and efficiently allow me to do things online that I never thought possible. And yet, there’s a dark side to the Web 2.0 craze for APIs and tools and importing and exporting data, and that is that we’ve taught our users to embrace man-in-the-middle attacks.  Every time I see a website asking me for my Facebook password I cringe, but nothing pales in comparison to the nightmare that is Mint.com.

I love Mint.com.  They have spectacular visual design, a great product, an entertaining and informative blog, and a great iPhone app.  I know tons of people who love Mint.com, and yet, when surveying my digital life with a critical eye, I know of no greater security risk than Mint.com.  It’s still astounding to me that Mint could grow from a small startup to being acquired by Intuit in the space of a few years and essentially retain unlimited liability by storing user’s logins and passwords to their entire financial lives.  Yikes. 

If I were turned to the dark side, I would immediately attempt to hit Mint for their millions of users credentials which provide me completely unfettered access to their accounts, most of which are not FDIC insured.  This means that when someone hacks Mint, they’ll be able to pull out all of my money, transfer it, etc., and I’ll be responsible because from the financial institution’s perspective they aren’t liable for me entrusting my credentials to a third party. Sure, Mint encrypts their password database, but somewhere that password is known or stored.  It has to be because they have to use my unencrypted credentials to login.  Sure, there are a bunch of ways they could monitor this access and mitigate risk, but at the end of the day there are usernames and passwords floating away.

There is simply no technical reason the financials institutions out there can’t work with Mint and every other API providers/consumers out there can’t implement an OAuth authentication solution.  For the nontechnical of us who are reading this, an OAuth solution is essentially a token based method of authentication.  A key based authentication mechanism doesn’t necessitate handing over your username and password to a third party, instead, you grant a key (and depending on the API, limited access) to Mint which can then login and grab the information you need.  If Mint gets compromised, your financial details might be stolen, but at least they can’t access the upstream account with the same type of access.  In fact, this is what I was really hoping would come out from the Intuit acquisition: for Quicken you used to have your financial institution give you a separate login or key for Quicken specifically. To be clear: this is originally the financial institution’s problem.  They should be providing OAuth based services for Mint and  others to consume.  However, this has now become Mint’s problem to address.  Also, hindsight is 20-20.  What may have started out as a great application for a developer to track his personal finances with an acceptable risk quotient has ballooned into one of the largest and best avenues for tracking finances in the world.

The simple fact is that today, when you change your financial institution credentials, Mint breaks, which means they’re scraping the content from financial institutions.  Financial institutions are in on it too – it should be easy to see that a large percentage of their traffic is coming from one domain.  Even those sites that use a two-factor “security questions” approach are accessed via Mint by saving all possible security questions!  Financial institutions could easily block Mint by adding CAPTCHAs to their login protocols, but since I personally know several users who have changed banks to use Mint, my guess is there’s sufficient pressure to maintain Mint’s access. Some might say that I’m being overly paranoid because we’re used to saving usernames and passwords on our local machines and while it is true that from a direct comparison perspective Mint probably has the security edge over my Macbook Pro, from a risk management perspective it’s quite a different story.  All of a sudden it pays to hire a team of evil programmers for a million bucks to gain access to Mint’s millions of users.  Consider too the fact that most people re-use a single username or password as much as possible – this means cracking a lower security database (a web forum, etc.) can leapfrog access into those same user’s Mint accounts.  The less we’re using usernames and passwords for services, the better. What’s the solution?  I think a three pronged approach should be considered by any modern technical service that holds data of value:

  • Institutions should provide rich APIs in the first place and aggressively prevent screen scraping.
  • APIs should clearly segregate between “read only” and “read and write” access levels.  Mint.com can “read” my financial data but can’t “write” and pull money out of my account for example.  API access could further be segmented to only allow access to pieces of data (e.g. financial sums only and not transactions, or both, etc.)
  • APIs should use account credentials for access, but instead should be key or token based.

This might sound complicated, but in practice it’s very straightforward.  I simply login to authorize a request made by an application (anyone authorized a Netflix device recently?) and that’s it. In an increasingly networked world, application service providers bear increased responsibility to provide safe computing to users.  The old standards of storing usernames and passwords within applications need to change to reflect a different risk model.  This means both providers (financial institutions) and consumers (Mint.com) of data.  I want to use Mint and recommend it to others so I’m hoping that they can bring their clout to bear and work things out with financial institutions to solve this problem.

Distributed Software Teams

At Sentry Data Systems, we have a very distributed technology organization.  The majority of our technical staff does not work from our Deerfield Beach headquarters.  Instead, we have our developers, implementation staff, tech support, and infrastructure personnel spread out across the country, and even a satellite office located in the midwest.  Everyone is an employee, and we don’t do any offshoring, but we are most certainly not geographically close to each other.If you’d asked me five years ago if I thought this would be a good approach to take, I would have rather emphatically told you no.  In fact, I resisted it pretty strenuously for quite a while.  You had to be a senior developer, having spent significant time on site (at least a year), and working remote was a reserved privilege.  While we had a few guys working remotely, it wasn’t the majority you see, so Bad Things couldn’t happen, but we still had folks dialing in right from the beginning.  And yet, in hindsight, it may be one of the factors that helps us squeeze more productivity out of our staff, helps them produce higher quality code, and allows us to get the leg up on competition.For starters, it forced us extremely early on to invest in systems, processes, and a way of working that brought everything we did online.  Project management, change control, bug tracking, issue tracking, source control, testing, collaboration, documentation, document management, communication, all of these things needed to be ubiquitous and consistently used by the entire staff.  If things weren’t accessible online, that meant Bob in Utah wasn’t going to be able to contribute, learn, participate, or even know about it.The second major factor that a distributed team gives us is a national recruiting footprint.  We’re not just going up against Acme Software in our back yard down here in Fort Lauderdale (South Florida has its own disadvantages for hiring technology workers), we’re getting to compete for the top talent across the US in every job market.  Our pool of potential applicants increases by an order of magnitude or more, which really amps up the talent level and allows us to be super picky.Third, I recently came across this article recently which was discussing some research from Microsoft, exploring traditional myths about Software Development, and they touched on the fact that distributed teams in their experience don’t have a negative impact on team performance.  They rightly point out that this flies in the face of a “one of the most cherished beliefs of software development” but they also illustrate how any worker would much rather talk to someone knowledgeable on their team 4,000 miles away than a less knowledgeable guy next door.  Makes sense, and it jives with our experience as well, but I can’t say I expected this outcome at first.Are there drawbacks?  Sure.  It’s nice to have everyone over for a barbeque on a long weekend, and that can’t happen.  It’s fun to walk by and joke with everyone while making the rounds in the morning, and that’s harder to do, but we still manage to interact a good deal as a team.  The flip side is it’s nice for the remote guys to be able to live where they want,  stay in touch with family and friends, and yet still have a great job at a fun company.  This really contributes to retention – we’ve had several guys move several times in the last few years, which I count as a “save” on losing an employee each time.If you’re considering running your organization’s software teams in a distributed fashion, here’s some things you’ll want to make sure you’ve got covered:

  • Excellent communication methods: cell phones, VoIP phones for extension dialing off the corporate network, private instant messaging network, email, and more.
  • Organizational Discipline: People in the organization need to understand that they will often be interacting with remote individuals, and that they can’t cherry pick projects to those who are in the office.  Yes, a phone call is not as nice as face-to-face, but often it’s more productive.
  • Team-Based Activities are Still Key: This is an easy one for us.  We play video games every Friday afternoon/evening.  Combination of shooters (Team Fortress 2) and other games (DoTA and HoN) and the games are part of the employee start up paperwork.
  • Everything Must be Online: Bug tracking, brainstorming, documentation, everything.  A major advantage this gives you is it’s a head start on preparing for audits or other certifications (SAS70, etc.) you might need to complete as an organization as everything will be easily accessible.
  • You Still Need to Be Involved: If you like to walk around and say hi to everyone each day like I do in the office, you still need to do it “online” via instant messenger or phone call.
  • Figure out if a Satellite Office Makes Sense: We found that we had roughly 5 people clustered in one city, so we sprung for a satellite office.  It’s a cheap thing to do and helps our recruiting in that area.
  • One Timezone: We work on US Eastern time.  You can live where you want, but you’re going to work that timezone.  This is critical, in my opinion and while it does mean the guys in California are up at 5AM, it’s not the end of the world and really helps keep things simple from a scheduling and planning perspective, and maintains the ability for quick communication.

It probably isn’t for every organization, but it’s really worked out for us, and it’s definitely something we’ve grown organically and will continue to improve.

    How to Access Your Django Models from External Python Scripts

    Spent a good portion of time on this over the weekend, and it turned out to be frustrating enough and the answers available incomplete enough for me to want to document this here briefly.I wanted to be able to populate my Django models from an external script by simply calling MyObject.save() .  My particular case was scraping some information off of a website using BeautifulSoup then inserting it into my Django application, where it would show up and be editable from the admin section.  It turned out that I had a major problem with environment variables, so after a lot of reading and some trial and error, the following should work for you.

    import osimport syssys.path.append('/path/to/the/directory/above/your/project')os.environ['DJANGO_SETTINGS_MODULE'] = 'yourproject.settings'from  yourproject.yourapp.models import YourModel

    To be a little more clear: f your project is in “/Users/username/myproject”, append “/Users/username” to your system path.  This clears up your environment so that when you set the location of your django settings file using the dot notation, it knows where to look.As always, if you have any questions or comments or a better way to do this, let me know.  I’m using this on Snow Leopard with a trunk SVN checkout of Django.

    A Cloud is Only as Good as Its Backup Strategy

    Saw this morning where T-Mobile and Microsoft (their subsidiary “Danger”, really) have lost all data stored by users of the Sidekick phone, due to some type of failure with their cloud storage.  We’re starting to see stories like this pop up more and more where so-called clouds fail due to some really simple reasons like not backing up (this instance), not having a redundant data center (the recent Authorize.net issue), or a botched software upgrade (Gmail and Google’s recent issues).Trust is the new currency that really matters in an increasingly distributed and technically delivered world.  Regardless of whether you’re using a “cloud computing” resource or a more traditionally provided computing resource, you’re relying on that provider to fulfill their commitments and think about preventing disasters.Implementing good security, good backup strategies, disaster recover planning…these things are all very difficult.  They are even more difficult when you’re talking about the volumes of data (multiple terrabytes, possibly even petabytes) and users (millions) that clouds are designed for.  The good news is that these are all solvable problems.Here’s what I’d look for in a cloud that I’m using:

    • published disaster recovery policy
    • independent review of some type of outside auditor (SAS70 or an equivalent)
    • published backup strategy
    • multi-homed setup (different geographic regions)
    • a publicly accessible status page that details system health and open issues.

    It looks like in this particular instance a botched SAN upgrade might be to blame.  Not having any backups or a mirrored SAN is really troubling, and even more troubling is the fact that this was the case when the design of the phone is such that it requires the cloud to be up for the phone to retrieve and use any of its contacts, pictures, etc.  Cloud computing isn’t the problem here, but it does make for a grabby headline.Read about it here.

    Update 10/12/2009:

    Apparently there were major issues with employees, morale, and the product that Danger was providing was already two years late.  Read some more anonymous details here.