We Need Viable Search Engine Competition, Now

It’s become clear to me that we desperately need a viable competitor (or two) in the search engine space. A somewhat related thought I’ve been having is the (probably inaccurate) sensation that bringing out a viable competitor to Google may not be nearly as hard as it has appeared for the last decade.

We need competitors now. Most websites see more than 80% of their search engine traffic arriving from just Google, and this is not a good long term recipe for a vibrant internet.

Inherent Conflict of Interest
Google’s revenue model of placing paid ads next to organic search results operates under the (publicly accepted) belief that there’s a secure “Chinese wall” between the paid and organic functions. It was even more secure, some argued, because ultimately the short-term conflict between receiving revenue for rankings (paid) vs. displaying the best rankings (organic) was not a long-term conflict. Better organic results were always in Google’s interest, because these competitive results maintained their dominance and user’s trust. And so we believed. To be fair, I feel that Google does a somewhat decent job in this area, but I continue to feel that the user experience of Adwords exhibits various dark patterns (more about this here) and Google’s corporate inertia seems to be focused on a walled garden approach with G+ and Android. Lets just say that I’m no longer going to blindly trust Google in the face of a worrying conflict of interest that’s central to their most valuable product. Declining empires under siege are the ones you have to be careful of, after all.

Vulnerabile to Manipulation
Is there anything worse than “SEO”? The very idea of this industry, filled with people whose sole job is to attempt to manipulate Google is bad enough, but the fact that “black hat” SEO can produce material gains is genuinely worrying. Having had to clean up a mess created by a black hat (who insisted he wasn’t) and now in the middle of another mess of toxic back links that may or may not be generated by a competitor, the whole thing is just annoying, wasteful, and embarrassing for Google. I get that they’re trying to clean this up with Penguin and Panda and the various versions therein.

Arbitrary and Corrupt
When RapGenius violated Google’s SEO guidelines, they were only caught due to a public revelation on Hacker News, then immediately penalised by a human (to compensate for where their algorithm failed), then they were permitted to communicate directly with google to discuss ways out of this mess. Not it appears they’ve been fast-tracked back into the listings, albeit at somewhat of a disadvantage.

All aspects of this rub me the wrong way –

  1. Google is making arbitrary rules on how sites should behave, because they have a monopoly. If they didn’t have a monopoly, they might not be able to make these arbitrary rules, and others might not follow them.
  2. Google needs these rules, because Google’s rankings are apparently trivial to game. Build a ton of links and make sure you don’t over-optimise your link text. That’ll do it for most key phrases, apparently, as long as you’re not completely obvious. There’s a clear incentive for “Bad Guys” to win using“Bad Ways”, that penalises good sites just trying to get on with business. Does anyone actually believe that the ridiculously obvious, poorly written link farms that Google catches periodically are the only examples out there? Smarter people doing a better job are gaming google all the time, and it appears to be getting worse.
  3. Google feels the right to at any time, and with zero due process, transparency, or appeal, to manually penalise sites who successfully ignore their rules yet exhibit a high ranking. This is not transparent, fair, or reliable. It is scary for legitimate businesses, and this kind of instability should not be the norm, but it is.
  4. The only organisations or individuals who can actually engage with Google over a penalisation or problem in any meaningful way are Silicon Valley favourites or companies backed by influential VCs, or [insert some other not-avaible-to-the-public recourse here]. This is the definition of corruption.

We Need A Competitive Alternative
Competition could provide a healthy response to many of these items. I don’t think regulation is the answer, but it may become one if these trends continue and intensify. A different revenue model could remove the conflict of interest, a better or different algorithm could be less prone to manipulation, and a search engine that prided itself on a transparent and efficient arbitration process for disputes with regards to rankings could win users trust. Of course, Google could also work on these problems themselves, but it seems like they’re more or less happy with the current state of affairs.

Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype, and have missed the enormous opportunity that exists right in front of our noses. Does the next search engine have to be as big, involved in as many things, employ as many people, and fight on the same footing to be accomplish the goal of providing a counterpoint to Google?

Time will tell.

The Efforts of a Generation – Some Thoughts on PPC Advertising

I’ve been spending the last month or so setting up, honing, and dialing in our Google Pay Per Click (PPC) strategy.  I have a few thoughts on this experience that I’d like to share and see what the current conventional wisdom might be on these topics.

A little background – we’re decidedly in the “long tail” camp of online advertising, which means we’re going after search terms that are very specific.  We’re not bidding on keywords like “cheap software” or something that would broadly appeal to all consumers.  Our target market is training companies who need software to help them manage their training businesses.  Search terms like “training management software”.  Even terms like “training business” are much too broad for us as this would usually refer to people starting a personal training business, and while they could use our software, we’re more geared to the type of training company that delivers many different classroom style courses (online and offline).

Is Google Adwords Designed to Scam You (or at least Mislead)?

Google’s motto is famously “Do No Evil”, but I believe that the way Adwords is configured by default is designed poorly at best and possible designed to flat out scam you.  Let me explain.  When you start a new campaign and begin adding keywords, Google accepts keywords you enter as a “broad match” type, which their documentation describes as “synonyms, related searches, and other relevant variations”.  This is totally reasonable on the face of it, and I’ll bet that most if not all campaigns get started with these parameters.  They’re the default and everyone would like to include synonyms!

Here’s the problem – Google buries the only way to sanity check which keywords your campaign is actually being clicked through on in a very hard to find option.  They also don’t update these stats until 24-48 hours later, when every other stat available through Adwords seems to be updating hourly or better.

Screen_shot_2012-07-24_at_12

What does this mean?  It means that the default report you see which shows your keyphrases and the clicks via those keywords is not accurate – users aren’t actually clicking on those keyphrases, they’re often clicking on things that have almost zero relevance to your campaign particularly for those campaigns that are designed to go after “long tail” keywords.

Instead of showing you the real keywords that you’re paying for, Google hides these away on another report in a dropdown menu labeled “Keyword details”.

Google repeats this tactic of (at least from what I can see) in not allowing you to whitelist display network sites.  They allow you to select sites on which you’d like to run your ads, but they also require you to run on “automatic placement” display network sites which were all 100pct irrelevant and probably would be to most companies.  Sites like screensaver downloads, free software download sites, or a wide variety of other sites that seemed designed to be Adwords honeypot properties.  From what I can tell you can either disable all display network advertising, or login every day and manually “Exclude” sites where you’ve potentially wasted money the previous day.

Why does Google do this?  It’s clearly in their interest to extract as much revenue as possible, but the thing that’s weird to me is the entire approach here seems to be designed in a way to ensure a negative customer experience, at the expense of a few bucks of revenue.  Consider the client who doesn’t care, has unlimited budget, and wants to blast out to as wide an audience as possible – let him start with a specific audience, then suggest that he loosen up his criteria and increase his spend.  In fact, Google already has this tool – their “Opportunities” tab.  The flip side is the long tail client who without really careful exploration and attention is going to get hosed.  I’m sure there are millions of small adwords advertisers (mostly small businesses) who are getting taken for a ride and will never know it.

At the very least, the keyword clickthrough report should have a one-click drilldown to the actual keywords clicked to help users see and hone their campaigns.  I mean, certainly this is one of the most valuable reports (if not the most).  This report only increases in value as you widen your net of keyphrases.  Why effectively hide it?  Because you might rethink your spend when you see what you’re actually paying for.

Another really annoying thing possibly designed for lockin reaons (although probably just an oversight) – you can’t export your keywords.  Yes, it’s true.  You have to “spreadsheet edit” then copy and paste them out.

This is Probably a Conflict of Interest

It’s a really uncomfortable feeling to know that you’re bidding on keywords within Google Adwords while trying to optimise for those same keywords via Google Webmaster Tools.  This is a massive conflict of interest and Google would be well served to break these into separate companies or at least detail what kind of protections they have in place to prevent the obvious issues that could crop up should the teams begin to collude together.

Click Fraud

Google allows you to pay for ads based on conversions, which is something we haven’t experimented with just yet, but on the face of it seems like a good idea.  Good fora business and for Google as in theory it makes click fraud irrelevant.  Who cares if they fradulently clicked the ad as long a they didn’t buy your product?

Since we’ve been live, we’ve noticed 1-2 signups per week that come with good names, reasonable email addresses, and plausible phone numbers.  Some of the phone numbers are even real, but are innacurate.  I’m pretty sure these are due to click fraud that’s designed to earn money for sites that get paid based on conversions.  If you’re a site owner, you won’t know how your ads are being paid for but it’s a good bet that no matter what a “real signup” will look less like click fraud and could net you a few extra bucks.

I’m not sure what the best solution here, but I am sure that click fraud is probably responsible for at least 10pct of clicks (more for higher value products or “shadier” industries).  I wonder if Google releases its suspected rate of click fraud over time?  That would be really interesting.

The Alternatives are Poor

Indeed, it has been said that democracy is the worst form of government except all those other forms that have been tried from time to time

– Winston Churchill, Speech in the House of Commons, 1947

The alternatives to Google Adwords seem to be Facebook, LinkedIn, and Bing (which services Yahoo as well).

Bing’s ad platform doesn’t work with Google’s Chrome browser.  Their coupons they send you for free ad credit upon signup don’t work for companies outside of the USA.  Fail.

LinkedIn’s ad platform currently has a bug where your ad previews don’t work for reporting or management purposes. This means you stare at a report of metrics by ad and can’t actually see which ads are generating those metrics.  You have to clone the ad, then cancel, then go back to the report to even see which ad your stats are referencing!  LinkedIn also only lets you target based on things like title or group associations.  You can’t actually target companies by any kind of meaningful industry criteria from what I can tell.  It’s genuinely bizarre.  Their tool also has about 1/10th the information that Google’s platform gives you.

Each of these is unbelievably painful compared to Google’s options.  It’s really hard for me to believe that multibillion dollar companies are built upon these platforms (exception of Bing), or at least see them as their route to profitability.  If an entire generation of techies gave their lives to these platforms, they should be ashamed of themselves.

I haven’t setup our Facebook campaign yet, so we’ll leave them aside just now.

Google Should be Ripe for Disruption, but Isn’t

The really sad thing about this, is there’s very little chance of Google getting dethroned or even having someone to compete agains in the next couple of years.  It’s possible mobile ad networks will provide some heat, but it seems really fragmented and Google already targets these through its existing Adwords tools.

Google is Getting Copied! So What?

There’s been an uproar (at least in tech circles) over the past week about allegations from Google that Microsoft’s Bing search engine was copying results from Google’s offering.  There goes Microsoft again, being evil!  There goes Google again, being good!  At least, that’s the popular sentiment.  However, the whole thing struck me as bizarre.  Technology companies have for years copied each other’s offerings, putting minimal twists or variations in their own products:

  • Microsoft Windows copies Apple’s MacOS.
  • Lotus copies Visicalc (the first spreadsheet).
  • Google copies every search engine in existence when they start.
  • Google’s Android OS copies Apple’s iOS.

Now, most people would respond that this isn’t the sort of detailed copying that Google’s objecting to.  Macro-copying of a product or service is OK – that’s just old fashioned competition.But what about copying a particular feature?  That seems to be OK by Google as well:

These were both good ideas, and were major differentiators from a user’s perspective for Bing.  Bing’s attractive backgrounds before starting a search were a classy very non-Google approach to take that immediately set it apart from it’s main competition.  Google felt like copying it was not an issue at all.Macro copying of an entire service or product line is OK with Google.  Copying specific features is OK with Google.  Maybe Google has an issue with the specificity of Microsoft’s copying.  They don’t like the very precise manner with which Bing lifts its results.  Still, Google has done this before as well when they copied Yahoo’s marketing in an almost pixel perfect ripoff.Now, my example provided above is admittedly much smaller in scope and much easier to have happen accidentally by some ill-advised graphic design intern, but I think it’s ridiculous to post about Bing copying Google when it’s a fact that all search engines routinely spend significant time analyzing their competitors.  Google has plenty of research and monitoring directed towards Bing, Yahoo, and others, otherwise how would they have known Bing was swiping their results?  In other words, we’re back to the copying that Google is comfortable with – a little higher on the food chain, a little more abstracted.What Bing and Microsoft are doing is certainly dirty, and annoying.  But it’s also stupid, lazy, and incredibly shortsighted – they might be improving their results temporarily but now they’re struggling in a PR battle they’re sure to lose.  They’re also not working on improving their own rankings or understanding why Google’s results are what they are.  Still, I can’t help but feel Google’s righteous indignation is a little over-the-top on the eve of it releasing several major copied products: its Facebook killer, Honeycomb (the tablet version of Android), a Groupon killer (Update: A reader provided this link showing an almost perfect ripoff from their beta site), and many more.Iteration is part of competition, and the way to beat a competitor (especially one who copies a little too closely) is to keep iterating.  Google exposed Bing, but I can’t help but wonder if they should have just ignored the whole thing, chuckled to themselves, and kept working.  I wonder how many millions went over to Bing and tried it after this announcement.