It’s become clear to me that we desperately need a viable competitor (or two) in the search engine space. A somewhat related thought I’ve been having is the (probably inaccurate) sensation that bringing out a viable competitor to Google may not be nearly as hard as it has appeared for the last decade.
We need competitors now. Most websites see more than 80% of their search engine traffic arriving from just Google, and this is not a good long term recipe for a vibrant internet.
Inherent Conflict of Interest
Google’s revenue model of placing paid ads next to organic search results operates under the (publicly accepted) belief that there’s a secure “Chinese wall” between the paid and organic functions. It was even more secure, some argued, because ultimately the short-term conflict between receiving revenue for rankings (paid) vs. displaying the best rankings (organic) was not a long-term conflict. Better organic results were always in Google’s interest, because these competitive results maintained their dominance and user’s trust. And so we believed. To be fair, I feel that Google does a somewhat decent job in this area, but I continue to feel that the user experience of Adwords exhibits various dark patterns (more about this here) and Google’s corporate inertia seems to be focused on a walled garden approach with G+ and Android. Lets just say that I’m no longer going to blindly trust Google in the face of a worrying conflict of interest that’s central to their most valuable product. Declining empires under siege are the ones you have to be careful of, after all.
Vulnerabile to Manipulation
Is there anything worse than “SEO”? The very idea of this industry, filled with people whose sole job is to attempt to manipulate Google is bad enough, but the fact that “black hat” SEO can produce material gains is genuinely worrying. Having had to clean up a mess created by a black hat (who insisted he wasn’t) and now in the middle of another mess of toxic back links that may or may not be generated by a competitor, the whole thing is just annoying, wasteful, and embarrassing for Google. I get that they’re trying to clean this up with Penguin and Panda and the various versions therein.
Arbitrary and Corrupt
When RapGenius violated Google’s SEO guidelines, they were only caught due to a public revelation on Hacker News, then immediately penalised by a human (to compensate for where their algorithm failed), then they were permitted to communicate directly with google to discuss ways out of this mess. Not it appears they’ve been fast-tracked back into the listings, albeit at somewhat of a disadvantage.
All aspects of this rub me the wrong way –
- Google is making arbitrary rules on how sites should behave, because they have a monopoly. If they didn’t have a monopoly, they might not be able to make these arbitrary rules, and others might not follow them.
- Google needs these rules, because Google’s rankings are apparently trivial to game. Build a ton of links and make sure you don’t over-optimise your link text. That’ll do it for most key phrases, apparently, as long as you’re not completely obvious. There’s a clear incentive for “Bad Guys” to win using“Bad Ways”, that penalises good sites just trying to get on with business. Does anyone actually believe that the ridiculously obvious, poorly written link farms that Google catches periodically are the only examples out there? Smarter people doing a better job are gaming google all the time, and it appears to be getting worse.
- Google feels the right to at any time, and with zero due process, transparency, or appeal, to manually penalise sites who successfully ignore their rules yet exhibit a high ranking. This is not transparent, fair, or reliable. It is scary for legitimate businesses, and this kind of instability should not be the norm, but it is.
- The only organisations or individuals who can actually engage with Google over a penalisation or problem in any meaningful way are Silicon Valley favourites or companies backed by influential VCs, or [insert some other not-avaible-to-the-public recourse here]. This is the definition of corruption.
We Need A Competitive Alternative
Competition could provide a healthy response to many of these items. I don’t think regulation is the answer, but it may become one if these trends continue and intensify. A different revenue model could remove the conflict of interest, a better or different algorithm could be less prone to manipulation, and a search engine that prided itself on a transparent and efficient arbitration process for disputes with regards to rankings could win users trust. Of course, Google could also work on these problems themselves, but it seems like they’re more or less happy with the current state of affairs.
Is PageRank really the indomitable tech of our generation? Nobody can do better algorithmically, or integrate some kind of crowd sourced feedback, or measure browsing time and habits, or simply hand tune some of the most competitive key phrases? I’m sure I’m oversimplifying, but I wonder if we haven’t all been hypnotised by the complexity, much of which is marketing hype, and have missed the enormous opportunity that exists right in front of our noses. Does the next search engine have to be as big, involved in as many things, employ as many people, and fight on the same footing to be accomplish the goal of providing a counterpoint to Google?
Time will tell.
PEEBS.ORG 
[…] via Hacker News https://peebs.org/2014/01/04/we-need-viable-search-engine-competition-now/ […]
It’s funny you should mention the term “Chinese Wall”. It’s well known among historians that the Great Wall of China failed to prevent invasions because the guards were constantly taking bribes from the enemy.
We started a new search engine a few months ago. Privacy focused, transparent and with a new concept for the ranking, Its called Blippex, you can find more about it here: https://www.blippex.org/about and here: http://blippex.github.io/updates/2013/07/11/launch.html
Bringing out a strong competitor to Google is hard. Ask Microsoft. Which, by the way, has been very successful in building both its relevancy and market share. But the challenge in going up against Google is that it works. People get results they want. It’s like having a best friend and then someone you don’t know coming along saying “Hey, you need a viable best friend competition. Choose me.” You’re not going to dump your best friend for someone new unless your best friend is no longer such a good friend. And for people who search on Google, it consistently does an excellent job in that regard.
The conflict with ads and editorial is real. I would agree that Google has done a very good job generally in keeping that divide clean. But the issue is no less real for Bing or any search engine competitor. IE, this isn’t a “Google thing.” You only solve if if you have search engines people pay for — and that’s been tried in the past and doesn’t work. I highly doubt it’ll suddenly work now.
SEO is not about the sole job of manipulating Google. If that were the case, Google would hardly offer its own guide to SEO. SEO is about ensuring that you have done best practices to improve the chances of becoming visible in any free search listings.
For example, helping an online business create a Google+ Local profile that lists important information about their company, which can help them better be found for relevant searches, is SEO. Helping someone understand how to do authorship, so they can have their author pictures appear next to listings, is SEO. Helping someone understand that the site they built out of all images and Flash is like trying to play video on the radio is SEO.
You’ve clearly had a bad experience with someone who claimed to be an SEO. As a result, you kind of seem like you’ll dismiss the entire industry. That’s like saying that all PR is bad, even though a good PR person can be a blessing to a reporter. Or that all plumbers are bad, because you had a bad one. Or bad doctors. Or bad web designers. And on and on and on.
It’s entirely unclear that Rap Genius had any fast-track with Google or special conversations. They’ve said things to suggest that, which also read like things they’d say to reassure their investors that they have some special way to fix stuff. But it sounds like they got a manual penalty through the regular notification system, acted to fix that, put in a reconsideration request all without necessarily going through something special.
So, you’re making an assumption that might not be true (which I can understand because of the spin Rap Genius has put on things). What I’d agree is that they definitely got back in faster than other sites that might have done the same thing. The reason is simply because they are a recognizable brand that ultimately, Google can’t removed. This “are you big enough to only get a wrist-slap” is well known to anyone in the SEO space. And yeah, it definitely sucks for the smaller sites where Google can get away hitting them harder and for longer because no one will miss them if they’re gone.
But, by the way, that’s also a check against those arbitrary rules you don’t like. Google can’t just wipe some things out because the ultimate check-and-balance is its users. If someone expects to find Rap Genius when looking for it, they’d better darn well find it — or they’re going to get angry at Google. So any site that wants this type of protection needs to make themselves a recognizable brand that doesn’t depend solely on hoping people just find them through Google.
Also, Google doesn’t set arbitrary rules because its a monopoly. It sets them because ultimately, it is a private publisher that has a First Amendment right (with court cases that back this) to do what it wants with its results. It’s like saying the New York Time has arbitrary rules about what stories it will cover (it does) and how it will cover them (it does) and can only do that because it’s a monopoly (it’s not).
Google’s monopoly status (which itself admits to) doesn’t change it having rules about what it will publish except in some very limited cases relating to competitors (and with the recent FTC ruling, Google’s in the clear on that).
Google’s most arbitrary rules, most convoluted ones, are about links. No, it’s not catching all the crud and really acts as if it has no choice but to keep adding to the list of link no-nos where it’s becoming easier to say that the only safe link is no link at all. It sucks. I won’t drop a link, but if you really want to understand what a mess this is, search for “Links: The Broken Ballot Box Used By Google & Bing.” Sadly, we’re pretty much stuck with this. Google’s rules have only gotten worse and more confusing since I wrote that last year.
It’s incorrect, however, to say Google has no due process, transparency or appeal in all this. It indeed has a process for dealing with manual penalties, which includes notifications that go out, explanations of what happened and a reconsideration process. This is exactly what Rap Genius did. This is exactly what many sites do that have problems. The system could improve in many ways, but it’s not anywhere as sparse as you describe. And no, it’s not only big VC companies that interact this way. I’ve first hand seen and known about businesses large and small that have dealt with Google.
In the end, I get your general frustration. I feel it myself. Plenty of those terrible SEOs you want to sideline know it in far greater depth to its inconsistencies that you, as well — and don’t like it either.
The answer, for one, is that it’s also really hard to run a search engine. It’s really easy to think you know what it should do, and toss out things like monopoly and regulation, but it’s much harder when you’re dealing with some of the issues. And if you want to really understand that, try searching for “Could You Walk In Google’s Shoes? Making Tough Calls With Search Listings.”
Another answer is that if Google would simply not count links its considers spammy, rather than this insane penalty system, a lot of the fear and confusion out there would go away. But Google seems to think forcing people to “disavow” links and trying to get links removed is a useful punishment or fear generator. So far, that hasn’t seemed to bite them on the PR front. SEOs understand how stupid this is, how bad it is for any site — but hey, people have just dismissed the SEOs saying this for ages as just some SEO thing. It’s not. It’s a publisher thing.
The last answer is that Google needs to get off links, in the same way that the world needs to get off fossil fuel. Links are a polluted signal that Google keeps trying to maintain in some “clean coal” fashion but continuing to make rules. Oh, now widget links are bad. Oh, press releases don’t count. Oh, guest posts are an issue. The list keeps growing longer and longer so that it’s, like I said, becoming easier to think no link should count.
The social signal shows great promise — one more thing to search for, “Could You Walk In Google’s Shoes? Making Tough Calls With Search Listings,” but both Google and Bing continue to dismiss this as not being strong enough, open to various issues. My take is that they’ve both invested so much in parsing links for signals that they haven’t yet made the investment needed to harness social signals properly.
No one signal will win it all, of course. Search for “The Periodic Table Of SEO Success Factors” to get just some of the major things that go into ranking results. But I do feel that easing off the huge emphasis given to links would solve some of the concerns you’ve outlined above and that many have.
Unpredicted timing for this discussion – but, I switched my iMac and iPad over to Bing this past weekend. Wanting to give the latest of Bing a chance against what I experience as a Google deteriorating, shallow.
Lasted two days. It doesn’t especially matter what I do with the results of my searches. What mattered was that they required getting even more dross out of the way. That when I wanted a range of choices that range wasn’t wide enough within the protocols examined.
Back with Google till the next effort somewhere or other.
Google’s rules are neither arbitrary nor corrupt. Web sites score on the combination of search term relevance and commercialism. Sites that rank high on the “sells stuff” scale rank less well organically than those that are providing free information. SEO is good when it helps search engines to correctly rank a site, and bad when it tries to fool them. Fooling a search engine is rather like shoplifting because it sends money to people who did not earn it under the rules of commerce. Being punished by a search engine is like being banned from a store for shoplifting.
Sites like Amazon earn better organic placement than small sites because they are not depending on search engines to drive their traffic. They have built their own “eyeball inventory”, so Google can’t afford to omit their pages from search results. Google’s value proposition with Adwords is to let sellers have access to the marketplace of searchers that Google built without having to make Amazon sized investments to get it. Google expects sites that sell stuff to participate as Adwords buyers in order to get access to the market. That is not a form of extortion as I have often seen it described, but the business model that makes what is now the best search engine available possible. Most of the complaints I have seen about Google come from people who want Google to give them something for nothing.
I believe Google has done its job in the traditional search engine arena to the extreme. But how to provide users with better search results remains huge room to be desired.
The most important thing is to start out from user needs and re-think the next generation of search engine out of the current definition and mechanism.
If you are seriously interested in the via solution for this topic, you may want to visit the blog page here:
http://bit.ly/1if23eh
or on Kickstarter project here:
http://kck.st/JNqv8z
The market really needs a good alternative to Google right now. Bing is a small player that wouldn’t even still be alive if it wasn’t backed up by Microsoft’s billions. DuckDuckGo is ok, but nothing special. I had some hopes that Facebook would both respect user privacy as well as form a viable search alternative to Google using some of their social data, but this is increasingly looking like both won’t happen. Facebook neither respects user privacy and the social data is being gamed by the types of companies listed at http://www.buylikesreviews.com for instance. I think the answer lays not in algorithms or social ranking, but a new approach to looking at data. Maybe some small company out there will look at search in a new way or maybe some Mathematician might figure out a new way to rank items. Whatever happens, it’s clear that the market needs some better competition for Google soon because their results are increasingly bad and even the IRS provides better customer service than them.
[…] of going up against a major company? Do […]
So what happened to blippex? There was a comment up there from the founders of blippex, but now blippex is dead it seems. How did it die? That’s a matter of curiosity.
Welcome to more than a few years ago.
Google & NSA’s endgame should be clear to those with a good intellect, search results doesn’t have much to do with it except aquire funding and information.
The algorithm shift that erased the private rankings was the single most clear step google took where they abundantly showed that they are not good guys, they work for those who work to limit access to information (the aristocracy/plutocracy). The DDoS’ing of scroogle by third party was another clear step against having an actual connection to the online communities. What is next? Sending home engineers and others with a list of talkingpoints for larger weekends (like NSA et al) to impose a corrupt view on people – oh right with passive censorship and even active censorship they allready do that.
There are serious alternatives to google, you might have to temporary accept that a state (and thereby the 5 eyes) know what you do, but in a very less intruding way.