A few years ago anything you wanted to search for would generate hundreds of pages of results. From sources all over. If you structured your query well you could dramatically cut down the irrelevant results and then spend a bit of time skimming through the offerings until you found your target.
Searching felt like a hunt, it could take you to far flung corners of the web, and hitting the jackpot on some obscure forum was part of the fun.
Now searching anything feels incredibly frustrating. You know what you are looking for is out there somewhere but it feels impossible getting to it. Basically we have only two main routes now, Google and Bing (and I swear Bing is just copying Google’s homework). I’ve tried a handful of others and they all fall very short.
Something has changed dramatically under the hood.
It seems like the results are very shallow and no amount of prodding will get either search engine to produce anything besides the laziest output. Any subject has a limited pool of sites they lean on and then pretend that that’s all of the Internet.
It generalizes very quickly and if you try to counter by putting things in quotations it gives up.
Even in the rare moments I’ve buckled and used their stupid AI, it takes even more time browbeating the program to just deliver the damned specifics.
What is going on??
How do we get our old search back?
I swear every day of trying to navigate the web feels worse than the day before.
How did we get here?
There’s got to be something better, right? Or something we can do to pave a new structure that benefits the denizens of the web better than the lazy slop our tech overlords scrape off the table for us. What can we do?
Ever since the deployment of machine generated results for search engines, there has always been a competition between the search engines and various others using search engine optimization to get to the top of the results. It isn’t worth it for Alphabet or Microsoft to fix their search engines if they can push you to AI, so their non-AI results naturally degrade.
Search quality has been degrading for years, even before AI was involved.
But it feels it got much worse much faster over the last 2 years.
Because data is never free. Google, duckduckgo, kagi, they all cost money one way or another whether that be a subscription service or by forcing websites to be shit so your only option is using their LLMs. Yes the reason why the internet has gone to shit is because of LLMs
I’ve went through hundreds of search results on duckduckgo, even when I make it intentionally broad and genetic that wikipedia would cover it, it still doesn’t come as the first search result. Search engines have gone to shit because LLMs make the most money in terms of data surveillance. Meaning (take for example google) they have a profile of you, they have queries to train, they have data that can be sold to advertisers. The only reason LLMs are being pushed so much isn’t because its intelligent, it’s because it makes the most money.
There is a “search engine” that hasn’t gone to shit, and that’s called marginalia search, It’s not exactly a search engine where you can type a url and it will bring you to a page, Instead it compliments your general search engine as a research aid and you can find some golden sites.
I’ve personally used it for data sheets on IC electronics and their implementations. Unfortunately I’ve found it insufferably difficult on duckduckgo (like multiple pages deep for a datasheet, something that should pop up on the first page), and I don’t trust the output of LLMs (large lying models).
So yea, hopefully this helps.
I’ll have to check that out. Thanks
Others have addressed what I’m about to say, but there is an answer older than LLMs: advertising.
Google made an amazing search product, one used by so so many people from all over. This made them a perfect platform for advertising, and as time passed and it became their primary focus, they realized they could make even more money people spent more time on their own site.
How do you do that, when the product people are there to use is aimed to redirect you elsewhere? Make it take longer to find the thing they are looking for. Search was incentived to work less well so that users saw more ads and for longer.
Integrating AI into search results introduced new ways to keep people on Google itself, including by presenting the answers themselves directly. But it started earlier.
I read an article about it somewhere, but, can’t find it. Basically the guy in charge of search realized they were at the highest level of users they can get so the only way to increase search traffic is to make it just bad enough you have to search different phrasing a couple of times to get the result therefore increasing search traffic.
The thing that is eternally forgotten when people ask these sorts of questions, or perhaps it is just deliberately pushed to the back of their minds is that the corporations providing these services are not in the business of providing services, they are in the business of making profit, primarily by finding ways to push as much advertising as is possible. Remembering that is really the only true answer to these sorts of questions.
“Number go up” is all that counts.
https://www.wheresyoured.at/the-men-who-killed-google/
These emails — which I encourage you to look up — tell a dramatic story about how Google’s finance and advertising teams, led by Raghavan with the blessing of CEO Sundar Pichai, actively worked to make Google worse to make the company more money.
I read an article about it somewhere, but, can’t find it.
Is this intentional ironic?
Lmao no it wasnt I did search for it too
I’ve read that, too. Probably that same article. I’ve read a few other explanations, about how SEO “ruined” the Internet. How they’ve deliberately handicapped search so you’d use AI, etc.
I feel like all of this is true but there is also more to all these stories than meets the eye.
To me it feels like back in the the day everyone just collectively allowed the tech giants to be lifeguards. But then we find out that they’ve been taking money on the side to let people shit in the pool.
Now the pool is completely filled with the commercial sectors turds and the “lifeguards” want to sell us a scooper to get our carefully curated results out, sans logs.
This one maybe ? https://www.wheresyoured.at/the-men-who-killed-google/
Yep
Diabolical
Check out SearxNG.
SearxNG is just a meta search engine that uses other search engines, so it’ll have the same issues as the upstream search engines.
I tried using an instance on a public page and after about a month I gave up. Maybe it was configured poorly but it was real hit or miss. Sometimes it’d deliver but just as often the results were nonsensical.
There’s probably some instance that’s really dialed in but I quit looking after the first go around.
Because it makes AI much more appealing.
No more reading about the recipe author’s grandma and how the food reminds them of home. AI will extract the useful info and ignore the SEO bullshit.
Lol no
Or it might tell you to put glue in your pizza
“AI” is why you now get thirty different pages talking around the topic you’ve searched for but not actually answering the question, reaffirming that they’re talking about what you’re searching for by repeating a useless bit of information about it, patting you on the back for thinking up such a nuanced topic and how only the smartest people think to ask this question, then three paragraphs of tangentially related information with an ad inbetween each one, before you find out that the page doesn’t actually have the information you’re after because the page was written by clankers.
Which is probably the intent of these cogsucking companies pushing this slop; flood the zone with useless trash, convince everyone that this sudden increase in quantity of websites necessitates another 900GW of data centre capacity right up in the ass of the nearest low income area.
Episode of the Better Offline podcast: The Man That Destroyed Google Search
I do think that part of the problem is that so much up-to-date information is now trapped in discord, also that people are generally using forums and the like less to solve their technical problems because they’re privately querying robots. This on top of enshittification… We did not know how good we had it 20 years ago
These points are valid in their own right but I don’t think they explain how Google has become much worse at finding things that really are in plain text on a website somewhere. Things that could be found, that is now buried.
SEO got really big around 2016 and Google has done nothing to help users get around their algorithms being gamed. Instead, it appears they’ve leaned in to the fact that companies were ready to pay for better treatment by the Search.
That and everything is a YouTube video now. Or at least that’s what the search company that makes money off their video service wants you to think.
Greed
You might try to give one of the alternative search providers a try. There’s duck duck go for the USA, and Qwant for the EU.
I think they intentionally made search engines worse to push people into using LLM’s
Which are amazing now, but not to worry, LLMs will be complete dog shit too once they pull the rug.
They already are dogshit.
If your LLM prompt isn’t about something common or mainstream is has an incorrect response most of the time. Unfortunately due to how they work, they will present the incorrect answers with the same confidence as correct answers.
LLMs seem like their only use is writing reports or resumes that no one needs to read. Everything else is better done by hand.
Google in particular getting worse to get you to click through more results pages or to run more queries so they can sell more ads on each of those is well attested. I used Bing for a while and didn’t think it was better or worse than newly bad Google. So now I use an equally shitty non-US search engine (start page) and just give up sooner.
In their defence, I think the exponentially increasing content of the internet, and the not insignificant portion that just sits there dead or unmaintained, plus the bits and bytes that just disappear as subscriptions or support run out are difficult problems. It’s getting ever harder to write an algorithm to sort through that as well. That plus greed equals shit.
Pre-Google we had very shitty search results. Than Google became the Kleenex, the word everybody used. Then Google got worse. While my faith in “American Intelligence” is somewhat limited, I think we will get back to better results soon. What Google had to do (build big server farms to look at all of the internet to index it) may become trivially easy for a sufficiently good LLM eventually. Although we’re not there yet. Given that it’s not just Microsoft or Google fighting in that market for dominance, I live in hope that competition from OpenAI etc. lights a fire under their arses.
Good points.
Legacy stuff is definitely getting buried under the new wave. Exacerbated by the fact that “new” sells while old “costs somebody to maintain”.
Speaking of costs, I guess the free lunch is about over. Besides the cost to access, most of the web was essentially free. A little bit of banner advertising paid for a simple website. But between then and now it’s becoming clear that the continued expansion of increasingly invasive marketing and data collection to pay for larger and large sites, and their complexity, is not sustainable.
non-US search engine (start page)
Startpage is owned by an American advertising company.
Try Qwant or Ecosia. They’re working together to develop an independent search index (Staan) and it’s already in use in France and Germany.
Startpage is owned by an American advertising company.
Eff me, really? Can you back this up? I only see a company in The Netherlands.
Just check their Wikipedia page.
Startpage is owned and operated by Surfboard Holding BV, which is majority owned by System1, an American internet advertising company.
I use kagi and metager. They’ve solved it for the most part for me.
How come you use both? Do they have different use cases or specialities?
Because I bought metager first and have a bunch of tokens left 😅
As a kagi user I agree! It’s worth the paid service. My results are good, and their features to pin/demote sites make personalization actually personal and not some black box algorithm.
Kagi is definitely worth every penny. It feels like keyword searching in the 2010s, and that’s a huge compliment. Their small web tool has led me to some fun blogs for my rss feed, but there are a lot of ai bro blogs on there too which is incredibly ironic.
The goal of a search engine is to make the company money, not to aid you in finding what you are looking for.
We’re in between a rock and a hard place. The only way to shake the grip of enshitified companies is to either turn to government/tax backed public utility type incentives, which opens up a whole other can of worms, or lean on crowdsourcing, which is very hit or miss.
Because “Fuck you, pay me”
It’s simple. Cause Search engines transformed to ad engines.
