Hacker News

points by lukeschlather 5 days ago | 0 comments

I think we've reached a point where reading the AI summary (which I think is Gemini Flash Lite) is probably going to be more reliable than actually reading the top 10 search results. The top 10 search results were often blogspam even in a pre-AI world. I feel like either you trust Google to retrieve information or you don't (and Gemini has problems but so does Google Search, and are you really sure the latter is worse than the former?)

It has been over a decade since Google Search was anything but a machine-learning powered word association engine. I miss old Google Search but it's dead and I trust Gemini more than what Search has turned into.

[–] _ache_ 5 days ago

That is NOT my experience. I used to have a dump of screenshots where Google AI from search is absolutely wrong. In my experience, it doesn't work and "quoting" things never told in the source (the exact contrary sometime), responding in hindi, responding in spanish, responding in mix of languages.

Most of the time it's quicker, sometimes it's just bad.

[–] kakugawa 5 days ago

It hallucinates so frequently, that I've found it unusable. Just ask it to provide sources for all of its claims, then watch how many times it reverses its original answer. It's mind-boggling to me. I've repeatedly tried to give Google AI Search a chance, but after weeks of seeing this behavior I can't trust it anymore.

Imagine winning search by being the fastest and most accurate, then replacing it w/ constant hallucinated errors.

[–] runarberg 5 days ago

For me, I use it maybe ones or twice a month when my traditional search on DuckDuckGo fails. And, anecdotally, it is overwhelmingly wrong, and worse, confidently so.

Given my sample is biased as I am using it for an outlier of difficult web searches (including web searches localized to Iceland). But I would think that if your parent was correct, at least majority of my uses would be correct, but in my experience, Gemini fails most of the the time traditional web search fails. So to me at least, it is effectively useless.

For people who use it in place of normal web search, it is probably worse then useless, because of how confidently wrong it is.

[–] dpoloncsak 5 days ago

I think they model the model they use for these, probably at more cost to them. They've been much better as of late, imo

[–] smallerize 5 days ago

They what the model?

[–] dpoloncsak 4 days ago

I meant 'upgraded'....oops haha. Too late to edit.

'I think they upgraded the model...."

[–] warkdarrior 5 days ago

The model they model to model this is modeled by a model they model.

[–] k12sosse 4 days ago

The police police police police the police police.

[–] panarky 5 days ago

The argument is not that Google's AI summaries are infallible.

The argument is that there's so much garbage on the open web that clicking through to the source has a lower signal-to-noise ratio than the AI summary.

This rings very true in my experience.

Google's AI summaries definitely make mistakes, but in most cases they're higher signal than the source websites.

[–] andrewflnr 5 days ago

You're underestimating how obviously, laughably wrong the AI summary often is.

I really think Google is making an error using such a lightweight model for the page. That's one of my only firsthand exposures to genAI, and it's awful. Use a decent model, or if that's too expensive don't put up a summary at all.

[–] kakugawa 4 days ago

They used to prioritize accuracy first, then speed second.

Now it's just speed. This is a grave error.

[–] _ache_ 4 days ago

Once it quotes a gov website, exactly the contrary of the official source. Why ? Because on the official website you have checkboxes and the div shown is the one corresponding to your case and most surely the AI summary didn't have information about which one is which and just mix everything up.

[–] contextfree 4 days ago

"reading the AI summary is ... more reliable than actually reading the top 10 search results."

I don't quite understand what this even means. I both use AI chatbots frequently and still use keyword-based search engines frequently as well. When I use keyword-based search engines, it's usually because I want to see what sort of websites, discussions, resources, online hangouts, etc. exist around a given term, or just how the term is used. I often don't have a specific question or task in mind and just kind of want to wallow in the search results for a while, sort of like if I moved to a new city and just wanted to walk around different neighborhoods. otoh when I use an AI chatbot, it's usually because I have a specific question to ask, and often I don't know what the relevant search terms would even be. The use cases seem fundamentally different and I don't know how one could be "more reliable" than the other.

[–] BoppreH 5 days ago

I haven't checked this month, but the few times I used Google Search this year I saw egregious errors in the AI summary ~50% of the time. Things like confidently citing a statistic whose number appeared in the top 10 results, but misrepresenting what it pertains to.

[–] Forgeties79 5 days ago

All we’re doing is adding another layer between us and the information. Google flash is still pulling on the same poor information. But it’s not as good at discerning what is useful and what is not as we are (people going way too far out on a limb outside of their domain expertise not withstanding, which is another part of the problem and why this stuff can be dangerous)

[–] dawnerd 4 days ago

If you searched, whats my ip, you used to get your ip. Now you get an ai answer telling you to use one of the websites. How is that helpful?

[–] deathanatos 4 days ago

Oh, it's better than that for me:

> Your public IP address as seen by external servers is 66.249.68.3 (or check your live details via ifconfig.me or What Is My IP Address).

The AI just hallucinates¹ an IP. (I.e., that is most definitely not my IP address.)

(¹it could possibly be invoking a tool here, I suppose. Either way, the answer is slop.)

[–] MatejKafka 5 days ago

Use Kagi, it mostly excludes spam.

[–] convolvatron 5 days ago

at least before I was able a find a blog post or a discussion or fleck of documentation. Now I've pretty much stopped using Google because I've wasted so much time having been assured by Gemini that something is entirely possible when in fact its very much not. now I have to dig to find the canonical reference for a question, at least SEO people didn't spend a lot of energy trying to front rust crates.

This is all very tragic. we built a magic network where we could stash all the worlds information and access it from anywhere, except its been trashed by people with sleazy motivations trying to get between us and what we'd like to see.

[–] jimbokun 5 days ago

Can’t access the article, but would assume the schools are more concerned about students having Google AI doing their work for them, rather than concern about using the AI summary for search results?

[–] themanualstates 4 days ago

Isn’t finding / vetting / analyzing / citing sources precisely the work students are required to do to teach them media literacy skills?

[–] in_cahoots 5 days ago

Yes, this is the concern. Thankfully Google has a 'helpful' AI chatbot / tutor product that students can use instead of search!

[–] 121789 5 days ago

Not my experience at all. I can usually trust chatgpt or Claude to give me accurate answers for queries but google summaries are frequently wrong

[–] hamdingers 5 days ago

Why would a summary of unreliable blogspam results be more reliable than the blogspam results themselves?

I find it easier to recognize bullshit from its source vs an LLM's confident and well written regurgitation of it.

[–] exmadscientist 5 days ago

I consistently get the impression that Gemini is better at sorting through the trash than whatever ranks SERPs these days is.

(Obviously far from perfect, often far from even good... just better.)

Which is not a good look for the Search teams/execs.

[–] rbits 4 days ago

If Gemini's good at sorting through search results, and they're already using Gemini on every search, why don't they use it to help filter out the SEO spam?

[–] exmadscientist 23 hours ago

It's got to be either scale (one Gemini call per query versus one each per SERP result, or something silly like that) or just internal turf wars.

Probably the latter.

Like I said, not a good look for Search.

[–] sailfast 5 days ago

Only if the answer is already on Reddit.

[–] zamadatix 5 days ago

The summary isn't limited to the top 10 search results as sources so it need not be only based on blogspam like the first page of search results is/was. Whether or not it's actually better or not is still its own debate beside that point though.

[–] __MatrixMan__ 5 days ago

Because the summarizer knows what blogspam is.

[–] k12sosse 4 days ago

Funny how many RSSINGS pages still turn up in traditional Google search though

[–] snowwrestler 5 days ago

The AI summary is just summarizing the top results. It’s not Gemini.

So I can’t understand how one would trust the summary but not what is being summarized.

[–] smallerize 5 days ago

A recent study from oumi.ai found that the summaries contradict the sources they cite 10% of the time.

[–] snailmailman 5 days ago

Anecdotally, the percentage is way higher than 10% for some kinds of searches. I don't always use google, but i've tried it dozens of times the past few weeks to test their AI Summary / AI Overview, and it has yet to get anything correct. I've had to use other search engines or AIs literally every time.

mistakes vary from "obviously wrong" to "looks vaguely correct, but the source states something different" to "looks plausible, but the cited page is completely irrelevant, the AI clearly made this up" Even for straight fact lookups, I've seen it cite pages and claim one number, and then i click through and the source matches the vague shape of the answer but has a different number.

Its just constant hallucinations. "thats a common problem, just change this <setting that does not exist> or set <option that does not exist>". Full gemini seems more capable, but whatever google is throwing at the top of their free searches is awful IME.

[–] smallerize 4 days ago

I'm guessing they're only counting one failure mode. But I think it's the one that would surprise most boosters. Even if it found a relevant page, and the information is fed into the model's context right there, the output is still completely backwards a fair fraction of the time.

[–] treems 4 days ago

I still find search results pretty good, if you use a tool like uBlacklist on DuckDuckGo. I prefer them to AI at times, as AI homogenises things.

[–] ZnerflBerp 5 days ago

Google is trying to behaviorally condition children.

The relative effectiveness of the search is immaterial.

This is gross.

[–] Melatonic 5 days ago

From my experience it's worse than any of the normal Gemini models

[–] AlienRobot 4 days ago

What? No. You're completely wrong. I frequently ask about "X" and the summary cites a single random Reddit post that doesn't even MENTION "X" as source for its assertions. That literally never happened before AI.

[–] emsign 4 days ago

How is this related?

[–] RajuChacha108 5 days ago

Absolutely. Google has shown the courage to admit that their bread and butter business is changing permanently. they are adapting which the large cos rarely do.

My mother never used search. She found it too hard to manage. She uses Gemini all the time, it gives answers in her language and she does not need to type. She is incredibly happy.

My daughter talks to Gemini. The idea of going through links seems primitive to her.

The future of interface is chat/talking not going through clickbait.

I think schools should have some basic control on how kids can use AI but saying that they want to freeze the learning experience in 2020 is simply not right. It is like saying we wont allow whiteboards and markers.

[–] suddenlybananas 5 days ago

Google shouldn't be making that choice for everyone.

[–] WarmWash 4 days ago

Thankfully, no one is forced to use google.

[–] suddenlybananas 4 days ago

It's an effective monopoly.

[–] WarmWash 4 days ago

Even worse, it's not even Google's doing.

People just really, really, really hate paying for stuff. Hell Google even lets you continue to use their services even if you block their ads. They're that generous, err, evil.

[–] rockskon 4 days ago

Google wasn't exactly losing money with search results before.

If anything they're losing far more money with AI.

[–] dawnerd 4 days ago

Something I keep saying, AI people have a problem with consent.

[–] suddenlybananas 4 days ago

They refuse to understand that people could value different things than them, things like "understanding", "effort", "knowledge".

[–] preg_match 4 days ago

The question we need to be asking is does AI enhance or help learning?

NOT “do kids use AI?”

NOT “do professionals use AI?”

And not whatever other question we can conjure up. At the end of the day, if AI makes learning worse, then it doesn’t fucking matter how ubiquitous or awesome or cool it is. Because we’re sacrificing other skills that WE KNOW and have established that we need.