I switched the search settings not to use Google by default in web browsers on all my devices after reading the blog post that the Head of Google Search published in response to the many reports of problems in Google’s “AI Overviews”.
Practically every point in that blog post is either a meaningless generality written in corporatespeak or a demonstrable lie. You don’t need specialized engineering knowledge or access to internal information to see it. You just need common sense.
User feedback shows that with AI Overviews, people have higher satisfaction with their search results…
Which people? Everyone? I don’t. I sharply reduced my use of Google search because I no longer trust it.
… and they’re asking longer, more complex questions that they know Google can now help with.
The word “help” is doing a lot of work here. Google can output a piece of text in response. Is this piece of text actually helpful?
AI Overviews work very differently than chatbots and other LLM products that people may have tried out.
No, they don’t. They work exactly the same. Both technologies automatically produce some text that was not written by a human.
They’re not simply generating an output based on training data.
No. They are, in fact, simply generating an output based on training data.
When AI Overviews get it wrong, it’s usually for other reasons: misinterpreting queries, misinterpreting a nuance of language on the web, or not having a lot of great information available. (These are challenges that occur with other Search features too.)
This is one of the few true things in this blog post, but it shows why this feature is completely pointless!
I mean, it’s nice that she doesn’t blame the users here for writing bad queries, but admits that the software that her team developed is bad at interpreting them.
And here’s an even more important thing: Despite the long-standing impression that “you can find everything on Google”, the “AI” innovations of the last couple of years help us realize that there are actually many topics about which there is not a lot of info online. And large language models are not going to solve this problem.
This approach is highly effective.
What does this even mean? “We are able to show more ads and improve our bottom line for the last quarter?”
Overall, our tests show that our accuracy rate for AI Overviews is on par with another popular feature in Search — featured snippets — which also uses AI systems to identify and show key info with links to web content.
This is probably the biggest lie of all in that whle post.
There is no comprehensive test or measure for accuracy! It is logically impossible to make one!
At most, there is some internal metric that middle managers present to senior managers, and it may show that the rate is “positive” according to internal company logic. However, it has absolutely nothing to do with what millions of web users actually need.
This is comparable to metrics of quality of machine translation, such as BLEU and NIST. There are methodologies and formulas behind them, but they are only useful for discussions among researchers, developers, and product and project managers, and they have very limited usefulness at predicting the correctness of the translation of a text that hasn’t yet been tested. Developers have to use those metrics because project managers love metrics, but most of them admit that they are not very good, and such a metric can never become perfect.
In a small number of cases, we have seen AI Overviews misinterpret language on webpages and present inaccurate information.
Yes, thanks again for admitting that computers are not supposed to interpret language in the first place. Humans are supposed to do it.
I could go on, but I have better things to do, like publishing three longish blog posts of my own. One is coming very soon, and it’s going to be fun, at least for me.
In response to accusations of monopolistic behavior, Google has been saying for years that competition is just a click away. It’s true, and it’s good. My experience with DuckDuckGo in the last few days has been perfectly fine.
That said, Google should still be tried for monopolistic behavior. And I kind of wish that there was regulation that prevents the deliberate destruction of fundamental public goods operated by commercial companies, but I guess that it would be very hard to legislate.
In the meantime, let’s try not to be silent about Google’s lies, and let’s consider using the competitors.
Aharoni in Unicode 