I found this thread on Mastodon where both John Mueller and Danny Sullivan from Google were helping internet legend Tim Bray with some SEO issues he was having with Google Search. Yes, two Googlers helped him out, which is rare, but it’s happened before.
Tim Bray is famous, well, internet famous, he has a Wikipedia entry that reads: “Timothy William Bray (born June 21, 1955) is a Canadian software developer, environmentalist, political activist, and one of the co-authors of the original XML specification. He worked for Amazon Web Services from December 2014 to May 2020 when he resigned due to concerns about firing whistleblowers. Previously, he was an employee of Google, Sun Microsystems, and Digital Equipment Corporation (DEC). Bray also founded or co-founded several start-ups such as Antarctica Systems.”
So when someone famous on the internet complains about Google, Google notices, well, I think Danny Sullivan at least notices.
I have reported before that Google is losing its memory – see https://www.tbray.org/ongoing/When/201x/2018/01/15/Google-is-losing-its-memory
It’s getting worse. I was looking for a blog article earlier this year mentioning a bike accident I had been in and I remembered that I had been wearing a Bontrager helmet (recommended btw) so I searched for “bontrager” through
bontrager site: tbray.org
and Google can’t find it. DuckDuckGo and Bing can handle the exact same string.
Search used to be important to Google.
So Danny and John stepped in to help with some SEO issues with the site and also said they would send feedback to the correct teams on Google Search.
Danny first came in, noted it, and wrote: “If you have an example you want to share in the future, happy to see it. That’s different than not having a page indexed: @timbray I see @johnmu gave you an answer on what could be tripping us up in this case”
John then dug a little deeper and found some issues, he said:
Hi Tim, I work with the Google search people. I took a quick look here and will pass a note internally.
To cut to the chase, what happened here is that we indexed https://www.tbray.org/ongoing/goto-potd/ from your site while you were redirecting to your E-Bike article, so we indexed that content under ” potd url”. Then the “potd” content changed (I guess this is in the layout), we indexed it and lost the E-bike content.
There are a few ways to fix this:
– Google could figure it out and handle it on its own. I passed this on, so we can improve systems, but it’s a weird edge case, in my opinion.
– block the “potd” URL with robots.txt so that search engines cannot detect it.
– use link-rel-canonical annotations on individual pages to make it more likely that Google will pick those URLs.
If you’d like examples of the latter two, happy to dig up a few.
He then had a bit of back and forth on some SEO questions with Tim, all useful for us to review as well. Here are some of those responses:
We usually fetch the robots.txt file once a day (it depends a bit on the site, but since it’s a static file, we try to cache it to reduce the load on the server). I guess “tomorrow” (depending on time zone :-)), it will stop tracking that.
With that URL being indexed against the article, I suspect it will take longer for the systems to realize they need to re-evaluate the situation (I’d guess a week or so, but impossible to say).
It can be tricky when our systems think they’ve already seen the content, just at another URL (there’s a lot of duplication on the web).
However, in general, it is rare that we index everything from a website. This can result in even closely linked pages from the home page not being indexed. I don’t want to set the expectation that a technically clean site will always have everything in search, because that’s almost never the case.
Danny also answered some general questions about how Google Search works in that thread:
No, we do not de-prioritize older content (nor was this April 2022 post “old”). We try to show as much useful content as we can. In this particular case, it is likely that the indexing fails for a technical reason.
We index pages. Old and new. We also rank pages. Old and new. Sometimes it can help to rank pages that are more recent, for example if there is a trending issue. I think overall it makes sense. This explains more about how #Google #search makes use of freshness in ranking.
I think that when someone who is famous on the Internet, is more followed and respected online, complains about Google Search, it gets Google’s attention more than a normal person. But at the same time, it makes sense, because others follow them more, too, and Google, at least from a PR perspective, wants to address those concerns before someone like me writes about them. Also, I think Danny just follows Tim, so he probably saw it that way anyway.
But yeah, being internet famous doesn’t hurt when it comes to getting support from Google with SEO questions. By the way, it doesn’t mean that Google will push a button to magically make Tim’s site rank higher…
forum discussion on Mastodon.