fb-pixel

1300 565 610

Google PartnerDigital Marketing Associate

1300 565 610

Group-39972

Why Google Gemini private chats were indexed (& now dropping off) | Technical SEO Analysis

Xugar Blog
Sagar Sethi Entrepreneur
Gagan Ghotra
10/03/2024
SPREAD THE LOVE!

In order to compete with OpenAI’s ChatGPT Google launched its version of LLM based chatbot on 6th February last year as a beta and over time it expanded to different countries. Bard uses LaMDA - Language Model for Dialogue Applications machine learning model, which was first showcased by Google back in 2021. But after that there was no direct product released by Google which utilised this model. Bard was the first product which used this model.

Despite using one of the most advanced and technically complex models, users didn’t find Bard more useful compared to ChatGPT that’s why behind the scene Google have been working on training new language models which are more robust and can compete with models developed by OpenAI.

On 6th Dec 2023, Google’s CEO Sundar Pichai and DeepMind CEO Demis Hassabis made an announcement showcasing Google’s new set of Language Models which are Gemini Ultra, Gemini Pro and Gemini Nano. In the announcement blog post, they also mentioned that Gemini Ultra performs way better than competitor model GPT-4 on different industry benchmarks.

Google CEO Announcing launch of Gemini Chatbot

On the same day of announcing new Gemini model, Google shift Bard from using LaMDA as underlying model to Gemini Pro and later on in just 2 months. Google rebranded Bard as Gemini. 

That’s why they moved bard.google.com to gemini.google.com

Earlier bard subdomain can be used for accessing the chat interface but exactly on 8th Feb Google changed this and then for accessing chat interface gemini.google.com subdomain have to be used. Even this redirect was set up in not an ideal way.

It should have been just one redirect from bard.google.com to gemini.google.com but interestingly just 3 days after this subdomain was redirected I noticed that there are 4 redirects in between.

I’m not exactly sure what’s intended purpose of Google’s engineers and SEOs to have 4 redirects when ideally it should be just one.

Another technical error that their team made while migrating this is robots.txt
Google engineers missed on implementing the same robots.txt roles which they ended up implementing after I highlighted the issue of Bard private conversions being indexed and ranking on Google.

Google engineers migrated subdomain from Bard to Gemini but missed on following best practices for technical SEO which is necessary to ensure accurate visibility of pages from the site in Google’s index.

Let me walk you through what the exact issue right now is and how Google engineers (or may be SEOs) can solve this.

Issue - Gemini Chats Showing Up in Google Search

As of 12 Feb just 4 days after rebranding and domain redirecting I noticed that private chat URLs agave started to show up in Google search which means these URLs are both crawable and indexable by Google (same issue like Bard that I highlighted on Twitter back in Sep 2023)

This lead to Gemini URLs of format gemini.google.com/share/(numbers & alphabets string)

All these private chats should not be indexed and show up in Google Search.

Technical Deep Dive - Why Gemini Chats got indexed?

Alright these chat URLs were showing up but the next question is WHY?

Any Gemini or previously Bard Chat can be shared with others by creating a publicly shareable link (same can be done ChatGPT as well) but when those links started to show up in blog posts, tweets or linkedin posts. Googlebot picked those up and started to index while they shouldn’t be.

Let me explain this further by showing an actual example of a Gemini conversation and creating a shareable link.

Here I’m asking Google Bard “Why Xugar is best SEO Agency in Melbourne” then after following couple of steps, I got a shareable link.

Xugar

Xugar

Xugar

Now this URL of format https://g.co/gemini/share/404e23b88462 when accessed redirects to
https://gemini.google.com/share/404e23b88462

Same happens for all of Shareable URLs created by Gemini. I think that initial creation of URL shows up as g.co because that’s one of most common way to share urls on web nowadays (just like bit .ly)

Ok now when this Gemini URL shows up somewhere on Internet and Googlebot finds it then of course it will try to fetch it and see what’s there on this page (of course that’s how Googlebot works)

So Googlebot fetches this URL https://gemini.google.com/share/404e23b88462 then checks it robots file for this domain allows it to crawl the content. 

But it's robots file in first 2 weeks of February had following content

Xugar

Its clear that /share/ is disallowed which means, Google can’t fetch the content of gemini urls having /share/ in it.

But because gemini is on google.com subdomain that’s why Google systems may be considering that these URLs are important enough. Yes without looking at content, Google systems made the decision to index these and therefore in search console indexing report it would have shown up as indexed but blocked by robots.txt

But digging more it can be clearly noticed that these /share/ URLs have no-index tag implemented. But Googlebot can’t even see that tag because it's blocked from crawling the page using robots file. As Googlebot can’t crawl content of page that’s why its unable to see no index tag and its just going ahead with indexing of those /share/ URLs.

Xugar

Alright I noticed this indexing of URLs for first time on 12th Feb and since then I have highlighted it multiple times on Twitter and even have tagged multiple Googlers so that they can pass this to their engineers/SEO and get this issue fixed.

Tweet from 17 Feb

Tweet from 19 Feb
Tweet from 26 Feb

Eventually I think that after mentioning the same thing again and again, it got noticed by Googlers on Twitter and they end up making some changes on 27 Feb which ultimately lead to Gemini chats dropping off from Google index.

Let me explain how Google fixed this technical issue?

They made changes to robots.txt file on 27 Feb and that’s how issue got resolved.

26 Feb robots.txt for gemini.google.com was
User-agent: *
Allow: /app/download
Disallow: /app/
Disallow: /chat/
Disallow: /share/
Sitemap: https://gemini .google. com/sitemap. xml

27 Feb robots.txt for gemini.google.com was
User-agent: *
Allow: /app/download
Disallow: /app/
Disallow: /chat/
Sitemap: https://gemini. google. com/sitemap. xml

Difference
Google removed 
Disallow: /share/ 
Which means now Googlebot be able to crawl the private chat URLs which have format 
https://gemini. google. com/share/(some numbers and alphabets)
Then there's a NO INDEX tag implemented on all these pages (as I have explained and showed in the screenshot above)

What was happening before
With robots file Google blocked crawling of /share/ and implemented NO INDEX tag on all of /share/ chat pages. 
But for some reason Google was fetching the page URL but crawling of it being blocked by robot it was not able to see NO INDEX tag.
Which means "Indexed, though blocked by robots.txt"

What's happening right now

Now with /share/ not being disallowed from robots file, Google is able to fetch these pages, then process them and see NO INDEX tag which is implemented there.

That's why most of these /share/ pages have dropped out of index now and I can only see 10 now (on 5th March), my best guess soon those will also be gone from index and as of 10th March I can just see 3 of these remaining in index. Which I think will drop off in the coming weeks.

Xugar

Update on 15 March

Just to check it again and see if URLs are still showing up in Search Results, today I again did site:gemini.google.com/share/ query. Google Search just showed me 2 chat URLs, I think that Google have not fetched these URLs again after 26th when change in robots file was implemented by Google.

Google Search Results page screenshot showing results for site:gemini.google.com/share/

I will keep on checking this over upcoming weeks and will update the story here.

Update on 23 March

I can still see same 2 URLs showing up in Search Results for query site:gemini.google.com/share/

I think that perhaps Google have not yet fetched these 2 URLs that's why its not yet released that indexing status of these URLs have been changed and now there's a no index tag implemented on these pages.

MORE ARTICLES FROM SAGAR

[psac_post_carousel slide_show="3" dots="true" arrows="false" autoplay="false" autoplay_interval="3000" speed="800" design="design-2" show_category="false" show_author="false" show_date="false" show_read_more="false"]

SCALE YOUR BUSINESS & DOMINATE YOUR INDUSTRY!

Logo Agency
arrow

We promise not to send you spam and keep your data safe!

arrow

We promise not to send you spam and keep your data safe!

arrow

We promise not to send you spam and keep your data safe!

arrow

We promise not to send you spam and keep your data safe!

Top Arrow
We still promise not to send you spam and keep your data safe!
Sagar Sethi

Leave a Reply

Your email address will not be published. Required fields are marked *