View all AI news articles

Take Control: How to Stop Google's Bard AI from Learning from Your Website

May 17, 2024


In a world where data is the new oil, Google's Bard AI and other machine learning models have been feasting on web content for training purposes. But now, Google is offering website owners a way to opt out of this data collection. By simply adding a line to your website's robots.txt file, you can prevent Google from using your content to train its AI models. This move comes after growing concerns about ethical data collection and the need for user consent.

The Mechanism

If you're a website owner and you want to opt out, the process is straightforward. All you need to do is disallow "User-Agent: Google-Extended" in your site's robots.txt file. This file serves as a guide for web crawlers, telling them which parts of your website they can or cannot access. Here's a guide on how to edit your robots.txt file.

Ethical Concerns

Google claims to develop its AI models ethically, but the reality is a bit more complicated. According to Danielle Romain, Google's VP of Trust, the company has heard from web publishers who want more control over how their content is used for AI training. However, this seems like a belated realization, as Google and other tech giants have already trained their models on vast amounts of data without explicit consent from users.

The Power of Choice

The company frames this new option as a way for you to "help improve Bard and Vertex AI generative APIs." In other words, it's not about Google taking something from you; it's about whether you're willing to contribute. But this framing is problematic, given that Google has already benefited from unrestricted access to web data.

Other Platforms Taking Action

Interestingly, Medium has also announced that it will be blocking such crawlers universally until a more granular solution is available. Read Medium's announcement here.

The Bottom Line

While this move by Google appears to be a step in the right direction, it's clear that the tech giant is playing catch-up when it comes to ethical data collection. If ethical data usage were truly a priority, this setting would have been available years ago.


How do I opt out of Google's Bard AI training?

You can opt out by adding "Disallow: User-Agent: Google-Extended" to your website's robots.txt file.

Is this move by Google truly ethical?

The ethics are debatable, as Google has already used a large amount of web data for training without explicit consent.

Are other platforms doing anything similar?

Yes, Medium has announced that it will block such crawlers until a more refined solution is available.


Recent articles

View all articles