Protecting Your Art Archives | Foundation Web Design & Development

How to edit your Shopify robots.txt file to block AI Crawlers

November 24, 2023November 24, 2023

So… You want to add some of the information to block AI crawlers from continually scraping your data. Easy enough with a WordPress plugin, and Squarespace has a toggle, but what about Shopify?

The help center document on their robots.txt file seems very daunting. Especially with that big ol warning about losing all of your traffic. But I assure you, it’s actually very easy to do, and very easy to revert if you’ve done it incorrectly.

Which web crawlers are associated with AI crawlers?

November 17, 2023October 1, 2025

While initially building out my Simple NoAI & NoImageAI plugin for WordPress, a meta directive was all we had at the time. Now that crawlers are using robots.txt to determine whether a site has opted in for web crawling, more well-behaved crawlers are looking for them. Therefor, it’s important to have a list of crawlers, what they’re noted to be used for, and their user agents in case you want to block them.

This information as of Oct 1st, 2025 is up to date, but may change. I will try to keep this list updated as much as I can. Information on this list is also supplemented by Cloudflare’s list of blockable AI crawlers.

Utilizing Robots.txt to Block AI Crawlers

August 8, 2023November 24, 2023

Here I am, back again to pass along some knowledge on how to protect you and your art against generative AI. In fact, I’ve got a few posts happening now, and I’m sure more will come as the technology evolves, so I’m going to start a new category for blog posts under “protecting your art“. You can always subscribe to get blog updates from me if you’re interested in hearing more about these things as they emerge.

Today I’m going to teach you a little bit about the inner workings of web crawlers and how to utilize a robots.txt file to block certain crawlers from accessing your website.

What is DeviantArt’s new “noai” and “noimageai” meta tag and how to install it

November 12, 2022November 24, 2023

Yesterday, the internet was taken by storm by DeviantArt’s announcement of some new services and features. Many people focused on the fact that DeviantArt announced an AI generator of their own, and that by default, images uploaded to DeviantArt were opted in to being allowed to be used for AI training (though not specifically DeviantArt’s AI called “DreamUp”). DeviantArt has since reversed its stance on opt in (now making opt out the default, which should have been the case to begin with), but there were a lot of different, loosely related ideas that were part of this announcement that, unfortunately, were missed. One of these is the release of the “noai” and “noimageai” meta tags, which will help you have more control over whether or not your art is used to train AI image generators.