Add GPTBot to robots.txt

lizrea · web-flow · commit 4bc82a19a5f5 · 2024-04-24T09:34:21.000-05:00
gptbot is very aggressive, let's disallow them along with the other very aggressive bots.
diff --git a/code/web/robots.txt b/code/web/robots.txt
@@ -80,6 +80,9 @@ Disallow: /
 User-Agent: MJ12bot
 Disallow: /
 
+User-Agent: GPTBot
+Disallow: /
+
 User-agent: 360Spider
 Disallow: /
 
