points by londons_explore 1 day ago | 0 comments
Behind nearly every AI bot is some human who will see real benefit from scraping my site.
I just make sure all my pages are super lightweight to cache and serve from cloudflare and let the bots do what they like.
If I put something on the internet for all to see, I want bots to be able to see it too.
I've been on the other side of bot blocks - when I ask my LLM to buy me X and it comes back and says the only place it could buy from is some Bulgarian seller because everywhere else has anti-bot scripts, I make sure to give that Bulgarian site my business.
[–] robinpie 1 day ago