While I am fully sympathetic to the bot scraping problem and support high quality public data on an open internet, I can't help but think the whole system needs a re-architecting.

Can we re-deploy the internet as statically hosted data available through a defined api? Something like DNS for data; eventually consistent and evenly distributed. Either the data is open and ubiquitous, or closed and inaccessible.