Re: cvsweb anti-bot protection (was: Retrieving MAC address from struct ifnet)

[Christof Meerwald <cmeerw%cmeerw.org@localhost>]

On Thu, Jul 03, 2025 at 11:17:51AM -0400, Mouse wrote:
> Another possible reason is that I don't speak HTTPS; I consider it
> plausble the LLM scrapers have drunk the "HTTPS is the One True Way"
> koolaid and aren't even trying HTTP.

I am seeing the exact opposite from GPTBot: it tries http on every
single request, gets a 301 redirect to https and then a
Strict-Transport-Security header as well. Yet it keeps trying http
first every single time.


Christof

-- 
https://cmeerw.org                             sip:cmeerw at cmeerw.org
mailto:cmeerw at cmeerw.org                   xmpp:cmeerw at cmeerw.org