Custom HTTP authentication header for Googlebot user agent crawling

[leoncheng57]

Hi Mintlify Team,

We are considering to migrate to Mintlify, and will need to crawl our Mintlify subdomain using Googlebot user agents (e.g., Googlebot, Googlebot Smartphone) because our monitoring is designed to detect issues that might specifically affect Googlebot. Currently, Mintlify blocks these requests unless they come from Google's IP ranges, which prevents us from running our checks.

Can there be a feature to add an option to safelist our crawler—ideally via setting a custom HTTP authentication header or similar mechanism—so we can use Googlebot user agents without being blocked. This would let us maintain our monitoring setup and ensure our docs are always accessible to Googlebot.

Thanks for considering this. Let us know if you need more details.