Skip to content

Whitelist Google's inspection & verification fetchers - #502

Open
Yuri989 wants to merge 3 commits into
crowdsecurity:masterfrom
Yuri989:feat/google-tool-fetchers
Open

Yuri989 wants to merge 3 commits into
crowdsecurity:masterfrom
Yuri989:feat/google-tool-fetchers

Conversation

@Yuri989

@Yuri989 Yuri989 commented Nov 4, 2026 •

Copy link
Copy Markdown

Google-InspectionTool is the fetcher behind Search Console URL Inspection and the Rich Results Test, Google-Site-Verification is the one that retrieves Search Console verification tokens, and Schema-Markup-Validator is validator.schema.org — the Schema Markup Validator, operated by Schema.org and hosted by Google. All three read a single URL on a user's request — they are not crawls — and a site behind a bot challenge returns them the challenge page instead, which breaks structured-data validation and domain verification.

They are added to whitelists/benign_bots/legit_bots with the forward-confirmed rDNS the other entries use: googlebot.com for Google-InspectionTool (Google documents it in its common crawlers category, same mask as the other Googlebot-range crawlers) and, for Google-Site-Verification and Schema-Markup-Validator, the mask Google documents for its user-triggered fetchers — google-proxy-***-***-***-***.google.com — anchored on that label ((^|\.)google-proxy-[0-9-]+\.google\.com$), the same shape the existing google-adsbot.json uses for ^rate-limited-proxy-[0-9-]+\.google\.com$. The two tokens share that single mask because they are one family, the same way the existing entries carry several tokens against one family (bingbot|adidxbot|bingpreview on search.msn.com, Googlebot|GoogleOther on googlebot.com). Schema.org publishes no bot identity of its own, so the trust here is Google's documented family mask plus the tool's own user agent — and the family/tool pairing is kept tight: either token from the crawler family (crawl-***-***-***-***.googlebot.com) is still challenged.

Anchoring matters: an unanchored googleusercontent.com alternative also matches any *.bc.googleusercontent.com address, i.e. the PTR any Google Cloud instance gets by default, which would make the entry forgeable. Both halves stay load-bearing — the User-Agent alone verifies nothing.

Verified against a local appsec with the challenge collection loaded, the file mounted into legit_bots/ and the rule active: 74.125.215.12 and 192.178.15.69 (google-proxy-***-***-***-***.google.com, both inside user-triggered-fetchers-google.json) with the Google-Site-Verification UA and with the Schema-Markup-Validator UA → allowed; the same user agents from 1.2.3.4, from 34.136.1.1 (*.bc.googleusercontent.com) and from the rate-limited-proxy-* family → challenged; either user agent from crawl-***-***-***-***.googlebot.com → challenged; a browser UA from a google-proxy address → challenged; stock googlebot control unaffected.

Fetcher strings and masks from https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests , https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers , https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers and https://schema.org/docs/validator.html — observed source addresses resolve to crawl-***-***-***-***.googlebot.com (e.g. 66.249.66.1, inside common-crawlers.json 66.249.66.0/27) and google-proxy-***-***-***-***.google.com (e.g. 192.178.15.69 and 74.125.215.12, both inside user-triggered-fetchers-google.json).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant