Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Google-InspectionToolis the fetcher behind Search Console URL Inspection and the Rich Results Test,Google-Site-Verificationis the one that retrieves Search Console verification tokens, andSchema-Markup-Validatoris validator.schema.org — the Schema Markup Validator, operated by Schema.org and hosted by Google. All three read a single URL on a user's request — they are not crawls — and a site behind a bot challenge returns them the challenge page instead, which breaks structured-data validation and domain verification.They are added to whitelists/benign_bots/legit_bots with the forward-confirmed rDNS the other entries use:
googlebot.comforGoogle-InspectionTool(Google documents it in its common crawlers category, same mask as the other Googlebot-range crawlers) and, forGoogle-Site-VerificationandSchema-Markup-Validator, the mask Google documents for its user-triggered fetchers —google-proxy-***-***-***-***.google.com— anchored on that label ((^|\.)google-proxy-[0-9-]+\.google\.com$), the same shape the existinggoogle-adsbot.jsonuses for^rate-limited-proxy-[0-9-]+\.google\.com$. The two tokens share that single mask because they are one family, the same way the existing entries carry several tokens against one family (bingbot|adidxbot|bingpreviewonsearch.msn.com,Googlebot|GoogleOtherongooglebot.com). Schema.org publishes no bot identity of its own, so the trust here is Google's documented family mask plus the tool's own user agent — and the family/tool pairing is kept tight: either token from the crawler family (crawl-***-***-***-***.googlebot.com) is still challenged.Anchoring matters: an unanchored
googleusercontent.comalternative also matches any*.bc.googleusercontent.comaddress, i.e. the PTR any Google Cloud instance gets by default, which would make the entry forgeable. Both halves stay load-bearing — the User-Agent alone verifies nothing.Verified against a local appsec with the challenge collection loaded, the file mounted into
legit_bots/and the rule active:74.125.215.12and192.178.15.69(google-proxy-***-***-***-***.google.com, both insideuser-triggered-fetchers-google.json) with theGoogle-Site-VerificationUA and with theSchema-Markup-ValidatorUA → allowed; the same user agents from1.2.3.4, from34.136.1.1(*.bc.googleusercontent.com) and from therate-limited-proxy-*family → challenged; either user agent fromcrawl-***-***-***-***.googlebot.com→ challenged; a browser UA from agoogle-proxyaddress → challenged; stockgooglebotcontrol unaffected.Fetcher strings and masks from https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests , https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers , https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers and https://schema.org/docs/validator.html — observed source addresses resolve to
crawl-***-***-***-***.googlebot.com(e.g. 66.249.66.1, inside common-crawlers.json 66.249.66.0/27) andgoogle-proxy-***-***-***-***.google.com(e.g. 192.178.15.69 and 74.125.215.12, both inside user-triggered-fetchers-google.json).