* docs(scan): correct crawl's false robots.txt claim
the doc comment said robots.txt is respected because colly honors it
by default, but colly's default is IgnoreRobotsTxt=true and Crawl never
flips it, so Disallow rules were never enforced. state the actual,
intentional behavior instead: a recon crawler has no reason to treat
Disallow as a scope boundary.
pin the behavior with a regression test so it isn't "fixed" into a
partial robots.txt implementation by accident later.
* fix(scan): cap crawl request fanout
MaxDepth only bounds recursion depth; colly's own MaxRequests budget
was left at 0 (unlimited) and Crawl set no Limit rule, so a hostile
page fanning out to an attacker-controlled number of links got every
one of them fetched at that depth (b^d). cap the total number of
requests a single Crawl call will make with a fixed internal budget.
TestCrawl_BoundsRequestFanout proves a 2500-link fanout page no longer
drives the server past maxCrawlRequests+1 actual fetches.
* fix(scan): stop redirects from escaping crawl scope
Crawl called SetClient(httpx.Client(timeout)), which replaces colly's
whole *http.Client and discards the CheckRedirect colly installs to
re-check AllowedDomains on every redirect hop. with it gone, redirects
fell back to net/http's default: follow up to 10 hops to any host. an
in-scope page could 302 the crawler to an arbitrary off-scope host
(e.g. a cloud metadata endpoint) and it would be fetched.
colly's own AllowedDomains only matches on hostname anyway, which
would still let a same-host, different-port redirect through (an
internal service reachable on the same IP). build our own http.Client
on top of httpx's shared transport instead, with a CheckRedirect that
requires the redirect target to share the exact host:port of the
request that produced it, and caps the hop count ourselves since a
custom CheckRedirect drops net/http's own 10-redirect default.
TestCrawl_RedirectRejectsOffScopeHost proves the off-scope host is no
longer fetched; TestCrawl_RedirectFollowsSameHost proves ordinary
same-host redirects still are.
-crawl spiders same-host links/scripts/forms through the shared httpx
client so proxy/headers/rate-limit and robots.txt are honored, bounded
by -crawl-depth. -passive pulls subdomains from keyless ct feeds (crt.sh,
certspotter) and historical urls from wayback, each source isolated so
one feed being down doesn't sink the rest and the target sees no traffic.
