mirror of
https://github.com/lunchcat/sif.git
synced 2026-07-28 14:37:01 -07:00
TigahandGitHub
788d20c18e
* docs(scan): correct crawl's false robots.txt claim the doc comment said robots.txt is respected because colly honors it by default, but colly's default is IgnoreRobotsTxt=true and Crawl never flips it, so Disallow rules were never enforced. state the actual, intentional behavior instead: a recon crawler has no reason to treat Disallow as a scope boundary. pin the behavior with a regression test so it isn't "fixed" into a partial robots.txt implementation by accident later. * fix(scan): cap crawl request fanout MaxDepth only bounds recursion depth; colly's own MaxRequests budget was left at 0 (unlimited) and Crawl set no Limit rule, so a hostile page fanning out to an attacker-controlled number of links got every one of them fetched at that depth (b^d). cap the total number of requests a single Crawl call will make with a fixed internal budget. TestCrawl_BoundsRequestFanout proves a 2500-link fanout page no longer drives the server past maxCrawlRequests+1 actual fetches. * fix(scan): stop redirects from escaping crawl scope Crawl called SetClient(httpx.Client(timeout)), which replaces colly's whole *http.Client and discards the CheckRedirect colly installs to re-check AllowedDomains on every redirect hop. with it gone, redirects fell back to net/http's default: follow up to 10 hops to any host. an in-scope page could 302 the crawler to an arbitrary off-scope host (e.g. a cloud metadata endpoint) and it would be fetched. colly's own AllowedDomains only matches on hostname anyway, which would still let a same-host, different-port redirect through (an internal service reachable on the same IP). build our own http.Client on top of httpx's shared transport instead, with a CheckRedirect that requires the redirect target to share the exact host:port of the request that produced it, and caps the hop count ourselves since a custom CheckRedirect drops net/http's own 10-redirect default. TestCrawl_RedirectRejectsOffScopeHost proves the off-scope host is no longer fetched; TestCrawl_RedirectFollowsSameHost proves ordinary same-host redirects still are.