feat(scan): add web crawler and passive subdomain/url discovery

-crawl spiders same-host links/scripts/forms through the shared httpx
client so proxy/headers/rate-limit and robots.txt are honored, bounded
by -crawl-depth. -passive pulls subdomains from keyless ct feeds (crt.sh,
certspotter) and historical urls from wayback, each source isolated so
one feed being down doesn't sink the rest and the target sees no traffic.
This commit is contained in:
vmfunc
2026-06-09 17:57:42 -07:00
committed by vmfunc
parent 9401aa669e
commit dbe79c495e
10 changed files with 787 additions and 1 deletions
+9
View File
@@ -98,6 +98,15 @@ reflected xss probe.
.B \-framework
framework detection with cve lookup.
.TP
.B \-crawl
web crawler; spiders same\-host links, scripts and forms, respecting robots.txt.
.TP
.BR \-crawl\-depth " \fIn\fR"
max crawl recursion depth (default 2).
.TP
.B \-passive
passive subdomain and historical url discovery from third\-party feeds (zero traffic to the target).
.TP
.B \-noscan
skip the base url scan (robots.txt, etc).
.SH OPTIONS