/* ·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· : : : █▀ █ █▀▀ · Blazing-fast pentesting suite : : ▄█ █ █▀ · BSD 3-Clause License : : : : (c) 2022-2026 vmfunc, xyzeva, : : lunchcat alumni & contributors : : : ·━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━· */ package scan import ( "fmt" "net/http" "net/http/httptest" "sync/atomic" "testing" "time" ) // crawlSite serves a small link graph: // // / -> links /a and an off-host page; references script.js, form action /submit // /a -> links /b // /b -> links /c (only reachable at depth 3) // /c -> leaf func crawlSite(t *testing.T) *httptest.Server { t.Helper() mux := http.NewServeMux() // no robots restrictions; colly fetches this before crawling. mux.HandleFunc("/robots.txt", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNotFound) }) mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { if r.URL.Path != "/" { http.NotFound(w, r) return } _, _ = w.Write([]byte(` a off `)) }) mux.HandleFunc("/a", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(`b`)) }) mux.HandleFunc("/b", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(`c`)) }) mux.HandleFunc("/c", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(`leaf`)) }) srv := httptest.NewServer(mux) t.Cleanup(srv.Close) return srv } func urlsContain(urls []string, want string) bool { for i := 0; i < len(urls); i++ { if urls[i] == want { return true } } return false } func TestCrawl_FindsLinkedPagesAndAssets(t *testing.T) { srv := crawlSite(t) result, err := Crawl(srv.URL, 3, 5*time.Second, "") if err != nil { t.Fatalf("Crawl: %v", err) } // links, scripts and forms must all be recorded, resolved to absolute urls. wants := []string{ srv.URL + "/a", srv.URL + "/b", srv.URL + "/c", srv.URL + "/script.js", srv.URL + "/submit", } for _, w := range wants { if !urlsContain(result.URLs, w) { t.Errorf("expected crawl to find %q, got %v", w, result.URLs) } } // AllowedDomains must keep the off-host link out of the result set. if urlsContain(result.URLs, "https://off-host.example/x") { t.Errorf("off-host link should be excluded, got %v", result.URLs) } } func TestCrawl_RespectsDepth(t *testing.T) { srv := crawlSite(t) // depth 1: only links found on the root page (/a, /script.js, /submit) are // recorded; /b lives one hop deeper and must not appear. result, err := Crawl(srv.URL, 1, 5*time.Second, "") if err != nil { t.Fatalf("Crawl: %v", err) } if !urlsContain(result.URLs, srv.URL+"/a") { t.Errorf("depth 1 should find /a, got %v", result.URLs) } if urlsContain(result.URLs, srv.URL+"/b") { t.Errorf("depth 1 must not reach /b, got %v", result.URLs) } if urlsContain(result.URLs, srv.URL+"/c") { t.Errorf("depth 1 must not reach /c, got %v", result.URLs) } } func TestCrawl_Dedupes(t *testing.T) { // a page that links the same target twice must yield a single entry. mux := http.NewServeMux() mux.HandleFunc("/robots.txt", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNotFound) }) mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/dup" { _, _ = w.Write([]byte(`leaf`)) return } _, _ = w.Write([]byte(`1 2`)) }) srv := httptest.NewServer(mux) defer srv.Close() result, err := Crawl(srv.URL, 2, 5*time.Second, "") if err != nil { t.Fatalf("Crawl: %v", err) } count := 0 for _, u := range result.URLs { if u == srv.URL+"/dup" { count++ } } if count != 1 { t.Errorf("expected /dup once after dedupe, got %d in %v", count, result.URLs) } } func TestCrawl_ResultType(t *testing.T) { r := &CrawlResult{} if r.ResultType() != "crawl" { t.Errorf("ResultType = %q, want crawl", r.ResultType()) } } // an in-scope page 302-redirecting to a separate host must not be followed: // colly's own CheckRedirect only re-checks AllowedDomains, which matches on // hostname alone, so a same-host redirect to a different port (as an // internal service or metadata endpoint reachable on the same host would be) // slipped through until Crawl installed its own host:port-strict check. func TestCrawl_RedirectRejectsOffScopeHost(t *testing.T) { var offScopeHits int64 offScope := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { atomic.AddInt64(&offScopeHits, 1) _, _ = w.Write([]byte("should never be fetched")) })) defer offScope.Close() mux := http.NewServeMux() mux.HandleFunc("/robots.txt", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNotFound) }) mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/go" { http.Redirect(w, r, offScope.URL+"/secret", http.StatusFound) return } _, _ = fmt.Fprint(w, `follow me`) }) target := httptest.NewServer(mux) defer target.Close() if _, err := Crawl(target.URL, 3, 5*time.Second, ""); err != nil { t.Fatalf("Crawl: %v", err) } if got := atomic.LoadInt64(&offScopeHits); got != 0 { t.Errorf("redirect escaped scope: off-scope host fetched %d time(s)", got) } } // a redirect that stays on the same host:port must still be followed; the // scope check must not break ordinary same-site redirects (e.g. login // bounces, trailing-slash normalization). func TestCrawl_RedirectFollowsSameHost(t *testing.T) { mux := http.NewServeMux() mux.HandleFunc("/robots.txt", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNotFound) }) mux.HandleFunc("/go", func(w http.ResponseWriter, r *http.Request) { http.Redirect(w, r, "/landed", http.StatusFound) }) mux.HandleFunc("/landed", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(`leaf`)) }) mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { _, _ = fmt.Fprint(w, `follow me`) }) target := httptest.NewServer(mux) defer target.Close() result, err := Crawl(target.URL, 3, 5*time.Second, "") if err != nil { t.Fatalf("Crawl: %v", err) } if !urlsContain(result.URLs, target.URL+"/go") { t.Errorf("expected same-host redirect source /go to be recorded, got %v", result.URLs) } } // a hostile page fanning out to far more links than any real page would must // not turn a bounded-depth crawl into an unbounded one. child pages are // leaves with no links of their own, so this exercises maxCrawlRequests // rather than the (unbounded, by design) link count off a single page. func TestCrawl_BoundsRequestFanout(t *testing.T) { fanout := maxCrawlRequests + 500 var hits int64 mux := http.NewServeMux() mux.HandleFunc("/robots.txt", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusNotFound) }) mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { atomic.AddInt64(&hits, 1) if r.URL.Path != "/" { _, _ = w.Write([]byte("leaf")) return } for i := 0; i < fanout; i++ { _, _ = fmt.Fprintf(w, `l`, i) } }) target := httptest.NewServer(mux) defer target.Close() if _, err := Crawl(target.URL, 1, 30*time.Second, ""); err != nil { t.Fatalf("Crawl: %v", err) } // +1 for the root page itself; the rest must be capped at maxCrawlRequests. if got, want := atomic.LoadInt64(&hits), int64(maxCrawlRequests+1); got > want { t.Errorf("expected at most %d requests (request cap), server received %d", want, got) } } // robots.txt is intentionally NOT honored: sif is a recon/pentest crawler and // Disallow rules are not a scope boundary it should respect. This pins the // intentional behavior so it isn't "fixed" into a partial robots.txt // implementation by accident later. func TestCrawl_DoesNotHonorRobots(t *testing.T) { var secretHits int64 mux := http.NewServeMux() mux.HandleFunc("/robots.txt", func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte("User-agent: *\nDisallow: /\n")) }) mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/secret" { atomic.AddInt64(&secretHits, 1) _, _ = w.Write([]byte("leaf")) return } _, _ = fmt.Fprint(w, `s`) }) target := httptest.NewServer(mux) defer target.Close() if _, err := Crawl(target.URL, 2, 5*time.Second, ""); err != nil { t.Fatalf("Crawl: %v", err) } if got := atomic.LoadInt64(&secretHits); got == 0 { t.Errorf("expected Disallow:/ path to be fetched (robots.txt is not honored), got %d hits", got) } }