-crawl spiders same-host links/scripts/forms through the shared httpx
client so proxy/headers/rate-limit and robots.txt are honored, bounded
by -crawl-depth. -passive pulls subdomains from keyless ct feeds (crt.sh,
certspotter) and historical urls from wayback, each source isolated so
one feed being down doesn't sink the rest and the target sees no traffic.
three active web-vuln probes wired into the per-target loop:
- cors: crafts attacker origins (evil sentinel, null, prefix/suffix
bypass, http downgrade) and flags responses that reflect them in
access-control-allow-origin, ranking reflection+credentials high.
- redirect: injects a controlled sentinel host plus bypass variants
(//, https:/, backslash, null-byte, userinfo @) into redirect-prone
params and catches 30x location, meta-refresh and js redirects that
resolve off-site.
- xss: injects a unique canary wrapped in breaking chars, classifies
the reflection context (html/attribute/script) and reports only the
chars that survive unescaped where they matter, so escaped
reflections don't false-positive.
all route through httpx.Client so proxy/-H/-cookie/-rate-limit apply.
hermetic httptest coverage plus integration testbed entries.
the -js pipeline already pulls every <script> into a buffer but only
mined supabase jwts from it. reuse that buffer to run a credential
regex bank (aws/github/slack/stripe/google keys, pem blocks, plus
entropy-gated generic apikey/secret/token assignments) and a
linkfinder-style endpoint extractor that resolves relatives to
absolute urls. both dedupe across scripts and surface through the
existing js logger and result struct, no new flag.
every scanner spun up its own &http.Client, so there was no single place
to apply a proxy, custom headers, a cookie or a rate limit. add an
internal/httpx package that builds one configured transport at startup and
hand it to every scanner via httpx.Client(timeout), keeping behavior
identical when nothing is set (plain client when Configure was never
called).
- httpx.Configure wires -proxy (http/https/socks5), -H/--header, -cookie
and -rate-limit into a package-level RoundTripper that paces via a
rate.Limiter and only sets headers the caller hasn't already, so a
scanner's explicit api key still wins.
- route the scan/wordlist downloads that used http.DefaultClient through
the shared client too; ports tcp dialing is left untouched.
- clamp -threads to a floor of 1: it feeds wg.Add across the scanners, so
0 was a silent no-op and a negative value panicked the waitgroup.
document the new flags in the readme, usage docs and man page.
the remaining hardcoded base urls had no test seam, so their drivers could
only be exercised against the live apis. promote them to package vars (matching
the dirlist/git/ports pattern from #112) and route dnslist's per-host probes
through an injectable transport, then add integration tests that pin each at a
local httptest fixture. defaults equal the old const values so behavior is
unchanged.
concurrent workers (-threads 40) all hit the same milestone bucket on
increment, spamming ~10 duplicate [25%] lines. track the last printed
bucket under p.mu and only print when it advances.
the pinned nixpkgs shipped go 1.25.5 but go.mod now needs >= 1.25.7, so the
flake build failed (GOTOOLCHAIN=local). bump the lock to a nixpkgs with go
1.26.3, and refresh the stale vendorHash for the current deps. `nix build`
and `nix run github:vmfunc/sif` work again.
- recenter the detection confidence (sigmoid centered at 0.3) so a single weak
signature match no longer clears the 0.5 threshold. before, sigmoid(0) was 0.5
so *any* match counted as a detection - that's the magento-on-a-plain-page
false positive from the live run. real detections match ~50%+ of signature
weight, so the existing detector tests are unaffected
- getVulnerabilities matched affected versions with a raw string prefix, so "4.2"
also matched "4.20"; match only on dotted boundaries now
- break confidence ties on name so the picked framework is deterministic
- add regression tests for the confidence floor and the version boundary
- make the four wordlist base urls (dirlist/dnslist/git/ports) package vars
instead of consts so tests can repoint them at a local fixture; the default
values are byte-for-byte unchanged
- add internal/scan/integration_test.go behind a //go:build integration tag: it
stands up a local "vulnerable app" httptest server with planted artifacts and
runs git/dirlist/cms/headers/sql/lfi/ports against it, asserting real findings
- go.yml runs them via `go test -tags=integration`; the default test run is
untouched (the tag keeps them out)
- document the integration run in docs/development.md
- add internal/version: resolve from the release ldflag, else the go build
info (module tag / vcs revision), else "dev"
- show the version on the boot banner and for `sif version`
- Makefile now stamps `make` builds via git describe (matching the release ci),
so local/go-install builds report a real version instead of "dev"
- patchnotes.ShowOnce skips pseudo/dev versions so non-release builds dont make
a doomed github call
- document sif version / sif patchnote / SIF_NO_PATCHNOTES in the readme + usage
doSupabaseRequest and the signup call used a bare http.Client{} with no
timeout, so a slow supabase project could hang the whole js scan. thread the
scan's --timeout down through ScanSupabase into every supabase request.
`strings.Split(url, "://")[1]` was copy-pasted in 18 spots and panics on a
schemeless target (index out of range). add a small stripScheme helper in the
scan package - and a guarded equivalent in logger, which cant import scan - so
a bare host degrades gracefully instead of crashing the scan.
adds man/sif.1 covering the targets/scans/options/modules flags, the
patchnote and version commands, env vars (incl SIF_NO_PATCHNOTES), and a
few examples. the install/uninstall make targets now drop it in
$(PREFIX)/share/man/man1.
- `sif patchnote` (also `-pn`) fetches the latest github release and renders
its notes with glamour
- on the first run of a new version those notes are shown once, then recorded
so they dont show again - best-effort, so dev builds, the SIF_NO_PATCHNOTES
opt-out, and any network failure stay quiet
- wire up `var version` so the release `-X main.version` ldflag actually lands,
and add `sif version`
- the readme headline used -all, which isn't a real flag (goflags fatals
on unknown flags), so the three -all examples now use actual flags
- document the new -sh security-header scan in the readme table, usage.md
and scans.md, and fix the -headers section (it dumps headers; -sh grades
them)
- bump the documented go version 1.23 -> 1.25 to match go.mod
strips the emoji from the subdomaintakeover/cloudstorage/supabase log
prefixes and trims comments that just restate the code (the
parameters/returns block on SubdomainTakeover, a couple of "this provides
type safety" notes) so the scanners read like whois.go/headers.go.
internal/worker.Pool[T,R] (run/runwithfilter/foreach) was added with tests
but never wired into anything - the scanners all hand-roll their own
waitgroup loops. dead weight, so remove it.
- git.go and dork.go appended to a shared results slice from every worker
goroutine with no mutex - a real race that -race never caught since
neither has a test. guard the appends like dirlist/dnslist already do.
- progress.go's non-tty milestone path divided by total with no guard, so
a zero-total bar panicked when output was piped/redirected. bail early
on total <= 0 to match the tty branch, and add an output test for it.
adds a -sh/--security-headers scan that flags missing or weak response
headers (hsts, csp, x-frame-options, x-content-type-options,
referrer-policy, permissions-policy, coop) and headers that leak server
internals (server, x-powered-by, ...). hsts is only graded over https
where it actually applies. wired into App.Run and the module results.
rolls the (c) 2022-2025 banner to 2022-2026 across all go files, the
startup banner in sif.go, and the header-check workflow's expected
format. comment-only, nothing else changes.
The nuclei-templates tarball is fetched over the network and its entry
names flowed directly into os.Mkdir/os.Create, so a malicious or
compromised archive could write outside the extraction directory
("Zip Slip", CWE-22). Resolve each entry against the working directory
and reject any path that escapes it before touching the filesystem.
CodeQL flagged this as a high-severity alert on the lines this branch
already touched. gosec's G305 fires on filepath.Join with archive data
regardless of the traversal guard, so it's excluded with a note.
celeste