ship findings to chat/webhook sinks after a scan so continuous recon can
alert on what it turns up. each provider is one POST through httpx.Client,
so the global proxy/rate-limit/header config applies and there's no extra
http stack. config resolves env-first (SLACK_WEBHOOK_URL, DISCORD_WEBHOOK_URL,
TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID, NOTIFY_WEBHOOK_URL), overridable by a
notify-compatible yaml file so existing projectdiscovery/notify configs port
over. -notify enables it, -notify-severity gates on the finding severity
ladder (default medium), -notify-config points at the yaml. wired after the
scan loop on the severity-filtered finding set; no provider configured is a
silent no-op.
re-scans become a monitor: -diff snapshots each target's normalized
findings to a per-target json file and, on the next run, surfaces only
the delta (+ new / - gone) against the last snapshot, then overwrites it
so each run diffs against the previous one. behavior is unchanged when
-diff is off.
new internal/store keys the set-difference off finding.Key (already
stable across runs) and uses only encoding/json + os - no new deps.
snapshot files are sanitized per target (no traversal), written 0600
under 0750 dirs. -store picks the location: explicit dir, else the log
dir, else <user-config>/sif/state. a missing snapshot is a clean
baseline, a corrupt one self-heals on the next save.
static i%threads stride partitions assign each item to a fixed worker, so a
goroutine stuck on a slow or timing-out host stalls while the rest idle behind
it (head-of-line blocking). add internal/pool.Each: workers all pull from one
shared channel, so a slow item only blocks its own worker and the others keep
draining. migrate git, ports, robots (scan), dnslist, dork, dirlist and
subdomaintakeover off the stride loops; per-item work, mutex-guarded appends and
progress increments are unchanged, results were already unordered.
jwt fetches the target once then analyzes every harvested token offline:
flags alg:none, the rs256->hs256 confusion surface, missing/expired exp
and plaintext sensitive claims, and cracks a small bundled weak-hmac list.
openapi probes the conventional spec paths, parses json/yaml and enumerates
paths plus unauthenticated operations. favicon computes the shodan-style
mmh3 hash (python base64.encodebytes chunking, signed int32) for tech
fingerprinting and the http.favicon.hash pivot, pinned by a golden test.
sif can now slot into unix pipelines. stdin is drained for targets when
it's a pipe (keyed off stdin's mode, not stdout), alongside -u/-f. naked
hosts are accepted and default to https://; explicit http(s) is kept,
other schemes rejected. -silent routes all banner/spinner/log chrome to
stderr and prints one normalized finding per line to stdout via
finding.Flatten, so `subfinder | sif -silent | notify` works.
go only returns a conn to the idle pool when the body is read to EOF
before Close. the header-only and early-return scan paths closed an
unread body, leaking the conn and forcing a fresh dial each request.
route those close sites through httpx.DrainClose so the tuned pool from
phase 1 actually gets reused. body-read paths (scanner/io.ReadAll) are
left untouched.
dnslist previously http-probed every wordlist candidate through the
blocking os resolver, so a big list meant a request per dead name and a
wildcard zone flooded results. resolve each candidate first via a new
internal/dnsx (retryabledns over a bundled 1.1.1.1/8.8.8.8/9.9.9.9 pool,
promoted to a direct dep), fingerprint the apex with random nonexistent
labels to detect catch-all zones, and http-probe only the names that
actually resolve and aren't wildcard. add -resolvers to override the
pool. resolverFn is a package-level seam so the dnsx tests stay
hermetic; the dnslist newDNSResolver seam keeps the integration test
network-free.
scan results live in ~two dozen structs with no shared shape, so every
consumer that wants "what did this run turn up" reimplements the
type-switch. add internal/finding: an ordered Severity (info<low<medium<
high<critical, with parse/compare) and Flatten, the single type-switch
that collapses every scan result struct into flat, severity-ranked
Findings keyed module:identifier for stable dedup/diff.
wire collectFindings off Flatten in the run loop so notify and diff
(later bundles) build on one normalization path instead of re-deriving
it; the report path keeps emitting raw json blobs unchanged. expose
JavascriptScanResult.SupabaseFindings so the js internals stay private.
the guard test iterates a representative instance of every ResultType
and fails if Flatten lacks a case (falls through to :unhandled) - so a
new scanner can't ship without a Flatten case landing too.
the shared transport was a bare DefaultTransport.Clone() with the stock
MaxIdleConnsPerHost=2, and call-sites only close response bodies without
draining them - so go could never return a conn to the idle pool and every
request re-dialed. high thread counts just thrashed the dialer.
- plumb Threads through Options into buildTransport; size MaxIdleConnsPerHost
to the worker count (floored) so concurrent workers on one host pool instead
of re-dialing, MaxIdleConns=512, MaxConnsPerHost=0, IdleConnTimeout=90s,
ForceAttemptHTTP2. the socks5 branch gets its own keepalive net.Dialer so it
doesn't lose os-level pooling under proxy.Direct.
- add DrainClose to read (capped) and close a body so the conn is reusable.
- benchmark proves it: 50 sequential requests reuse 1 conn tuned vs 50 bare.
adds an httpx-style -probe scanner reporting liveness, final status, page
title, server header and the redirect chain, plus -sarif/-markdown export
flags that serialize the collected run after the scan loop. the report
serializers live in a decoupled internal/report package consuming a raw-json
result model so they never import scan types.
the old scanner surfaced every response that wasn't 404/403, so modern SPA
catch-all 200s flooded the output and made -dirlist near-useless. add ffuf-style
matching:
- -mc/-fc/-fr and -fs/-fw filter by status, regex, body size and word count;
bodies are read through a capped io.LimitReader so size/word counts are
deterministic and memory stays flat. filters win over matches.
- -ac auto-calibrates the soft-404 baseline from a few deterministic
non-existent paths and drops responses matching that wildcard shape.
- -w overrides the size switch with a local file or remote list (fetched through
the shared client so proxy/rate-limit apply); -e appends extensions per word.
size and words are added to DirectoryResult for the json output.
the yaml module engine (the user-facing extensibility surface) had 0%
test coverage. add table-driven tests for the matcher types
(status/word/regex + and/or + negative), checkWords/checkRegex (incl
invalid-pattern fail-closed under AND, skip under OR), runExtractors
(regex capture groups, group-index bounds, part selection),
substituteVariables and generateHTTPRequests (path x payload expansion),
and ParseYAMLModule on valid + malformed yaml. drive ExecuteHTTPModule
end-to-end against an httptest server through the shared httpx client so
matcher hits and extractor captures are exercised for real. coverage
0% -> 93.7%.
also: ExecuteDNSModule/ExecuteTCPModule were stubs returning an empty
result with nil error, so a type:dns/type:tcp module silently reported
"0 findings" - indistinguishable from a real clean scan. make them
return ErrUnsupportedModuleType (sentinel, wrapped with the module id) so
the existing caller logs a clear failure instead. a test pins the new
behavior.
bodyclose is excluded for test files in .golangci.yml: the synthetic
*http.Response fixtures carry no socket, mirroring the existing _test.go
slack for errcheck/noctx/gosec.
four recon-flagged bugs, each with a focused test:
- dnslist fired both http and https per candidate and counted a "found"
on any non-error response (incl 404 and wildcard catch-all redirects),
so every host double-counted and a wildcard-dns host flooded results.
probe http then https with per-subdomain dedupe, gate on a meaningful
(2xx, non-redirect) status, and stop chasing redirects so a catch-all
301 reads as a redirect instead of a 200.
- fetchRobotsTXT recursed on every 301 Location with no depth limit and
no visited set, so an A->B->A loop blew the stack. bound it to a named
hop cap and a visited set, iteratively.
- framework cve lookup used best.version ("unknown" when the detector
only fingerprints the framework) and threw away the version
ExtractVersionOptimized dug out of the body, missing CVEs. reconcile
via resolveVersion, preferring the extracted concrete version.
- subdomain takeover flagged a dangling cname whenever a no-such-host
coincided with ANY cname (LookupCNAME echoes the host back for plain A
records). only flag when the cname points off-host at a known
takeoverable provider.
-crawl spiders same-host links/scripts/forms through the shared httpx
client so proxy/headers/rate-limit and robots.txt are honored, bounded
by -crawl-depth. -passive pulls subdomains from keyless ct feeds (crt.sh,
certspotter) and historical urls from wayback, each source isolated so
one feed being down doesn't sink the rest and the target sees no traffic.
three active web-vuln probes wired into the per-target loop:
- cors: crafts attacker origins (evil sentinel, null, prefix/suffix
bypass, http downgrade) and flags responses that reflect them in
access-control-allow-origin, ranking reflection+credentials high.
- redirect: injects a controlled sentinel host plus bypass variants
(//, https:/, backslash, null-byte, userinfo @) into redirect-prone
params and catches 30x location, meta-refresh and js redirects that
resolve off-site.
- xss: injects a unique canary wrapped in breaking chars, classifies
the reflection context (html/attribute/script) and reports only the
chars that survive unescaped where they matter, so escaped
reflections don't false-positive.
all route through httpx.Client so proxy/-H/-cookie/-rate-limit apply.
hermetic httptest coverage plus integration testbed entries.
the -js pipeline already pulls every <script> into a buffer but only
mined supabase jwts from it. reuse that buffer to run a credential
regex bank (aws/github/slack/stripe/google keys, pem blocks, plus
entropy-gated generic apikey/secret/token assignments) and a
linkfinder-style endpoint extractor that resolves relatives to
absolute urls. both dedupe across scripts and surface through the
existing js logger and result struct, no new flag.
every scanner spun up its own &http.Client, so there was no single place
to apply a proxy, custom headers, a cookie or a rate limit. add an
internal/httpx package that builds one configured transport at startup and
hand it to every scanner via httpx.Client(timeout), keeping behavior
identical when nothing is set (plain client when Configure was never
called).
- httpx.Configure wires -proxy (http/https/socks5), -H/--header, -cookie
and -rate-limit into a package-level RoundTripper that paces via a
rate.Limiter and only sets headers the caller hasn't already, so a
scanner's explicit api key still wins.
- route the scan/wordlist downloads that used http.DefaultClient through
the shared client too; ports tcp dialing is left untouched.
- clamp -threads to a floor of 1: it feeds wg.Add across the scanners, so
0 was a silent no-op and a negative value panicked the waitgroup.
document the new flags in the readme, usage docs and man page.
the remaining hardcoded base urls had no test seam, so their drivers could
only be exercised against the live apis. promote them to package vars (matching
the dirlist/git/ports pattern from #112) and route dnslist's per-host probes
through an injectable transport, then add integration tests that pin each at a
local httptest fixture. defaults equal the old const values so behavior is
unchanged.
concurrent workers (-threads 40) all hit the same milestone bucket on
increment, spamming ~10 duplicate [25%] lines. track the last printed
bucket under p.mu and only print when it advances.
the pinned nixpkgs shipped go 1.25.5 but go.mod now needs >= 1.25.7, so the
flake build failed (GOTOOLCHAIN=local). bump the lock to a nixpkgs with go
1.26.3, and refresh the stale vendorHash for the current deps. `nix build`
and `nix run github:vmfunc/sif` work again.
- recenter the detection confidence (sigmoid centered at 0.3) so a single weak
signature match no longer clears the 0.5 threshold. before, sigmoid(0) was 0.5
so *any* match counted as a detection - that's the magento-on-a-plain-page
false positive from the live run. real detections match ~50%+ of signature
weight, so the existing detector tests are unaffected
- getVulnerabilities matched affected versions with a raw string prefix, so "4.2"
also matched "4.20"; match only on dotted boundaries now
- break confidence ties on name so the picked framework is deterministic
- add regression tests for the confidence floor and the version boundary
- make the four wordlist base urls (dirlist/dnslist/git/ports) package vars
instead of consts so tests can repoint them at a local fixture; the default
values are byte-for-byte unchanged
- add internal/scan/integration_test.go behind a //go:build integration tag: it
stands up a local "vulnerable app" httptest server with planted artifacts and
runs git/dirlist/cms/headers/sql/lfi/ports against it, asserting real findings
- go.yml runs them via `go test -tags=integration`; the default test run is
untouched (the tag keeps them out)
- document the integration run in docs/development.md
- add internal/version: resolve from the release ldflag, else the go build
info (module tag / vcs revision), else "dev"
- show the version on the boot banner and for `sif version`
- Makefile now stamps `make` builds via git describe (matching the release ci),
so local/go-install builds report a real version instead of "dev"
- patchnotes.ShowOnce skips pseudo/dev versions so non-release builds dont make
a doomed github call
- document sif version / sif patchnote / SIF_NO_PATCHNOTES in the readme + usage
doSupabaseRequest and the signup call used a bare http.Client{} with no
timeout, so a slow supabase project could hang the whole js scan. thread the
scan's --timeout down through ScanSupabase into every supabase request.
`strings.Split(url, "://")[1]` was copy-pasted in 18 spots and panics on a
schemeless target (index out of range). add a small stripScheme helper in the
scan package - and a guarded equivalent in logger, which cant import scan - so
a bare host degrades gracefully instead of crashing the scan.
adds man/sif.1 covering the targets/scans/options/modules flags, the
patchnote and version commands, env vars (incl SIF_NO_PATCHNOTES), and a
few examples. the install/uninstall make targets now drop it in
$(PREFIX)/share/man/man1.
- `sif patchnote` (also `-pn`) fetches the latest github release and renders
its notes with glamour
- on the first run of a new version those notes are shown once, then recorded
so they dont show again - best-effort, so dev builds, the SIF_NO_PATCHNOTES
opt-out, and any network failure stay quiet
- wire up `var version` so the release `-X main.version` ldflag actually lands,
and add `sif version`
- the readme headline used -all, which isn't a real flag (goflags fatals
on unknown flags), so the three -all examples now use actual flags
- document the new -sh security-header scan in the readme table, usage.md
and scans.md, and fix the -headers section (it dumps headers; -sh grades
them)
- bump the documented go version 1.23 -> 1.25 to match go.mod
strips the emoji from the subdomaintakeover/cloudstorage/supabase log
prefixes and trims comments that just restate the code (the
parameters/returns block on SubdomainTakeover, a couple of "this provides
type safety" notes) so the scanners read like whois.go/headers.go.
vmfunc