modules/recon/metabase-api-exposure.yaml flags a Metabase instance that exposes
a live setup token without authentication, keyed on a non-null uuid token paired
with the anonymous tracking setting, then extracts the version tag. A live token
is the pre-auth chain behind CVE-2023-38646; a patched instance reports it as
null and is left alone.
modules/recon/zeppelin-api-exposure.yaml flags an Apache Zeppelin server that
discloses its version and build commit over the anonymous version api, keyed on
the version banner paired with the git commit id, then extracts the version. The
endpoint stays anonymous even on a shiro-secured instance, so this is rated as a
version leak rather than an auth bypass.
modules/recon/jupyter-api-exposure.yaml flags a Jupyter server whose status api
answers without a token, keyed on the activity, connections and kernels fields
it reports, then extracts the running kernel count.
internal/modules/analytics_ui_exposure_test.go drives the three modules end to
end through ExecuteHTTPModule and asserts the leak alongside the near misses a
strict review wants pinned: each service with one keying field missing, a
patched metabase that nulls its token, a generic version response, a plain 200
and a 404.
verify: go test ./internal/modules, each matcher and extractor proven to bite
(break -> red, restore -> green).
shared/multi-SAN certificates list unrelated names, so collecting every
crt.sh/certspotter entry reported off-scope hosts as subdomains of the
target. keep only the target or a subdomain of it.
the data:// wrapper payload contains the base64 "PD9waHA" that the
base64-php evidence pattern matches, so a server reflecting the request
path was flagged as a critical inclusion. skip the self-reflected match.
isAdminPanel's default branch (the generic /db/, /sql/, /mongodb/ paths) matched
the keyword "query", which is a substring of jQuery and querySelector. any site
whose catch-all returns 200/403/401 at one of those paths and runs any javascript
was reported as a database admin panel at high severity.
drop "query". the remaining keywords (database, sql, mysql, postgresql, mongodb)
only match db-topical pages, so real generic interfaces are still detected via a
sibling keyword.
detectJoomla matched any body containing the lowercase word "joomla", so pages
that merely mention it (joomla hosting marketing like hostinger.com) were
misreported as Joomla. the canonical Joomla! generator tag is capitalised, so
the lowercase match also missed sites whose only tell is that tag.
match the Joomla! generator meta tag and the joomla asset paths
(/media/vendor/joomla, /media/system/js/core.js) instead. keying on the meta
attribute rather than the plain tagline also rejects prose that quotes it.
pointsAtSentinel parsed the reflected Location with url.Parse, which only
finds the authority when the scheme is followed by "//". the redirect probe
already sends "https:/host" and "https:host" payloads, but a server that
reflected them landed the host in Opaque or the path, so the off-site
redirect went unreported. normalise the scheme slash count to "//" before
parsing so those forms resolve to the host a browser would navigate to.
scoped to http(s), so an opaque "mailto:host" is left untouched.
the ghost detector keyed on the bare "ghost-" body substring, which fires on
any page using the common "ghost-button" or skeleton-loader CSS class (e.g. a
WordPress theme), scoring 0.65 and reporting it as Ghost CMS. key on the
canonical `<meta name="generator" content="Ghost` tag instead, matching the
existing astro detector. the /ghost/api/ path and Ghost header markers are
unchanged.
a ghost site that emits neither the generator meta nor a /ghost/api/
reference is no longer detected; the Ghost response header alone scores
below the detection floor.
the phoenix and adonis detectors matched bare substrings ("phx-", "phoenix",
"adonis") that fire on unrelated pages: a "phx-" css class on a phoenix,
arizona site, or any markup containing the word "adonis". replace them with
markers the frameworks actually emit. phoenix keys on the liveview container
attributes data-phx-main, data-phx-session and data-phx-static; adonis on its
default adonis-session cookie.
this narrows detection: plain (non-liveview) phoenix and session-less adonis
apis are no longer matched. the markers we now key on (liveview's container
attributes, adonis's default session cookie) are ones ordinary prose cannot
forge. each detector gains a true-positive test and a false-positive tripwire.
the engine declared size matchers and kv extractors but the executor
dropped them (size fell through to the default case, kv was never read).
wire both: size matches the response body length in bytes, kv records
every response header as a key-value pair namespaced by the extractor
name.
this unblocks the headers.go conversion in #52, which needs a full header
dump the known-set regex extractors cannot reproduce; the headers.yaml
module and the headers.go removal are a separate follow-up. the extractor
is named kv to match docs/modules.md (the struct comment said kval). the
declared json extractor stays deferred since it needs a json-path
dependency and a path-syntax decision.
refs #52
cover three platforms the built-in cms scanner misses (it only handles
wordpress, drupal and joomla). markers are structural: generator meta,
framework-specific js init and asset paths, not bare brand strings, so a
page that merely mentions the cms does not match. ghost also extracts its
version from the generator meta.
the yaml module set had wordpress and drupal but not joomla, while the legacy
internal/scan/cms.go detects all three. the new module fills that gap so
--all-modules and -mt cms cover joomla too.
matches the generator meta (version-independent, joomla 1.5 through 5) plus
structural markers /media/system/js/core.js, /media/jui/ and
joomla-script-options, on the root and /administrator/. verified against live
and archived joomla sites, with no false positives on pages that only mention
joomla. version comes from a versioned generator or a leaked
X-Content-Encoded-By header.
additive: cms.go is untouched, the -cms scan is unchanged. whether converted
scanners should also run in the default flow is the open question on #52.
refs #52
ship findings to chat/webhook sinks after a scan so continuous recon can
alert on what it turns up. each provider is one POST through httpx.Client,
so the global proxy/rate-limit/header config applies and there's no extra
http stack. config resolves env-first (SLACK_WEBHOOK_URL, DISCORD_WEBHOOK_URL,
TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID, NOTIFY_WEBHOOK_URL), overridable by a
notify-compatible yaml file so existing projectdiscovery/notify configs port
over. -notify enables it, -notify-severity gates on the finding severity
ladder (default medium), -notify-config points at the yaml. wired after the
scan loop on the severity-filtered finding set; no provider configured is a
silent no-op.
re-scans become a monitor: -diff snapshots each target's normalized
findings to a per-target json file and, on the next run, surfaces only
the delta (+ new / - gone) against the last snapshot, then overwrites it
so each run diffs against the previous one. behavior is unchanged when
-diff is off.
new internal/store keys the set-difference off finding.Key (already
stable across runs) and uses only encoding/json + os - no new deps.
snapshot files are sanitized per target (no traversal), written 0600
under 0750 dirs. -store picks the location: explicit dir, else the log
dir, else <user-config>/sif/state. a missing snapshot is a clean
baseline, a corrupt one self-heals on the next save.
static i%threads stride partitions assign each item to a fixed worker, so a
goroutine stuck on a slow or timing-out host stalls while the rest idle behind
it (head-of-line blocking). add internal/pool.Each: workers all pull from one
shared channel, so a slow item only blocks its own worker and the others keep
draining. migrate git, ports, robots (scan), dnslist, dork, dirlist and
subdomaintakeover off the stride loops; per-item work, mutex-guarded appends and
progress increments are unchanged, results were already unordered.
jwt fetches the target once then analyzes every harvested token offline:
flags alg:none, the rs256->hs256 confusion surface, missing/expired exp
and plaintext sensitive claims, and cracks a small bundled weak-hmac list.
openapi probes the conventional spec paths, parses json/yaml and enumerates
paths plus unauthenticated operations. favicon computes the shodan-style
mmh3 hash (python base64.encodebytes chunking, signed int32) for tech
fingerprinting and the http.favicon.hash pivot, pinned by a golden test.
sif can now slot into unix pipelines. stdin is drained for targets when
it's a pipe (keyed off stdin's mode, not stdout), alongside -u/-f. naked
hosts are accepted and default to https://; explicit http(s) is kept,
other schemes rejected. -silent routes all banner/spinner/log chrome to
stderr and prints one normalized finding per line to stdout via
finding.Flatten, so `subfinder | sif -silent | notify` works.
go only returns a conn to the idle pool when the body is read to EOF
before Close. the header-only and early-return scan paths closed an
unread body, leaking the conn and forcing a fresh dial each request.
route those close sites through httpx.DrainClose so the tuned pool from
phase 1 actually gets reused. body-read paths (scanner/io.ReadAll) are
left untouched.
dnslist previously http-probed every wordlist candidate through the
blocking os resolver, so a big list meant a request per dead name and a
wildcard zone flooded results. resolve each candidate first via a new
internal/dnsx (retryabledns over a bundled 1.1.1.1/8.8.8.8/9.9.9.9 pool,
promoted to a direct dep), fingerprint the apex with random nonexistent
labels to detect catch-all zones, and http-probe only the names that
actually resolve and aren't wildcard. add -resolvers to override the
pool. resolverFn is a package-level seam so the dnsx tests stay
hermetic; the dnslist newDNSResolver seam keeps the integration test
network-free.
scan results live in ~two dozen structs with no shared shape, so every
consumer that wants "what did this run turn up" reimplements the
type-switch. add internal/finding: an ordered Severity (info<low<medium<
high<critical, with parse/compare) and Flatten, the single type-switch
that collapses every scan result struct into flat, severity-ranked
Findings keyed module:identifier for stable dedup/diff.
wire collectFindings off Flatten in the run loop so notify and diff
(later bundles) build on one normalization path instead of re-deriving
it; the report path keeps emitting raw json blobs unchanged. expose
JavascriptScanResult.SupabaseFindings so the js internals stay private.
the guard test iterates a representative instance of every ResultType
and fails if Flatten lacks a case (falls through to :unhandled) - so a
new scanner can't ship without a Flatten case landing too.
the shared transport was a bare DefaultTransport.Clone() with the stock
MaxIdleConnsPerHost=2, and call-sites only close response bodies without
draining them - so go could never return a conn to the idle pool and every
request re-dialed. high thread counts just thrashed the dialer.
- plumb Threads through Options into buildTransport; size MaxIdleConnsPerHost
to the worker count (floored) so concurrent workers on one host pool instead
of re-dialing, MaxIdleConns=512, MaxConnsPerHost=0, IdleConnTimeout=90s,
ForceAttemptHTTP2. the socks5 branch gets its own keepalive net.Dialer so it
doesn't lose os-level pooling under proxy.Direct.
- add DrainClose to read (capped) and close a body so the conn is reusable.
- benchmark proves it: 50 sequential requests reuse 1 conn tuned vs 50 bare.
adds an httpx-style -probe scanner reporting liveness, final status, page
title, server header and the redirect chain, plus -sarif/-markdown export
flags that serialize the collected run after the scan loop. the report
serializers live in a decoupled internal/report package consuming a raw-json
result model so they never import scan types.
the old scanner surfaced every response that wasn't 404/403, so modern SPA
catch-all 200s flooded the output and made -dirlist near-useless. add ffuf-style
matching:
- -mc/-fc/-fr and -fs/-fw filter by status, regex, body size and word count;
bodies are read through a capped io.LimitReader so size/word counts are
deterministic and memory stays flat. filters win over matches.
- -ac auto-calibrates the soft-404 baseline from a few deterministic
non-existent paths and drops responses matching that wildcard shape.
- -w overrides the size switch with a local file or remote list (fetched through
the shared client so proxy/rate-limit apply); -e appends extensions per word.
size and words are added to DirectoryResult for the json output.
the yaml module engine (the user-facing extensibility surface) had 0%
test coverage. add table-driven tests for the matcher types
(status/word/regex + and/or + negative), checkWords/checkRegex (incl
invalid-pattern fail-closed under AND, skip under OR), runExtractors
(regex capture groups, group-index bounds, part selection),
substituteVariables and generateHTTPRequests (path x payload expansion),
and ParseYAMLModule on valid + malformed yaml. drive ExecuteHTTPModule
end-to-end against an httptest server through the shared httpx client so
matcher hits and extractor captures are exercised for real. coverage
0% -> 93.7%.
also: ExecuteDNSModule/ExecuteTCPModule were stubs returning an empty
result with nil error, so a type:dns/type:tcp module silently reported
"0 findings" - indistinguishable from a real clean scan. make them
return ErrUnsupportedModuleType (sentinel, wrapped with the module id) so
the existing caller logs a clear failure instead. a test pins the new
behavior.
bodyclose is excluded for test files in .golangci.yml: the synthetic
*http.Response fixtures carry no socket, mirroring the existing _test.go
slack for errcheck/noctx/gosec.
four recon-flagged bugs, each with a focused test:
- dnslist fired both http and https per candidate and counted a "found"
on any non-error response (incl 404 and wildcard catch-all redirects),
so every host double-counted and a wildcard-dns host flooded results.
probe http then https with per-subdomain dedupe, gate on a meaningful
(2xx, non-redirect) status, and stop chasing redirects so a catch-all
301 reads as a redirect instead of a 200.
- fetchRobotsTXT recursed on every 301 Location with no depth limit and
no visited set, so an A->B->A loop blew the stack. bound it to a named
hop cap and a visited set, iteratively.
- framework cve lookup used best.version ("unknown" when the detector
only fingerprints the framework) and threw away the version
ExtractVersionOptimized dug out of the body, missing CVEs. reconcile
via resolveVersion, preferring the extracted concrete version.
- subdomain takeover flagged a dangling cname whenever a no-such-host
coincided with ANY cname (LookupCNAME echoes the host back for plain A
records). only flag when the cname points off-host at a known
takeoverable provider.
-crawl spiders same-host links/scripts/forms through the shared httpx
client so proxy/headers/rate-limit and robots.txt are honored, bounded
by -crawl-depth. -passive pulls subdomains from keyless ct feeds (crt.sh,
certspotter) and historical urls from wayback, each source isolated so
one feed being down doesn't sink the rest and the target sees no traffic.
three active web-vuln probes wired into the per-target loop:
- cors: crafts attacker origins (evil sentinel, null, prefix/suffix
bypass, http downgrade) and flags responses that reflect them in
access-control-allow-origin, ranking reflection+credentials high.
- redirect: injects a controlled sentinel host plus bypass variants
(//, https:/, backslash, null-byte, userinfo @) into redirect-prone
params and catches 30x location, meta-refresh and js redirects that
resolve off-site.
- xss: injects a unique canary wrapped in breaking chars, classifies
the reflection context (html/attribute/script) and reports only the
chars that survive unescaped where they matter, so escaped
reflections don't false-positive.
all route through httpx.Client so proxy/-H/-cookie/-rate-limit apply.
hermetic httptest coverage plus integration testbed entries.
the -js pipeline already pulls every <script> into a buffer but only
mined supabase jwts from it. reuse that buffer to run a credential
regex bank (aws/github/slack/stripe/google keys, pem blocks, plus
entropy-gated generic apikey/secret/token assignments) and a
linkfinder-style endpoint extractor that resolves relatives to
absolute urls. both dedupe across scripts and surface through the
existing js logger and result struct, no new flag.
every scanner spun up its own &http.Client, so there was no single place
to apply a proxy, custom headers, a cookie or a rate limit. add an
internal/httpx package that builds one configured transport at startup and
hand it to every scanner via httpx.Client(timeout), keeping behavior
identical when nothing is set (plain client when Configure was never
called).
- httpx.Configure wires -proxy (http/https/socks5), -H/--header, -cookie
and -rate-limit into a package-level RoundTripper that paces via a
rate.Limiter and only sets headers the caller hasn't already, so a
scanner's explicit api key still wins.
- route the scan/wordlist downloads that used http.DefaultClient through
the shared client too; ports tcp dialing is left untouched.
- clamp -threads to a floor of 1: it feeds wg.Add across the scanners, so
0 was a silent no-op and a negative value panicked the waitgroup.
document the new flags in the readme, usage docs and man page.
