detect cloudflare waf, sucuri, incapsula, f5 big-ip asm and modsecurity
from the headers, cookies and block-page strings the waf itself sets, so
a site that merely names the vendor in prose does not trip them. grouped
in one file so the category can be reviewed or excluded as a unit.
* refactor(fingerprint): make the favicon tech table the single source
move the hash->tech map from internal/scan into internal/fingerprint
beside the hash function, exposed as LookupFaviconTech. the scan
Favicon path now resolves tech through it instead of a private map.
drop the demo-sync guard test: it existed only to keep scan's map in
sync with the yaml demo module, and that map no longer exists.
* feat(modules): name the matched tech in favicon evidence
favicon module findings now read the tech straight from the shared
fingerprint table, so a canonical hit reports "favicon mmh3=<n>
tech=<name>" instead of a bare hash. unknown hashes are unaffected.
the probe title field is documented as an httpx-style page title, but
extractTitle returned the raw regex capture, so a title like
"Tom & Jerry" was reported verbatim instead of "Tom & Jerry".
run it through html.UnescapeString before trimming so the reported
title matches the rendered page.
* fix(scan): tune js secret rules for false positives and missed formats
drop stripe pk_ publishable keys (public by design) and require a digit in
the generic secret value so camelCase identifiers stop tripping the entropy
gate. add stripe rk_ restricted keys, github fine-grained pat (github_pat_),
encrypted pem headers and version-anchored slack xapp tokens; drop the
trailing word boundary on the aws-secret and google rules so keys ending in
/ + or - still match.
renames the "stripe live key" rule to "stripe secret key", which changes
the rule label in the json findings output.
* feat(scan): detect more provider keys in javascript
add unique-prefix credential rules to the js secret bank: gitlab pat
(glpat-), anthropic api/admin keys (sk-ant-), npm tokens (npm_), google
oauth client secrets (GOCSPX-), stripe webhook secrets (whsec_), shopify
app tokens (shp[at|ss|pa|ca]_), sendgrid keys (SG.) and slack incoming
webhook urls. all ride the no-entropy slot since the prefix alone is
proof, so they carry near-zero false-positive risk.
* fix(logger): create log dirs and flatten url log filenames
CreateFile/Write kept the '/' from a target's url path when building the
log filename, so a target like http://host:port/ tried to open
<dir>/host:port/.log whose parent directory never existed. os.OpenFile
failed and scanTarget aborted the whole target before any scanning ran.
fold every '/' and '\\' run in the sanitized url into a single '_' via a
shared logPath helper so CreateFile and Write always resolve to the same
flat file, and mkdir the parent defensively in getWriter as a second line
of defense.
* fix(scan): route module status lines through the locking sink
subdomaintakeover, cloudstorage, and the next.js framework detector
printed their status/error lines with a bare fmt.Println straight to
os.Stdout, bypassing the output package's sink. under -concurrency>1
only the sink is lock-wrapped, so these lines could interleave or tear
against lines other targets write concurrently.
swap them for output.ScanStart / output.Error, matching every other
scanner in the package.
* fix(store): make snapshot filenames injective and writes atomic
sanitize() folds every separator run (and a literal '_') to one '_', so
distinct targets like https://a.com/x and https://a.com//x, or
host:8443/path and host_8443_path, sanitized to the identical string and
shared one snapshot file - the second target's Save silently clobbered
the first's baseline. Save's os.WriteFile also wasn't atomic, so two
targets racing on a collided path (or -concurrency>1 in general) could
interleave partial writes.
pathFor now appends 16 hex chars of the target's sha256 to the sanitized
name, so distinct targets never collide, and Save writes through a temp
file in the same dir followed by os.Rename so a reader always sees a
complete snapshot or the previous one, never a partial write.
this changes the on-disk snapshot filename scheme: existing users' -diff
baselines under the old sanitize()-only names won't be found and the
next run re-baselines from scratch (every finding reports as newly
added once). noting this as a deliberate tradeoff for a local hardening
pass rather than silently orphaning them.
* fix(store): check the deferred temp-file cleanup error
golangci-lint (errcheck) flagged the bare defer os.Remove(tmpPath) in
writeFileAtomic; wrap it so the discard is explicit.
* fix(js): stop the scanning spinner on early script-parse errors
htmlquery.Parse and QueryAll failures returned before spin.Stop(), leaving
the "Scanning JavaScript files" spinner running forever on those paths while
every other error branch in the function stops it first.
* fix(js): stop reporting quoted regex-flag literals as endpoints
Quoted regex flag combos like "/gi" or "/su" in ordinary JS (e.g.
str.replace("/gi", x)) matched the root-relative path alternative in
endpointRegex and were reported as endpoints. Denylist the known flag
bigrams by exact match instead of raising minEndpointLen, so real short
endpoints like /v1, /me, /ws still extract normally.
* fix(js): keep supabase findings from other projects on a per-project error
ScanSupabase iterates every JWT found in a script and can discover several
distinct supabase projects. Four error paths inside that loop (reading the
signup response, parsing it, fetching the openapi spec, and a missing
Paths field) returned nil and the whole error, discarding any results
already collected for earlier, successfully scanned projects. Skip the
broken project with continue instead so the scan still returns what it
found.
* test(js): drop flaky wall-clock redos timing probe
go's regexp is re2 and linear by construction; the timing threshold added
only flake under -race load, not coverage. the regex-flag-literal false
positive it sat next to is guarded by TestRegexFlagEndpointsNotReported.
* fix(detectors): let django csrf body field detect
csrfmiddlewaretoken is a hidden form body field django templates
render, never a header, so marking it HeaderOnly meant it could never
match and a real django form page (csrf field plus csrftoken cookie)
went undetected.
* fix(detectors): tighten aspnet, angular, astro and nextjs markers
* fix(frameworks): surface header version patterns to extraction
version extraction only ever searched the response body, so
header-shaped patterns like ASP.NET's "X-AspNet-Version: x.y.z" or
Flask's "Werkzeug/x.y.z" Server header could never match even when the
detector itself fired off that same header. add
ExtractVersionFromResponse, which also searches canonical header
lines, and point aspnet and flask at it. fix the aspnet header regexes
to match case-insensitively, since Go canonicalizes header names
("X-AspNet-Version" becomes "X-Aspnet-Version") and the old literal
pattern never matched the canonical form.
* test(detectors): pin fp/fn corpus for framework detection
promote the ad-hoc probe/sweep scratch tests used to find these
defects into a proper regression file: for each fix, assert both the
real-product positive still detects and the prose/other-product
negative does not, plus a sweep of unrelated pages against every
registered detector.
* docs(scan): correct crawl's false robots.txt claim
the doc comment said robots.txt is respected because colly honors it
by default, but colly's default is IgnoreRobotsTxt=true and Crawl never
flips it, so Disallow rules were never enforced. state the actual,
intentional behavior instead: a recon crawler has no reason to treat
Disallow as a scope boundary.
pin the behavior with a regression test so it isn't "fixed" into a
partial robots.txt implementation by accident later.
* fix(scan): cap crawl request fanout
MaxDepth only bounds recursion depth; colly's own MaxRequests budget
was left at 0 (unlimited) and Crawl set no Limit rule, so a hostile
page fanning out to an attacker-controlled number of links got every
one of them fetched at that depth (b^d). cap the total number of
requests a single Crawl call will make with a fixed internal budget.
TestCrawl_BoundsRequestFanout proves a 2500-link fanout page no longer
drives the server past maxCrawlRequests+1 actual fetches.
* fix(scan): stop redirects from escaping crawl scope
Crawl called SetClient(httpx.Client(timeout)), which replaces colly's
whole *http.Client and discards the CheckRedirect colly installs to
re-check AllowedDomains on every redirect hop. with it gone, redirects
fell back to net/http's default: follow up to 10 hops to any host. an
in-scope page could 302 the crawler to an arbitrary off-scope host
(e.g. a cloud metadata endpoint) and it would be fetched.
colly's own AllowedDomains only matches on hostname anyway, which
would still let a same-host, different-port redirect through (an
internal service reachable on the same IP). build our own http.Client
on top of httpx's shared transport instead, with a CheckRedirect that
requires the redirect target to share the exact host:port of the
request that produced it, and caps the hop count ourselves since a
custom CheckRedirect drops net/http's own 10-redirect default.
TestCrawl_RedirectRejectsOffScopeHost proves the off-scope host is no
longer fetched; TestCrawl_RedirectFollowsSameHost proves ordinary
same-host redirects still are.
the google-search client only swaps spaces for '+' and drops the term
into the query string verbatim, so a dork carrying a raw '#' or '&' cut
the request url short at the fragment or split it into stray query
params. the search then ran against a fragment of the intended dork.
encode the term up front so the whole dork survives.
truncateBanner sliced by byte offset, so a multibyte utf-8 rune landing
across the limit was cut mid-sequence and emitted invalid utf-8. shodan
banners routinely carry non-ascii bytes, so this corrupted logged and
printed output. count runes instead so the cut lands on a boundary.
* fix(scan): honour the caller context when downloading the ports list
the common-scope ports fetch built its request with context.TODO(),
so a cancelled run or -max-time deadline could not abort a hung list
download even though the dialer already used the passed context.
* test(scan): drop redundant comment restating release channel receive
GetPagesRouterScripts fetched _buildManifest.js with httpx.Client(0),
which sets no client timeout, so a slow or hostile manifest host could
hang the whole scan on the response read. thread the caller's scan
timeout through instead, matching every other fetch in the package.
the wayback line scanner capped each line at 1mb, so a single archived
url with a huge query string made scanner.Err return ErrTooLong and the
whole feed was discarded. read lines without a per-line cap, still bounded
by the passiveMaxBytes read limit.
drupal's generator meta only exposes a bare major, so the extracted
version is "10"/"9". versionAffected only matched when the affected
entry was a component-prefix of the version, never the reverse, so a
bare major never matched a dotted affected entry and every drupal cve
was silently missed. match on dotted boundaries in either direction.
numericClaim only accepted exp as a json number, but some emitters
serialize it as a quoted string. a token with a past string exp was
reported as missing exp instead of expired, an expiry false negative
and a missing-claim false positive on the same token. also parse a
string value via strconv.ParseFloat, falling back to absent when it
does not parse.
hsts grading was gated on the scheme of the originally requested url,
but the client follows redirects, so an http target that redirects to
https skipped the hsts check entirely and dropped a high-severity
finding. decide the scheme from the final response request url instead,
falling back to the requested url only when no response request is set.
paths was typed as a map of operation structs, so a spec with a $ref
string path item (valid in openapi 3.1) failed to decode and the whole
spec was dropped with no finding. decode path items into a generic tree
and extract per-operation security manually, treating a non-array
security value as absent so it inherits the global default rather than
being reported as an anonymous endpoint.
pointsAtSentinel parsed the reflected location without the whitespace
and control-byte removal a browser applies, so payloads like a
space-prefixed scheme-relative host or a tab embedded in the scheme
parsed to an empty host and a real off-site redirect was missed. strip
leading and trailing c0-or-space and all tab, cr and lf before parsing,
matching the navigation-time normalization, without removing mid-string
spaces so a same-site target is never reclassified as off-site.
the http scheme downgrade origin was queued against every target, so on
a plain-http target the http://host probe is the target's own origin and
any correct same-origin reflection was reported as a high-severity
downgrade. mark that origin https-only and skip it when the target
scheme is not https, where a downgrade is meaningless, while still
firing it on https targets that genuinely trust the http origin.
checkDatabaseErrors matched db error regexes against each response with
no baseline, so a site serving a shared error template, custom 500 or
waf block page containing a db error string reported an identical
high-severity sql injection finding on every probed path. capture the
main-url response as a baseline and skip, per pattern, any error string
already present in it while still checking other patterns, so only
differential probe-caused errors are reported.
the lfi scan matched evidence regexes against each response with no
baseline, so a page whose static content matched a pattern (a blog
containing an /etc/passwd-shaped line, a code sample with <?php) was
reported as an lfi finding for every parameter and payload combination.
take one baseline get of the unmodified target and exclude, per
evidence class, any pattern already present in it, so only
payload-caused disclosures are counted.
the live-response path flagged any host whose body matched a provider
fingerprint without checking dns, so a page merely containing a
provider 404 string was reported as a takeover. resolve the cname and
only confirm when it maps to the same provider; a contradicting cname
kills the flag. providers with a body signature but no apex to
correlate, and unresolvable lookups, degrade to a medium potential
finding via a new confidence field rather than confirming or dropping.
Crawl only capped recursion depth, not breadth per level. A link-heavy page
(pagination, faceted search) drives fetch count toward branching^depth with no
ceiling: a depth-3 crawl of a page linking 40 fresh urls issued 1641 real
fetches.
Cap total fetches at a page budget via an atomic counter in an OnRequest hook
that aborts once exceeded, and add a Truncated flag on CrawlResult so callers
can tell a run was cut short.
a detector accuracy audit surfaced two classes of bug in the framework
detectors.
bare-brand header false positives: header-only signatures matched a
brand name as a substring across every header name and value, so a
detector fired on any response that merely referenced the brand (a
vendor cdn named in a link or csp value, a cookie sharing the prefix).
add an optional Header field to Signature that scopes a header-only
match to one named header's value, and apply it (or a structural
anchor) per detector:
- express: "Express" scoped to x-powered-by, was firing on an
express_checkout cookie.
- flask: "Werkzeug" scoped to the server header.
- symfony: dropped the bare "symfony" word (symfony sets no such
header, it fired on symfony.com links); the x-debug-token header is
the marker.
- shopify: key on the x-shopify response headers instead of the bare
"Shopify" word, which fired on a cdn.shopify.com link.
- remix: dropped the bare "remix"/"_remix" substrings that fired on a
track_remix.mp3 asset; window.__remixContext is the definitive
marker.
- spring boot: anchor the whitelabel title in its h1 tag context so a
tutorial discussing the error does not fire.
the gin and fastapi detectors are removed: gin keyed on the
"gin-gonic" import-path string (appears in tutorials, never in a real
gin response) and fastapi on bare words matching the projects' doc
domains. neither framework advertises itself in a response header or a
non-prose body marker, so there is no clean passive signal to anchor
on.
version mis-extraction: drop the low-confidence ".*?" version
fallbacks (rails, django, laravel, spring), whose unbounded gap
grabbed the first version-shaped number after the framework word and
reported an unrelated asset's cache-buster when no real version was
present. let isValidVersionString accept a single integer so a bare
major such as drupal's "Drupal 10" is no longer rejected as "unknown".
each false positive and version bug is covered by a regression test.
Ran `gofmt -w .` accross the repo to fix formatting drift.
Mechanical `gofmt -w .` only. No functional or behavioural changes.
CONTRIBUTING.md requires gofmt-clean code; these files had slipped.
rename the go module path from github.com/dropalldatabases/sif to
github.com/vmfunc/sif across go.mod, all imports, the golangci exclude
list, release install docs and docs. pure string rename, no logic change.
#180 added the calibrate bool to SQL but the integration-tagged test
(only built under -tags=integration, outside normal CI) still called the
4-arg form. pass false (no calibration) to restore behavior.
replace bare gin/api/cake/svelte/ember/backbone/meteor substrings with
specific markers (Backbone.Model, ember-application, __meteor_runtime_config__,
CAKEPHP cookie, svelte/internal) so prose and CORS headers stop matching.
adds paired false-positive/positive coverage per framework.
calibrate against reflecting catch-alls whose body size tracks path
length so exact-shape calibration no longer misses them; -ac now drives
both dirlist and sql. updates the admin-panel query test to the new SQL
signature. adds soft-404 + calibration coverage.
CMS only flagged Drupal on X-Drupal-Cache: HIT or the Drupal 7 Drupal.settings
marker, so Drupal 8-11 went undetected: a MISS cache header was ignored, and
cdn-fronted sites serve none of those markers in the body at all. verified live
that london.gov.uk and georgia.gov (both Drupal) were missed.
key on the Drupal-specific headers instead: any X-Drupal-Cache or
X-Drupal-Dynamic-Cache, plus X-Generator naming Drupal. these survive cdn
caching. drupalSettings (8+) and Drupal.settings (7) cover uncached bodies.
load yaml-defined detectors from ~/.config/sif/signatures (AppData\Local
on windows), mirroring the user-modules convention, so a framework sif
does not ship can be detected without a rebuild. they load lazily once
per run from DetectFramework and register alongside the built-ins.
each file is one detector, scored by the same weighted signature match as
the built-ins. confidence is linear rather than their sigmoid (importing
it would cycle), so a detector clears the 0.5 threshold once its matched
weights pass half. a name matching a built-in overrides it and inherits
that built-in's version patterns and cves, the same as a user module. a
single unparseable file warns and is skipped rather than failing the scan.
implements the custom signature support help-wanted item in contributing.
detectWordPress treated any 2xx on /wp-login.php, /wp-admin/ or /wp-config.php
as wordpress, but the http client follows redirects and many sites soft-404
(200 for every path) or catch-all redirect to their homepage. a probed path
then returns 200 without being wordpress at all, so plain static sites and
marketing pages were flagged as wordpress.
read the probed response and require one of the existing wordpress markers in
its body before counting it. a real /wp-login.php still references wp-includes
assets, so genuine detection holds.
classifyXSSContext defaulted any non-html, non-script reflection to attribute context. a reflection echoed into element text with the angle brackets escaped but quotes left raw (common for encoders limited to < > &) was then reported as a high-severity attribute injection, even though a quote in text content has no delimiter to break out of.
classify a reflection as attribute only when the canary actually lands inside an open tag; everything else is inert element text and yields no finding. the in-tag test is a byte-scan, not a parser, so rare malformed markup can still mis-bucket.
the ghost and pantheon entries keyed on each platform's older unclaimed-domain
copy, which the live pages no longer serve, so a real dangling subdomain went
undetected. verified against live unclaimed subdomains: <random>.ghost.io now
returns "Failed to resolve DNS path for this host" and <random>.pantheonsite.io
returns a "404 - Unknown site" page. key on those instead.
ghost also serves a bare "Site unavailable." on some hosts; the body match holds
one substring per service, so it keys on the distinctive arm and a page showing
only "Site unavailable." is not detected.
The cors probe followed up to three redirects, then read the cors headers off the final response. Its own comment said the cap existed so the headers came from the host we asked about, so the code contradicted its intent: a target that only redirects to a permissive third party had that party's misconfiguration reported against it.
Stop following redirects, matching the open-redirect scanner. The redirect cap was a const shared with the xss probe, so xss now owns its own xssMaxRedirects with its follow behavior unchanged.
the extractPotentialBuckets TODO asked to handle non-adjacent label
combos and strip the tld. strip the trailing tld label so we stop
guessing it ("com", "com-s3", "example-com"), and pair every label
with every other rather than only adjacent ones, so a deep host like
shop.cdn.example yields shop-example too.
this widens the candidate set (and the s3 probes) per host. multi-part
suffixes like .co.uk still leave a junk label; publicsuffix would
refine that as a follow-up.
scanLines used a default bufio.Scanner (64k token cap) and never
checked scanner.Err(), so a wordlist line past the cap silently halted
the scan and dropped that line plus everything after it. return the
error so the wordlist loaders fail loud instead of running against a
truncated list.
move FaviconHash and its base64 chunking into a leaf package so the scan check and the module engine share one implementation instead of each carrying a copy. no behavior change; the golden test moves with it.
detectWordPress flagged any page whose body merely contained the word
"wordpress", so sites that only mention it (wp-hosting marketing, comparison
pages) were misreported: acquia.com, a Drupal site, was detected as WordPress.
wp-content, wp-includes and wp-json already identify a real install and appear
on every wordpress page, so the bare word adds only false positives. drop it.
