mirror of
https://github.com/lunchcat/sif.git
synced 2026-07-28 14:37:01 -07:00
TigahandGitHub
0d96838ee4
Crawl only capped recursion depth, not breadth per level. A link-heavy page (pagination, faceted search) drives fetch count toward branching^depth with no ceiling: a depth-3 crawl of a page linking 40 fresh urls issued 1641 real fetches. Cap total fetches at a page budget via an atomic counter in an OnRequest hook that aborts once exceeded, and add a Truncated flag on CrawlResult so callers can tell a run was cut short.