Skip to content

fix(crawlers)!: Keep statistics and sessions between run() calls - #2449

Draft
Mantisus wants to merge 3 commits into
apify:v2from
Mantisus:keep-state-across-runs
Draft

Mantisus wants to merge 3 commits into
apify:v2from
Mantisus:keep-state-across-runs

Conversation

@Mantisus

Copy link
Copy Markdown
Collaborator

Description

  • A repeated run() continues the crawl: it adds to the statistics of the earlier runs, error trackers included, and keeps the sessions of the session pool. max_requests_per_crawl counts the requests of the earlier runs too, so a run that starts with the limit already reached stops right away. To count from zero, call crawler.statistics.reset() before the next run().
  • The same change for Crawlee JS is proposed in v4: A repeated run() continues the queue but resets the statistics the crawler built crawlee#4397.

Issues

Testing

  • Unit and integration tests for repeated runs with own and passed statistics and session pools.

✍️ Drafted by Claude Code

@Mantisus Mantisus self-assigned this Nov 15, 2026
@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 15, 2026
@Mantisus
Mantisus force-pushed the keep-state-across-runs branch from e3dc508 to 41411cc Compare November 15, 2026 22:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant