Skip to content

refactor(crawlers)!: Enter and exit only the objects a crawler creates - #2445

Open
Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:ownership
Open

Mantisus wants to merge 2 commits into
apify:v2from
Mantisus:ownership

Conversation

@Mantisus

@Mantisus Mantisus commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

Description

  • A crawler now enters and exits only the objects it creates. A session pool, browser pool, HTTP client, rendering type predictor or PydanticAI extractor passed to it is the caller's to enter before run(), and run() raises otherwise. The crawler no longer closes it after a run or resets it on the next one. Statistics stay the exception: the crawler enters them on every run, so crawlers can share an instance one after another, but not at the same time.
  • PydanticAiDirectExtractor becomes an async context manager, and session pools, statistics, HTTP clients and the default rendering type predictor can be entered again after a failed or cancelled entry or exit. For details, see "Crawlers manage only what they create" in the upgrade guide. The docs examples and project templates follow in a separate PR.

Issues

Testing

  • New unit and integration tests for passed and own objects of every crawler, and for failed or cancelled entries and exits.

✍️ Drafted by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant