Skip to content

Commit b8dc178

Browse files
committed
enter and exit only the objects a crawler creates
1 parent 7de320a commit b8dc178

28 files changed

Lines changed: 861 additions & 193 deletions

File tree

‎docs/06_upgrading/upgrading_to_v2.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -470,6 +470,35 @@ class MyHttpClient(HttpClient):
470470

471471
`SitemapRequestLoader.fetch_next_request` skips a URL while its request is still in progress, for example when another sitemap lists it too. In v1, it returned a request for the URL again, and `get_total_count` counted the URL twice.
472472

473+
## Crawlers manage only what they create
474+
475+
A crawler now enters and exits only the objects it creates, the same as Crawlee JS. An object you pass is yours to enter and exit. The `Statistics` and the event managers are the exceptions, since the crawler enters them on every run. You can enter an event manager yourself as well. In v1, the crawler entered a passed object that wasn't active yet and exited it when its run finished, even while another crawler still used it.
476+
477+
What you do with an object you pass depends on its type:
478+
479+
- `SessionPool`, `BrowserPool`, an HTTP client and the `rendering_type_predictor` of `AdaptivePlaywrightCrawler`: enter it with `async with` before you run the crawler, or `run()` raises a `RuntimeError`. Keep it open until every crawler using it has finished.
480+
- The `extractor` of `PydanticAiCrawler`: enter it the same way. `PydanticAiDirectExtractor` is now an async context manager, like `PydanticAiSelectorExtractor`. A selector extractor enters and exits its `fallback` itself, so don't enter the fallback yourself, and give each selector extractor its own. If your subclass of `BasePydanticAiHtmlExtractor` overrides `__aenter__` or `__aexit__`, call `super()`. A custom extractor that implements the `PydanticAiHtmlExtractor` protocol directly needs the new `active` property, `__aenter__` and `__aexit__`.
481+
- `Statistics`: don't enter it yourself, or `run()` raises a `RuntimeError`. In v1, the crawler left entered statistics alone. Crawlers that run one after another can share an instance, including another crawler's statistics. The instance adds up their runs, so the `max_requests_per_crawl` of the second crawler counts the requests of the first one too. When a crawler runs again, it resets the instance, including what the other crawlers counted. Two crawlers can't run with the same instance at the same time.
482+
483+
For example:
484+
485+
```python
486+
# v1
487+
crawler = ParselCrawler(session_pool=SessionPool(max_pool_size=10))
488+
await crawler.run(urls)
489+
490+
# v2
491+
session_pool = SessionPool(max_pool_size=10)
492+
crawler = ParselCrawler(session_pool=session_pool)
493+
494+
async with session_pool:
495+
await crawler.run(urls)
496+
```
497+
498+
When a crawler runs again, it resets only a session pool it created. A `SessionPool` you pass keeps its sessions. In v1, a repeated `run()` reset a passed pool too.
499+
500+
`RenderingTypePredictor` has a new `active` property, which `initialize()` and `clear()` of the base class keep up to date. If your predictor overrides them, call `super()`.
501+
473502
## Resetting statistics and the session pool
474503

475504
`Statistics.reset()` is now synchronous and resets only the in-memory statistics, the same as Crawlee JS. In v1, it also cleared the persisted record. To clear the record too, call the new `reset_store()` once the statistics are exited:

‎src/crawlee/_utils/context.py‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,42 @@
33
import inspect
44
from collections.abc import Callable
55
from functools import wraps
6-
from typing import Any, TypeVar, cast
6+
from typing import TYPE_CHECKING, Any, Protocol, TypeVar, cast
7+
8+
if TYPE_CHECKING:
9+
from types import TracebackType
710

811
T = TypeVar('T', bound=Callable[..., Any])
912

1013

14+
class Collaborator(Protocol):
15+
"""An async context manager that reports through `active` whether it's entered."""
16+
17+
@property
18+
def active(self) -> bool:
19+
"""Whether the object is entered and not exited yet."""
20+
21+
async def __aenter__(self) -> Any:
22+
"""Prepare the object for use.
23+
24+
Raises:
25+
RuntimeError: If the object is already active.
26+
"""
27+
28+
async def __aexit__(
29+
self,
30+
exc_type: type[BaseException] | None,
31+
exc_value: BaseException | None,
32+
exc_traceback: TracebackType | None,
33+
/,
34+
) -> Any:
35+
"""Release what the object holds.
36+
37+
Raises:
38+
RuntimeError: If the object is not active.
39+
"""
40+
41+
1142
def ensure_context(method: T) -> T:
1243
"""Ensure the (async) context manager is initialized before executing the method.
1344

‎src/crawlee/autoscaling/_types.py‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from pydantic import Field
88
from pydantic.dataclasses import dataclass as pydantic_dataclass
99

10+
from crawlee._utils.context import Collaborator
1011
from crawlee._utils.docs import docs_group
1112

1213

@@ -104,12 +105,13 @@ class ConcurrencyConsumer:
104105

105106

106107
@docs_group('Autoscaling')
107-
class ConcurrencySystemInterface(Protocol):
108+
class ConcurrencySystemInterface(Collaborator, Protocol):
108109
"""The contract between an `AutoscaledPool` and the component that governs its concurrency.
109110
110111
The system answers whether there is free compute for one more task and tracks the budget that tasks are booked
111112
against. `ConcurrencySystem` is the default implementation. Anything matching this contract can take its place,
112-
without depending on its internals.
113+
without depending on its internals. An implementation is an async context manager, and `AutoscaledPool.run` refuses
114+
to run on one that isn't entered.
113115
114116
Every allocation method is told which `ConcurrencyConsumer` is asking, so an implementation may allocate per
115117
consumer. `ConcurrencySystem` does not, and serves whoever asks first, which can starve a consumer that joins
@@ -127,10 +129,6 @@ def desired_concurrency(self) -> int:
127129
def current_concurrency(self) -> int:
128130
"""The number of parallel tasks currently booked against this system."""
129131

130-
@property
131-
def active(self) -> bool:
132-
"""Whether the system is ready to be booked against. `AutoscaledPool.run` refuses to run when it is not."""
133-
134132
def has_capacity_for_task(self, consumer: ConcurrencyConsumer) -> bool:
135133
"""Report whether one more task may start right now on behalf of `consumer`.
136134

‎src/crawlee/crawlers/_adaptive_playwright/_adaptive_playwright_crawler.py‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -134,7 +134,8 @@ def __init__(
134134
135135
Args:
136136
rendering_type_predictor: Object that implements RenderingTypePredictor and is capable of predicting which
137-
rendering method should be used. If None, then `DefaultRenderingTypePredictor` is used.
137+
rendering method should be used. If None, then `DefaultRenderingTypePredictor` is used. Keep a
138+
predictor passed in open with `async with` until every crawler using it has finished.
138139
result_checker: Function that evaluates whether crawling result is valid or not.
139140
result_comparator: Function that compares two crawling results and decides whether they are equivalent.
140141
static_parser: Implementation of `AbstractHttpParser`. Parser that will be used for static crawling.
@@ -143,11 +144,10 @@ def __init__(
143144
save_response_cookies: Whether to save the response cookies to the session, in both the static and the
144145
browser sub crawler.
145146
statistics: A custom `Statistics[AdaptivePlaywrightCrawlerStatisticState]` instance, allowing the use of
146-
non-default configuration.
147+
non-default configuration. The crawler enters it on every run, so don't enter it yourself.
147148
kwargs: Additional keyword arguments to pass to the underlying `BasicCrawler`.
148149
"""
149150
# Adaptive crawling related.
150-
self.rendering_type_predictor = rendering_type_predictor or DefaultRenderingTypePredictor()
151151
self.result_checker = result_checker or (lambda _: True)
152152
self.result_comparator = result_comparator or create_default_comparator(result_checker)
153153

@@ -163,6 +163,8 @@ def __init__(
163163

164164
super().__init__(statistics=adaptive_statistics, **kwargs)
165165

166+
self.rendering_type_predictor = self._own_or_borrow(rendering_type_predictor, DefaultRenderingTypePredictor)
167+
166168
# Sub crawlers related.
167169
playwright_crawler_specific_kwargs = playwright_crawler_specific_kwargs or _PlaywrightCrawlerAdditionalOptions()
168170

@@ -228,13 +230,14 @@ async def adaptive_post_navigation_hook_pw(context: PlaywrightPostNavCrawlingCon
228230
static_crawler.post_navigation_hook(adaptive_post_navigation_hook_static)
229231
playwright_crawler.post_navigation_hook(adaptive_post_navigation_hook_pw)
230232

231-
self._additional_context_managers = [
232-
*self._additional_context_managers,
233-
self.rendering_type_predictor,
234-
static_crawler.statistics,
235-
playwright_crawler.statistics,
236-
playwright_crawler._browser_pool, # noqa: SLF001 # Intentional access to private member.
237-
]
233+
# The sub crawlers never run themselves, so this crawler enters and exits their statistics.
234+
self._owned_context_managers.extend([static_crawler.statistics, playwright_crawler.statistics])
235+
236+
# The default pool is the one the browser sub crawler built.
237+
self._own_or_borrow(
238+
playwright_crawler_specific_kwargs.get('browser_pool'),
239+
lambda: playwright_crawler._browser_pool, # noqa: SLF001 # Intentional access to private member.
240+
)
238241

239242
# Sub crawler pipeline related
240243
self._pw_context_pipeline = playwright_crawler._context_pipeline # noqa:SLF001 # Intentional access to private member.

‎src/crawlee/crawlers/_adaptive_playwright/_rendering_type_predictor.py‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,11 @@ def __init__(self) -> None:
6868
# Flag to indicate the state.
6969
self._active = False
7070

71+
@property
72+
def active(self) -> bool:
73+
"""Whether the predictor is initialized and not cleared since."""
74+
return self._active
75+
7176
@abstractmethod
7277
def predict(self, request: Request) -> RenderingTypePrediction:
7378
"""Get `RenderingTypePrediction` based on the input request.
@@ -164,7 +169,11 @@ async def initialize(self) -> None:
164169
await super().initialize()
165170

166171
if not self._state.is_initialized:
167-
await self._state.initialize()
172+
try:
173+
await self._state.initialize()
174+
except BaseException:
175+
await super().clear()
176+
raise
168177

169178
@override
170179
async def clear(self) -> None:

0 commit comments

Comments
 (0)