fix google provider DataprocSubmitTrigger deferred tasks stuck forever#62082
fix google provider DataprocSubmitTrigger deferred tasks stuck forever#62082olegkachur-e wants to merge 1 commit intoapache:mainfrom
Conversation
boring-cyborg
bot
added
area:providers
provider:google
|
Running prek locally and then committing changes will resolve the failing static checks. |
SameerMesiah97
left a comment
SameerMesiah97
left a comment
There was a problem hiding this comment.
Looks fine conceptually. It looks like it is doing a sync call within the triggerer, which is not ideal. But this seems like it was present before.
Please fix CI.
| async def run(self): | ||
| hook = self.get_async_hook() | ||
| # Trigger client cache with sync call get_credentials(), evaluated once. | ||
| await hook.get_job_client(region=self.region) |
There was a problem hiding this comment.
Since the client is now initialized once and reused for the trigger lifetime, do we rely on google-auth’s automatic credential refresh? Just checking that long-running triggers won’t fail due to token expiry.
There was a problem hiding this comment.
Yes, we rely on refresh logic, introduced in BaseGoogle* hooks classes.
- To prevent tasks getting stuck in the deffered state, as a result of sync_hook calls thread stuck on retrieveing credentials. Observed on secrets storage connection retrival.
olegkachur-e
force-pushed
the
dataproc_trigger_fix
branch
from
ff88a30 to
735fb65
Compare
Thank you for the review! Yes, there were sync calls under the hood all the time, this change aim to reduce it's number but not running it on each loop iteration. |
3 participants
Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.