Launch job from the task queue

[gabor-lbl]

The problem: one of our users needs to launch 100s of jobs at the same time. They don't want to rewrite their workflow. In the NERSC implementation jobs are launched async and waiting for this queue to clear out can time out. Even if we make job submission api for NERSC sync, starting 100-s of jobs will eventually use up all api resources, or the user will hit the api rate limits.

This solution re-uses the existing task queue (used for file system commands) with no updates needed by facility implementations.

The downside is that Job.id is reused to contain a task_id. Maybe it would be better to have an explicit Job.task_id field and make Job.id null-able?

[gabor-lbl]

@cjh1 fyi

[juztas]

My vote would go that everything (FS/Compute) is always async and uses a 202 Accepted return. But that does change the API Semantics (old submit/new submit)... Should we separate this as a new /v2/api/compute//job/{resource_id:str} that is asynchronous?
I would like to not to change anything in graduated v1, meaning not to affect other users by changing semantics/contract. I would see the best to have it under /v2 (as beta), and we can later deprecate /v1 compute. /v2 also allows us to add new stuff quickly for other features.

[gabor-lbl]

Sounds good to me and I like the 202 response idea.

I can start a v2 branch with this change that we could run in parallel with v1. Let me know if you want me to do this.

[gabor-lbl]

I will also move delete and update to the async model.

[juztas]

@gabor-lbl Sound great! Feel free to proceed. I will adapt to your changes for all new stuff that is pending, too!

[gabor-lbl]

Sounds good, I will do this next week (I'm out tomorrow 4/17).