A plea for a declarative api for provider/profile updates

[mroark1m]

Currently it seems to me like provider and profile updating is a painful exercise that requires you to have a lot of knowledge about the state of the system in order to update it. This complicates simple source control of profiles and providers definitions outside of the built-ins and makes automation of creating and updating those things unnecessarily hard.

For profiles, You have to fetch the file out using the tool, eliminate the providers (or ask it to), and update a resource version just to update the profile.

Additionally, you have to precheck if the thing exists before you create it because updating it is a whole separate command. So any automation I want to build for creating this now has to handle a check if/else around creating or updating each object and I have to remember to update the file 'just so' and make sure also that I do things in the correct order because of dependencies.

IMHO, a declarative and idempotent style is preferable. I should be able to declare my provider and profile in a single file and just say 'make it so' and then, assuming things can be applied, that's what I get. If it needs to be created, create it, if it needs to be updated, update it. Handle the dependencies in the order needed. See terraform for reference.

If I can just update a single file in place rather than importing and exporting potentially different and multiple files, then the whole problem with stale files mentioned in the docs goes away, there is no extra copy to get stale. There is just the one file that says declaratively what I want. If I want to version it, I'll version it in a version control system.

Another weird thing here is the lint command rejects profiles if they already exist. How do you lint a file that has a change and is already created? Will you provide also an update-lint command? Linting does not and should not affect the runtime system so why check it at all? No other tool I know of lints against what is in the runtime system, to me this violates the principle of 'least surprise'. At most that should be a warning or an opt-in check or something.

I grant that I have not thought through all the aspects regarding secret rotation or other things that have to be stateful, but those should probably be kept outside the declarative definition.

[johntmyers]

Thanks for the feedback. Would you be interested in putting some more concrete ideas together on a more ergonomic approach?

We haven't gotten a lot of concrete feedback on providers v2 so this is the time to make improvements before we stabilize the APIs for beta and make providers v2 the default.

[mroark1m]

I can give it a shot, I'll see what I can come up with.

[mroark1m]

Well, this is not so concrete, but I'm having some trouble figuring out the use case for the resource_version blocking functionality. It seems to be there to prevent concurrent updates from multiple sources accidentally reverting some change to the config by blocking updates unless the version is incremented, but when would that happen?

I can see that in a shared deployment with centralized administration, one does need to have a single source of truth. In my experience, most companies manage this with a gitops style workflow, which will enable a multi environment (test -> prod) deployment pipeline. You would resolve conflicts before you get to the deployment stage by deploying from a specific branch and you would be forced to handle merges and conflicts there, likely with reviews. Tracking what version is deployed does have a value for ops, I've seen that usually done by tracking the sha of what was deployed in the runtime system or just in the deployment pipeline. I guess you could do the same with the resource_version by just forcing a version update on merge, but it doesn't seem to me like this should block deployment and I consider it kind of an anti-pattern in gitops. A better approach might be to just track the sha as a plain property, which could be handled in the deployment pipeline directly. Also this allows for rollbacks in a normal fashion, if you have to positively increment the version number to do a rollback, that's an ops pain, it would be simpler to just deploy the previous version directly.

In the single user case, it seems like the easiest thing would be to just have a set of config files that you can update to set up these things, and have it all read at gateway startup time (which is what I was suggesting in my first post), there should not be a separate update from someone else to worry about.

There is the ability to alter the config at runtime, which is nice to shorten the cycle during testing. But if it's a single user doing that, do they need to be forced to increment the version to avoid overwriting their own config?

I guess there could be use cases like shared deployment with also shared administration and no source of truth outside the runtime system, but I wonder if that's really a common deployment model. It doesn't seem to allow for multi-environment (test->prod) deployment unless you export and import as your deployment pipeline rather than use a version control system.