Implement custom protoc plugin to generate OTLP JSON class definitions

[herin049]

Description

This PR implements a custom protoc plugin to automatically generate OTLP JSON class definitions. Consensus was reached deciding that a custom protoc plugin would be the best route for adding support for OTLP JSON while minimizing the number of runtime dependencies (for additional context reference #4886). This PR is a refined version of the first part of a set of changes that are in #4902. Please view the draft PR for an overview of how these packages will be used.

Fixes #1003

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

tox run -e $(tox --listenvs | grep opentelemetry-protojson | tr '\n' ',')

Does This PR Require a Contrib Repo Change?

• Yes. - Link to PR:
• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[mmanciop]

I am very much looking forward to this landing: protobuf is a pretty toxic dependency when automatically injecting Python SDKs with the OpenTelemetry Injector

[pmcollins]

I think it would be valuable to add a test that validates the generated JSON by feeding it to google.protobuf.json_format.Parse. Maybe you could then compare dictionary representations of the original object with that of the resulting proto object for equality.

[herin049]

I think it would be valuable to add a test that validates the generated JSON by feeding it to google.protobuf.json_format.Parse. Maybe you could then compare dictionary representations of the original object with that of the resulting proto object for equality.

Just to make sure I understand your suggestion properly, currently I have an integration test which runs the plugin to generate the ProtoJSON Python classes and tests the serialization/deserialization. Are you suggesting that we also generate the corresponding Protobuf definitions and then perform the JSON serialization using the ProtoJSON classes, load the JSON using the Protobuf definitions and then compare the attributes on the ProtoJSON and Protobuf classes?

[herin049]

@pmcollins Since we mostly care about serialization for the SDK, I added tests to use google.protobuf.json_format.MessageToDict and verify that the contents are identical. Let me know what you think, and if there are another other test cases that might be useful to add.

[pmcollins]

Thanks @herin049. My suggestion is to simulate a Collector receiving JSON. In other words, end-to-end tests that create SDK objects typically sent to an exporter for emission (traces, metrics, and logs), serialize those objects into JSON using the functionality in this PR, send that JSON to our simulated Collector (google.protobuf.json_format.Parse), and make sure that the "Collector" produces data that matches what was fed into the other end of the pipe.

[herin049]

@pmcollins I will be adding the opentelemetry-exporter-otlp-json-common and opentelemetry-exporter-otlp-json-http packages in a follow up PR to limit the size of this PR. The opentelemetry-exporter-otlp-json-common will be analogous to the existing opentelemetry-exporter-otlp-proto-common package and will include the logic to convert SDK objects to the ProtoJSON objects in the generated opentelemetry-proto-json package added in this PR. Do we want to add the roundtrip tests you are describing into the opentelemetry-exporter-otlp-proto-common package, and defer this to the next PR? I am open to including them in this PR as well, just not sure where exactly we would want these tests to live. Feel free to look at the draft PR #4902 for additional context.

[pmcollins]

Makes sense @herin049 -- can be in a subsequent PR.

[herin049]

@aabmass I have rewritten the commit history to separate out the addition of the opentelemetry-codegen-json package an the auto generated opentelemetry-proto-json package with a final commit to integrate these packages into the rest of the repo. Hopefully this makes reviewing a bit easier, let me know if you would prefer that I break down these changes further.

[aabmass]

It's a bit large LGTM. I will look over the generated code too

[aabmass]

Oh, does this runtime module need to be available to the generated code as a dependency? I.e. they need to install opentelemetry-codegen-json?

[herin049]

No. I have added a single json_codec.py file in the runtime module which I manually write into the root of the generated package. Alternatively, I could have opted to create another package for these utility functions, but personally I think creating another package for this is overkill. Let me know your thoughts.

[aabmass]

Would it be reasonable to put this in a base class to try to reduce the code size a bit? Maybe there are downsides, lmk

[herin049]

I've opted to add a @json_serde class decorator

[aabmass]

Since it's not super obvious to me, what's the need of using builtins? To avoid accidental name clashes?

[herin049]

Yes, I used this to prevent name clashes. It might be overkill, but I figured it wouldn't hurt. This is also the convention that the mypy protobuf library used which I took inspiration from.

[MikeGoldsmith]

Looks good to me outside of what @aabmass raised. Found one minor typo 👍🏻