Avro write support - will you accept a patch?

[martin-traverse]

Describe the enhancement requested

Hi,

We need full Avro read / write support in our project and are working on an implementation. I had a look at what already exists in arrow-java, I think it would be fairly straightforward to extend what is there to get full read/write support in the Arrow Java project. Here is what I am proposing:

• A set of producers to handle the Avro data structures, mirroring the existing consumers
• Handle the high level file structure (header, embedded schema and block structure)
• Support for compressed blocks (using the existing codecs in the Avro project)
• High level APIs for read / write, including incremental read (block by block, corresponding to the VSR)

The last point is important for us because we handle streaming data, if we can check a whole block is available before reading it we should be able to prevent avoid on IO calls.

If I draft a PR along these lines, would there be interest to help me refine it and get it into arrow-java? If not we can do our own implementation which will be simpler because we don't need all the features and data types, but I think the delta is not that large and IMO it would be a good thing to have in the Arrow Java toolkit.

Thoughts welcome!

[lidavidm]

On principle I think a write implementation would be welcome. Is there a good way to split up the work so that we aren't reviewing large blocks of code (and you aren't being asked to revise large blocks of code)?

[martin-traverse]

Hm, I could do just the producers and the code to build a producer tree from a schema? That's probably the first / smallest bit that can be self-contained, although still well over half the implementation I would guess! Perhaps a simpler pattern to review and talk about though, then add the higher level pieces on top as a second (and maybe third) step?

[lidavidm]

That would be great, even if it's a lot of code as long as it's logically separate then that works (I assume it's some repetitive code for each type?)

[martin-traverse]

Yes that is exactly how it is ;-)

I've made a start so there should be a draft PR for review in the next few days. I think there might be some interesting questions around the correct representation for some of the types, that we'll be able to discuss just looking at the producers. Once the draft PR is ready I'll share my comments / questions.

[lidavidm]

Sounds good, thank you!

[jbonofre]

It sounds interesting. I think it's a valuable feature. Happy to review as well.

[martin-traverse]

Hello - I have raised a draft PR #638 which lays out the structure of the producers and builds the producer tree for a VSR. I put some points for discussion in the PR. Please do take a look and let me know if it's going in the right direction. If it is I'll add some of the missing pieces and start with the test coverage. Per our discussion above, this PR will focus on the core data producers so we can round-trip the content of a single block. Assuming that is ok, I can look at the container file next and some higher-level functionality.