HDDS-15916. Native memory leak in ECKeyOutputStream: encoder.release() not called in close(), causing ISA-L native memory accumulation

[anuragp010]

What changes were proposed in this pull request?

Any long lived client JVM which is writing EC Keys using ECKeyOutputStream leaks native memory. This occurs because ECKeyOutputStream.close() never calls encoder.release(). When NativeRSRawEncoder is in use by ECKeyOutputStream, each key creation results in acquiring native memory to store lookup tables used in calculation of Parity Data. This native memory is only freed by calling release(). This PR adds encoder.release() in ECKeyOutputStream.close() to free the native memory.

A similar native memory leak is present through the decoder in ECBlockReconstructedStripeInputStream. This PR adds a conditional decoder.release() in ECBlockReconstructedStripeInputStream.close() to free the native memory.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15916

How was this patch tested?

1. Unit Test - Ensures encoder.release() is called.
2. Local Docker compose cluster:

Env Setup:

1. Build with -Phadoop-native-lib:
   mvn clean package -Pdist -Phadoop-native-lib -DskipTests -DskipShade -DskipRecon -DskipDocs

2. Update docker-compose.yaml to include isa-l libraries pre starting S3Gateway:

s3g:
    <<: *common-config
    environment:
      OZONE_OPTS:
      <<: *replication
    ports:
      - 9878:9878
      - 19878:19878
    command: [ "bash","-c","sudo dnf install -y epel-release && sudo dnf install -y isa-l && exec
      ozone s3g" ]

3. Use linux/amd64 as the default platform:
   export DOCKER_DEFAULT_PLATFORM=linux/amd64

4. Start the cluster:
   docker compose up -d --scale datanode=5

Running the test:

1. Create an S3 bucket.
2. Create many objects for key put's through the s3 gateway:

mkdir -p /tmp/many && cd /tmp/many          # 20,000 tiny objects = 20,000 EC keys
for i in $(seq 1 20000); do echo x > f$i; done

3. Put the keys through s3g:
   aws s3 $EP cp --recursive --storage-class STANDARD_IA /tmp/many s3://leaktest/r1/

Results:

The RSS (Resident Set Size) for the s3 gateway process was captured after the runs with the fix and without the fix.
Commands used:

1. Trigger manual GC: jcmd $PID GC.run
2. Grep the Process RSS: grep VmRSS /proc/$PID/status

Post-GC RSS floor of the S3 Gateway JVM

Round	Without fix	Δ/round	With fix	Δ/round
baseline	564 MB	—	568 MB	—
1	789 MB	+225	807 MB	+239
2	913 MB	+124	830 MB	+23
3	993 MB	+80	845 MB	+16
4	1105 MB	+112	831 MB	−14
net (r1→r4)		+316 MB		+25 MB, flat

Without the fix, post each run, there is a clear upward trend in RSS of the S3Gateway process.

[jojochuang]

looks good. Maybe we want to do the same for the decoder object in ECBlockReconstructedStripeInputStream?

[sodonnel]

Yea it looks like the decoder in ECBlockReconstructedStripeInputStream has the same issue, so it would be good to fix both together here.

[anuragp010]

Thanks for the review @jojochuang and @sodonnel . I'll update the PR to include a fix for ECBlockReconstructedStripeInputStream .