[bug] Fix Flink TaskManager Metaspace retention via shared Hadoop ReflectionUtils cache - #9827
Open
zhang-arvin wants to merge 1 commit into
Open
zhang-arvin wants to merge 1 commit into
zhang-arvin wants to merge 1 commit into
Conversation
…pache#9795) Parquet 1.15.x CodecFactory instantiates codecs through Hadoop's ReflectionUtils.CONSTRUCTOR_CACHE, a static ConcurrentHashMap with strong keys living in the shared AppClassLoader. Paimon shades parquet but not Hadoop, so the shaded ZstandardCodec class loaded by each job's ChildFirstClassLoader is pinned there forever (~18 MB Metaspace per finished Flink job, never reclaimed). Parquet 1.16.0 rewrites CodecFactory to construct codecs directly (DirectCodecFactory) and no longer routes through the shared ReflectionUtils cache, so the job classloader is reclaimable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #9795
Root cause
Every finished Flink batch job shipping the Paimon connector as a user jar leaks ~18.2 MB TaskManager Metaspace permanently. The retained object is the job's
ChildFirstClassLoaderclass fororg.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec, pinned by Hadoop'sReflectionUtils.CONSTRUCTOR_CACHE(static, strong keys) in the sharedAppClassLoader.Paimon shades parquet (1.15.2 on release-1.3) but deliberately does not shade Hadoop, so parquet's
CodecFactoryhands the job-classloader-loaded codec class into the shared Hadoop utility that never evicts production-side.Fix
Parquet 1.16.0 rewrites
CodecFactoryto instantiate codecs directly (DirectCodecFactory) instead of going throughReflectionUtils.newInstance, so the strong-keyed shared cache is no longer on the path. The job classloader becomes garbage-collectible after job completion. main/2.1-SNAPSHOT already runs 1.16.0; this backports the version to release-1.3.Verification
CodecFactory: invokesReflectionUtils.newInstance→ shared cache. 1.16.0CodecFactoryconstructor buildsDirectCodecFactorydirectly — no ReflectionUtils reference.Reporter's verification method (from issue thread)
Heap-dump count of user-classloader keys in
CONSTRUCTOR_CACHEshould drop to 0 after upgrade.