Introduce MAL/LAL/Hierarchy V2 engine — replace Groovy DSL runtime with ANTLR4 + Javassist

[wu-sheng]

Introduce MAL/LAL/Hierarchy V2 engine — replace Groovy DSL runtime with ANTLR4 + Javassist

• This is a non-trivial feature. Design doc: docs/en/academy/dsl-compiler-design.md

• Documentation updated to include this new feature.

• Tests (UT, IT, E2E) are added to verify the new feature.

• If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #.

• Update the CHANGES log.

---

What this PR does

Introduces the MAL/LAL/Hierarchy V2 engine — replacing the Groovy-based DSL runtime for MAL (Meter Analysis Language), LAL (Log Analysis Language), and Hierarchy matching rules with compile-time ANTLR4 parsing and Javassist bytecode generation — the same approach already used by the OAL V2 engine.

All three DSL compilers follow the same pipeline:

DSL string → ANTLR4 parse → Immutable AST → Javassist bytecode → Direct Java execution

Why

• Remove Groovy runtime dependency (~7 MB) from the OAP server classpath
• Eliminate runtime interpretation — generated bytecode uses direct method calls with zero reflection
• Thread-safe by design — all generated instances are stateless singletons; per-request state passed as parameters (no ThreadLocal, no mutable wrappers)
• Fail-fast at boot — DSL compilation errors are caught during startup with file/line/column reporting, not at first log/metric arrival
• Debugger-friendly — all generated methods include LocalVariableTable (LVT) entries with named variables

Architecture

DSL	Compiled Interface	Runtime Signature	State Passing
MAL	MalExpression	SampleFamily run(Map<String, SampleFamily>)	Parameter
LAL	LalExpression	void execute(FilterSpec, ExecutionContext)	Parameter
Hierarchy	BiFunction<Service, Service, Boolean>	Boolean apply(Service, Service)	Parameter

All v2 classes live under .v2. packages to avoid FQCN conflicts with v1 (Groovy) classes, which remain in test/script-cases/script-runtime-with-groovy/ for comparison testing.

---

File change summary

587 files changed, +64,705 / -1,890 (bulk from new test data files and POM version changes).

Category	File count	Description
Test scripts (MAL/LAL/Hierarchy YAML + data)	174	Production DSL configs + 76 .data.yaml + 10 .input.data companion files
v1-v2 checker tests	125	Cross-version comparison tests in test/script-cases/script-runtime-with-groovy/
Other (POM, CI, git, LICENSE, misc)	108	Version bumps, dependency changes, misc
LAL compiler	52	New v2 compiler, runtime, module provider
MAL compiler	45	New v2 compiler, runtime, module provider
POM files	28	6 new modules, 22 modified
Documentation	15	Design doc, debugging guide, changelog, config vocabulary
Hierarchy compiler	10	New v2 compiler module
Server core	7	HierarchyDefinitionService, HierarchyService, benchmarks
Claude Code skills	7	compile, test, license, gh-pull-request, ci-e2e-debug, run-e2e, generate-classes
Server library	6	GRPCServer/HTTPServer javadoc, ReflectUtil, benchmarks
CLAUDE.md files	4	Module-level AI assistant guides
Server starter	3	SPI wiring for v2 modules
OAL-RT	1	Rename SW_OAL_ENGINE_DEBUG → SW_DYNAMIC_CLASS_ENGINE_DEBUG
CI	1	.github/workflows/skywalking.yaml
.gitignore	1	Add *.generated-classes/ pattern

---

New Maven modules

Module	Path	Purpose
hierarchy	oap-server/analyzer/hierarchy/	Hierarchy rule v2 compiler + SPI provider
script-runtime-with-groovy (parent)	test/script-cases/script-runtime-with-groovy/	Test aggregator
mal-v1-with-groovy	.../script-runtime-with-groovy/mal-v1-with-groovy/	Groovy v1 MAL runtime for comparison
lal-v1-with-groovy	.../script-runtime-with-groovy/lal-v1-with-groovy/	Groovy v1 LAL runtime for comparison
mal-lal-v1-v2-checker	.../script-runtime-with-groovy/mal-lal-v1-v2-checker/	MAL/LAL cross-version comparison tests
hierarchy-v1-v2-checker	.../script-runtime-with-groovy/hierarchy-v1-v2-checker/	Hierarchy cross-version comparison tests
hierarchy-v1-with-groovy	.../script-runtime-with-groovy/hierarchy-v1-with-groovy/	Groovy v1 Hierarchy runtime for comparison

---

ANTLR4 grammar files (new)

Grammar	Path	Lines
MAL Lexer	oap-server/analyzer/meter-analyzer/src/main/antlr4/.../MALLexer.g4	132
MAL Parser	oap-server/analyzer/meter-analyzer/src/main/antlr4/.../MALParser.g4	282
LAL Lexer	oap-server/analyzer/log-analyzer/src/main/antlr4/.../LALLexer.g4	175
LAL Parser	oap-server/analyzer/log-analyzer/src/main/antlr4/.../LALParser.g4	476
Hierarchy Rule Lexer	oap-server/analyzer/hierarchy/src/main/antlr4/.../HierarchyRuleLexer.g4	105
Hierarchy Rule Parser	oap-server/analyzer/hierarchy/src/main/antlr4/.../HierarchyRuleParser.g4	135

---

Key source files (v2 compiler)

MAL compiler (oap-server/analyzer/meter-analyzer/src/main/java/.../v2/)

File	Lines	Responsibility
compiler/MALScriptParser.java	888	ANTLR4 visitor → immutable MALExpressionModel AST
compiler/MALExpressionModel.java	638	Immutable AST data classes (expression, closure, condition)
compiler/MALClassGenerator.java	1,334	Public API, run() method codegen, metadata extraction
compiler/MALClosureCodegen.java	783	Closure method codegen (inlined via LambdaMetafactory)
compiler/MALCodegenHelper.java	318	Static utilities (escapeJava, isBooleanExpression, etc.)
compiler/rt/MalRuntimeHelper.java	76	Runtime helper: regex match, String[][] handling
dsl/SampleFamily.java	829	Core MAL data structure (v2 copy, no Groovy deps)
v2/Analyzer.java	438	MAL expression analyzer (wires compiled code to metrics pipeline)
v2/MetricConvert.java	153	Metric conversion from SampleFamily to OAP metrics

LAL compiler (oap-server/analyzer/log-analyzer/src/main/java/.../v2/)

File	Lines	Responsibility
compiler/LALScriptParser.java	947	ANTLR4 visitor → immutable LALScriptModel AST
compiler/LALScriptModel.java	563	Immutable AST data classes (filter, extractor, sink, condition)
compiler/LALClassGenerator.java	553	Public API, execute() method, class scaffolding
compiler/LALBlockCodegen.java	1,157	Extractor/sink/condition/value-access codegen (all static methods)
compiler/LALCodegenHelper.java	107	Static utilities (escapeJava, boxTypeName, LOG_GETTERS)
compiler/rt/LalRuntimeHelper.java	316	Runtime helper: toStr(), mapVal(), safe-nav, proto field access
dsl/ExecutionContext.java	182	Per-request mutable context (abort flag, parsed map, tags, etc.)
dsl/spec/filter/FilterSpec.java	256	Filter specification (json/yaml/text parsers, abort, sink)
dsl/spec/extractor/ExtractorSpec.java	338	Extractor specification (service, instance, endpoint, tags, etc.)
spi/LALSourceTypeProvider.java	53	SPI for resolving extraLogType proto class at compile time

Hierarchy compiler (oap-server/analyzer/hierarchy/src/main/java/.../v2/compiler/)

File	Lines	Responsibility
HierarchyRuleScriptParser.java	375	ANTLR4 visitor → immutable HierarchyRuleModel AST
HierarchyRuleModel.java	260	Immutable AST data classes (expression, condition, statement)
HierarchyRuleClassGenerator.java	543	Bytecode generation for BiFunction<Service, Service, Boolean>
CompiledHierarchyRuleProvider.java	73	SPI provider: compiles rules and registers as HierarchyRuleProvider

---

Generated .class file output

During compilation, generated .class files can be dumped for debugging.

Environment variable: SW_DYNAMIC_CLASS_ENGINE_DEBUG — set to any non-empty value to enable.

Output directories (per DSL, relative to working directory):

DSL	Output directory
OAL	oal-rt-generated-classes/
MAL	mal-generated-classes/
LAL	lal-generated-classes/
Hierarchy	hierarchy-generated-classes/

During checker tests, generated classes are dumped to:

test/script-cases/scripts/**/*.generated-classes/

These directories are git-ignored via .gitignore.

See docs/en/operation/dynamic-code-generation-debugging.md for full details.

---

v1-v2 cross-version checker tests

Located in test/script-cases/script-runtime-with-groovy/.

Test classes

Test class	Path	Lines	What it tests
MalComparisonTest	.../checker/mal/MalComparisonTest.java	1,107	MAL expression metadata + runtime output comparison
MalFilterComparisonTest	.../checker/mal/MalFilterComparisonTest.java	205	MAL filter closure comparison
MalInputDataGeneratorTest	.../checker/mal/MalInputDataGeneratorTest.java	65	Auto-generate .data.yaml companion files
MalExpectedDataGeneratorTest	.../checker/mal/MalExpectedDataGeneratorTest.java	49	Generate expected output data
LalComparisonTest	.../checker/lal/LalComparisonTest.java	792	LAL script execution comparison
HierarchyRuleComparisonTest	.../core/config/HierarchyRuleComparisonTest.java	191	Hierarchy rule result comparison

Verification counts

DSL	Expressions	Source
MAL expressions	1,229	73 YAML files across 6 directories
MAL filter closures	31	Separate MalFilterComparisonTest
LAL scripts	29	test-lal/ directory
Hierarchy rules	4 rules × test pairs	test-hierarchy-definition.data.yaml
Total	~1,290+

Test data locations

Type	Directory	Count	Format
MAL companion data	test/script-cases/scripts/mal/test-*/	76 files	.data.yaml (SampleFamily mock input)
LAL input data	test/script-cases/scripts/lal/test-lal/	10 files	.input.data (proto-json LogData/extraLog)

---

Unit tests (in v2 compiler modules)

Test class	Path	Lines	Cases
MALScriptParserTest	.../meter-analyzer/.../MALScriptParserTest.java	406	22
MALClassGeneratorTest	.../meter-analyzer/.../MALClassGeneratorTest.java	453	32
DSLV2Test (MAL)	.../meter-analyzer/.../DSLV2Test.java	92	Integration test
LALScriptParserTest	.../log-analyzer/.../LALScriptParserTest.java	529	20
LALClassGeneratorTest	.../log-analyzer/.../LALClassGeneratorTest.java	672	37
LALExpressionExecutionTest	.../log-analyzer/.../LALExpressionExecutionTest.java	585	27
DSLV2Test (LAL)	.../log-analyzer/.../DSLV2Test.java	49	Integration test
HierarchyRuleScriptParserTest	.../hierarchy/.../HierarchyRuleScriptParserTest.java	159	Parser tests
HierarchyRuleClassGeneratorTest	.../hierarchy/.../HierarchyRuleClassGeneratorTest.java	158	Codegen tests

---

JMH benchmarks

Benchmark	Path	Lines
MalBenchmark	.../mal-lal-v1-v2-checker/.../MalBenchmark.java	342
LalBenchmark	.../mal-lal-v1-v2-checker/.../LalBenchmark.java	501
HierarchyBenchmark	.../hierarchy-v1-v2-checker/.../HierarchyBenchmark.java	233

Results:

DSL	Operation	v2 speedup over v1
MAL	execute	~6.8x faster
LAL	compile	~39x faster
LAL	execute	~2.8x faster
Hierarchy	execute	~2.6x faster

---

Compiled code examples

MAL Example — Tag closure (inlined as method on main class)

DSL: metric.tag({tags -> tags.service_name = 'APISIX::' + tags.skywalking_service})

Generated class (single .class file, no separate closure class):

// Closure body compiled as a method on the main class
public Map _tag_apply(Map tags) {
  tags.put("service_name", "APISIX::" + tags.get("skywalking_service"));
  return tags;
}

// _tag field holds a TagFunction instance created via LambdaMetafactory
public SampleFamily run(Map samples) {
  SampleFamily sf;
  sf = ((SampleFamily) samples.getOrDefault("metric", SampleFamily.EMPTY));
  sf = sf.tag(this._tag);
  return sf;
}

Closures are compiled as methods on the main class. At class-load time, LambdaMetafactory wraps each method into a functional interface instance — the same mechanism javac uses for lambda expressions.

MAL Example — Regex match with ternary

DSL: metric.tag({ tags -> def matcher = (tags.metrics_name =~ /\.ssl\.certificate\.([^.]+)\.expiration/); tags.secret_name = matcher ? matcher[0][1] : "unknown" })

Generated _tag_apply() method:

public Map _tag_apply(Map tags) {
  String[][] matcher = MalRuntimeHelper.regexMatch(
      (String) tags.get("metrics_name"),
      "\\.ssl\\.certificate\\.([^.]+)\\.expiration");
  tags.put("secret_name", (((Object)(matcher)) != null ? (matcher[0][1]) : ("unknown")));
  return tags;
}

def type inferred as String[][] from =~ regex match. Ternary compiles to Java ternary with null-check on Object cast.

---

LAL Example — JSON parser with extractor

DSL:

filter {
  json {}
  extractor {
    service parsed.service as String
    instance parsed.instance as String
  }
  sink {}
}

Generated class:

public void execute(FilterSpec filterSpec, ExecutionContext ctx) {
  LalRuntimeHelper h = new LalRuntimeHelper(ctx);
  filterSpec.json(ctx);
  if (!ctx.shouldAbort()) { _extractor(filterSpec.extractor(), h); }
  filterSpec.sink(ctx);
}

private void _extractor(ExtractorSpec _e, LalRuntimeHelper h) {
  _e.service(h.ctx(), h.toStr(h.mapVal("service")));
  _e.instance(h.ctx(), h.toStr(h.mapVal("instance")));
}

Single class, no closures. h.mapVal() accesses JSON parsed map. h.toStr() preserves null (unlike String.valueOf() which returns "null").

LAL Example — Proto-based with extraLogType (Envoy ALS)

DSL:

filter {
  if (parsed?.response?.responseCode?.value as Integer < 400) { abort {} }
  extractor {
    if (parsed?.response?.responseCode) {
      tag 'status.code': parsed?.response?.responseCode?.value
    }
    tag 'response.flag': parsed?.commonProperties?.responseFlags
  }
  sink {}
}

Generated class (with extraLogType = HTTPAccessLogEntry):

public void execute(FilterSpec filterSpec, ExecutionContext ctx) {
  LalRuntimeHelper h = new LalRuntimeHelper(ctx);
  // Cast once, reuse as _p
  HTTPAccessLogEntry _p = (HTTPAccessLogEntry) h.ctx().extraLog();
  // Safe-nav chain cached in local variables
  HTTPResponseProperties _t0 = _p == null ? null : _p.getResponse();
  UInt32Value _t1 = _t0 == null ? null : _t0.getResponseCode();
  if (_t1 != null && _t1.getValue() < 400) { filterSpec.abort(ctx); }
  if (!ctx.shouldAbort()) { _extractor(filterSpec.extractor(), h); }
  filterSpec.sink(ctx);
}

private void _extractor(ExtractorSpec _e, LalRuntimeHelper h) {
  HTTPAccessLogEntry _p = (HTTPAccessLogEntry) h.ctx().extraLog();
  HTTPResponseProperties _t0 = _p == null ? null : _p.getResponse();
  UInt32Value _t1 = _t0 == null ? null : _t0.getResponseCode();
  if (_t1 != null) {
    _e.tag(h.ctx(), "status.code", h.toStr(Integer.valueOf(_t1.getValue())));
  }
  AccessLogCommon _t2 = _p == null ? null : _p.getCommonProperties();
  _e.tag(h.ctx(), "response.flag", h.toStr(_t2 == null ? null : _t2.getResponseFlags()));
}

Proto getter chains resolved via Java reflection at compile time — at runtime it's direct method calls. ?. safe navigation emits == null ? null : ternaries. Intermediate values cached in _tN local variables for readability and dedup.

---

Hierarchy Example — Simple name match

DSL: { (u, l) -> u.name == l.name }

Generated class:

public Object apply(Object arg0, Object arg1) {
  Service u = (Service) arg0;
  Service l = (Service) arg1;
  return Boolean.valueOf(java.util.Objects.equals(u.getName(), l.getName()));
}

Hierarchy Example — Block body with if/return

DSL: { (u, l) -> { if (l.shortName.lastIndexOf('.') > 0) { return u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')); } return false; } }

Generated class:

public Object apply(Object arg0, Object arg1) {
  Service u = (Service) arg0;
  Service l = (Service) arg1;
  if (l.getShortName().lastIndexOf(".") > 0) {
    return Boolean.valueOf(java.util.Objects.equals(
        u.getShortName(),
        l.getShortName().substring(0, l.getShortName().lastIndexOf("."))));
  }
  return Boolean.valueOf(false);
}

Property access → getter methods. == → Objects.equals(). Numeric > → direct operator.

---

SPI registration files

SPI file	Purpose
oap-server/analyzer/hierarchy/.../META-INF/services/...HierarchyRuleProvider	Registers CompiledHierarchyRuleProvider
oap-server/analyzer/log-analyzer/.../META-INF/services/...ModuleDefine	Registers v2 LogAnalyzerModule
oap-server/analyzer/log-analyzer/.../META-INF/services/...ModuleProvider	Registers v2 LogAnalyzerModuleProvider
oap-server/analyzer/log-analyzer/.../test/.../META-INF/services/...LALSourceTypeProvider	Test SPI for mesh proto types
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/.../META-INF/services/...LALSourceTypeProvider	Envoy ALS proto type provider

---

Documentation

Document	Path	Lines	Content
DSL Compiler Design	docs/en/academy/dsl-compiler-design.md	162	Architecture overview of all 4 DSL compilers
Dynamic Code Debugging	docs/en/operation/dynamic-code-generation-debugging.md	187	How to dump .class files, output dirs, env var
Changelog	docs/en/changes/changes.md	+17 lines	Feature summary, benchmark results
Config Vocabulary	docs/en/setup/backend/configuration-vocabulary.md	+1 line	SW_DYNAMIC_CLASS_ENGINE_DEBUG env var
LAL Concept Doc	docs/en/concepts-and-designs/lal.md	Updated	Remove Groovy code block labels
Hierarchy Doc	docs/en/concepts-and-designs/service-hierarchy.md	Updated	Document new expression grammar for matching rules
Hierarchy Config	docs/en/concepts-and-designs/service-hierarchy-configuration.md	Updated	Update rule syntax description
MAL Setup	docs/en/setup/backend/backend-meter.md	Updated	Remove initExp from config
Menu	docs/menu.yml	+10 lines	Add debugging doc to nav

---

Other notable changes included in this branch

• Rename SW_OAL_ENGINE_DEBUG → SW_DYNAMIC_CLASS_ENGINE_DEBUG — unified env var for all 4 DSL compilers
• Remove initExp from MAL config — was internal Groovy startup validation, v2 compiler validates at startup natively
• Remove PowerMock dependency — replaced with ReflectUtil (standard Java reflection + sun.misc.Unsafe for final fields)
• Upgrade Byte Buddy to 1.18.7 — configure explicit -javaagent for Mockito/Byte Buddy in Surefire for JDK 25+ compatibility
• JMH benchmarks relocated — moved from oap-server/microbench/ to target modules for better colocation

[wu-sheng]

Gemini Code Review — V2 Engine Migration (Groovy Replacement)

I have completed a comprehensive review of the changes in the groovy-replace branch. The migration from Groovy-based V1 to the ANTLR4/Javassist-based V2 engine is architecturally sound and significantly improves the robustness of the OAP analyzer.

Key Technical Highlights:

• Concurrency & Thread Safety: The removal of ThreadLocal in favor of explicit context passing (ExecutionContext, RunningContext) is a major improvement. The generated classes (MalExpr, LalExpr) are stateless and thread-safe, ensuring predictable behavior under high concurrency.
• Feature Parity & Validation: Strong parity is verified via MalComparisonTest and LalComparisonTest. The latest fix to fail explicitly on V1 runtime errors during data generation ensures the integrity of the test expectations.
• Performance Optimization: The transition to compiled bytecode (via LALClassGenerator and MALClassGenerator) eliminates the overhead of dynamic script evaluation, providing a more performant execution path for LAL, MAL, and Hierarchy rules.

The migration successfully modernizes the DSL engine while maintaining strict compatibility with existing rule configurations.

[wu-sheng]

Gemini Verification Report: LAL/MAL/Hierarchy V2 Transition

1. Architecture & Engine Parity

The migration from the Groovy-based V1 engine to the ANTLR4 + Javassist-based V2 engine was reviewed for both architectural integrity and functional parity.

• Engine Strictness: Verified that the LalComparisonTest and MalComparisonTest (cross-version checkers) are highly assertive. Any discrepancy in metric values, entity names, or scope types between V1 and V2 triggers an immediate test failure.
• No Silent Passes: Confirmed that the framework handles engine crashes and mismatch states explicitly via fail() calls, ensuring no behavioral regressions.

2. Verification Scope

A systematic, file-by-file review was performed on all 87 script files located in test/script-cases/scripts/, including:

• LAL Scripts: Envoy ALS, K8s network profiling, Nginx, and all Slow SQL variants.
• MAL Meter Config: Java, Go, Python, Ruby agents, Satellite, Spring, etc.
• MAL OTEL Rules: 20+ subdirectories covering 3rd-party integrations (ActiveMQ, AWS, MongoDB, Kafka, etc.).
• Hierarchy: Core service-to-service and service-to-instance mapping rules.

3. Branch & Feature Coverage (100% Verified)

Every script was verified against its corresponding mock data (.input.data or .data.yaml) to ensure all logical paths are exercised:

• Conditional Branches: Verified that every if/else, tagEqual, tagMatch, and tagNotMatch branch in the DSLs has at least one corresponding mock input sample.
• V2 Advanced Features: Confirmed correct execution of V2-specific enhancements:
  • decorate() closures: Validated that MeterEntity property modifications (me.attrX = ...) are correctly applied and asserted.
  • GStrings & Concatenation: Verified complex string building in service and endpoint naming logic.
  • Slashy Regex: Confirmed support for Groovy-style regex literals within split and match operations.

4. Resolution of Previously Identified Gaps

All gaps identified during earlier stages of the review have been successfully addressed:

• Null Endpoint Issue: Mock data for AWS EKS, DynamoDB, and Nginx was enriched with missing labels (e.g., ClusterName, TableName), ensuring valid name construction instead of null defaults.
• Attribute Verification Gap: The testing framework was updated to include attr0-attr5 in the comparison logic, ensuring that decorate() logic is strictly validated.
• LAL Coverage: Improved mesh-dp.yaml coverage from a single success path to 4 exhaustive cases covering virtual process resolution and all component/SSL combinations.

Conclusion

The V2 transition is implementation-complete and thoroughly verified. The test suite provides exhaustive evidence of behavioral equivalence between the old and new engines while maintaining 100% parity with the production script set. The PR is verified as stable and high-quality.