Skip to content

feat(go): add filter push-down and predicate API for Go binding#216

Open
luoyuxia wants to merge 5 commits intoapache:mainfrom
luoyuxia:support-fiter-push-down-go
Open

feat(go): add filter push-down and predicate API for Go binding#216
luoyuxia wants to merge 5 commits intoapache:mainfrom
luoyuxia:support-fiter-push-down-go

Conversation

@luoyuxia
Copy link
Copy Markdown
Contributor

@luoyuxia luoyuxia commented Apr 6, 2026

Purpose

Add filter push-down and predicate API support to the Go binding, enabling Go callers to filter table scans with native Go values.

Linked issue: close #214

Brief change log

  • [bindings/c/src/types.rs] Add paimon_predicate, paimon_datum (tagged flat struct with Timestamp/Decimal/Bytes fields), ReadBuilderState.filter, TableScanState
  • [bindings/c/src/result.rs] Add paimon_result_predicate
  • [bindings/c/src/table.rs] Add 10 leaf predicate functions, 3 combinators (and/or/not), with_filter, fix scan to propagate filters via TableScanState; add integer datum auto-coercion across integer-family types; fix empty string datum handling
  • [bindings/go/predicate.go] PredicateBuilder type with short methods (Eq, NotEq, Lt, Le, Gt, Ge, IsNull, IsNotNull, In, NotIn); Predicate.And/Or/Not combinator methods (return errors, no panics); Go value types (Date, Time, Timestamp, LocalZonedTimestamp, Decimal with full i128, Bytes); automatic type inference from Go literals; FFI wrappers
  • [bindings/go/predicate/predicate.go] New sub-package with variadic And/Or convenience functions and Not wrapper
  • [bindings/go/read_builder.go] Add ReadBuilder.WithFilter method (nil-safe, no-op)
  • [bindings/go/table.go] Add Table.PredicateBuilder() factory method
  • [bindings/go/types.go] Add Go mirror types for datum/predicate FFI structs
  • [bindings/go/tests/paimon_test.go] Add TestReadWithFilter with subtests, refactor shared test helpers

Tests

  • TestReadWithFilter/EqualById: filters id = 1, verifies 1 matching row
  • TestReadWithFilter/EmptyStringEqual: verifies Eq("name", "") is accepted and returns 0 rows
  • Refactored existing tests to use shared openTestTable and readRows helpers

API and Format

Go API usage:

import pred "github.com/apache/paimon-rust/bindings/go/predicate"

// Create a predicate builder from a table
pb := table.PredicateBuilder()

// Native Go literals — type auto-inferred
idEq, _ := pb.Eq("id", 1)
nameEq, _ := pb.Eq("name", "alice")

// Dedicated types for temporal/special types
datePred, _ := pb.Eq("dt", paimon.Date(19000))
tsPred, _ := pb.Eq("ts", paimon.Timestamp{Millis: 1700000000000, Nanos: 0})
decPred, _ := pb.Eq("amount", paimon.NewDecimal(12345, 10, 2))

// IN predicate with variadic args
inPred, _ := pb.In("id", 1, 2, 3)

// Combinator methods on Predicate
combined, _ := idEq.And(nameEq)
combined, _ := idEq.Or(nameEq)
negated, _ := idEq.Not()

// Variadic convenience functions (sub-package)
filter, _ := pred.And(pred1, pred2, pred3)
filter, _ := pred.Or(pred1, pred2)
negated, _ := pred.Not(pred1)

// Apply filter
rb.WithFilter(filter)

Documentation

N/A

@luoyuxia luoyuxia marked this pull request as draft April 6, 2026 04:44
@luoyuxia luoyuxia force-pushed the support-fiter-push-down-go branch 9 times, most recently from 3608a37 to 4e3ace4 Compare April 6, 2026 07:28
luoyuxia and others added 2 commits April 6, 2026 15:45
Add complete predicate support to the Go binding, enabling filter
push-down for table scans. This includes:

- C FFI layer: predicate construction functions (equal, not_equal,
  less_than, greater_than, is_null, is_in, etc.), combinators
  (and/or/not), with_filter on ReadBuilder, and fix scan to
  propagate filters via TableScanState
- Go binding: Predicate type, Go-native value types (Date, Time,
  Timestamp, LocalZonedTimestamp, Decimal, Bytes), automatic
  type inference from Go literals (int, string, bool, etc.),
  and ReadBuilder.WithFilter method
- Tests: TestReadWithFilter with shared test helpers

Closes apache#215

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix empty string predicate rejection: allow str_data=null with
  str_len=0 in datum_from_c, matching Bytes branch semantics
- Add integer datum auto-coercion in C FFI layer so Go int literals
  work on BIGINT columns (and vice versa with range checking)
- Change predicate combinators to method style: pred.And(other),
  pred.Or(other), pred.Not() instead of package-level functions
- Expand Decimal to full i128 via Lo/Hi int64 fields with NewDecimal
  convenience constructor
- WithFilter(nil) is now a no-op instead of panicking
- Add regression subtests for empty string predicates and nil filter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@luoyuxia luoyuxia force-pushed the support-fiter-push-down-go branch from 8e0e972 to 09d4d64 Compare April 6, 2026 07:47
luoyuxia and others added 2 commits April 6, 2026 15:52
- Rename And/Or/Not to PredicateAnd/PredicateOr/PredicateNot
- OrPredicate test: assert matching rows are present instead of exact
  count, since filter pushdown prunes at split level not row level

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TableReadState was missing data_predicates, so paimon_table_read_to_arrow
reconstructed TableRead with an empty predicate set, bypassing the
row-level parquet filtering added in apache#208.

- Add data_predicates field to TableReadState
- Add TableRead::data_predicates() accessor
- Pass stored predicates when constructing TableRead in to_arrow
- Restore exact row assertion in OrPredicate test

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@luoyuxia luoyuxia force-pushed the support-fiter-push-down-go branch from e1b69ee to ceff52f Compare April 6, 2026 13:31
Compound predicates (Or/And/Not) are not yet supported at the reader
level, so the OrPredicate test was silently returning all rows. Replace
with a single-leaf EqualById test that exercises filter push-down with
correct results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@luoyuxia luoyuxia force-pushed the support-fiter-push-down-go branch from ceff52f to 2f47899 Compare April 6, 2026 13:44
@luoyuxia luoyuxia marked this pull request as ready for review April 7, 2026 08:08
Copy link
Copy Markdown
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you create a documentation for Go API?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support filter push down in go binding

2 participants