Repository navigation
Query execution time difference between deltatable QueryBuilder and using DataFusion directly. #1140
Description
Activity
For whoever picks this up, it may be worth first investigating if it's due to the table provider or the ffi interface. I would probably try to reproduce in a pure rust ecosystem without ffi. Then we could add in ffi (still in rust) to eliminate any potential other sources of issues.
@timsaucer the QueryBuilder is the same tableprovider but just executed in delta-rs directly and then returning a Lazy RecordBatchReader, which you collect with read_all
I gave this a try with the following:
deltalake 1.5.0 datafusion 52.3.0 pyarrow 20.0.0generated some data with the following script generate_delta_traces.py and run some queries against that delta file, no special datafusion configuration
def run_delta_query_builder(): QueryBuilder().register("tbl", dt).execute(sql).read_all() def run_datafusion(): ctx = SessionContext() ctx.register_table("tbl", dt) ctx.sql(sql).collect() def run_datafusion_single_partition(): config = SessionConfig().with_target_partitions(1) ctx = SessionContext(config) ctx.register_table("tbl", dt) ctx.sql(sql).collect()I run the following query which has 0.1% selectivity
sql = """ SELECT * FROM tbl WHERE ("MlRepoId" = 1089) AND ("TracingProjectId" = '222fde49-1f7a-4752-8ec1-06bcdbf570c5') AND ("TraceId" = '8728990bd3d11fa91a688e9d9964bca1') AND ("SpanId" = '82c0a65e80000450') """the results I got are really similar, there doesn't seem to be a performance regression in the python side
Delta QueryBuilder: avg=0.002449s over 100 runs DataFusion: avg=0.002464s over 100 runs DataFusion (single partition): avg=0.001728s over 100 runsbecause I didn't see any performance issues, I didn't dive into the rust side yet.
@timsaucer @ion-elgreco: do you think there is something else we need to check here? otherwise I think we can close this issue.
Reacted by Ion KoutsourisHi @timsaucer , could you please us closing this issue ?
Describe the bug
What happened:
There are two ways of querying a Delta Table using DataFusion.
As we can see in the above result, I am noticing a massive difference in the query execution time.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I was expecting a near-identical execution time.
Additional context
delta-io/delta-rs#3517 (comment)