From ef8f1e8c0203485c5c14609b0217bc673afb3ec7 Mon Sep 17 00:00:00 2001 From: Gabriel Date: Mon, 13 Jul 2026 22:09:23 +0800 Subject: [PATCH] [fix](be) Fix Parquet timestamp decoding and export defaults ### What problem does this PR solve? Issue Number: None Related PR: #65446 Problem Summary: Timestamp decoding had two correctness problems in scanner v2. Parquet INT96 does not carry a timezone annotation, so applying the Doris SQL session timezone shifted raw wall-clock fields written by Trino/UTC-style writers. ORC nanoseconds were truncated inconsistently across row decoding, statistics, and SARG predicates, which could also produce unsafe pruning at millisecond statistic boundaries. Add an explicit `hive.parquet.time-zone` property for scanner-v2 Parquet INT96 reads. By default scanner v2 preserves raw INT96 wall-clock fields; when configured, it interprets them with the requested unambiguous IANA timezone or valid UTC offset. Reject ambiguous short aliases such as `CST`, share timezone parsing across HMS catalogs and file TVFs, and propagate the setting through generic external-table and TVF interfaces for Hive and Hudi while leaving Iceberg semantics unchanged. Change Parquet export to write INT64 logical timestamps by default and require `enable_int96_timestamps` to be exactly `true` or `false`. Update export regressions to exercise the new default directly while retaining explicit INT96 only for legacy Hive interoperability coverage. Add an end-to-end regression that exports INT96 in one session timezone and reads it through scanner v2 in another session timezone with `hive.parquet.time-zone` configured. Apply half-up nanosecond rounding consistently to ORC rows and statistics, use conservative SARG bounds, and fall back to exact row filtering where negating a widened timestamp envelope would be unsafe. The legacy v1 reader remains unchanged by design. ### Release note Parquet export now writes INT64 logical timestamps by default. Set `enable_int96_timestamps=true` to request INT96. When scanner v2 reads legacy INT96 files that were normalized by a writer timezone, set `hive.parquet.time-zone` to that timezone. The property accepts unambiguous IANA timezone IDs or valid UTC offsets; ambiguous short aliases such as `CST` are rejected. Existing Hive and Hudi catalogs containing normalized INT96 files should be checked during upgrade. ### Check List (For Author) - Test: - Build: `./build.sh --fe` passed on the designated remote host. - Checkstyle: `cd fe && mvn checkstyle:check -pl fe-core` passed on the designated remote host with 0 violations. - Unit Test: 39 targeted FE tests covering scan propagation, HMS/TVF timezone validation, Parquet properties, and export commands passed on the designated remote host. - Unit Test: targeted Parquet and ORC BE unit tests passed on the designated remote host. - Regression test: `test_outfile_parquet` passed on an isolated scanner-v2 cluster on the designated remote host and verified default Parquet export/stream-load round-trip results. - Regression test: `test_hive_read_parquet` and `test_hive_read_parquet_complex_type` passed on the designated remote host. - Regression test: `test_doris_int96_round_trip` passed in generated-output and comparison modes on the designated remote host, verifying that the configured compatibility timezone overrides a different read session timezone. - Regression test: `test_hdfs_parquet_group0` passed completely on an isolated scanner-v2 cluster on the designated remote host. - Regression test: `paimon_timestamp_types` passed completely for JNI and native readers on the isolated cluster on the designated remote host. - Regression test: TeamCity builds 994160 and 994162 produced matching millisecond-precision real results for the updated ARRAY, MAP, and STRUCT expectations. - Regression test: the three S3 outfile suites were attempted on the designated remote host but were blocked before the changed assertions because the configured OSS credential returned `InvalidAccessKeyId`. - Behavior changed: Yes. Scanner v2 preserves raw INT96 wall-clock values by default, Parquet export defaults to INT64 timestamps, timezone aliases are rejected, and ORC timestamp rounding and pruning are consistent and conservative. - Does this need documentation: Yes. https://github.com/apache/doris-website/pull/3982 --- .../core/data_type_serde/data_type_serde.cpp | 30 +- be/src/format_v2/orc/orc_reader.cpp | 20 +- be/src/format_v2/orc/orc_search_argument.cpp | 196 +++++++- be/src/format_v2/parquet/parquet_reader.cpp | 19 +- be/src/format_v2/parquet/parquet_reader.h | 5 +- be/src/format_v2/parquet/parquet_scan.cpp | 4 +- be/src/format_v2/parquet/parquet_scan.h | 2 + .../parquet/reader/column_reader.cpp | 6 +- .../format_v2/parquet/reader/column_reader.h | 4 +- .../parquet/reader/parquet_leaf_reader.cpp | 17 +- .../parquet/reader/parquet_leaf_reader.h | 6 +- .../parquet/reader/scalar_column_reader.cpp | 4 +- .../parquet/reader/scalar_column_reader.h | 11 +- be/src/format_v2/table_reader.cpp | 6 +- be/test/format_v2/orc/orc_reader_test.cpp | 429 +++++++++++++++++- .../parquet/parquet_leaf_reader_test.cpp | 34 +- .../format_v2/parquet/parquet_reader_test.cpp | 136 +++++- .../parquet/parquet_serde_reader_test.cpp | 141 ++++-- .../common/util/FileFormatConstants.java | 1 + .../doris/common/util/FileFormatUtils.java | 31 ++ .../doris/datasource/ExternalTable.java | 5 + .../doris/datasource/FileQueryScanNode.java | 18 + .../datasource/hive/HMSExternalCatalog.java | 18 +- .../datasource/hive/HMSExternalTable.java | 8 + .../ParquetFileFormatProperties.java | 14 +- .../java/org/apache/doris/load/ExportJob.java | 8 + .../trees/plans/commands/ExportCommand.java | 9 +- .../ExternalFileTableValuedFunction.java | 19 + .../FileTableValuedFunction.java | 5 + .../tablefunction/TableValuedFunctionIf.java | 4 + .../datasource/FileQueryScanNodeTest.java | 151 ++++++ ...HMSExternalCatalogParquetTimeZoneTest.java | 88 ++++ .../ParquetFileFormatPropertiesTest.java | 16 +- .../org/apache/doris/load/ExportJobTest.java | 48 ++ .../plans/commands/ExportCommandTest.java | 44 ++ .../ExternalFileTableValuedFunctionTest.java | 30 ++ gensrc/thrift/PlanNodes.thrift | 3 + .../test_outfile_parquet_array_type.out | 8 +- .../test_outfile_parquet_complex_type.out | 40 +- .../parquet/test_outfile_parquet_map_type.out | 16 +- .../parquet/test_doris_int96_round_trip.out | 7 + .../paimon/paimon_timestamp_types.out | 1 - .../test_hdfs_orc_group4_orc_files.out | 2 +- .../test_hdfs_orc_group6_orc_files.out | Bin 11033 -> 11026 bytes .../tvf/test_hdfs_parquet_group1.out | 6 +- .../tvf/test_hdfs_parquet_group2.out | 96 ++-- .../tvf/test_hdfs_parquet_group4.out | Bin 106870 -> 106962 bytes .../tvf/test_hdfs_parquet_group5.out | Bin 613345 -> 613345 bytes .../tvf/test_hdfs_parquet_group6.out | 220 ++++----- .../tvf/test_local_tvf_with_complex_type.out | 50 +- ...local_tvf_with_complex_type_element_at.out | 1 - .../external_table_p0/tvf/test_tvf_p0.out | 10 +- ...st_nested_types_insert_into_with_s3.groovy | 4 + .../parquet/test_outfile_parquet.groovy | 4 +- .../test_doris_int96_round_trip.groovy | 75 +++ .../parquet/test_hive_read_parquet.groovy | 19 +- ...test_hive_read_parquet_complex_type.groovy | 35 +- .../hive/ddl/test_hive_ctas.groovy | 3 +- .../hive/test_complex_types.groovy | 3 +- .../hive/test_external_catalog_hive.groovy | 3 +- ...est_external_catalog_hive_partition.groovy | 3 +- .../hive/test_hive_basic_type.groovy | 3 +- .../hive/test_hive_compress_type.groovy | 3 +- .../test_hive_get_schema_from_table.groovy | 3 +- .../hive/test_hive_page_index.groovy | 3 +- .../hive/test_hive_schema_evolution.groovy | 3 +- .../write/test_hive_text_write_insert.groovy | 3 +- .../hive/write/test_hive_write_insert.groovy | 3 +- .../write/test_hive_write_partitions.groovy | 3 +- .../test_iceberg_insert_overwrite.groovy | 1 + .../write/test_iceberg_write_insert.groovy | 1 + .../paimon/paimon_timestamp_types.groovy | 6 +- .../tvf/test_hdfs_parquet_group0.groovy | 15 +- ...al_tvf_with_complex_type_element_at.groovy | 24 +- 74 files changed, 1902 insertions(+), 365 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HMSExternalCatalogParquetTimeZoneTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/load/ExportJobTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java create mode 100644 regression-test/data/external_table_p0/export/hive_read/parquet/test_doris_int96_round_trip.out create mode 100644 regression-test/suites/external_table_p0/export/hive_read/parquet/test_doris_int96_round_trip.groovy diff --git a/be/src/core/data_type_serde/data_type_serde.cpp b/be/src/core/data_type_serde/data_type_serde.cpp index 16526bc172b213..4dafca97aa3433 100644 --- a/be/src/core/data_type_serde/data_type_serde.cpp +++ b/be/src/core/data_type_serde/data_type_serde.cpp @@ -474,6 +474,24 @@ int64_t find_struct_child_index(const ::orc::Type& type, const std::string& fiel return -1; } +struct RoundedOrcTimestamp { + int64_t seconds; + uint64_t microseconds; +}; + +RoundedOrcTimestamp round_orc_timestamp_to_microseconds(int64_t seconds, int64_t nanoseconds) { + constexpr int64_t NANOS_PER_SECOND = 1000000000; + constexpr int64_t NANOS_PER_MICROSECOND = 1000; + constexpr int64_t MICROS_PER_SECOND = 1000000; + DORIS_CHECK(nanoseconds >= 0 && nanoseconds < NANOS_PER_SECOND); + // Doris stores six fractional digits, so use half-up rounding and carry 999999500ns into the + // next second instead of silently truncating the ORC value. + const auto rounded_microseconds = + (nanoseconds + NANOS_PER_MICROSECOND / 2) / NANOS_PER_MICROSECOND; + return {.seconds = seconds + rounded_microseconds / MICROS_PER_SECOND, + .microseconds = cast_set(rounded_microseconds % MICROS_PER_SECOND)}; +} + Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColumnView& orc_view, const cctz::time_zone& timezone) { const auto* orc_batch = dynamic_cast(orc_view.batch); @@ -493,8 +511,10 @@ Status decode_timestamp_orc_values(IColumn& nested_column, const OrcDecodedColum } auto& value = reinterpret_cast&>(data[old_data_size + row]); - value.from_unixtime(orc_batch->data[source_row], timezone); - value.set_microsecond(cast_set(orc_batch->nanoseconds[source_row] / 1000)); + const auto timestamp = round_orc_timestamp_to_microseconds( + orc_batch->data[source_row], orc_batch->nanoseconds[source_row]); + value.from_unixtime(timestamp.seconds, timezone); + value.set_microsecond(timestamp.microseconds); } return Status::OK(); } @@ -518,8 +538,10 @@ Status decode_timestamp_tz_orc_values(IColumn& nested_column, continue; } auto& value = data[old_data_size + row]; - value.from_unixtime(orc_batch->data[source_row], utc_time_zone); - value.set_microsecond(cast_set(orc_batch->nanoseconds[source_row] / 1000)); + const auto timestamp = round_orc_timestamp_to_microseconds( + orc_batch->data[source_row], orc_batch->nanoseconds[source_row]); + value.from_unixtime(timestamp.seconds, utc_time_zone); + value.set_microsecond(timestamp.microseconds); } return Status::OK(); } diff --git a/be/src/format_v2/orc/orc_reader.cpp b/be/src/format_v2/orc/orc_reader.cpp index 9a2aeb538977dc..88462abbcdd03f 100644 --- a/be/src/format_v2/orc/orc_reader.cpp +++ b/be/src/format_v2/orc/orc_reader.cpp @@ -500,7 +500,15 @@ DateV2Value datetime_v2_from_orc_millis(int64_t millis, int millis_remainder += 1000; } const auto extra_nanos = std::max(nanos_tail, 0); - const auto microseconds = cast_set(millis_remainder * 1000 + extra_nanos / 1000); + constexpr int64_t NANOS_PER_MICROSECOND = 1000; + constexpr int64_t MICROS_PER_SECOND = 1000000; + // Stripe statistics split the timestamp into milliseconds and the remaining nanoseconds. Use + // the same half-up rule as row decoding so zone-map pruning observes identical values. + const auto rounded_extra_microseconds = + (extra_nanos + NANOS_PER_MICROSECOND / 2) / NANOS_PER_MICROSECOND; + const auto microseconds_with_carry = millis_remainder * 1000 + rounded_extra_microseconds; + seconds += microseconds_with_carry / MICROS_PER_SECOND; + const auto microseconds = cast_set(microseconds_with_carry % MICROS_PER_SECOND); DateV2Value value; value.from_unixtime(seconds, timezone); value.set_microsecond(microseconds); @@ -2111,6 +2119,16 @@ Status OrcReader::get_aggregate_result(const format::FileAggregateRequest& reque return Status::NotSupported( "ORC TIMESTAMP min/max statistics are unsafe before writer version ORC-135"); } + if (leaf_type->getKind() == ::orc::TypeKind::TIMESTAMP_INSTANT && + !_enable_mapping_timestamp_tz) { + // Raw timestamp order does not preserve local DATETIMEV2 order across a DST fold. + int32_t fixed_offset_seconds = 0; + if (!TimezoneUtils::try_get_fixed_offset_seconds(_state->timezone_obj, + &fixed_offset_seconds)) { + return Status::NotSupported( + "ORC timestamp min/max pushdown requires a fixed-offset timezone"); + } + } auto& aggregate_column = result->columns[column_idx]; aggregate_column.projection = request_column.projection; diff --git a/be/src/format_v2/orc/orc_search_argument.cpp b/be/src/format_v2/orc/orc_search_argument.cpp index 244109c12ce6e6..5cee9cf46c116c 100644 --- a/be/src/format_v2/orc/orc_search_argument.cpp +++ b/be/src/format_v2/orc/orc_search_argument.cpp @@ -50,6 +50,7 @@ #ifdef BE_TEST #include "util/debug_points.h" #endif +#include "util/timezone_utils.h" namespace doris::format::orc { namespace { @@ -722,6 +723,8 @@ std::optional<::orc::Literal> make_orc_literal(const OrcSargColumn& sarg_column, case ::orc::PredicateDataType::TIMESTAMP: { DORIS_CHECK(sarg_column.orc_type != nullptr); static const cctz::time_zone utc0 = cctz::utc_time_zone(); + // Plain TIMESTAMP statistics use UTC coordinates for wall-clock values. TIMESTAMP_INSTANT + // literals instead represent instants derived from the session-local DATETIMEV2 value. const auto& literal_timezone = sarg_column.orc_type->getKind() == ::orc::TypeKind::TIMESTAMP_INSTANT ? timezone : utc0; @@ -978,6 +981,15 @@ std::optional make_comparison_literal_for_sarg( if (!literal.has_value()) { return std::nullopt; } + if (column.predicate_type == ::orc::PredicateDataType::TIMESTAMP) { + const auto timestamp = literal->getTimestamp(); + // ORC reconstructs negative timestamp statistics with truncating division, which can + // produce a non-canonical nanos component. The exact epoch is unsafe too because its + // half-up lower bound is -500ns. Leave these predicates to Doris row filtering. + if (timestamp.second < 0 || (timestamp.second == 0 && timestamp.nanos == 0)) { + return std::nullopt; + } + } return OrcSargComparisonLiteral { .literal = *literal, .normalized_op = normalized_op, @@ -1052,6 +1064,12 @@ std::optional> make_in_literals_for_sarg( if (!literal.has_value()) { return std::nullopt; } + if (column.predicate_type == ::orc::PredicateDataType::TIMESTAMP) { + const auto timestamp = literal->getTimestamp(); + if (timestamp.second < 0 || (timestamp.second == 0 && timestamp.nanos == 0)) { + return std::nullopt; + } + } literals.push_back(*literal); } if (literals.empty()) { @@ -1177,6 +1195,145 @@ OrcSargNode make_not_node(OrcSargNode child) { }; } +OrcSargNode make_literal_node(OrcSargNodeKind kind, const OrcSargColumn& column, + ::orc::Literal literal) { + std::vector<::orc::Literal> literals; + literals.push_back(std::move(literal)); + return OrcSargNode { + .kind = kind, + .column = column, + .literals = std::move(literals), + .children = {}, + }; +} + +OrcSargNode make_and_node(std::vector children) { + DORIS_CHECK(!children.empty()); + return OrcSargNode { + .kind = OrcSargNodeKind::AND, + .column = std::nullopt, + .literals = {}, + .children = std::move(children), + }; +} + +::orc::Literal shift_timestamp_literal(const ::orc::Literal& literal, int32_t nanos_delta) { + constexpr int64_t NANOS_PER_SECOND = 1000000000; + const auto timestamp = literal.getTimestamp(); + int64_t seconds = timestamp.second; + int64_t nanos = timestamp.nanos + nanos_delta; + if (nanos < 0) { + --seconds; + nanos += NANOS_PER_SECOND; + } else if (nanos >= NANOS_PER_SECOND) { + ++seconds; + nanos -= NANOS_PER_SECOND; + } + return {seconds, cast_set(nanos)}; +} + +::orc::Literal floor_timestamp_literal_to_millis(const ::orc::Literal& literal) { + constexpr int32_t NANOS_PER_MILLISECOND = 1000000; + const auto timestamp = literal.getTimestamp(); + return {timestamp.second, timestamp.nanos / NANOS_PER_MILLISECOND * NANOS_PER_MILLISECOND}; +} + +::orc::Literal ceil_timestamp_literal_to_millis(const ::orc::Literal& literal) { + constexpr int32_t NANOS_PER_MILLISECOND = 1000000; + const auto timestamp = literal.getTimestamp(); + const auto remainder = timestamp.nanos % NANOS_PER_MILLISECOND; + if (remainder == 0) { + return literal; + } + return shift_timestamp_literal(literal, NANOS_PER_MILLISECOND - remainder); +} + +std::pair<::orc::Literal, ::orc::Literal> timestamp_rounding_bounds(const ::orc::Literal& literal) { + constexpr int32_t HALF_MICROSECOND_NANOS = 500; + // All raw ORC values in [lower, upper) round half-up to this Doris microsecond. Round these + // boundaries toward the side that enlarges the SARG match set because ORC statistics retain + // only millisecond precision. + return {shift_timestamp_literal(literal, -HALF_MICROSECOND_NANOS), + shift_timestamp_literal(literal, HALF_MICROSECOND_NANOS)}; +} + +OrcSargNode make_timestamp_equals_node(const OrcSargColumn& column, const ::orc::Literal& literal) { + const auto [lower_bound, upper_bound] = timestamp_rounding_bounds(literal); + std::vector children; + children.push_back(make_not_node(make_literal_node( + OrcSargNodeKind::LESS_THAN, column, floor_timestamp_literal_to_millis(lower_bound)))); + children.push_back(make_literal_node(OrcSargNodeKind::LESS_THAN, column, + ceil_timestamp_literal_to_millis(upper_bound))); + return make_and_node(std::move(children)); +} + +std::optional make_timestamp_comparison_node(const OrcSargComparison& comparison) { + const auto [lower_bound, upper_bound] = timestamp_rounding_bounds(comparison.literal); + switch (comparison.normalized_op) { + case TExprOpcode::GE: + return make_not_node(make_literal_node(OrcSargNodeKind::LESS_THAN, comparison.column, + floor_timestamp_literal_to_millis(lower_bound))); + case TExprOpcode::GT: + return make_not_node(make_literal_node(OrcSargNodeKind::LESS_THAN, comparison.column, + floor_timestamp_literal_to_millis(upper_bound))); + case TExprOpcode::LE: + return make_literal_node(OrcSargNodeKind::LESS_THAN, comparison.column, + ceil_timestamp_literal_to_millis(upper_bound)); + case TExprOpcode::LT: + return make_literal_node(OrcSargNodeKind::LESS_THAN, comparison.column, + ceil_timestamp_literal_to_millis(lower_bound)); + case TExprOpcode::EQ: + return make_timestamp_equals_node(comparison.column, comparison.literal); + case TExprOpcode::NE: + return std::nullopt; + default: + DORIS_CHECK(false) << "Unsupported normalized ORC timestamp SARG comparison op " + << comparison.normalized_op; + __builtin_unreachable(); + } +} + +OrcSargNode make_timestamp_in_node(const OrcSargColumn& column, + const std::vector<::orc::Literal>& literals) { + DORIS_CHECK(!literals.empty()); + if (literals.size() == 1) { + return make_timestamp_equals_node(column, literals.front()); + } + auto min_literal = literals.front(); + auto max_literal = literals.front(); + for (const auto& literal : literals) { + if (literal.getTimestamp() < min_literal.getTimestamp()) { + min_literal = literal; + } + if (max_literal.getTimestamp() < literal.getTimestamp()) { + max_literal = literal; + } + } + const auto lower_bound = + floor_timestamp_literal_to_millis(timestamp_rounding_bounds(min_literal).first); + const auto upper_bound = + ceil_timestamp_literal_to_millis(timestamp_rounding_bounds(max_literal).second); + std::vector children; + children.push_back( + make_not_node(make_literal_node(OrcSargNodeKind::LESS_THAN, column, lower_bound))); + children.push_back(make_literal_node(OrcSargNodeKind::LESS_THAN, column, upper_bound)); + return make_and_node(std::move(children)); +} + +bool can_emit_timestamp_sarg(const OrcSargColumn& column, const cctz::time_zone& timezone) { + if (column.predicate_type != ::orc::PredicateDataType::TIMESTAMP) { + return true; + } + DORIS_CHECK(column.orc_type != nullptr); + if (column.orc_type->getKind() == ::orc::TypeKind::TIMESTAMP) { + return true; + } + // An unmapped TIMESTAMP_INSTANT is decoded to a local DATETIMEV2. A named timezone may map + // that local timestamp to multiple instants during a DST fold. + int32_t offset_seconds = 0; + return TimezoneUtils::try_get_fixed_offset_seconds(timezone, &offset_seconds); +} + OrcSargNode make_comparison_node(OrcSargComparison comparison) { OrcSargNodeKind kind; bool negate = false; @@ -1208,14 +1365,7 @@ OrcSargNode make_comparison_node(OrcSargComparison comparison) { __builtin_unreachable(); } - std::vector<::orc::Literal> literals; - literals.push_back(std::move(comparison.literal)); - OrcSargNode node { - .kind = kind, - .column = comparison.column, - .literals = std::move(literals), - .children = {}, - }; + auto node = make_literal_node(kind, comparison.column, std::move(comparison.literal)); if (negate) { return make_not_node(std::move(node)); } @@ -1308,12 +1458,12 @@ class OrcSargCompiler { case TExprOpcode::LT: case TExprOpcode::EQ: case TExprOpcode::NE: - return compile_comparison(*normalized); + return compile_comparison(*normalized, is_negated); case TExprOpcode::EQ_FOR_NULL: return compile_null_safe_equal(*normalized); case TExprOpcode::FILTER_IN: case TExprOpcode::FILTER_NOT_IN: - return compile_in(*normalized); + return compile_in(*normalized, is_negated); case TExprOpcode::INVALID_OPCODE: return compile_is_null(*normalized); default: @@ -1383,7 +1533,7 @@ class OrcSargCompiler { return make_not_node(std::move(*child)); } - std::optional compile_comparison(const VExprSPtr& expr) const { + std::optional compile_comparison(const VExprSPtr& expr, bool is_negated) const { if (expr->node_type() == TExprNodeType::NULL_AWARE_BINARY_PRED) { return std::nullopt; } @@ -1391,6 +1541,14 @@ class OrcSargCompiler { if (!comparison.has_value()) { return std::nullopt; } + if (comparison->column.predicate_type == ::orc::PredicateDataType::TIMESTAMP) { + // Timestamp comparisons use a conservative millisecond envelope. Negating it turns + // false-positive margin into false negatives, so leave exact negation to Doris. + if (is_negated || !can_emit_timestamp_sarg(comparison->column, _timezone)) { + return std::nullopt; + } + return make_timestamp_comparison_node(*comparison); + } return make_comparison_node(std::move(*comparison)); } @@ -1404,10 +1562,16 @@ class OrcSargCompiler { if (!comparison.has_value()) { return std::nullopt; } + if (comparison->column.predicate_type == ::orc::PredicateDataType::TIMESTAMP) { + if (!can_emit_timestamp_sarg(comparison->column, _timezone)) { + return std::nullopt; + } + return make_timestamp_equals_node(comparison->column, comparison->literal); + } return make_comparison_node(std::move(*comparison)); } - std::optional compile_in(const VExprSPtr& expr) const { + std::optional compile_in(const VExprSPtr& expr, bool is_negated) const { if (expr->node_type() == TExprNodeType::NULL_AWARE_IN_PRED || expr->children().size() < 2) { return std::nullopt; } @@ -1421,6 +1585,14 @@ class OrcSargCompiler { if (!literals.has_value()) { return std::nullopt; } + if (column->predicate_type == ::orc::PredicateDataType::TIMESTAMP) { + // NOT IN negates the conservative equality envelopes and is unsafe for pruning. + if (is_negated || expr->op() == TExprOpcode::FILTER_NOT_IN || + !can_emit_timestamp_sarg(*column, _timezone)) { + return std::nullopt; + } + return make_timestamp_in_node(*column, *literals); + } auto node = make_in_node(*column, std::move(*literals)); if (expr->op() == TExprOpcode::FILTER_NOT_IN) { node = make_not_node(std::move(node)); diff --git a/be/src/format_v2/parquet/parquet_reader.cpp b/be/src/format_v2/parquet/parquet_reader.cpp index 753b3628bfa19b..0bcff2fc7e81d1 100644 --- a/be/src/format_v2/parquet/parquet_reader.cpp +++ b/be/src/format_v2/parquet/parquet_reader.cpp @@ -42,6 +42,7 @@ #include "format_v2/parquet/reader/column_reader.h" #include "io/io_common.h" #include "runtime/runtime_state.h" +#include "util/timezone_utils.h" namespace doris::format::parquet { @@ -52,6 +53,7 @@ struct ParquetReaderScanState { ParquetScanScheduler scheduler; const RuntimeState* runtime_state = nullptr; const cctz::time_zone* timezone = nullptr; + std::optional int96_timezone; bool enable_bloom_filter = false; bool enable_page_cache = false; bool enable_strict_mode = false; @@ -328,10 +330,11 @@ ParquetReader::ParquetReader(std::shared_ptr& system_p std::unique_ptr& file_description, std::shared_ptr io_ctx, RuntimeProfile* profile, std::optional global_rowid_context, - bool enable_mapping_timestamp_tz) + bool enable_mapping_timestamp_tz, std::string hive_parquet_time_zone) : FileReader(system_properties, file_description, io_ctx, profile), _global_rowid_context(global_rowid_context), - _enable_mapping_timestamp_tz(enable_mapping_timestamp_tz) {} + _enable_mapping_timestamp_tz(enable_mapping_timestamp_tz), + _hive_parquet_time_zone(std::move(hive_parquet_time_zone)) {} ParquetReader::~ParquetReader() = default; @@ -350,6 +353,15 @@ Status ParquetReader::init(RuntimeState* state) { state != nullptr && state->query_options().enable_parquet_filter_by_bloom_filter; _state->enable_page_cache = state != nullptr && state->query_options().enable_parquet_file_page_cache; + if (!_hive_parquet_time_zone.empty()) { + cctz::time_zone int96_timezone; + if (!TimezoneUtils::find_cctz_time_zone(_hive_parquet_time_zone, int96_timezone)) { + return Status::InvalidArgument("Invalid hive.parquet.time-zone: {}", + _hive_parquet_time_zone); + } + _state->int96_timezone = int96_timezone; + _state->scheduler.set_int96_timezone(&*_state->int96_timezone); + } if (state != nullptr) { _state->runtime_state = state; _state->timezone = &state->timezone_obj(); @@ -637,7 +649,8 @@ Status ParquetReader::get_aggregate_result(const format::FileAggregateRequest& r row_group, _state->file_context.schema->num_columns(), &row_group_plan.page_skip_plans, _parquet_profile.page_skip_profile(), _state->timezone, _state->enable_strict_mode, - _parquet_profile.scan_profile().column_reader_profile); + _parquet_profile.scan_profile().column_reader_profile, + _state->int96_timezone ? &*_state->int96_timezone : nullptr); std::unique_ptr shape_reader; RETURN_IF_ERROR(column_reader_factory.create_count_shape_reader( root_schema, &count_projection, &shape_reader)); diff --git a/be/src/format_v2/parquet/parquet_reader.h b/be/src/format_v2/parquet/parquet_reader.h index 6c8e88cc27a9b6..24794d80692763 100644 --- a/be/src/format_v2/parquet/parquet_reader.h +++ b/be/src/format_v2/parquet/parquet_reader.h @@ -17,6 +17,7 @@ #include #include +#include #include #include "common/status.h" @@ -46,7 +47,8 @@ class ParquetReader : public format::FileReader { std::unique_ptr& file_description, std::shared_ptr io_ctx, RuntimeProfile* profile, std::optional global_rowid_context = std::nullopt, - bool enable_mapping_timestamp_tz = false); + bool enable_mapping_timestamp_tz = false, + std::string hive_parquet_time_zone = ""); ~ParquetReader() override; Status init(RuntimeState* state) override; @@ -89,6 +91,7 @@ class ParquetReader : public format::FileReader { std::optional _global_rowid_context; // global RowId context size_t _batch_size = ParquetScanScheduler::DEFAULT_READ_BATCH_SIZE; bool _enable_mapping_timestamp_tz = false; // whether UTC timestamps are mapped to TIMESTAMPTZ + std::string _hive_parquet_time_zone; // explicit INT96 timezone; empty disables conversion }; } // namespace doris::format::parquet diff --git a/be/src/format_v2/parquet/parquet_scan.cpp b/be/src/format_v2/parquet/parquet_scan.cpp index 1abddc777aa442..0626faedce17fd 100644 --- a/be/src/format_v2/parquet/parquet_scan.cpp +++ b/be/src/format_v2/parquet/parquet_scan.cpp @@ -714,8 +714,8 @@ Status ParquetScanScheduler::open_next_row_group( ParquetColumnReaderFactory column_reader_factory( _current_row_group, file_context.schema->num_columns(), &row_group_plan.page_skip_plans, - _page_skip_profile, _timezone, _enable_strict_mode, - _scan_profile.column_reader_profile); + _page_skip_profile, _timezone, _enable_strict_mode, _scan_profile.column_reader_profile, + _int96_timezone); for (const auto& col : request.predicate_columns) { const auto local_id = col.local_id(); if (local_id == format::ROW_POSITION_COLUMN_ID) { diff --git a/be/src/format_v2/parquet/parquet_scan.h b/be/src/format_v2/parquet/parquet_scan.h index c656b146cefeaa..adcb8b174abe24 100644 --- a/be/src/format_v2/parquet/parquet_scan.h +++ b/be/src/format_v2/parquet/parquet_scan.h @@ -125,6 +125,7 @@ class ParquetScanScheduler { } void set_condition_cache_context(std::shared_ptr ctx); void set_timezone(const cctz::time_zone* timezone) { _timezone = timezone; } + void set_int96_timezone(const cctz::time_zone* timezone) { _int96_timezone = timezone; } void set_enable_strict_mode(bool enable_strict_mode) { _enable_strict_mode = enable_strict_mode; } @@ -218,6 +219,7 @@ class ParquetScanScheduler { int64_t _merge_read_slice_size = -1; std::optional _global_rowid_context; const cctz::time_zone* _timezone = nullptr; + const cctz::time_zone* _int96_timezone = nullptr; bool _enable_strict_mode = false; int64_t _batch_size = DEFAULT_READ_BATCH_SIZE; std::shared_ptr _condition_cache_ctx; diff --git a/be/src/format_v2/parquet/reader/column_reader.cpp b/be/src/format_v2/parquet/reader/column_reader.cpp index 352fbbd7c3d215..b88c9ebdf55514 100644 --- a/be/src/format_v2/parquet/reader/column_reader.cpp +++ b/be/src/format_v2/parquet/reader/column_reader.cpp @@ -210,13 +210,15 @@ ParquetColumnReaderFactory::ParquetColumnReaderFactory( std::shared_ptr<::parquet::RowGroupReader> row_group, int num_leaf_columns, const std::map* page_skip_plans, ParquetPageSkipProfile page_skip_profile, const cctz::time_zone* timezone, - bool enable_strict_mode, ParquetColumnReaderProfile column_reader_profile) + bool enable_strict_mode, ParquetColumnReaderProfile column_reader_profile, + const cctz::time_zone* int96_timezone) : _row_group(std::move(row_group)), _record_readers(static_cast(num_leaf_columns)), _dictionary_record_readers(static_cast(num_leaf_columns)), _page_skip_plans(page_skip_plans), _page_skip_profile(page_skip_profile), _timezone(timezone), + _int96_timezone(int96_timezone), _enable_strict_mode(enable_strict_mode), _column_reader_profile(column_reader_profile) {} @@ -243,7 +245,7 @@ Status ParquetColumnReaderFactory::make_scalar_column_reader( : nullptr; *reader = std::make_unique(column_schema, std::move(record_reader), page_skip_plan, _timezone, _enable_strict_mode, - _column_reader_profile); + _column_reader_profile, _int96_timezone); return Status::OK(); } diff --git a/be/src/format_v2/parquet/reader/column_reader.h b/be/src/format_v2/parquet/reader/column_reader.h index 51dbd44c11c226..ae56bfff1268c5 100644 --- a/be/src/format_v2/parquet/reader/column_reader.h +++ b/be/src/format_v2/parquet/reader/column_reader.h @@ -156,7 +156,8 @@ class ParquetColumnReaderFactory { ParquetPageSkipProfile page_skip_profile = {}, const cctz::time_zone* timezone = nullptr, bool enable_strict_mode = false, - ParquetColumnReaderProfile column_reader_profile = {}); + ParquetColumnReaderProfile column_reader_profile = {}, + const cctz::time_zone* int96_timezone = nullptr); Status create(const ParquetColumnSchema& column_schema, const format::LocalColumnIndex* projection, @@ -225,6 +226,7 @@ class ParquetColumnReaderFactory { nullptr; // page-index pruning result ParquetPageSkipProfile _page_skip_profile; // page skip profile const cctz::time_zone* _timezone = nullptr; // timezone + const cctz::time_zone* _int96_timezone = nullptr; // explicit timezone for legacy INT96 bool _enable_strict_mode = false; // strict mode ParquetColumnReaderProfile _column_reader_profile; // column reader profile }; diff --git a/be/src/format_v2/parquet/reader/parquet_leaf_reader.cpp b/be/src/format_v2/parquet/reader/parquet_leaf_reader.cpp index fd261ef5219d27..7c217fd7765a0e 100644 --- a/be/src/format_v2/parquet/reader/parquet_leaf_reader.cpp +++ b/be/src/format_v2/parquet/reader/parquet_leaf_reader.cpp @@ -376,7 +376,15 @@ Status ParquetLeafReader::append_values(const ParquetLeafBatch& batch, int64_t r view.decimal_scale = _type_descriptor.decimal_scale; view.fixed_length = _type_descriptor.fixed_length; view.timestamp_is_adjusted_to_utc = _type_descriptor.timestamp_is_adjusted_to_utc; - view.timezone = _timezone; + // INT96 has no timezone annotation, so the writer's convention cannot be inferred from the + // file. Match Trino's explicit hive.parquet.time-zone behavior instead of guessing from + // created_by or using the SQL session timezone: + // - With no catalog property, keep the raw wall clock. For example, raw 2021-01-01 10:11 in a + // Trino/UTC file stays 10:11 instead of becoming 18:11 in an Asia/Shanghai session. + // - With hive.parquet.time-zone=Asia/Shanghai, interpret a legacy Hive-normalized raw 02:11 as + // a UTC epoch and convert it back to the original local wall clock 10:11. + // TIMESTAMPTZ intentionally ignores this pointer in its SerDe and keeps the same UTC instant. + view.timezone = batch._value_kind == DecodedValueKind::INT96 ? _int96_timezone : _timezone; view.enable_strict_mode = _enable_strict_mode; view.null_map = null_map == nullptr || null_map->empty() ? nullptr : null_map->data(); const bool read_dense_for_nullable = batch._read_dense_for_nullable && view.null_map != nullptr; @@ -469,7 +477,8 @@ ParquetLeafReader::ParquetLeafReader( std::shared_ptr<::parquet::internal::RecordReader> record_reader, ParquetColumnReaderProfile profile, const cctz::time_zone* timezone, bool enable_strict_mode, - std::function decoded_value_appender) + std::function decoded_value_appender, + const cctz::time_zone* int96_timezone) : _descriptor(descriptor), _type_descriptor(type_descriptor), _type(std::move(type)), @@ -477,6 +486,7 @@ ParquetLeafReader::ParquetLeafReader( _record_reader(std::move(record_reader)), _profile(profile), _timezone(timezone), + _int96_timezone(int96_timezone), _enable_strict_mode(enable_strict_mode), _decoded_value_appender(std::move(decoded_value_appender)) {} @@ -736,7 +746,8 @@ Status ParquetLeafReader::build_nested_batch_from_leaf_batch( batch->values_column = value_type->create_column(); if (values_written > 0) { ParquetLeafReader value_reader(_descriptor, _type_descriptor, value_type, _name, - _record_reader, _profile, _timezone, _enable_strict_mode); + _record_reader, _profile, _timezone, _enable_strict_mode, + nullptr, _int96_timezone); RETURN_IF_ERROR(value_reader.append_values(leaf_batch, values_written, &value_nulls, batch->values_column)); } diff --git a/be/src/format_v2/parquet/reader/parquet_leaf_reader.h b/be/src/format_v2/parquet/reader/parquet_leaf_reader.h index b396b35fd1f32c..b54b04ada17ad7 100644 --- a/be/src/format_v2/parquet/reader/parquet_leaf_reader.h +++ b/be/src/format_v2/parquet/reader/parquet_leaf_reader.h @@ -103,7 +103,8 @@ class ParquetLeafReader { ParquetColumnReaderProfile profile = {}, const cctz::time_zone* timezone = nullptr, bool enable_strict_mode = false, std::function - decoded_value_appender = nullptr); + decoded_value_appender = nullptr, + const cctz::time_zone* int96_timezone = nullptr); Status read_batch(int64_t batch_rows, ParquetLeafBatch* batch, int64_t* rows_read) const; @@ -166,7 +167,8 @@ class ParquetLeafReader { _record_reader; // Arrow physical column reader (shared ownership) ParquetColumnReaderProfile _profile; // profile counters const cctz::time_zone* _timezone = nullptr; // timezone for timestamp conversion - bool _enable_strict_mode = false; // strict mode for type mismatch errors + const cctz::time_zone* _int96_timezone = nullptr; + bool _enable_strict_mode = false; // strict mode for type mismatch errors std::function _decoded_value_appender; }; diff --git a/be/src/format_v2/parquet/reader/scalar_column_reader.cpp b/be/src/format_v2/parquet/reader/scalar_column_reader.cpp index bdbd9ac8779e05..8ee1236a9e5401 100644 --- a/be/src/format_v2/parquet/reader/scalar_column_reader.cpp +++ b/be/src/format_v2/parquet/reader/scalar_column_reader.cpp @@ -121,13 +121,15 @@ ScalarColumnReader::ScalarColumnReader( const ParquetColumnSchema& column_schema, std::shared_ptr<::parquet::internal::RecordReader> record_reader, const ParquetPageSkipPlan* page_skip_plan, const cctz::time_zone* timezone, - bool enable_strict_mode, ParquetColumnReaderProfile profile) + bool enable_strict_mode, ParquetColumnReaderProfile profile, + const cctz::time_zone* int96_timezone) : ParquetColumnReader(column_schema, column_schema.type, profile), _descriptor(column_schema.descriptor), _type_descriptor(column_schema.type_descriptor), _record_reader(std::move(record_reader)), _page_skip_plan(page_skip_plan), _timezone(timezone), + _int96_timezone(int96_timezone), _enable_strict_mode(enable_strict_mode), _nested_batch(std::make_unique()) {} diff --git a/be/src/format_v2/parquet/reader/scalar_column_reader.h b/be/src/format_v2/parquet/reader/scalar_column_reader.h index 5342baa803eca0..1bac5e8c968769 100644 --- a/be/src/format_v2/parquet/reader/scalar_column_reader.h +++ b/be/src/format_v2/parquet/reader/scalar_column_reader.h @@ -50,7 +50,8 @@ class ScalarColumnReader final : public ParquetColumnReader { std::shared_ptr<::parquet::internal::RecordReader> record_reader, const ParquetPageSkipPlan* page_skip_plan = nullptr, const cctz::time_zone* timezone = nullptr, bool enable_strict_mode = false, - ParquetColumnReaderProfile profile = {}); + ParquetColumnReaderProfile profile = {}, + const cctz::time_zone* int96_timezone = nullptr); ~ScalarColumnReader() override; Status read(int64_t rows, MutableColumnPtr& column, int64_t* rows_read) override; @@ -88,7 +89,8 @@ class ScalarColumnReader final : public ParquetColumnReader { ParquetLeafReader leaf_reader() const { return ParquetLeafReader(_descriptor, _type_descriptor, _type, _name, _record_reader, - _profile, _timezone, _enable_strict_mode); + _profile, _timezone, _enable_strict_mode, nullptr, + _int96_timezone); } void advance_rows_read(int64_t rows); @@ -102,8 +104,9 @@ class ScalarColumnReader final : public ParquetColumnReader { const ParquetPageSkipPlan* _page_skip_plan = nullptr; // page-index pruning result (may be nullptr) const cctz::time_zone* _timezone = nullptr; // timezone - bool _enable_strict_mode = false; // strict mode - int64_t _row_group_rows_read = 0; // rows read in the current row group (cursor) + const cctz::time_zone* _int96_timezone = nullptr; + bool _enable_strict_mode = false; // strict mode + int64_t _row_group_rows_read = 0; // rows read in the current row group (cursor) std::unique_ptr _nested_batch; // intermediate result for nested reads }; diff --git a/be/src/format_v2/table_reader.cpp b/be/src/format_v2/table_reader.cpp index 5f37538c1f2639..e31a6f50bd8c27 100644 --- a/be/src/format_v2/table_reader.cpp +++ b/be/src/format_v2/table_reader.cpp @@ -753,10 +753,14 @@ Status TableReader::create_file_reader(std::unique_ptr* reader) { const bool enable_mapping_timestamp_tz = _scan_params != nullptr && _scan_params->__isset.enable_mapping_timestamp_tz && _scan_params->enable_mapping_timestamp_tz; + const std::string hive_parquet_time_zone = + _scan_params != nullptr && _scan_params->__isset.hive_parquet_time_zone + ? _scan_params->hive_parquet_time_zone + : ""; if (_format == FileFormat::PARQUET) { *reader = std::make_unique( _system_properties, _current_task->data_file, _io_ctx, _scanner_profile, - _global_rowid_context, enable_mapping_timestamp_tz); + _global_rowid_context, enable_mapping_timestamp_tz, hive_parquet_time_zone); return Status::OK(); } if (_format == FileFormat::ORC) { diff --git a/be/test/format_v2/orc/orc_reader_test.cpp b/be/test/format_v2/orc/orc_reader_test.cpp index 6809186f115b59..f572f260bc4659 100644 --- a/be/test/format_v2/orc/orc_reader_test.cpp +++ b/be/test/format_v2/orc/orc_reader_test.cpp @@ -275,6 +275,7 @@ class TableSlotRef : public VSlotRef { constexpr int64_t ROW_COUNT = 5; constexpr int64_t PRIMITIVE_ROW_COUNT = 3; +constexpr int64_t TIMESTAMP_ROUNDING_ROW_COUNT = 4; constexpr int64_t COMPLEX_ROW_COUNT = 3; constexpr int64_t DEEP_NESTED_ROW_COUNT = 4; constexpr int64_t DEEP_NESTED_BATCH_CAPACITY = 16; @@ -2475,13 +2476,20 @@ class NullableGreaterThanExpr final : public VExpr { using ValueType = typename PrimitiveTypeTraits::CppType; NullableGreaterThanExpr(int column_id, DataTypePtr type, const Field& value, - std::string column_name, bool literal_on_left = false) + std::string column_name, bool literal_on_left = false, + TExprOpcode::type comparison_op = TExprOpcode::GT) : VExpr(std::make_shared(), false), _column_id(column_id), _value(value.get()), + _comparison_op(comparison_op), _expr_name("NullableGreaterThanExpr") { _node_type = TExprNodeType::BINARY_PRED; - _opcode = literal_on_left ? TExprOpcode::LT : TExprOpcode::GT; + if (literal_on_left) { + DORIS_CHECK(comparison_op == TExprOpcode::GT); + _opcode = TExprOpcode::LT; + } else { + _opcode = comparison_op; + } auto slot = TableSlotRef::create_shared(column_id, column_id, -1, make_nullable(type), column_name); auto literal = TableLiteral::create_shared(std::move(type), value); @@ -2504,8 +2512,24 @@ class NullableGreaterThanExpr final : public VExpr { result_data.resize(count); for (size_t row = 0; row < count; ++row) { const size_t input_row = selector == nullptr ? row : (*selector)[row]; - result_data[row] = - !nullable_column.is_null_at(input_row) && input.get_element(input_row) > _value; + if (nullable_column.is_null_at(input_row)) { + result_data[row] = false; + continue; + } + const auto input_value = input.get_element(input_row); + switch (_comparison_op) { + case TExprOpcode::GT: + result_data[row] = input_value > _value; + break; + case TExprOpcode::LT: + result_data[row] = input_value < _value; + break; + case TExprOpcode::NE: + result_data[row] = input_value != _value; + break; + default: + DORIS_CHECK(false) << "Unsupported test comparison opcode " << _comparison_op; + } } result_column = std::move(result); return Status::OK(); @@ -2516,6 +2540,7 @@ class NullableGreaterThanExpr final : public VExpr { private: const int _column_id; const ValueType _value; + const TExprOpcode::type _comparison_op; const std::string _expr_name; }; @@ -3068,6 +3093,48 @@ void write_primitive_orc_file(const std::string& file_path) { out.write(memory_stream.getData(), static_cast(memory_stream.getLength())); } +void write_timestamp_rounding_orc_file(const std::string& file_path, + int64_t row_count = TIMESTAMP_ROUNDING_ROW_COUNT, + int64_t first_source_row = 0) { + auto type = std::unique_ptr<::orc::Type>(::orc::Type::buildTypeFromString( + "struct")); + + MemoryOutputStream memory_stream(1024 * 1024); + ::orc::WriterOptions options; + options.setCompression(::orc::CompressionKind_NONE); + options.setMemoryPool(::orc::getDefaultPool()); + // ORC treats GMT as its canonical zero-offset writer timezone. + options.setTimezoneName("GMT"); + auto writer = ::orc::createWriter(*type, &memory_stream, options); + auto batch = writer->createRowBatch(row_count); + auto& struct_batch = dynamic_cast<::orc::StructVectorBatch&>(*batch); + auto& timestamp_batch = dynamic_cast<::orc::TimestampVectorBatch&>(*struct_batch.fields[0]); + auto& timestamp_instant_batch = + dynamic_cast<::orc::TimestampVectorBatch&>(*struct_batch.fields[1]); + + static constexpr std::array seconds {1, 2, 1609459199, + -2}; + static constexpr std::array nanoseconds { + 111001900, 345678499, 999999999, 999998500}; + for (int64_t row = 0; row < row_count; ++row) { + const auto source_row = first_source_row + row; + timestamp_batch.data[row] = seconds[source_row]; + timestamp_batch.nanoseconds[row] = nanoseconds[source_row]; + timestamp_instant_batch.data[row] = seconds[source_row]; + timestamp_instant_batch.nanoseconds[row] = nanoseconds[source_row]; + } + struct_batch.numElements = row_count; + timestamp_batch.numElements = row_count; + timestamp_instant_batch.numElements = row_count; + + writer->add(*batch); + writer->close(); + + std::ofstream out(file_path, std::ios::binary); + out.write(memory_stream.getData(), static_cast(memory_stream.getLength())); +} + void fill_complex_struct_column(::orc::StructVectorBatch& struct_col) { auto& struct_a = dynamic_cast<::orc::LongVectorBatch&>(*struct_col.fields[0]); auto& struct_b = dynamic_cast<::orc::StringVectorBatch&>(*struct_col.fields[1]); @@ -4192,6 +4259,8 @@ void write_multi_stripe_orc_sarg_types_file(const std::string& file_path) { ::orc::WriterOptions options; options.setCompression(::orc::CompressionKind_NONE); options.setMemoryPool(::orc::getDefaultPool()); + // ORC treats GMT as its canonical zero-offset writer timezone. + options.setTimezoneName("GMT"); options.setStripeSize(1); options.setDictionaryKeySizeThreshold(0); auto writer = ::orc::createWriter(*type, &memory_stream, options); @@ -4316,7 +4385,8 @@ void write_two_stripe_constant_date_file(const std::string& file_path, int64_t f void write_two_stripe_constant_timestamp_file(const std::string& file_path, int64_t first_timestamp_second, int64_t second_timestamp_second, - std::string_view writer_timezone = "GMT") { + std::string_view writer_timezone = "GMT", + int64_t nanoseconds = 123000000) { auto type = std::unique_ptr<::orc::Type>( ::orc::Type::buildTypeFromString("struct")); @@ -4339,7 +4409,7 @@ void write_two_stripe_constant_timestamp_file(const std::string& file_path, payloads.reserve(ROWS_PER_STRIPE); for (int64_t row = 0; row < ROWS_PER_STRIPE; ++row) { timestamp_batch.data[row] = timestamp_second; - timestamp_batch.nanoseconds[row] = 123000000; + timestamp_batch.nanoseconds[row] = nanoseconds; payloads.push_back(std::string(2048, static_cast('a' + row % 26))); set_string_value(payload_batch, row, payloads.back()); } @@ -4470,6 +4540,8 @@ format::LocalColumnIndex struct_child_projection(int32_t root_field_id, int32_t class NewOrcReaderTest : public testing::Test { protected: + static void SetUpTestSuite() { TimezoneUtils::load_timezones_to_cache(); } + enum class DirectInPredicateShape { ROOT, AND, @@ -4957,6 +5029,61 @@ TEST_F(NewOrcReaderTest, AggregatePushdownTimestampMinMaxFallsBackForPreOrc135Wr EXPECT_EQ(fallback_values, baseline_values); } +TEST_F(NewOrcReaderTest, AggregatePushdownTimestampMinMaxMatchesNegativeRowDecoding) { + const auto file_path = (_test_dir / "aggregate_negative_timestamp.orc").string(); + // Unlike ORC SARG reconstruction, the aggregate path canonicalizes negative millisecond + // remainders before rounding. Its statistics result must match the decoded row value. + write_timestamp_rounding_orc_file(file_path, 1, 3); + + auto reader = create_reader_for_path(file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); + ASSERT_TRUE(reader->init(&state).ok()); + + auto request = std::make_shared(); + request->non_predicate_columns = {field_projection(0)}; + ASSERT_TRUE(reader->open(request).ok()); + + format::FileAggregateRequest aggregate_request; + aggregate_request.agg_type = TPushAggOp::type::MINMAX; + aggregate_request.columns.push_back( + {.projection = format::LocalColumnIndex::top_level(format::LocalColumnId(0))}); + format::FileAggregateResult aggregate_result; + auto status = reader->get_aggregate_result(aggregate_request, &aggregate_result); + ASSERT_TRUE(status.ok()) << status; + ASSERT_EQ(aggregate_result.columns.size(), 1); + EXPECT_EQ(aggregate_result.count, 1); + EXPECT_TRUE(aggregate_result.columns[0].has_min); + EXPECT_TRUE(aggregate_result.columns[0].has_max); + const auto expected = make_datetime_v2(1969, 12, 31, 23, 59, 58, 999999); + EXPECT_EQ(aggregate_result.columns[0].min_value.get(), expected); + EXPECT_EQ(aggregate_result.columns[0].max_value.get(), expected); +} + +TEST_F(NewOrcReaderTest, AggregatePushdownTimestampMinMaxRejectsNamedTimezone) { + const auto multi_stripe_file_path = (_test_dir / "aggregate_timestamp_dst_fold.orc").string(); + // 2021-11-07 08:30Z and 09:00Z straddle the America/Los_Angeles fall-back transition. + write_multi_stripe_orc_timestamp_instant_sarg_file(multi_stripe_file_path, 1636273800, + 1636275600); + + auto reader = create_reader_for_path(multi_stripe_file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("America/Los_Angeles"); + ASSERT_TRUE(reader->init(&state).ok()); + + auto request = std::make_shared(); + request->non_predicate_columns = {field_projection(0)}; + ASSERT_TRUE(reader->open(request).ok()); + + format::FileAggregateRequest aggregate_request; + aggregate_request.agg_type = TPushAggOp::type::MINMAX; + aggregate_request.columns.push_back( + {.projection = format::LocalColumnIndex::top_level(format::LocalColumnId(0))}); + format::FileAggregateResult aggregate_result; + auto status = reader->get_aggregate_result(aggregate_request, &aggregate_result); + EXPECT_TRUE(status.is()) << status; +} + TEST_F(NewOrcReaderTest, AggregatePushdownTimestampInstantMinMaxUsesTimestampTzWhenMapped) { const auto multi_stripe_file_path = (_test_dir / "aggregate_timestamp_instant_tz.orc").string(); write_multi_stripe_orc_timestamp_instant_sarg_file(multi_stripe_file_path); @@ -9215,6 +9342,47 @@ TEST_F(NewOrcReaderTest, SargTimestampConjunctPrunesStripes) { EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 200); } +TEST_F(NewOrcReaderTest, SargPlainTimestampFixedOffsetUsesUtcWallClockLiteral) { + const auto multi_stripe_file_path = + (_test_dir / "sarg_timestamp_session_timezone.orc").string(); + // Plain TIMESTAMP uses UTC coordinates for wall-clock statistics even when writer and session + // timezones have a nonzero offset. + write_two_stripe_constant_timestamp_file(multi_stripe_file_path, 0, 3600, "Asia/Shanghai"); + ASSERT_EQ(get_orc_stripe_count(multi_stripe_file_path), 2); + + auto reader = create_reader_for_path(multi_stripe_file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+08:00"); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + const auto literal = Field::create_field(make_datetime_v2(1970, 1, 1, 8, 30)); + auto request = std::make_shared(); + request->predicate_columns = {field_projection(0)}; + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared>( + 0, remove_nullable(schema[0].type), literal, "timestamp_col", false, + TExprOpcode::LT))); + ASSERT_TRUE(reader->open(request).ok()); + + bool eof = false; + size_t result_rows = 0; + while (!eof) { + Block block = build_file_block({schema[0]}); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + result_rows += rows; + } + + EXPECT_EQ(result_rows, 200); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 1); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 1); + EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 200); +} + TEST_F(NewOrcReaderTest, SargTimestampInstantConjunctUsesSessionTimezone) { const auto multi_stripe_file_path = (_test_dir / "sarg_timestamp_instant_timezone.orc").string(); @@ -9249,9 +9417,11 @@ TEST_F(NewOrcReaderTest, SargTimestampInstantConjunctUsesSessionTimezone) { } EXPECT_EQ(result_rows, 200); - EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 1); - EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 1); - EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 200); + // Named timezones can map a local timestamp to multiple instants during a DST fold, so Doris + // must leave timestamp stripe pruning disabled and apply the predicate after decoding. + EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 0); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 0); + EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 0); } TEST_F(NewOrcReaderTest, SargTimestampInstantRepeatedCivilTimeDoesNotPruneStripes) { @@ -9525,6 +9695,7 @@ TEST_F(NewOrcReaderTest, SargTimestampInListConjunctPrunesStripes) { auto reader = create_reader_for_path(multi_stripe_file_path); RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); ASSERT_TRUE(reader->init(&state).ok()); std::vector schema; @@ -9556,13 +9727,14 @@ TEST_F(NewOrcReaderTest, SargTimestampInListConjunctPrunesStripes) { EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 200); } -TEST_F(NewOrcReaderTest, SargTimestampNotInListConjunctPrunesStripes) { +TEST_F(NewOrcReaderTest, SargTimestampNotInListConjunctDoesNotUseUnsafeEnvelope) { const auto multi_stripe_file_path = (_test_dir / "sarg_timestamp_not_in.orc").string(); write_two_stripe_constant_timestamp_file(multi_stripe_file_path, 0, 1609459200); ASSERT_EQ(get_orc_stripe_count(multi_stripe_file_path), 2); auto reader = create_reader_for_path(multi_stripe_file_path); RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); ASSERT_TRUE(reader->init(&state).ok()); std::vector schema; @@ -9589,9 +9761,52 @@ TEST_F(NewOrcReaderTest, SargTimestampNotInListConjunctPrunesStripes) { } EXPECT_EQ(result_rows, 200); - EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 1); - EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 1); - EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 200); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 0); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 0); + EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 0); +} + +TEST_F(NewOrcReaderTest, SargTimestampNamedTimezoneFoldDoesNotPruneMatches) { + const auto file_path = (_test_dir / "sarg_timestamp_dst_fold.orc").string(); + write_two_stripe_constant_timestamp_file(file_path, 1636273800, 1636277400, + "America/Los_Angeles"); + ASSERT_EQ(get_orc_stripe_count(file_path), 2); + + auto reader = create_reader_for_path(file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("America/Los_Angeles"); + int32_t offset_seconds = 0; + ASSERT_FALSE( + TimezoneUtils::try_get_fixed_offset_seconds(state.timezone_obj(), &offset_seconds)); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->predicate_columns = {field_projection(0)}; + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared>( + 0, remove_nullable(schema[0].type), + std::vector {Field::create_field( + make_datetime_v2(2021, 11, 7, 1, 30, 0, 123000))}, + "timestamp_col"))); + ASSERT_TRUE(reader->open(request).ok()); + + bool eof = false; + size_t result_rows = 0; + while (!eof) { + Block block = build_file_block({schema[0]}); + size_t rows = 0; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + result_rows += rows; + } + + EXPECT_EQ(result_rows, 400); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 0); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups_by_min_max, 0); + EXPECT_EQ(reader->reader_statistics().filtered_group_rows, 0); } TEST_F(NewOrcReaderTest, SargConjunctPruningPreservesRowPosition) { @@ -9927,6 +10142,194 @@ TEST_F(NewOrcReaderTest, ReadPrimitiveTypesWithNulls) { EXPECT_EQ(schema[15].type->to_string(*block.get_by_position(15).column, NULL_ROW), "NULL"); } +TEST_F(NewOrcReaderTest, ReadTimestampNanosecondsRoundsToMicroseconds) { + const auto file_path = (_test_dir / "timestamp_rounding.orc").string(); + write_timestamp_rounding_orc_file(file_path); + auto reader = create_reader_for_path(file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + Block block = build_file_block(schema); + auto request = std::make_shared(); + request->non_predicate_columns = {field_projection(0), field_projection(1)}; + ASSERT_TRUE(reader->open(request).ok()); + + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + EXPECT_FALSE(eof); + ASSERT_EQ(rows, TIMESTAMP_ROUNDING_ROW_COUNT); + + static constexpr std::array expected { + "1970-01-01 00:00:01.111002", "1970-01-01 00:00:02.345678", + "2021-01-01 00:00:00.000000", "1969-12-31 23:59:58.999999"}; + for (size_t column_id = 0; column_id < schema.size(); ++column_id) { + for (size_t row = 0; row < expected.size(); ++row) { + EXPECT_EQ(schema[column_id].type->to_string(*block.get_by_position(column_id).column, + row), + expected[row]); + } + } +} + +TEST_F(NewOrcReaderTest, TimestampSargMatchesRoundedMicroseconds) { + const auto file_path = (_test_dir / "timestamp_rounding_sarg.orc").string(); + write_timestamp_rounding_orc_file(file_path, 1); + auto reader = create_reader_for_path(file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->predicate_columns = {field_projection(0)}; + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared>( + 0, remove_nullable(schema[0].type), + std::vector {Field::create_field( + make_datetime_v2(1970, 1, 1, 0, 0, 1, 111002))}, + "timestamp_col"))); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block({schema[0]}); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + EXPECT_FALSE(eof); + ASSERT_EQ(rows, 1); + EXPECT_EQ(schema[0].type->to_string(*block.get_by_position(0).column, 0), + "1970-01-01 00:00:01.111002"); +} + +TEST_F(NewOrcReaderTest, TimestampSargMillisecondBoundsNeverPruneMatches) { + auto scan_rows = [&](std::string_view suffix, TExprOpcode::type comparison_op, + uint32_t literal_microseconds, bool not_in = false) { + const auto file_path = (_test_dir / fmt::format("timestamp_sarg_{}.orc", suffix)).string(); + // The first stripe rounds 1.111001900 to 1.111002, while the second rounds the same + // fractional part at second 10. Millisecond statistics must never make the SARG stricter + // than Doris's exact microsecond row comparison. + write_two_stripe_constant_timestamp_file(file_path, 1, 10, "GMT", 111001900); + auto reader = create_reader_for_path(file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); + EXPECT_TRUE(reader->init(&state).ok()); + + std::vector schema; + EXPECT_TRUE(reader->get_schema(&schema).ok()); + EXPECT_EQ(schema.size(), 2); + + const auto literal = Field::create_field( + make_datetime_v2(1970, 1, 1, 0, 0, 1, literal_microseconds)); + auto request = std::make_shared(); + request->predicate_columns = {field_projection(0)}; + if (not_in) { + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared>( + 0, remove_nullable(schema[0].type), std::vector {literal}, + "timestamp_col", true))); + } else { + request->conjuncts.push_back(VExprContext::create_shared( + std::make_shared>( + 0, remove_nullable(schema[0].type), literal, "timestamp_col", false, + comparison_op))); + } + EXPECT_TRUE(reader->open(request).ok()); + + bool eof = false; + size_t result_rows = 0; + while (!eof) { + Block block = build_file_block({schema[0]}); + size_t rows = 0; + EXPECT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + result_rows += rows; + } + return std::pair {result_rows, reader->reader_statistics().filtered_row_groups}; + }; + + EXPECT_EQ(scan_rows("gt", TExprOpcode::GT, 111001).first, 400); + EXPECT_EQ(scan_rows("lt", TExprOpcode::LT, 111003).first, 200); + const auto not_equal = scan_rows("ne", TExprOpcode::NE, 111001); + EXPECT_EQ(not_equal.first, 400); + EXPECT_EQ(not_equal.second, 0); + const auto not_in = scan_rows("not_in", TExprOpcode::INVALID_OPCODE, 111001, true); + EXPECT_EQ(not_in.first, 400); + EXPECT_EQ(not_in.second, 0); +} + +TEST_F(NewOrcReaderTest, NegativeTimestampRoundingBypassesUnsafeSarg) { + const auto file_path = (_test_dir / "negative_timestamp_rounding_sarg.orc").string(); + write_timestamp_rounding_orc_file(file_path, 1, 3); + auto reader = create_reader_for_path(file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->predicate_columns = {field_projection(0)}; + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared>( + 0, remove_nullable(schema[0].type), + std::vector {Field::create_field( + make_datetime_v2(1969, 12, 31, 23, 59, 58, 999999))}, + "timestamp_col"))); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block({schema[0]}); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + EXPECT_FALSE(eof); + ASSERT_EQ(rows, 1); + EXPECT_EQ(schema[0].type->to_string(*block.get_by_position(0).column, 0), + "1969-12-31 23:59:58.999999"); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 0); +} + +TEST_F(NewOrcReaderTest, NegativeTimestampNotInEpochBypassesUnsafeSarg) { + const auto file_path = (_test_dir / "negative_timestamp_not_in_epoch.orc").string(); + write_timestamp_rounding_orc_file(file_path, 1, 3); + auto reader = create_reader_for_path(file_path); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("+00:00"); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 2); + + auto request = std::make_shared(); + request->predicate_columns = {field_projection(0)}; + request->conjuncts.push_back( + VExprContext::create_shared(std::make_shared>( + 0, remove_nullable(schema[0].type), + std::vector { + Field::create_field(make_datetime_v2(1970, 1, 1))}, + "timestamp_col", true))); + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block({schema[0]}); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + EXPECT_FALSE(eof); + ASSERT_EQ(rows, 1); + EXPECT_EQ(schema[0].type->to_string(*block.get_by_position(0).column, 0), + "1969-12-31 23:59:58.999999"); + EXPECT_EQ(reader->reader_statistics().filtered_row_groups, 0); +} + TEST_F(NewOrcReaderTest, ReadHive11DecimalUsesBatchScale) { const auto file_path = find_repo_file( "contrib/apache-orc/java/core/src/test/resources/orc-file-11-format.orc"); diff --git a/be/test/format_v2/parquet/parquet_leaf_reader_test.cpp b/be/test/format_v2/parquet/parquet_leaf_reader_test.cpp index 0d0f9a2f8567cc..38c080b7624410 100644 --- a/be/test/format_v2/parquet/parquet_leaf_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_leaf_reader_test.cpp @@ -22,6 +22,7 @@ #include #include +#include #include #include #include @@ -76,7 +77,8 @@ struct CapturedDecodedView { ParquetLeafReader make_spy_leaf_reader(ParquetTypeDescriptor descriptor, DataTypePtr type, CapturedDecodedView* captured, const cctz::time_zone* timezone = nullptr, - bool enable_strict_mode = false) { + bool enable_strict_mode = false, + const cctz::time_zone* int96_timezone = nullptr) { auto appender = [captured](MutableColumnPtr&, const DecodedColumnView& view) { captured->value_kind = view.value_kind; captured->time_unit = view.time_unit; @@ -117,7 +119,7 @@ ParquetLeafReader make_spy_leaf_reader(ParquetTypeDescriptor descriptor, DataTyp return Status::OK(); }; return ParquetLeafReader(nullptr, descriptor, std::move(type), "leaf", nullptr, {}, timezone, - enable_strict_mode, std::move(appender)); + enable_strict_mode, std::move(appender), int96_timezone); } } // namespace @@ -354,6 +356,34 @@ TEST(ParquetLeafReaderTest, DecodedColumnViewCarriesDescriptorSessionAndNullMapF EXPECT_TRUE(captured.null_map_is_null); } +TEST(ParquetLeafReaderTest, Int96UsesOnlyExplicitHiveParquetTimezone) { + ParquetTypeDescriptor descriptor; + descriptor.physical_type = ::parquet::Type::INT96; + auto type = std::make_shared(); + cctz::time_zone session_timezone; + cctz::time_zone hive_parquet_timezone; + ASSERT_TRUE(cctz::load_time_zone("America/Los_Angeles", &session_timezone)); + ASSERT_TRUE(cctz::load_time_zone("Asia/Shanghai", &hive_parquet_timezone)); + + CapturedDecodedView captured; + auto reader = make_spy_leaf_reader(descriptor, type, &captured, &session_timezone, false, + &hive_parquet_timezone); + const std::array int96_value {}; + ParquetLeafBatch batch; + batch._value_kind = DecodedValueKind::INT96; + batch._fixed_values = int96_value.data(); + batch._values_written = 1; + auto column = type->create_column(); + + ASSERT_TRUE(reader.append_values(batch, 1, nullptr, column).ok()); + EXPECT_EQ(captured.timezone, &hive_parquet_timezone); + + captured.timezone = &session_timezone; + auto default_reader = make_spy_leaf_reader(descriptor, type, &captured, &session_timezone); + ASSERT_TRUE(default_reader.append_values(batch, 1, nullptr, column).ok()); + EXPECT_EQ(captured.timezone, nullptr); +} + TEST(ParquetLeafReaderTest, DecodedColumnViewCapturesBinaryFixedLengthAndFloat16Override) { ParquetTypeDescriptor binary_descriptor; binary_descriptor.physical_type = ::parquet::Type::FIXED_LEN_BYTE_ARRAY; diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp b/be/test/format_v2/parquet/parquet_reader_test.cpp index c5bf251c1553e2..67ab90a5cbc69d 100644 --- a/be/test/format_v2/parquet/parquet_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_reader_test.cpp @@ -37,10 +37,12 @@ #include "core/assert_cast.h" #include "core/block/block.h" +#include "core/column/column_array.h" #include "core/column/column_nullable.h" #include "core/column/column_string.h" #include "core/column/column_vector.h" #include "core/data_type/data_type_array.h" +#include "core/data_type/data_type_date_or_datetime_v2.h" #include "core/data_type/data_type_map.h" #include "core/data_type/data_type_nullable.h" #include "core/data_type/data_type_number.h" @@ -66,6 +68,7 @@ #include "storage/index/zone_map/zonemap_filter_result.h" #include "storage/segment/condition_cache.h" #include "storage/utils.h" +#include "util/timezone_utils.h" namespace doris { namespace { @@ -728,6 +731,37 @@ void write_int96_timestamp_parquet_file(const std::string& file_path) { arrow_builder.build())); } +void write_nested_int96_timestamp_parquet_file(const std::string& file_path) { + auto timestamp_type = arrow::timestamp(arrow::TimeUnit::MICRO); + auto value_builder = + std::make_shared(timestamp_type, arrow::default_memory_pool()); + arrow::ListBuilder list_builder(arrow::default_memory_pool(), value_builder, + arrow::list(arrow::field("element", timestamp_type, true))); + + ASSERT_TRUE(list_builder.Append().ok()); + ASSERT_TRUE(value_builder->Append(1735660800000000LL).ok()); + ASSERT_TRUE(value_builder->Append(1735660800123456LL).ok()); + ASSERT_TRUE(list_builder.Append().ok()); + ASSERT_TRUE(value_builder->Append(1735689600000000LL).ok()); + std::shared_ptr array; + ASSERT_TRUE(list_builder.Finish(&array).ok()); + auto table = arrow::Table::Make( + arrow::schema({arrow::field("timestamps", array->type(), true)}), {array}); + + auto file_result = arrow::io::FileOutputStream::Open(file_path); + ASSERT_TRUE(file_result.ok()) << file_result.status(); + std::shared_ptr out = *file_result; + ::parquet::WriterProperties::Builder writer_builder; + writer_builder.version(::parquet::ParquetVersion::PARQUET_2_6); + writer_builder.data_page_version(::parquet::ParquetDataPageVersion::V2); + writer_builder.compression(::parquet::Compression::UNCOMPRESSED); + ::parquet::ArrowWriterProperties::Builder arrow_builder; + arrow_builder.enable_force_write_int96_timestamps(); + PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(*table, arrow::default_memory_pool(), out, + ROW_COUNT, writer_builder.build(), + arrow_builder.build())); +} + void write_int_pair_parquet_file(const std::string& file_path, int64_t row_group_size = ROW_COUNT) { auto schema = arrow::schema({ arrow::field("id", arrow::int32(), false), @@ -1136,7 +1170,7 @@ class NewParquetReaderTest : public testing::Test { RuntimeProfile* profile = nullptr, bool enable_mapping_timestamp_tz = false, std::shared_ptr io_ctx = nullptr, std::optional global_rowid_context = std::nullopt, - bool is_immutable = false) const { + bool is_immutable = false, std::string hive_parquet_time_zone = "") const { auto system_properties = std::make_shared(); system_properties->system_type = TFileType::FILE_LOCAL; auto file_description = std::make_unique(); @@ -1147,7 +1181,8 @@ class NewParquetReaderTest : public testing::Test { file_description->is_immutable = is_immutable; return std::make_unique( system_properties, file_description, std::move(io_ctx), profile, - global_rowid_context, enable_mapping_timestamp_tz); + global_rowid_context, enable_mapping_timestamp_tz, + std::move(hive_parquet_time_zone)); } std::filesystem::path _test_dir; @@ -1312,6 +1347,103 @@ TEST_F(NewParquetReaderTest, GetSchemaMapsInt96ToTimestampTzWhenTimestampTzMappi EXPECT_EQ(remove_nullable(schema[0].type)->get_scale(), 6); } +TEST_F(NewParquetReaderTest, Int96TimezoneUsesCatalogPropertyInsteadOfSessionTimezone) { + TimezoneUtils::load_timezones_to_cache(); + write_int96_timestamp_parquet_file(_file_path); + + auto read_first_value = [&](const std::string& hive_parquet_time_zone, + bool enable_mapping_timestamp_tz = false) { + auto reader = create_reader(0, -1, nullptr, enable_mapping_timestamp_tz, nullptr, + std::nullopt, false, hive_parquet_time_zone); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + // A session timezone must never implicitly opt an INT96 column into legacy conversion. + state.set_timezone("America/Los_Angeles"); + auto status = reader->init(&state); + if (!status.ok()) { + ADD_FAILURE() << status; + return std::string {}; + } + std::vector schema; + status = reader->get_schema(&schema); + if (!status.ok()) { + ADD_FAILURE() << status; + return std::string {}; + } + auto request = std::make_shared(); + request->non_predicate_columns = {field_projection(0)}; + status = reader->open(request); + if (!status.ok()) { + ADD_FAILURE() << status; + return std::string {}; + } + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + status = reader->get_block(&block, &rows, &eof); + if (!status.ok()) { + ADD_FAILURE() << status; + return std::string {}; + } + if (rows != 3) { + ADD_FAILURE() << "Expected 3 rows, got " << rows; + return std::string {}; + } + return block.get_by_position(0).type->to_string(*block.get_by_position(0).column, 0); + }; + + // A Trino/UTC-style file stores the wall clock 2024-12-31 16:00 directly. With the property + // absent, Doris preserves it even though the SQL session is America/Los_Angeles. + EXPECT_EQ(read_first_value(""), "2024-12-31 16:00:00.000000"); + // A legacy Hive writer configured for Asia/Shanghai can normalize local 2025-01-01 00:00 to + // raw INT96 2024-12-31 16:00. The matching catalog property reverses that normalization. + EXPECT_EQ(read_first_value("Asia/Shanghai"), "2025-01-01 00:00:00.000000"); + // TIMESTAMPTZ preserves the instant even when the INT96 compatibility timezone is configured. + EXPECT_EQ(read_first_value("Asia/Shanghai", true), "2024-12-31 16:00:00.000000+00:00"); +} + +TEST_F(NewParquetReaderTest, NestedInt96UsesHiveParquetTimezone) { + TimezoneUtils::load_timezones_to_cache(); + write_nested_int96_timestamp_parquet_file(_file_path); + auto reader = + create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, "Asia/Shanghai"); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + state.set_timezone("America/Los_Angeles"); + ASSERT_TRUE(reader->init(&state).ok()); + + std::vector schema; + ASSERT_TRUE(reader->get_schema(&schema).ok()); + ASSERT_EQ(schema.size(), 1); + auto request = std::make_shared(); + request->non_predicate_columns = {field_projection(0)}; + ASSERT_TRUE(reader->open(request).ok()); + + Block block = build_file_block(schema); + size_t rows = 0; + bool eof = false; + ASSERT_TRUE(reader->get_block(&block, &rows, &eof).ok()); + ASSERT_EQ(rows, 2); + const auto& arrays = nullable_nested_column(block, 0); + ASSERT_EQ(arrays.get_offsets(), ColumnArray::Offsets64({2, 3})); + const auto& nullable_elements = assert_cast(arrays.get_data()); + const auto& timestamps = + assert_cast(nullable_elements.get_nested_column()); + const auto* array_type = + assert_cast(remove_nullable(schema[0].type).get()); + const auto element_type = remove_nullable(array_type->get_nested_type()); + EXPECT_EQ(element_type->to_string(timestamps, 0), "2025-01-01 00:00:00.000000"); + EXPECT_EQ(element_type->to_string(timestamps, 1), "2025-01-01 00:00:00.123456"); + EXPECT_EQ(element_type->to_string(timestamps, 2), "2025-01-01 08:00:00.000000"); +} + +TEST_F(NewParquetReaderTest, RejectsInvalidHiveParquetTimezone) { + auto reader = + create_reader(0, -1, nullptr, false, nullptr, std::nullopt, false, "Not/A_Timezone"); + RuntimeState state {TQueryOptions(), TQueryGlobals()}; + auto status = reader->init(&state); + EXPECT_FALSE(status.ok()); + EXPECT_NE(status.to_string().find("Invalid hive.parquet.time-zone"), std::string::npos); +} + TEST_F(NewParquetReaderTest, ReadSingleRowGroupThenEof) { auto reader = create_reader(); RuntimeState state {TQueryOptions(), TQueryGlobals()}; diff --git a/be/test/format_v2/parquet/parquet_serde_reader_test.cpp b/be/test/format_v2/parquet/parquet_serde_reader_test.cpp index c35138e3263723..96b7fbeed0dade 100644 --- a/be/test/format_v2/parquet/parquet_serde_reader_test.cpp +++ b/be/test/format_v2/parquet/parquet_serde_reader_test.cpp @@ -17,10 +17,12 @@ #include #include +#include #include #include #include +#include #include #include #include @@ -37,6 +39,7 @@ #include "core/column/column_vector.h" #include "core/data_type/data_type.h" #include "core/data_type/data_type_nullable.h" +#include "core/data_type/data_type_timestamptz.h" #include "core/types.h" #include "format_v2/parquet/parquet_column_schema.h" #include "format_v2/parquet/reader/column_reader.h" @@ -146,13 +149,17 @@ class ParquetSerdeReaderTest : public testing::Test { } void write_table(const std::string& file_path, const std::shared_ptr& table, - std::shared_ptr<::parquet::ArrowWriterProperties> arrow_properties = nullptr) { + std::shared_ptr<::parquet::ArrowWriterProperties> arrow_properties = nullptr, + bool enable_dictionary = true) { auto file_result = arrow::io::FileOutputStream::Open(file_path); ASSERT_TRUE(file_result.ok()) << file_result.status(); ::parquet::WriterProperties::Builder writer_builder; writer_builder.version(::parquet::ParquetVersion::PARQUET_2_6); writer_builder.data_page_version(::parquet::ParquetDataPageVersion::V2); writer_builder.compression(::parquet::Compression::UNCOMPRESSED); + if (!enable_dictionary) { + writer_builder.disable_dictionary(); + } if (arrow_properties == nullptr) { ::parquet::ArrowWriterProperties::Builder arrow_builder; arrow_properties = arrow_builder.build(); @@ -238,14 +245,42 @@ class ParquetSerdeReaderTest : public testing::Test { return _fields.size(); } - std::unique_ptr create_reader(size_t field_idx) const { - ParquetColumnReaderFactory factory(_row_group, _file_reader->metadata()->num_columns()); + std::unique_ptr create_reader( + size_t field_idx, const cctz::time_zone* int96_timezone = nullptr) const { + ParquetColumnReaderFactory factory(_row_group, _file_reader->metadata()->num_columns(), + nullptr, {}, nullptr, false, {}, int96_timezone); std::unique_ptr reader; auto st = factory.create(*_fields[field_idx], &reader); EXPECT_TRUE(st.ok()) << st; return reader; } + void write_int96_timestamp_file(const std::string& file_path, bool enable_dictionary) { + auto field = arrow::field("col_datetime", arrow::timestamp(arrow::TimeUnit::MICRO), false); + auto array = build_timestamp_array( + arrow::timestamp(arrow::TimeUnit::MICRO), + {0, 1234567, 1609459200000000, 1615714200000000, 1615717800000000}); + auto table = arrow::Table::Make(arrow::schema({field}), {array}); + + ::parquet::ArrowWriterProperties::Builder arrow_builder; + arrow_builder.enable_force_write_int96_timestamps(); + write_table(file_path, table, arrow_builder.build(), enable_dictionary); + } + + void expect_int96_timestamp_strings(const cctz::time_zone* int96_timezone, + const std::vector& expected) const { + ASSERT_EQ(expected.size(), ROW_COUNT); + auto reader = create_reader(0, int96_timezone); + ASSERT_NE(reader, nullptr); + auto column = _fields[0]->type->create_column(); + int64_t rows_read = 0; + ASSERT_TRUE(reader->read(ROW_COUNT, column, &rows_read).ok()); + ASSERT_EQ(rows_read, ROW_COUNT); + for (int64_t row = 0; row < ROW_COUNT; ++row) { + EXPECT_EQ(_fields[0]->type->to_string(*column, row), expected[row]); + } + } + template void read_and_validate(const std::string& name, Validator validator) const { const auto field_idx = find_field_idx(name); @@ -422,37 +457,75 @@ TEST_F(ParquetSerdeReaderTest, ReadAllSupportedPhysicalAndLogicalTypes) { }); } -TEST_F(ParquetSerdeReaderTest, ReadInt96TimestampAsDateTimeV2) { - const auto file_path = (_test_dir / "int96_timestamp.parquet").string(); - auto field = arrow::field("col_datetime", arrow::timestamp(arrow::TimeUnit::MICRO), false); - auto array = build_timestamp_array(arrow::timestamp(arrow::TimeUnit::MICRO), - {0, 1234567, 1609459200000000, 1609459201000000, -1}); - auto table = arrow::Table::Make(arrow::schema({field}), {array}); - - ::parquet::ArrowWriterProperties::Builder arrow_builder; - arrow_builder.enable_force_write_int96_timestamps(); - _fields.clear(); - _file_reader.reset(); - _row_group.reset(); - write_table(file_path, table, arrow_builder.build()); - open_file(file_path); - - ASSERT_EQ(_fields.size(), 1); - EXPECT_EQ(_fields[0]->type_descriptor.physical_type, ::parquet::Type::INT96); - EXPECT_EQ(_fields[0]->type_descriptor.extra_type_info, ParquetExtraTypeInfo::IMPALA_TIMESTAMP); - ASSERT_TRUE(supports_record_reader(_fields[0]->type_descriptor)); - ASSERT_EQ(remove_nullable(_fields[0]->type)->get_primitive_type(), TYPE_DATETIMEV2); - - auto reader = create_reader(0); - ASSERT_NE(reader, nullptr); - auto column = _fields[0]->type->create_column(); - int64_t rows_read = 0; - ASSERT_TRUE(reader->read(ROW_COUNT, column, &rows_read).ok()); - ASSERT_EQ(rows_read, ROW_COUNT); - EXPECT_EQ(_fields[0]->type->to_string(*column, 0), "1970-01-01 00:00:00.000000"); - EXPECT_EQ(_fields[0]->type->to_string(*column, 1), "1970-01-01 00:00:01.234567"); - EXPECT_EQ(_fields[0]->type->to_string(*column, 2), "2021-01-01 00:00:00.000000"); - EXPECT_EQ(_fields[0]->type->to_string(*column, 4), "1969-12-31 23:59:59.999999"); +TEST_F(ParquetSerdeReaderTest, ReadInt96TimestampUsesConfiguredParquetTimezoneLikeTrino) { + cctz::time_zone shanghai; + cctz::time_zone los_angeles; + ASSERT_TRUE(cctz::load_time_zone("Asia/Shanghai", &shanghai)); + ASSERT_TRUE(cctz::load_time_zone("America/Los_Angeles", &los_angeles)); + + for (const bool enable_dictionary : {false, true}) { + const auto file_path = + (_test_dir / (enable_dictionary ? "int96_dict.parquet" : "int96_plain.parquet")) + .string(); + write_int96_timestamp_file(file_path, enable_dictionary); + _fields.clear(); + _file_reader.reset(); + _row_group.reset(); + open_file(file_path); + + ASSERT_EQ(_fields.size(), 1); + EXPECT_EQ(_fields[0]->type_descriptor.physical_type, ::parquet::Type::INT96); + EXPECT_EQ(_fields[0]->type_descriptor.extra_type_info, + ParquetExtraTypeInfo::IMPALA_TIMESTAMP); + ASSERT_TRUE(supports_record_reader(_fields[0]->type_descriptor)); + ASSERT_EQ(remove_nullable(_fields[0]->type)->get_primitive_type(), TYPE_DATETIMEV2); + const auto encodings = _file_reader->metadata()->RowGroup(0)->ColumnChunk(0)->encodings(); + const bool has_dictionary = + std::find(encodings.begin(), encodings.end(), + ::parquet::Encoding::RLE_DICTIONARY) != encodings.end() || + std::find(encodings.begin(), encodings.end(), + ::parquet::Encoding::PLAIN_DICTIONARY) != encodings.end(); + EXPECT_EQ(has_dictionary, enable_dictionary); + + // The default disables INT96 timezone conversion, so the raw wall-clock value is stable + // regardless of the Doris session timezone. + expect_int96_timestamp_strings(nullptr, + {"1970-01-01 00:00:00.000000", "1970-01-01 00:00:01.234567", + "2021-01-01 00:00:00.000000", "2021-03-14 09:30:00.000000", + "2021-03-14 10:30:00.000000"}); + expect_int96_timestamp_strings(&shanghai, + {"1970-01-01 08:00:00.000000", "1970-01-01 08:00:01.234567", + "2021-01-01 08:00:00.000000", "2021-03-14 17:30:00.000000", + "2021-03-14 18:30:00.000000"}); + // These two UTC instants straddle the 2021 spring-forward transition and verify that the + // configured zone applies its DST rules rather than a fixed offset. + expect_int96_timestamp_strings(&los_angeles, + {"1969-12-31 16:00:00.000000", "1969-12-31 16:00:01.234567", + "2020-12-31 16:00:00.000000", "2021-03-14 01:30:00.000000", + "2021-03-14 03:30:00.000000"}); + + // INT96 is mapped to TIMESTAMPTZ when the scan enables timestamp-with-time-zone mapping. + // Passing a non-UTC reader timezone must not change its instant; the timezone only affects + // how that instant is rendered later. + auto timestamp_tz_type = std::make_shared(6); + _fields[0]->type = timestamp_tz_type; + _fields[0]->type_descriptor.doris_type = timestamp_tz_type; + auto timestamp_tz_reader = create_reader(0, &shanghai); + ASSERT_NE(timestamp_tz_reader, nullptr); + auto timestamp_tz_column = timestamp_tz_type->create_column(); + int64_t rows_read = 0; + ASSERT_TRUE(timestamp_tz_reader->read(ROW_COUNT, timestamp_tz_column, &rows_read).ok()); + ASSERT_EQ(rows_read, ROW_COUNT); + const auto& timestamp_tz_values = + assert_cast(*timestamp_tz_column); + const auto utc = cctz::utc_time_zone(); + EXPECT_EQ(timestamp_tz_values.get_element(0).to_string(utc, 6), + "1970-01-01 00:00:00.000000+00:00"); + EXPECT_EQ(timestamp_tz_values.get_element(1).to_string(utc, 6), + "1970-01-01 00:00:01.234567+00:00"); + EXPECT_EQ(timestamp_tz_values.get_element(3).to_string(shanghai, 6), + "2021-03-14 17:30:00.000000+08:00"); + } } } // namespace diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatConstants.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatConstants.java index 774ee4e6e838f6..6245983b3d4e22 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatConstants.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatConstants.java @@ -50,6 +50,7 @@ public class FileFormatConstants { public static final String PROP_ENCLOSE = "enclose"; public static final String PROP_ENABLE_MAPPING_VARBINARY = "enable_mapping_varbinary"; public static final String PROP_ENABLE_MAPPING_TIMESTAMP_TZ = "enable_mapping_timestamp_tz"; + public static final String PROP_HIVE_PARQUET_TIME_ZONE = "hive.parquet.time-zone"; // decimal(p,s) public static final Pattern DECIMAL_TYPE_PATTERN = Pattern.compile("decimal\\((\\d+),(\\d+)\\)"); diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatUtils.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatUtils.java index 15240f103b0e51..d9e9efb0ed3e2f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/FileFormatUtils.java @@ -21,16 +21,47 @@ import org.apache.doris.catalog.PrimitiveType; import org.apache.doris.catalog.ScalarType; import org.apache.doris.common.AnalysisException; +import org.apache.doris.common.DdlException; import org.apache.doris.common.FeNameFormat; import com.google.common.base.Strings; +import java.time.DateTimeException; +import java.time.ZoneId; import java.util.List; +import java.util.Locale; import java.util.Optional; import java.util.regex.Matcher; public class FileFormatUtils { + public static String parseHiveParquetTimeZone(String value) throws DdlException { + String timeZone = value.trim(); + if (timeZone.isEmpty()) { + return ""; + } + + try { + String upperCaseTimeZone = timeZone.toUpperCase(Locale.ROOT); + boolean isUtcOrGmt = upperCaseTimeZone.equals("UTC") || upperCaseTimeZone.equals("GMT"); + boolean isShortAlias = ZoneId.SHORT_IDS.containsKey(upperCaseTimeZone) + || upperCaseTimeZone.equals("CST") || upperCaseTimeZone.equals("PRC"); + if (!isUtcOrGmt && isShortAlias) { + throw new DateTimeException("Ambiguous short timezone aliases are not supported"); + } + String standardizedTimeZone = TimeUtils.checkTimeZoneValidAndStandardize(timeZone); + if ((standardizedTimeZone.startsWith("UTC") || standardizedTimeZone.startsWith("GMT")) + && standardizedTimeZone.length() > 3) { + TimeUtils.checkTimeZoneValidAndStandardize(standardizedTimeZone.substring(3)); + } + return ZoneId.of(standardizedTimeZone).getId(); + } catch (DdlException | DateTimeException e) { + throw new DdlException("The parameter " + FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE + + " must be an IANA timezone or UTC offset in the range -12:00 to +14:00; " + + "short timezone aliases are not supported, value is " + timeZone, e); + } + } + public static boolean isCsv(String formatStr) { return FileFormatConstants.FORMAT_CSV.equalsIgnoreCase(formatStr) || FileFormatConstants.FORMAT_CSV_WITH_NAMES.equalsIgnoreCase(formatStr) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java index 24de55133be39c..926080174654ec 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java @@ -26,6 +26,7 @@ import org.apache.doris.catalog.TableIndexes; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.Pair; +import org.apache.doris.common.UserException; import org.apache.doris.common.io.Text; import org.apache.doris.common.io.Writable; import org.apache.doris.common.util.PropertyAnalyzer; @@ -230,6 +231,10 @@ public String getMetaCacheEngine() { return "default"; } + public String getHiveParquetTimeZone() throws UserException { + return ""; + } + @Override public String getMysqlType() { return getType().toMysqlType(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java index 55da4f30ad7043..7ae158f399e063 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileQueryScanNode.java @@ -45,6 +45,7 @@ import org.apache.doris.spi.Split; import org.apache.doris.system.Backend; import org.apache.doris.tablefunction.ExternalFileTableValuedFunction; +import org.apache.doris.tablefunction.TableValuedFunctionIf; import org.apache.doris.thrift.TColumnCategory; import org.apache.doris.thrift.TExternalScanRange; import org.apache.doris.thrift.TFileAttributes; @@ -191,6 +192,10 @@ protected void initSchemaParams() throws UserException { // Set enable_mapping_varbinary from catalog or TVF params.setEnableMappingVarbinary(getEnableMappingVarbinary()); params.setEnableMappingTimestampTz(getEnableMappingTimestampTz()); + String hiveParquetTimeZone = getHiveParquetTimeZone(); + if (!hiveParquetTimeZone.isEmpty()) { + params.setHiveParquetTimeZone(hiveParquetTimeZone); + } } private void updateRequiredSlots() throws UserException { @@ -677,6 +682,19 @@ protected boolean getEnableMappingTimestampTz() { return false; } + protected String getHiveParquetTimeZone() throws UserException { + TableIf table = getTargetTable(); + if (table instanceof ExternalTable) { + return ((ExternalTable) table).getHiveParquetTimeZone(); + } + if (table instanceof FunctionGenTable) { + FunctionGenTable functionGenTable = (FunctionGenTable) table; + TableValuedFunctionIf tvf = functionGenTable.getTvf(); + return tvf.getHiveParquetTimeZone(); + } + return ""; + } + protected abstract List getPathPartitionKeys() throws UserException; protected abstract TableIf getTargetTable() throws UserException; diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java index e4157cf72ddadf..3df91cd29bddca 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java @@ -20,6 +20,8 @@ import org.apache.doris.catalog.Env; import org.apache.doris.common.DdlException; import org.apache.doris.common.ThreadPoolManager; +import org.apache.doris.common.util.FileFormatConstants; +import org.apache.doris.common.util.FileFormatUtils; import org.apache.doris.common.util.Util; import org.apache.doris.datasource.CatalogProperty; import org.apache.doris.datasource.ExternalCatalog; @@ -65,7 +67,6 @@ public class HMSExternalCatalog extends ExternalCatalog { // But notice that if set to true, the default value of column will be ignored because we cannot get default value // from remoteTable object. public static final String GET_SCHEMA_FROM_TABLE = "get_schema_from_table"; - private static final int FILE_SYSTEM_EXECUTOR_THREAD_NUM = 16; private ThreadPoolExecutor fileSystemExecutor; private SpiSwitchingFileSystem spiFileSystem; @@ -123,9 +124,24 @@ public void checkProperties() throws DdlException { throw new DdlException( "The parameter " + PARTITION_CACHE_TTL_SECOND + " is wrong, value is " + partitionCacheTtlSecond); } + + getHiveParquetTimeZone(); catalogProperty.checkMetaStoreAndStorageProperties(AbstractHiveProperties.class); } + /** + * Timezone used to reverse legacy Parquet INT96 writer normalization. This intentionally + * follows Trino's hive.parquet.time-zone catalog property instead of the SQL session timezone. + * An empty value keeps INT96 wall-clock fields unchanged. A configured value applies to Hive + * and Hudi Parquet files in this catalog because Parquet metadata cannot reliably identify the + * writer's timezone convention. Iceberg timestamp semantics are defined by its table format + * and intentionally ignore this property. + */ + public String getHiveParquetTimeZone() throws DdlException { + return FileFormatUtils.parseHiveParquetTimeZone(catalogProperty.getOrDefault( + FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, "")); + } + @Override protected synchronized void initPreExecutionAuthenticator() { if (executionAuthenticator == null) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java index e1311237a603d5..23ec991acbb104 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java @@ -579,6 +579,14 @@ public DLAType getDlaType() { return dlaType; } + @Override + public String getHiveParquetTimeZone() throws UserException { + if (getDlaType() == DLAType.ICEBERG) { + return ""; + } + return ((HMSExternalCatalog) getCatalog()).getHiveParquetTimeZone(); + } + @Override public TTableDescriptor toThrift() { List schema = getFullSchema(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatProperties.java index 9c4d5f2ae494fd..e30a2eec602b80 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatProperties.java @@ -61,7 +61,7 @@ public class ParquetFileFormatProperties extends FileFormatProperties { private TParquetCompressionType parquetCompressionType = TParquetCompressionType.SNAPPY; private boolean parquetDisableDictionary = false; private TParquetVersion parquetVersion = TParquetVersion.PARQUET_1_0; - private boolean enableInt96Timestamps = true; + private boolean enableInt96Timestamps = false; public ParquetFileFormatProperties() { super(TFileFormatType.FORMAT_PARQUET, FileFormatProperties.FORMAT_PARQUET); @@ -86,7 +86,7 @@ public void analyzeFileFormatProperties(Map formatProperties, bo //save the enable int96 timestamp property if (formatProperties.containsKey(ENABLE_INT96_TIMESTAMPS)) { - this.enableInt96Timestamps = Boolean.valueOf(formatProperties.get(ENABLE_INT96_TIMESTAMPS)).booleanValue(); + this.enableInt96Timestamps = parseEnableInt96Timestamps(formatProperties.get(ENABLE_INT96_TIMESTAMPS)); } // save all parquet prefix property @@ -110,6 +110,16 @@ public void analyzeFileFormatProperties(Map formatProperties, bo } } + public static boolean parseEnableInt96Timestamps(String value) { + if ("true".equalsIgnoreCase(value)) { + return true; + } + if ("false".equalsIgnoreCase(value)) { + return false; + } + throw new AnalysisException(ENABLE_INT96_TIMESTAMPS + + " should be true or false, but is " + value); + } @Override public void fullTResultFileSinkOptions(TResultFileSinkOptions sinkOptions) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java index e286bb3e3bacb2..4b311f9319d7e1 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java +++ b/fe/fe-core/src/main/java/org/apache/doris/load/ExportJob.java @@ -38,6 +38,7 @@ import org.apache.doris.common.io.Writable; import org.apache.doris.common.util.TimeUtils; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.property.fileformat.ParquetFileFormatProperties; import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.analyzer.UnboundRelation; import org.apache.doris.nereids.analyzer.UnboundSlot; @@ -123,6 +124,8 @@ public class ExportJob implements Writable { private String columns; @SerializedName("format") private String format; + @SerializedName("enableInt96Timestamps") + private String enableInt96Timestamps; @SerializedName("timeoutSecond") private int timeoutSecond; @SerializedName("maxFileSize") @@ -484,6 +487,11 @@ private Map convertOutfileProperties() { outfileProperties.put(ExportCommand.COMPRESS_TYPE, compressType); } + if (enableInt96Timestamps != null) { + outfileProperties.put(ParquetFileFormatProperties.ENABLE_INT96_TIMESTAMPS, + enableInt96Timestamps); + } + if (!maxFileSize.isEmpty()) { outfileProperties.put(OutFileClause.PROP_MAX_FILE_SIZE, maxFileSize); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java index 3fadf9bc05aba4..19663686f636d4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExportCommand.java @@ -38,6 +38,7 @@ import org.apache.doris.common.util.PropertyAnalyzer; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.property.fileformat.ParquetFileFormatProperties; import org.apache.doris.load.ExportJob; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.exceptions.AnalysisException; @@ -92,6 +93,7 @@ public class ExportCommand extends Command implements NeedAuditEncryption, Forwa .add(PropertyAnalyzer.PROPERTIES_LINE_DELIMITER) .add(PropertyAnalyzer.PROPERTIES_TIMEOUT) .add("format") + .add(ParquetFileFormatProperties.ENABLE_INT96_TIMESTAMPS) .add(OutFileClause.PROP_WITH_BOM) .add(COMPRESS_TYPE) .build(); @@ -291,6 +293,12 @@ private ExportJob generateExportJob(ConnectContext ctx, Map file exportJob.setFormat(fileProperties.getOrDefault(LoadCommand.KEY_IN_PARAM_FORMAT_TYPE, "csv") .toLowerCase()); + String enableInt96Timestamps = fileProperties.get(ParquetFileFormatProperties.ENABLE_INT96_TIMESTAMPS); + if (enableInt96Timestamps != null) { + ParquetFileFormatProperties.parseEnableInt96Timestamps(enableInt96Timestamps); + } + exportJob.setEnableInt96Timestamps(enableInt96Timestamps); + // set withBom exportJob.setWithBom(fileProperties.getOrDefault(OutFileClause.PROP_WITH_BOM, "false")); @@ -431,4 +439,3 @@ public String toDigest() { return sb.toString(); } } - diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java index 6c7962e7267033..8304e19c8b5948 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunction.java @@ -34,6 +34,7 @@ import org.apache.doris.catalog.VariantType; import org.apache.doris.common.AnalysisException; import org.apache.doris.common.Config; +import org.apache.doris.common.DdlException; import org.apache.doris.common.ErrorCode; import org.apache.doris.common.Pair; import org.apache.doris.common.UserException; @@ -131,6 +132,7 @@ public abstract class ExternalFileTableValuedFunction extends TableValuedFunctio protected Optional resourceName = Optional.empty(); public FileFormatProperties fileFormatProperties; + private String hiveParquetTimeZone = ""; private long tableId; public abstract TFileType getTFileType(); @@ -224,6 +226,14 @@ protected Map parseCommonProperties(Map properti FileFormatConstants.PROP_ENABLE_MAPPING_TIMESTAMP_TZ, "false"); fileFormatProperties.enableMappingTimestampTz = Boolean.parseBoolean(enableMappingTimestampTzStr); + String hiveParquetTimeZone = getOrDefaultAndRemove(copiedProps, + FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, ""); + try { + this.hiveParquetTimeZone = FileFormatUtils.parseHiveParquetTimeZone(hiveParquetTimeZone); + } catch (DdlException e) { + throw new AnalysisException(e.getMessage(), e); + } + fileFormatProperties.analyzeFileFormatProperties(copiedProps, true); if (fileFormatProperties instanceof CsvFileFormatProperties @@ -255,6 +265,11 @@ public List getFileStatuses() { return fileStatuses; } + @Override + public String getHiveParquetTimeZone() { + return hiveParquetTimeZone; + } + public TFileAttributes getFileAttributes() { return fileFormatProperties.toTFileAttributes(); } @@ -478,6 +493,10 @@ private PFetchTableSchemaRequest getFetchTableStructureRequest() throws TExcepti // table function fetch schema, whether to enable mapping varbinary fileScanRangeParams.setEnableMappingVarbinary(fileFormatProperties.enableMappingVarbinary); fileScanRangeParams.setEnableMappingTimestampTz(fileFormatProperties.enableMappingTimestampTz); + String hiveParquetTimeZone = getHiveParquetTimeZone(); + if (!hiveParquetTimeZone.isEmpty()) { + fileScanRangeParams.setHiveParquetTimeZone(hiveParquetTimeZone); + } if (getTFileType() == TFileType.FILE_STREAM) { fileStatuses.add(new TBrokerFileStatus("", false, -1, true)); diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/FileTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/FileTableValuedFunction.java index 2cb5bedb4463ce..03d260c9f861f5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/FileTableValuedFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/FileTableValuedFunction.java @@ -109,6 +109,11 @@ public TFileAttributes getFileAttributes() { return delegateTvf.getFileAttributes(); } + @Override + public String getHiveParquetTimeZone() { + return delegateTvf.getHiveParquetTimeZone(); + } + @Override public void checkAuth(ConnectContext ctx) { delegateTvf.checkAuth(ctx); diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java index a36c289aca182c..e637317b1165e4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/TableValuedFunctionIf.java @@ -115,6 +115,10 @@ public static TableValuedFunctionIf getTableFunction(String funcName, Map properties = new HashMap<>(); + properties.put("type", "hms"); + properties.put("hive.metastore.uris", "thrift://localhost:9083"); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, "8:00"); + HMSExternalCatalog catalog = new HMSExternalCatalog(1, "hms", null, properties, ""); + catalog.checkProperties(); + + HMSExternalTable hmsTable = Mockito.mock(HMSExternalTable.class, Mockito.CALLS_REAL_METHODS); + Mockito.doReturn(catalog).when(hmsTable).getCatalog(); + Mockito.doReturn(DLAType.HIVE).when(hmsTable).getDlaType(); + Mockito.doReturn(Collections.emptyList()).when(hmsTable).getBaseSchema(false); + Mockito.doReturn(Collections.emptyList()).when(hmsTable).getFullSchema(); + SessionVariable sessionVariable = new SessionVariable(); + sessionVariable.setTimeZone("America/Los_Angeles"); + TestFileQueryScanNode node = new TestFileQueryScanNode(sessionVariable); + node.setTargetTable(hmsTable); + node.getTupleDescriptor().setTable(hmsTable); + + node.initSchemaParamsForTest(); + + Assert.assertEquals("+08:00", node.getFileScanRangeParams().getHiveParquetTimeZone()); + Assert.assertNotEquals(sessionVariable.getTimeZone(), + node.getFileScanRangeParams().getHiveParquetTimeZone()); + + SessionVariable differentSessionVariable = new SessionVariable(); + differentSessionVariable.setTimeZone("Asia/Tokyo"); + TestFileQueryScanNode nodeWithDifferentSession = new TestFileQueryScanNode(differentSessionVariable); + nodeWithDifferentSession.setTargetTable(hmsTable); + nodeWithDifferentSession.getTupleDescriptor().setTable(hmsTable); + nodeWithDifferentSession.initSchemaParamsForTest(); + + Assert.assertEquals(node.getFileScanRangeParams().getHiveParquetTimeZone(), + nodeWithDifferentSession.getFileScanRangeParams().getHiveParquetTimeZone()); + } + + @Test + public void testHiveParquetTimezoneIsNotSetForHmsIcebergTable() throws Exception { + HMSExternalCatalog catalog = Mockito.mock(HMSExternalCatalog.class); + Mockito.when(catalog.getHiveParquetTimeZone()).thenReturn("Asia/Shanghai"); + HMSExternalTable hmsTable = Mockito.mock(HMSExternalTable.class, Mockito.CALLS_REAL_METHODS); + Mockito.doReturn(catalog).when(hmsTable).getCatalog(); + Mockito.doReturn(DLAType.ICEBERG).when(hmsTable).getDlaType(); + Mockito.doReturn(Collections.emptyList()).when(hmsTable).getBaseSchema(false); + Mockito.doReturn(Collections.emptyList()).when(hmsTable).getFullSchema(); + + TestFileQueryScanNode node = new TestFileQueryScanNode(new SessionVariable()); + node.setTargetTable(hmsTable); + node.getTupleDescriptor().setTable(hmsTable); + node.initSchemaParamsForTest(); + + Assert.assertFalse(node.getFileScanRangeParams().isSetHiveParquetTimeZone()); + } + + @Test + public void testHiveParquetTimezoneIsSetForHmsHudiTable() throws Exception { + HMSExternalCatalog catalog = Mockito.mock(HMSExternalCatalog.class); + Mockito.when(catalog.getHiveParquetTimeZone()).thenReturn("Asia/Shanghai"); + HMSExternalTable hmsTable = Mockito.mock(HMSExternalTable.class, Mockito.CALLS_REAL_METHODS); + Mockito.doReturn(catalog).when(hmsTable).getCatalog(); + Mockito.doReturn(DLAType.HUDI).when(hmsTable).getDlaType(); + Mockito.doReturn(Collections.emptyList()).when(hmsTable).getBaseSchema(false); + Mockito.doReturn(Collections.emptyList()).when(hmsTable).getFullSchema(); + + TestFileQueryScanNode node = new TestFileQueryScanNode(new SessionVariable()); + node.setTargetTable(hmsTable); + node.getTupleDescriptor().setTable(hmsTable); + node.initSchemaParamsForTest(); + + Assert.assertEquals("Asia/Shanghai", node.getFileScanRangeParams().getHiveParquetTimeZone()); + } + + @Test + public void testHiveParquetTimezoneComesFromTableValuedFunction() throws Exception { + ExternalFileTableValuedFunction tvf = Mockito.mock(ExternalFileTableValuedFunction.class); + tvf.fileFormatProperties = new ParquetFileFormatProperties(); + Mockito.when(tvf.getHiveParquetTimeZone()).thenReturn("Asia/Shanghai"); + FunctionGenTable functionGenTable = Mockito.mock(FunctionGenTable.class); + Mockito.when(functionGenTable.getTvf()).thenReturn(tvf); + Mockito.when(functionGenTable.getBaseSchema(false)).thenReturn(Collections.emptyList()); + Mockito.when(functionGenTable.getFullSchema()).thenReturn(Collections.emptyList()); + + TestFileQueryScanNode node = new TestFileQueryScanNode(new SessionVariable()); + node.setTargetTable(functionGenTable); + node.getTupleDescriptor().setTable(functionGenTable); + node.initSchemaParamsForTest(); + + Assert.assertEquals("Asia/Shanghai", node.getFileScanRangeParams().getHiveParquetTimeZone()); + } + + @Test + public void testHiveParquetTimezoneComesFromFileTableValuedFunctionDelegate() throws Exception { + Map properties = new HashMap<>(); + properties.put("uri", "s3://bucket/path/file.parquet"); + properties.put("s3.endpoint", "s3.us-east-1.amazonaws.com"); + properties.put("s3.region", "us-east-1"); + properties.put("s3.access_key", "access-key"); + properties.put("s3.secret_key", "secret-key"); + properties.put(FileFormatConstants.PROP_FORMAT, FileFormatConstants.FORMAT_PARQUET); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, "Asia/Shanghai"); + + boolean originalRunningUnitTest = FeConstants.runningUnitTest; + FeConstants.runningUnitTest = true; + try { + FileTableValuedFunction tvf = new FileTableValuedFunction(properties); + tvf.fileFormatProperties = new ParquetFileFormatProperties(); + FunctionGenTable functionGenTable = new FunctionGenTable( + -1, "file", TableIf.TableType.TABLE_VALUED_FUNCTION, Collections.emptyList(), tvf); + + TestFileQueryScanNode node = new TestFileQueryScanNode(new SessionVariable()); + node.setTargetTable(functionGenTable); + node.getTupleDescriptor().setTable(functionGenTable); + node.initSchemaParamsForTest(); + + Assert.assertEquals("Asia/Shanghai", node.getFileScanRangeParams().getHiveParquetTimeZone()); + } finally { + FeConstants.runningUnitTest = originalRunningUnitTest; + } + } + + @Test + public void testHiveParquetTimezoneIgnoresJdbcQueryTableValuedFunction() throws Exception { + JdbcQueryTableValueFunction tvf = Mockito.mock( + JdbcQueryTableValueFunction.class, Mockito.CALLS_REAL_METHODS); + FunctionGenTable functionGenTable = new FunctionGenTable( + -1, "query", TableIf.TableType.TABLE_VALUED_FUNCTION, Collections.emptyList(), tvf); + TestFileQueryScanNode node = new TestFileQueryScanNode(new SessionVariable()); + node.setTargetTable(functionGenTable); + node.getTupleDescriptor().setTable(functionGenTable); + + node.initSchemaParamsForTest(); + + Assert.assertFalse(node.getFileScanRangeParams().isSetHiveParquetTimeZone()); + } + @Test public void testUpdateRequiredSlotsPreservesInlineDefaultValueExpr() throws Exception { SessionVariable sv = new SessionVariable(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HMSExternalCatalogParquetTimeZoneTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HMSExternalCatalogParquetTimeZoneTest.java new file mode 100644 index 00000000000000..6ecdf384bfb8de --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HMSExternalCatalogParquetTimeZoneTest.java @@ -0,0 +1,88 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource.hive; + +import org.apache.doris.common.DdlException; +import org.apache.doris.common.util.FileFormatConstants; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +public class HMSExternalCatalogParquetTimeZoneTest { + private static Map baseProperties() { + Map properties = new HashMap<>(); + properties.put("type", "hms"); + properties.put("hive.metastore.uris", "thrift://localhost:9083"); + return properties; + } + + @Test + public void testHiveParquetTimeZoneDefaultsToDisabled() throws Exception { + HMSExternalCatalog catalog = new HMSExternalCatalog(1, "hms", null, baseProperties(), ""); + Assertions.assertEquals("", catalog.getHiveParquetTimeZone()); + } + + @Test + public void testHiveParquetTimeZoneAcceptsIanaZoneAndUtcOffset() throws Exception { + for (String timezone : new String[] { + "Asia/Shanghai", "+08:00", "UTC", "+14:00", "-12:00", "UTC+14:00", "GMT-12:00" + }) { + Map properties = baseProperties(); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, timezone); + HMSExternalCatalog catalog = new HMSExternalCatalog(1, "hms", null, properties, ""); + catalog.checkProperties(); + Assertions.assertEquals(timezone, catalog.getHiveParquetTimeZone()); + } + } + + @Test + public void testHiveParquetTimeZoneCanonicalizesShorthandOffset() throws Exception { + Map properties = baseProperties(); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, "8:00"); + HMSExternalCatalog catalog = new HMSExternalCatalog(1, "hms", null, properties, ""); + catalog.checkProperties(); + Assertions.assertEquals("+08:00", catalog.getHiveParquetTimeZone()); + } + + @Test + public void testHiveParquetTimeZoneRejectsAmbiguousShortAliases() { + for (String timezone : new String[] {"AET", "CST", "PRC"}) { + Map properties = baseProperties(); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, timezone); + HMSExternalCatalog catalog = new HMSExternalCatalog(1, "hms", null, properties, ""); + DdlException exception = Assertions.assertThrows(DdlException.class, catalog::checkProperties); + Assertions.assertTrue(exception.getMessage().contains("short timezone aliases are not supported")); + } + } + + @Test + public void testHiveParquetTimeZoneRejectsInvalidZone() { + for (String timezone : new String[] { + "Not/A_Timezone", "+15:00", "-13:00", "UTC+15:00", "GMT-13:00" + }) { + Map properties = baseProperties(); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, timezone); + HMSExternalCatalog catalog = new HMSExternalCatalog(1, "hms", null, properties, ""); + DdlException exception = Assertions.assertThrows(DdlException.class, catalog::checkProperties); + Assertions.assertTrue(exception.getMessage().contains(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE)); + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatPropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatPropertiesTest.java index 4d140b2ba57037..0a8831585de4d0 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatPropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/fileformat/ParquetFileFormatPropertiesTest.java @@ -47,7 +47,7 @@ public void testAnalyzeFileFormatProperties() { Assert.assertEquals(TParquetCompressionType.SNAPPY, parquetFileFormatProperties.getParquetCompressionType()); Assert.assertEquals(false, parquetFileFormatProperties.isParquetDisableDictionary()); - Assert.assertTrue(parquetFileFormatProperties.isEnableInt96Timestamps()); + Assert.assertFalse(parquetFileFormatProperties.isEnableInt96Timestamps()); } @Test @@ -106,6 +106,20 @@ public void testEnableInt96Timestamps() { Assert.assertFalse(parquetFileFormatProperties.isEnableInt96Timestamps()); } + @Test + public void testEnableInt96TimestampsRejectsInvalidBoolean() { + for (String value : new String[] {"yes", "1", " true"}) { + Map properties = new HashMap<>(); + properties.put("enable_int96_timestamps", value); + try { + parquetFileFormatProperties.analyzeFileFormatProperties(properties, true); + Assert.fail("Expected invalid boolean value to be rejected: " + value); + } catch (AnalysisException e) { + Assert.assertTrue(e.getMessage().contains("enable_int96_timestamps")); + } + } + } + @Test public void testParquetVersion() { Map properties = new HashMap<>(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/load/ExportJobTest.java b/fe/fe-core/src/test/java/org/apache/doris/load/ExportJobTest.java new file mode 100644 index 00000000000000..e197eb202c1a90 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/load/ExportJobTest.java @@ -0,0 +1,48 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.load; + +import org.apache.doris.analysis.BrokerDesc; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.datasource.property.fileformat.ParquetFileFormatProperties; + +import org.junit.Assert; +import org.junit.Test; + +import java.util.Map; + +public class ExportJobTest { + + @Test + public void testEnableInt96TimestampsOutfileProperty() { + ExportJob exportJob = new ExportJob(1); + exportJob.setFormat("parquet"); + exportJob.setMaxFileSize(""); + exportJob.setBrokerDesc(new BrokerDesc("local")); + + Map outfileProperties = + Deencapsulation.invoke(exportJob, "convertOutfileProperties"); + Assert.assertFalse(outfileProperties.containsKey( + ParquetFileFormatProperties.ENABLE_INT96_TIMESTAMPS)); + + exportJob.setEnableInt96Timestamps("false"); + outfileProperties = Deencapsulation.invoke(exportJob, "convertOutfileProperties"); + Assert.assertEquals("false", outfileProperties.get( + ParquetFileFormatProperties.ENABLE_INT96_TIMESTAMPS)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java new file mode 100644 index 00000000000000..981b7be805b898 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ExportCommandTest.java @@ -0,0 +1,44 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.trees.plans.commands; + +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.datasource.property.fileformat.ParquetFileFormatProperties; + +import com.google.common.collect.ImmutableMap; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Map; +import java.util.Optional; + +public class ExportCommandTest { + + @Test + void testEnableInt96TimestampsProperty() { + Map properties = ImmutableMap.of( + ParquetFileFormatProperties.ENABLE_INT96_TIMESTAMPS, "false"); + ExportCommand exportCommand = new ExportCommand( + Collections.singletonList("test_table"), Collections.emptyList(), Optional.empty(), + "file:///tmp/export", properties, Optional.empty()); + + Assertions.assertDoesNotThrow( + () -> Deencapsulation.invoke(exportCommand, "checkPropertyKey", properties)); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java index e5b06bd5dd4101..34df496439589c 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/tablefunction/ExternalFileTableValuedFunctionTest.java @@ -28,11 +28,41 @@ import com.google.common.collect.Maps; import org.junit.Assert; import org.junit.Test; +import org.mockito.Mockito; import java.util.List; import java.util.Map; public class ExternalFileTableValuedFunctionTest { + @Test + public void testHiveParquetTimeZoneIsCanonicalizedAndRemovedFromStorageProperties() + throws AnalysisException { + ExternalFileTableValuedFunction tvf = Mockito.mock( + ExternalFileTableValuedFunction.class, Mockito.CALLS_REAL_METHODS); + Map properties = Maps.newHashMap(); + properties.put(FileFormatConstants.PROP_FORMAT, FileFormatConstants.FORMAT_PARQUET); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, "8:00"); + + Map storageProperties = tvf.parseCommonProperties(properties); + + Assert.assertEquals("+08:00", tvf.getHiveParquetTimeZone()); + Assert.assertFalse(storageProperties.containsKey(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE)); + } + + @Test + public void testHiveParquetTimeZoneRejectsAmbiguousShortAlias() { + ExternalFileTableValuedFunction tvf = Mockito.mock( + ExternalFileTableValuedFunction.class, Mockito.CALLS_REAL_METHODS); + Map properties = Maps.newHashMap(); + properties.put(FileFormatConstants.PROP_FORMAT, FileFormatConstants.FORMAT_PARQUET); + properties.put(FileFormatConstants.PROP_HIVE_PARQUET_TIME_ZONE, "CST"); + + AnalysisException exception = Assert.assertThrows( + AnalysisException.class, () -> tvf.parseCommonProperties(properties)); + + Assert.assertTrue(exception.getMessage().contains("short timezone aliases are not supported")); + } + @Test public void testCsvSchemaParse() { Config.enable_date_conversion = true; diff --git a/gensrc/thrift/PlanNodes.thrift b/gensrc/thrift/PlanNodes.thrift index b546649933c239..4ae117d94819c5 100644 --- a/gensrc/thrift/PlanNodes.thrift +++ b/gensrc/thrift/PlanNodes.thrift @@ -560,6 +560,9 @@ struct TFileScanRangeParams { 32: optional map es_docvalue_context // ES fields field→keyword mappings 33: optional map es_fields_context + // HMS catalog property hive.parquet.time-zone. When absent, format_v2 keeps INT96 wall-clock + // values unchanged. When present, only INT96 TIMESTAMP values are converted with this zone. + 34: optional string hive_parquet_time_zone } struct TFileRangeDesc { diff --git a/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_array_type.out b/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_array_type.out index 497fcc7d7b8190..2d11f7eda340a2 100644 --- a/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_array_type.out +++ b/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_array_type.out @@ -60,12 +60,12 @@ 7 doris7 [null, null, null, "2017-10-01 00:00:00", "2011-10-01 01:23:59"] -- !select_load_datetime -- -1 doris1 ["2017-10-01 00:00:00.000000", "2011-10-01 01:23:59.000000"] -2 doris2 ["2017-10-01 00:00:00.000000", "2011-10-01 01:23:59.000000"] +1 doris1 ["2017-10-01 00:00:00.000", "2011-10-01 01:23:59.000"] +2 doris2 ["2017-10-01 00:00:00.000", "2011-10-01 01:23:59.000"] 3 doris3 [] -5 doris5 ["2017-10-01 00:00:00.000000", null, "2017-10-01 00:00:00.000000"] +5 doris5 ["2017-10-01 00:00:00.000", null, "2017-10-01 00:00:00.000"] 6 doris6 [null, null, null] -7 doris7 [null, null, null, "2017-10-01 00:00:00.000000", "2011-10-01 01:23:59.000000"] +7 doris7 [null, null, null, "2017-10-01 00:00:00.000", "2011-10-01 01:23:59.000"] -- !select_base_varchar -- 1 doris1 ["2017-10-01 00:00:00", "2011-10-01 01:23:59"] diff --git a/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_complex_type.out b/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_complex_type.out index cd7fe1e40fdb2d..8d8eb55024f07c 100644 --- a/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_complex_type.out +++ b/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_complex_type.out @@ -116,26 +116,26 @@ 10 doris_10 {"user_id":10, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00", "city":null, "age":null, "sex":null, "bool_col":null, "int_col":null, "bigint_col":null, "largeint_col":null, "float_col":null, "double_col":null, "char_col":null, "decimal_col":null} -- !select_load7 -- -1 doris_1 {"user_id":1, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":1, "sex":1, "bool_col":1, "int_col":1, "bigint_col":1, "largeint_col":"1", "float_col":1.1, "double_col":1.1, "char_col":"char1_1234", "decimal_col":1.000000000} -2 doris_2 {"user_id":2, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":2, "sex":2, "bool_col":1, "int_col":2, "bigint_col":2, "largeint_col":"2", "float_col":2.2, "double_col":2.2, "char_col":"char2_1234", "decimal_col":2.000000000} -3 doris_3 {"user_id":3, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":3, "sex":3, "bool_col":1, "int_col":3, "bigint_col":3, "largeint_col":"3", "float_col":3.3, "double_col":3.3, "char_col":"char3_1234", "decimal_col":3.000000000} -4 doris_4 {"user_id":4, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":4, "sex":4, "bool_col":1, "int_col":4, "bigint_col":4, "largeint_col":"4", "float_col":4.4, "double_col":4.4, "char_col":"char4_1234", "decimal_col":4.000000000} -5 doris_5 {"user_id":5, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":5, "sex":5, "bool_col":1, "int_col":5, "bigint_col":5, "largeint_col":"5", "float_col":5.5, "double_col":5.5, "char_col":"char5_1234", "decimal_col":5.000000000} -6 doris_6 {"user_id":6, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":6, "sex":6, "bool_col":1, "int_col":6, "bigint_col":6, "largeint_col":"6", "float_col":6.6, "double_col":6.6, "char_col":"char6_1234", "decimal_col":6.000000000} -7 doris_7 {"user_id":7, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":7, "sex":7, "bool_col":1, "int_col":7, "bigint_col":7, "largeint_col":"7", "float_col":7.7, "double_col":7.7, "char_col":"char7_1234", "decimal_col":7.000000000} -8 doris_8 {"user_id":8, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":8, "sex":8, "bool_col":1, "int_col":8, "bigint_col":8, "largeint_col":"8", "float_col":8.8, "double_col":8.800000000000001, "char_col":"char8_1234", "decimal_col":8.000000000} -9 doris_9 {"user_id":9, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":9, "sex":9, "bool_col":1, "int_col":9, "bigint_col":9, "largeint_col":"9", "float_col":9.9, "double_col":9.9, "char_col":"char9_1234", "decimal_col":9.000000000} -10 doris_10 {"user_id":10, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":null, "age":null, "sex":null, "bool_col":null, "int_col":null, "bigint_col":null, "largeint_col":null, "float_col":null, "double_col":null, "char_col":null, "decimal_col":null} +1 doris_1 {"user_id":1, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":1, "sex":1, "bool_col":1, "int_col":1, "bigint_col":1, "largeint_col":"1", "float_col":1.1, "double_col":1.1, "char_col":"char1_1234", "decimal_col":1.000000000} +2 doris_2 {"user_id":2, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":2, "sex":2, "bool_col":1, "int_col":2, "bigint_col":2, "largeint_col":"2", "float_col":2.2, "double_col":2.2, "char_col":"char2_1234", "decimal_col":2.000000000} +3 doris_3 {"user_id":3, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":3, "sex":3, "bool_col":1, "int_col":3, "bigint_col":3, "largeint_col":"3", "float_col":3.3, "double_col":3.3, "char_col":"char3_1234", "decimal_col":3.000000000} +4 doris_4 {"user_id":4, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":4, "sex":4, "bool_col":1, "int_col":4, "bigint_col":4, "largeint_col":"4", "float_col":4.4, "double_col":4.4, "char_col":"char4_1234", "decimal_col":4.000000000} +5 doris_5 {"user_id":5, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":5, "sex":5, "bool_col":1, "int_col":5, "bigint_col":5, "largeint_col":"5", "float_col":5.5, "double_col":5.5, "char_col":"char5_1234", "decimal_col":5.000000000} +6 doris_6 {"user_id":6, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":6, "sex":6, "bool_col":1, "int_col":6, "bigint_col":6, "largeint_col":"6", "float_col":6.6, "double_col":6.6, "char_col":"char6_1234", "decimal_col":6.000000000} +7 doris_7 {"user_id":7, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":7, "sex":7, "bool_col":1, "int_col":7, "bigint_col":7, "largeint_col":"7", "float_col":7.7, "double_col":7.7, "char_col":"char7_1234", "decimal_col":7.000000000} +8 doris_8 {"user_id":8, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":8, "sex":8, "bool_col":1, "int_col":8, "bigint_col":8, "largeint_col":"8", "float_col":8.8, "double_col":8.800000000000001, "char_col":"char8_1234", "decimal_col":8.000000000} +9 doris_9 {"user_id":9, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":9, "sex":9, "bool_col":1, "int_col":9, "bigint_col":9, "largeint_col":"9", "float_col":9.9, "double_col":9.9, "char_col":"char9_1234", "decimal_col":9.000000000} +10 doris_10 {"user_id":10, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":null, "age":null, "sex":null, "bool_col":null, "int_col":null, "bigint_col":null, "largeint_col":null, "float_col":null, "double_col":null, "char_col":null, "decimal_col":null} -- !select_load7 -- -1 doris_1 {"user_id":1, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":1, "sex":1, "bool_col":1, "int_col":1, "bigint_col":1, "largeint_col":"1", "float_col":1.1, "double_col":1.1, "char_col":"char1_1234", "decimal_col":1.000000000} -2 doris_2 {"user_id":2, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":2, "sex":2, "bool_col":1, "int_col":2, "bigint_col":2, "largeint_col":"2", "float_col":2.2, "double_col":2.2, "char_col":"char2_1234", "decimal_col":2.000000000} -3 doris_3 {"user_id":3, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":3, "sex":3, "bool_col":1, "int_col":3, "bigint_col":3, "largeint_col":"3", "float_col":3.3, "double_col":3.3, "char_col":"char3_1234", "decimal_col":3.000000000} -4 doris_4 {"user_id":4, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":4, "sex":4, "bool_col":1, "int_col":4, "bigint_col":4, "largeint_col":"4", "float_col":4.4, "double_col":4.4, "char_col":"char4_1234", "decimal_col":4.000000000} -5 doris_5 {"user_id":5, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":5, "sex":5, "bool_col":1, "int_col":5, "bigint_col":5, "largeint_col":"5", "float_col":5.5, "double_col":5.5, "char_col":"char5_1234", "decimal_col":5.000000000} -6 doris_6 {"user_id":6, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":6, "sex":6, "bool_col":1, "int_col":6, "bigint_col":6, "largeint_col":"6", "float_col":6.6, "double_col":6.6, "char_col":"char6_1234", "decimal_col":6.000000000} -7 doris_7 {"user_id":7, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":7, "sex":7, "bool_col":1, "int_col":7, "bigint_col":7, "largeint_col":"7", "float_col":7.7, "double_col":7.7, "char_col":"char7_1234", "decimal_col":7.000000000} -8 doris_8 {"user_id":8, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":8, "sex":8, "bool_col":1, "int_col":8, "bigint_col":8, "largeint_col":"8", "float_col":8.8, "double_col":8.800000000000001, "char_col":"char8_1234", "decimal_col":8.000000000} -9 doris_9 {"user_id":9, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":"Beijing", "age":9, "sex":9, "bool_col":1, "int_col":9, "bigint_col":9, "largeint_col":"9", "float_col":9.9, "double_col":9.9, "char_col":"char9_1234", "decimal_col":9.000000000} -10 doris_10 {"user_id":10, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000000", "city":null, "age":null, "sex":null, "bool_col":null, "int_col":null, "bigint_col":null, "largeint_col":null, "float_col":null, "double_col":null, "char_col":null, "decimal_col":null} +1 doris_1 {"user_id":1, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":1, "sex":1, "bool_col":1, "int_col":1, "bigint_col":1, "largeint_col":"1", "float_col":1.1, "double_col":1.1, "char_col":"char1_1234", "decimal_col":1.000000000} +2 doris_2 {"user_id":2, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":2, "sex":2, "bool_col":1, "int_col":2, "bigint_col":2, "largeint_col":"2", "float_col":2.2, "double_col":2.2, "char_col":"char2_1234", "decimal_col":2.000000000} +3 doris_3 {"user_id":3, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":3, "sex":3, "bool_col":1, "int_col":3, "bigint_col":3, "largeint_col":"3", "float_col":3.3, "double_col":3.3, "char_col":"char3_1234", "decimal_col":3.000000000} +4 doris_4 {"user_id":4, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":4, "sex":4, "bool_col":1, "int_col":4, "bigint_col":4, "largeint_col":"4", "float_col":4.4, "double_col":4.4, "char_col":"char4_1234", "decimal_col":4.000000000} +5 doris_5 {"user_id":5, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":5, "sex":5, "bool_col":1, "int_col":5, "bigint_col":5, "largeint_col":"5", "float_col":5.5, "double_col":5.5, "char_col":"char5_1234", "decimal_col":5.000000000} +6 doris_6 {"user_id":6, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":6, "sex":6, "bool_col":1, "int_col":6, "bigint_col":6, "largeint_col":"6", "float_col":6.6, "double_col":6.6, "char_col":"char6_1234", "decimal_col":6.000000000} +7 doris_7 {"user_id":7, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":7, "sex":7, "bool_col":1, "int_col":7, "bigint_col":7, "largeint_col":"7", "float_col":7.7, "double_col":7.7, "char_col":"char7_1234", "decimal_col":7.000000000} +8 doris_8 {"user_id":8, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":8, "sex":8, "bool_col":1, "int_col":8, "bigint_col":8, "largeint_col":"8", "float_col":8.8, "double_col":8.800000000000001, "char_col":"char8_1234", "decimal_col":8.000000000} +9 doris_9 {"user_id":9, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":"Beijing", "age":9, "sex":9, "bool_col":1, "int_col":9, "bigint_col":9, "largeint_col":"9", "float_col":9.9, "double_col":9.9, "char_col":"char9_1234", "decimal_col":9.000000000} +10 doris_10 {"user_id":10, "date":"2017-10-01", "datetime":"2017-10-01 00:00:00.000", "city":null, "age":null, "sex":null, "bool_col":null, "int_col":null, "bigint_col":null, "largeint_col":null, "float_col":null, "double_col":null, "char_col":null, "decimal_col":null} diff --git a/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_map_type.out b/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_map_type.out index f210bf62460cda..8c58581f95f858 100644 --- a/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_map_type.out +++ b/regression-test/data/export_p0/outfile/parquet/test_outfile_parquet_map_type.out @@ -218,14 +218,14 @@ 8 doris8 {"2025-12-31 12:01:41":524524, "2006-02-19 09:01:02":2534} -- !select_load11 -- -1 doris1 {"2023-04-20 01:02:03.000000":null, "2018-04-20 10:40:35.000000":123} -2 doris2 {"2000-04-20 00:00:00.000000":-2147483648, "1967-12-31 12:24:56.000000":2147483647} -3 doris3 {"2023-01-01 00:00:00.000000":1246, "2023-02-27 00:01:02.000000":5646} +1 doris1 {"2023-04-20 01:02:03.000":null, "2018-04-20 10:40:35.000":123} +2 doris2 {"2000-04-20 00:00:00.000":-2147483648, "1967-12-31 12:24:56.000":2147483647} +3 doris3 {"2023-01-01 00:00:00.000":1246, "2023-02-27 00:01:02.000":5646} 4 doris4 {} 5 doris5 {} 6 \N \N 7 doris7 \N -8 doris8 {"2025-12-31 12:01:41.000000":524524, "2006-02-19 09:01:02.000000":2534} +8 doris8 {"2025-12-31 12:01:41.000":524524, "2006-02-19 09:01:02.000":2534} -- !select_base12 -- 1 doris1 {"2023-04-20":null, "2018-04-20":123} @@ -278,14 +278,14 @@ 8 doris8 {"2025-12-31 11:22:33":"min_largeint", "2006-02-19 00:44:55":"max_largeint"} -- !select_load14 -- -1 doris1 {"2023-04-20 12:20:03.000000":"null", "2018-04-20 12:59:59.000000":null} -2 doris2 {"2000-04-20 23:59:59.000000":"-2147483648", "1967-12-31 00:00:00.000000":"2147483647"} -3 doris3 {"2023-01-01 07:24:54.000000":"1246", "2023-02-27 15:12:13.000000":"5646"} +1 doris1 {"2023-04-20 12:20:03.000":"null", "2018-04-20 12:59:59.000":null} +2 doris2 {"2000-04-20 23:59:59.000":"-2147483648", "1967-12-31 00:00:00.000":"2147483647"} +3 doris3 {"2023-01-01 07:24:54.000":"1246", "2023-02-27 15:12:13.000":"5646"} 4 doris4 {} 5 doris5 {} 6 \N \N 7 doris7 \N -8 doris8 {"2025-12-31 11:22:33.000000":"min_largeint", "2006-02-19 00:44:55.000000":"max_largeint"} +8 doris8 {"2025-12-31 11:22:33.000":"min_largeint", "2006-02-19 00:44:55.000":"max_largeint"} -- !select_base15 -- 1 doris1 {100:"null", 111:"b"} diff --git a/regression-test/data/external_table_p0/export/hive_read/parquet/test_doris_int96_round_trip.out b/regression-test/data/external_table_p0/export/hive_read/parquet/test_doris_int96_round_trip.out new file mode 100644 index 00000000000000..5cbdb0c83ce1bc --- /dev/null +++ b/regression-test/data/external_table_p0/export/hive_read/parquet/test_doris_int96_round_trip.out @@ -0,0 +1,7 @@ +-- This file is automatically generated. You should know what you did if you want to edit this +-- !session_timezone -- +America/Los_Angeles + +-- !int96_round_trip -- +1 2023-04-20T00:00 2023-04-20T00:00:00.123456 +2 1970-01-01T08:00 1970-01-01T08:00:00.654321 diff --git a/regression-test/data/external_table_p0/paimon/paimon_timestamp_types.out b/regression-test/data/external_table_p0/paimon/paimon_timestamp_types.out index 9df5553d1493c5..4267bc26cbbbe6 100644 --- a/regression-test/data/external_table_p0/paimon/paimon_timestamp_types.out +++ b/regression-test/data/external_table_p0/paimon/paimon_timestamp_types.out @@ -136,4 +136,3 @@ -- !ltz_ntz_simple15 -- 2024-01-02T10:12:34.123456 - diff --git a/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group4_orc_files.out b/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group4_orc_files.out index 1f5ea9741485ca..855cae3466835c 100644 --- a/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group4_orc_files.out +++ b/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group4_orc_files.out @@ -1,6 +1,6 @@ -- This file is automatically generated. You should know what you did if you want to edit this -- !test_0 -- -2022-06-10T05:26:22.753999 2022-06-09T21:26:22.753999 +2022-06-10T05:26:22.754 2022-06-09T21:26:22.754 -- !test_1 -- 1 John Doe diff --git a/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.out b/regression-test/data/external_table_p0/tvf/orc_tvf/test_hdfs_orc_group6_orc_files.out index 85b52a6b28fb6f86b7db14324fbdd7cc675df3d1..dcf3cdb63ef76fe74b72fceaa26c98782814c146 100644 GIT binary patch delta 13 UcmbOkHYsewE-gm0$%nMU0VwkYc>n+a delta 15 XcmbOfHZyF)F0ILPwKym5(Fy|qIHCrg diff --git a/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group1.out b/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group1.out index 55ae6199bfb7ad..51c9f773485999 100644 --- a/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group1.out +++ b/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group1.out @@ -90,9 +90,9 @@ [{"latitude":0, "longitude":180}, {"latitude":0, "longitude":0}] -- !test_17 -- -1 1880-01-01T15:58:41 -2 1884-01-01T16:05:43 -3 1990-01-01T16:00 +1 1880-01-01T07:52:58 +2 1884-01-01T08:00 +3 1990-01-01T08:00 -- !test_18 -- [{"latitude":0, "longitude":0}, null, {"latitude":0, "longitude":180}] diff --git a/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group2.out b/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group2.out index 79b63e41cc1b4d..7b1744aa72f61d 100644 --- a/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group2.out +++ b/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group2.out @@ -122,14 +122,14 @@ apple_banana_mango9 3 Cecilia 2022-11-16T02:32:09.123534 -- !test_17 -- -1001-01-07T17:07:46.123456 1001-01-07T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-08T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-09T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-10T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-11T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-12T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-13T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-14T17:07:46.123456 +1001-01-07T09:02:03.123456 1001-01-07T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-08T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-09T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-10T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-11T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-12T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-13T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-14T09:02:03.123456 -- !test_18 -- 0.00 @@ -151,14 +151,14 @@ apple_banana_mango9 2 -- !test_20 -- -1001-01-07T17:07:46.123456 1001-01-07T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-08T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-09T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-10T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-11T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-12T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-13T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-14T17:07:46.123456 +1001-01-07T09:02:03.123456 1001-01-07T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-08T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-09T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-10T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-11T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-12T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-13T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-14T09:02:03.123456 -- !test_21 -- 1001-01-07 1001-01-07 @@ -181,14 +181,14 @@ apple_banana_mango9 1001-01-07T17:07:46.123 1001-01-14T17:07:46.123 -- !test_23 -- -1001-01-07T17:07:46.123456 1001-01-07T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-08T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-09T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-10T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-11T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-12T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-13T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-14T17:07:46.123456 +1001-01-07T09:02:03.123456 1001-01-07T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-08T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-09T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-10T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-11T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-12T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-13T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-14T09:02:03.123456 -- !test_24 -- false 1 2 3 10 1.2 val_1 val_1 HEARTS false 1 2 3 10 1.2 val_1 val_1 HEARTS ["arr_1", "arr_2", "arr_3"] [1] {1:"val_1", 2:"val_2", 3:"val_3"} {1:[{"nestedintscolumn":[1, 2, 3], "nestedstringcolumn":"val_1"}, {"nestedintscolumn":[2, 3, 4], "nestedstringcolumn":"val_2"}, {"nestedintscolumn":[3, 4, 5], "nestedstringcolumn":"val_3"}], 2:[{"nestedintscolumn":[1, 2, 3], "nestedstringcolumn":"val_1"}, {"nestedintscolumn":[2, 3, 4], "nestedstringcolumn":"val_2"}, {"nestedintscolumn":[3, 4, 5], "nestedstringcolumn":"val_3"}], 3:[{"nestedintscolumn":[1, 2, 3], "nestedstringcolumn":"val_1"}, {"nestedintscolumn":[2, 3, 4], "nestedstringcolumn":"val_2"}, {"nestedintscolumn":[3, 4, 5], "nestedstringcolumn":"val_3"}]} @@ -238,14 +238,14 @@ true 8 9 10 80 8.199999999999999 val_8 val_8 SPADES true 8 9 10 80 8.19999999999 9.00 -- !test_29 -- -1001-01-07T17:07:46.123456 1001-01-07T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-08T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-09T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-10T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-11T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-12T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-13T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-14T17:07:46.123456 +1001-01-07T09:02:03.123456 1001-01-07T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-08T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-09T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-10T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-11T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-12T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-13T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-14T09:02:03.123456 -- !test_30 -- \N @@ -265,14 +265,14 @@ true 8 9 10 80 8.199999999999999 val_8 val_8 SPADES true 8 9 10 80 8.19999999999 1970-01-01T08:00:00.010 -- !test_33 -- -1001-01-07T17:07:46.123456 1001-01-07T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-08T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-09T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-10T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-11T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-12T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-13T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-14T17:07:46.123456 +1001-01-07T09:02:03.123456 1001-01-07T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-08T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-09T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-10T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-11T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-12T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-13T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-14T09:02:03.123456 -- !test_34 -- 1001-01-07 1001-01-07 @@ -295,12 +295,12 @@ true 8 9 10 80 8.199999999999999 val_8 val_8 SPADES true 8 9 10 80 8.19999999999 1001-01-07T17:07:46.123 1001-01-14T17:07:46.123 -- !test_36 -- -1001-01-07T17:07:46.123456 1001-01-07T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-08T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-09T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-10T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-11T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-12T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-13T17:07:46.123456 -1001-01-07T17:07:46.123456 1001-01-14T17:07:46.123456 +1001-01-07T09:02:03.123456 1001-01-07T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-08T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-09T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-10T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-11T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-12T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-13T09:02:03.123456 +1001-01-07T09:02:03.123456 1001-01-14T09:02:03.123456 diff --git a/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group4.out b/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group4.out index d694f2db1415544f537bc1b42caa02a54785ff0c..e05cee918971e3dee254fe382bc57e14bcbb3e0d 100644 GIT binary patch delta 1751 zcmd^A-D?zA6z6PqX0w~tL?v~z4G}-1(OK`k^R+X@1cg4>2RF*JB7VfwkF};0LupGx zp|sF;eXvAs&45Hmz*MRFAXm82Ak_LM!^*}|jvH6W8^_b2_*Gay(wKG8(pPYSL47BcS z0LuH-Wijf=nNXX>0=(d;xSe2Kb( zlg@I27~*p0)$|<7txb2(eX;R7apb)pm6WdOK|Z3b`%Eed$p73nQqg~WeOz>HKd01g zr+a>ZYF?l8ey;%GZQU(=UdgK`j;0fIS@CU2$uwAwMp-2Qe>T#>&-Sma8b9w} zA9FkN#lCDU$eYK8qK^v_=;2T67Z8__G<8g-5kn)oM=0MtockaW+e9OXpxAJU=@Kh% z7o0RB)Yb?-)FUxPS{0xo$y25qriHHKx!xVl3)FEx2 zjl7Et5a!PpV|5ujGnm2h2X!rq)lpAhiK(WrkmB+62Z?0x*<#|>M5b0;_8?W^ir-6N zT)cQ#N`_$Nnw_#@D9`*|j)9w>{F~&5)wlSb=_4`m*i1(n+bSk!PpFvX$1neZ*Rx+M zWC{Qj8>?=+rk;0n$Fj@}HIO&RLuz`2xwPOiE3Y$$*hYpr455NXZI3d)ONEvKu@H9+ zB#P~rW}p%=6K4yp4cDZ3h7{7R8bkx@L9|X#-9H<=i7s!dFt)VTCcnJ{I)KQn9k30m z!^$KMYc$^y`S5GB16pO51Cxt@%K2Mha~*cq@AY6`3WtKZMEC#jzzkQC-1RMe^lW(2D!P&;0;HwJ!9P hVFL<-6)kU+VWRdo^4}SXr}ecDpT>NjABNwP{{ebO^$!35 delta 1609 zcmdT^-)mGy6y|KRcfG51V<5(?D-BXKn%v!)xj*NEDHeRNFKSsaB@vCpL~IL9OGB%) zRr8Mnv-n6y)JT z-%{Ygh}Ig+t$!K6R&?Hrz}0fT^VKx)Gl%-**`f}$1hP^ZtAAmtTk==>zSc-ATX}Oj zc%>&uQ6_)xsp<_R*uM24=H8b3Tz>c3sdUCN2o1Z*4Y$i3+rP}j6@!Ic-$rG62jj%O zlk(1<%Mi!Nv4Jnr;Njl4mrznqBtIKolMVW17Tlnt(jH8WbT`76zq=_NHPU-aW`pIX$bP*)?a*lB9G@Dm}|p^2j*7hDG5suQskbC={&Pm2D$Kpk1^B@p9LPpn3nCh zUOg;6Y48(yFA66ip6&e^Qq~ZPLZ43aqKL$Lgie8+GW(9 zSw7)KlX2o&S;`qgoSY=m79Pm-hNyWna( zFD0dRGte93sXpZ}(*|5j9E201orLLD z4!}`B`{jmLiEI0ynsn1W2#H0+JNseN^KA#wKMZFZ$5U4hKyy5gB_&1=!n!(-mm&t& zwAdYnk!9Q=23p2My=y$5y!jR^Ph*h6`})6R@YK{OoYZ2*)f*o{0ViCH9)oj@s$zN! dHlxJ)ad?#6|3aKwM74S<;Jrm@^$ED&^gquD&l>;$ diff --git a/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group5.out b/regression-test/data/external_table_p0/tvf/test_hdfs_parquet_group5.out index c7b9542f5c766242348f9f4638642042d2c28067..ccf2d100dfdcf2599e23dd91b71bf9b1246ba64d 100644 GIT binary patch delta 880 zcmX|9y-!p@5YH{|PUko-;R+KBhVXG%=zGk5y?yTmIIyrXQIktdjs^`HgAqYZ6r+a$ zgdoJPXR?wYiIs)TDa92Q+SvF%SkPG5D3z_Vcf?|LW_N${o8RutezUyaEH{2gkd7^M$8Qs=t8P3LyKSx;7Zm=>}(U0~8j*-aP3X(2a9f#7@f+++u_ zeT65qp3ToPH0h`O9hZKxbAisbJL`GkzU+5}E)*FB;C%8_eNI3EEC<^hAhx;O~a$#2n zV7gDWcvOO65E6(h0?Ekdq6Jcx4N@^v6__B^lYv2PBBw~kK60OvB4w8cVa92eiIPDW zDvtU;LQn;5*$`aoB@Pm70Ir`Igr(q+*?I|CgF_iYR533Bd zoeW@(VQ4=FZf~p^fi361jlffCyuFC<(lyuEp9u4snnyno+KhqME{g27AyP%d*X)X*NHb%5N8u9bygLfp zv@4FmW7<6*gPnHj$T-|@M<2#X8Ic$}QGrD}ry*y*RA9@UMfUsz>{28|^I!s&dWh<9 qw%*qObM#}Fm+4aJxcys$W3JE?Ai2sEXpe4rdamTRC)j~`KK~CV|J|hk delta 880 zcmX|9J5N+W6wV#)uAXHfVG9!shVWP{^j_x7>plW3SXh~;$tEUCg9eSkh@d8l(Zv8l z5QC3vGLj&Pm4(eJ#TLfI#>W3(L1ST~RJNXBiN(3+n>*k4oij6MuUXz}mK(o4FV6mW zb%s2fV~Y$$w$IiXn(Sd)=2Dlx;nI#j@{{fJ^lz)Z&ID%N#U+>C7VAzYvfhq`meY2}yaUm0SLbz~_zz0j ziK1GfqPjQ6eXtPm*N~agw`5KVI4)wvAo+Q;e-w3I0}#l z0;?F?l|H!KYpDBF=v2s%CyBtQMdC>ybQO*D`{Bw7AyX+*Bp-}p!w-Eh5J^>VyW9^` zy}HU(Dpbl%Xo!T0NHPvk9FT-%15_-g8%lZNibY&UvFk`C0mlHbDui7cfN9sWOqC3c zQONawhEP=@EgOVuJ#k7lA_}!KLLo7X40sd>;ksEFgsbEsqf}vNBGfuLWat26ccfey zrs+NGjWTpIjO=O|rktlql?u6_x(CfuU(zYKXtE*L$dpBzcKWOZ1Ea8Ro`YxST*$WEZ zIL$8wYqWh^!z)JT&KtPp@;w8qKHW99UIecl5_T&hQ$o7g5kn`u5z7^BWeigvsL3l0hUr2?BRbbcIm95T)QaaimotHarPUjv?} k8{-9;EtZbkzco1K3QYp1RG9?r(=KM`N // select ... into outfile ... def uuid = UUID.randomUUID().toString() + def int96Property = enableInt96Timestamps == null ? "" : + ",\n \"enable_int96_timestamps\" = \"${enableInt96Timestamps}\"" outfile_path = "/user/doris/tmp_data/${uuid}" uri = "${defaultFS}" + "${outfile_path}/exp_" @@ -74,8 +76,7 @@ suite("test_hive_read_parquet", "p0,external") { FORMAT AS ${format} PROPERTIES ( "fs.defaultFS"="${defaultFS}", - "hadoop.username" = "${hdfsUserName}", - "enable_int96_timestamps" = "true" + "hadoop.username" = "${hdfsUserName}"${int96Property} ); """ logger.info("outfile success path: " + res[0][3]); @@ -135,7 +136,8 @@ suite("test_hive_read_parquet", "p0,external") { qt_select_base1 """ SELECT * FROM ${export_table_name} ORDER BY user_id; """ // test outfile to hdfs - def outfile_url = outfile_to_HDFS() + // The default Doris OUTFILE path uses INT64 timestamps. + def outfile_url = outfile_to_HDFS(null) // create hive table create_hive_table(hive_table, hive_column_define) @@ -249,9 +251,8 @@ suite("test_hive_read_parquet", "p0,external") { qt_select_base2 """ SELECT * FROM ${export_table_name} t ORDER BY user_id; """ // test outfile to hdfs - def outfile_url = outfile_to_HDFS() - // create hive table - create_hive_table(hive_table, hive_column_define) + // The default Doris OUTFILE path uses INT64 timestamps. + def outfile_url = outfile_to_HDFS(null) qt_select_tvf2 """ select * from HDFS( "uri" = "${outfile_url}0.parquet", @@ -259,6 +260,10 @@ suite("test_hive_read_parquet", "p0,external") { "format" = "${format}"); """ + // Hive 2 and 3 do not support Parquet INT64 logical timestamps. Keep the Doris + // OUTFILE/TVF path on INT64, and use INT96 only for the legacy Hive compatibility check. + outfile_to_HDFS(true) + create_hive_table(hive_table, hive_column_define) qt_hive_docker_02 """ SELECT * FROM ${hive_database}.${hive_table};""" } finally { diff --git a/regression-test/suites/external_table_p0/export/hive_read/parquet/test_hive_read_parquet_complex_type.groovy b/regression-test/suites/external_table_p0/export/hive_read/parquet/test_hive_read_parquet_complex_type.groovy index 94c516478b534b..5fa054d44f9006 100644 --- a/regression-test/suites/external_table_p0/export/hive_read/parquet/test_hive_read_parquet_complex_type.groovy +++ b/regression-test/suites/external_table_p0/export/hive_read/parquet/test_hive_read_parquet_complex_type.groovy @@ -90,9 +90,11 @@ suite("test_hive_read_parquet_complex_type", "p0,external") { hive_docker """ ${create_table_str} """ } - def outfile_to_HDFS = { + def outfile_to_HDFS = { enableInt96Timestamps -> // select ... into outfile ... def uuid = UUID.randomUUID().toString() + def int96Property = enableInt96Timestamps == null ? "" : + ",\n \"enable_int96_timestamps\" = \"${enableInt96Timestamps}\"" outfile_path = "/user/doris/tmp_data/${uuid}" uri = "${defaultFS}" + "${outfile_path}/exp_" @@ -102,8 +104,7 @@ suite("test_hive_read_parquet_complex_type", "p0,external") { INTO OUTFILE "${uri}" FORMAT AS ${format} PROPERTIES ( - "hadoop.username" = "${hdfsUserName}", - "enable_int96_timestamps" = "true" + "hadoop.username" = "${hdfsUserName}"${int96Property} ); """ logger.info("outfile success path: " + res[0][3]); @@ -143,7 +144,8 @@ suite("test_hive_read_parquet_complex_type", "p0,external") { qt_select_base1 """ SELECT * FROM ${export_table_name} t ORDER BY user_id; """ // test outfile to hdfs - def outfile_url = outfile_to_HDFS() + // The default Doris OUTFILE path uses INT64 timestamps. + def outfile_url = outfile_to_HDFS(null) // create hive table create_hive_table(hive_table, hive_field_define) @@ -182,7 +184,8 @@ suite("test_hive_read_parquet_complex_type", "p0,external") { qt_select_base2 """ SELECT * FROM ${export_table_name} t ORDER BY user_id; """ // test outfile to hdfs - def outfile_url = outfile_to_HDFS() + // The default Doris OUTFILE path uses INT64 timestamps. + def outfile_url = outfile_to_HDFS(null) // create hive table create_hive_table(hive_table, hive_field_define) @@ -223,7 +226,8 @@ suite("test_hive_read_parquet_complex_type", "p0,external") { qt_select_base3 """ SELECT * FROM ${export_table_name} t ORDER BY user_id; """ // test outfile to hdfs - def outfile_url = outfile_to_HDFS() + // The default Doris OUTFILE path uses INT64 timestamps. + def outfile_url = outfile_to_HDFS(null) // create hive table create_hive_table(hive_table, hive_field_define) @@ -268,17 +272,24 @@ suite("test_hive_read_parquet_complex_type", "p0,external") { qt_select_base4 """ SELECT * FROM ${export_table_name} t ORDER BY user_id; """ // test outfile to hdfs - def outfile_url = outfile_to_HDFS() - - // create hive table - create_hive_table(hive_table, hive_field_define) - - qt_select_tvf4 """ select * from HDFS( + // The default Doris OUTFILE path uses INT64 timestamps. + def outfile_url = outfile_to_HDFS(null) + + qt_select_tvf4 """ select user_id, name, + cast(s_info as struct) + from HDFS( "uri" = "${outfile_url}0.parquet", "hadoop.username" = "${hdfsUserName}", "format" = "${format}"); """ + // Hive 2 and 3 do not support Parquet INT64 logical timestamps. Keep the Doris + // OUTFILE/TVF path on INT64, and use INT96 only for the legacy Hive compatibility check. + outfile_to_HDFS(true) + create_hive_table(hive_table, hive_field_define) qt_hive_docker_04 """ SELECT * FROM ${hive_database}.${hive_table};""" } finally { diff --git a/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy b/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy index 64c127c030dab4..15b562a082a020 100644 --- a/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy +++ b/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy @@ -569,7 +569,8 @@ suite("test_hive_ctas", "p0,external") { 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}', - 'use_meta_cache' = 'true' + 'use_meta_cache' = 'true', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """switch ${catalog_name}""" diff --git a/regression-test/suites/external_table_p0/hive/test_complex_types.groovy b/regression-test/suites/external_table_p0/hive/test_complex_types.groovy index 40a815a4688d6b..ac59d835b603be 100644 --- a/regression-test/suites/external_table_p0/hive/test_complex_types.groovy +++ b/regression-test/suites/external_table_p0/hive/test_complex_types.groovy @@ -30,7 +30,8 @@ suite("test_complex_types", "p0,external") { sql """drop catalog if exists ${catalog_name}""" sql """create catalog if not exists ${catalog_name} properties ( "type"="hms", - 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}' + 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" logger.info("catalog " + catalog_name + " created") sql """switch ${catalog_name};""" diff --git a/regression-test/suites/external_table_p0/hive/test_external_catalog_hive.groovy b/regression-test/suites/external_table_p0/hive/test_external_catalog_hive.groovy index 50a72ddff075b3..e23cd7f67c1cab 100644 --- a/regression-test/suites/external_table_p0/hive/test_external_catalog_hive.groovy +++ b/regression-test/suites/external_table_p0/hive/test_external_catalog_hive.groovy @@ -33,7 +33,8 @@ suite("test_external_catalog_hive", "p0,external") { sql """ create catalog if not exists ${catalog_name} properties ( 'type'='hms', - 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}' + 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' ); """ diff --git a/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy b/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy index beef3b2254b739..85bd031a3048fe 100644 --- a/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy +++ b/regression-test/suites/external_table_p0/hive/test_external_catalog_hive_partition.groovy @@ -30,7 +30,8 @@ suite("test_external_catalog_hive_partition", "p0,external") { sql """ create catalog if not exists ${catalog_name} properties ( 'type'='hms', - 'hive.metastore.uris' = 'thrift://${extHiveHmsHost}:${extHiveHmsPort}' + 'hive.metastore.uris' = 'thrift://${extHiveHmsHost}:${extHiveHmsPort}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' ); """ diff --git a/regression-test/suites/external_table_p0/hive/test_hive_basic_type.groovy b/regression-test/suites/external_table_p0/hive/test_hive_basic_type.groovy index 1a5965694935a1..9829388457e54c 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_basic_type.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_basic_type.groovy @@ -39,7 +39,8 @@ suite("test_hive_basic_type", "p0,external") { sql """CREATE CATALOG ${catalog_name} PROPERTIES ( 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', - 'hadoop.username' = 'hive' + 'hadoop.username' = 'hive', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """switch ${catalog_name}""" diff --git a/regression-test/suites/external_table_p0/hive/test_hive_compress_type.groovy b/regression-test/suites/external_table_p0/hive/test_hive_compress_type.groovy index 5742ddfd12984a..6dfeb2873c512c 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_compress_type.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_compress_type.groovy @@ -31,7 +31,8 @@ suite("test_hive_compress_type", "p0,external") { sql """drop catalog if exists ${catalog_name}""" sql """create catalog if not exists ${catalog_name} properties ( "type"="hms", - 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}' + 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """use `${catalog_name}`.`multi_catalog`""" diff --git a/regression-test/suites/external_table_p0/hive/test_hive_get_schema_from_table.groovy b/regression-test/suites/external_table_p0/hive/test_hive_get_schema_from_table.groovy index 772cf4d01f2bac..7c63764be7ccf8 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_get_schema_from_table.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_get_schema_from_table.groovy @@ -78,7 +78,8 @@ suite("test_hive_get_schema_from_table", "p0,external") { 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', 'hadoop.username' = 'hive', - 'get_schema_from_table' = 'true' + 'get_schema_from_table' = 'true', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """switch ${catalog_name}""" diff --git a/regression-test/suites/external_table_p0/hive/test_hive_page_index.groovy b/regression-test/suites/external_table_p0/hive/test_hive_page_index.groovy index 113494275622f5..7a28188ffb37e8 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_page_index.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_page_index.groovy @@ -34,7 +34,8 @@ suite("test_hive_page_index", "p0,external") { sql """drop catalog if exists ${catalog_name}""" sql """create catalog if not exists ${catalog_name} properties ( "type"="hms", - 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}' + 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """switch ${catalog_name}""" diff --git a/regression-test/suites/external_table_p0/hive/test_hive_schema_evolution.groovy b/regression-test/suites/external_table_p0/hive/test_hive_schema_evolution.groovy index 746a956f07eee4..0f94daedc48b17 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_schema_evolution.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_schema_evolution.groovy @@ -67,7 +67,8 @@ suite("test_hive_schema_evolution", "p0,external") { sql """drop catalog if exists ${catalog_name}""" sql """create catalog if not exists ${catalog_name} properties ( "type"="hms", - 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}' + 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """switch ${catalog_name}""" sql """use `${catalog_name}`.`default`""" diff --git a/regression-test/suites/external_table_p0/hive/write/test_hive_text_write_insert.groovy b/regression-test/suites/external_table_p0/hive/write/test_hive_text_write_insert.groovy index 4f59b19f231b46..e867e0744361ce 100644 --- a/regression-test/suites/external_table_p0/hive/write/test_hive_text_write_insert.groovy +++ b/regression-test/suites/external_table_p0/hive/write/test_hive_text_write_insert.groovy @@ -893,7 +893,8 @@ suite("test_hive_text_write_insert", "p0,external") { sql """create catalog if not exists ${catalog_name} properties ( 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', - 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}' + 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """use `${catalog_name}`.`write_test`""" logger.info("hive sql: " + """ use `write_test` """) diff --git a/regression-test/suites/external_table_p0/hive/write/test_hive_write_insert.groovy b/regression-test/suites/external_table_p0/hive/write/test_hive_write_insert.groovy index 4942bc2383d715..6f5251afe75972 100644 --- a/regression-test/suites/external_table_p0/hive/write/test_hive_write_insert.groovy +++ b/regression-test/suites/external_table_p0/hive/write/test_hive_write_insert.groovy @@ -892,7 +892,8 @@ INSERT INTO all_types_par_${format_compression}_${catalog_name}_q03 sql """create catalog if not exists ${catalog_name} properties ( 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', - 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}' + 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """use `${catalog_name}`.`write_test`""" logger.info("hive sql: " + """ use `write_test` """) diff --git a/regression-test/suites/external_table_p0/hive/write/test_hive_write_partitions.groovy b/regression-test/suites/external_table_p0/hive/write/test_hive_write_partitions.groovy index 6a2e3fe4a5351f..8dd0cac5cb7d0b 100644 --- a/regression-test/suites/external_table_p0/hive/write/test_hive_write_partitions.groovy +++ b/regression-test/suites/external_table_p0/hive/write/test_hive_write_partitions.groovy @@ -224,7 +224,8 @@ suite("test_hive_write_partitions", "p0,external") { sql """create catalog if not exists ${catalog_name} properties ( 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', - 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}' + 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai' );""" sql """use `${catalog_name}`.`write_test`""" logger.info("hive sql: " + """ use `write_test` """) diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_insert_overwrite.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_insert_overwrite.groovy index 0acc21bc90ac1b..2b3414d633c89c 100644 --- a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_insert_overwrite.groovy +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_insert_overwrite.groovy @@ -831,6 +831,7 @@ suite("test_iceberg_insert_overwrite", "p0,external") { 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai', 'use_meta_cache' = 'true' );""" diff --git a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_insert.groovy b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_insert.groovy index 8b8973e416592e..5aef540d63c02e 100644 --- a/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_insert.groovy +++ b/regression-test/suites/external_table_p0/iceberg/write/test_iceberg_write_insert.groovy @@ -826,6 +826,7 @@ suite("test_iceberg_write_insert", "p0,external") { 'type'='hms', 'hive.metastore.uris' = 'thrift://${externalEnvIp}:${hms_port}', 'fs.defaultFS' = 'hdfs://${externalEnvIp}:${hdfs_port}', + 'hive.parquet.time-zone' = 'Asia/Shanghai', 'use_meta_cache' = 'true' );""" diff --git a/regression-test/suites/external_table_p0/paimon/paimon_timestamp_types.groovy b/regression-test/suites/external_table_p0/paimon/paimon_timestamp_types.groovy index dd3fcaadac2934..b2b0f3f66800d2 100644 --- a/regression-test/suites/external_table_p0/paimon/paimon_timestamp_types.groovy +++ b/regression-test/suites/external_table_p0/paimon/paimon_timestamp_types.groovy @@ -23,6 +23,7 @@ suite("paimon_timestamp_types", "p0,external") { return } + def originalTimeZone = sql("SELECT @@time_zone")[0][0] try { String catalog_name = "paimon_timestamp_types" String minio_port = context.config.otherConfigs.get("iceberg_minio_port") @@ -44,6 +45,8 @@ suite("paimon_timestamp_types", "p0,external") { sql """use flink_paimon;""" logger.info("use test_paimon_db") + // The expected timestamp_ltz values are defined in the Asia/Shanghai timezone. + sql """set time_zone = 'Asia/Shanghai'""" def test_ltz_ntz = { table -> qt_ltz_ntz2 """ select * from ${table} """ @@ -108,10 +111,11 @@ suite("paimon_timestamp_types", "p0,external") { test_ltz_ntz_simple("test_timestamp_ntz_ltz_simple_parquet") } finally { sql """set force_jni_scanner=false""" + sql """set time_zone = '${originalTimeZone}'""" } // TODO: - // 1. Fix: native read + parquet + timestamp(7/8/9) (ts7,ts8,ts9), it will be 8 hour more + // 1. Fix: FileScannerV1 + parquet + timestamp(7/8/9) (ts7,ts8,ts9), it will be 8 hour more // 2. paimon bugs: native read + orc + timestamp_ltz. // In the Shanghai time zone, the read data will be 8 hours less, // because the data written by Flink to the orc file is UTC, but the time zone saved in the orc file is Shanghai. diff --git a/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy b/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy index 531bc0deac22b1..9e95864b123ecc 100644 --- a/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy +++ b/regression-test/suites/external_table_p0/tvf/test_hdfs_parquet_group0.groovy @@ -86,7 +86,8 @@ suite("test_hdfs_parquet_group0", "p0,external") { order_qt_test_8 """ select * from HDFS( "uri" = "${uri}", "hadoop.username" = "${hdfsUserName}", - "format" = "parquet") limit 10; """ + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai") limit 10; """ uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_parquet/group0/delta_encoding_optional_column.parquet" @@ -212,7 +213,8 @@ suite("test_hdfs_parquet_group0", "p0,external") { order_qt_test_26 """ select * from HDFS( "uri" = "${uri}", "hadoop.username" = "${hdfsUserName}", - "format" = "parquet") limit 10; """ + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai") limit 10; """ uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_parquet/group0/dict-page-offset-zero.parquet" @@ -361,14 +363,16 @@ suite("test_hdfs_parquet_group0", "p0,external") { order_qt_test_47 """ select * from HDFS( "uri" = "${uri}", "hadoop.username" = "${hdfsUserName}", - "format" = "parquet") limit 10; """ + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai") limit 10; """ uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_parquet/group0/alltypes_tiny_pages_plain.parquet" order_qt_test_48 """ select * from HDFS( "uri" = "${uri}", "hadoop.username" = "${hdfsUserName}", - "format" = "parquet") limit 10; """ + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai") limit 10; """ uri = "${defaultFS}" + "/user/doris/tvf_data/test_hdfs_parquet/group0/single_nan.parquet" @@ -417,7 +421,8 @@ suite("test_hdfs_parquet_group0", "p0,external") { order_qt_test_55 """ select * from HDFS( "uri" = "${uri}", "hadoop.username" = "${hdfsUserName}", - "format" = "parquet") limit 10; """ + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai") limit 10; """ } finally { } } diff --git a/regression-test/suites/external_table_p0/tvf/test_local_tvf_with_complex_type_element_at.groovy b/regression-test/suites/external_table_p0/tvf/test_local_tvf_with_complex_type_element_at.groovy index 28ddf9c7a51fe5..2ad30283df2ff6 100644 --- a/regression-test/suites/external_table_p0/tvf/test_local_tvf_with_complex_type_element_at.groovy +++ b/regression-test/suites/external_table_p0/tvf/test_local_tvf_with_complex_type_element_at.groovy @@ -95,40 +95,48 @@ suite("test_local_tvf_with_complex_type_element_at", "p0,external") { select * from local( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet"); """ + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai"); """ qt_sql """ select count(*) from local( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet"); """ + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai"); """ qt_sql """ select arr_arr[1][1] from local ( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet");""" + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai");""" qt_sql """ select arr_map[1] from local ( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet");""" + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai");""" qt_sql """ select arr_map[1]["WdTnFb-LHW8Nel-laB-HCQA"] from local ( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet");""" + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai");""" qt_sql """ select map_map["W1iF16-DE1gzJx-avC-Mrf6"]["HJVQSC-46l3xm7-J6c-moIH"] from local ( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet");""" + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai");""" qt_sql """ select map_arr[1] from local ( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet");""" + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai");""" qt_sql """ select map_arr[1][7] from local ( "file_path" = "${outFilePath}/t.parquet", "backend_id" = "${be_id}", - "format" = "parquet");""" + "format" = "parquet", + "hive.parquet.time-zone" = "Asia/Shanghai");""" }