Skip to content

test: add non-AQE DPP edge case coverage for native Parquet scans#4037

Merged
mbutrovich merged 1 commit intoapache:mainfrom
mbutrovich:more_non_aqe_dpp_tests
Apr 23, 2026
Merged

test: add non-AQE DPP edge case coverage for native Parquet scans#4037
mbutrovich merged 1 commit intoapache:mainfrom
mbutrovich:more_non_aqe_dpp_tests

Conversation

@mbutrovich
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Related to #4033. Improves test coverage for #4011.

Rationale for this change

While working on AQE DPP broadcast reuse for Iceberg (#4033), we identified several non-AQE DPP scenarios that #4011 supports but doesn't explicitly test: multiple broadcast joins, broadcast exchange fallback, empty DPP results, and dual partition filter resolution. Adding these tests strengthens confidence in #4011's implementation and documents edge cases for future work on V1 Parquet AQE DPP (#3510).

What changes are included in this PR?

Four new non-AQE DPP tests in CometExecSuite:

  • non-AQE DPP with two separate broadcast joins: Fact table partitioned by two columns, each joined to a different dimension table. Verifies convertSubqueryBroadcasts handles multiple independent DPP filters.
  • non-AQE DPP fallback when broadcast exchange is not Comet: Disables Comet BHJ/broadcast exchange so convertSubqueryBroadcasts encounters a Spark BroadcastExchangeExec and correctly skips conversion. Verifies correct results via Spark's standard subquery path.
  • non-AQE DPP with empty broadcast result: Dimension filter matches no rows, DPP prunes all partitions. Verifies empty result without errors.
  • non-AQE DPP resolves both outer and inner partition filters: CometNativeScanExec.partitionFilters and CometScanExec.partitionFilters contain different InSubqueryExec instances. Both must be resolved for partition selection to work. Verifies correct results, which requires both filter sets to be resolved.

How are these changes tested?

The PR is entirely tests. All four tests use AQE=false to exercise #4011's non-AQE DPP code paths (PlanDynamicPruningFilters, convertSubqueryBroadcasts, ReuseExchangeAndSubquery).

@mbutrovich mbutrovich marked this pull request as ready for review April 22, 2026 18:28
Copy link
Copy Markdown
Member

@andygrove andygrove left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding more tests @mbutrovich

@mbutrovich mbutrovich merged commit 33f2764 into apache:main Apr 23, 2026
74 checks passed
@mbutrovich mbutrovich deleted the more_non_aqe_dpp_tests branch April 23, 2026 01:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants