test: unignore DynamicPartitionPruning static scan metrics test#4038
Merged
andygrove merged 1 commit intoapache:mainfrom Apr 22, 2026
Merged
Conversation
DPP support for native_datafusion scan was added in apache#4011. Update the Spark SQL test diffs so the test runs under native_datafusion by removing the IgnoreCometNativeDataFusion tag and adding a CometNativeScanExec case to getFactScan.
andygrove
force-pushed
the
unignore-dpp-static-scan-metrics-3442
branch
from
6aa57ec to
f55095a
Compare
mbutrovich
approved these changes
Apr 22, 2026
Contributor
mbutrovich
left a comment
mbutrovich
left a comment
There was a problem hiding this comment.
Approved pending CI. Sorry I missed this one. Good catch @andygrove!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #3442.
Rationale for this change
Non-AQE DPP support for
CometNativeScanExecwas added in #4011. TheDynamicPartitionPruningV1Suite."static scan metrics"test had been taggedIgnoreCometNativeDataFusion("...issues/3442")and was left ignored by that PR because the test'sgetFactScanhelper only matchedFileSourceScanExec/BatchScanExec/CometScanExec, notCometNativeScanExec. With the native_datafusion DPP path in place, adding aCometNativeScanExeccase lets the test run.What changes are included in this PR?
For the Spark 3.4.3, 3.5.8, and 4.0.1 test diffs:
IgnoreCometNativeDataFusion("...issues/3442")from the"static scan metrics"test.CometNativeScanExeccase to thefind(plan)block ingetFactScan.How are these changes tested?
Ran the test locally against Spark 3.5.8 with Comet enabled and
SCAN_AUTO(resolves to native_datafusion):The AEOn variant is ignored by Spark's own
DisableAdaptiveExecutiontag, not Comet. Spark SQL CI will cover 3.4.3 and 4.0.1.