Skip to content

[CALCITE-7440] Preserve correlated scope mapping in RelToSqlConverter#4828

Open
bvolpato wants to merge 4 commits intoapache:mainfrom
bvolpato:bvolpato/calcite-7440-rel2sql-correl-scope
Open

[CALCITE-7440] Preserve correlated scope mapping in RelToSqlConverter#4828
bvolpato wants to merge 4 commits intoapache:mainfrom
bvolpato:bvolpato/calcite-7440-rel2sql-correl-scope

Conversation

@bvolpato
Copy link

@bvolpato bvolpato commented Mar 9, 2026

Summary

RelToSqlConverter can lose correlation context during PostgreSQL round-trip conversion after the semi-join rewrite pipeline, causing variable $cor1 is not found.

Reproduction query

WITH product_keys AS (
  SELECT p."product_id",
         (SELECT MAX(p3."product_id")
          FROM "foodmart"."product" p3
          WHERE p3."product_id" = p."product_id") AS "mx"
  FROM "foodmart"."product" p
)
SELECT DISTINCT pk."product_id"
FROM product_keys pk
LEFT JOIN "foodmart"."product" p2 USING ("product_id")
WHERE pk."product_id" IN (
  SELECT p4."product_id"
  FROM "foodmart"."product" p4
)

Reproducer rules

RuleSets.ofList(
    CoreRules.FILTER_SUB_QUERY_TO_MARK_CORRELATE,
    CoreRules.PROJECT_SUB_QUERY_TO_MARK_CORRELATE,
    CoreRules.MARK_TO_SEMI_OR_ANTI_JOIN_RULE,
    CoreRules.SEMI_JOIN_JOIN_TRANSPOSE)

Error before this PR

java.lang.NullPointerException: variable $cor1 is not found
  at org.apache.calcite.rel.rel2sql.SqlImplementor.getAliasContext(SqlImplementor.java:1590)

Fix

  • Preserve correlation mappings when visiting Aggregate inputs during RelToSql conversion.
  • Keep the alias-context fallback scoped to the empty-map case only.

Test

  • Added RelToSqlConverterTest.testPostgresqlRoundTripCorrelatedProjectWithSemiJoinRules
  • Verified the retained regression fails on main and passes with this patch
  • Ran:
    JAVA_HOME=$(/usr/libexec/java_home -v 17) ./gradlew :core:test --tests org.apache.calcite.rel.rel2sql.RelToSqlConverterTest.testPostgresqlRoundTripCorrelatedProjectWithSemiJoinRules

Disclaimer

The changes were done with the assistance of Codex (gpt-5.3-codex) in response to an actual bug when using Calcite in production. The changes were manually verified prior to sending this to review.

@bvolpato bvolpato force-pushed the bvolpato/calcite-7440-rel2sql-correl-scope branch 3 times, most recently from 6309f8d to 5447f29 Compare March 10, 2026 04:40
@bvolpato bvolpato marked this pull request as ready for review March 10, 2026 04:49
@bvolpato bvolpato marked this pull request as draft March 10, 2026 12:39
@bvolpato bvolpato force-pushed the bvolpato/calcite-7440-rel2sql-correl-scope branch 2 times, most recently from 4449f5a to f11eed6 Compare March 10, 2026 13:01
@bvolpato bvolpato marked this pull request as ready for review March 11, 2026 02:15
@bvolpato
Copy link
Author

bvolpato commented Mar 13, 2026

Thanks for the review! Updated this PR with both requested changes:

Reduced the regression to the minimal reproducer on main (empty rules was a mistake):

  • FILTER_SUB_QUERY_TO_MARK_CORRELATE
  • PROJECT_SUB_QUERY_TO_MARK_CORRELATE
  • MARK_TO_SEMI_OR_ANTI_JOIN_RULE
  • SEMI_JOIN_JOIN_TRANSPOSE

Added a plain round-trip test without extra rules:

  • testPostgresqlRoundTripCorrelatedProject

Both new/updated tests pass locally in this branch.

@bvolpato bvolpato force-pushed the bvolpato/calcite-7440-rel2sql-correl-scope branch from 6365d37 to 565cb6a Compare March 13, 2026 12:10
@bvolpato bvolpato marked this pull request as draft March 13, 2026 12:32
@bvolpato bvolpato force-pushed the bvolpato/calcite-7440-rel2sql-correl-scope branch from 565cb6a to a685206 Compare March 13, 2026 12:50
@bvolpato bvolpato marked this pull request as ready for review March 16, 2026 21:57
@xiedeyantu
Copy link
Member

Could we please consider @asolimando 's suggestion in Jira? Personally, I don't think this particular fix is ​​entirely appropriate. Incidentally, regarding the structure of the test case: you might want to refer to other examples for guidance. Please avoid using try-catch blocks; if an error occurs, it should be allowed to surface naturally.

@bvolpato
Copy link
Author

@xiedeyantu I addressed the test-structure part of your note:

  • removed the try-catch from testPostgresqlRoundTripCorrelatedProjectWithSemiJoinRules
  • updated the PR description so it matches the retained reproducer and current fix

On the Jira alias suggestion: I considered it, but I kept this PR scoped to the failure mode reproduced here. The retained regression still fails on main and passes with this patch, while changing alias generation in visit(Correlate) would be a broader behavior change than the NPE fix in this PR.

If you want, I can also split out the alias-cleanup direction as a separate follow-up.

@sonarqubecloud
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants