feat(datafusion): add AsyncCatalogProvider for dynamic catalog support#2174
feat(datafusion): add AsyncCatalogProvider for dynamic catalog support#2174Shekharrajak wants to merge 1 commit intoapache:mainfrom
Conversation
Shekharrajak
changed the title
blackmwk
left a comment
blackmwk
left a comment
There was a problem hiding this comment.
Thanks @Shekharrajak for this pr. Async catalog may not be the right direction, since its usage requires extra setup in datafusion. What I was thinking was to have a scheduled background task to refresh it.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
@blackmwk, so how the API will look like ? Can you please share ? |
2 participants
Which issue does this PR close?
Closes #1163
What changes are included in this PR?
Add IcebergAsyncCatalogProvider and IcebergAsyncSchemaProvider implementing DataFusion's async catalog traits
Fetch table/schema metadata on-demand instead of capturing snapshot at creation time
Are these changes tested?
new unit tests covering