Skip to content

Import executor clean up#1975

Open
vish-cs wants to merge 1 commit intodatacommonsorg:masterfrom
vish-cs:maifest
Open

Import executor clean up#1975
vish-cs wants to merge 1 commit intodatacommonsorg:masterfrom
vish-cs:maifest

Conversation

@vish-cs
Copy link
Copy Markdown
Contributor

@vish-cs vish-cs commented Apr 22, 2026

No description provided.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes redundant logging in the file uploader, updates workflow IDs in logs, and modifies the import helper to use a more specific file pattern for graph paths. Additionally, the manifest configuration was updated to include graph_data_path and remove cron schedules and resource limits. Review feedback highlighted a naming inconsistency between the code and the manifest for the graph data path attribute, as well as a concern regarding the redirection of command output to a log file in the download script, which could hinder debugging.

Comment thread import-automation/workflow/import-helper/main.py
Comment thread scripts/entities/download.sh
Returns:
A ValidationResult object.
"""
if summary is None:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't summary be None for an import that has never been run (or run successfully) where there is no previous successful run to diff against?

'EMPTY_IMPORT_CHECK',
message="Differ summary is missing.")

if 'current_obs_size' not in summary or 'current_schema_size' not in summary:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the validation report.json instead of differ for checking empty import output or even just check the import output files rows for imports that may have disabled differ and validations?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants