Download error fix - CensusCountyBusinessPatterns#1961
Download error fix - CensusCountyBusinessPatterns#1961kartik-s21 wants to merge 1 commit intodatacommonsorg:masterfrom
Conversation
There was a problem hiding this comment.
Code Review
This pull request improves the reliability and security of data ingestion scripts by adding exception handling for network requests, implementing path traversal checks during zip extraction, and filtering for specific file extensions. It also includes minor code cleanups in the energy dataset scripts and test files. Feedback suggests addressing an indentation issue, removing an unused variable, and extending error handling to include corrupted zip files.
saanikaaa
left a comment
There was a problem hiding this comment.
I see un_energy changes in this PR. Pls rectify and make separate PR for seperate imports
d32756f to
9384c28
Compare
9384c28 to
1214c1a
Compare
Rectified to Two separate imports. |
The main.py script for CensusCountyBusinessPatterns was failing during the data download phase.The script calculates the default "data_end_year" as two years prior to the current year (in 2026, it defaults to 2026-2=2024), but the source has not yet published the 2024 dataset. This caused the script to crash with Error 404: File Not Found.
So what Is done -
Modified scripts/census_county_business_patterns/main.py to make the download process more resilient: