You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
When the opendata.scot_pipeline is run, the /data folder in the_od_bods repo isn't replaced. This means the data/ folder is completely out of date unless someone manually replaces it.
This causes issues if publishers or sources are removed from the pipeline refresh, but the original aged data files remain in the folder. It also causes compatibility issues on contributors' local machines, especially if there are structural changes to data files expected downstream.
Describe the solution you'd like
Add a clean up step to the pipeline to nuke all existing csvs before we run all the scripts
Start committing to the od bods repo as well on the weekly sync to keep it up to date
Describe alternatives you've considered
Start writing all the csvs to a temp directory that goes away after pipeline run
Additional context
Rejected writing to temp directory because it’s a significant effort, don’t think it’s worth it if we have our eyes on issue OpenDataScotland/the_od_bods#163 already
Is your feature request related to a problem? Please describe.
When the opendata.scot_pipeline is run, the /data folder in the_od_bods repo isn't replaced. This means the data/ folder is completely out of date unless someone manually replaces it.
This causes issues if publishers or sources are removed from the pipeline refresh, but the original aged data files remain in the folder. It also causes compatibility issues on contributors' local machines, especially if there are structural changes to data files expected downstream.
Describe the solution you'd like
Describe alternatives you've considered
Additional context
Rejected writing to temp directory because it’s a significant effort, don’t think it’s worth it if we have our eyes on issue OpenDataScotland/the_od_bods#163 already
Originally discussed on slack 21 Oct 2022: https://opendatascotland.slack.com/archives/C02HEHDL8AY/p1666356294060289
The text was updated successfully, but these errors were encountered: