Problem
The focus of this project is on handling data in a way that is reproducible and update-able.
Description
- How often should the data be gathered and analyzed/rendered?
- What is the strategy for gathering data over multiple days (due to query limits)?
- What is the strategy for ensuring automated updates do not result in broken/incomplete state if they don't complete successfully?
- Should scripts wait until completion to write data to file(s)?
- Can the various tasks be run using GitHub Actions?
- How should the gathered data be stored and formatted?
- What naming conventions and purge/overwrite rules should be used to facilitate multi-day data gathering?
- Should data be stored in a plaintext (immediately readable) or binary format (compressed, SQLite, etc.)?
- Are there opportunities for code deduplication?
Alternatives
Do everything manually 😩
Additional context
(Suggestions welcomed! Please comment if you have a relevant links to share.)
Problem
The focus of this project is on handling data in a way that is reproducible and update-able.
Description
Alternatives
Do everything manually 😩
Additional context
(Suggestions welcomed! Please comment if you have a relevant links to share.)