Skip to content

Optimise sim2_daily database update

At each startup the database is started from zero which takes 5 minutes and fill tha RAM up to 90 Gb...

radis-api-1  | INFO [2025-06-18 10:38:35] Starting SIM2 daily downloads...
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_1958-1959.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_1960-1969.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_1970-1979.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_1980-1989.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_1990-1999.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_2000-2009.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_2010-2019.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_previous-2020-202505.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Use cached file data/datagouvfr/QUOT_SIM2_latest-20250501-20250617.csv.gz
radis-api-1  | INFO [2025-06-18 10:38:36] Loading data/datagouvfr/QUOT_SIM2_1958-1959.csv.gz...
radis-api-1  | INFO [2025-06-18 10:38:40] Loading data/datagouvfr/QUOT_SIM2_1960-1969.csv.gz...
radis-api-1  | INFO [2025-06-18 10:39:07] Merging QUOT_SIM2_1960-1969.csv.gz...
radis-api-1  | INFO [2025-06-18 10:39:07] Loading data/datagouvfr/QUOT_SIM2_1970-1979.csv.gz...
radis-api-1  | INFO [2025-06-18 10:39:31] Merging QUOT_SIM2_1970-1979.csv.gz...
radis-api-1  | INFO [2025-06-18 10:39:31] Loading data/datagouvfr/QUOT_SIM2_1980-1989.csv.gz...
radis-api-1  | INFO [2025-06-18 10:39:56] Merging QUOT_SIM2_1980-1989.csv.gz...
radis-api-1  | INFO [2025-06-18 10:39:56] Loading data/datagouvfr/QUOT_SIM2_1990-1999.csv.gz...
radis-api-1  | INFO [2025-06-18 10:40:20] Merging QUOT_SIM2_1990-1999.csv.gz...
radis-api-1  | INFO [2025-06-18 10:40:20] Loading data/datagouvfr/QUOT_SIM2_2000-2009.csv.gz...
radis-api-1  | INFO [2025-06-18 10:40:44] Merging QUOT_SIM2_2000-2009.csv.gz...
radis-api-1  | INFO [2025-06-18 10:40:44] Loading data/datagouvfr/QUOT_SIM2_2010-2019.csv.gz...
radis-api-1  | INFO [2025-06-18 10:41:07] Merging QUOT_SIM2_2010-2019.csv.gz...
radis-api-1  | INFO [2025-06-18 10:41:07] Loading data/datagouvfr/QUOT_SIM2_previous-2020-202505.csv.gz...
radis-api-1  | INFO [2025-06-18 10:41:19] Merging QUOT_SIM2_previous-2020-202505.csv.gz...
radis-api-1  | INFO [2025-06-18 10:41:19] Loading data/datagouvfr/QUOT_SIM2_latest-20250501-20250617.csv.gz...
radis-api-1  | INFO [2025-06-18 10:41:19] Merging QUOT_SIM2_latest-20250501-20250617.csv.gz...
radis-api-1  | INFO [2025-06-18 10:41:19] Writing dataset to data/sim2_daily ...
radis-api-1  | INFO [2025-06-18 10:43:02] Dataset data/sim2_daily created.

Instead of merging the whole database in RAM, rows must be merges smartly and only adding one day if the last start has been run for one day.

Edited by David Dorchies