DGNum/demarches-normaliennes
15
1
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity
23210 commits 2 branches 1166 tags 85 MiB
Commit graph

93 commits

Author SHA1 Message Date
mfo
d729c2f193
tech(perf): reschedule this job at 4am when our workers does not do anything 2024-10-10 11:21:05 +02:00
mfo
08ec45443d
BREAKING(sidekiq.queues): rationalize queues. now have critical, default, low 
Dear instances, make sure to update your sidekiq processes. Indead, we are adopting a new strategy for daemons that process our background
jobs.

Now their is only 3 queues on sidekiq (still, we keep archive/export on delayed job for now).

The big idea : allow any worker to deal with any queue (capacity mgmt) :
- critical: must be procesed now
- default: will be processed asap
- low: do it after everything else

FYI: we had 50% of our server capacity on reserved queues. Leading to
bottle neck processing (reserved queues/worker not processing other
queues). We HAD TO fix it. Sorry for the inconvenience

TBH, this is an idea of Colin Darie <colin@darie.eu>. I'm just pushing
it forward.

Co-Authored-By: Colin Darie <colin@darie.eu>
 2024-09-24 21:29:11 +02:00
Colin Darie
77fb0553fc
chore: enable freeze string literals by comment 2024-08-22 09:26:48 +02:00
simon lehericey
7c1b26930b
tech: move cron tab to specific time 2024-06-21 17:15:01 +02:00
simon lehericey
a10a59b265
perf: only load id digest 2024-04-19 08:48:17 +02:00
simon lehericey
8f8b63ac6a
fix: load the dols day by day 2024-04-19 08:48:17 +02:00
simon lehericey
d419508b68
fix: reduce a join query size 2024-04-15 10:34:39 +02:00
simon lehericey
34fae2831f
refactor: add dol_move_batch job which are enqueued by the cron 2024-04-11 10:58:34 +02:00
Colin Darie
dfbe53dcc5
fix(cron): really schedule jobs on sidekiq instead of running them 2024-04-10 18:03:47 +02:00
Colin Darie
25892f594c
fix(jobs): don't enqueue crons to default adapter anymore 2024-04-08 18:27:31 +02:00
LeSim
12406643d9
Merge pull request #10268 from demarches-simplifiees/sidekiq_cron 
Tech: déplace les cron job sur sidekiq
 2024-04-08 08:32:06 +00:00
Paul Chavard
282ce1322c fix(stats): more accurat stats about deleted dossiers 2024-04-04 19:00:48 +02:00
simon lehericey
42af5a9149
make cron_job work with both delayed_job and sidekiq 2024-04-04 16:38:38 +02:00
Martin
be47152096 tech(export_job): sometimes ExportJob are OOMed, in those cases jobs are stuck and never retried. release lock and increase attemps 2024-03-26 15:29:04 +01:00
Colin Darie
0da2843f26
fix(rubocop): Rails/FindEach 2024-02-12 09:33:17 +01:00
simon lehericey
97f335c36a add a cron job to send expiration notice 2024-01-26 10:18:16 +01:00
simon lehericey
016c392914 tech: add env var to disable cron scheduling 2023-12-14 17:12:19 +01:00
simon lehericey
e75d87e76b chore(pjs_migration): remove now obsolete migration_jobs 2023-12-14 14:10:09 +01:00
Martin
fd81936baf correctif(users.expires): maj l'appel a process_expired car le service est instanciable maintenant 2023-11-21 10:18:58 +01:00
Martin
81826e2509 amelioration(Cron::ExpiredUsersDeletionJob): ne retente jamais ce job si il crash [pour eviter de se lancer ds une boucle infinie de suppression] 2023-11-17 11:33:14 +01:00
Martin
1716a931c5 amelioration(Procedure.procedure_expires_when_termine_enabled): active l'expiration des dossiers terminés sur les procedure ne l'ayant pas activé jusqu'à présent. La cible est de faire ça sur 20 Procedure quotidiennement 2023-11-17 11:33:14 +01:00
Martin
fb238ff7ac tech(refactor): deplace l'echelonnement des taches cron liés a l'expiration dans un module dédié 2023-11-17 11:33:14 +01:00
Martin
99529c611d tech(refactor): deplace les service d'expiration usager/dossier dans un namespace dedié 2023-11-17 11:33:14 +01:00
Martin
b989af5b47 amelioration(ExpiredUsersDeletionService): limite le nombre d'envoie d'email sur le job de suppression des utilisateurs 2023-11-17 11:33:14 +01:00
Martin
252b3e4719 tech(expires-user.cron): ajoute une tache cron [débrayable] pour faire tourner la tache de suppression des usagers inutile 
attention, pour notre instance, avec ma db anonymiser :
```
ExpiredUsersDeletionService.find_expired_user.pluck(:id).size
=> 1795515
```
 2023-11-17 11:33:14 +01:00
simon lehericey
ee471614eb tech(pjs): add pjs migration cron 2023-07-12 12:05:25 +02:00
Colin Darie
5db80ee6de
feat(sva): cron accepts dossiers for each sva procedure 2023-07-11 10:01:47 +02:00
Paul Chavard
7136c96a36 feat(procedure): validate external links 2023-07-06 10:27:43 +02:00
Martin
6f2bd8f27c correctif(expiration): re-active l'expiration des dossiers termines 2023-06-29 09:22:48 +02:00
Martin
cc4d124068 tech(expires): pause car on est entrain de trop enqueue de jobs 2023-06-27 16:30:52 +02:00
Martin
a0ceee96bd amelioration(email.resume_hebdomadaire): envoie le mail a 4h du matin sur une periode de 3h 
tech(
Co-authored-by: Colin Darie <colin@darie.eu>
 2023-06-26 17:47:53 +02:00
Martin
d45a250075 amelioration(mail): ajoute d'un simili rate limiter pour envoyer les mails sur des fenetres de temps ayant une limite 2023-06-26 17:28:27 +02:00
Martin
61d620c280 amelioration(expiration.dossiers): evite d'envoyer tous les mails d'un coup. Donc supprime les dossiers en brouillon a 22h, les dossiers en construction a 14h, les dossiers termine a 7h. 2023-06-26 15:51:26 +02:00
Colin Darie
17f5fb4a51
refactor(declarative): process dossier in job isolation 2023-06-14 12:06:22 +02:00
Colin Darie
f7029615a9
refactor(email): spread sending of instructeur digest emails over time 
On envoi parfois plus de 20K mails, ce qui pose des problèmes de délivrance
et quotas si on le fait d'un coup.
On étale donc un peu dans la durée ces envois pour pas limiter le dépassement
de quota.
 2023-05-09 11:32:43 +02:00
Colin Darie
e8e785b953
refactor(rubocop): fix Rails/ToFormattedS 2023-05-03 11:35:30 +02:00
Martin
9dec6f1611 correctif(procedure.declarative): ETQ administrateur d'une procedure declarative, certains de mes dossiers restent en construction [ex: l'object storage est down, le dossier reste bloqué] 2023-04-21 15:45:45 +02:00
Colin Darie
160178b8f8
chore(rubocop): fix Rails/ExpandedDateRange 2023-04-19 12:55:13 +02:00
Colin Darie
6b87c290cb fix(cron): PurgeOldEmailEvent in batches 2023-03-08 15:31:10 +01:00
Colin Darie
43613009d6
fix(job): Cron::WeeklyOverviewJob simplified and respect dolist API rate limiting 2023-03-06 14:39:55 +01:00
Colin Darie
f8f14830a6
fix(missing-antivirus-job): retry when backend is unavailable 2023-03-06 14:39:55 +01:00
Colin Darie
e9585632d9 fix(job): hopefully fix pg timeouts in Cron::PurgeUnattachedBlobsJob 2023-03-02 13:42:04 +01:00
Martin
f4cc8078b4 amelioration(EmailEvent): purge les email_events vieux de plus de un mois 
Update app/models/email_event.rb

Co-authored-by: Colin Darie <colin@darie.eu>

Update app/jobs/cron/purge_old_email_event_job.rb

Co-authored-by: Colin Darie <colin@darie.eu>
 2023-02-06 10:35:57 +01:00
Damien Le Thiec
d5bfe84381
Nettoyage des dossiers préremplis abandonnés (#8341) 
* Clean prefilled dossiers
 2023-01-12 16:14:42 +01:00
Paul Chavard
6e18090fb3 refactor(virus_scan_result): use column instead of metadata on blob 2023-01-04 12:12:11 +01:00
Christophe Robillard
daa4411abe Revert "unschedule export demarches opendata" 
This reverts commit da0738c661.
`ExportAndPublishDemarchesPubliqjesJob` is now schedulable
 2022-12-20 12:52:56 +01:00
Christophe Robillard
672e56a099 schedule job during a no-busy time 2022-12-16 18:02:36 +01:00
Paul Chavard
ed4d5cb36a refactor(demarche): make declarative demarche processing syncroneous 2022-12-06 11:41:25 +01:00
Martin
ea8bd13bcf poc(batch_operation.purge): nettoie les batch terminé ou inutiles 2022-12-06 11:28:48 +01:00
Paul Chavard
b89f6e2d53 feat(operation_log): add Cron::DossierOperationLogMoveToColdStorageJob 2022-11-22 18:06:05 +01:00