DGNum/demarches-normaliennes
15
1
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity
21929 commits 2 branches 1166 tags 85 MiB
Commit graph

333 commits

Author SHA1 Message Date
Colin Darie
d20114f978 chore(sentry): use "procedure" tag everywhere for a better debug experience 2023-03-08 13:48:44 +01:00
Colin Darie
43613009d6
fix(job): Cron::WeeklyOverviewJob simplified and respect dolist API rate limiting 2023-03-06 14:39:55 +01:00
Colin Darie
f8f14830a6
fix(missing-antivirus-job): retry when backend is unavailable 2023-03-06 14:39:55 +01:00
Colin Darie
e9585632d9 fix(job): hopefully fix pg timeouts in Cron::PurgeUnattachedBlobsJob 2023-03-02 13:42:04 +01:00
sebastiencarceles
f52554b5a3 review: update value with async fetch 
1 - spec cover the job which fetches external data

2 - refactor the job with guard clauses

3 - delegate update operation to the champ itself

4 - annuaire education: override the update operation
to let the value be populated by the fetched data

5 - prefilling: don't fetch data synchronously
 2023-02-28 14:53:08 +01:00
Paul Chavard
f6b528e497 chore(geo_area): normalize all geo_areas 2023-02-15 11:58:59 +01:00
sebastiencarceles
f0ffae8320 migrate(champs): normalize departements 2023-02-13 16:32:34 +01:00
Colin Darie
a4329b6699 chore(export/job): set procedure_id for sentry 2023-02-09 11:06:54 +01:00
Colin Darie
964f3829c6 fix(export/job): race condition when export was already generated 2023-02-09 11:06:54 +01:00
Martin
f4cc8078b4 amelioration(EmailEvent): purge les email_events vieux de plus de un mois 
Update app/models/email_event.rb

Co-authored-by: Colin Darie <colin@darie.eu>

Update app/jobs/cron/purge_old_email_event_job.rb

Co-authored-by: Colin Darie <colin@darie.eu>
 2023-02-06 10:35:57 +01:00
Lisa Durand
120fbb0ef7 Overload the finished_at 2023-02-01 17:06:04 +01:00
Lisa Durand
d0a131dc55 change the way we set the finished_at on batch operation job to fix alert display bug 2023-02-01 17:06:04 +01:00
Paul Chavard
dd2d3d4626 fix(task): improuve after party backfill_dossiers_repetitions 2023-02-01 15:27:07 +01:00
Paul Chavard
8bf0d9c5ec fix(dossier): backfill missing repetition champs 2023-02-01 10:36:46 +01:00
Damien Le Thiec
3a8a50a216
Allow prefill pays type de champs (#8344) 
* Allow prefill pays type de champs

* Avoid conditional prefil params for pays champ

* Clean pays data with batch update

* Fix bug and add test batch update pays value

* Improve performance batch_update_pays

* Fix associated country code problem

* Fix after party task name

* Format country name if needed in batch update
 2023-01-18 12:52:38 +01:00
Colin Darie
784b0458fe feat(dolist): consolidate dispatched events with delivered emails 2023-01-18 09:56:40 +01:00
Damien Le Thiec
5e26acb0e1
Make date and datetime prefillable (#8304) 
* Make date and datetime prefillable

* Format in ISO8601 format
 2023-01-12 17:42:02 +01:00
Damien Le Thiec
d5bfe84381
Nettoyage des dossiers préremplis abandonnés (#8341) 
* Clean prefilled dossiers
 2023-01-12 16:14:42 +01:00
Paul Chavard
dfe65136be refactor(repetition): add row_id to champs 2023-01-05 18:04:54 +01:00
Martin
018a94f43b correctif(batch_operation): empeche de changer l'etat d'un dossier en cas d'erreur. et s'assure de delinker le dossier en cas de dossier invalid 2023-01-05 16:20:18 +01:00
Paul Chavard
6e18090fb3 refactor(virus_scan_result): use column instead of metadata on blob 2023-01-04 12:12:11 +01:00
Paul Chavard
d469bca0ae refactor(watermark): use column instead of metadata on blob 2023-01-04 12:07:08 +01:00
Christophe Robillard
edffd0a7a8 extract ds_opendata_enabled 2022-12-20 12:52:56 +01:00
Christophe Robillard
daa4411abe Revert "unschedule export demarches opendata" 
This reverts commit da0738c661.
`ExportAndPublishDemarchesPubliqjesJob` is now schedulable
 2022-12-20 12:52:56 +01:00
Christophe Robillard
672e56a099 schedule job during a no-busy time 2022-12-16 18:02:36 +01:00
Paul Chavard
ed4d5cb36a refactor(demarche): make declarative demarche processing syncroneous 2022-12-06 11:41:25 +01:00
Martin
ea8bd13bcf poc(batch_operation.purge): nettoie les batch terminé ou inutiles 2022-12-06 11:28:48 +01:00
Martin
eaf72162da poc(batch_operation): some rewrite to avoid various conflict (when an instructeur try to create a job with an incompatible dossier regarding the current task). also soome cleanup to isole spec in least involved model 2022-12-06 11:28:48 +01:00
Martin
4266ab93c5 poc(batch_operation): track dossier processed safely 2022-12-06 11:28:48 +01:00
Martin
beb39027d0 poc(batch_operation.ui): implement simple ui to trigger a batch of current page 2022-12-06 11:28:48 +01:00
Martin
7df86c50fb poc(batch_operations_controller): implement simple [not yet with procedure_presentation] action to create a batch operation 2022-12-06 11:28:48 +01:00
Martin
b11dbafc59 poc(batch_operation_job): implement archive dossiers with async behaviour, lock, run_at, finished_at, failed_dossier_ids, success_dossier_ids 2022-12-06 11:28:48 +01:00
Colin Darie
b13c5e56f6 refactor(piece_justificative): UX follows mockups 2022-12-05 10:47:10 +01:00
Paul Chavard
b89f6e2d53 feat(operation_log): add Cron::DossierOperationLogMoveToColdStorageJob 2022-11-22 18:06:05 +01:00
Kara Diaby
4d6ff768c4 Ajoute les dossiers déposés dans les statistiques Datagouv 2022-11-03 10:24:04 +01:00
Paul Chavard
1fd1878370 feat(jobs): add request_id to jobs payload 2022-10-12 14:32:23 +02:00
Damien Le Thiec
0131a41266
Merge branch 'main' into feature/add_rna_type_de_champs 2022-10-05 12:38:24 +02:00
Damien Le Thiec
0d9f02c88d First draft autofetch data for RNA 2022-10-04 22:11:01 +02:00
Christophe Robillard
da0738c661 unschedule export demarches opendata 2022-09-30 07:26:48 +02:00
Martin
d9432ccd3b fix(etablissement.as_degraded_mode): backfill missing data via a cron 2022-09-23 15:12:25 +02:00
Colin Darie
67cbc61e1e chore(services): update etablissement adresse & geocode when siret changes 2022-09-06 09:23:52 +02:00
Martin
45e0c15eb0 feat(PurgeManagerAdministrateurSessionsJob): also purge instructeur assign_tos 2022-08-23 13:28:10 +02:00
Kara Diaby
ec3118e82c Add concern for Datagouv schedulable jobs 2022-07-29 12:25:27 +02:00
Kara Diaby
a156529394 modify export descriptif demarches job 2022-07-29 11:17:22 +02:00
Kara Diaby
882ee74f88 Comptes créés avec FC par mois : CRON Job 2022-07-29 11:17:22 +02:00
Kara Diaby
dc1d0ce071 Démarches supprimées par mois : CRON Job 2022-07-29 11:17:22 +02:00
Kara Diaby
4a29162c6c Démarches closes par mois : CRON Job 2022-07-29 11:17:22 +02:00
Kara Diaby
999059269e Instructeurs connectés par mois : CRON Job 2022-07-29 11:17:21 +02:00
Kara Diaby
8e9d1eeb6f Nombre de comptes créés par mois : CRON 2022-07-29 11:17:21 +02:00
Kara Diaby
b48a48980a Nombre de dossiers créés par mois : CRON 2022-07-29 11:17:21 +02:00
Kara Diaby
9fa54c3cc8 Nombre de démarches créés par mois : CRON 2022-07-29 11:17:21 +02:00
Kara Diaby
e0dc36926e Nombre d'instructeurs créés par mois : CRON 2022-07-29 11:17:21 +02:00
Kara Diaby
a0a3c6c0ee Administrateurs crées par mois : CRON Job 2022-07-29 11:17:20 +02:00
Colin Darie
197fa54ea4 fix(archives): purge stuck pending archives 
Cf 0cb096b3d6440a4da05ca2902ed6fa72a93e6579
 2022-07-27 15:26:59 +02:00
Colin Darie
f1957e51f8 fix(exports): purge stuck (pending) exports, not generated or failed 
Avec l'autre PR, ça reproduit le comportement d'avant, c'est à dire
que quoiqu'il arrive un export est purgé :
- soit 16h après sa génération (on a 16h pour le télécharger)
- soit 12h après sa création, et qu'il est bloqué

Auparavant, tous les exports étaient purgés au bout de 3h quelle que soit
le statut.
 2022-07-27 15:26:59 +02:00
Martin
3e56fdd1d7 fix(export): when it takes more than 3 hours, exports are purge before being generated. make it possible to have an export that takes more than 3 hours and share this behaviour with kind of same class archive 2022-07-25 10:48:54 +02:00
Christophe Robillard
906dd4b509 remove useless DATAGOUV_TMP_DIR env 2022-07-21 21:54:15 +02:00
Christophe Robillard
0e3e6eb62a add schedulable? class method to cron jobs 2022-07-21 21:54:15 +02:00
Christophe Robillard
bed6167010 use specific namespace for datagouv cron jobs 2022-07-21 21:54:15 +02:00
Christophe Robillard
b6c96301ef export and publish opendata demarches 2022-07-21 21:54:15 +02:00
Martin
7fb85c97cb feat(Administrateur::ArchivesControllers): add archives #index and #create for admin 2022-07-04 14:18:46 +02:00
mfo
011fcd40d8
Merge pull request #7457 from betagouv/US/fix-webhook 
fix(WebHookJob): missing find procedure
 2022-06-10 17:30:28 +02:00
Martin
165db34c4a fix(WebHookJob): missing find procedure 2022-06-10 17:28:07 +02:00
Martin
b84c2966c4 fix(DeclarativeProceduresJob.perform_now): avoid to crash other demarche when one fail 2022-06-10 14:59:36 +02:00
Martin
d39b3b09ed fix(webhook.perform): ensure to enqueue job with current state/updated_at for consistency reason 2022-06-01 15:54:55 +02:00
Christophe Robillard
f14cd7b022 remove warning mail for dubious procedures 2022-05-25 14:44:53 +02:00
Paul Chavard
9e0b3b642f cleanup(sendinblue): remove sendinblue tracking 2022-05-06 11:14:44 +02:00
simon lehericey
da97ad1858 destroy all admins 2022-05-03 12:14:02 +02:00
Martin
3c0e3f725d feat(administrateur/procedures#publication): show dubious to administrateur 
Update app/views/administrateurs/procedures/publication.html.haml

Co-authored-by: Paul Chavard <paul.chavard@beta.gouv.fr>
 2022-04-27 15:13:22 +02:00
Martin
dbcf21a555 feat(archive): extract archive status management within job to simplify the main service as well as to isolate this part for a merge with exports csv/xslx [maybe?] 
Update app/dashboards/archive_dashboard.rb

Co-authored-by: LeSim <mail@simon.lehericey.net>
 2022-03-31 13:35:49 +02:00
LeSim
fdc4e5bfbc
Merge pull request #6928 from adullact/feature/6886-watermark-url 
Watermark depuis une URL distante
 2022-02-08 11:45:26 +01:00
Paul Chavard
0268511264 fix(manager): remove admins added for a day 
fix #6855
 2022-02-08 11:18:44 +01:00
Jon
88d3a6e4c3
feat: can have remote watemark file 2022-02-04 18:55:16 +01:00
Paul Chavard
403e71b41f feat(procedure): only purge procedures without dossiers 2022-01-18 11:22:04 +01:00
simon lehericey
f0b0e7fd9a Switch to usage of zip unix binary to create archive. Also use a dedicated queue for DelayedJob 
use dedicated archives queue

As the used disk space will increase, we want a fined grain control

move zip logic in dedicated method

zip

wip

wip

fix(spec): pass spec in green

tech(improvements): avoid File.delete(folder), favor FileUtils.remove_entry_secure which is safer. Also wrap most of code that open file within blocks so it is cleaned when the block ends. Lastly use  attachement.download to avoid big memory pressure [download in chunk, write in chunk] otherwise big file [124>1GO] are loaded in memory. what if we run multiple jobs/download in parallel ?

fix(spec): try to retry with grace

clean(procedure_archive_service_spec.rb): better retry [avoid to rewrite on open file]

lint(things): everything
 2021-12-13 16:37:04 +01:00
kara Diaby
9ee9389ba1 add the good retry_on StaleObjectError 2021-11-05 13:32:21 +01:00
Paul Chavard
d5d3995dad fix(dossier): fix dossier.avis cascade 2021-10-26 11:16:29 +02:00
Paul Chavard
1561ea82f6 fix(transfer): manually nullify staled transfers references 2021-10-21 13:54:26 +02:00
Paul Chavard
2a3a9dd822 feat(revisions): rebase dossiers brouillons 2021-10-20 17:26:09 +02:00
Paul Chavard
4a947f9135 feat(manager): add become administrateur button in manager (with 24h expiration) 2021-10-07 11:32:12 +02:00
simon lehericey
e4484eb6d9 discard expert not found job error 2021-10-04 12:57:37 +02:00
Paul Chavard
5985755229 feat(dossier): add purge stale dossier transfers job 2021-09-08 14:39:46 +02:00
Christophe Robillard
ddea0b9a45 purge old sendinblue mails 2021-08-20 08:52:39 +02:00
Paul Chavard
ffa8c0c80a feat(dossiers): enable dossiers termine expiration behind feature flag 
feature flag "procedure_process_expired_dossiers_termine" controls if a procedure has expiration
enabled on dossiers termine

re #3796
 2021-08-18 16:11:35 +01:00
Christophe Robillard
66cc0dd08d Revert "Revert "Expose dossier PDF export as IO"" 
This reverts commit 362093eff0.
 2021-06-24 21:01:59 +02:00
krichtof
362093eff0
Revert "Expose dossier PDF export as IO" 2021-06-24 19:21:37 +02:00
Paul Chavard
48dca8bad1 Execute AchiveCreationJob in exports queue 
Co-authored-by: Christophe Robillard <christophe.robillard@beta.gouv.fr>
 2021-06-24 17:49:58 +02:00
Christophe Robillard
f40d96fbd2 Revert "Revert "Export de tous les dossier d'une démarche"" 
This reverts commit d9a588b52e.
 2021-04-29 17:29:47 +02:00
krichtof
d9a588b52e
Revert "Export de tous les dossier d'une démarche" 2021-04-29 16:07:18 +02:00
Pierre de La Morinerie
684af77e35 jobs: extract an RetryOnTranscientErrors concern 2021-04-29 14:08:12 +02:00
Christophe Robillard
aa407d5607 place le cronjob PurgeStaleArchives dans le bon répertoire 2021-04-28 11:40:58 +02:00
Christophe Robillard
8bee53fe77 rend plus robuste la création de l'archive 2021-04-28 11:40:58 +02:00
Christophe Robillard
609a1aa45b add archive creation and purge archives jobs 2021-04-28 11:40:58 +02:00
simon lehericey
e636e3a752 add merge_and_update_metadata method 2021-04-06 18:11:26 +02:00
simon lehericey
d552e364fc retry 5 times on integrity error and then block processing 2021-04-06 18:03:39 +02:00
Pierre de La Morinerie
75a1046315 active_storage: refactor concerns 
Follow-up of #5953.

Refactor the concerns with two goals:

- Getting closer from the way ActiveStorage adds its own hooks.
  Usually ActiveStorage does this using an `Attachment#after_create`
  hook, which then delegates to the blob to enqueue the job.
- Enqueuing each job only once. By hooking on `Attachment#after_create`,
  we guarantee each job will be added only once.

We then let the jobs themselves check if they are relevant or not, and
retry or discard themselves if necessary.

We also need to update the tests a bit, because Rails'
`perform_enqueued_jobs(&block)` test helper doesn't honor the `retry_on`
clause of jobs. Instead it forwards the exception to the caller – which
makes the test fail.

Instead we use the inline version of `perform_enqueued_jobs()`, without
a block, which properly ignores errors catched by retry_on.
 2021-03-16 11:49:14 +01:00
Pierre de La Morinerie
08094e5384 jobs: retry on ActiveStorage::IntegrityError during virus scan 
This is the same setting than what ActiveStorage::AnalyzeJob uses.
 2021-03-16 10:52:12 +01:00
Paul Chavard
d24ee27cac Try to reduce the number of external data fetches 
Only fetch external data if the external_id is the current one and data is not yet fetched
 2021-03-11 14:56:37 +01:00
Paul Chavard
21d5da379f remove unused job 2021-03-11 14:56:37 +01:00
Paul Chavard
7eee14feef Add ChampFetchExternalDataJob 2021-02-17 19:10:16 +01:00
Pierre de La Morinerie
150ddab660 zeitwerk: Api -> API 2021-02-09 13:07:30 +01:00
Paul Chavard
9156754cf6 After maximum ammount of retries discard the job and log exception 2021-02-05 18:56:27 +01:00
Paul Chavard
41c3a98d7d Update Raven references to use Sentry 2021-01-28 19:46:36 +01:00
Paul Chavard
0633d43868 Stop crashing if attachment can not be watermarked and mark blob as invalid 2021-01-27 16:49:09 +01:00
simon lehericey
650a598698 ensure a buggy procedure does not block the others from being closed 2021-01-19 10:42:16 +01:00
Paul Chavard
97ce8f312b Use external_id to fetch annuaire_education data 2021-01-15 11:50:02 +01:00
Paul Chavard
5c9f2e8783 Add api education adapter and job 2021-01-14 17:57:48 +01:00
simon lehericey
1348b883e3 Add cron to complete missing analysis 2020-12-17 11:05:33 +01:00
kara Diaby
f3c4040d2d add grace period to deleted dossiers 2020-12-16 15:00:26 +01:00
Paul Chavard
3c6e30a326 Show titre identite link only once watermark is added 2020-12-10 16:50:03 +01:00
clemkeirua
513d4f6ff1 move all the cron jobs in a dedicated directory 2020-12-08 13:22:43 +00:00
simon lehericey
b187244a29 extract and refactor api errors 2020-12-04 17:07:30 +01:00
simon lehericey
46c355beb2 Add comments 2020-12-03 11:52:01 +01:00
simon lehericey
f56235c7ee retry exponentially on api entreprise timeout 2020-12-03 11:52:01 +01:00
simon lehericey
9eab310565 [#5675] for all api entreprise job, retry on 502, 503 2020-12-02 18:25:31 +01:00
Paul Chavard
18dfacb933 Make watermark configurable 2020-11-25 16:19:06 +01:00
Paul Chavard
371179dc5b Watermark titres identite 2020-11-25 16:19:06 +01:00
simon lehericey
c563956a9f retry each day during 5 days to fetch attestation social 2020-10-28 15:53:58 +00:00
simon lehericey
ad01d4dee5 really faster stats by precomputing some of them 2020-10-06 16:12:01 +02:00
clemkeirua
245e9e59c7 do not run ApiEntreprise jobs on missing etablissements 2020-09-28 17:36:38 +02:00
Paul Chavard
dea78c49f6 Run jobs in named queues 2020-09-23 08:22:17 +02:00
simon lehericey
5459c2fa7f Fix uninitialized excon constant 2020-09-08 18:36:24 +02:00
Pierre de La Morinerie
292dc84eb4 jobs: ignore deleted file in Virus Scan 
We have errors in production where the job starts correctly (i.e. the
blob exists), but `blob.open` fails with a `ActiveStorage::FileNotFound`
error.

When checking later in production, the blob has been deleted.

This points to the blob (and the file) being deleted during the virus
scan job.

In that case, ignore the error (rather than retrying the job).
 2020-09-03 11:00:59 +02:00
simon lehericey
49aa426d1b application job swallow BadRequest error and retry 2020-09-02 16:59:54 +02:00
Paul Chavard
5ceae8235b Ignore type_de_champ procedure_id 2020-09-02 11:26:26 +02:00
Paul Chavard
ec72fdd164 Remove migration service 2020-09-02 11:26:26 +02:00
Pierre de La Morinerie
3c91cfc83c jobs: add health-related keywords to dubious procedures scanner 
We are not certified for hosting health-related data yet.
 2020-08-25 12:04:10 +00:00
Christophe Robillard
f2dcb61521 transform job in task for set dossiers_last_updated_at 2020-08-12 16:10:15 +02:00
Pierre de La Morinerie
54a4db1c47 config: flip config.active_storage.queues 
ActiveStorage jobs are now moved to their own queue.

For consistency, we also move our own analysis jobs (VirusScannerJob)
on the same `:active_storage_analysis` queue.
 2020-08-05 16:48:36 +02:00
clemkeirua
952a43058f last_commentaire_updated_at does not take system email into account 2020-07-30 11:52:41 +02:00
clemkeirua
153c6aebd7 add temp job to set last_updated_at_* values for dossier 
Co-authored-by: Christophe Robillard <christophe.robillard@beta.gouv.fr>
 2020-07-30 10:31:14 +02:00
Paul Chavard
bb5a90da76 fix active_revision_id -> active_revision.id 2020-07-23 15:07:25 +02:00
Paul Chavard
7d421d471c Fix revision migration job 2020-07-23 09:56:34 +02:00
Paul Chavard
bc09b61684 Add migration service and task 2020-07-21 19:35:30 +02:00
Pierre de La Morinerie
7a67f1a802 jobs: ignore missing blob during virus scan 
We currently have many failed VirusScannerJob enqueued, because the
underlying blob is missing.

This PR fixes the issue by discarding the job in those cases (because if
the blob is gone, the job is never going to succeed).

The implementation is based on a similar issue encoutered by the
ActiveStorage::AnalyzeJob. See 06f8baf73c
 2020-07-13 14:35:41 +02:00
Christophe Robillard
d20d91bc07 no sentry reporting for api-entreprise 2020-07-08 17:58:33 +02:00
Christophe Robillard
954d41ce4e fetch effectifs mensuels for may month 2020-06-24 10:23:00 +02:00
Paul Chavard
e89f0846bb Fix job max attempts 2020-06-16 15:57:18 +02:00
Christophe Robillard
f8b170836c report to sentry for BadFormatRequest and ResourceNotFound exceptions 2020-06-11 23:09:50 +02:00
Christophe Robillard
b90513f456 no sentry report when error 400 for exercices 2020-06-11 18:16:19 +02:00
Paul Chavard
c4896b45ad Prepare to remove dossier procedure_id 2020-06-11 16:13:44 +02:00
Christophe Robillard
8df9ae3a8d ask only effectifs for april 2020-06-10 17:41:13 +02:00
Christophe Robillard
6535324ae9 ask effectifs only for 2020/02 2020-06-04 09:18:01 +02:00
Christophe Robillard
705487dab6 define max_attempts for jobs 2020-06-02 20:21:11 +02:00
Christophe Robillard
8c66b2cab1 rescue ResourceNotFound and BadFormatRequest errors 
we don't have to retry this kind of errors, but just to send it to
sentry
 2020-05-20 11:09:54 +02:00
Christophe Robillard
f4ebb5d107 create api_entreprise jobs 2020-05-20 11:09:54 +02:00
Paul Chavard
fcd136c566 Enable deletion of expired dossiers en_construction 
fix #4977
 2020-04-21 10:41:54 +02:00
Paul Chavard
61eaca928f remove WarnExpiringDossiersJob 2020-04-09 11:11:51 +02:00
Paul Chavard
6dc1f3e2c4 Use human readable expressions instead of cron gibberish 
https://github.com/floraison/fugit#fugitnat
 2020-04-07 12:44:58 +02:00
Paul Chavard
58c126308c Add DiscardedProceduresDeletionJob job 2020-03-31 17:08:28 +02:00