Christophe Robillard
f04e121a6f
add extrait_kbis to handle nom_commercial and capital_social
2023-05-26 13:37:20 +02:00
Christophe Robillard
ca922005f9
add admin update default zones job
2023-05-16 09:40:53 +02:00
Colin Darie
79f450a422
Merge pull request #9021 from colinux/feat-en-construction-fork
...
ETQ usager, je modifie et soumets à nouveau mon dossier “en construction"
2023-05-10 19:46:40 +00:00
Colin Darie
f6ebfe0e47
fix(dossier): sync search_terms after merge
...
Use a job instead of a lot of reloads
2023-05-09 18:49:52 +02:00
Paul Chavard
025bd5beaf
feat(dossier): dossier can be forked for editing
2023-05-09 15:34:10 +02:00
Colin Darie
f7029615a9
refactor(email): spread sending of instructeur digest emails over time
...
On envoi parfois plus de 20K mails, ce qui pose des problèmes de délivrance
et quotas si on le fait d'un coup.
On étale donc un peu dans la durée ces envois pour pas limiter le dépassement
de quota.
2023-05-09 11:32:43 +02:00
Colin Darie
e8e785b953
refactor(rubocop): fix Rails/ToFormattedS
2023-05-03 11:35:30 +02:00
Paul Chavard
1d10e59e63
fix(commune): fix ‘undefined’ departement codes
2023-04-27 12:07:43 +02:00
Martin
9dec6f1611
correctif(procedure.declarative): ETQ administrateur d'une procedure declarative, certains de mes dossiers restent en construction [ex: l'object storage est down, le dossier reste bloqué]
2023-04-21 15:45:45 +02:00
Colin Darie
160178b8f8
chore(rubocop): fix Rails/ExpandedDateRange
2023-04-19 12:55:13 +02:00
Paul Chavard
def32a3615
chore(dossier): remove fallback from commune champ
2023-04-11 10:49:01 +02:00
Paul Chavard
a6e3ea4839
fix(dossier): normalize champs commune data
2023-04-07 16:02:13 +02:00
Colin Darie
4288f28ac8
chore(job): same max_run_time between ArchiveCreationJob and ExportJob
2023-03-09 15:37:13 +01:00
Colin Darie
697aeeb348
refactor(job): DossieRebaseJob in new queue so they don't interfere with others
...
Ces jobs sont empilés par milliers d'un coup et peuvent mettre des dizaines
de minutes à s'écouler. On ne veut pas que ça interfère avec d'autres jobs,
notamment les scans antivirus qui, si retardés, empilent des "poll" pour
connaitre le status du fichier.
2023-03-09 15:18:14 +01:00
Colin Darie
86f15b8c05
chore(job): config delayed job for long-living export jobs
2023-03-08 18:37:41 +01:00
Colin Darie
6b87c290cb
fix(cron): PurgeOldEmailEvent in batches
2023-03-08 15:31:10 +01:00
Colin Darie
d20114f978
chore(sentry): use "procedure" tag everywhere for a better debug experience
2023-03-08 13:48:44 +01:00
Colin Darie
43613009d6
fix(job): Cron::WeeklyOverviewJob simplified and respect dolist API rate limiting
2023-03-06 14:39:55 +01:00
Colin Darie
f8f14830a6
fix(missing-antivirus-job): retry when backend is unavailable
2023-03-06 14:39:55 +01:00
Colin Darie
e9585632d9
fix(job): hopefully fix pg timeouts in Cron::PurgeUnattachedBlobsJob
2023-03-02 13:42:04 +01:00
sebastiencarceles
f52554b5a3
review: update value with async fetch
...
1 - spec cover the job which fetches external data
2 - refactor the job with guard clauses
3 - delegate update operation to the champ itself
4 - annuaire education: override the update operation
to let the value be populated by the fetched data
5 - prefilling: don't fetch data synchronously
2023-02-28 14:53:08 +01:00
Paul Chavard
f6b528e497
chore(geo_area): normalize all geo_areas
2023-02-15 11:58:59 +01:00
sebastiencarceles
f0ffae8320
migrate(champs): normalize departements
2023-02-13 16:32:34 +01:00
Colin Darie
a4329b6699
chore(export/job): set procedure_id for sentry
2023-02-09 11:06:54 +01:00
Colin Darie
964f3829c6
fix(export/job): race condition when export was already generated
2023-02-09 11:06:54 +01:00
Martin
f4cc8078b4
amelioration(EmailEvent): purge les email_events vieux de plus de un mois
...
Update app/models/email_event.rb
Co-authored-by: Colin Darie <colin@darie.eu>
Update app/jobs/cron/purge_old_email_event_job.rb
Co-authored-by: Colin Darie <colin@darie.eu>
2023-02-06 10:35:57 +01:00
Lisa Durand
120fbb0ef7
Overload the finished_at
2023-02-01 17:06:04 +01:00
Lisa Durand
d0a131dc55
change the way we set the finished_at on batch operation job to fix alert display bug
2023-02-01 17:06:04 +01:00
Paul Chavard
dd2d3d4626
fix(task): improuve after party backfill_dossiers_repetitions
2023-02-01 15:27:07 +01:00
Paul Chavard
8bf0d9c5ec
fix(dossier): backfill missing repetition champs
2023-02-01 10:36:46 +01:00
Damien Le Thiec
3a8a50a216
Allow prefill pays type de champs ( #8344 )
...
* Allow prefill pays type de champs
* Avoid conditional prefil params for pays champ
* Clean pays data with batch update
* Fix bug and add test batch update pays value
* Improve performance batch_update_pays
* Fix associated country code problem
* Fix after party task name
* Format country name if needed in batch update
2023-01-18 12:52:38 +01:00
Colin Darie
784b0458fe
feat(dolist): consolidate dispatched events with delivered emails
2023-01-18 09:56:40 +01:00
Damien Le Thiec
5e26acb0e1
Make date and datetime prefillable ( #8304 )
...
* Make date and datetime prefillable
* Format in ISO8601 format
2023-01-12 17:42:02 +01:00
Damien Le Thiec
d5bfe84381
Nettoyage des dossiers préremplis abandonnés ( #8341 )
...
* Clean prefilled dossiers
2023-01-12 16:14:42 +01:00
Paul Chavard
dfe65136be
refactor(repetition): add row_id to champs
2023-01-05 18:04:54 +01:00
Martin
018a94f43b
correctif(batch_operation): empeche de changer l'etat d'un dossier en cas d'erreur. et s'assure de delinker le dossier en cas de dossier invalid
2023-01-05 16:20:18 +01:00
Paul Chavard
6e18090fb3
refactor(virus_scan_result): use column instead of metadata on blob
2023-01-04 12:12:11 +01:00
Paul Chavard
d469bca0ae
refactor(watermark): use column instead of metadata on blob
2023-01-04 12:07:08 +01:00
Christophe Robillard
edffd0a7a8
extract ds_opendata_enabled
2022-12-20 12:52:56 +01:00
Christophe Robillard
daa4411abe
Revert "unschedule export demarches opendata"
...
This reverts commit da0738c661
.
`ExportAndPublishDemarchesPubliqjesJob` is now schedulable
2022-12-20 12:52:56 +01:00
Christophe Robillard
672e56a099
schedule job during a no-busy time
2022-12-16 18:02:36 +01:00
Paul Chavard
ed4d5cb36a
refactor(demarche): make declarative demarche processing syncroneous
2022-12-06 11:41:25 +01:00
Martin
ea8bd13bcf
poc(batch_operation.purge): nettoie les batch terminé ou inutiles
2022-12-06 11:28:48 +01:00
Martin
eaf72162da
poc(batch_operation): some rewrite to avoid various conflict (when an instructeur try to create a job with an incompatible dossier regarding the current task). also soome cleanup to isole spec in least involved model
2022-12-06 11:28:48 +01:00
Martin
4266ab93c5
poc(batch_operation): track dossier processed safely
2022-12-06 11:28:48 +01:00
Martin
beb39027d0
poc(batch_operation.ui): implement simple ui to trigger a batch of current page
2022-12-06 11:28:48 +01:00
Martin
7df86c50fb
poc(batch_operations_controller): implement simple [not yet with procedure_presentation] action to create a batch operation
2022-12-06 11:28:48 +01:00
Martin
b11dbafc59
poc(batch_operation_job): implement archive dossiers with async behaviour, lock, run_at, finished_at, failed_dossier_ids, success_dossier_ids
2022-12-06 11:28:48 +01:00
Colin Darie
b13c5e56f6
refactor(piece_justificative): UX follows mockups
2022-12-05 10:47:10 +01:00
Paul Chavard
b89f6e2d53
feat(operation_log): add Cron::DossierOperationLogMoveToColdStorageJob
2022-11-22 18:06:05 +01:00
Kara Diaby
4d6ff768c4
Ajoute les dossiers déposés dans les statistiques Datagouv
2022-11-03 10:24:04 +01:00
Paul Chavard
1fd1878370
feat(jobs): add request_id to jobs payload
2022-10-12 14:32:23 +02:00
Damien Le Thiec
0131a41266
Merge branch 'main' into feature/add_rna_type_de_champs
2022-10-05 12:38:24 +02:00
Damien Le Thiec
0d9f02c88d
First draft autofetch data for RNA
2022-10-04 22:11:01 +02:00
Christophe Robillard
da0738c661
unschedule export demarches opendata
2022-09-30 07:26:48 +02:00
Martin
d9432ccd3b
fix(etablissement.as_degraded_mode): backfill missing data via a cron
2022-09-23 15:12:25 +02:00
Colin Darie
67cbc61e1e
chore(services): update etablissement adresse & geocode when siret changes
2022-09-06 09:23:52 +02:00
Martin
45e0c15eb0
feat(PurgeManagerAdministrateurSessionsJob): also purge instructeur assign_tos
2022-08-23 13:28:10 +02:00
Kara Diaby
ec3118e82c
Add concern for Datagouv schedulable jobs
2022-07-29 12:25:27 +02:00
Kara Diaby
a156529394
modify export descriptif demarches job
2022-07-29 11:17:22 +02:00
Kara Diaby
882ee74f88
Comptes créés avec FC par mois : CRON Job
2022-07-29 11:17:22 +02:00
Kara Diaby
dc1d0ce071
Démarches supprimées par mois : CRON Job
2022-07-29 11:17:22 +02:00
Kara Diaby
4a29162c6c
Démarches closes par mois : CRON Job
2022-07-29 11:17:22 +02:00
Kara Diaby
999059269e
Instructeurs connectés par mois : CRON Job
2022-07-29 11:17:21 +02:00
Kara Diaby
8e9d1eeb6f
Nombre de comptes créés par mois : CRON
2022-07-29 11:17:21 +02:00
Kara Diaby
b48a48980a
Nombre de dossiers créés par mois : CRON
2022-07-29 11:17:21 +02:00
Kara Diaby
9fa54c3cc8
Nombre de démarches créés par mois : CRON
2022-07-29 11:17:21 +02:00
Kara Diaby
e0dc36926e
Nombre d'instructeurs créés par mois : CRON
2022-07-29 11:17:21 +02:00
Kara Diaby
a0a3c6c0ee
Administrateurs crées par mois : CRON Job
2022-07-29 11:17:20 +02:00
Colin Darie
197fa54ea4
fix(archives): purge stuck pending archives
...
Cf 0cb096b3d6440a4da05ca2902ed6fa72a93e6579
2022-07-27 15:26:59 +02:00
Colin Darie
f1957e51f8
fix(exports): purge stuck (pending) exports, not generated or failed
...
Avec l'autre PR, ça reproduit le comportement d'avant, c'est à dire
que quoiqu'il arrive un export est purgé :
- soit 16h après sa génération (on a 16h pour le télécharger)
- soit 12h après sa création, et qu'il est bloqué
Auparavant, tous les exports étaient purgés au bout de 3h quelle que soit
le statut.
2022-07-27 15:26:59 +02:00
Martin
3e56fdd1d7
fix(export): when it takes more than 3 hours, exports are purge before being generated. make it possible to have an export that takes more than 3 hours and share this behaviour with kind of same class archive
2022-07-25 10:48:54 +02:00
Christophe Robillard
906dd4b509
remove useless DATAGOUV_TMP_DIR env
2022-07-21 21:54:15 +02:00
Christophe Robillard
0e3e6eb62a
add schedulable? class method to cron jobs
2022-07-21 21:54:15 +02:00
Christophe Robillard
bed6167010
use specific namespace for datagouv cron jobs
2022-07-21 21:54:15 +02:00
Christophe Robillard
b6c96301ef
export and publish opendata demarches
2022-07-21 21:54:15 +02:00
Martin
7fb85c97cb
feat(Administrateur::ArchivesControllers): add archives #index and #create for admin
2022-07-04 14:18:46 +02:00
mfo
011fcd40d8
Merge pull request #7457 from betagouv/US/fix-webhook
...
fix(WebHookJob): missing find procedure
2022-06-10 17:30:28 +02:00
Martin
165db34c4a
fix(WebHookJob): missing find procedure
2022-06-10 17:28:07 +02:00
Martin
b84c2966c4
fix(DeclarativeProceduresJob.perform_now): avoid to crash other demarche when one fail
2022-06-10 14:59:36 +02:00
Martin
d39b3b09ed
fix(webhook.perform): ensure to enqueue job with current state/updated_at for consistency reason
2022-06-01 15:54:55 +02:00
Christophe Robillard
f14cd7b022
remove warning mail for dubious procedures
2022-05-25 14:44:53 +02:00
Paul Chavard
9e0b3b642f
cleanup(sendinblue): remove sendinblue tracking
2022-05-06 11:14:44 +02:00
simon lehericey
da97ad1858
destroy all admins
2022-05-03 12:14:02 +02:00
Martin
3c0e3f725d
feat(administrateur/procedures#publication): show dubious to administrateur
...
Update app/views/administrateurs/procedures/publication.html.haml
Co-authored-by: Paul Chavard <paul.chavard@beta.gouv.fr>
2022-04-27 15:13:22 +02:00
Martin
dbcf21a555
feat(archive): extract archive status management within job to simplify the main service as well as to isolate this part for a merge with exports csv/xslx [maybe?]
...
Update app/dashboards/archive_dashboard.rb
Co-authored-by: LeSim <mail@simon.lehericey.net>
2022-03-31 13:35:49 +02:00
LeSim
fdc4e5bfbc
Merge pull request #6928 from adullact/feature/6886-watermark-url
...
Watermark depuis une URL distante
2022-02-08 11:45:26 +01:00
Paul Chavard
0268511264
fix(manager): remove admins added for a day
...
fix #6855
2022-02-08 11:18:44 +01:00
Jon
88d3a6e4c3
feat: can have remote watemark file
2022-02-04 18:55:16 +01:00
Paul Chavard
403e71b41f
feat(procedure): only purge procedures without dossiers
2022-01-18 11:22:04 +01:00
simon lehericey
f0b0e7fd9a
Switch to usage of zip unix binary to create archive. Also use a dedicated queue for DelayedJob
...
use dedicated archives queue
As the used disk space will increase, we want a fined grain control
move zip logic in dedicated method
zip
wip
wip
fix(spec): pass spec in green
tech(improvements): avoid File.delete(folder), favor FileUtils.remove_entry_secure which is safer. Also wrap most of code that open file within blocks so it is cleaned when the block ends. Lastly use attachement.download to avoid big memory pressure [download in chunk, write in chunk] otherwise big file [124>1GO] are loaded in memory. what if we run multiple jobs/download in parallel ?
fix(spec): try to retry with grace
clean(procedure_archive_service_spec.rb): better retry [avoid to rewrite on open file]
lint(things): everything
2021-12-13 16:37:04 +01:00
kara Diaby
9ee9389ba1
add the good retry_on StaleObjectError
2021-11-05 13:32:21 +01:00
Paul Chavard
d5d3995dad
fix(dossier): fix dossier.avis cascade
2021-10-26 11:16:29 +02:00
Paul Chavard
1561ea82f6
fix(transfer): manually nullify staled transfers references
2021-10-21 13:54:26 +02:00
Paul Chavard
2a3a9dd822
feat(revisions): rebase dossiers brouillons
2021-10-20 17:26:09 +02:00
Paul Chavard
4a947f9135
feat(manager): add become administrateur button in manager (with 24h expiration)
2021-10-07 11:32:12 +02:00
simon lehericey
e4484eb6d9
discard expert not found job error
2021-10-04 12:57:37 +02:00
Paul Chavard
5985755229
feat(dossier): add purge stale dossier transfers job
2021-09-08 14:39:46 +02:00
Christophe Robillard
ddea0b9a45
purge old sendinblue mails
2021-08-20 08:52:39 +02:00
Paul Chavard
ffa8c0c80a
feat(dossiers): enable dossiers termine expiration behind feature flag
...
feature flag "procedure_process_expired_dossiers_termine" controls if a procedure has expiration
enabled on dossiers termine
re #3796
2021-08-18 16:11:35 +01:00
Christophe Robillard
66cc0dd08d
Revert "Revert "Expose dossier PDF export as IO""
...
This reverts commit 362093eff0
.
2021-06-24 21:01:59 +02:00
krichtof
362093eff0
Revert "Expose dossier PDF export as IO"
2021-06-24 19:21:37 +02:00
Paul Chavard
48dca8bad1
Execute AchiveCreationJob in exports queue
...
Co-authored-by: Christophe Robillard <christophe.robillard@beta.gouv.fr>
2021-06-24 17:49:58 +02:00
Christophe Robillard
f40d96fbd2
Revert "Revert "Export de tous les dossier d'une démarche""
...
This reverts commit d9a588b52e
.
2021-04-29 17:29:47 +02:00
krichtof
d9a588b52e
Revert "Export de tous les dossier d'une démarche"
2021-04-29 16:07:18 +02:00
Pierre de La Morinerie
684af77e35
jobs: extract an RetryOnTranscientErrors
concern
2021-04-29 14:08:12 +02:00
Christophe Robillard
aa407d5607
place le cronjob PurgeStaleArchives dans le bon répertoire
2021-04-28 11:40:58 +02:00
Christophe Robillard
8bee53fe77
rend plus robuste la création de l'archive
2021-04-28 11:40:58 +02:00
Christophe Robillard
609a1aa45b
add archive creation and purge archives jobs
2021-04-28 11:40:58 +02:00
simon lehericey
e636e3a752
add merge_and_update_metadata method
2021-04-06 18:11:26 +02:00
simon lehericey
d552e364fc
retry 5 times on integrity error and then block processing
2021-04-06 18:03:39 +02:00
Pierre de La Morinerie
75a1046315
active_storage: refactor concerns
...
Follow-up of #5953 .
Refactor the concerns with two goals:
- Getting closer from the way ActiveStorage adds its own hooks.
Usually ActiveStorage does this using an `Attachment#after_create`
hook, which then delegates to the blob to enqueue the job.
- Enqueuing each job only once. By hooking on `Attachment#after_create`,
we guarantee each job will be added only once.
We then let the jobs themselves check if they are relevant or not, and
retry or discard themselves if necessary.
We also need to update the tests a bit, because Rails'
`perform_enqueued_jobs(&block)` test helper doesn't honor the `retry_on`
clause of jobs. Instead it forwards the exception to the caller – which
makes the test fail.
Instead we use the inline version of `perform_enqueued_jobs()`, without
a block, which properly ignores errors catched by retry_on.
2021-03-16 11:49:14 +01:00
Pierre de La Morinerie
08094e5384
jobs: retry on ActiveStorage::IntegrityError during virus scan
...
This is the same setting than what ActiveStorage::AnalyzeJob uses.
2021-03-16 10:52:12 +01:00
Paul Chavard
d24ee27cac
Try to reduce the number of external data fetches
...
Only fetch external data if the external_id is the current one and data is not yet fetched
2021-03-11 14:56:37 +01:00
Paul Chavard
21d5da379f
remove unused job
2021-03-11 14:56:37 +01:00
Paul Chavard
7eee14feef
Add ChampFetchExternalDataJob
2021-02-17 19:10:16 +01:00
Pierre de La Morinerie
150ddab660
zeitwerk: Api -> API
2021-02-09 13:07:30 +01:00
Paul Chavard
9156754cf6
After maximum ammount of retries discard the job and log exception
2021-02-05 18:56:27 +01:00
Paul Chavard
41c3a98d7d
Update Raven references to use Sentry
2021-01-28 19:46:36 +01:00
Paul Chavard
0633d43868
Stop crashing if attachment can not be watermarked and mark blob as invalid
2021-01-27 16:49:09 +01:00
simon lehericey
650a598698
ensure a buggy procedure does not block the others from being closed
2021-01-19 10:42:16 +01:00
Paul Chavard
97ce8f312b
Use external_id to fetch annuaire_education data
2021-01-15 11:50:02 +01:00
Paul Chavard
5c9f2e8783
Add api education adapter and job
2021-01-14 17:57:48 +01:00
simon lehericey
1348b883e3
Add cron to complete missing analysis
2020-12-17 11:05:33 +01:00
kara Diaby
f3c4040d2d
add grace period to deleted dossiers
2020-12-16 15:00:26 +01:00
Paul Chavard
3c6e30a326
Show titre identite link only once watermark is added
2020-12-10 16:50:03 +01:00
clemkeirua
513d4f6ff1
move all the cron jobs in a dedicated directory
2020-12-08 13:22:43 +00:00
simon lehericey
b187244a29
extract and refactor api errors
2020-12-04 17:07:30 +01:00
simon lehericey
46c355beb2
Add comments
2020-12-03 11:52:01 +01:00
simon lehericey
f56235c7ee
retry exponentially on api entreprise timeout
2020-12-03 11:52:01 +01:00
simon lehericey
9eab310565
[ #5675 ] for all api entreprise job, retry on 502, 503
2020-12-02 18:25:31 +01:00
Paul Chavard
18dfacb933
Make watermark configurable
2020-11-25 16:19:06 +01:00
Paul Chavard
371179dc5b
Watermark titres identite
2020-11-25 16:19:06 +01:00
simon lehericey
c563956a9f
retry each day during 5 days to fetch attestation social
2020-10-28 15:53:58 +00:00
simon lehericey
ad01d4dee5
really faster stats by precomputing some of them
2020-10-06 16:12:01 +02:00
clemkeirua
245e9e59c7
do not run ApiEntreprise jobs on missing etablissements
2020-09-28 17:36:38 +02:00
Paul Chavard
dea78c49f6
Run jobs in named queues
2020-09-23 08:22:17 +02:00
simon lehericey
5459c2fa7f
Fix uninitialized excon constant
2020-09-08 18:36:24 +02:00
Pierre de La Morinerie
292dc84eb4
jobs: ignore deleted file in Virus Scan
...
We have errors in production where the job starts correctly (i.e. the
blob exists), but `blob.open` fails with a `ActiveStorage::FileNotFound`
error.
When checking later in production, the blob has been deleted.
This points to the blob (and the file) being deleted during the virus
scan job.
In that case, ignore the error (rather than retrying the job).
2020-09-03 11:00:59 +02:00
simon lehericey
49aa426d1b
application job swallow BadRequest error and retry
2020-09-02 16:59:54 +02:00
Paul Chavard
5ceae8235b
Ignore type_de_champ procedure_id
2020-09-02 11:26:26 +02:00
Paul Chavard
ec72fdd164
Remove migration service
2020-09-02 11:26:26 +02:00
Pierre de La Morinerie
3c91cfc83c
jobs: add health-related keywords to dubious procedures scanner
...
We are not certified for hosting health-related data yet.
2020-08-25 12:04:10 +00:00
Christophe Robillard
f2dcb61521
transform job in task for set dossiers_last_updated_at
2020-08-12 16:10:15 +02:00
Pierre de La Morinerie
54a4db1c47
config: flip config.active_storage.queues
...
ActiveStorage jobs are now moved to their own queue.
For consistency, we also move our own analysis jobs (VirusScannerJob)
on the same `:active_storage_analysis` queue.
2020-08-05 16:48:36 +02:00
clemkeirua
952a43058f
last_commentaire_updated_at does not take system email into account
2020-07-30 11:52:41 +02:00
clemkeirua
153c6aebd7
add temp job to set last_updated_at_* values for dossier
...
Co-authored-by: Christophe Robillard <christophe.robillard@beta.gouv.fr>
2020-07-30 10:31:14 +02:00
Paul Chavard
bb5a90da76
fix active_revision_id -> active_revision.id
2020-07-23 15:07:25 +02:00
Paul Chavard
7d421d471c
Fix revision migration job
2020-07-23 09:56:34 +02:00
Paul Chavard
bc09b61684
Add migration service and task
2020-07-21 19:35:30 +02:00