Commit graph

313 commits

Author SHA1 Message Date
simon lehericey
ea361643be types_de_champ -> types_de_champs_public 2022-05-03 11:05:24 +02:00
Martin
79c473ecf9 fix(ProcedureExportService.to_zip): prebatch dossiers only for to_csv,to_xlsx,to_ods. also add spec around to_zip 2022-04-22 05:13:51 +02:00
Martin
a3c5bf58ac bug(ProcedureExportService.to_zip): use in_batches on Array 2022-04-22 05:13:51 +02:00
Paul Chavard
433c01b1e6 Revert "Merge pull request #7137 from betagouv/faster_pdf"
This reverts commit 9da44bd913, reversing
changes made to ebac71796c.
2022-04-12 19:22:07 +02:00
Paul Chavard
2832ea0286 refactor(export): return blob from to_* methods 2022-04-12 14:54:54 +02:00
Paul Chavard
561b83781e refactor(archive): remove dependencie on archive from ArchiveUploader 2022-04-12 14:54:54 +02:00
Paul Chavard
cf8c084a59 refactor: extract download_and_zip in to a shared service 2022-04-12 14:54:54 +02:00
simon lehericey
4bbe2db668 only export safe pj 2022-04-12 12:05:08 +02:00
simon lehericey
4d3412daf5 batch it 2022-04-12 11:56:56 +02:00
simon lehericey
e2130cc2d3 add witness dossier 2022-04-07 12:06:13 +02:00
simon lehericey
951d964701 clean pj_service test 2022-04-07 12:06:13 +02:00
simon lehericey
34b0578d70 pj_and_path only take dossier id 2022-04-07 12:06:13 +02:00
simon lehericey
7ac1288905 pj_service take a dossier collection ! 2022-04-07 12:06:13 +02:00
Martin
9e8807d12a feat(ArchiveUploader.upload_with_chunking_wrapper): retry once on error 2022-04-05 15:11:21 +02:00
Martin
c72ba43c55 feat(ArchiveUploader.upload_with_chunking_wrapper): expect to retry call to wrapper once 2022-04-05 15:11:21 +02:00
simon lehericey
1f98f75ccc remove unused method 2022-04-05 11:55:14 +02:00
simon lehericey
4c3b9979e9 fix test 2022-04-05 11:55:14 +02:00
Martin
dbcf21a555 feat(archive): extract archive status management within job to simplify the main service as well as to isolate this part for a merge with exports csv/xslx [maybe?]
Update app/dashboards/archive_dashboard.rb

Co-authored-by: LeSim <mail@simon.lehericey.net>
2022-03-31 13:35:49 +02:00
Martin
ab2caaa5f7 fix(ProcedureArchiveService.zip_root_folder): should take archive instance otherwise when we generate many archive for the same procedure, errors may occures 2022-03-30 16:29:54 +02:00
Martin
98c1fb8abc feat(archive): drop old feature 2022-03-18 14:26:09 +01:00
Martin
5739150f15 feat(service/archive_uploader): add an archive uploader class to upload files thru a custom script which handle file encryption of massive file (bigger than 4Go)
Update doc/object-storange-and-data-encryption.md

Co-authored-by: LeSim <mail@simon.lehericey.net>

Update app/services/archive_uploader.rb

Co-authored-by: LeSim <mail@simon.lehericey.net>

Update doc/object-storange-and-data-encryption.md

Co-authored-by: Pierre de La Morinerie <kemenaran@gmail.com>

clean(doc): align document file name and document h1

clean(review): refactore based on various comments

clean(review): refactore based on various comments
2022-03-16 14:56:21 +01:00
Jon
97feca6305 feat(ClamAV): add config to disable clamav usage 2022-02-15 09:15:47 +01:00
Paul Chavard
3d8471e064 fix(dossier): do not send notification on expiration when dossier is already deleted 2022-01-19 17:52:53 +01:00
Paul Chavard
7937e58caa fix(archives): only export dossiers in archive groupe_instructeurs
fix #6793
2022-01-18 11:16:20 +01:00
Martin
383a92bcd9 fix(parallel_download_queue): tophoeus does not like raise from a request handler [crash straight to first frame] 2022-01-07 14:24:37 +01:00
Martin
3eb1c1a421 feat(async_backend): switch to typhoeus 2022-01-04 16:27:23 +01:00
Martin
ce1b189dcd refactor(DownloadManager): extract parallel download in dedicated class. move error management in custom class for procedure exports using the didicated class 2022-01-04 16:17:03 +01:00
Martin
2ed9cccba0 extract download all attachments in dedicated class using async/async-http for better perf 2022-01-04 16:17:03 +01:00
Christophe Robillard
f9609b0b0b fix zone specs
by reenabling task before the next spec is run
2021-12-23 09:58:31 +01:00
Pierre de La Morinerie
4efc1053f9 specs: make dossiers deletion specs more robust
It would fail when ran on some dates, like "21 Dec. 2021".
2021-12-21 17:20:32 +01:00
Christophe Robillard
822c754581 add namespace for populate_zones task 2021-12-16 17:20:06 +01:00
Christophe Robillard
882f92268c update zone to procedures 2021-12-16 17:20:06 +01:00
François Vantomme
14e0c68266 feat(api particulier): add MESRI field 2021-12-16 16:45:18 +01:00
Martin
5ad0ee8ca9 fix(spec.flacky): try to use extractor in a more conventional way to extract file content 2021-12-14 14:18:59 +01:00
Martin
f7bc387e44 feat(flipper): use new way of generating archive only on some procedure flipped with new actor :zip_using_binary 2021-12-13 16:37:04 +01:00
simon lehericey
f0b0e7fd9a Switch to usage of zip unix binary to create archive. Also use a dedicated queue for DelayedJob
use dedicated archives queue

As the used disk space will increase, we want a fined grain control

move zip logic in dedicated method

zip

wip

wip

fix(spec): pass spec in green

tech(improvements): avoid File.delete(folder), favor FileUtils.remove_entry_secure which is safer. Also wrap most of code that open file within blocks so it is cleaned when the block ends. Lastly use  attachement.download to avoid big memory pressure [download in chunk, write in chunk] otherwise big file [124>1GO] are loaded in memory. what if we run multiple jobs/download in parallel ?

fix(spec): try to retry with grace

clean(procedure_archive_service_spec.rb): better retry [avoid to rewrite on open file]

lint(things): everything
2021-12-13 16:37:04 +01:00
François Vantomme
6ab8ac0662 feat (api particulier): add Pole emploi field 2021-12-08 10:36:25 +01:00
François Vantomme
2110ef072c feat(api particulier): add DGFiP field 2021-12-07 11:03:05 +01:00
Paul Chavard
2ba05bfb4f fix(dossier): use depose_at instead of en_construction_at 2021-12-06 15:49:17 +01:00
Martin
1bb868714c fix(spec/lint/review): lint and fix spec of previous commits, also fix based on tchak feedback 2021-12-06 07:05:17 +01:00
Pierre de La Morinerie
970c3e4b2b specs: avoid executing business logic in Procedure factory
Calling business logic in a factory is a code-smell, because it
usually requires the object to be saved into database, and may have
unintended consequences when the business logic is changed.

Also, this allows to just build a published procedure, without saving it
to the database.
2021-11-23 08:07:07 +01:00
Paul Chavard
71939c650b fix(export): remove wrong columns from repetition spreadsheets
This fix prevent repetition children types de champ from being pulled from cloned procedures. stable_id is stable across revisions but also across cloned procedures.
2021-11-10 20:45:32 +01:00
Pierre de La Morinerie
b06183d7bf specs: rename Dossier's with_all_champs to with_populated_champs
This better reflects what the trait is used for, and disambiguate it
from the similarly named trait in the Procedure factory.
2021-11-03 19:38:01 +01:00
Pierre de La Morinerie
d0e87a08cf services: cache zxcvbn dictionaries per-thread
Before, every time a password was tested, the dictionaries were parsed
again by zxcvbn.

Parsing dictionaries is slow: it may take up to ~1s. This doesn't matter
that much in production, but it makes tests very slow (because we tend
to create a lot of User records).

With this changes, the initializer tester is shared between calls, class
instances and threads. It is lazily loaded on first use, in order not to
slow down the application boot sequence.

This uses ~20 Mo of memory (only once for all threads), but makes tests
more that twice faster.

For instance, model tests go from **8m 21s** to **3m 26s**.

NB:
An additionnal optimization could be to preload the tester on
boot, before workers are forked, to take advantage of Puma copy-on-write
mechanism. In this way all forked workers would use the same cached
instance.

But:

- We're not actually sure this would work properly. What if Ruby updates
  an interval ivar on the class, and this forces the OS to copy the
  whole data structure in each fork?
- Puma phased restarts are not compatible with copy-on-write anyway.

So we're avoiding this optimisation for now, and take the extra 20 Mo
per worker.
2021-10-25 12:04:56 +02:00
simon lehericey
d68129b34d add cnaf type de champ 2021-10-12 14:26:40 +02:00
kara Diaby
cbedef996b tests 2021-10-05 15:00:21 +02:00
simon lehericey
9fc195f0e5 add test to dossier_projection_service 2021-09-07 15:17:06 +02:00
Paul Chavard
8e1bfb469f fix(dossier): send expiration notifications 2 weeks prior to supression instead of a month 2021-08-26 11:28:57 +02:00
Christophe Robillard
e06f11f33f add code insee libelle for commune export 2021-08-19 08:04:47 +02:00
Paul Chavard
ffa8c0c80a feat(dossiers): enable dossiers termine expiration behind feature flag
feature flag "procedure_process_expired_dossiers_termine" controls if a procedure has expiration
enabled on dossiers termine

re #3796
2021-08-18 16:11:35 +01:00