DGNum/demarches-normaliennes
15
1
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity
17333 commits 250 branches 1166 tags 78 MiB
Commit graph

557 commits

Author SHA1 Message Date
Martin
ce1b189dcd refactor(DownloadManager): extract parallel download in dedicated class. move error management in custom class for procedure exports using the didicated class 2022-01-04 16:17:03 +01:00
Martin
2ed9cccba0 extract download all attachments in dedicated class using async/async-http for better perf 2022-01-04 16:17:03 +01:00
Christophe Robillard
882f92268c update zone to procedures 2021-12-16 17:20:06 +01:00
Martin
721d926c0d wip(fix): chdir 2021-12-13 16:37:04 +01:00
mfo
148be50c86 Update app/services/procedure_archive_service.rb 
Co-authored-by: LeSim <mail@simon.lehericey.net>
 2021-12-13 16:37:04 +01:00
Martin
f7bc387e44 feat(flipper): use new way of generating archive only on some procedure flipped with new actor :zip_using_binary 2021-12-13 16:37:04 +01:00
simon lehericey
f0b0e7fd9a Switch to usage of zip unix binary to create archive. Also use a dedicated queue for DelayedJob 
use dedicated archives queue

As the used disk space will increase, we want a fined grain control

move zip logic in dedicated method

zip

wip

wip

fix(spec): pass spec in green

tech(improvements): avoid File.delete(folder), favor FileUtils.remove_entry_secure which is safer. Also wrap most of code that open file within blocks so it is cleaned when the block ends. Lastly use  attachement.download to avoid big memory pressure [download in chunk, write in chunk] otherwise big file [124>1GO] are loaded in memory. what if we run multiple jobs/download in parallel ?

fix(spec): try to retry with grace

clean(procedure_archive_service_spec.rb): better retry [avoid to rewrite on open file]

lint(things): everything
 2021-12-13 16:37:04 +01:00
Pierre de La Morinerie
0e35bc609d notifications: don't preload dossiers on instructeurs 
This request currently times out almost every night in production.

It's because although Instructeurs are loaded in batches (default batch
size is 1000), loading all dossiers for 1000 instructeurs is slow.

Turns out the code executed after this query to compute notifications
doesn't even use these dossiers. Indeed it is faster not to preload
them: both the initial query and the total treatment time are shorter.

Here's a quick benchmark made locally (but using production data):

- Before this commit:

	Benchmark.measure { pp Instructeur.includes(assign_to: { procedure: :dossiers }).where(assign_tos: { daily_email_notifications_enabled: true }).limit(100).m
ap(&:email_notification_data) }

Only the initial query : 35s
Total time : 97s

- Without preloading dossiers:

	Benchmark.measure { pp Instructeur.includes(assign_to: :procedure).where(assign_tos: { daily_email_notifications_enabled: true }).limit(100).m
ap(&:email_notification_data) }

Only the initial query : 0.08s (400x faster)
Total time : 29s (3,3x faster)

Plus it doesn't timeout, of course.
 2021-12-09 12:10:00 +01:00
Martin
1bb868714c fix(spec/lint/review): lint and fix spec of previous commits, also fix based on tchak feedback 2021-12-06 07:05:17 +01:00
Kara Diaby
24ba7b6633 modify dossier projection service 2021-11-26 09:45:13 +01:00
simon lehericey
5234a1854c manage AgentConnect callback 2021-11-23 14:17:59 +01:00
simon lehericey
898df449d4 redirect to AgentConnect 2021-11-23 14:17:59 +01:00
simon lehericey
d2432e34eb AgentConnect UI 2021-11-23 14:17:59 +01:00
Paul Chavard
f6b8689a97 fix(revisions): fix repetitions export with revisions 2021-11-03 18:20:48 +01:00
Paul Chavard
0e2f09dd6f fix(dossiers): wrap dossier discard in a transaction 
By doing this we ensure that deleted_dossier are not created when dossier is not discarded
 2021-11-02 18:17:35 +01:00
Pierre de La Morinerie
d0e87a08cf services: cache zxcvbn dictionaries per-thread 
Before, every time a password was tested, the dictionaries were parsed
again by zxcvbn.

Parsing dictionaries is slow: it may take up to ~1s. This doesn't matter
that much in production, but it makes tests very slow (because we tend
to create a lot of User records).

With this changes, the initializer tester is shared between calls, class
instances and threads. It is lazily loaded on first use, in order not to
slow down the application boot sequence.

This uses ~20 Mo of memory (only once for all threads), but makes tests
more that twice faster.

For instance, model tests go from **8m 21s** to **3m 26s**.

NB:
An additionnal optimization could be to preload the tester on
boot, before workers are forked, to take advantage of Puma copy-on-write
mechanism. In this way all forked workers would use the same cached
instance.

But:

- We're not actually sure this would work properly. What if Ruby updates
  an interval ivar on the class, and this forces the OS to copy the
  whole data structure in each fork?
- Puma phased restarts are not compatible with copy-on-write anyway.

So we're avoiding this optimisation for now, and take the extra 20 Mo
per worker.
 2021-10-25 12:04:56 +02:00
Paul Chavard
1ca8192864 perf(export): load dossiers.champs in batches 2021-10-20 16:52:38 +02:00
kara Diaby
0bd879ec9b modify pj service 2021-10-05 15:00:21 +02:00
Paul Chavard
44eb0ada4f fix(i18n): add Kosovo to countries list 
Kosovo is not part of ISO 3166 as of time of writing. https://en.wikipedia.org/wiki/ISO_3166-2:RS#cite_ref-1
 2021-09-08 12:11:25 +02:00
Paul Chavard
96cbbc0192 fix(virus scan): prevent virus scan on archives and signatures uploads 2021-09-07 14:21:48 +02:00
Paul Chavard
084e48c8e3 fix(dossier_projection_service): select external_id on champs 2021-09-07 14:07:55 +02:00
Christophe Robillard
031b74759f archives: move bugreport 2021-08-05 09:24:20 +02:00
Christophe Robillard
38e3469e9a add bug report to archive 2021-08-03 14:51:41 +02:00
François Vantomme
17b659539f Feat (API Particulier): new encryption service 2021-07-30 11:18:44 +02:00
Christophe Robillard
67c0d93eae Debugging archive generation 
debugging exception that occurs sometime when trying to download pj during
archive generation
 2021-07-29 11:12:07 +02:00
Paul Chavard
ce27999bc3 Refactor clone_attachments 2021-07-01 16:21:46 +02:00
Christophe Robillard
cd36f2622f fix download FakeAttachment 2021-06-24 21:02:55 +02:00
Christophe Robillard
66cc0dd08d Revert "Revert "Expose dossier PDF export as IO"" 
This reverts commit 362093eff0.
 2021-06-24 21:01:59 +02:00
krichtof
362093eff0
Revert "Expose dossier PDF export as IO" 2021-06-24 19:21:37 +02:00
Paul Chavard
f39944a02e Create procedure sub folder in archives 
Co-authored-by: Christophe Robillard <christophe.robillard@beta.gouv.fr>
 2021-06-24 17:50:17 +02:00
Paul Chavard
b73d504f8d Expose dossier PDF export as IO 
Co-authored-by: Christophe Robillard <christophe.robillard@beta.gouv.fr>
 2021-06-24 17:49:24 +02:00
Paul Chavard
19195008e8 Expose removed types_de_champ in exports 2021-06-23 09:40:07 +02:00
kara Diaby
87ce71f27d fix import service 2021-06-21 14:11:05 +02:00
Christophe Robillard
fdec9b2fd5 archives: don't update dossiers 
this fix avoid to touch dossier after attaching pdf_export_for_instructeur
 2021-06-18 11:00:57 +02:00
kara Diaby
6353104ecb modify service 2021-06-15 19:36:46 +02:00
François Vantomme
c9b1095d1e Refactor (Rubocop): replace map{ … }.compact by filter_map 
In Ruby 2.7, Enumerable#filter_map has been added.
This cop identifies places where map { … }.compact can be replaced by filter_map.

See: https://docs.rubocop.org/rubocop-performance/cops_performance.html#performancemapcompact
 2021-06-11 15:38:09 +02:00
Paul Chavard
990ae10399 GraphQL byte_size should be BigInt 2021-06-03 10:53:17 +02:00
Paul Chavard
18ce027b84 Update dossier serialize query to stop using deprecated fileds 2021-05-26 11:12:06 +02:00
Paul Chavard
01c558953b Remove API GEO legacy adapter 2021-05-24 11:50:16 +02:00
simon lehericey
63311eceb0 avoid serializing long id list in case of huge procedure, search on it before 2021-05-20 12:08:04 +02:00
simon lehericey
17617fba43 address champ needs the data column 2021-05-19 15:59:18 +02:00
simon lehericey
a46000dc1f ensure to_s is used on specialized champ 2021-05-19 09:39:20 +02:00
Paul Chavard
e043645a88 cleanup tests 2021-05-18 11:42:10 +02:00
kara Diaby
fdde55f675 modify service 2021-05-18 11:42:10 +02:00
simon lehericey
9fd1c604a1 display follower instructeur emails in alphabetic order 2021-05-04 16:38:34 +02:00
Christophe Robillard
01f180022b prend en compte uniquement les pj pour estimer la taille d'un dossier 2021-05-04 12:50:12 +02:00
Christophe Robillard
3d04221dab generate pdf export for dossier only on demand 2021-04-29 17:30:38 +02:00
Christophe Robillard
f40d96fbd2 Revert "Revert "Export de tous les dossier d'une démarche"" 
This reverts commit d9a588b52e.
 2021-04-29 17:29:47 +02:00
krichtof
d9a588b52e
Revert "Export de tous les dossier d'une démarche" 2021-04-29 16:07:18 +02:00
simon lehericey
7b94c2de88 remove extra Dossier call in projection 2021-04-29 11:46:00 +02:00
simon lehericey
5cfb4114ce one query per table 2021-04-29 11:23:41 +02:00
Christophe Robillard
09870c918d prevent race conditions when creating archives 2021-04-28 11:40:58 +02:00
Christophe Robillard
dfbe004122 rename content_type to time_span_type for archives 2021-04-28 11:40:58 +02:00
Christophe Robillard
9134114c2e fix rubocop offenses 2021-04-28 11:40:58 +02:00
Christophe Robillard
8bee53fe77 rend plus robuste la création de l'archive 2021-04-28 11:40:58 +02:00
Christophe Robillard
07cc4fa97f add service to create procedure archive 2021-04-28 11:40:58 +02:00
Christophe Robillard
fba0d78153 generate pdf export inside pieces_justificatives_service 2021-04-28 11:40:58 +02:00
Paul Chavard
c20ad5ca17 Remove the link between commentaire and user 2021-04-22 15:10:47 +01:00
simon lehericey
5bfd4ecbbf add DossierProjectionService 2021-04-22 10:46:04 +02:00
Pierre de La Morinerie
185c74d891 dossier: fix SQL deprecation error 
ActiveRecord 6.1 disallows passing strings to the `order()` function, to
prevent SQL injections.

Here we know that the order string is safe: `ts_vector` is constant,
and `ts_query` is properly escaped.

Wrap the SQL fragment in Arel.sql to bypass the error.
 2021-03-25 13:24:53 +01:00
kara Diaby
fff6725799 modify api and serializer 2021-03-24 17:47:24 +01:00
kara Diaby
2930de1015 remove useless code in models 2021-03-24 17:47:24 +01:00
Paul Chavard
9c3e72c84a Use revision instead of groupe instructeur as dossier procedure link 2021-03-09 14:47:23 +01:00
Paul Chavard
96a832bc19 Add SerializerService 2021-03-02 12:42:22 +01:00
Pierre de La Morinerie
98be7e2076 Fix new Rubocop warnings 2021-02-16 16:31:30 +00:00
Christophe Robillard
a288a13805 disable france connect authentication for admin and instructeurs 2021-02-10 18:13:47 +01:00
Pierre de La Morinerie
150ddab660 zeitwerk: Api -> API 2021-02-09 13:07:30 +01:00
Pierre de La Morinerie
0d101f3f1a views: remove old design's leftbars 2021-02-02 15:33:09 +01:00
Paul Chavard
6ea44eefa9 fix upstream change in i18n 
c4dfcda202
 2021-01-28 19:46:36 +01:00
Christophe Robillard
792b53beeb import several instructeurs for a routee procedure 
Co-authored-by: simon lehericey <mail@simon.lehericey.net>
 2021-01-20 15:03:17 +01:00
Paul Chavard
a2e87cbb56 Add attestation, justificatifs, operation_logs and bill_signatures to dossier export 2020-12-16 16:34:31 +01:00
simon lehericey
b187244a29 extract and refactor api errors 2020-12-04 17:07:30 +01:00
Paul Chavard
59086cc728 Fix geo length computations 2020-10-01 15:09:38 +02:00
Paul Chavard
4dfc1c719f Fix geo areas computations 2020-10-01 15:09:38 +02:00
Paul Chavard
1bd59c72e5 Include in exports types_de_champ from all revisions 2020-09-30 13:42:38 +02:00
Keirua
3fa628235e Nettoyage de I18n.transliterate pour les noms de feuilles d'exports 
Co-authored-by: LeSim <mail@simon.lehericey.net>
 2020-09-29 15:50:19 +00:00
clemkeirua
9714b607aa ensure valid excel worksheet names 2020-09-29 15:50:19 +00:00
simon lehericey
1eb780b0c9 convert sheet_name to ascii before tuncate to 30 to ensure length < 31 bytes 2020-09-29 14:11:06 +02:00
clemkeirua
bf02238662 fix for missing user_sign_in_count 2020-09-28 15:55:18 +02:00
clemkeirua
9b324614b8 transform_values instead of map+to_h 2020-09-25 12:53:11 +00:00
clemkeirua
c19b653581 higher level length sanitization for worksheet length 2020-09-24 14:59:46 +02:00
Paul Chavard
ec72fdd164 Remove migration service 2020-09-02 11:26:26 +02:00
Fabrice Gangler
1a43f888fb Feat: allow to disable France-Connect 
Refs: #5440
 2020-09-01 15:51:10 +02:00
Pierre de La Morinerie
7d4c748432 services: make etablissement save raise on error 
Before, if saving the etablissement failed, the error would be ignored silently.
 2020-08-18 14:56:40 +02:00
Paul Chavard
733b8bd904 Add published revision to all non brouillon procedures 2020-07-22 11:27:49 +02:00
Paul Chavard
bc09b61684 Add migration service and task 2020-07-21 19:35:30 +02:00
Pierre de La Morinerie
acc1cb1abd dossier: export PJs from private champs 
Fix #5297
 2020-06-24 15:42:30 +02:00
Pierre de La Morinerie
6f14e7783f dossier: refactor piece_justificative service 2020-06-24 15:42:30 +02:00
Paul Chavard
20705d6e30 Do not send draft norifications to users on inactive démarches 2020-05-27 16:36:46 +02:00
Christophe Robillard
fb87a09c02 doesn't update dossier etablissement for siret controller 2020-05-26 18:13:20 +02:00
Christophe Robillard
ba5fa9aa4b update ApiEntrepriseService 
There is now the `create_etablissement` method which
create etablissement with EtablissementAdapter
and enqueue api_entreprise jobs to retrieve
all informations we can get based on SIRET
 2020-05-20 11:09:54 +02:00
Pierre de La Morinerie
9f1407b6d7 expiration: fix the mailer arguments 
The mailers expect serializable arguments, but were given
ActiveRecord::Relation objects instead. This made the mailers throw an
exception.

But how was that possible ? This code is tested, and the tests were
green.

Well, the specs spy on the mailer implementation, in order to check that
the mailers methods were properly called. Fair enough.

But if the specs mock the mailer code (instead of calling the original
implementation), we may not notice that the original implementation
rejects our method parameters.

Here this is the case: once we actually call the original implementation
the tests start to fail, because some arguments are not converted from
an ActiveRecord::Relation to a serializable array.

This is fixed by ensuring that the mailer code is executed (and doesn't
throw an exception).
 2020-05-18 16:24:08 +02:00
Christophe Robillard
b12226c667 fetch and store bilans bdf 2020-04-30 16:15:16 +02:00
Christophe Robillard
3ddb5a4cb4 store attestation fiscale and display 2020-04-30 11:39:54 +02:00
Christophe Robillard
d74b14c205 fetch attestation sociale url 2020-04-29 14:25:55 +02:00
Christophe Robillard
453adf3c74 instructeurs: affiche l'effectif moyen annuel n-1 2020-04-23 09:53:27 +02:00
Paul Chavard
b849fc6fc3 Mark as sent or destroy right after each notification is sent to enable a better recovery in case of mid loop failures 2020-04-23 09:36:09 +02:00
Paul Chavard
31943f7d9c Add handeling of expired processed dossiers to deletion service 2020-04-22 15:39:10 +02:00
Christophe Robillard
db3ef195ad fetch and show effectif mensuel 
- fetch and store effectif mensuel (at the creation of the dossier or
when updating a siret field

- show effectif mensuel only for instructeurs
 2020-04-21 14:22:11 +02:00
clemkeirua
2d68d7dc59 fix to_tsquery disallowed chars 2020-04-20 15:15:22 +02:00
Paul Chavard
22604013d0 Expose multiple user selections on champ carte 2020-04-15 18:07:09 +02:00
Paul Chavard
52d2ace823 Remove quartiers prioritaires 2020-04-14 10:29:59 +02:00
clemkeirua
98337f2334 recherche usager full-text avec 1 resultat 2020-04-09 09:42:31 +02:00
Paul Chavard
7ba4c513e6 Refactor notify near deletion mailers 2020-04-01 17:40:52 +02:00
Christian Lautier
806e5d5eed eIDAS level added to France Connect authorization call 2020-03-26 14:07:32 +01:00
Paul Chavard
5b5ae5a7a5 Do not send notifications or create operation logs on brouillon demarches 2020-03-25 15:04:52 +01:00
Paul Chavard
5c2eba1dd1 Always add a reason to dossier deletion 2020-03-24 09:30:11 +01:00
Paul Chavard
cd0acb1344 Cleanup dossier mailers 2020-03-19 16:52:18 +01:00
simon lehericey
ede61ee22b refacto 2020-03-18 15:25:02 +01:00
simon lehericey
50bb2e6a08 refacto 2020-03-18 15:25:02 +01:00
simon lehericey
f35c0518b7 test for #delete_expired_en_construction_and_notify 2020-03-18 15:25:02 +01:00
Paul Chavard
fe1cd2a47b Delete and notify en_construction dossiers 2020-03-18 15:25:02 +01:00
Paul Chavard
f35ccc9d92 Move process_expired_dossiers_brouillon to a service 2020-03-18 15:25:02 +01:00
Paul Chavard
ae2cfdd44e Update browser gem 
`modern?` method was removed in version 4
 2020-03-17 11:25:20 +01:00
Paul Chavard
c281347da1 Remove restclient 2020-03-17 11:12:14 +01:00
kara Diaby
4a70a7fcc8 remove email_notifications_enabled column because of the new daily_email_notifications_enabled 2020-02-27 11:09:19 +01:00
Christophe Robillard
53f420c748 admin: fix sql column ambiguity 
administrateurs_instructeurs and users (loaded by
Administrateur.default_scope) have administrateur_id column with same
name. This commit indicates which column to use for GROUP query.
 2020-02-04 16:21:42 +01:00
maatinito
9672d892e9 Bug with time zone: Time.zone.today looses time_zone and may breaks database comparisons 2019-12-11 17:34:10 +01:00
Paul Chavard
7b947feae4 Rename demarche archivée to demarche close 2019-11-28 15:07:16 +01:00
clemkeirua
9503a363e5 4378 Ajout des PJ de la messagerie dans le zip des PJs du dossier 2019-11-25 10:36:32 +01:00
Paul Chavard
0c6705f7fd Drop old export service 2019-11-21 10:25:40 +01:00
Paul Chavard
990c867c2e [GraphQL] Add more filters to dossiers 2019-11-07 22:07:42 +01:00
Paul Chavard
f22b39b7c5 Handle non unique champ repetable labels 
fix #4466
 2019-11-07 14:27:52 +01:00
Paul Chavard
2a61ed5b1c Export etablissement information in csv 
fix #4440
 2019-11-07 14:27:52 +01:00
simon lehericey
5643e671a0 Code use user.active? 2019-11-05 14:10:39 +01:00
simon lehericey
8ae592fe25 [fix #4361] administrateur active 2019-10-24 18:32:48 +02:00
clemkeirua
4a6893d88b migrate sendinblue API to v3 2019-10-22 10:06:53 +02:00
simon lehericey
f577484aa2 Add groupe_instructeur_label to export v1 2019-09-19 12:30:12 +02:00
simon lehericey
8660d4af30 Instructeur: only export its dossiers 2019-09-19 12:24:40 +02:00
Paul Chavard
8a691d534a Fix AdministrateurUsageStatisticsService 2019-09-17 12:37:37 +02:00
Paul Chavard
3b8d3b7967 Add Style/CollectionMethods to rubocop 2019-09-12 14:48:18 +02:00
Paul Chavard
c370c2f475 Cleanup FlipFlop 2019-09-10 23:52:44 +02:00
clemkeirua
feb9c32537 search attachments inside bloc repetables 2019-09-10 15:30:44 +02:00
simon lehericey
3cc8f0c792 Link Dossier and GroupeInstructeur 2019-09-03 06:47:02 +02:00
Paul Chavard
15148a4fe1 Fix invalid characters in repetition champs sheet_name for xls export 2019-08-28 10:47:58 +02:00
simon lehericey
b7f8bb2fea Simplify ip_service 2019-08-20 13:29:29 +02:00
simon lehericey
dab1519b8c Remove credenticals synchronisation logic 2019-08-16 11:35:19 +02:00
simon lehericey
e3fd1cc33a Remove weird switch_devise_profile_service 2019-08-13 15:15:16 +02:00
simon lehericey
3fde2a6f70 Rename gestionnaire in code to instructeur 2019-08-12 13:47:01 +02:00
maatinito
0b0ef8a318 #3928 Zxcvbn service to compute password complexity 2019-08-01 17:12:14 +02:00
Paul Chavard
c75e39884e Save selection utilisateur as geo area 2019-08-01 11:38:09 +02:00
Paul Chavard
b266915892 Add repetitions to api 2019-08-01 10:04:35 +02:00
Paul Chavard
5a9468aa95 Fix repetition data export 2019-07-30 17:17:36 +02:00
Pierre de La Morinerie
0da0a2b446 model: remove support for cloning procedures with old pj 2019-07-30 16:11:17 +02:00
Pierre de La Morinerie
e975fe4ade tasks: remove task to migrate old pieces justificatives 2019-07-30 16:11:17 +02:00
Pierre de La Morinerie
5502f2e42e dossier: remove UI for uploading old pieces justificatives 2019-07-30 16:11:16 +02:00
Pierre de La Morinerie
80efe27ff2 pj migration: retry when changing the content type fails 2019-07-16 17:51:29 +02:00
clemkeirua
25f81f1d3c download a dossier as zip with all attachments 2019-07-16 09:11:25 +02:00
Paul Chavard
3cb39c2840 Refactor message attachements to use active_storage 2019-07-10 15:35:29 +02:00
Chaïb Martinez
0fdae2456b We will not quibble 
Signed-off-by: Chaïb Martinez <chaibax@gmail.com>
 2019-07-08 14:20:49 +02:00