DGNum/demarches-normaliennes
15
1
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity

BREAKING(sidekiq.queues): rationalize queues. now have critical, default, low

Browse source 
Dear instances, make sure to update your sidekiq processes. Indead, we are adopting a new strategy for daemons that process our background
jobs.

Now their is only 3 queues on sidekiq (still, we keep archive/export on delayed job for now).

The big idea : allow any worker to deal with any queue (capacity mgmt) :
- critical: must be procesed now
- default: will be processed asap
- low: do it after everything else

FYI: we had 50% of our server capacity on reserved queues. Leading to
bottle neck processing (reserved queues/worker not processing other
queues). We HAD TO fix it. Sorry for the inconvenience

TBH, this is an idea of Colin Darie <colin@darie.eu>. I'm just pushing
it forward.

Co-Authored-By: Colin Darie <colin@darie.eu>
This commit is contained in:
mfo 2024-09-24 20:58:26 +02:00
parent 0cbb296e7d
commit 08ec45443d
No known key found for this signature in database
GPG key ID: 7CE3E1F5B794A8EC
9 changed files with 8 additions and 10 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
app/jobs/batch_operation_enqueue_all_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class BatchOperationEnqueueAllJob < ApplicationJob class BatchOperationEnqueueAllJob < ApplicationJob
queue_as :mailers # hotfix queue_as :critical
def perform(batch_operation) def perform(batch_operation)
batch_operation.enqueue_all batch_operation.enqueue_all

2
app/jobs/batch_operation_process_one_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class BatchOperationProcessOneJob < ApplicationJob class BatchOperationProcessOneJob < ApplicationJob
queue_as :mailers # hotfix queue_as :critical
retry_on StandardError, attempts: 1 # default 5, for now no retryable behavior retry_on StandardError, attempts: 1 # default 5, for now no retryable behavior
def perform(batch_operation, dossier) def perform(batch_operation, dossier)

2
app/jobs/cron/cron_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class Cron::CronJob < ApplicationJob class Cron::CronJob < ApplicationJob
queue_as :cron queue_as :default
class_attribute :schedule_expression class_attribute :schedule_expression
class << self class << self

2
app/jobs/dossier_index_search_terms_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class DossierIndexSearchTermsJob < ApplicationJob class DossierIndexSearchTermsJob < ApplicationJob
queue_as :low_priority queue_as :low
discard_on ActiveRecord::RecordNotFound discard_on ActiveRecord::RecordNotFound

2
app/jobs/dossier_operation_log_move_to_cold_storage_batch_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class DossierOperationLogMoveToColdStorageBatchJob < ApplicationJob class DossierOperationLogMoveToColdStorageBatchJob < ApplicationJob
queue_as :low_priority queue_as :low
def perform(ids) def perform(ids)
DossierOperationLog.where(id: ids) DossierOperationLog.where(id: ids)

2
app/jobs/dossier_rebase_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class DossierRebaseJob < ApplicationJob class DossierRebaseJob < ApplicationJob
queue_as :low_priority # they are massively enqueued, so don't interfere with others especially antivirus queue_as :low # they are massively enqueued, so don't interfere with others especially antivirus
# If by the time the job runs the Dossier has been deleted, ignore the rebase # If by the time the job runs the Dossier has been deleted, ignore the rebase
discard_on ActiveRecord::RecordNotFound discard_on ActiveRecord::RecordNotFound

2
app/jobs/helpscout_create_conversation_job.rb
View file

@ -1,8 +1,6 @@
# frozen_string_literal: true # frozen_string_literal: true
class HelpscoutCreateConversationJob < ApplicationJob class HelpscoutCreateConversationJob < ApplicationJob
queue_as :default
class FileNotScannedYetError < StandardError class FileNotScannedYetError < StandardError
end end

2
app/jobs/procedure_sva_svr_process_dossier_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class ProcedureSVASVRProcessDossierJob < ApplicationJob class ProcedureSVASVRProcessDossierJob < ApplicationJob
queue_as :sva queue_as :critical
def perform(dossier) def perform(dossier)
dossier.process_sva_svr! dossier.process_sva_svr!

2
app/jobs/web_hook_job.rb
View file

@ -1,7 +1,7 @@
# frozen_string_literal: true # frozen_string_literal: true
class WebHookJob < ApplicationJob class WebHookJob < ApplicationJob
queue_as :webhooks_v1 queue_as :default
TIMEOUT = 10 TIMEOUT = 10