BREAKING(sidekiq.queues): rationalize queues. now have critical, default, low

Dear instances, make sure to update your sidekiq processes. Indead, we are adopting a new strategy for daemons that process our background
jobs.

Now their is only 3 queues on sidekiq (still, we keep archive/export on delayed job for now).

The big idea : allow any worker to deal with any queue (capacity mgmt) :
- critical: must be procesed now
- default: will be processed asap
- low: do it after everything else

FYI: we had 50% of our server capacity on reserved queues. Leading to
bottle neck processing (reserved queues/worker not processing other
queues). We HAD TO fix it. Sorry for the inconvenience

TBH, this is an idea of Colin Darie <colin@darie.eu>. I'm just pushing
it forward.

Co-Authored-By: Colin Darie <colin@darie.eu>
This commit is contained in:
mfo 2024-09-24 20:58:26 +02:00
parent 0cbb296e7d
commit 08ec45443d
No known key found for this signature in database
GPG key ID: 7CE3E1F5B794A8EC
9 changed files with 8 additions and 10 deletions

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class BatchOperationEnqueueAllJob < ApplicationJob
queue_as :mailers # hotfix
queue_as :critical
def perform(batch_operation)
batch_operation.enqueue_all

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class BatchOperationProcessOneJob < ApplicationJob
queue_as :mailers # hotfix
queue_as :critical
retry_on StandardError, attempts: 1 # default 5, for now no retryable behavior
def perform(batch_operation, dossier)

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class Cron::CronJob < ApplicationJob
queue_as :cron
queue_as :default
class_attribute :schedule_expression
class << self

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class DossierIndexSearchTermsJob < ApplicationJob
queue_as :low_priority
queue_as :low
discard_on ActiveRecord::RecordNotFound

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class DossierOperationLogMoveToColdStorageBatchJob < ApplicationJob
queue_as :low_priority
queue_as :low
def perform(ids)
DossierOperationLog.where(id: ids)

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class DossierRebaseJob < ApplicationJob
queue_as :low_priority # they are massively enqueued, so don't interfere with others especially antivirus
queue_as :low # they are massively enqueued, so don't interfere with others especially antivirus
# If by the time the job runs the Dossier has been deleted, ignore the rebase
discard_on ActiveRecord::RecordNotFound

View file

@ -1,8 +1,6 @@
# frozen_string_literal: true
class HelpscoutCreateConversationJob < ApplicationJob
queue_as :default
class FileNotScannedYetError < StandardError
end

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class ProcedureSVASVRProcessDossierJob < ApplicationJob
queue_as :sva
queue_as :critical
def perform(dossier)
dossier.process_sva_svr!

View file

@ -1,7 +1,7 @@
# frozen_string_literal: true
class WebHookJob < ApplicationJob
queue_as :webhooks_v1
queue_as :default
TIMEOUT = 10