DGNum/demarches-normaliennes
15
1
Fork 0
Code Issues 1 Pull requests Projects Releases Packages Wiki Activity
demarches-normaliennes/app/jobs/virus_scanner_job.rb

37 lines
1.2 KiB
Ruby
Raw Normal View History

Save virus scan status to blob metadata
2019-05-02 11:37:35 +02:00
class VirusScannerJob < ApplicationJob
active_storage: refactor concerns Follow-up of #5953. Refactor the concerns with two goals: - Getting closer from the way ActiveStorage adds its own hooks. Usually ActiveStorage does this using an `Attachment#after_create` hook, which then delegates to the blob to enqueue the job. - Enqueuing each job only once. By hooking on `Attachment#after_create`, we guarantee each job will be added only once. We then let the jobs themselves check if they are relevant or not, and retry or discard themselves if necessary. We also need to update the tests a bit, because Rails' `perform_enqueued_jobs(&block)` test helper doesn't honor the `retry_on` clause of jobs. Instead it forwards the exception to the caller – which makes the test fail. Instead we use the inline version of `perform_enqueued_jobs()`, without a block, which properly ignores errors catched by retry_on.
2021-03-11 14:42:57 +01:00
class FileNotAnalyzedYetError < StandardError
end
config: flip config.active_storage.queues ActiveStorage jobs are now moved to their own queue. For consistency, we also move our own analysis jobs (VirusScannerJob) on the same `:active_storage_analysis` queue.
2020-07-13 17:48:09 +02:00
queue_as :active_storage_analysis
jobs: ignore deleted file in Virus Scan We have errors in production where the job starts correctly (i.e. the blob exists), but `blob.open` fails with a `ActiveStorage::FileNotFound` error. When checking later in production, the blob has been deleted. This points to the blob (and the file) being deleted during the virus scan job. In that case, ignore the error (rather than retrying the job).
2020-09-03 11:00:59 +02:00
# If by the time the job runs the blob has been deleted, ignore the error
jobs: ignore missing blob during virus scan We currently have many failed VirusScannerJob enqueued, because the underlying blob is missing. This PR fixes the issue by discarding the job in those cases (because if the blob is gone, the job is never going to succeed). The implementation is based on a similar issue encoutered by the ActiveStorage::AnalyzeJob. See https://github.com/rails/rails/commit/06f8baf73c677303353c79609838caaeec3ccc9a
2020-07-13 14:31:21 +02:00
discard_on ActiveRecord::RecordNotFound
jobs: ignore deleted file in Virus Scan We have errors in production where the job starts correctly (i.e. the blob exists), but `blob.open` fails with a `ActiveStorage::FileNotFound` error. When checking later in production, the blob has been deleted. This points to the blob (and the file) being deleted during the virus scan job. In that case, ignore the error (rather than retrying the job).
2020-09-03 11:00:59 +02:00
# If the file is deleted during the scan, ignore the error
discard_on ActiveStorage::FileNotFoundError
active_storage: refactor concerns Follow-up of #5953. Refactor the concerns with two goals: - Getting closer from the way ActiveStorage adds its own hooks. Usually ActiveStorage does this using an `Attachment#after_create` hook, which then delegates to the blob to enqueue the job. - Enqueuing each job only once. By hooking on `Attachment#after_create`, we guarantee each job will be added only once. We then let the jobs themselves check if they are relevant or not, and retry or discard themselves if necessary. We also need to update the tests a bit, because Rails' `perform_enqueued_jobs(&block)` test helper doesn't honor the `retry_on` clause of jobs. Instead it forwards the exception to the caller – which makes the test fail. Instead we use the inline version of `perform_enqueued_jobs()`, without a block, which properly ignores errors catched by retry_on.
2021-03-11 14:42:57 +01:00
# If the file is not analyzed yet, retry later (to avoid clobbering metadata)
retry_on FileNotAnalyzedYetError, wait: :exponentially_longer, attempts: 10
jobs: retry on ActiveStorage::IntegrityError during virus scan This is the same setting than what ActiveStorage::AnalyzeJob uses.
2021-03-11 18:36:25 +01:00
# If for some reason the file appears invalid, retry for a while
retry 5 times on integrity error and then block processing
2021-04-06 14:59:12 +02:00
retry_on(ActiveStorage::IntegrityError, attempts: 5, wait: 5.seconds) do |job, _error|
blob = job.arguments.first
metadata = {
virus_scan_result: ActiveStorage::VirusScanner::INTEGRITY_ERROR,
scanned_at: Time.zone.now
}
blob.update!(metadata: blob.metadata.merge(metadata))
end
jobs: retry on ActiveStorage::IntegrityError during virus scan This is the same setting than what ActiveStorage::AnalyzeJob uses.
2021-03-11 18:36:25 +01:00
Save virus scan status to blob metadata
2019-05-02 11:37:35 +02:00
def perform(blob)
active_storage: refactor concerns Follow-up of #5953. Refactor the concerns with two goals: - Getting closer from the way ActiveStorage adds its own hooks. Usually ActiveStorage does this using an `Attachment#after_create` hook, which then delegates to the blob to enqueue the job. - Enqueuing each job only once. By hooking on `Attachment#after_create`, we guarantee each job will be added only once. We then let the jobs themselves check if they are relevant or not, and retry or discard themselves if necessary. We also need to update the tests a bit, because Rails' `perform_enqueued_jobs(&block)` test helper doesn't honor the `retry_on` clause of jobs. Instead it forwards the exception to the caller – which makes the test fail. Instead we use the inline version of `perform_enqueued_jobs()`, without a block, which properly ignores errors catched by retry_on.
2021-03-11 14:42:57 +01:00
if !blob.analyzed? then raise FileNotAnalyzedYetError end
if blob.virus_scanner.done? then return end
Save virus scan status to blob metadata
2019-05-02 11:37:35 +02:00
metadata = extract_metadata_via_virus_scanner(blob)
blob.update!(metadata: blob.metadata.merge(metadata))
end
def extract_metadata_via_virus_scanner(blob)
ActiveStorage::VirusScanner.new(blob).metadata
end
end
Reference in a new issue Copy permalink