mdebray/tvl-depot
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
tvl-depot/users/flokli/archeology
History
Florian Klink abe099b6ba fix(users/flokli/archeology/parse_bucket_logs): fix regex and skip 
It seems the regex is not perfect, it choked on a single log line:

```
Nov 13 03:10:19 archeology-ec2 59nkrwmih3ywaxrgxqj79pn395fs6m17-parse-bucket-logs-continuously[11105]: Code: 117. DB::Exception: Line "d57bd890fbd1ae16625bdb8168064125e013198099b7e1b3c24878a4d03c3ab8 nix-cache [12/Nov/2023:09:13:02 +0000] xxx.xx.xxx.xxx - VB7SJVZ108DSSN67 REST.POST.OBJECT index.html "POST /index.html HTTP/1.1" 405 MethodNotAllowed 348 - 4 - "-" "Mozilla/5.0 (Macintosh;                 Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML,                 like Gecko) Chrome/39.0.2171.95 Safari/537.36" - 0bFdGKbi0n9JHXU1a2hijcJwmYdc6lG2xgbdozc3wS6mlUkBE7ssrQCHIDdOLebo78o2cGbhivY= - ECDHE-RSA-AES128-GCM-SHA256 - nix-cache.s3.amazonaws.com TLSv1.2 - -" doesn't match the regexp.: (in file/uri log/2023-11-12-10-19-50-80805A702ECF65EB): (at row 5)
```

This was due to the user-agent field. The regex is now fixed.

The request itself is fun (someone trying to POST an index.html to the
bucket), and we should probably filter this on the Fastly side already,
not via IAM,

In any case, there's no point failing to parse if a single line doesn't
match the regex - we can just skip them.

For the sake of completeness, logs for that day have been reprocessed
and reuploaded.

Change-Id: Id98a7167a381cda06d150ad5118ee9e70ead277e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10034
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-11-14 21:46:09 +00:00
..
default.nix feat(users/flokli/archeology): add AWS config to shell 2023-11-12 22:43:26 +00:00
OWNERS feat(users/flokli/archeology): init parse-bucket-logs 2023-11-11 12:24:23 +00:00
parse_bucket_logs.rs fix(users/flokli/archeology/parse_bucket_logs): fix regex and skip 2023-11-14 21:46:09 +00:00
README.md feat(users/flokli/archeology): init parse-bucket-logs 2023-11-11 12:24:23 +00:00

README.md

archeology

This directory contains various scripts and helpers used for nix-archeology tasks.

It's used from some of the archeology instances, as well as standalone.