Commit graph

96 commits

Author SHA1 Message Date
Profpatsch
278f56d95b feat(users/Profpatsch/whatcd-resolver): factor out handlers
First stab at factoring out handlers into a generalized handler
function.
This is still kind of confusing, but can be simplified later.

Change-Id: I42da047de83f6d489337d57059f85f793313443a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11245
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-24 15:17:06 +00:00
Profpatsch
eeb5e7abd6 feat(users/Profpatsch/whatcd-resolver): trace http requests
Move the http calls into their own module, so we can trace the request
and provide a simple copy-to-replay command.

We have to work around a bug in the otel library, which would limit
our attribute value length to 128 bytes because it uses the wrong
option value.

~~~

`ifExists` is finally made more useful for dealing with optional
attributes in e.g. lists.

Change-Id: Iafab523e9ec4b00136db43f31fdc12aeefb7f77c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11241
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-23 19:51:59 +00:00
Profpatsch
0b78998509 feat(users/Profpatsch/MyPrelude): add Pretty module
Change-Id: Id774963178ba358447699d0297a6a1fbef5ac8fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11240
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-23 19:51:59 +00:00
Profpatsch
1ae5e20c98 chore(users/Profpatsch/whatcd-resolver): JsonLd module
Change-Id: Ia2bd60b8449592ef1f79ac4877554958eb0b0407
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11239
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-23 05:48:47 +00:00
Profpatsch
c2856dc2cd chore(users/Profpatsch/whatcd-resolver): Transmission & Redacted
Move the functionality into two coarse modules.

There’s still the question about whether functions that change the
database tables should be in their own storage module, but let’s see
if it gets too confusing.

Change-Id: Ied1d47b353dd4597ffea35f111f440aad22e981d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11238
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-03-23 05:27:30 +00:00
Profpatsch
32c5674962 feat(users/Profpatsch/whatcd-resolver): render anon json+ld objs
Some json+ld objects cannot be expanded any further, they are missing
the `@id` tag. Now we also render them as objects.

Change-Id: I1c8f26f3c34e69420c349e66a3ce5a36dc55a1ea
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11173
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-17 01:36:56 +00:00
Profpatsch
3b9fb1aa60 feat(users/Profpatsch/whatcd-resolver): add autorefresh
Adds a little polling mechanism that compares against an ID that is
generated anew every time the server is restarted.

Works well together with shortcuttable.

Change-Id: Icc6745b599e43881c14349794feaf5794cfe6777
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11172
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-17 01:36:56 +00:00
Profpatsch
efa5fe1239 feat(users/Profpatsch/whatcd-resolver): add services & readme
Gives a story for running both jaeger and the reverse proxy, and the
program proper.

`postgres` is still run from the client.

Adds a little readme.

Change-Id: I11185028541c0214e152fd76c9c72b7e2eb11122
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11171
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-17 01:21:14 +00:00
Profpatsch
11a2098e0b feat(users/Profpatsch/my-prelude): update libraries
The latest and greatest!

Change-Id: I34c0e9f41b3b3cc727d9ea89c7ce6a43271b3170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11169
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-16 22:36:55 +00:00
Profpatsch
981c7fef0e feat(users/Profpatsch/whatcd-resolver): log json+ld fetching
This traces the target of a json+ld fetch.

We also simplify the telemetry stuff by using a pseudo-class
`MonadOtel` everywhere.

I wonder if we can get rid of passing the span to `assertM`, because
it’s kind of an antipattern to be honest.

Change-Id: I1448d643c909a29684fa1ae54037177ba2c20639
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11166
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-16 22:36:55 +00:00
Profpatsch
0b06dda9a6 refactor(users/Profpatsch/whatcd-resolver): move AppT & Html out
These functions are just general setup and html helpers, the main file
is getting a bit long otherwise.

Change-Id: I194e9f7f4caa4ce204d510c885dcf5af63d0e76e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11165
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-16 22:36:55 +00:00
Profpatsch
7e63addad3 chore(third_party/haskell): pa-label: 0.1.0.1 -> 0.1.1.0
chore(users/Profpatsch/whatcd-resolver): use more from Label

New label version contains these helpers.

Change-Id: I3ef2ae9b92f2c798842ae001588b2c6d9f3ab971
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11162
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-16 22:36:55 +00:00
Profpatsch
9a7246ea1d chore(users/Profpatsch/whatcd-resolver): slight changes
Change-Id: I57b0fcf9bd3953951dd0cffbee1fbfab5abbeb47
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11089
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
2024-03-03 13:31:27 +00:00
Profpatsch
7c2bdf84b9 fix(users/Profpatsch/whatcd-resolver): pin to postgres 14
I haven’t thought about how to migrate this when the postgres version
changes, so for now we just pin to 14 and worry about it when we reach
EOL lol

Change-Id: Ib72ec7835c660db636a95fb5af8c9ce40897b1ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11085
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2024-03-03 13:30:51 +00:00
Profpatsch
c869f9627e chore(users/Profpatsch): misc small improvements
Change-Id: I8fc128391196da22f03bac76a6c621f2cee73334
Reviewed-on: https://cl.tvl.fyi/c/depot/+/11084
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2024-03-03 13:25:10 +00:00
Profpatsch
7129749968 feat(users/Profpatsch/whatcd-resolver): also link json ld fields
We can cross-reference all of these to schema.org, it should work for
most of the fields.

Change-Id: I38d8dbc7e964764886ddd156c4148bcf3ee376f3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9744
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-10-15 23:05:28 +00:00
Profpatsch
81b790af1d feat(users/Profpatsch/whatcd-resolver): start checking musicbrainz
Ideally we can figure out how to search for single songs by grepping
through musicbrainz. For this we kinda need the jsonld results, so
this is a first step which visualizes the structure and makes it
easy-ish to lazily traverse it.

Change-Id: Ieca21674dee8e8c2dacbab4f2f15ccbe067da647
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9743
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-10-15 22:57:25 +00:00
Profpatsch
0a98f8ec3b chore(third_party/haskell): update pa packages
Change-Id: I8abcb479b0f5c0bd6ed1abc3c9618c2362ff835a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9740
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-10-15 18:58:41 +00:00
Profpatsch
8e811fe625 feat(users/Profpatsch/whatcd-resolver): more otel traces
Change-Id: I5094b64f202eeedb57510a25850bba2edd9ec36f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9725
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-10-14 18:01:11 +00:00
Vincent Ambo
47bc5bb4bc chore(3p/sources): bump channels & overlays (2023-10-08)
Change-Id: I9eeaffad53aba278cd9082c5b4c3e7b1c5f6cc22
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9585
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: tazjin <tazjin@tvl.su>
2023-10-09 09:09:10 +00:00
Profpatsch
acfc49efc2 feat(users/Profpatsch/MonadPostgres): trace db queries
Experiment of how to instrument a lib I’m using to trace instead of
log.

Now that we added MonadTracer to Transaction, we can drop the unlifted `inSpanT`.

Change-Id: Iea891a58cfb33a0837978611456c33aefcccf0d7
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9491
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-09-29 17:25:26 +00:00
Profpatsch
0dcc72a31c feat(users/Profpatsch/whatcd-resolver): add basic otel tracing
For it to work, you need otel (e.g. jaeger) to run on port 4317.

Change-Id: I36f0493b9be26af256769ae5af8916029036a76e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9488
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-09-29 17:25:26 +00:00
Profpatsch
3863a2ebd6 refactor(users/Profpatsch/whatcd-resolver): more efficient inserts
Instead of inserting torrents and every tour group seperately, insert
the tour groups and then the torrents in one go (unzipped).

I finally found a good use for list comprehensions, flattening nested
lists.

Change-Id: I7dfc765ad058dff3afb3b03887141b334a4b1988
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9486
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
2023-09-29 17:11:20 +00:00
Profpatsch
6c360f2b64 refactor(users/Profpatsch/whatcd-resolver): naive combined insert
Still n+1, but now we got the IO where we want it to make the database
inserts efficient.

Change-Id: I2ee36ea41bf186cbeb5581b0df802bb3611769fe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9485
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-09-29 17:11:20 +00:00
Profpatsch
7bc739ec5f refactor(users/Profpatsch/whatcd-resolver): move inserts out
Now we can move the I/O into a where block.

Change-Id: Ib5334948f3d11ca120ce0b7a46c67f8500fdab3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9484
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
2023-09-29 17:08:16 +00:00
Profpatsch
7157e2baed refactor(users/Profpatsch/whatcd-resolver): prepare to split IO
Returning an I/O action was a good first approximation, but leads to a
n+1 query problem, making the whole shebang pretty slow after doing a
search.

Thus we need to split data & I/O, so we can be more clever in the next
commit.

Change-Id: Ieb2f8d5445f1258047da9b121b977c0b8d2dd7f8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9483
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-09-29 17:08:15 +00:00
Profpatsch
3de9601764 feat(users/Profpatsch/whatcd-resolver): add executable
Change-Id: I28f5c26443b9a878f8f374af2c7935831d17ab12
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9355
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-09-17 18:05:42 +00:00
Profpatsch
07b0dff209 refactor(users/Profpatsch/whatcd-resolver): rename main function
Change-Id: I8ce8ef1cc8a35358d20610be7c8bdadefd3aa30f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9354
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-09-17 18:05:42 +00:00
Profpatsch
d111a0fda8 feat(users/Profpatsch/whatcd-resolver): misc improvements
* run on port 9092 (transmission runs on 9091)
* run postgres on port 5431 instead of 5432 (to not interfere)
* only search for albums for now
* correctly handle missing torrent file in SELECT

Change-Id: I20125f7731c9b80a9e8ea05b726adfb1244a24bc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9335
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-09-15 12:44:06 +00:00
Profpatsch
8cfe6bc99b feat(users/Profpatsch/whatcd-resolver): query all pages in search
If more than one page is returned by the search, query all of them.

Also add an ON CONFLICT clause in case the torrent group already
exists, to update it.

This function is getting a bit unwieldy (plus it suffers from an n+1
problem), but ok.

Change-Id: Ib505a2be8286d658ae44a3fe124a4fb42d0fc0c6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9334
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-09-15 12:34:04 +00:00
Profpatsch
33fa42a1a3 chore(users/Profpatsch): Update postgres module n stuff
Improvements from “upstream”, fresh served.

Change-Id: I60e02835730f6a65739eaa729f3e3eed1a0693e6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9025
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-08-08 20:03:22 +00:00
Profpatsch
8dac21b444 feat(users/Profpatsch/whatcd-resolver): Prefer remasters
Usually remasters just … sound better. Thus let’s give them a boost in
the selection score. In case they are horrible they should have
significantly less people seeding them and less snatches than their
originals anyway (maybe the multiplier should be 1.5 or 2 instead?)

Change-Id: I6817091f42b7bcb75eaf57a6aa635b5ba468f99e
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9014
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
2023-08-07 14:03:01 +00:00
Profpatsch
3e5a2ea57f chore(users/Profpatsch): Move Multipart2 into new webstuff package
Change-Id: I903f1b554beed1240d2a9cf14ff44d1f3cb41ec5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9013
Tested-by: BuildkiteCI
Autosubmit: Profpatsch <mail@profpatsch.de>
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-08-06 16:08:23 +00:00
Profpatsch
b41af857ae chore(users/Profpatsch/whatcd-resolver): build with nix
Change-Id: I2d6ee30671bb60a96bcbfede1fd7a17fcf7c3ca8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/9012
Reviewed-by: Profpatsch <mail@profpatsch.de>
Autosubmit: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-08-06 11:21:38 +00:00
Profpatsch
57bab040ed chore(users/Profpatsch): move utils to my-prelude
I want to use these in multiple projects.

Change-Id: I5dfdad8614bc5835e59df06f724de78acae78d42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8971
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-16 20:15:49 +00:00
Profpatsch
43feacb64b feat(users/Profpatsch/whatcd-resolver): Show percent done
Change-Id: I6d7852570bdca807e4d4fff01d72de9f1084fd42
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8910
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-14 08:03:14 +00:00
Profpatsch
50c27b6ba1 feat(users/Profpatsch/whatcd-resolver): Somewhat fix torrent state
I feel like I’m slowly but steadily coding myself into a corner here,
have to rething the whole state thing.

Anyway, now the refresh will display roughly the same information as
the interactive one, which is *a* first step I guess.

Change-Id: I8820c2e321e6e8c9eba0f2f1cc70ce07a044621c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8906
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-07-14 08:03:14 +00:00
Profpatsch
12d23b3e64 feat(users/Profpatsch/whatcd-resolver): Add torrent & basic status
This is a bit dirty, ideally we have a single polling loop that uses
`hx-swap-oob` to fill all status fields in the table (to avoid O(n)
looping requests).

Change-Id: I78ab392964cf00e39424002fe48cb35a60af184a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8875
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-07-14 08:03:14 +00:00
Profpatsch
9504914a59 feat(users/Profpatsch/whatcd-resolver): Download torrent file
Change-Id: I75422a1fc4f94e8aa856f1ea1b2dbec42360c7ac
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8874
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-07-14 08:03:14 +00:00
Profpatsch
4ec27ed088 feat(users/Profpatsch/whatcd-resolver): Display transmission torrent
Change-Id: I1a45dd4c7fa798c161545abf545017be1f83a8f9
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8873
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-14 08:03:14 +00:00
Profpatsch
fa0b7d0804 feat(users/Profpatsch/whatcd-resolver): Remove dynatable
In favor of all-server-side rendering.

Change-Id: I439c31cc6184cd1f6f77843819eebfb396e0ace8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8872
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-14 08:03:14 +00:00
Profpatsch
68a9037d17 feat(users/Profpatsch/whatcd-resolver): Add server-side search
Change-Id: Ifbbe3bca6988b0a090f456ae8d9dbaa808c89e19
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8867
Tested-by: BuildkiteCI
Reviewed-by: Profpatsch <mail@profpatsch.de>
2023-07-14 08:03:14 +00:00
Profpatsch
5cfdd259df feat(users/Profpatsch/whatcd-resolver): Cache searches & web UI
When looking up stuff on the tracker, cache the results in our
database and display the best torrent matches in a simple web UI.

Change-Id: Iba8417fbdd3ea812765ab0289a1d5b03b7c2be81
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8857
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-14 08:03:14 +00:00
Profpatsch
70da4318f5 feat(users/Profpatsch/whatcd-resolver): INSERT red search results
Change-Id: Ice7fdb2e265cfb99734ed41d17b62ac98f7a4869
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8840
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-14 08:03:14 +00:00
Profpatsch
98e38339f2 feat(users/Profpatsch/whatcd-resolver): add initial redacted API
Change-Id: I3143a932646d61b7661df4e4da4edf24a360f6cc
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8833
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-14 08:03:14 +00:00
Profpatsch
07b976ccd8 feat(users/Profpatsch): init whatcd-resolver
Change-Id: Ieb377fb8caa60e716703153dfeca5173f9a6779d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8830
Reviewed-by: Profpatsch <mail@profpatsch.de>
Tested-by: BuildkiteCI
2023-07-14 08:03:14 +00:00