Commit graph

27 commits

Author SHA1 Message Date
Vincent Ambo
5cae0026f5 fix(tazjin/tgsa): fix cloud folder ID for translations
Change-Id: Icc2fa07902507a9abcf43ab3076419fefac8552d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10749
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2024-02-08 20:55:27 +00:00
Vincent Ambo
31099094ec chore(tazjin/tgsa): bump cargo dependencies
Fixes:

- RUSTSEC-2023-0044
- RUSTSEC-2023-0018

Change-Id: Ifc1acce5696f9ec584ac7790d3a99f8ad7d28707
Reviewed-on: https://cl.tvl.fyi/c/depot/+/10014
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-11-12 20:39:12 +00:00
Vincent Ambo
fcda068235 feat(nix/sparseTree): add optional name argument
This allows pinning the name of the sparse tree derivation, which
stops the continous rebuilding of tvix-store-proto dependents.

I've opted to let the function take an attribute set instead and
refactored the call sites appropriately.

Change-Id: I3e57785094b1adbfffa24caf9f1c3384844fa200
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8965
Reviewed-by: grfn <grfn@gws.fyi>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-08-08 20:11:56 +00:00
Vincent Ambo
b399dad0ff refactor(tazjin/tgsa): automatic clippy lint fixes
Change-Id: I30c380bd538dd5c697e529e7e18fe87b8bfea371
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8829
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-06-19 11:36:04 +00:00
Vincent Ambo
332a821100 feat(tazjin/tgsa): replace translation backend
The GPT backend is cool, but it's also very slow, prone to request
errors and quite expensive.

This switches to Yandex Translate instead which for all posts that I
tested seems to be totally fine.

Change-Id: I5217113995b701508a83e7782eb1325957996719
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8826
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-06-18 12:43:15 +00:00
Vincent Ambo
24cbf93729 chore(tazjin/tgsa): log translation requests
Change-Id: Ia4a5059bc82ad337fc9648d65bab638a0c2a7682
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8627
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-24 20:42:14 +00:00
Vincent Ambo
38042ea445 feat(tazjin/tgsa): add gpt-3 powered message translation feature
this is slow and often overloaded, but it's kind of cool when it
works. this translation method deals much better with the kind of
slang you'd see in telegram posts than any other method.

Change-Id: I7e4c845eb382f0eac627c4237b492c8e40dae574
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8625
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-05-24 16:28:33 +00:00
Vincent Ambo
eadcfbbfab feat(tazjin/tgsa): support extracting fallback message from preview
some telegram channels do not allow embedding of messages, but do
allow a preview to be shown on twitter. this preview is just embedded
in the html, and can be scraped out if no message was found.

technically this preview also contains image links, but they are to
very low resolution, thumbnail-style images so i decided not to
include them here.

Change-Id: Ifb89f9fbde8140d577a5ee3af6e60b04232e53e3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8480
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-04-12 12:31:12 +00:00
Vincent Ambo
cd80c00a6b chore(tazjin/tgsa): drop rouille's default features
we don't need these and they add a bunch of unnecessary deps.

Change-Id: I88a30ec8443090a2c61934b35848bea6f1d9597a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/8479
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
2023-04-12 12:31:12 +00:00
Vincent Ambo
96181f917a chore(tazjin/tgsa): bump all dependencies
Change-Id: Id90cf6b74999e81680c589847a8b2f64eefc37ca
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6875
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-10-06 22:53:04 +00:00
sterni
0c178a0ef6 chore(3p/sources): Bump channels & overlays
Upstream nixpkgs removed a lot of aliases this time, so we needed to do
the following transformations. It's a real shame that aliases only
really become discoverable easily when they are removed.

* runCommandNoCC -> runCommand
* gmailieer -> lieer
  We also need to work around the fact that home-manager hasn't catched
  on to this rename.
* mysql -> mariadb
* pkgconfig -> pkg-config
  This also affects our Nix fork which needs to be bumped.
* prometheus_client -> prometheus-client
* rxvt_unicode -> rxvt-unicode-unwrapped
* nix-review -> nixpkgs-review
* oauth2_proxy -> oauth2-proxy

Additionally, some Go-related builders decided to drop support for
passing the sha256 hash in directly, so we need to use the generic hash
arguments.

Change-Id: I84aaa225ef18962937f8616a9ff064822f0d5dc3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6792
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: grfn <grfn@gws.fyi>
Reviewed-by: flokli <flokli@flokli.de>
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2022-09-28 08:02:31 +00:00
Vincent Ambo
742e2a611b chore(tazjin/tgsa): bump cargo dependencies
Change-Id: Id52ab2937e32385324ba93f4df7e1da17a1f4eec
Reviewed-on: https://cl.tvl.fyi/c/depot/+/6119
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-08-22 14:12:00 +00:00
Vincent Ambo
9b395edaa5 chore(tazjin/tgsa): remove note about long image links
this is no longer the case since image links are now forwarded by tgsa
with a cache

Change-Id: I36216efb5f372115e082a11a51c04736f102debb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5612
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-05-16 08:20:22 +00:00
Vincent Ambo
73c08943b8 feat(tazjin/tgsa): use permanent media links in bbcode
uses the tgsa-proxied media links in the message bbcode, leading to
stable image serving even if telegram swaps cdn addresses around, as
long as their embed page is någorlunda the same.

Change-Id: I50af1b3512d4e429fae4a2b3d10395664169e7a1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5611
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-05-15 00:01:31 +00:00
Vincent Ambo
b308361a73 feat(tazjin/tgsa): add stable redirects to telegram images
telegram occasionally changes the upstream urls at which images are
served, with old/new partially overlapping in time afaict.

with this commit, a new endpoint is added to tgsa at

  /img/$user/$post_id/$img_idx

which serves a redirect to the actual image url that telegram had
returned within the last hour (i.e. as per the cache).

next step is to use these image urls in the bbcode itself.

Change-Id: I9aa5cb56bc444cbe796868346c67f2e1e1b79413
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5610
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-14 23:50:50 +00:00
Vincent Ambo
f200b1265f refactor(tazjin/tgsa): move error handling one layer up
this lays the groundwork for adding another handler and returning
handler results as `anyhow::Result<rouille::Response>`.

needed for the image redirect stuff.

Change-Id: I909bd9c2f46f42ea759d50662d7bc36c1f408ed3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5609
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-14 23:50:50 +00:00
Vincent Ambo
e3d09c3446 refactor(tazjin/tgsa): factor out cache access helper
factor out a function to access telegram posts from the cache,
fetching them anew if required.

a small behavioural change means that the program now takes a write
lock when fetching a post, to avoid simultaneously fetching the same
post many times (quite likely once it serves image redirects).

Change-Id: If9c1429f86fb118dab90834f349cb222709c3a31
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5608
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-05-14 23:50:50 +00:00
Vincent Ambo
89f25b431d feat(tazjin/tgsa): Expire cached telegram entries after one hour
Telegram expires certain links in messages after some (unknown) amount
of time; this commit is the first step to working around that by
providing an image URL proxy.

In order to do that, we tick two important boxes here:

1. Store an extensible data structure in the cache.
2. Expire them periodically.

Change-Id: Iba192d8b71db4493c942d1baf5680bd086f8f60b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5607
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-05-14 22:31:31 +00:00
Vincent Ambo
e0b9d9b1cd fix(tazjin/tgsa): Preserve newlines in messages
Change-Id: I9e2d7038357a5510898d153740ae0c0e1f7a8a3a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5483
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-04-20 11:02:44 +00:00
Vincent Ambo
094b8cc7af style(tazjin/tgsa): Move attribution outside of quote body
When quoting a post containing a tgsa quote, the entire quote was
previously erased. Doing it this way will leave the Telegram link in it.

Change-Id: I53bfadd0615734f7e219b20509858d9b73c908a4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5482
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-04-20 09:37:34 +00:00
Vincent Ambo
eca7954758 chore(tgsa): Remove shortening of image links
The link shortening was in place because posts with image URLs that
were too long would not get recognised as valid image links.

astral has fixed this after my report:

  https://forums.somethingawful.com/showthread.php?threadid=3999866

I verified this with the example post that has a bunch of images and
such in it.

Thanks astral!

Change-Id: I60e90fd52020ee5464c931a4dde91382c8a149ab
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5481
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-04-19 22:07:13 +00:00
Vincent Ambo
856dfa62a1 feat(tazjin/tgsa): Handle messages with audio attached
Audio can not be embedded on these stupid dead comedy forums, but
people can click through to listen.

Change-Id: I6e28636e69e424bb8cbc6b92963d1b28b3c04bf6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5478
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-04-17 22:12:43 +00:00
Vincent Ambo
6c8841c176 chore(tazjin/tgsa): Update quote footer text
Change-Id: I39ea4a40d9265d06193e3bddede157b10066da06
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5477
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-04-17 20:01:27 +00:00
Vincent Ambo
91198685b4 feat(tazjin/tgsa): cache generated bbcode in-memory per message
Change-Id: Id36266ef20585bfbdffb197ab08aba2978cb86cb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5471
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2022-04-16 21:31:50 +00:00
Vincent Ambo
792de59c64 feat(tazjin/tgsa): add "web interface" for this
Change-Id: I1619f0ceec2f83f4728d7de006a08b2ce6148e59
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5468
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-16 20:35:01 +00:00
Vincent Ambo
f6b8bf8b41 feat(tazjin/tgsa): Support linked video thumbnails
I couldn't figure out how to get the ID for the `single=1` mode for
linking to a video directly, but linking to the post in embed mode
should also work.

Change-Id: Iebbd62724c36cee227a7c24968617ab418e8bd0c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5467
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-16 20:35:01 +00:00
Vincent Ambo
3d58dfb4ab feat(tazjin/tgsa): Implement initial tg embed -> BBcode features
Supports only posts with plain photos, and loses all message
formatting, but it's getting there.

Change-Id: I9e4afcf3072d1e0724521ccbdc1338fe4f8d5ebe
Reviewed-on: https://cl.tvl.fyi/c/depot/+/5466
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2022-04-16 20:35:01 +00:00