Commit graph

18532 commits

Author SHA1 Message Date
Vincent Ambo
5d856ac2e9 feat(tvix/derivation): add path_with_references
This allows the calculation of a store path for a plain string that
potentially contains references. These paths are used for
`builtins.toFile` (and potentially other features of C++ Nix).

Change-Id: Ic507c7f264f362b5e6e628255869e5a4fbe4d788
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7906
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-23 12:55:24 +00:00
Vincent Ambo
be7f64ac87 refactor(tvix/derivation): minor structure simplification
Fixes some module comments and embeds the `compress_hash` function in
the `derivation` module, as it was not used outside of this module
anyways.

Change-Id: I6c5c92b3f0c03c2cdcbcfc2f813909a968c4d44c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7905
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-23 12:55:24 +00:00
Vincent Ambo
5719763fd3 feat(tvix/eval): support builtins implemented in Nix itself
This makes it possible to inject builtins into the builtin set that
are written in Nix code, and which at runtime are represented by a
thunk that will compile them the first time they are used.

Change-Id: Ia632367328f66fb2f26cb64ae464f8f3dc9c6d30
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7891
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-22 20:48:53 +00:00
Vincent Ambo
8513a58b37 docs(tvix/eval): update some outdated comments
These don't apply anymore since the "antidote-CL".

Change-Id: I40ee73ef43d44bbfc650a8fe6c2b33263dd06959
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7890
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-22 19:39:00 +00:00
Aaqa Ishtyaq
59c3383b10 fix(corp/tvixbolt): set output to eval value.
This CL fixes the bug where output of a nix
evaluation is not set.

Change-Id: I8ae2759a7ec26e1de2e57dd43302129347a8c302
Signed-off-by: Aaqa Ishtyaq <aaqaishtyaq@gmail.com>
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7896
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-22 16:35:49 +00:00
Vincent Ambo
ed8dd4acd7 feat(corp/data-import): add import of OR 'translations' table
The original dataset contains translations into different languages,
but only the English ones are imported here.

Note that translations are for lemmata only.

Change-Id: Ifb9c32c25fda44c38ad899efca9d205c520c0fa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7895
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-22 16:13:09 +00:00
sterni
2b308c64b9 feat(sterni/aoc/2022): day16 part 1 k solution
Change-Id: I4d141e0d911157ae17f12066b859297836d4858b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7897
Autosubmit: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Reviewed-by: sterni <sternenseemann@systemli.org>
2023-01-22 14:36:01 +00:00
Vincent Ambo
8eeb5d3bcc feat(corp/data-import): add import of OR 'words_forms' table
This is the full morphological set table for all the words from the
lemmata table, which they don't call it that.

Change-Id: I6f5be673c5f59f11e36bd8c8c935844a7d4fd170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7894
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 17:49:33 +00:00
Vincent Ambo
429c0d00c4 feat(corp/data-import): add import of OpenRussian 'words' table
This is actually the lemmata table of this corpus, not the forms of
all words (they're in a separate table).

Change-Id: I89a2c2817ccce840f47406fa2a636f4ed3f49154
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7893
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 17:49:33 +00:00
Vincent Ambo
ee0c0ee951 chore(corp/data-import): make OR data archive available in env
Change-Id: Idacf42743051eae0cf7010f952a4f91af17ad708
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7892
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 17:49:33 +00:00
Florian Klink
1a84e3e6d8 chore(tvix/store): add fastcdc crate
This is used for content-defined chunking.

Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 15:12:45 +00:00
Florian Klink
4614db2f83 docs(tvix): link to IRC channel and mailing list
I omitted the link to the commit history. Most people see this README
through some git interface anyways, so it felt a bit redundant.

Change-Id: I92cf6d08b83ef680fe37df29d3d546cad020955a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7888
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
2023-01-21 15:08:38 +00:00
Florian Klink
758b94f47c docs(tvix): move most of //tvix/eval README up to //tvix
While moving the CLI out of the evaluator, we forgot to update the
README in //tvix/eval. Move this up to //tvix, so people know where
to start.

Keep the instructions on how to build only `//tvix/eval` in `//tvix/
eval/README.md`.

Change-Id: Ie2755e8b5a0056225dbf3a0ee040f70f7f6a1f27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7887
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 15:08:38 +00:00
Florian Klink
90979d39f3 feat(tvix/store/directory): validate Directory and sizes
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.

Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 10:48:17 +00:00
Florian Klink
d8e0fa8e5e feat(tvix/store/directory): deduplicate Directory messages
We can omit sending Directory messages to clients that have already been
sent in the same stream.

We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.

Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Florian Klink
e719da53be feat(tvix/store): add SledDirectoryService
Change-Id: I8e97a59da2c6623f4cca746efa741097a6dbe5b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7871
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 10:48:17 +00:00
Adam Joseph
22b9e6ff09 refactor(tvix/eval): administer antidote for poison
The codebase contains a lot of complexity and odd roundabout
handling for shadowing globals.  I'm pretty sure none of this is
necessary, and all of it disappears if you simply make the globals
part of the ordinary identifier resolution chain, with their own
scope up above the root scope.  Then the ordinary shadowing routines
do the right thing, and no special cases or new terminology are
required.

This commit does that.

Note by tazjin: This commit was originally abandoned when Adam decided
not to take away reviewer bandwidth for this at the time (eval was
still in a much earlier stage). As we've recently done some
significant refactoring of globals initialisation this came up again,
and it seems we can easily cover the use-cases of the poison tracking
in other ways now, so I've rebased, updated and resurrected the CL.

Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib3309a47a7b31fa5bf10466bade0d876b76ae462
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7089
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-21 10:19:15 +00:00
Florian Klink
ab8486e5b8 chore(tvix/store): add tonic-mock
Upstream seems to be dead, so we're using https://github.com/tyrchen/
tonic-mock/pull/3 here.

According to https://github.com/tyrchen/tonic-mock/pull/1#issuecomment-
1241164173, we might not need this crate at all, but for now, it gets
the job done and is less code to write in the tests.

Change-Id: Ia77fa19b998a5bbabd0311cc714b85a2ee30f36a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7869
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-21 09:34:15 +00:00
Florian Klink
a8b13a0b57 refactor(tvix/store): simplify test a bit
Import more things, and use expect_err to unpack the response.

Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-21 09:34:15 +00:00
William Carroll
e20c0d2fbf fix(wpcarro/slx): Fix LTE/GTE parsing error
Fix: `i += 2`. Welp!
Change-Id: I06061f0c5bb5283c8b85bd3f5a6e52e2eb59d4f5
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7885
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
2023-01-20 23:50:27 +00:00
William Carroll
7f37cfb184 test(wpcarro/slx): Add (basic) tests
Tests with React? ...wat?

Change-Id: I95ccd08bb6e66e9d74a63a596b5f844f9dab0361
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7884
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
2023-01-20 23:33:22 +00:00
William Carroll
6945601ef3 feat(wpcarro/slx): Support grouping
I'm considering rearchitecting the parser to align with the "Crafting
Interpreters" wisdom. I don't really want to do that right now, and the current
implementation fits my use-case. If I discover bugs or want to add more
features, I'll revisit the issue.

Change-Id: Ia767933519ea9fc0fe954cb142f21b02e13a1f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7883
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
2023-01-20 23:33:22 +00:00
Vincent Ambo
7442558b33 refactor(tvix/eval): keep globals alive through VM struct
This forces users to pass the fully constructed set of globals to the
VM, making it harder to accidentally "lose" the set while weak
references to it still exist.

This doesn't modify any functionality, but is laying the foundation
for simplifying some of the builtins behaviour that has grown more
complex again.

Change-Id: I5120f97861c65dc46d90b8a4e2c92ad32cc53e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7877
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 22:48:13 +00:00
Vincent Ambo
6d03e31060 docs(tvix/eval): add doc comments to VM fields
Change-Id: Ia4857c217de15aec8b61e1abd39e22c50e2d816a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7876
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
2023-01-20 22:48:13 +00:00
sterni
23a5d730f7 chore(3p/sources): Bump channels & overlays
* //3p/overlays/haskell: clean up stale override

* //3p/overlays/tvl: work around mullvad build failure

* //3p/gerrit: update dependencies' output hash

Change-Id: Iaea849fb199bb3d059e067c237ca939b3558a766
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7867
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: sterni <sternenseemann@systemli.org>
2023-01-20 22:31:09 +00:00
William Carroll
bfa662eadf feat(wpcarro/slx): Include naive optimizations
Empty queries and empty lists.

Change-Id: Iacd429feb9d6a0b48ffcac682c3dbd4a0a164134
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7880
Tested-by: BuildkiteCI
Autosubmit: wpcarro <wpcarro@gmail.com>
Reviewed-by: wpcarro <wpcarro@gmail.com>
2023-01-20 18:34:32 +00:00
William Carroll
f91785bcc2 feat(wpcarro/slx): Support EQ operator
Naturally...

Change-Id: I9802a12db65eb07ed820e6ec1b56a9528001d0b8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7879
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
2023-01-20 18:34:31 +00:00
William Carroll
9df188ad03 feat(wpcarro/slx): Support numeric comparisons
I was (and still ~am) a bit leery of supporting this (scope creep?), but I need
it in two of my personal projects all within the first O(days) of using this. So
I'm thinking that if this tool is going to be a workhorse, I'll need to
sacrifice some purity for practicality. Future me will find out the real
answer...

Change-Id: Ia71a8cf6627062440476b638d2c194c2c9ac97c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7878
Autosubmit: wpcarro <wpcarro@gmail.com>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
2023-01-20 16:09:11 +00:00
Vincent Ambo
259faea2b2 feat(tvix/cli): add errors module with drv construction errors
These will be threaded through to eval through the new `TvixError`
variant.

Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-20 15:39:51 +00:00
Vincent Ambo
7d0456fa0e feat(tvix/eval): let builtin macro capture external state
This adds a feature to the `#[builtins]` macro which lets users
specify an additional state type to (optionally) thread through to
builtins when constructing them.

This makes it possible for builtins-macro users to pass external state
handles (specifically, in our case, known path tracking) into a set of
builtins.

Change-Id: I3ade20d333fc3ba90a80822cdfa5f87a9cfada75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7840
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-20 15:39:51 +00:00
Vincent Ambo
f12f938166 refactor(tvix/eval): directly return builtin tuples from macro
All invocations of the builtin macro had to previously filter through
the `builtin_tuple` function, but it's more sensible to directly
return these from the macro.

Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 15:39:51 +00:00
Vincent Ambo
972c867b36 feat(tvix/eval): add error contexts to annotate error kinds
This makes it possible for users to add additional context to an
error, which will then be rendered as an additional secondary span in
the formatted error output.

We should strive to do this basically anywhere errors are raised that
can occur multiple times, *especially* during type casts. This was
triggered by me debugging a type cast error attached to a fairly
large-ish span (a builtin invocation).

Change-Id: I51be41fabee00cf04de973935daf34fe6424e76f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
2023-01-20 15:39:51 +00:00
William Carroll
148a63ae7e fix(wpcarro/slx): Forward config to all functions
Oops...

Change-Id: I985a1a10e3009107ca2b8f65e5377f36fe0531fa
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7875
Autosubmit: wpcarro <wpcarro@gmail.com>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
2023-01-19 23:23:44 +00:00
William Carroll
509e356bb8 feat(wpcarro/slx.js): Support JavaScript simple-select impl
See README.md

Change-Id: I6a50e34398c42aabe3cceba160be006f1867eca4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7874
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
2023-01-19 18:12:59 +00:00
Vincent Ambo
0dfe460fbb docs(corp/data-import): document OpenRussian format
This is the second dataset I want to integrate as it contains some
more practically useful, but somewhat less structured, information.

Change-Id: Ib46b2597a33e76f59e030f889a0961ecc5a144eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7873
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 21:58:02 +00:00
Vincent Ambo
db26825eec chore(corp/data-import): namespace tables for OpenCorpora data
I'm changing strategies to importing both OC and another dataset
before continuing to normalise the data, as it might be easier to do
in a set of table-constructing queries inside of SQLite with all raw
data in place.

Change-Id: I26b41af80586fc1bfd8e26a6be20579068a82507
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7872
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 21:58:02 +00:00
Florian Klink
9822fa387a feat(tvix/derivation): more checks for output hashes and encoding
Change-Id: Idebad60c3bf9daf94d04a36bb73ac0dd767f9e79
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7856
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
083e24bbb1 feat(tvix/derivation): add validate_output_paths flag for validate
This allows calling validate() on a derivation that doesn't have its
output paths already calculated yet. It allows offloading some of the
error checking in builtins.derivation* to be offloaded to that function.

Change-Id: Ib4aeadc0eb6583ef8cd765f33e9a9ec32be62729
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7848
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
b1e8fe7212 feat(tvix/derivation): check for other invalid output names
This uses the exposed StorePath::validate_name method to check for other
invalid output names (for which it would not be possible to construct a
store path of).

Change-Id: Ia3f65e19a07ef164f9f64013a5f37cbac99eb8e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7855
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
4a256dda09 docs(tvix/store): fix docstring for StorePath::to_absolute_path
Change-Id: I8b36c85fe11e4fb62e5d28f6900cd80d89a5cc41
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7854
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
0f7e4bec66 feat(tvix/store): rename and expose StorePath::validate_name
Change-Id: I8a16c214c7c644756d9d54187beba8c80ccfb39c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7853
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
6e7f06d942 refactor(tvix/derivation): use DerivationError in Output::validate
Change-Id: I7dbd3b8ff9ef92acddde2e579fb24b8311c34d8f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7852
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 17:13:23 +00:00
Florian Klink
eebb3ce028 refactor(tvix/derivation): rename {ValidateDerivation,}Error
This is now used in more than just validate().

Change-Id: I69c3ad6cb5f3ad60a636fe2ea05d432aebe8e53b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7851
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 17:13:23 +00:00
Florian Klink
0aad4e2246 feat(tvix/derivation): also fail if output name is called drv
`drv` is an invalid output name too, as this would cause a
`builtins.derivation` call to return an attrset with a `drvPath` key
(which already exists) and has a different meaning.

Also handle errors during store path construction, and return our own
error type, instead of the ParseStorePathError.

Change-Id: Ib7952dde1d5cf18a0e210928df7c57b5939b7678
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7850
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
2023-01-18 17:13:23 +00:00
Vincent Ambo
c891833414 feat(corp/data-import): build morphology database in derivation
This makes the actual imported database of the ~whole Russian
language (all lemmas, grammemes, forms etc.) a Nix build target which
is built in CI.

This still needs schema normalisation (it's fairly directly mapped to
the raw data), but it's already starting to be a useful data set.

This also happens to be a pretty cool demonstration of the power of
Nix. You can do `nix-build -A corp.russian.data-import.database` and
out comes a perfectly valid SQLite database with a valid external data
import!

Change-Id: I5d6d15e67d0e4a7ff590fad06252be34f5d561fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7866
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 15:44:06 +00:00
Vincent Ambo
0ed6583edc feat(corp/data-import): let users specify output path
Change-Id: I61ad021c7a5318b099f3adc8bc6aedef65500974
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7865
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 15:44:06 +00:00
Vincent Ambo
476e312c06 feat(corp/data-import): parse and import links
Change-Id: Iebdbc8f884f28064d7b00b8f8808b5030fa3d05c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7864
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 15:44:06 +00:00
Vincent Ambo
dc55ea3201 feat(corp/data-import): parse and import link types
Change-Id: Iae01d1dc6894117dc693b4690d8bc79861212ae6
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7863
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 15:44:06 +00:00
Vincent Ambo
3f0b1d8e0b fix(corp/data-import): commit the final transaction, too
Otherwise up to 1000 elements might be missing.

Change-Id: I20d6238424eec27f0e758e7737c9c31bcb81b23d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7862
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
2023-01-18 15:44:06 +00:00
Vincent Ambo
6986aa5824 feat(corp/data-import): insert OpenCorpora data into SQLite
This is an initial and kind of dumb table structure, but there's some
massaging that needs to be done before this makes more sense.

Change-Id: I441288b684ef86be507099bcc4ebf984598789c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7861
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
2023-01-18 15:44:06 +00:00