The original dataset contains translations into different languages,
but only the English ones are imported here.
Note that translations are for lemmata only.
Change-Id: Ifb9c32c25fda44c38ad899efca9d205c520c0fa3
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7895
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is the full morphological set table for all the words from the
lemmata table, which they don't call it that.
Change-Id: I6f5be673c5f59f11e36bd8c8c935844a7d4fd170
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7894
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is actually the lemmata table of this corpus, not the forms of
all words (they're in a separate table).
Change-Id: I89a2c2817ccce840f47406fa2a636f4ed3f49154
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7893
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is used for content-defined chunking.
Change-Id: I10345372cecb9a643cc51ca45aa5b77d2a05198a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7889
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I omitted the link to the commit history. Most people see this README
through some git interface anyways, so it felt a bit redundant.
Change-Id: I92cf6d08b83ef680fe37df29d3d546cad020955a
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7888
Reviewed-by: tazjin <tazjin@tvl.su>
Reviewed-by: sterni <sternenseemann@systemli.org>
Tested-by: BuildkiteCI
Autosubmit: flokli <flokli@flokli.de>
While moving the CLI out of the evaluator, we forgot to update the
README in //tvix/eval. Move this up to //tvix, so people know where
to start.
Keep the instructions on how to build only `//tvix/eval` in `//tvix/
eval/README.md`.
Change-Id: Ie2755e8b5a0056225dbf3a0ee040f70f7f6a1f27
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7887
Reviewed-by: sterni <sternenseemann@systemli.org>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This calls out to Directory::validate() for all received Directory
messages, and also makes sure the sizes we refer a Directory message as
matches the sizes that have been calculated.
Change-Id: I316f9191d5872ee4ba6d78b9a4326f069b22fa63
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7882
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
We can omit sending Directory messages to clients that have already been
sent in the same stream.
We can also omit storing a Directory message if we already have it -
they're content-addressed anyways.
Change-Id: Iba44565e07157a83a033177a2ffbdddced64ba5c
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7881
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
The codebase contains a lot of complexity and odd roundabout
handling for shadowing globals. I'm pretty sure none of this is
necessary, and all of it disappears if you simply make the globals
part of the ordinary identifier resolution chain, with their own
scope up above the root scope. Then the ordinary shadowing routines
do the right thing, and no special cases or new terminology are
required.
This commit does that.
Note by tazjin: This commit was originally abandoned when Adam decided
not to take away reviewer bandwidth for this at the time (eval was
still in a much earlier stage). As we've recently done some
significant refactoring of globals initialisation this came up again,
and it seems we can easily cover the use-cases of the poison tracking
in other ways now, so I've rebased, updated and resurrected the CL.
Co-Authored-By: Vincent Ambo <tazjin@tvl.su>
Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: Ib3309a47a7b31fa5bf10466bade0d876b76ae462
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7089
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
Import more things, and use expect_err to unpack the response.
Change-Id: Ia319dd4d126b8d0e1df585234710d825a33a0002
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7868
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
I'm considering rearchitecting the parser to align with the "Crafting
Interpreters" wisdom. I don't really want to do that right now, and the current
implementation fits my use-case. If I discover bugs or want to add more
features, I'll revisit the issue.
Change-Id: Ia767933519ea9fc0fe954cb142f21b02e13a1f40
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7883
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Autosubmit: wpcarro <wpcarro@gmail.com>
This forces users to pass the fully constructed set of globals to the
VM, making it harder to accidentally "lose" the set while weak
references to it still exist.
This doesn't modify any functionality, but is laying the foundation
for simplifying some of the builtins behaviour that has grown more
complex again.
Change-Id: I5120f97861c65dc46d90b8a4e2c92ad32cc53e03
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7877
Autosubmit: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
I was (and still ~am) a bit leery of supporting this (scope creep?), but I need
it in two of my personal projects all within the first O(days) of using this. So
I'm thinking that if this tool is going to be a workhorse, I'll need to
sacrifice some purity for practicality. Future me will find out the real
answer...
Change-Id: Ia71a8cf6627062440476b638d2c194c2c9ac97c4
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7878
Autosubmit: wpcarro <wpcarro@gmail.com>
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
These will be threaded through to eval through the new `TvixError`
variant.
Change-Id: Ia0d3f8710dcf26bb95015cd2a6a2b2911f06343f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7842
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This adds a feature to the `#[builtins]` macro which lets users
specify an additional state type to (optionally) thread through to
builtins when constructing them.
This makes it possible for builtins-macro users to pass external state
handles (specifically, in our case, known path tracking) into a set of
builtins.
Change-Id: I3ade20d333fc3ba90a80822cdfa5f87a9cfada75
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7840
Reviewed-by: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
All invocations of the builtin macro had to previously filter through
the `builtin_tuple` function, but it's more sensible to directly
return these from the macro.
Change-Id: I45600ba84d56c9528d3e92570461c319eea595ce
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7825
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This makes it possible for users to add additional context to an
error, which will then be rendered as an additional secondary span in
the formatted error output.
We should strive to do this basically anywhere errors are raised that
can occur multiple times, *especially* during type casts. This was
triggered by me debugging a type cast error attached to a fairly
large-ish span (a builtin invocation).
Change-Id: I51be41fabee00cf04de973935daf34fe6424e76f
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7849
Tested-by: BuildkiteCI
Reviewed-by: flokli <flokli@flokli.de>
This is the second dataset I want to integrate as it contains some
more practically useful, but somewhat less structured, information.
Change-Id: Ib46b2597a33e76f59e030f889a0961ecc5a144eb
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7873
Tested-by: BuildkiteCI
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
I'm changing strategies to importing both OC and another dataset
before continuing to normalise the data, as it might be easier to do
in a set of table-constructing queries inside of SQLite with all raw
data in place.
Change-Id: I26b41af80586fc1bfd8e26a6be20579068a82507
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7872
Autosubmit: tazjin <tazjin@tvl.su>
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This allows calling validate() on a derivation that doesn't have its
output paths already calculated yet. It allows offloading some of the
error checking in builtins.derivation* to be offloaded to that function.
Change-Id: Ib4aeadc0eb6583ef8cd765f33e9a9ec32be62729
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7848
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
Tested-by: BuildkiteCI
This uses the exposed StorePath::validate_name method to check for other
invalid output names (for which it would not be possible to construct a
store path of).
Change-Id: Ia3f65e19a07ef164f9f64013a5f37cbac99eb8e0
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7855
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
This is now used in more than just validate().
Change-Id: I69c3ad6cb5f3ad60a636fe2ea05d432aebe8e53b
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7851
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
`drv` is an invalid output name too, as this would cause a
`builtins.derivation` call to return an attrset with a `drvPath` key
(which already exists) and has a different meaning.
Also handle errors during store path construction, and return our own
error type, instead of the ParseStorePathError.
Change-Id: Ib7952dde1d5cf18a0e210928df7c57b5939b7678
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7850
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
Autosubmit: flokli <flokli@flokli.de>
This makes the actual imported database of the ~whole Russian
language (all lemmas, grammemes, forms etc.) a Nix build target which
is built in CI.
This still needs schema normalisation (it's fairly directly mapped to
the raw data), but it's already starting to be a useful data set.
This also happens to be a pretty cool demonstration of the power of
Nix. You can do `nix-build -A corp.russian.data-import.database` and
out comes a perfectly valid SQLite database with a valid external data
import!
Change-Id: I5d6d15e67d0e4a7ff590fad06252be34f5d561fd
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7866
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Otherwise up to 1000 elements might be missing.
Change-Id: I20d6238424eec27f0e758e7737c9c31bcb81b23d
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7862
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>
This is an initial and kind of dumb table structure, but there's some
massaging that needs to be done before this makes more sense.
Change-Id: I441288b684ef86be507099bcc4ebf984598789c8
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7861
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
Google Cloud Run uses images to define services, so:
```shell
$ mg build :image
$ docker load <./result
$ docker tag website:latest gcr.io/wpcarros-infrastructure/website:latest
$ docker push gcr.io/wpcarros-infrastructure/website:latest
```
And then restart the service with `:latest`.
TODO: Figure-out some ~sane CI solution (maybe personal Buildkite).
Change-Id: I5734e3344779552aba7e0478321ba99610204e29
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7735
Reviewed-by: wpcarro <wpcarro@gmail.com>
Tested-by: BuildkiteCI
Trying to prune my monthly GCP bill, which is ~$60. Will run my website as a
stateless Google Cloud Run service and see if that'll help.
I still need to figure out what to do with my Quassel instance...
Change-Id: I934b55029f14132af74cabde5e0ddb9e2d3bb933
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7734
Tested-by: BuildkiteCI
Reviewed-by: wpcarro <wpcarro@gmail.com>
Adds the beginning of a tool which can import OpenCorpora data into a
SQLite database. This is quite a lot of toil and there's probably a
better way to do this, but overall becoming this intimately familiar
with the data structures is quite helpful for understanding what I
can/can't do with only this dataset.
Change-Id: Ieab33a8ce07ea4ac87917b9c8132226bbc6523b1
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7859
Reviewed-by: tazjin <tazjin@tvl.su>
Tested-by: BuildkiteCI
