Commit graph

10846 commits

Author SHA1 Message Date
Loo Rong Jie
f1c17a303a Revert migration for //absl/strings/... 2018-10-09 23:33:46 +08:00
Loo Rong Jie
73b908fbcf Migrate some internal base targets 2018-10-09 23:33:45 +08:00
Loo Rong Jie
027ebfdb8f Use ABSL_RUN_TESTS instead of BUILD_TESTING 2018-10-09 23:33:45 +08:00
Loo Rong Jie
fcc476cce5 Implement header-only target as INTERFACE lib 2018-10-09 23:33:44 +08:00
Loo Rong Jie
ac533e18db [CMake] Implement absl_cc_library as Bazel's cc_library
cc_library
2018-10-09 23:33:44 +08:00
Eelco Dolstra
9617a04354
Merge pull request #2454 from dtzWill/update/nixpkgs-1809
bump base nixpkgs used by default 18.03 -> 18.09
2018-10-09 13:56:33 +02:00
Vincent Ambo
623a84f6f8 refactor: Require directory instead of file path for cursors
The previous change, which makes journaldriver write the cursor
position in two steps, requires that journaldriver can write files
adjacent to the cursor position file itself.

Instead of simply guessing that this is possible (e.g. by changing the
file suffix), expect the user to provide a directory that
journaldriver can work with.
2018-10-09 11:38:41 +02:00
Vincent Ambo
61b2577a19 fix: Refuse to write empty journald cursors and inform users
Exits from `persist_cursor` early if the cursor received from journald
is an empty string.

We don't currently know if this actually happens (please see #2 for
more details), so an error message has been added that asks users to
report this if it ever occurs.
2018-10-09 11:38:41 +02:00
Vincent Ambo
c1ab78c05a fix: Write cursor into temporary file and move it
This deals with a potential issue where creating a new file in place
of an existing cursor position file may cause position files to be
empty.

The cause for this is that the newly created file will truncate the
previous content, if journaldriver is then terminated before it
completes the cursor write to this file, it will not have written a
valid cursor (or anything at all).

Potentially relates to #2
2018-10-09 11:38:41 +02:00
Eelco Dolstra
01bd66bf83
Merge pull request #2452 from ElvishJerricco/verify-sigs-overflow
Fix overflow when verifying signatures of content addressable paths
2018-10-08 12:03:03 +02:00
Vincent Ambo
7ec507f8cb chore: Bump version to 1.1.0
... in preparation of a new release!
2018-10-06 01:04:56 +02:00
Vincent Ambo
55c4943df1 docs(README): Add notes about error reporting & minor improvements 2018-10-06 00:51:26 +02:00
Vincent Ambo
86c25cc226 refactor: Replace reqwest library with ureq
This replaces reqwest with the more simplistic ureq library for
performing required HTTP requests.

Reqwest comes with a lot of (tokio-based) machinery for
high-performance requesting that is a bit out of scope for
journaldriver's needs.

This clocks in at 62 (!) fewer dependencies after the change, with
equivalent functionality. Wew.
2018-10-05 23:58:01 +02:00
Abseil Team
e821380d69 Export of internal Abseil changes.
--
e09635c907d471ab46e14fb8091988c92cb4e78e by Abseil Team <absl-team@google.com>:

Fix syntax error in documentation '}' -> ')'

PiperOrigin-RevId: 215744854

--
a9e2d036850748137355f4a07083f1052d5c3766 by Abseil Team <absl-team@google.com>:

WrapUnique's comment about avoiding explicit template specification
doesn't make clear what alternative there is when this hurts
readability. Add a comment about the approved alternative.

PiperOrigin-RevId: 215634917

--
596b8ff41ff70b27bff3a2369038c0fe7d13ae85 by Greg Falcon <gfalcon@google.com>:

Allow for a small amount of slop in timeout tests, as not all OSs we support give us the perfect timing behavior we're testing for here.

PiperOrigin-RevId: 215615172
GitOrigin-RevId: e09635c907d471ab46e14fb8091988c92cb4e78e
Change-Id: I30721294bac86510a49bb7f405149fc74c532abb
2018-10-05 10:53:08 -04:00
Eelco Dolstra
0fda9b22c7
Remove unnecessary typecast 2018-10-04 13:16:30 +02:00
Eelco Dolstra
38b960136d
Merge pull request #2468 from bhipple/docs/fixedOutputHashes
Docs: update fixed output hashes
2018-10-04 09:20:26 +02:00
Benjamin Hipple
65453e2d77 Docs: update fixed output hashes
`fetchurl` will now throw if given an `md5`, and the hashes have generally
been upgraded to avoid it and use `sha256` as a default. This updates the
documentation examples in the manual accordingly.
2018-10-04 02:37:15 +00:00
Abseil Team
f21d187b80 Export of internal Abseil changes.
--
1c1d6e2404dfc6caa022b335df5acdac6da50fe1 by Derek Mauro <dmauro@google.com>:

Fix the internal namespacing in unaligned_access.h

PiperOrigin-RevId: 215434506

--
17d4400aebf025a230690fc1c7a968ef8d85bbba by Eric Fiselier <ericwf@google.com>:

gtest depends on the GCC extension allowing variadic macros
to be passed a empty parameter pack for ..., but LLVM diagnoses
this as a GNU extension.

This patch suppresses the warning when building the absl tests.

PiperOrigin-RevId: 215426161

--
f2c49dde23a9f445b9de963f1bbe840ebb568b30 by Eric Fiselier <ericwf@google.com>:

Use EXPECT_DEATH_IF_SUPPORTED instead of EXPECT_DEATH.

This avoids breaking the test when gtest doesn't support
death tests.

PiperOrigin-RevId: 215423849

--
cd687c1e121709603f4fc3726b534f6a9c52cc89 by Eric Fiselier <ericwf@google.com>:

Disable LLVM's -Wmissing-variable-declarations in tests.

GCC's configuration already disables this via -Wno-missing-declarations,
this change makes LLVM do the same.

The warning would otherwise flag most tests which use ABSL_FLAG.

PiperOrigin-RevId: 215407429

--
d14098824c84e3a8c8f6fb920e0335fb48fe2010 by Eric Fiselier <ericwf@google.com>:

Fix local variable shadowing in city hash implementation.

PiperOrigin-RevId: 215407249

--
4b5e140ba743f0d231790a26c49083abb4329e2c by Abseil Team <absl-team@google.com>:

Make raw_hash_set::reserve 2X fast when reserve doesn't do any allocation.
Make raw_hash_set::reserve ~1% faster when reserve does some (128~4k) allocation.

PiperOrigin-RevId: 215348727

--
461161e65e04b801480aa117af2534c594654ccf by Eric Fiselier <ericwf@google.com>:

Internal change

PiperOrigin-RevId: 215272283

--
50413ae31ad3d3a177257416acd8ede47a17bff2 by Eric Fiselier <ericwf@google.com>:

Internal Change

PiperOrigin-RevId: 215233183

--
477be54c43d61019a8fe4e190e340eb52737d383 by Abseil Team <absl-team@google.com>:

Clarify misleading comment on ABSL_ATTRIBUTE_UNUSED

PiperOrigin-RevId: 215185496

--
2cafa2b5287507d3a946682aee9ab13af6d471c9 by Matt Kulukundis <kfm@google.com>:

Add support for absl::Hash to various absl in types.

PiperOrigin-RevId: 215039569

--
082248901991aa3d29be0ea3689c7f213cf0fd83 by Derek Mauro <dmauro@google.com>:

Remove an instance of HAS_GLOBAL_STRING from hash_function_defaults.h

PiperOrigin-RevId: 214989094

--
b929f61907f0786a6133e3a9d7287e339c0a0acb by Derek Mauro <dmauro@google.com>:

Internal import of Github #174
Fix code snippet in comment
https://github.com/abseil/abseil-cpp/pull/174

PiperOrigin-RevId: 214958849

--
f2c5e829eca11c352e121f56eefbf87083305023 by Derek Mauro <dmauro@google.com>:

Internal import of GitHub #173
Fix CMake build for absl::container.
https://github.com/abseil/abseil-cpp/pull/173

PiperOrigin-RevId: 214957796

--
d704f860f9fddafb99e34e6c5032e49f73874e10 by Abseil Team <absl-team@google.com>:

Internal change

PiperOrigin-RevId: 214828181
GitOrigin-RevId: 1c1d6e2404dfc6caa022b335df5acdac6da50fe1
Change-Id: I551de2b1ba0cbaf6856cd5959358cf6651179dea
2018-10-03 09:19:28 -04:00
Eelco Dolstra
d93db0ace0
Merge pull request #2464 from Profpatsch/docs-builtins-add-ids
manual: add XML IDs to all builtin list entries
2018-10-02 15:45:59 +02:00
Profpatsch
05cfc71cab manual: add XML IDs to all builtin list entries 2018-10-02 15:38:16 +02:00
Eelco Dolstra
aa5e47b2f4
Merge pull request #2456 from grahamc/s3-substituters
Update docs to describe how s3:// URLS does in fact support endpoint, region, and profile for upload
2018-10-01 22:47:00 +02:00
Graham Christensen
47ed3b282f
Merge pull request #2462 from matthewbauer/patch-10
nix-profile-daemon: remove cruft
2018-10-01 14:28:29 -04:00
Matthew Bauer
9cc876fb11
nix-profile-daemon: remove cruft
This removes part of the PATH that were being added automatically in multi-user installs:

- $HOME/.nix-profile/lib/kde4/libexec - shouldn't be needed anymore, we are now using kde5
- @localstatedir@/nix/profiles/default/lib/kde4/libexec - same as above
- @localstatedir@/nix/profiles/default - shouldn't ever contain binaries
2018-10-01 13:26:59 -05:00
Eelco Dolstra
5b7cfa487e
Add missing file 2018-10-01 11:58:54 +02:00
Loo Rong Jie
5441bbe1db Fix code snippet in comment (#174) 2018-09-28 13:41:11 -04:00
Stephan Dollberg
5aae0cffae Fix CMake build (#173)
* Fix CMake build after Swissmap addition

`absl::container` library now contains source files so needs to be
respectively built for CMake as well.

Switches `absl::container` from being a header library to a source
library.

* Add raw_hash_set_test to CMake test build
2018-09-28 13:35:51 -04:00
Eelco Dolstra
1e7b8deea7
Check requiredSystemFeatures for local builds
For example, this prevents a "kvm" build on machines that don't have
KVM.

Fixes #2012.
2018-09-28 16:10:27 +02:00
Eelco Dolstra
7ae7a38c9a
Move structured attrs handling into a separate class
This is primarily because Derivation::{can,will}BuildLocally() depends
on attributes like preferLocalBuild and requiredSystemFeatures, but it
can't handle them properly because it doesn't have access to the
structured attributes.
2018-09-28 14:32:46 +02:00
Eelco Dolstra
99d4bb2d4c
Remove obsolete check on system features 2018-09-28 13:28:15 +02:00
Eelco Dolstra
c9ba33870e
Support special attributes in structured attributes derivations
E.g. __noChroot and allowedReferences now work correctly. We also now
check that the attribute type is correct. For instance, instead of

  allowedReferences = "out";

you have to write

  allowedReferences = [ "out" ];

Fixes #2453.
2018-09-28 12:43:01 +02:00
Graham Christensen
51cbeec49a
Update docs to describe how s3:// URLS does in fact support endpoint, region, and profile for upload 2018-09-27 17:01:39 -04:00
Eelco Dolstra
63786cbd3b
S3BinaryCacheStore: Don't create buckets
This meant that making a typo in an s3:// URI would cause a bucket to
be created. Also it didn't handle eventual consistency very well. Now
it's up to the user to create the bucket.
2018-09-27 21:34:41 +02:00
Abseil Team
48cd2c3f35 Export of internal Abseil changes.
--
4eacae3ff1b14b1d309e8092185bc10e8a6203cf by Derek Mauro <dmauro@google.com>:

Release SwissTable - a fast, efficient, cache-friendly hash table.

https://www.youtube.com/watch?v=ncHmEUmJZf4

PiperOrigin-RevId: 214816527

--
df8c3dfab3cfb2f4365909a84d0683b193cfbb11 by Derek Mauro <dmauro@google.com>:

Internal change

PiperOrigin-RevId: 214785288

--
1eabd5266bbcebc33eecc91e5309b751856a75c8 by Abseil Team <absl-team@google.com>:

Internal change

PiperOrigin-RevId: 214722931

--
2ebbfac950f83146b46253038e7dd7dcde9f2951 by Derek Mauro <dmauro@google.com>:

Internal change

PiperOrigin-RevId: 214701684
GitOrigin-RevId: 4eacae3ff1b14b1d309e8092185bc10e8a6203cf
Change-Id: I9ba64e395b22ad7863213d157b8019b082adc19d
2018-09-27 15:28:12 -04:00
Eelco Dolstra
f74e0b4786
Merge pull request #2455 from grahamc/add-interactive-path
nix-shell: add bashInteractive to the start of the PATH, set SHELL
2018-09-27 21:15:50 +02:00
Graham Christensen
f11acbaf17
nix-shell: add bashInteractive to the start of the PATH, set SHELL
Tools which re-exec `$SHELL` or `$0` or `basename $SHELL` or even just
`bash` will otherwise  get the non-interactive bash, providing a
broken shell for the same reasons described in
https://github.com/NixOS/nixpkgs/issues/27493.

Extends c94f3d5575
2018-09-27 15:12:56 -04:00
Eelco Dolstra
7cba4214a8
Don't talk about a "current folder build output"
Presumably this refers to ./default.nix but the support for that in
'nix' is tenuous. Also folders are a Mac thing.
2018-09-27 21:01:19 +02:00
Vincent Ambo
45afa18846 feat(postgres): Compatibility with new associated error type
Changes the implementation of action execution to deal with the
returned associated errors.

The only sensible constraint on those errors that I could think of for
now is `Debug`, meaning that errors are now persisted as debug
messages.

This is not as nice to work with for a future implementation of
retryable actions as the equivalent in Haskell, but maybe an idea
shows up underway. The main issue is that most of the common error
types will not be implementing Serde traits, so serialization to/from
the same error type is difficult.

Adding an implementation constraint for JSON serialisation on error
types (i.e. `S::Error: Serialize + Deserialize`) would probably cause
headaches for users, especially if they are trying to use an
out-of-the-box error type or an error type wrapping foreign errors.

Det ska'kke være lett ...
2018-09-26 23:19:34 +02:00
Vincent Ambo
c4b94d8d2d feat(door): Use failure::Error as associated error type
Implements the associated error type for the FSM trait as
failure::Error. This makes it possible to fail gracefully in all
actions, for example in the updated definition of the `NotifyIRC`
action which now appends to a file.
2018-09-26 23:18:26 +02:00
Daiderd Jordan
ed25753501
nix doctor: reimplement profile warning without gcroot check
Calculating roots seems significantly slower on darwin compared to
linux.  Checking for /profile/ links could show some false positives but
should still catch most issues.
2018-09-26 23:13:17 +02:00
Vincent Ambo
37590ae0f6 feat(core): Add associated 'Error' type to FSM trait
Adds an associated 'Error' type that can be returned by actions when
an interpretation fails.
2018-09-26 23:05:50 +02:00
Will Dietz
d4f78a6b64 bump base nixpkgs used by default 18.03 -> 18.09 2018-09-26 15:37:13 -05:00
Vincent Ambo
406a90e8d6 feat(postgres): Implement initial (synchronous) actoin execution
Implements a simple model for executing actions which will run them in
sequence, synchronously, after advancing an FSM and committing the
initial transaction.

Note that multiple things are not yet taken into account:

* Error handling of actions (they can not currently fail)
* Retrying of actions
* Concurrency model

I started out by implementing the concurrency model similarly to the
green-threading method used in Hamingja (but using OS threads), but
slowly noticed that it may not be the best way to do that. It needs a
little bit of discussion.

Either way for most actions this method is fast enough to work for
implementing things on top of Finito's model.
2018-09-26 22:31:42 +02:00
Eelco Dolstra
9750430003
Ensure download thread liveness
* Don't wait forever for the client to remove data from the
  buffer. This does mean that the buffer can grow without bounds
  (e.g. when downloading is faster than writing to disk), but meh.

* Don't hold the state lock while calling the sink. The sink could
  take any amount of time to process the data (in particular when it's
  actually a coroutine), so we don't want to block the download
  thread.
2018-09-26 21:47:34 +02:00
Eelco Dolstra
98b2cc2e6e
Untabify 2018-09-26 21:39:06 +02:00
Eelco Dolstra
ed78582847
sinkToSource(): Start the coroutine lazily
In particular this causes copyStorePath() from HttpBinaryCacheStore to
only start a download if needed. E.g. if the destination LocalStore
goes to sleep waiting for the path lock and another process creates
the path, then LocalStore::addToStore() will never read from the
source so we don't have to do the download.
2018-09-26 21:19:34 +02:00
Vincent Ambo
3891ba84d5 test(postgres): Add test for insert_machine and advance
Adds a test for the two most important functions in Finito's
PostgreSQL backend.

These actually require a local Postgres database to be available when
running. Currently the connection details are hardcoded in the test.
2018-09-26 18:44:16 +02:00
Vincent Ambo
7e5592f0d1 fix(postgres): Minor fixes in Postgres queries and handling 2018-09-26 18:43:53 +02:00
Vincent Ambo
6254d05620 feat(door): Add serde instances for door FSM types 2018-09-26 18:43:24 +02:00
Abseil Team
e291c279e4 Export of internal Abseil changes.
--
87c9505f1d41b52486630ae4404b9060b89a6369 by Abseil Team <absl-team@google.com>:

Fix wording in a comment.

PiperOrigin-RevId: 214606367
GitOrigin-RevId: 87c9505f1d41b52486630ae4404b9060b89a6369
Change-Id: Icb95299b88e65840dfbc415b515412b24d578ce9
2018-09-26 12:23:01 -04:00
Vincent Ambo
6e35c083bf refactor(postgres): Minor changes to match actual table schema 2018-09-26 18:20:04 +02:00