tvl-depot/absl/base/internal/spinlock_wait.cc

82 lines
2.8 KiB
C++
Raw Normal View History

2017-09-19 22:54:40 +02:00
// Copyright 2017 The Abseil Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
2017-09-19 22:54:40 +02:00
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// The OS-specific header included below must provide two calls:
// AbslInternalSpinLockDelay() and AbslInternalSpinLockWake().
2017-09-19 22:54:40 +02:00
// See spinlock_wait.h for the specs.
#include <atomic>
#include <cstdint>
#include "absl/base/internal/spinlock_wait.h"
#if defined(_WIN32)
#include "absl/base/internal/spinlock_win32.inc"
#elif defined(__linux__)
#include "absl/base/internal/spinlock_linux.inc"
#elif defined(__akaros__)
#include "absl/base/internal/spinlock_akaros.inc"
2017-09-19 22:54:40 +02:00
#else
#include "absl/base/internal/spinlock_posix.inc"
#endif
namespace absl {
ABSL_NAMESPACE_BEGIN
2017-09-19 22:54:40 +02:00
namespace base_internal {
// See spinlock_wait.h for spec.
uint32_t SpinLockWait(std::atomic<uint32_t> *w, int n,
const SpinLockWaitTransition trans[],
base_internal::SchedulingMode scheduling_mode) {
Export of internal Abseil changes. -- ee19e203eca970ff88e8f25ce4e19c32e143b988 by Jon Cohen <cohenjon@google.com>: Exception safety testing no longer uses absl::optional PiperOrigin-RevId: 220336204 -- 460666eb0b316a8b4aeedc589644d53b05251bd1 by Derek Mauro <dmauro@google.com>: Rework SwissTable SSE2 support - Use SSE2 on MSVC when available https://github.com/abseil/abseil-cpp/issues/210 - Emulate _mm_cmpgt_epi8 with other SSE2 instructions when using -funsigned-char under GCC https://github.com/abseil/abseil-cpp/issues/209 PiperOrigin-RevId: 220312351 -- 1f4318ecedf8d539b7b698eb803d613ad6b69278 by Abseil Team <absl-team@google.com>: Change CollectPerfectRatios to use 10 trials to smooth out the outliers in the sample. PiperOrigin-RevId: 220286579 -- 6755abc2673553a7f578bb29c6e9ca8d991bc9c8 by Abseil Team <absl-team@google.com>: Internal change PiperOrigin-RevId: 220274307 -- 8645b6187329ebf0aaf3c2de2888ba44466cd879 by Abseil Team <absl-team@google.com>: * #endif for a header guard should reference the guard macro in a comment PiperOrigin-RevId: 220206868 -- 3987a7ad11319230910931cd2468b60b3fd1b85c by Gennadiy Civil <misterg@google.com>: Internal Change PiperOrigin-RevId: 220136674 -- cc908c1db2ee0d4523dc813e33f600583bb986c5 by Abseil Team <absl-team@google.com>: absl: fix backoff logic in SpinLockWait There are 3 bugs in loop variable handling: 1. It starts with 0, but AbslInternalSpinLockDelay ignores loop == 0. So it does not actually wait when it should. 2. loop is incremented after successful state changes, but it should not (why would be increase backoff delay after that?). 3. loop is incremented after CAS failures, but it should not (why would be increase backoff delay after that?). Use the same handling of loop as used in SpinLock. PiperOrigin-RevId: 220136079 -- a0a1c6ef5910ebd28e07215d7df03cc0da0b3eed by Abseil Team <absl-team@google.com>: absl: relax unnecessarily strong memory ordering in SpinLock::SlowLock We don't need to acquire visibility over anything when setting kSpinLockSleeper. Replace the confusing and unnecessarily strong memory order with relaxed. PiperOrigin-RevId: 220023380 -- c50858b51af28b9fca1a62616324f85f3e84ea74 by Tom Manshreck <shreck@google.com>: Update comments in flat_hash_map, node_hash_{set, map} and the containers developer guide PiperOrigin-RevId: 219938692 -- e87b7d1a5f61e165b1c44d3b16d8d967197cdfce by CJ Johnson <johnsoncj@google.com>: Rearranges the public methods of InlinedVector and cleans up the comments PiperOrigin-RevId: 219896257 -- f3234c466f792e0fc4bfd21fc7919dba5e679375 by CJ Johnson <johnsoncj@google.com>: Adds branch prediction to exceptional early exit cases of inlined vector's API PiperOrigin-RevId: 219887173 -- 4dfccf1a81ca0425912d3da25a8470f78c532ce4 by CJ Johnson <johnsoncj@google.com>: Fixes the InlinedVector public interface to use the allocator type references instead of assuming the type Also cleans up some cruft in formatting and comments PiperOrigin-RevId: 219878876 -- 4bb6a2b892abb10bd6a424db7e94ed8640802470 by Tom Manshreck <shreck@google.com>: Add comments on constructor and assignment operator support to flat_hash_set PiperOrigin-RevId: 219825338 -- c23f973e2f7f4feea0da36bf8a9c3f8a8954bb74 by Abseil Team <absl-team@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 219823847 GitOrigin-RevId: ee19e203eca970ff88e8f25ce4e19c32e143b988 Change-Id: I288c927ca481dc57340420dbb4c278a05cf15e83
2018-11-06 22:01:25 +01:00
int loop = 0;
for (;;) {
2017-09-19 22:54:40 +02:00
uint32_t v = w->load(std::memory_order_acquire);
int i;
for (i = 0; i != n && v != trans[i].from; i++) {
}
if (i == n) {
Export of internal Abseil changes. -- ee19e203eca970ff88e8f25ce4e19c32e143b988 by Jon Cohen <cohenjon@google.com>: Exception safety testing no longer uses absl::optional PiperOrigin-RevId: 220336204 -- 460666eb0b316a8b4aeedc589644d53b05251bd1 by Derek Mauro <dmauro@google.com>: Rework SwissTable SSE2 support - Use SSE2 on MSVC when available https://github.com/abseil/abseil-cpp/issues/210 - Emulate _mm_cmpgt_epi8 with other SSE2 instructions when using -funsigned-char under GCC https://github.com/abseil/abseil-cpp/issues/209 PiperOrigin-RevId: 220312351 -- 1f4318ecedf8d539b7b698eb803d613ad6b69278 by Abseil Team <absl-team@google.com>: Change CollectPerfectRatios to use 10 trials to smooth out the outliers in the sample. PiperOrigin-RevId: 220286579 -- 6755abc2673553a7f578bb29c6e9ca8d991bc9c8 by Abseil Team <absl-team@google.com>: Internal change PiperOrigin-RevId: 220274307 -- 8645b6187329ebf0aaf3c2de2888ba44466cd879 by Abseil Team <absl-team@google.com>: * #endif for a header guard should reference the guard macro in a comment PiperOrigin-RevId: 220206868 -- 3987a7ad11319230910931cd2468b60b3fd1b85c by Gennadiy Civil <misterg@google.com>: Internal Change PiperOrigin-RevId: 220136674 -- cc908c1db2ee0d4523dc813e33f600583bb986c5 by Abseil Team <absl-team@google.com>: absl: fix backoff logic in SpinLockWait There are 3 bugs in loop variable handling: 1. It starts with 0, but AbslInternalSpinLockDelay ignores loop == 0. So it does not actually wait when it should. 2. loop is incremented after successful state changes, but it should not (why would be increase backoff delay after that?). 3. loop is incremented after CAS failures, but it should not (why would be increase backoff delay after that?). Use the same handling of loop as used in SpinLock. PiperOrigin-RevId: 220136079 -- a0a1c6ef5910ebd28e07215d7df03cc0da0b3eed by Abseil Team <absl-team@google.com>: absl: relax unnecessarily strong memory ordering in SpinLock::SlowLock We don't need to acquire visibility over anything when setting kSpinLockSleeper. Replace the confusing and unnecessarily strong memory order with relaxed. PiperOrigin-RevId: 220023380 -- c50858b51af28b9fca1a62616324f85f3e84ea74 by Tom Manshreck <shreck@google.com>: Update comments in flat_hash_map, node_hash_{set, map} and the containers developer guide PiperOrigin-RevId: 219938692 -- e87b7d1a5f61e165b1c44d3b16d8d967197cdfce by CJ Johnson <johnsoncj@google.com>: Rearranges the public methods of InlinedVector and cleans up the comments PiperOrigin-RevId: 219896257 -- f3234c466f792e0fc4bfd21fc7919dba5e679375 by CJ Johnson <johnsoncj@google.com>: Adds branch prediction to exceptional early exit cases of inlined vector's API PiperOrigin-RevId: 219887173 -- 4dfccf1a81ca0425912d3da25a8470f78c532ce4 by CJ Johnson <johnsoncj@google.com>: Fixes the InlinedVector public interface to use the allocator type references instead of assuming the type Also cleans up some cruft in formatting and comments PiperOrigin-RevId: 219878876 -- 4bb6a2b892abb10bd6a424db7e94ed8640802470 by Tom Manshreck <shreck@google.com>: Add comments on constructor and assignment operator support to flat_hash_set PiperOrigin-RevId: 219825338 -- c23f973e2f7f4feea0da36bf8a9c3f8a8954bb74 by Abseil Team <absl-team@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 219823847 GitOrigin-RevId: ee19e203eca970ff88e8f25ce4e19c32e143b988 Change-Id: I288c927ca481dc57340420dbb4c278a05cf15e83
2018-11-06 22:01:25 +01:00
SpinLockDelay(w, v, ++loop, scheduling_mode); // no matching transition
} else if (trans[i].to == v || // null transition
2017-09-19 22:54:40 +02:00
w->compare_exchange_strong(v, trans[i].to,
std::memory_order_acquire,
std::memory_order_relaxed)) {
if (trans[i].done) return v;
}
}
}
static std::atomic<uint64_t> delay_rand;
// Return a suggested delay in nanoseconds for iteration number "loop"
int SpinLockSuggestedDelayNS(int loop) {
// Weak pseudo-random number generator to get some spread between threads
// when many are spinning.
uint64_t r = delay_rand.load(std::memory_order_relaxed);
r = 0x5deece66dLL * r + 0xb; // numbers from nrand48()
delay_rand.store(r, std::memory_order_relaxed);
if (loop < 0 || loop > 32) { // limit loop to 0..32
loop = 32;
}
Export of internal Abseil changes. -- 5804cc13b413412988248835459b90cd15ec43d9 by Abseil Team <absl-team@google.com>: Mark raw_hash_set::clear() with the ABSL_ATTRIBUTE_REINITIALIZES attribute. This prevents false positives in the clang-tidy check bugprone-use-after-move; it allows reset() to be called on a moved-from raw_hash_set without any warnings, and the raw_hash_set will thereafter be regarded as initialized again. PiperOrigin-RevId: 230717196 -- ff5961a5600ae19b69a9cba6912126cdf2858f38 by CJ Johnson <johnsoncj@google.com>: Swaps DisableIfIntegral<> for EnableIfInputIterator<> for Iterator member functions of InlinedVector PiperOrigin-RevId: 230559521 -- 3f9754ccbeecbd40f235c6f2465279e045ff51d9 by Derek Mauro <dmauro@google.com>: Import GitHub PR 254 https://github.com/abseil/abseil-cpp/pull/254 Fixes warnings from -Wclass-memaccess (base_internal::ThreadIdentity? with no trivial copy-assignment). PiperOrigin-RevId: 230536048 -- 8af03a654ce9a4a7f55384bc7eb1ed64878ac2ec by Chris Kennelly <ckennelly@google.com>: absl: cap SpinLock backoff to 4ms The current backoff logic has 3 problems: 1. It can produce too high values (up to 256ms), which can negatively affect tail latency. The value was chosen long time ago and now it's a good idea to reconsider it. 2. It does not have low bound, so on any iteration it can produce a very small value that will lead to unnecessary cpu consumption. 3. It does not increase low bound with the number of iterations. So if the SpinLock is actually somehow locked for a very prolonged time, a waiter can still wake periodically. Rework the logic to solve these problems. Add lower bound of 128us, no code should rely on absence of episodic delays in this range as they can occur everywhere. Lower upper bound to 4ms. A thread sleeping for 4ms does not consume significant cpu time (see below). Grow lower bound with the number of iterations. This is cpu consumption of a process doing usleep(x) in a loop (sampled with ps): 64us -> 4.0% 128us -> 2.7% 256us -> 3.5% 512us -> 2.8% 1024us -> 1.6% 2048us -> 0.6% 4096us -> 0.3% 8192us -> 0.0% Few millisecond sleeps do not consume significant time. PiperOrigin-RevId: 230534015 -- 37ebba92289ca556cb2412cd8b3cb4c1ead3def7 by Samuel Benzaquen <sbenza@google.com>: Add override and dispose hooks to the hashtable sampler. PiperOrigin-RevId: 230353438 -- 89c8f90175233ce9964eb3412df04e8a3cff0c0f by Andy Getzendanner <durandal@google.com>: Fix a comment typo. PiperOrigin-RevId: 229986838 GitOrigin-RevId: 5804cc13b413412988248835459b90cd15ec43d9 Change-Id: Iedb5e2cc9c0b924635c1c87b537780ab6b5b899f
2019-01-24 16:23:40 +01:00
const int kMinDelay = 128 << 10; // 128us
// Double delay every 8 iterations, up to 16x (2ms).
int delay = kMinDelay << (loop / 8);
// Randomize in delay..2*delay range, for resulting 128us..4ms range.
return delay | ((delay - 1) & static_cast<int>(r));
2017-09-19 22:54:40 +02:00
}
} // namespace base_internal
ABSL_NAMESPACE_END
2017-09-19 22:54:40 +02:00
} // namespace absl