tvl-depot/absl/base/internal/spinlock.cc

235 lines
9.3 KiB
C++
Raw Normal View History

2017-09-19 22:54:40 +02:00
// Copyright 2017 The Abseil Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
2017-09-19 22:54:40 +02:00
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
#include "absl/base/internal/spinlock.h"
#include <algorithm>
#include <atomic>
Changes imported from Abseil "staging" branch: - 06c8c67f5a564d00696e023060f05a5c34e7e164 IWYU | absl/base by Juemin Yang <jueminyang@google.com> - 2b1a054a09bda55843b449843b2a125741e936e7 Internal refactoring by Greg Miller <jgm@google.com> - f43f7f1f91bee26b5ddcd0c5bbbc47cb977aef77 Make std::hash<absl::optional<T>> to be standard compliant: by Xiaoyi Zhang <zhangxy@google.com> - 539bad2ebc22e610e1f292285a30a87945bc663e Update utility.h to Abseil standards by Tom Manshreck <shreck@google.com> - d05ec10a5f16a5d6640e0db91ecc7ab3ea971fd4 Add a test for absl::Barrier. by Derek Mauro <dmauro@google.com> - d707e27acb3c06f0d74c5f7ad7861e3841a5471f Run leak-checking tool over all outbound code. by Daniel Katz <katzdm@google.com> - 55f07f482a50422b8f99f7176374a19d0d473c5f Add alignas(16) to uint128. by Alex Strelnikov <strel@google.com> - 94999b7edde82308f736fb939501537ee9edbca6 Update attributes.h to Abseil standards by Tom Manshreck <shreck@google.com> - 321bed0061c41b53d0206ad4865528c00dd6d825 Test git merge + piper cl process by Juemin Yang <jueminyang@google.com> - 69920e7351a1053a7f4940bbde1768e839ce84bc Adds support for "/etc/localtime" as an embedded time zon... by Greg Miller <jgm@google.com> - 6839c06bf232903d3a9cbffa6eb2c960db78e67b Add copyright notices to inlined_vector code. by Greg Falcon <gfalcon@google.com> - 4e2714f6266263515cdfd31675c30c6ed6f98e1a Adding Apache 2.0 License by Gennadiy Civil <misterg@google.com> - 7402e7594016a4cd0a8b823fe6bc1bad1874bb85 IWYU | absl/utility by Juemin Yang <jueminyang@google.com> - 271a3812337eed97c412042738482688a80e19bd IWYU | absl/memory by Juemin Yang <jueminyang@google.com> - 32bda13a8098c2b06e25a5cf7bb782d6b79eb006 IWYU | absl/numeric by Juemin Yang <jueminyang@google.com> - 62d375cedc133108904bc06e340e303091a565df Remove "no_test"-annotations on span_test_noexceptions. by Daniel Katz <katzdm@google.com> - ebcbae9a55a93a7f1bb6862edc2715a6d9877206 Move CI-testing support files out of public-facing reposi... by Daniel Katz <katzdm@google.com> - d3f05eff4daa6030bfacb31cca0f9213fb702247 Fixes ToInt64Minutes() and ToInt64Hours() to properly sat... by Greg Miller <jgm@google.com> - b8dfae3facb6bb002622f083a10d14448f19e6e0 Fix typo. by Abseil Team <absl-team@google.com> - 150f03baa0afa231c2fc01597ea2321da586caba Update README.md by Yilei Yang <yileiyang@google.com> - 05276aa837dd081686518fd27bda4bd206ac4443 Adding Apache 2.0 License by Gennadiy Civil <misterg@google.com> - 37bf8e223e79ad06a195e28db9499e0c3d140f73 IWYU | absl/container by Juemin Yang <jueminyang@google.com> - 49164928f220978a32f88d16a55549bdf871daef IWYU | absl/algorithm by Juemin Yang <jueminyang@google.com> - a6804734e129039f9580a4fcd0f66425d0d0ac30 Move throw delegate wrappers to an internal namespace. by Greg Falcon <gfalcon@google.com> - ac83e73f67f593e2aff957b2be0b28e59c552a71 Fix error in comment stripping directives. by Greg Falcon <gfalcon@google.com> - e018a24185a984e787fb81a75fc35b74ad3a4d3d Update copyright headers all BUILD files by Gennadiy Civil <misterg@google.com> - a3be0990bfd76b0dec76bd85cecfa4dcec68b3ea Fix closing namespace comment typo. by Abseil Team <absl-team@google.com> - be3e3c4327e4f83949e0f29fd7a190d7eaa8b50b Update TODO by Abseil Team <absl-team@google.com> - f56a5d6f72685d92bb9c2905841b950d8177210c Add test-coverage for leak-sanitizer. by Daniel Katz <katzdm@google.com> - 7694bf161c7e00fdd08bfadc2aaf8e0fb09335f8 span.h: further touch up wording around the std::initiali... by Abseil Team <absl-team@google.com> - 3a12e081c0f8b359973e020d1e91f65356548ebc Update time.h to Abseil standards by Tom Manshreck <shreck@google.com> - 48d28f6468129420f4b20d451dca8e08012a7a77 Remove references to google from comments in Abseil. by Greg Falcon <gfalcon@google.com> - 773e34402d15fcad6370d5ed2430482d17db910d Rename the ExpectTime macro to match Abseil naming conven... by Greg Falcon <gfalcon@google.com> - 774d2ff1fe26c7313b301ff203e83e1aaac86627 Internal change. by Daniel Katz <katzdm@google.com> - 2e8a5830e95c8a1b839721bb2f1d4f5c85b9fb60 Fix typo in comment (missing '*' on a pointer). by Abseil Team <absl-team@google.com> - 458106feb707cf9609dd243713bde44aa9679e2a Correct capitalization: github -> GitHub by Yilei Yang <yileiyang@google.com> - af440725f02c2a83ca5cbaf176e1142f9e9d9b2f Update copyright headers by Gennadiy Civil <misterg@google.com> - 05b1118cce4ab87d23c33d48e64a96bcfec08761 Update copyright headers by Gennadiy Civil <misterg@google.com> - d5c6669a62d047156bb77055c5da03ee1b3c61b9 Update Abseil README to include descriptions of the inclu... by Tom Manshreck <shreck@google.com> - 3cd7e4663dddc840087469a6495f6cf433bfad8d Update copyright headers in //base by Tom Manshreck <shreck@google.com> - 7a876da657cd6698c5da2008a582d52eedc85dd1 Update strings overview with robust string library docume... by Tom Manshreck <shreck@google.com> - d9e3d0768d6f1c77d30992bdbef7b47ec92994bb Update copyright headers all BUILD files by Abseil Team <absl-team@google.com> - 6fe942728bceb0625f7c79b2840c4a6154d076b3 Make InlinedVector, FixedArray, and Span's at() throw on ... by Jon Cohen <cohenjon@google.com> - 5b52d5ec6cb9fbb07fc2e2fa020bd3eeb48c4953 Update clock.h to Abseil standards by Tom Manshreck <shreck@google.com> - c03c1ca3aee8bb7e40aa0315f6c432d31a72c30c Update //algorithm copyright headers by Tom Manshreck <shreck@google.com> - d46f40ddc596aaacb0459351d0e4aa6871289fa2 Temporarily prevent running mutex_test on crosstool17, wh... by Jon Cohen <cohenjon@google.com> - 61f11476189df68edfb9908308d677a91f03ff67 Update copyright headers in //container by Tom Manshreck <shreck@google.com> - 91832c00948954edf0b3dda12219c9a0202421ac Update copyright headers for //synchronization by Tom Manshreck <shreck@google.com> - 4e09100264b4585af6b4508ff35b9c627ac1f1ce Update copyright headers all BUILD files by Gennadiy Civil <misterg@google.com> - 13a0e8aebedec0f95b33750cbcd6b5548619b2a5 Update copyright headers in //memory by Tom Manshreck <shreck@google.com> - 63e1b9d4fdbcdf097e5276050ad1f76f0053e553 Update copyright headers for //strings (+ one from //nume... by Tom Manshreck <shreck@google.com> - 0108e7cfc50777a94c56d00e9c305161364df341 Convert ASSERT and EXPECT to ABSL_RAW_CHECK in helper fun... by Derek Mauro <dmauro@google.com> - 0122306fe47a3093248254a1b475c3a1d82abec5 Internal change by Abseil Team <absl-team@google.com> - 89c0c2698c98a12cd63172eeb02063b2f67e7c81 #absl Fix comment. by Abseil Team <absl-team@google.com> - 6621cc1ff54800e0aadb5e3071dbaa84b2077ceb Publishing contributing guidelines. by Gennadiy Civil <misterg@google.com> - e48c5be3c75e794b3e3827d40915b01fe1a1afc5 Avoid PRIdPTR. Cast to long long and use %lld insted. by Abseil Team <absl-team@google.com> - 2640ea4a260d89b94b07a3142660327e47db33fd No algorithmic changes. by Abseil Team <absl-team@google.com> - 1bc6c1bad17754f5d84963bf1d0db279402a0a1d Internal change. by Derek Mauro <dmauro@google.com> - 6845d24733e8c95bebde825ba78a2abfd9e35bdb span.h: fix up incorrect wording around lvalues. by Abseil Team <absl-team@google.com> - d8f5caee721e252e5f9b1080fb996363f498ac28 Add more exaustive Mutex testing. by Derek Mauro <dmauro@google.com> - e8b4cb053eb98858eef10cc53280b6ed5d6815a7 Change Span::subspan to not call into a deprecated constr... by Jon Cohen <cohenjon@google.com> - 49c36a82b3114926390557670aaaf0ea25b5760c s/std::size_t/size_t/g by Jon Cohen <cohenjon@google.com> - e17487c3c4d4a99f2fd8bc3e42176fc3171614d1 Account for the case of timeval::tv_sec being smaller than by Abseil Team <absl-team@google.com> - ecbb89d5fb98483e777c03d97ac02d7b7b54985e Alias absl::string_view to std::string_view when C++17 st... by Xiaoyi Zhang <zhangxy@google.com> - 6820e5a51459cdbb6a423cbae25a0cc839c85d44 Internal cleanup. by Xiaoyi Zhang <zhangxy@google.com> - 6976469b76a6faaf4111a24ddb37f40211ffadae More Span constexpr by Jon Cohen <cohenjon@google.com> - 8521c8956eee1125b7759eb272ec4a5a86fcefc5 #absl Fix comment. by Abseil Team <absl-team@google.com> - 20eae7a67fde5dd809aa47e5f7de8a493701645e Embed enough zoneinfo data to make time:time_test (under ... by Abseil Team <absl-team@google.com> - 841f5d98ceef4a423839ea73ee06c2f47a9b9680 Clean up macros in attributes.h | ATTRIBUTE_INITIAL_EXEC by Juemin Yang <jueminyang@google.com> - 83d8b36656e47919b5d0bac82eece897e195697e Update any.h comments to Abseil standards by Tom Manshreck <shreck@google.com> - bb3fae11d3459eeae2f63bfd22e65d3193187cc8 Update type_traits.h comments to Abseil standards by Tom Manshreck <shreck@google.com> - 992e1b07c0dec64271f8c44f22fd8df3734d0c47 Renamespace CycleClock code. by Greg Falcon <gfalcon@google.com> - 08d6fb0594098493ffbc0e737405182638122e7e Eliminate more existing lint warnings by Gennadiy Rozental <rogeeff@google.com> - 1cc6fdc71eb777497239f8c3e9168e6c9d40ea53 Update optional.h to Abseil standards by Tom Manshreck <shreck@google.com> - baa91747aa55009a9eb31b6072e33db06cfce2d2 Enforce internal namespace symbol reference policy by Gennadiy Rozental <rogeeff@google.com> - 939251e39342ce559e5d23fe43799671581f7cf5 Add CycleClock scaling shift to mitigate a possible overf... by Derek Mauro <dmauro@google.com> - c6dfdeecea0c7470938bed47c99ea2b2a95889d8 Add constexpr tests for absl::make_optional(). by Xiaoyi Zhang <zhangxy@google.com> - 509e949b992db33041d840746fbd05cc01cb206e Alias absl::optional to std::optional when C++17 std::opt... by Xiaoyi Zhang <zhangxy@google.com> - a1ae6d96a8826ba75281cac8632a766b5856acaf Remove no_test_* tags from span_test to increase test cov... by Xiaoyi Zhang <zhangxy@google.com> - 3c2a43cc09791723c8a324836629644ac44cb9c8 Remove accidental bits of Google-internal code and short ... by Greg Falcon <gfalcon@google.com> - 4874d49d496ac0b6ec36f4280a14b2159e7af930 Replaces the macro-generated Duration factory functions e... by Greg Miller <jgm@google.com> - 90e62695e03cb4a57e137ca0c3e116b1d802db57 Fix namespacing for a couple files in base/internal, and ... by Greg Falcon <gfalcon@google.com> - b0e6e00e34f967924849aaf8c123bba068f093e3 Publishing contributing guidelines. by Gennadiy Civil <misterg@google.com> - d74eafbccc3dffa6c25f9b6a2219425a24b5a959 Internal change by Abseil Team <absl-team@google.com> - 27477badbbf720265f5b9509b6c0e01913dc0a9f Update escaping.h comments to Abseil standards by Tom Manshreck <shreck@google.com> - 67002f55738319c2875197c3b6282de215ec250d adds absl namespace to debugging/stacktrace.h by Behzad Nouri <bnouri@google.com> - e608018f7faa384d5b202ac0a4c7a0d5166f4d9c Update string_view.h comments to Abseil standards by Tom Manshreck <shreck@google.com> - e884f04d4c648e01ed7dcde2fda80c24e8452047 Exclude strings/ files we are not releasing from OSS univ... by Gennadiy Rozental <rogeeff@google.com> - 3f4c4032ed520f2dd10a81d58ef4f399c001c5cc Strip out eventmanager reference from release. by Gennadiy Rozental <rogeeff@google.com> - fb0f1c204793c3792bad101dbaa734e7c2a35887 Fix copybara strip comments by Gennadiy Rozental <rogeeff@google.com> - 3eaaac942f77c3d41d63d414630403bfd0f6b70c Strip out style guide waivers. by Gennadiy Rozental <rogeeff@google.com> - 020e045058173178b51266b99a2a5dc9ed921960 substitute_test portability | MUST_USE_RESULT cast-to-voi... by Juemin Yang <jueminyang@google.com> - 86c093bf81d80ff537ed5e8b89225ce75a636220 Internal change. by Derek Mauro <dmauro@google.com> - 330375eb952fe78276e75631a28e750d5bfdb198 Prefer absl::FixedTimeZone() over loading "Etc/GMT[-+]<N>". by Abseil Team <absl-team@google.com> - 2e07ebee46a8201adc0dfd2c4ddb3df76e524357 Internal change. by Derek Mauro <dmauro@google.com> - 1f0c8b78c8ebd66f14cdf39fcba9f4c9986dcdca ::absl -> absl by Gennadiy Rozental <rogeeff@google.com> - ad163566d12ea08f1da2c23931eeacfffc564139 Avoid old style loops where possible. by Gennadiy Rozental <rogeeff@google.com> - bce2108818fe57b5617ce0090ddd4f753808f0a1 Update comments in str_cat.h in line with recent changes. by Abseil Team <absl-team@google.com> - cfd593a80f4897256f2ce1ea0be55dc14e3fcad4 Copybara-out gtl aliases. by Gennadiy Rozental <rogeeff@google.com> - 584f1524d717993c1a16093caccd9ed2b1e5409e Fix a warning for Windows/Kokoro time_test.cc. by Daniel Katz <katzdm@google.com> (And 562 more changes) GitOrigin-RevId: 06c8c67f5a564d00696e023060f05a5c34e7e164 Change-Id: I89907a6188fe7de05da400bf49ddfeba242aff8e
2017-09-20 02:15:26 +02:00
#include <limits>
2017-09-19 22:54:40 +02:00
- abacaab4b11a69dd4db627bd183571d7cabb8def Refinement to previous time.h edit (in this same github p... by Greg Falcon <gfalcon@google.com> - 64db19b773134c6c8004e3b23c9ca892efbf8bae Move SpinLock's adaptive spin count computation from a st... by Derek Mauro <dmauro@google.com> - 6f9533fb44a52485a7c2bbb9b4efc7bf8d6c359a Import of CCTZ from GitHub. by Abseil Team <absl-team@google.com> - a211d7255c986e8dd4ceada362c0d054a6a1969a Cleanup exception flags by Abseil Team <absl-team@google.com> - babdb29c590126fe9bba5229fe91034b5b5c358a Release time benchmarks. by Alex Strelnikov <strel@google.com> - 5803b32a3ff123d1fb57a0c471d199c818357c9f Release memutil microbenchmarks. by Alex Strelnikov <strel@google.com> - 5357d4890d30e80c53beb05af32500fb20e9402b Add parens around expansion of ABSL_PREDICT_{FALSE,TRUE} ... by Abseil Team <absl-team@google.com> - 32023f61a239a5f6b1c59e577bfe81b179bbcd2d Reformat build rule tag. by Alex Strelnikov <strel@google.com> - 833758ecf2b0cf7a42bbd50b5b127e416425c168 Release uint128 microbenchmarks. by Alex Strelnikov <strel@google.com> - c115a9bca1f944b90fdc78a56b2de176466b124f Disambiguate bitwise-not of size_type by Abseil Team <absl-team@google.com> - f6905f5b5f6e425792de646edafde440548d9346 Updates ConstructorTracker and TrackedObjects with 1) a m... by Abseil Team <absl-team@google.com> - 147c553bdd5d2db20a38f75c4d1ef973d6c709c5 Changes the absl::Duration factory functions to disallow ... by Greg Miller <jgm@google.com> - dba2b96d11b5264546b283ba452f2de1303b0f07 White space fix by Alex Strelnikov <strel@google.com> GitOrigin-RevId: abacaab4b11a69dd4db627bd183571d7cabb8def Change-Id: I6fa34f20d0b2f898e7b5475a603111413bb80a67
2018-06-08 17:14:48 +02:00
#include "absl/base/attributes.h"
2017-09-19 22:54:40 +02:00
#include "absl/base/internal/atomic_hook.h"
#include "absl/base/internal/cycleclock.h"
#include "absl/base/internal/spinlock_wait.h"
#include "absl/base/internal/sysinfo.h" /* For NumCPUs() */
- abacaab4b11a69dd4db627bd183571d7cabb8def Refinement to previous time.h edit (in this same github p... by Greg Falcon <gfalcon@google.com> - 64db19b773134c6c8004e3b23c9ca892efbf8bae Move SpinLock's adaptive spin count computation from a st... by Derek Mauro <dmauro@google.com> - 6f9533fb44a52485a7c2bbb9b4efc7bf8d6c359a Import of CCTZ from GitHub. by Abseil Team <absl-team@google.com> - a211d7255c986e8dd4ceada362c0d054a6a1969a Cleanup exception flags by Abseil Team <absl-team@google.com> - babdb29c590126fe9bba5229fe91034b5b5c358a Release time benchmarks. by Alex Strelnikov <strel@google.com> - 5803b32a3ff123d1fb57a0c471d199c818357c9f Release memutil microbenchmarks. by Alex Strelnikov <strel@google.com> - 5357d4890d30e80c53beb05af32500fb20e9402b Add parens around expansion of ABSL_PREDICT_{FALSE,TRUE} ... by Abseil Team <absl-team@google.com> - 32023f61a239a5f6b1c59e577bfe81b179bbcd2d Reformat build rule tag. by Alex Strelnikov <strel@google.com> - 833758ecf2b0cf7a42bbd50b5b127e416425c168 Release uint128 microbenchmarks. by Alex Strelnikov <strel@google.com> - c115a9bca1f944b90fdc78a56b2de176466b124f Disambiguate bitwise-not of size_type by Abseil Team <absl-team@google.com> - f6905f5b5f6e425792de646edafde440548d9346 Updates ConstructorTracker and TrackedObjects with 1) a m... by Abseil Team <absl-team@google.com> - 147c553bdd5d2db20a38f75c4d1ef973d6c709c5 Changes the absl::Duration factory functions to disallow ... by Greg Miller <jgm@google.com> - dba2b96d11b5264546b283ba452f2de1303b0f07 White space fix by Alex Strelnikov <strel@google.com> GitOrigin-RevId: abacaab4b11a69dd4db627bd183571d7cabb8def Change-Id: I6fa34f20d0b2f898e7b5475a603111413bb80a67
2018-06-08 17:14:48 +02:00
#include "absl/base/call_once.h"
2017-09-19 22:54:40 +02:00
// Description of lock-word:
// 31..00: [............................3][2][1][0]
//
// [0]: kSpinLockHeld
// [1]: kSpinLockCooperative
// [2]: kSpinLockDisabledScheduling
// [31..3]: ONLY kSpinLockSleeper OR
// Wait time in cycles >> PROFILE_TIMESTAMP_SHIFT
//
// Detailed descriptions:
//
// Bit [0]: The lock is considered held iff kSpinLockHeld is set.
//
// Bit [1]: Eligible waiters (e.g. Fibers) may co-operatively reschedule when
// contended iff kSpinLockCooperative is set.
//
// Bit [2]: This bit is exclusive from bit [1]. It is used only by a
// non-cooperative lock. When set, indicates that scheduling was
// successfully disabled when the lock was acquired. May be unset,
// even if non-cooperative, if a ThreadIdentity did not yet exist at
// time of acquisition.
//
// Bit [3]: If this is the only upper bit ([31..3]) set then this lock was
// acquired without contention, however, at least one waiter exists.
//
// Otherwise, bits [31..3] represent the time spent by the current lock
// holder to acquire the lock. There may be outstanding waiter(s).
namespace absl {
ABSL_NAMESPACE_BEGIN
2017-09-19 22:54:40 +02:00
namespace base_internal {
Export of internal Abseil changes -- 1bc4d36e13fb9175ea8cdaa00213aa9d4417c669 by Andy Getzendanner <durandal@google.com>: Fix pointer format specifier in documentation Import of https://github.com/abseil/abseil-cpp/pull/614 PiperOrigin-RevId: 293227540 -- c7b43b30493c4fb5f2ec3264672b08bfe1ea3709 by Abseil Team <absl-team@google.com>: Internal change. PiperOrigin-RevId: 293160245 -- 64439365e2b4a0b5e51ae0a7dafdb15912402dfd by Shahriar Rouf <nafi@google.com>: Add benchmarks for string_view: BM_CompareFirstOneLess and BM_CompareSecondOneLess. PiperOrigin-RevId: 293031676 -- b273b420cab24a6e3f487430987e09f4eb1caec4 by Greg Falcon <gfalcon@google.com>: Remove an unreachable line from charconv.cc. Fixes github issue #613. PiperOrigin-RevId: 292980167 -- 70babb5f7a3d9fdd00a2b3085c3c2b9fe0265c79 by Gennadiy Rozental <rogeeff@google.com>: Move GetFlag implementation into FlagImpl. This change will allow us to hide details of GetFlag overloads inside implementation detais. Eventually we'll migrate to a different implementation. No semantic changes in this CL. PiperOrigin-RevId: 292930847 -- 94bee7b7cc31e0167ee4b953281c1e78c96a574a by Abseil Team <absl-team@google.com>: Clarification in absl::Exponential documentation. PiperOrigin-RevId: 292912672 -- d6916d30c5c1d3ee9ae46d69ec0a166a760c99c7 by Derek Mauro <dmauro@google.com>: Make AtomicHook constant-initializable on Clang for Windows. Only mark AtomicHook as constant-initializable on platforms where it is actually constant-initializable. PiperOrigin-RevId: 292655939 GitOrigin-RevId: 1bc4d36e13fb9175ea8cdaa00213aa9d4417c669 Change-Id: I090b231a0ca0d92868e494ab5b3fa86c902889d5
2020-02-04 23:18:00 +01:00
ABSL_INTERNAL_ATOMIC_HOOK_ATTRIBUTES static base_internal::AtomicHook<void (*)(
const void *lock, int64_t wait_cycles)>
2017-09-19 22:54:40 +02:00
submit_profile_data;
void RegisterSpinLockProfiler(void (*fn)(const void *contendedlock,
int64_t wait_cycles)) {
submit_profile_data.Store(fn);
}
// Uncommon constructors.
SpinLock::SpinLock(base_internal::SchedulingMode mode)
: lockword_(IsCooperative(mode) ? kSpinLockCooperative : 0) {
ABSL_TSAN_MUTEX_CREATE(this, __tsan_mutex_not_static);
2017-09-19 22:54:40 +02:00
}
SpinLock::SpinLock(base_internal::LinkerInitialized,
base_internal::SchedulingMode mode) {
ABSL_TSAN_MUTEX_CREATE(this, 0);
2017-09-19 22:54:40 +02:00
if (IsCooperative(mode)) {
InitLinkerInitializedAndCooperative();
}
// Otherwise, lockword_ is already initialized.
}
// Static (linker initialized) spinlocks always start life as functional
// non-cooperative locks. When their static constructor does run, it will call
// this initializer to augment the lockword with the cooperative bit. By
// actually taking the lock when we do this we avoid the need for an atomic
// operation in the regular unlock path.
//
// SlowLock() must be careful to re-test for this bit so that any outstanding
// waiters may be upgraded to cooperative status.
void SpinLock::InitLinkerInitializedAndCooperative() {
Lock();
lockword_.fetch_or(kSpinLockCooperative, std::memory_order_relaxed);
Unlock();
}
// Monitor the lock to see if its value changes within some time period
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
// (adaptive_spin_count loop iterations). The last value read from the lock
// is returned from the method.
uint32_t SpinLock::SpinLoop() {
- abacaab4b11a69dd4db627bd183571d7cabb8def Refinement to previous time.h edit (in this same github p... by Greg Falcon <gfalcon@google.com> - 64db19b773134c6c8004e3b23c9ca892efbf8bae Move SpinLock's adaptive spin count computation from a st... by Derek Mauro <dmauro@google.com> - 6f9533fb44a52485a7c2bbb9b4efc7bf8d6c359a Import of CCTZ from GitHub. by Abseil Team <absl-team@google.com> - a211d7255c986e8dd4ceada362c0d054a6a1969a Cleanup exception flags by Abseil Team <absl-team@google.com> - babdb29c590126fe9bba5229fe91034b5b5c358a Release time benchmarks. by Alex Strelnikov <strel@google.com> - 5803b32a3ff123d1fb57a0c471d199c818357c9f Release memutil microbenchmarks. by Alex Strelnikov <strel@google.com> - 5357d4890d30e80c53beb05af32500fb20e9402b Add parens around expansion of ABSL_PREDICT_{FALSE,TRUE} ... by Abseil Team <absl-team@google.com> - 32023f61a239a5f6b1c59e577bfe81b179bbcd2d Reformat build rule tag. by Alex Strelnikov <strel@google.com> - 833758ecf2b0cf7a42bbd50b5b127e416425c168 Release uint128 microbenchmarks. by Alex Strelnikov <strel@google.com> - c115a9bca1f944b90fdc78a56b2de176466b124f Disambiguate bitwise-not of size_type by Abseil Team <absl-team@google.com> - f6905f5b5f6e425792de646edafde440548d9346 Updates ConstructorTracker and TrackedObjects with 1) a m... by Abseil Team <absl-team@google.com> - 147c553bdd5d2db20a38f75c4d1ef973d6c709c5 Changes the absl::Duration factory functions to disallow ... by Greg Miller <jgm@google.com> - dba2b96d11b5264546b283ba452f2de1303b0f07 White space fix by Alex Strelnikov <strel@google.com> GitOrigin-RevId: abacaab4b11a69dd4db627bd183571d7cabb8def Change-Id: I6fa34f20d0b2f898e7b5475a603111413bb80a67
2018-06-08 17:14:48 +02:00
// We are already in the slow path of SpinLock, initialize the
// adaptive_spin_count here.
ABSL_CONST_INIT static absl::once_flag init_adaptive_spin_count;
ABSL_CONST_INIT static int adaptive_spin_count = 0;
base_internal::LowLevelCallOnce(&init_adaptive_spin_count, []() {
adaptive_spin_count = base_internal::NumCPUs() > 1 ? 1000 : 1;
});
2017-09-19 22:54:40 +02:00
int c = adaptive_spin_count;
uint32_t lock_value;
do {
lock_value = lockword_.load(std::memory_order_relaxed);
} while ((lock_value & kSpinLockHeld) != 0 && --c > 0);
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
return lock_value;
2017-09-19 22:54:40 +02:00
}
void SpinLock::SlowLock() {
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
uint32_t lock_value = SpinLoop();
lock_value = TryLockInternal(lock_value, 0);
if ((lock_value & kSpinLockHeld) == 0) {
return;
}
2017-09-19 22:54:40 +02:00
// The lock was not obtained initially, so this thread needs to wait for
// it. Record the current timestamp in the local variable wait_start_time
// so the total wait time can be stored in the lockword once this thread
// obtains the lock.
int64_t wait_start_time = CycleClock::Now();
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
uint32_t wait_cycles = 0;
2017-09-19 22:54:40 +02:00
int lock_wait_call_count = 0;
while ((lock_value & kSpinLockHeld) != 0) {
// If the lock is currently held, but not marked as having a sleeper, mark
// it as having a sleeper.
if ((lock_value & kWaitTimeMask) == 0) {
// Here, just "mark" that the thread is going to sleep. Don't store the
// lock wait time in the lock as that will cause the current lock
// owner to think it experienced contention.
if (lockword_.compare_exchange_strong(
lock_value, lock_value | kSpinLockSleeper,
Export of internal Abseil changes. -- ee19e203eca970ff88e8f25ce4e19c32e143b988 by Jon Cohen <cohenjon@google.com>: Exception safety testing no longer uses absl::optional PiperOrigin-RevId: 220336204 -- 460666eb0b316a8b4aeedc589644d53b05251bd1 by Derek Mauro <dmauro@google.com>: Rework SwissTable SSE2 support - Use SSE2 on MSVC when available https://github.com/abseil/abseil-cpp/issues/210 - Emulate _mm_cmpgt_epi8 with other SSE2 instructions when using -funsigned-char under GCC https://github.com/abseil/abseil-cpp/issues/209 PiperOrigin-RevId: 220312351 -- 1f4318ecedf8d539b7b698eb803d613ad6b69278 by Abseil Team <absl-team@google.com>: Change CollectPerfectRatios to use 10 trials to smooth out the outliers in the sample. PiperOrigin-RevId: 220286579 -- 6755abc2673553a7f578bb29c6e9ca8d991bc9c8 by Abseil Team <absl-team@google.com>: Internal change PiperOrigin-RevId: 220274307 -- 8645b6187329ebf0aaf3c2de2888ba44466cd879 by Abseil Team <absl-team@google.com>: * #endif for a header guard should reference the guard macro in a comment PiperOrigin-RevId: 220206868 -- 3987a7ad11319230910931cd2468b60b3fd1b85c by Gennadiy Civil <misterg@google.com>: Internal Change PiperOrigin-RevId: 220136674 -- cc908c1db2ee0d4523dc813e33f600583bb986c5 by Abseil Team <absl-team@google.com>: absl: fix backoff logic in SpinLockWait There are 3 bugs in loop variable handling: 1. It starts with 0, but AbslInternalSpinLockDelay ignores loop == 0. So it does not actually wait when it should. 2. loop is incremented after successful state changes, but it should not (why would be increase backoff delay after that?). 3. loop is incremented after CAS failures, but it should not (why would be increase backoff delay after that?). Use the same handling of loop as used in SpinLock. PiperOrigin-RevId: 220136079 -- a0a1c6ef5910ebd28e07215d7df03cc0da0b3eed by Abseil Team <absl-team@google.com>: absl: relax unnecessarily strong memory ordering in SpinLock::SlowLock We don't need to acquire visibility over anything when setting kSpinLockSleeper. Replace the confusing and unnecessarily strong memory order with relaxed. PiperOrigin-RevId: 220023380 -- c50858b51af28b9fca1a62616324f85f3e84ea74 by Tom Manshreck <shreck@google.com>: Update comments in flat_hash_map, node_hash_{set, map} and the containers developer guide PiperOrigin-RevId: 219938692 -- e87b7d1a5f61e165b1c44d3b16d8d967197cdfce by CJ Johnson <johnsoncj@google.com>: Rearranges the public methods of InlinedVector and cleans up the comments PiperOrigin-RevId: 219896257 -- f3234c466f792e0fc4bfd21fc7919dba5e679375 by CJ Johnson <johnsoncj@google.com>: Adds branch prediction to exceptional early exit cases of inlined vector's API PiperOrigin-RevId: 219887173 -- 4dfccf1a81ca0425912d3da25a8470f78c532ce4 by CJ Johnson <johnsoncj@google.com>: Fixes the InlinedVector public interface to use the allocator type references instead of assuming the type Also cleans up some cruft in formatting and comments PiperOrigin-RevId: 219878876 -- 4bb6a2b892abb10bd6a424db7e94ed8640802470 by Tom Manshreck <shreck@google.com>: Add comments on constructor and assignment operator support to flat_hash_set PiperOrigin-RevId: 219825338 -- c23f973e2f7f4feea0da36bf8a9c3f8a8954bb74 by Abseil Team <absl-team@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 219823847 GitOrigin-RevId: ee19e203eca970ff88e8f25ce4e19c32e143b988 Change-Id: I288c927ca481dc57340420dbb4c278a05cf15e83
2018-11-06 22:01:25 +01:00
std::memory_order_relaxed, std::memory_order_relaxed)) {
2017-09-19 22:54:40 +02:00
// Successfully transitioned to kSpinLockSleeper. Pass
// kSpinLockSleeper to the SpinLockWait routine to properly indicate
// the last lock_value observed.
lock_value |= kSpinLockSleeper;
} else if ((lock_value & kSpinLockHeld) == 0) {
// Lock is free again, so try and acquire it before sleeping. The
// new lock state will be the number of cycles this thread waited if
// this thread obtains the lock.
lock_value = TryLockInternal(lock_value, wait_cycles);
continue; // Skip the delay at the end of the loop.
}
}
base_internal::SchedulingMode scheduling_mode;
if ((lock_value & kSpinLockCooperative) != 0) {
scheduling_mode = base_internal::SCHEDULE_COOPERATIVE_AND_KERNEL;
} else {
scheduling_mode = base_internal::SCHEDULE_KERNEL_ONLY;
}
// SpinLockDelay() calls into fiber scheduler, we need to see
// synchronization there to avoid false positives.
ABSL_TSAN_MUTEX_PRE_DIVERT(this, 0);
// Wait for an OS specific delay.
base_internal::SpinLockDelay(&lockword_, lock_value, ++lock_wait_call_count,
scheduling_mode);
ABSL_TSAN_MUTEX_POST_DIVERT(this, 0);
// Spin again after returning from the wait routine to give this thread
// some chance of obtaining the lock.
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
lock_value = SpinLoop();
wait_cycles = EncodeWaitCycles(wait_start_time, CycleClock::Now());
lock_value = TryLockInternal(lock_value, wait_cycles);
2017-09-19 22:54:40 +02:00
}
}
void SpinLock::SlowUnlock(uint32_t lock_value) {
base_internal::SpinLockWake(&lockword_,
false); // wake waiter if necessary
// If our acquisition was contended, collect contentionz profile info. We
// reserve a unitary wait time to represent that a waiter exists without our
// own acquisition having been contended.
if ((lock_value & kWaitTimeMask) != kSpinLockSleeper) {
const uint64_t wait_cycles = DecodeWaitCycles(lock_value);
ABSL_TSAN_MUTEX_PRE_DIVERT(this, 0);
submit_profile_data(this, wait_cycles);
ABSL_TSAN_MUTEX_POST_DIVERT(this, 0);
}
}
// We use the upper 29 bits of the lock word to store the time spent waiting to
// acquire this lock. This is reported by contentionz profiling. Since the
// lower bits of the cycle counter wrap very quickly on high-frequency
Export of internal Abseil changes -- 79913a12f0cad4baf948430315aabf53f03b6475 by Abseil Team <absl-team@google.com>: Don't inline (Un)LockSlow. PiperOrigin-RevId: 302502344 -- 6b340e80f0690655f24799c8de6707b3a95b8579 by Derek Mauro <dmauro@google.com>: Add hardening assertions to absl::optional's dereference operators PiperOrigin-RevId: 302492862 -- a9951bf4852d8c1aec472cb4b539830411270e4c by Derek Mauro <dmauro@google.com>: Correctly add hardware AES compiler flags under Linux X86-64 Fixes #643 PiperOrigin-RevId: 302490673 -- 314c3621ee4d57b6bc8d64338a1f1d48a69741d1 by Derek Mauro <dmauro@google.com>: Upgrade to hardening assertions in absl::Span::remove_prefix and absl::Span::remove_suffix PiperOrigin-RevId: 302481191 -- a142b8c6c62705c5f0d4fe3113150f0c0b7822b9 by Derek Mauro <dmauro@google.com>: Update docker containers to Bazel 2.2.0, GCC 9.3, and new Clang snapshot PiperOrigin-RevId: 302454042 -- afedeb70a2adc87010030c9ba6f06fe35ec26407 by Derek Mauro <dmauro@google.com>: Add hardening assertions for the preconditions of absl::FixedArray PiperOrigin-RevId: 302441767 -- 44442bfbc0a9a742df32f07cee86a47712efb8b4 by Derek Mauro <dmauro@google.com>: Fix new Clang warning about SpinLock doing operations on enums of different types PiperOrigin-RevId: 302430387 -- 69eaff7f97231779f696321c2ba8b88debf6dd9e by Derek Mauro <dmauro@google.com>: Convert precondition assertions to ABSL_HARDENING_ASSERT for absl::InlinedVector PiperOrigin-RevId: 302427894 -- 26b6db906a0942fd18583dc2cdd1bab32919d964 by Gennadiy Rozental <rogeeff@google.com>: Internal change PiperOrigin-RevId: 302425283 -- e62e81422979e922505d2cd9000e1de58123c088 by Derek Mauro <dmauro@google.com>: Add an option to build Abseil in hardened mode In hardened mode, the ABSL_HARDENING_ASSERT() macro is active even when NDEBUG is defined. This allows Abseil to perform runtime checks even in release mode. This should be used to implement things like bounds checks that could otherwise lead to security vulnerabilities. Use the new assertion in absl::string_view and absl::Span to test it. PiperOrigin-RevId: 302119187 GitOrigin-RevId: 79913a12f0cad4baf948430315aabf53f03b6475 Change-Id: I0cc3341fd333a1df313167bab72dc5a759c4a048
2020-03-23 21:16:18 +01:00
// processors we divide to reduce the granularity to 2^kProfileTimestampShift
2017-09-19 22:54:40 +02:00
// sized units. On a 4Ghz machine this will lose track of wait times greater
// than (2^29/4 Ghz)*128 =~ 17.2 seconds. Such waits should be extremely rare.
Export of internal Abseil changes -- 79913a12f0cad4baf948430315aabf53f03b6475 by Abseil Team <absl-team@google.com>: Don't inline (Un)LockSlow. PiperOrigin-RevId: 302502344 -- 6b340e80f0690655f24799c8de6707b3a95b8579 by Derek Mauro <dmauro@google.com>: Add hardening assertions to absl::optional's dereference operators PiperOrigin-RevId: 302492862 -- a9951bf4852d8c1aec472cb4b539830411270e4c by Derek Mauro <dmauro@google.com>: Correctly add hardware AES compiler flags under Linux X86-64 Fixes #643 PiperOrigin-RevId: 302490673 -- 314c3621ee4d57b6bc8d64338a1f1d48a69741d1 by Derek Mauro <dmauro@google.com>: Upgrade to hardening assertions in absl::Span::remove_prefix and absl::Span::remove_suffix PiperOrigin-RevId: 302481191 -- a142b8c6c62705c5f0d4fe3113150f0c0b7822b9 by Derek Mauro <dmauro@google.com>: Update docker containers to Bazel 2.2.0, GCC 9.3, and new Clang snapshot PiperOrigin-RevId: 302454042 -- afedeb70a2adc87010030c9ba6f06fe35ec26407 by Derek Mauro <dmauro@google.com>: Add hardening assertions for the preconditions of absl::FixedArray PiperOrigin-RevId: 302441767 -- 44442bfbc0a9a742df32f07cee86a47712efb8b4 by Derek Mauro <dmauro@google.com>: Fix new Clang warning about SpinLock doing operations on enums of different types PiperOrigin-RevId: 302430387 -- 69eaff7f97231779f696321c2ba8b88debf6dd9e by Derek Mauro <dmauro@google.com>: Convert precondition assertions to ABSL_HARDENING_ASSERT for absl::InlinedVector PiperOrigin-RevId: 302427894 -- 26b6db906a0942fd18583dc2cdd1bab32919d964 by Gennadiy Rozental <rogeeff@google.com>: Internal change PiperOrigin-RevId: 302425283 -- e62e81422979e922505d2cd9000e1de58123c088 by Derek Mauro <dmauro@google.com>: Add an option to build Abseil in hardened mode In hardened mode, the ABSL_HARDENING_ASSERT() macro is active even when NDEBUG is defined. This allows Abseil to perform runtime checks even in release mode. This should be used to implement things like bounds checks that could otherwise lead to security vulnerabilities. Use the new assertion in absl::string_view and absl::Span to test it. PiperOrigin-RevId: 302119187 GitOrigin-RevId: 79913a12f0cad4baf948430315aabf53f03b6475 Change-Id: I0cc3341fd333a1df313167bab72dc5a759c4a048
2020-03-23 21:16:18 +01:00
static constexpr int kProfileTimestampShift = 7;
// We currently reserve the lower 3 bits.
static constexpr int kLockwordReservedShift = 3;
2017-09-19 22:54:40 +02:00
uint32_t SpinLock::EncodeWaitCycles(int64_t wait_start_time,
int64_t wait_end_time) {
static const int64_t kMaxWaitTime =
Export of internal Abseil changes -- 79913a12f0cad4baf948430315aabf53f03b6475 by Abseil Team <absl-team@google.com>: Don't inline (Un)LockSlow. PiperOrigin-RevId: 302502344 -- 6b340e80f0690655f24799c8de6707b3a95b8579 by Derek Mauro <dmauro@google.com>: Add hardening assertions to absl::optional's dereference operators PiperOrigin-RevId: 302492862 -- a9951bf4852d8c1aec472cb4b539830411270e4c by Derek Mauro <dmauro@google.com>: Correctly add hardware AES compiler flags under Linux X86-64 Fixes #643 PiperOrigin-RevId: 302490673 -- 314c3621ee4d57b6bc8d64338a1f1d48a69741d1 by Derek Mauro <dmauro@google.com>: Upgrade to hardening assertions in absl::Span::remove_prefix and absl::Span::remove_suffix PiperOrigin-RevId: 302481191 -- a142b8c6c62705c5f0d4fe3113150f0c0b7822b9 by Derek Mauro <dmauro@google.com>: Update docker containers to Bazel 2.2.0, GCC 9.3, and new Clang snapshot PiperOrigin-RevId: 302454042 -- afedeb70a2adc87010030c9ba6f06fe35ec26407 by Derek Mauro <dmauro@google.com>: Add hardening assertions for the preconditions of absl::FixedArray PiperOrigin-RevId: 302441767 -- 44442bfbc0a9a742df32f07cee86a47712efb8b4 by Derek Mauro <dmauro@google.com>: Fix new Clang warning about SpinLock doing operations on enums of different types PiperOrigin-RevId: 302430387 -- 69eaff7f97231779f696321c2ba8b88debf6dd9e by Derek Mauro <dmauro@google.com>: Convert precondition assertions to ABSL_HARDENING_ASSERT for absl::InlinedVector PiperOrigin-RevId: 302427894 -- 26b6db906a0942fd18583dc2cdd1bab32919d964 by Gennadiy Rozental <rogeeff@google.com>: Internal change PiperOrigin-RevId: 302425283 -- e62e81422979e922505d2cd9000e1de58123c088 by Derek Mauro <dmauro@google.com>: Add an option to build Abseil in hardened mode In hardened mode, the ABSL_HARDENING_ASSERT() macro is active even when NDEBUG is defined. This allows Abseil to perform runtime checks even in release mode. This should be used to implement things like bounds checks that could otherwise lead to security vulnerabilities. Use the new assertion in absl::string_view and absl::Span to test it. PiperOrigin-RevId: 302119187 GitOrigin-RevId: 79913a12f0cad4baf948430315aabf53f03b6475 Change-Id: I0cc3341fd333a1df313167bab72dc5a759c4a048
2020-03-23 21:16:18 +01:00
std::numeric_limits<uint32_t>::max() >> kLockwordReservedShift;
2017-09-19 22:54:40 +02:00
int64_t scaled_wait_time =
Export of internal Abseil changes -- 79913a12f0cad4baf948430315aabf53f03b6475 by Abseil Team <absl-team@google.com>: Don't inline (Un)LockSlow. PiperOrigin-RevId: 302502344 -- 6b340e80f0690655f24799c8de6707b3a95b8579 by Derek Mauro <dmauro@google.com>: Add hardening assertions to absl::optional's dereference operators PiperOrigin-RevId: 302492862 -- a9951bf4852d8c1aec472cb4b539830411270e4c by Derek Mauro <dmauro@google.com>: Correctly add hardware AES compiler flags under Linux X86-64 Fixes #643 PiperOrigin-RevId: 302490673 -- 314c3621ee4d57b6bc8d64338a1f1d48a69741d1 by Derek Mauro <dmauro@google.com>: Upgrade to hardening assertions in absl::Span::remove_prefix and absl::Span::remove_suffix PiperOrigin-RevId: 302481191 -- a142b8c6c62705c5f0d4fe3113150f0c0b7822b9 by Derek Mauro <dmauro@google.com>: Update docker containers to Bazel 2.2.0, GCC 9.3, and new Clang snapshot PiperOrigin-RevId: 302454042 -- afedeb70a2adc87010030c9ba6f06fe35ec26407 by Derek Mauro <dmauro@google.com>: Add hardening assertions for the preconditions of absl::FixedArray PiperOrigin-RevId: 302441767 -- 44442bfbc0a9a742df32f07cee86a47712efb8b4 by Derek Mauro <dmauro@google.com>: Fix new Clang warning about SpinLock doing operations on enums of different types PiperOrigin-RevId: 302430387 -- 69eaff7f97231779f696321c2ba8b88debf6dd9e by Derek Mauro <dmauro@google.com>: Convert precondition assertions to ABSL_HARDENING_ASSERT for absl::InlinedVector PiperOrigin-RevId: 302427894 -- 26b6db906a0942fd18583dc2cdd1bab32919d964 by Gennadiy Rozental <rogeeff@google.com>: Internal change PiperOrigin-RevId: 302425283 -- e62e81422979e922505d2cd9000e1de58123c088 by Derek Mauro <dmauro@google.com>: Add an option to build Abseil in hardened mode In hardened mode, the ABSL_HARDENING_ASSERT() macro is active even when NDEBUG is defined. This allows Abseil to perform runtime checks even in release mode. This should be used to implement things like bounds checks that could otherwise lead to security vulnerabilities. Use the new assertion in absl::string_view and absl::Span to test it. PiperOrigin-RevId: 302119187 GitOrigin-RevId: 79913a12f0cad4baf948430315aabf53f03b6475 Change-Id: I0cc3341fd333a1df313167bab72dc5a759c4a048
2020-03-23 21:16:18 +01:00
(wait_end_time - wait_start_time) >> kProfileTimestampShift;
2017-09-19 22:54:40 +02:00
// Return a representation of the time spent waiting that can be stored in
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
// the lock word's upper bits.
uint32_t clamped = static_cast<uint32_t>(
Export of internal Abseil changes -- 79913a12f0cad4baf948430315aabf53f03b6475 by Abseil Team <absl-team@google.com>: Don't inline (Un)LockSlow. PiperOrigin-RevId: 302502344 -- 6b340e80f0690655f24799c8de6707b3a95b8579 by Derek Mauro <dmauro@google.com>: Add hardening assertions to absl::optional's dereference operators PiperOrigin-RevId: 302492862 -- a9951bf4852d8c1aec472cb4b539830411270e4c by Derek Mauro <dmauro@google.com>: Correctly add hardware AES compiler flags under Linux X86-64 Fixes #643 PiperOrigin-RevId: 302490673 -- 314c3621ee4d57b6bc8d64338a1f1d48a69741d1 by Derek Mauro <dmauro@google.com>: Upgrade to hardening assertions in absl::Span::remove_prefix and absl::Span::remove_suffix PiperOrigin-RevId: 302481191 -- a142b8c6c62705c5f0d4fe3113150f0c0b7822b9 by Derek Mauro <dmauro@google.com>: Update docker containers to Bazel 2.2.0, GCC 9.3, and new Clang snapshot PiperOrigin-RevId: 302454042 -- afedeb70a2adc87010030c9ba6f06fe35ec26407 by Derek Mauro <dmauro@google.com>: Add hardening assertions for the preconditions of absl::FixedArray PiperOrigin-RevId: 302441767 -- 44442bfbc0a9a742df32f07cee86a47712efb8b4 by Derek Mauro <dmauro@google.com>: Fix new Clang warning about SpinLock doing operations on enums of different types PiperOrigin-RevId: 302430387 -- 69eaff7f97231779f696321c2ba8b88debf6dd9e by Derek Mauro <dmauro@google.com>: Convert precondition assertions to ABSL_HARDENING_ASSERT for absl::InlinedVector PiperOrigin-RevId: 302427894 -- 26b6db906a0942fd18583dc2cdd1bab32919d964 by Gennadiy Rozental <rogeeff@google.com>: Internal change PiperOrigin-RevId: 302425283 -- e62e81422979e922505d2cd9000e1de58123c088 by Derek Mauro <dmauro@google.com>: Add an option to build Abseil in hardened mode In hardened mode, the ABSL_HARDENING_ASSERT() macro is active even when NDEBUG is defined. This allows Abseil to perform runtime checks even in release mode. This should be used to implement things like bounds checks that could otherwise lead to security vulnerabilities. Use the new assertion in absl::string_view and absl::Span to test it. PiperOrigin-RevId: 302119187 GitOrigin-RevId: 79913a12f0cad4baf948430315aabf53f03b6475 Change-Id: I0cc3341fd333a1df313167bab72dc5a759c4a048
2020-03-23 21:16:18 +01:00
std::min(scaled_wait_time, kMaxWaitTime) << kLockwordReservedShift);
2017-09-19 22:54:40 +02:00
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
if (clamped == 0) {
return kSpinLockSleeper; // Just wake waiters, but don't record contention.
}
// Bump up value if necessary to avoid returning kSpinLockSleeper.
const uint32_t kMinWaitTime =
Export of internal Abseil changes -- 79913a12f0cad4baf948430315aabf53f03b6475 by Abseil Team <absl-team@google.com>: Don't inline (Un)LockSlow. PiperOrigin-RevId: 302502344 -- 6b340e80f0690655f24799c8de6707b3a95b8579 by Derek Mauro <dmauro@google.com>: Add hardening assertions to absl::optional's dereference operators PiperOrigin-RevId: 302492862 -- a9951bf4852d8c1aec472cb4b539830411270e4c by Derek Mauro <dmauro@google.com>: Correctly add hardware AES compiler flags under Linux X86-64 Fixes #643 PiperOrigin-RevId: 302490673 -- 314c3621ee4d57b6bc8d64338a1f1d48a69741d1 by Derek Mauro <dmauro@google.com>: Upgrade to hardening assertions in absl::Span::remove_prefix and absl::Span::remove_suffix PiperOrigin-RevId: 302481191 -- a142b8c6c62705c5f0d4fe3113150f0c0b7822b9 by Derek Mauro <dmauro@google.com>: Update docker containers to Bazel 2.2.0, GCC 9.3, and new Clang snapshot PiperOrigin-RevId: 302454042 -- afedeb70a2adc87010030c9ba6f06fe35ec26407 by Derek Mauro <dmauro@google.com>: Add hardening assertions for the preconditions of absl::FixedArray PiperOrigin-RevId: 302441767 -- 44442bfbc0a9a742df32f07cee86a47712efb8b4 by Derek Mauro <dmauro@google.com>: Fix new Clang warning about SpinLock doing operations on enums of different types PiperOrigin-RevId: 302430387 -- 69eaff7f97231779f696321c2ba8b88debf6dd9e by Derek Mauro <dmauro@google.com>: Convert precondition assertions to ABSL_HARDENING_ASSERT for absl::InlinedVector PiperOrigin-RevId: 302427894 -- 26b6db906a0942fd18583dc2cdd1bab32919d964 by Gennadiy Rozental <rogeeff@google.com>: Internal change PiperOrigin-RevId: 302425283 -- e62e81422979e922505d2cd9000e1de58123c088 by Derek Mauro <dmauro@google.com>: Add an option to build Abseil in hardened mode In hardened mode, the ABSL_HARDENING_ASSERT() macro is active even when NDEBUG is defined. This allows Abseil to perform runtime checks even in release mode. This should be used to implement things like bounds checks that could otherwise lead to security vulnerabilities. Use the new assertion in absl::string_view and absl::Span to test it. PiperOrigin-RevId: 302119187 GitOrigin-RevId: 79913a12f0cad4baf948430315aabf53f03b6475 Change-Id: I0cc3341fd333a1df313167bab72dc5a759c4a048
2020-03-23 21:16:18 +01:00
kSpinLockSleeper + (1 << kLockwordReservedShift);
Export of internal Abseil changes. -- 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 by Jon Cohen <cohenjon@google.com>: Import of CCTZ from GitHub. PiperOrigin-RevId: 220523164 -- 59ef14fe7034a3148f1e9cef1f128b8ca264b444 by Jon Cohen <cohenjon@google.com>: Don't assume how much std::vector's constructors allocate in InlinedVector's test for scoped_allocator_adaptor support. PiperOrigin-RevId: 220464683 -- 6f8351be43a44a8f10bf20612b2cc744a4a911c7 by Jon Cohen <cohenjon@google.com>: Add VS Code and some Bazel output files to absl/.gitignore PiperOrigin-RevId: 220464362 -- 43fac22f8af6b6ed55309a784a9d25d837393d0e by Abseil Team <absl-team@google.com>: absl: fix SpinLock::EncodeWaitCycles If a thread has ever observed or set kSpinLockSleeper, it must never leave 0 in kWaitTimeMask because at this point it is expected to wake subsequent threads. Current calculations in EncodeWaitCycles can result in 0 in kWaitTimeMask and lead to missed wakeups. This is mostly theoretical today, because the futex call needs to finish within 128 cycles (futex can return immediately without waiting, but 128 cycles still look too low for this). But this can well fire in future if we bump granularity and/or threshold for recording contention. Use kSpinLockSleeper instead of 0. PiperOrigin-RevId: 220463123 -- def9b7e3d45c99d68cc52a4429256116d7f421f2 by Abseil Team <absl-team@google.com>: absl: optimize SpinLock::SlowLock Currently we record contention even after the first initial spin. This leads to several performance issues: 1. If we succeed in acquiring the lock after the initial spin, overall wait time can be within tens/hundreds of nanoseconds. Recording such low wait time looks completely unnecessary and excessive. From some point of view this is not even a wait, because we did not sleep. And, for example, Mutex does not record contention in this case. In majority of cases the lock should be acquired exactly during the initial spin, yet we still go through full overhead of submitting contention. 2. Whenever a thread submits contention it also calls FUTEX_WAKE (there is no way to understand if it's necessary or not when wait value is stored in the lock). So if there are just 2 threads and a brief contention, the second thread will still call FUTEX_WAKE which is completely unnecessary overhead. Don't record contention after the initial spin wait. FWIW this also removes 2 CycleClock::Now calls and EncodeWaitCycles from the common hot path. PiperOrigin-RevId: 220379972 -- 75b0c0cb214de904ea622f81ec3f4eabdc8695b0 by Derek Mauro <dmauro@google.com>: Supress MSVC warnings in raw_hash_set's use of TrailingZeros and LeadingZeros. https://github.com/abseil/abseil-cpp/issues/208 PiperOrigin-RevId: 220372204 GitOrigin-RevId: 178e7a9a76fc8fcd6df6335b59139cbe644a16b9 Change-Id: I3a66af4e050810a3274e45d4e055b2aa19ffba1b
2018-11-07 22:52:16 +01:00
if (clamped == kSpinLockSleeper) {
return kMinWaitTime;
}
return clamped;
2017-09-19 22:54:40 +02:00
}
uint64_t SpinLock::DecodeWaitCycles(uint32_t lock_value) {
// Cast to uint32_t first to ensure bits [63:32] are cleared.
const uint64_t scaled_wait_time =
static_cast<uint32_t>(lock_value & kWaitTimeMask);
Export of internal Abseil changes -- 79913a12f0cad4baf948430315aabf53f03b6475 by Abseil Team <absl-team@google.com>: Don't inline (Un)LockSlow. PiperOrigin-RevId: 302502344 -- 6b340e80f0690655f24799c8de6707b3a95b8579 by Derek Mauro <dmauro@google.com>: Add hardening assertions to absl::optional's dereference operators PiperOrigin-RevId: 302492862 -- a9951bf4852d8c1aec472cb4b539830411270e4c by Derek Mauro <dmauro@google.com>: Correctly add hardware AES compiler flags under Linux X86-64 Fixes #643 PiperOrigin-RevId: 302490673 -- 314c3621ee4d57b6bc8d64338a1f1d48a69741d1 by Derek Mauro <dmauro@google.com>: Upgrade to hardening assertions in absl::Span::remove_prefix and absl::Span::remove_suffix PiperOrigin-RevId: 302481191 -- a142b8c6c62705c5f0d4fe3113150f0c0b7822b9 by Derek Mauro <dmauro@google.com>: Update docker containers to Bazel 2.2.0, GCC 9.3, and new Clang snapshot PiperOrigin-RevId: 302454042 -- afedeb70a2adc87010030c9ba6f06fe35ec26407 by Derek Mauro <dmauro@google.com>: Add hardening assertions for the preconditions of absl::FixedArray PiperOrigin-RevId: 302441767 -- 44442bfbc0a9a742df32f07cee86a47712efb8b4 by Derek Mauro <dmauro@google.com>: Fix new Clang warning about SpinLock doing operations on enums of different types PiperOrigin-RevId: 302430387 -- 69eaff7f97231779f696321c2ba8b88debf6dd9e by Derek Mauro <dmauro@google.com>: Convert precondition assertions to ABSL_HARDENING_ASSERT for absl::InlinedVector PiperOrigin-RevId: 302427894 -- 26b6db906a0942fd18583dc2cdd1bab32919d964 by Gennadiy Rozental <rogeeff@google.com>: Internal change PiperOrigin-RevId: 302425283 -- e62e81422979e922505d2cd9000e1de58123c088 by Derek Mauro <dmauro@google.com>: Add an option to build Abseil in hardened mode In hardened mode, the ABSL_HARDENING_ASSERT() macro is active even when NDEBUG is defined. This allows Abseil to perform runtime checks even in release mode. This should be used to implement things like bounds checks that could otherwise lead to security vulnerabilities. Use the new assertion in absl::string_view and absl::Span to test it. PiperOrigin-RevId: 302119187 GitOrigin-RevId: 79913a12f0cad4baf948430315aabf53f03b6475 Change-Id: I0cc3341fd333a1df313167bab72dc5a759c4a048
2020-03-23 21:16:18 +01:00
return scaled_wait_time << (kProfileTimestampShift - kLockwordReservedShift);
2017-09-19 22:54:40 +02:00
}
} // namespace base_internal
ABSL_NAMESPACE_END
2017-09-19 22:54:40 +02:00
} // namespace absl