aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
Diffstat
-rw-r--r--Documentation/00-INDEX2
-rw-r--r--Documentation/DocBook/kernel-locking.tmpl2
-rw-r--r--Documentation/locking/lockdep-design.txt (renamed from Documentation/lockdep-design.txt)0
-rw-r--r--Documentation/locking/lockstat.txt (renamed from Documentation/lockstat.txt)2
-rw-r--r--Documentation/locking/mutex-design.txt (renamed from Documentation/mutex-design.txt)6
-rw-r--r--Documentation/locking/rt-mutex-design.txt (renamed from Documentation/rt-mutex-design.txt)0
-rw-r--r--Documentation/locking/rt-mutex.txt (renamed from Documentation/rt-mutex.txt)0
-rw-r--r--Documentation/locking/spinlocks.txt (renamed from Documentation/spinlocks.txt)14
-rw-r--r--Documentation/locking/ww-mutex-design.txt (renamed from Documentation/ww-mutex-design.txt)0
-rw-r--r--MAINTAINERS4
-rw-r--r--arch/x86/include/asm/rwlock.h49
-rw-r--r--arch/x86/include/asm/spinlock.h81
-rw-r--r--arch/x86/include/asm/spinlock_types.h4
-rw-r--r--arch/x86/lib/Makefile1
-rw-r--r--arch/x86/lib/rwlock.S44
-rw-r--r--drivers/gpu/drm/drm_modeset_lock.c2
-rw-r--r--include/linux/atomic.h36
-rw-r--r--include/linux/bitops.h20
-rw-r--r--include/linux/lockdep.h2
-rw-r--r--include/linux/mutex.h4
-rw-r--r--include/linux/rwsem.h2
-rw-r--r--include/linux/spinlock.h8
-rw-r--r--kernel/locking/mcs_spinlock.h3
-rw-r--r--kernel/locking/mutex.c416
-rw-r--r--kernel/locking/mutex.h2
-rw-r--r--kernel/locking/rtmutex.c2
-rw-r--r--kernel/locking/rwsem-xadd.c27
-rw-r--r--kernel/locking/semaphore.c12
-rw-r--r--kernel/sched/core.c16
-rw-r--r--lib/Kconfig.debug4
30 files changed, 282 insertions, 483 deletions
diff --git a/Documentation/00-INDEX b/Documentation/00-INDEX
index 27e67a98b7be..1750fcef1ab4 100644
--- a/Documentation/00-INDEX
+++ b/Documentation/00-INDEX
@@ -287,6 +287,8 @@ local_ops.txt
287 - semantics and behavior of local atomic operations. 287 - semantics and behavior of local atomic operations.
288lockdep-design.txt 288lockdep-design.txt
289 - documentation on the runtime locking correctness validator. 289 - documentation on the runtime locking correctness validator.
290locking/
291 - directory with info about kernel locking primitives
290lockstat.txt 292lockstat.txt
291 - info on collecting statistics on locks (and contention). 293 - info on collecting statistics on locks (and contention).
292lockup-watchdogs.txt 294lockup-watchdogs.txt
diff --git a/Documentation/DocBook/kernel-locking.tmpl b/Documentation/DocBook/kernel-locking.tmpl
index e584ee12a1e7..7c9cc4846cb6 100644
--- a/Documentation/DocBook/kernel-locking.tmpl
+++ b/Documentation/DocBook/kernel-locking.tmpl
@@ -1972,7 +1972,7 @@ machines due to caching.
1972 <itemizedlist> 1972 <itemizedlist>
1973 <listitem> 1973 <listitem>
1974 <para> 1974 <para>
1975 <filename>Documentation/spinlocks.txt</filename>: 1975 <filename>Documentation/locking/spinlocks.txt</filename>:
1976 Linus Torvalds' spinlocking tutorial in the kernel sources. 1976 Linus Torvalds' spinlocking tutorial in the kernel sources.
1977 </para> 1977 </para>
1978 </listitem> 1978 </listitem>
diff --git a/Documentation/lockdep-design.txt b/Documentation/locking/lockdep-design.txt
index 5dbc99c04f6e..5dbc99c04f6e 100644
--- a/Documentation/lockdep-design.txt
+++ b/Documentation/locking/lockdep-design.txt
diff --git a/Documentation/lockstat.txt b/Documentation/locking/lockstat.txt
index 72d010689751..7428773a1e69 100644
--- a/Documentation/lockstat.txt
+++ b/Documentation/locking/lockstat.txt
@@ -12,7 +12,7 @@ Because things like lock contention can severely impact performance.
12- HOW 12- HOW
13 13
14Lockdep already has hooks in the lock functions and maps lock instances to 14Lockdep already has hooks in the lock functions and maps lock instances to
15lock classes. We build on that (see Documentation/lockdep-design.txt). 15lock classes. We build on that (see Documentation/lokcing/lockdep-design.txt).
16The graph below shows the relation between the lock functions and the various 16The graph below shows the relation between the lock functions and the various
17hooks therein. 17hooks therein.
18 18
diff --git a/Documentation/mutex-design.txt b/Documentation/locking/mutex-design.txt
index ee231ed09ec6..60c482df1a38 100644
--- a/Documentation/mutex-design.txt
+++ b/Documentation/locking/mutex-design.txt
@@ -145,9 +145,9 @@ Disadvantages
145 145
146Unlike its original design and purpose, 'struct mutex' is larger than 146Unlike its original design and purpose, 'struct mutex' is larger than
147most locks in the kernel. E.g: on x86-64 it is 40 bytes, almost twice 147most locks in the kernel. E.g: on x86-64 it is 40 bytes, almost twice
148as large as 'struct semaphore' (24 bytes) and 8 bytes shy of the 148as large as 'struct semaphore' (24 bytes) and tied, along with rwsems,
149'struct rw_semaphore' variant. Larger structure sizes mean more CPU 149for the largest lock in the kernel. Larger structure sizes mean more
150cache and memory footprint. 150CPU cache and memory footprint.
151 151
152When to use mutexes 152When to use mutexes
153------------------- 153-------------------
diff --git a/Documentation/rt-mutex-design.txt b/Documentation/locking/rt-mutex-design.txt
index 8666070d3189..8666070d3189 100644
--- a/Documentation/rt-mutex-design.txt
+++ b/Documentation/locking/rt-mutex-design.txt
diff --git a/Documentation/rt-mutex.txt b/Documentation/locking/rt-mutex.txt
index 243393d882ee..243393d882ee 100644
--- a/Documentation/rt-mutex.txt
+++ b/Documentation/locking/rt-mutex.txt
diff --git a/Documentation/spinlocks.txt b/Documentation/locking/spinlocks.txt
index 97eaf5727178..ff35e40bdf5b 100644
--- a/Documentation/spinlocks.txt
+++ b/Documentation/locking/spinlocks.txt
@@ -105,9 +105,9 @@ never used in interrupt handlers, you can use the non-irq versions:
105 spin_unlock(&lock); 105 spin_unlock(&lock);
106 106
107(and the equivalent read-write versions too, of course). The spinlock will 107(and the equivalent read-write versions too, of course). The spinlock will
108guarantee the same kind of exclusive access, and it will be much faster. 108guarantee the same kind of exclusive access, and it will be much faster.
109This is useful if you know that the data in question is only ever 109This is useful if you know that the data in question is only ever
110manipulated from a "process context", ie no interrupts involved. 110manipulated from a "process context", ie no interrupts involved.
111 111
112The reasons you mustn't use these versions if you have interrupts that 112The reasons you mustn't use these versions if you have interrupts that
113play with the spinlock is that you can get deadlocks: 113play with the spinlock is that you can get deadlocks:
@@ -122,21 +122,21 @@ the other interrupt happens on another CPU, but it is _not_ ok if the
122interrupt happens on the same CPU that already holds the lock, because the 122interrupt happens on the same CPU that already holds the lock, because the
123lock will obviously never be released (because the interrupt is waiting 123lock will obviously never be released (because the interrupt is waiting
124for the lock, and the lock-holder is interrupted by the interrupt and will 124for the lock, and the lock-holder is interrupted by the interrupt and will
125not continue until the interrupt has been processed). 125not continue until the interrupt has been processed).
126 126
127(This is also the reason why the irq-versions of the spinlocks only need 127(This is also the reason why the irq-versions of the spinlocks only need
128to disable the _local_ interrupts - it's ok to use spinlocks in interrupts 128to disable the _local_ interrupts - it's ok to use spinlocks in interrupts
129on other CPU's, because an interrupt on another CPU doesn't interrupt the 129on other CPU's, because an interrupt on another CPU doesn't interrupt the
130CPU that holds the lock, so the lock-holder can continue and eventually 130CPU that holds the lock, so the lock-holder can continue and eventually
131releases the lock). 131releases the lock).
132 132
133Note that you can be clever with read-write locks and interrupts. For 133Note that you can be clever with read-write locks and interrupts. For
134example, if you know that the interrupt only ever gets a read-lock, then 134example, if you know that the interrupt only ever gets a read-lock, then
135you can use a non-irq version of read locks everywhere - because they 135you can use a non-irq version of read locks everywhere - because they
136don't block on each other (and thus there is no dead-lock wrt interrupts. 136don't block on each other (and thus there is no dead-lock wrt interrupts.
137But when you do the write-lock, you have to use the irq-safe version. 137But when you do the write-lock, you have to use the irq-safe version.
138 138
139For an example of being clever with rw-locks, see the "waitqueue_lock" 139For an example of being clever with rw-locks, see the "waitqueue_lock"
140handling in kernel/sched/core.c - nothing ever _changes_ a wait-queue from 140handling in kernel/sched/core.c - nothing ever _changes_ a wait-queue from
141within an interrupt, they only read the queue in order to know whom to 141within an interrupt, they only read the queue in order to know whom to
142wake up. So read-locks are safe (which is good: they are very common 142wake up. So read-locks are safe (which is good: they are very common
diff --git a/Documentation/ww-mutex-design.txt b/Documentation/locking/ww-mutex-design.txt
index 8a112dc304c3..8a112dc304c3 100644
--- a/Documentation/ww-mutex-design.txt
+++ b/Documentation/locking/ww-mutex-design.txt
diff --git a/MAINTAINERS b/MAINTAINERS
index 92ad10ef0cf1..f413abff3807 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5680,8 +5680,8 @@ M: Ingo Molnar <mingo@redhat.com>
5680L: linux-kernel@vger.kernel.org 5680L: linux-kernel@vger.kernel.org
5681T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core/locking 5681T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git core/locking
5682S: Maintained 5682S: Maintained
5683F: Documentation/lockdep*.txt 5683F: Documentation/locking/lockdep*.txt
5684F: Documentation/lockstat.txt 5684F: Documentation/locking/lockstat.txt
5685F: include/linux/lockdep.h 5685F: include/linux/lockdep.h
5686F: kernel/locking/ 5686F: kernel/locking/
5687 5687
diff --git a/arch/x86/include/asm/rwlock.h b/arch/x86/include/asm/rwlock.h
deleted file mode 100644
index a5370a03d90c..000000000000
--- a/arch/x86/include/asm/rwlock.h
+++ /dev/null
@@ -1,49 +0,0 @@
1#ifndef _ASM_X86_RWLOCK_H
2#define _ASM_X86_RWLOCK_H
3
4#include <asm/asm.h>
5
6#if CONFIG_NR_CPUS <= 2048
7
8#ifndef __ASSEMBLY__
9typedef union {
10 s32 lock;
11 s32 write;
12} arch_rwlock_t;
13#endif
14
15#define RW_LOCK_BIAS 0x00100000
16#define READ_LOCK_SIZE(insn) __ASM_FORM(insn##l)
17#define READ_LOCK_ATOMIC(n) atomic_##n
18#define WRITE_LOCK_ADD(n) __ASM_FORM_COMMA(addl n)
19#define WRITE_LOCK_SUB(n) __ASM_FORM_COMMA(subl n)
20#define WRITE_LOCK_CMP RW_LOCK_BIAS
21
22#else /* CONFIG_NR_CPUS > 2048 */
23
24#include <linux/const.h>
25
26#ifndef __ASSEMBLY__
27typedef union {
28 s64 lock;
29 struct {
30 u32 read;
31 s32 write;
32 };
33} arch_rwlock_t;
34#endif
35
36#define RW_LOCK_BIAS (_AC(1,L) << 32)
37#define READ_LOCK_SIZE(insn) __ASM_FORM(insn##q)
38#define READ_LOCK_ATOMIC(n) atomic64_##n
39#define WRITE_LOCK_ADD(n) __ASM_FORM(incl)
40#define WRITE_LOCK_SUB(n) __ASM_FORM(decl)
41#define WRITE_LOCK_CMP 1
42
43#endif /* CONFIG_NR_CPUS */
44
45#define __ARCH_RW_LOCK_UNLOCKED { RW_LOCK_BIAS }
46
47/* Actual code is in asm/spinlock.h or in arch/x86/lib/rwlock.S */
48
49#endif /* _ASM_X86_RWLOCK_H */
diff --git a/arch/x86/include/asm/spinlock.h b/arch/x86/include/asm/spinlock.h
index 54f1c8068c02..9295016485c9 100644
--- a/arch/x86/include/asm/spinlock.h
+++ b/arch/x86/include/asm/spinlock.h
@@ -187,7 +187,6 @@ static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
187 cpu_relax(); 187 cpu_relax();
188} 188}
189 189
190#ifndef CONFIG_QUEUE_RWLOCK
191/* 190/*
192 * Read-write spinlocks, allowing multiple readers 191 * Read-write spinlocks, allowing multiple readers
193 * but only one writer. 192 * but only one writer.
@@ -198,91 +197,15 @@ static inline void arch_spin_unlock_wait(arch_spinlock_t *lock)
198 * irq-safe write-lock, but readers can get non-irqsafe 197 * irq-safe write-lock, but readers can get non-irqsafe
199 * read-locks. 198 * read-locks.
200 * 199 *
201 * On x86, we implement read-write locks as a 32-bit counter 200 * On x86, we implement read-write locks using the generic qrwlock with
202 * with the high bit (sign) being the "contended" bit. 201 * x86 specific optimization.
203 */ 202 */
204 203
205/**
206 * read_can_lock - would read_trylock() succeed?
207 * @lock: the rwlock in question.
208 */
209static inline int arch_read_can_lock(arch_rwlock_t *lock)
210{
211 return lock->lock > 0;
212}
213
214/**
215 * write_can_lock - would write_trylock() succeed?
216 * @lock: the rwlock in question.
217 */
218static inline int arch_write_can_lock(arch_rwlock_t *lock)
219{
220 return lock->write == WRITE_LOCK_CMP;
221}
222
223static inline void arch_read_lock(arch_rwlock_t *rw)
224{
225 asm volatile(LOCK_PREFIX READ_LOCK_SIZE(dec) " (%0)\n\t"
226 "jns 1f\n"
227 "call __read_lock_failed\n\t"
228 "1:\n"
229 ::LOCK_PTR_REG (rw) : "memory");
230}
231
232static inline void arch_write_lock(arch_rwlock_t *rw)
233{
234 asm volatile(LOCK_PREFIX WRITE_LOCK_SUB(%1) "(%0)\n\t"
235 "jz 1f\n"
236 "call __write_lock_failed\n\t"
237 "1:\n"
238 ::LOCK_PTR_REG (&rw->write), "i" (RW_LOCK_BIAS)
239 : "memory");
240}
241
242static inline int arch_read_trylock(arch_rwlock_t *lock)
243{
244 READ_LOCK_ATOMIC(t) *count = (READ_LOCK_ATOMIC(t) *)lock;
245
246 if (READ_LOCK_ATOMIC(dec_return)(count) >= 0)
247 return 1;
248 READ_LOCK_ATOMIC(inc)(count);
249 return 0;
250}
251
252static inline int arch_write_trylock(arch_rwlock_t *lock)
253{
254 atomic_t *count = (atomic_t *)&lock->write;
255
256 if (atomic_sub_and_test(WRITE_LOCK_CMP, count))
257 return 1;
258 atomic_add(WRITE_LOCK_CMP, count);
259 return 0;
260}
261
262static inline void arch_read_unlock(arch_rwlock_t *rw)
263{
264 asm volatile(LOCK_PREFIX READ_LOCK_SIZE(inc) " %0"
265 :"+m" (rw->lock) : : "memory");
266}
267
268static inline void arch_write_unlock(arch_rwlock_t *rw)
269{
270 asm volatile(LOCK_PREFIX WRITE_LOCK_ADD(%1) "%0"
271 : "+m" (rw->write) : "i" (RW_LOCK_BIAS) : "memory");
272}
273#else
274#include <asm/qrwlock.h> 204#include <asm/qrwlock.h>
275#endif /* CONFIG_QUEUE_RWLOCK */
276 205
277#define arch_read_lock_flags(lock, flags) arch_read_lock(lock) 206#define arch_read_lock_flags(lock, flags) arch_read_lock(lock)
278#define arch_write_lock_flags(lock, flags) arch_write_lock(lock) 207#define arch_write_lock_flags(lock, flags) arch_write_lock(lock)
279 208
280#undef READ_LOCK_SIZE
281#undef READ_LOCK_ATOMIC
282#undef WRITE_LOCK_ADD
283#undef WRITE_LOCK_SUB
284#undef WRITE_LOCK_CMP
285
286#define arch_spin_relax(lock) cpu_relax() 209#define arch_spin_relax(lock) cpu_relax()
287#define arch_read_relax(lock) cpu_relax() 210#define arch_read_relax(lock) cpu_relax()
288#define arch_write_relax(lock) cpu_relax() 211#define arch_write_relax(lock) cpu_relax()
diff --git a/arch/x86/include/asm/spinlock_types.h b/arch/x86/include/asm/spinlock_types.h
index 73c4c007200f..5f9d7572d82b 100644
--- a/arch/x86/include/asm/spinlock_types.h
+++ b/arch/x86/include/asm/spinlock_types.h
@@ -34,10 +34,6 @@ typedef struct arch_spinlock {
34 34
35#define __ARCH_SPIN_LOCK_UNLOCKED { { 0 } } 35#define __ARCH_SPIN_LOCK_UNLOCKED { { 0 } }
36 36
37#ifdef CONFIG_QUEUE_RWLOCK
38#include <asm-generic/qrwlock_types.h> 37#include <asm-generic/qrwlock_types.h>
39#else
40#include <asm/rwlock.h>
41#endif
42 38
43#endif /* _ASM_X86_SPINLOCK_TYPES_H */ 39#endif /* _ASM_X86_SPINLOCK_TYPES_H */
diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 4d4f96a27638..7ef9a30e7dac 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -20,7 +20,6 @@ lib-y := delay.o misc.o cmdline.o
20lib-y += thunk_$(BITS).o 20lib-y += thunk_$(BITS).o
21lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o 21lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o
22lib-y += memcpy_$(BITS).o 22lib-y += memcpy_$(BITS).o
23lib-$(CONFIG_SMP) += rwlock.o
24lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o 23lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o
25lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o 24lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o
26 25
diff --git a/arch/x86/lib/rwlock.S b/arch/x86/lib/rwlock.S
deleted file mode 100644
index 1cad22139c88..000000000000
--- a/arch/x86/lib/rwlock.S
+++ /dev/null
@@ -1,44 +0,0 @@
1/* Slow paths of read/write spinlocks. */
2
3#include <linux/linkage.h>
4#include <asm/alternative-asm.h>
5#include <asm/frame.h>
6#include <asm/rwlock.h>
7
8#ifdef CONFIG_X86_32
9# define __lock_ptr eax
10#else
11# define __lock_ptr rdi
12#endif
13
14ENTRY(__write_lock_failed)
15 CFI_STARTPROC
16 FRAME
170: LOCK_PREFIX
18 WRITE_LOCK_ADD($RW_LOCK_BIAS) (%__lock_ptr)
191: rep; nop
20 cmpl $WRITE_LOCK_CMP, (%__lock_ptr)
21 jne 1b
22 LOCK_PREFIX
23 WRITE_LOCK_SUB($RW_LOCK_BIAS) (%__lock_ptr)
24 jnz 0b
25 ENDFRAME
26 ret
27 CFI_ENDPROC
28END(__write_lock_failed)
29
30ENTRY(__read_lock_failed)
31 CFI_STARTPROC
32 FRAME
330: LOCK_PREFIX
34 READ_LOCK_SIZE(inc) (%__lock_ptr)
351: rep; nop
36 READ_LOCK_SIZE(cmp) $1, (%__lock_ptr)
37 js 1b
38 LOCK_PREFIX
39 READ_LOCK_SIZE(dec) (%__lock_ptr)
40 js 0b
41 ENDFRAME
42 ret
43 CFI_ENDPROC
44END(__read_lock_failed)
diff --git a/drivers/gpu/drm/drm_modeset_lock.c b/drivers/gpu/drm/drm_modeset_lock.c
index 0dc57d5ecd10..3a02e5e3e9f3 100644
--- a/drivers/gpu/drm/drm_modeset_lock.c
+++ b/drivers/gpu/drm/drm_modeset_lock.c
@@ -35,7 +35,7 @@
35 * of extra utility/tracking out of our acquire-ctx. This is provided 35 * of extra utility/tracking out of our acquire-ctx. This is provided
36 * by drm_modeset_lock / drm_modeset_acquire_ctx. 36 * by drm_modeset_lock / drm_modeset_acquire_ctx.
37 * 37 *
38 * For basic principles of ww_mutex, see: Documentation/ww-mutex-design.txt 38 * For basic principles of ww_mutex, see: Documentation/locking/ww-mutex-design.txt
39 * 39 *
40 * The basic usage pattern is to: 40 * The basic usage pattern is to:
41 * 41 *
diff --git a/include/linux/atomic.h b/include/linux/atomic.h
index fef3a809e7cf..5b08a8540ecf 100644
--- a/include/linux/atomic.h
+++ b/include/linux/atomic.h
@@ -3,42 +3,6 @@
3#define _LINUX_ATOMIC_H 3#define _LINUX_ATOMIC_H
4#include <asm/atomic.h> 4#include <asm/atomic.h>
5 5
6/*
7 * Provide __deprecated wrappers for the new interface, avoid flag day changes.
8 * We need the ugly external functions to break header recursion hell.
9 */
10#ifndef smp_mb__before_atomic_inc
11static inline void __deprecated smp_mb__before_atomic_inc(void)
12{
13 extern void __smp_mb__before_atomic(void);
14 __smp_mb__before_atomic();
15}
16#endif
17
18#ifndef smp_mb__after_atomic_inc
19static inline void __deprecated smp_mb__after_atomic_inc(void)
20{
21 extern void __smp_mb__after_atomic(void);
22 __smp_mb__after_atomic();
23}
24#endif
25
26#ifndef smp_mb__before_atomic_dec
27static inline void __deprecated smp_mb__before_atomic_dec(void)
28{
29 extern void __smp_mb__before_atomic(void);
30 __smp_mb__before_atomic();
31}
32#endif
33
34#ifndef smp_mb__after_atomic_dec
35static inline void __deprecated smp_mb__after_atomic_dec(void)
36{
37 extern void __smp_mb__after_atomic(void);
38 __smp_mb__after_atomic();
39}
40#endif
41
42/** 6/**
43 * atomic_add_unless - add unless the number is already a given value 7 * atomic_add_unless - add unless the number is already a given value
44 * @v: pointer of type atomic_t 8 * @v: pointer of type atomic_t
diff --git a/include/linux/bitops.h b/include/linux/bitops.h
index cbc5833fb221..be5fd38bd5a0 100644
--- a/include/linux/bitops.h
+++ b/include/linux/bitops.h
@@ -32,26 +32,6 @@ extern unsigned long __sw_hweight64(__u64 w);
32 */ 32 */
33#include <asm/bitops.h> 33#include <asm/bitops.h>
34 34
35/*
36 * Provide __deprecated wrappers for the new interface, avoid flag day changes.
37 * We need the ugly external functions to break header recursion hell.
38 */
39#ifndef smp_mb__before_clear_bit
40static inline void __deprecated smp_mb__before_clear_bit(void)
41{
42 extern void __smp_mb__before_atomic(void);
43 __smp_mb__before_atomic();
44}
45#endif
46
47#ifndef smp_mb__after_clear_bit
48static inline void __deprecated smp_mb__after_clear_bit(void)
49{
50 extern void __smp_mb__after_atomic(void);
51 __smp_mb__after_atomic();
52}
53#endif
54
55#define for_each_set_bit(bit, addr, size) \ 35#define for_each_set_bit(bit, addr, size) \
56 for ((bit) = find_first_bit((addr), (size)); \ 36 for ((bit) = find_first_bit((addr), (size)); \
57 (bit) < (size); \ 37 (bit) < (size); \
diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h
index 72d6dea7fac1..74ab23176e9b 100644
--- a/include/linux/lockdep.h
+++ b/include/linux/lockdep.h
@@ -4,7 +4,7 @@
4 * Copyright (C) 2006,2007 Red Hat, Inc., Ingo Molnar <mingo@redhat.com> 4 * Copyright (C) 2006,2007 Red Hat, Inc., Ingo Molnar <mingo@redhat.com>
5 * Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com> 5 * Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra <pzijlstr@redhat.com>
6 * 6 *
7 * see Documentation/lockdep-design.txt for more details. 7 * see Documentation/locking/lockdep-design.txt for more details.
8 */ 8 */
9#ifndef __LINUX_LOCKDEP_H 9#ifndef __LINUX_LOCKDEP_H
10#define __LINUX_LOCKDEP_H 10#define __LINUX_LOCKDEP_H
diff --git a/include/linux/mutex.h b/include/linux/mutex.h
index 8d5535c58cc2..cc31498fc526 100644
--- a/include/linux/mutex.h
+++ b/include/linux/mutex.h
@@ -52,7 +52,7 @@ struct mutex {
52 atomic_t count; 52 atomic_t count;
53 spinlock_t wait_lock; 53 spinlock_t wait_lock;
54 struct list_head wait_list; 54 struct list_head wait_list;
55#if defined(CONFIG_DEBUG_MUTEXES) || defined(CONFIG_SMP) 55#if defined(CONFIG_DEBUG_MUTEXES) || defined(CONFIG_MUTEX_SPIN_ON_OWNER)
56 struct task_struct *owner; 56 struct task_struct *owner;
57#endif 57#endif
58#ifdef CONFIG_MUTEX_SPIN_ON_OWNER 58#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
@@ -133,7 +133,7 @@ static inline int mutex_is_locked(struct mutex *lock)
133 133
134/* 134/*
135 * See kernel/locking/mutex.c for detailed documentation of these APIs. 135 * See kernel/locking/mutex.c for detailed documentation of these APIs.
136 * Also see Documentation/mutex-design.txt. 136 * Also see Documentation/locking/mutex-design.txt.
137 */ 137 */
138#ifdef CONFIG_DEBUG_LOCK_ALLOC 138#ifdef CONFIG_DEBUG_LOCK_ALLOC
139extern void mutex_lock_nested(struct mutex *lock, unsigned int subclass); 139extern void mutex_lock_nested(struct mutex *lock, unsigned int subclass);
diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h
index 035d3c57fc8a..8f498cdde280 100644
--- a/include/linux/rwsem.h
+++ b/include/linux/rwsem.h
@@ -149,7 +149,7 @@ extern void downgrade_write(struct rw_semaphore *sem);
149 * static then another method for expressing nested locking is 149 * static then another method for expressing nested locking is
150 * the explicit definition of lock class keys and the use of 150 * the explicit definition of lock class keys and the use of
151 * lockdep_set_class() at lock initialization time. 151 * lockdep_set_class() at lock initialization time.
152 * See Documentation/lockdep-design.txt for more details.) 152 * See Documentation/locking/lockdep-design.txt for more details.)
153 */ 153 */
154extern void down_read_nested(struct rw_semaphore *sem, int subclass); 154extern void down_read_nested(struct rw_semaphore *sem, int subclass);
155extern void down_write_nested(struct rw_semaphore *sem, int subclass); 155extern void down_write_nested(struct rw_semaphore *sem, int subclass);
diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h
index 3f2867ff0ced..262ba4ef9a8e 100644
--- a/include/linux/spinlock.h
+++ b/include/linux/spinlock.h
@@ -197,7 +197,13 @@ static inline void do_raw_spin_unlock(raw_spinlock_t *lock) __releases(lock)
197 _raw_spin_lock_nest_lock(lock, &(nest_lock)->dep_map); \ 197 _raw_spin_lock_nest_lock(lock, &(nest_lock)->dep_map); \
198 } while (0) 198 } while (0)
199#else 199#else
200# define raw_spin_lock_nested(lock, subclass) _raw_spin_lock(lock) 200/*
201 * Always evaluate the 'subclass' argument to avoid that the compiler
202 * warns about set-but-not-used variables when building with
203 * CONFIG_DEBUG_LOCK_ALLOC=n and with W=1.
204 */
205# define raw_spin_lock_nested(lock, subclass) \
206 _raw_spin_lock(((void)(subclass), (lock)))
201# define raw_spin_lock_nest_lock(lock, nest_lock) _raw_spin_lock(lock) 207# define raw_spin_lock_nest_lock(lock, nest_lock) _raw_spin_lock(lock)
202#endif 208#endif
203 209
diff --git a/kernel/locking/mcs_spinlock.h b/kernel/locking/mcs_spinlock.h
index 23e89c5930e9..4d60986fcbee 100644
--- a/kernel/locking/mcs_spinlock.h
+++ b/kernel/locking/mcs_spinlock.h
@@ -56,9 +56,6 @@ do { \
56 * If the lock has already been acquired, then this will proceed to spin 56 * If the lock has already been acquired, then this will proceed to spin
57 * on this node->locked until the previous lock holder sets the node->locked 57 * on this node->locked until the previous lock holder sets the node->locked
58 * in mcs_spin_unlock(). 58 * in mcs_spin_unlock().
59 *
60 * We don't inline mcs_spin_lock() so that perf can correctly account for the
61 * time spent in this lock function.
62 */ 59 */
63static inline 60static inline
64void mcs_spin_lock(struct mcs_spinlock **lock, struct mcs_spinlock *node) 61void mcs_spin_lock(struct mcs_spinlock **lock, struct mcs_spinlock *node)
diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c
index ae712b25e492..dadbf88c22c4 100644
--- a/kernel/locking/mutex.c
+++ b/kernel/locking/mutex.c
@@ -15,7 +15,7 @@
15 * by Steven Rostedt, based on work by Gregory Haskins, Peter Morreale 15 * by Steven Rostedt, based on work by Gregory Haskins, Peter Morreale
16 * and Sven Dietrich. 16 * and Sven Dietrich.
17 * 17 *
18 * Also see Documentation/mutex-design.txt. 18 * Also see Documentation/locking/mutex-design.txt.
19 */ 19 */
20#include <linux/mutex.h> 20#include <linux/mutex.h>
21#include <linux/ww_mutex.h> 21#include <linux/ww_mutex.h>
@@ -106,6 +106,92 @@ void __sched mutex_lock(struct mutex *lock)
106EXPORT_SYMBOL(mutex_lock); 106EXPORT_SYMBOL(mutex_lock);
107#endif 107#endif
108 108
109static __always_inline void ww_mutex_lock_acquired(struct ww_mutex *ww,
110 struct ww_acquire_ctx *ww_ctx)
111{
112#ifdef CONFIG_DEBUG_MUTEXES
113 /*
114 * If this WARN_ON triggers, you used ww_mutex_lock to acquire,
115 * but released with a normal mutex_unlock in this call.
116 *
117 * This should never happen, always use ww_mutex_unlock.
118 */
119 DEBUG_LOCKS_WARN_ON(ww->ctx);
120
121 /*
122 * Not quite done after calling ww_acquire_done() ?
123 */
124 DEBUG_LOCKS_WARN_ON(ww_ctx->done_acquire);
125
126 if (ww_ctx->contending_lock) {
127 /*
128 * After -EDEADLK you tried to
129 * acquire a different ww_mutex? Bad!
130 */
131 DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock != ww);
132
133 /*
134 * You called ww_mutex_lock after receiving -EDEADLK,
135 * but 'forgot' to unlock everything else first?
136 */
137 DEBUG_LOCKS_WARN_ON(ww_ctx->acquired > 0);
138 ww_ctx->contending_lock = NULL;
139 }
140
141 /*
142 * Naughty, using a different class will lead to undefined behavior!
143 */
144 DEBUG_LOCKS_WARN_ON(ww_ctx->ww_class != ww->ww_class);
145#endif
146 ww_ctx->acquired++;
147}
148
149/*
150 * after acquiring lock with fastpath or when we lost out in contested
151 * slowpath, set ctx and wake up any waiters so they can recheck.
152 *
153 * This function is never called when CONFIG_DEBUG_LOCK_ALLOC is set,
154 * as the fastpath and opportunistic spinning are disabled in that case.
155 */
156static __always_inline void
157ww_mutex_set_context_fastpath(struct ww_mutex *lock,
158 struct ww_acquire_ctx *ctx)
159{
160 unsigned long flags;
161 struct mutex_waiter *cur;
162
163 ww_mutex_lock_acquired(lock, ctx);
164
165 lock->ctx = ctx;
166
167 /*
168 * The lock->ctx update should be visible on all cores before
169 * the atomic read is done, otherwise contended waiters might be
170 * missed. The contended waiters will either see ww_ctx == NULL
171 * and keep spinning, or it will acquire wait_lock, add itself
172 * to waiter list and sleep.
173 */
174 smp_mb(); /* ^^^ */
175
176 /*
177 * Check if lock is contended, if not there is nobody to wake up
178 */
179 if (likely(atomic_read(&lock->base.count) == 0))
180 return;
181
182 /*
183 * Uh oh, we raced in fastpath, wake up everyone in this case,
184 * so they can see the new lock->ctx.
185 */
186 spin_lock_mutex(&lock->base.wait_lock, flags);
187 list_for_each_entry(cur, &lock->base.wait_list, list) {
188 debug_mutex_wake_waiter(&lock->base, cur);
189 wake_up_process(cur->task);
190 }
191 spin_unlock_mutex(&lock->base.wait_lock, flags);
192}
193
194
109#ifdef CONFIG_MUTEX_SPIN_ON_OWNER 195#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
110/* 196/*
111 * In order to avoid a stampede of mutex spinners from acquiring the mutex 197 * In order to avoid a stampede of mutex spinners from acquiring the mutex
@@ -180,6 +266,129 @@ static inline int mutex_can_spin_on_owner(struct mutex *lock)
180 */ 266 */
181 return retval; 267 return retval;
182} 268}
269
270/*
271 * Atomically try to take the lock when it is available
272 */
273static inline bool mutex_try_to_acquire(struct mutex *lock)
274{
275 return !mutex_is_locked(lock) &&
276 (atomic_cmpxchg(&lock->count, 1, 0) == 1);
277}
278
279/*
280 * Optimistic spinning.
281 *
282 * We try to spin for acquisition when we find that the lock owner
283 * is currently running on a (different) CPU and while we don't
284 * need to reschedule. The rationale is that if the lock owner is
285 * running, it is likely to release the lock soon.
286 *
287 * Since this needs the lock owner, and this mutex implementation
288 * doesn't track the owner atomically in the lock field, we need to
289 * track it non-atomically.
290 *
291 * We can't do this for DEBUG_MUTEXES because that relies on wait_lock
292 * to serialize everything.
293 *
294 * The mutex spinners are queued up using MCS lock so that only one
295 * spinner can compete for the mutex. However, if mutex spinning isn't
296 * going to happen, there is no point in going through the lock/unlock
297 * overhead.
298 *
299 * Returns true when the lock was taken, otherwise false, indicating
300 * that we need to jump to the slowpath and sleep.
301 */
302static bool mutex_optimistic_spin(struct mutex *lock,
303 struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx)
304{
305 struct task_struct *task = current;
306
307 if (!mutex_can_spin_on_owner(lock))
308 goto done;
309
310 if (!osq_lock(&lock->osq))
311 goto done;
312
313 while (true) {
314 struct task_struct *owner;
315
316 if (use_ww_ctx && ww_ctx->acquired > 0) {
317 struct ww_mutex *ww;
318
319 ww = container_of(lock, struct ww_mutex, base);
320 /*
321 * If ww->ctx is set the contents are undefined, only
322 * by acquiring wait_lock there is a guarantee that
323 * they are not invalid when reading.
324 *
325 * As such, when deadlock detection needs to be
326 * performed the optimistic spinning cannot be done.
327 */
328 if (ACCESS_ONCE(ww->ctx))
329 break;
330 }
331
332 /*
333 * If there's an owner, wait for it to either
334 * release the lock or go to sleep.
335 */
336 owner = ACCESS_ONCE(lock->owner);
337 if (owner && !mutex_spin_on_owner(lock, owner))
338 break;
339
340 /* Try to acquire the mutex if it is unlocked. */
341 if (mutex_try_to_acquire(lock)) {
342 lock_acquired(&lock->dep_map, ip);
343
344 if (use_ww_ctx) {
345 struct ww_mutex *ww;
346 ww = container_of(lock, struct ww_mutex, base);
347
348 ww_mutex_set_context_fastpath(ww, ww_ctx);
349 }
350
351 mutex_set_owner(lock);
352 osq_unlock(&lock->osq);
353 return true;
354 }
355
356 /*
357 * When there's no owner, we might have preempted between the
358 * owner acquiring the lock and setting the owner field. If
359 * we're an RT task that will live-lock because we won't let
360 * the owner complete.
361 */
362 if (!owner && (need_resched() || rt_task(task)))
363 break;
364
365 /*
366 * The cpu_relax() call is a compiler barrier which forces
367 * everything in this loop to be re-loaded. We don't need
368 * memory barriers as we'll eventually observe the right
369 * values at the cost of a few extra spins.
370 */
371 cpu_relax_lowlatency();
372 }
373
374 osq_unlock(&lock->osq);
375done:
376 /*
377 * If we fell out of the spin path because of need_resched(),
378 * reschedule now, before we try-lock the mutex. This avoids getting
379 * scheduled out right after we obtained the mutex.
380 */
381 if (need_resched())
382 schedule_preempt_disabled();
383
384 return false;
385}
386#else
387static bool mutex_optimistic_spin(struct mutex *lock,
388 struct ww_acquire_ctx *ww_ctx, const bool use_ww_ctx)
389{
390 return false;
391}
183#endif 392#endif
184 393
185__visible __used noinline 394__visible __used noinline
@@ -277,91 +486,6 @@ __mutex_lock_check_stamp(struct mutex *lock, struct ww_acquire_ctx *ctx)
277 return 0; 486 return 0;
278} 487}
279 488
280static __always_inline void ww_mutex_lock_acquired(struct ww_mutex *ww,
281 struct ww_acquire_ctx *ww_ctx)
282{
283#ifdef CONFIG_DEBUG_MUTEXES
284 /*
285 * If this WARN_ON triggers, you used ww_mutex_lock to acquire,
286 * but released with a normal mutex_unlock in this call.
287 *
288 * This should never happen, always use ww_mutex_unlock.
289 */
290 DEBUG_LOCKS_WARN_ON(ww->ctx);
291
292 /*
293 * Not quite done after calling ww_acquire_done() ?
294 */
295 DEBUG_LOCKS_WARN_ON(ww_ctx->done_acquire);
296
297 if (ww_ctx->contending_lock) {
298 /*
299 * After -EDEADLK you tried to
300 * acquire a different ww_mutex? Bad!
301 */
302 DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock != ww);
303
304 /*
305 * You called ww_mutex_lock after receiving -EDEADLK,
306 * but 'forgot' to unlock everything else first?
307 */
308 DEBUG_LOCKS_WARN_ON(ww_ctx->acquired > 0);
309 ww_ctx->contending_lock = NULL;
310 }
311
312 /*
313 * Naughty, using a different class will lead to undefined behavior!
314 */
315 DEBUG_LOCKS_WARN_ON(ww_ctx->ww_class != ww->ww_class);
316#endif
317 ww_ctx->acquired++;
318}
319
320/*
321 * after acquiring lock with fastpath or when we lost out in contested
322 * slowpath, set ctx and wake up any waiters so they can recheck.
323 *
324 * This function is never called when CONFIG_DEBUG_LOCK_ALLOC is set,
325 * as the fastpath and opportunistic spinning are disabled in that case.
326 */
327static __always_inline void
328ww_mutex_set_context_fastpath(struct ww_mutex *lock,
329 struct ww_acquire_ctx *ctx)
330{
331 unsigned long flags;
332 struct mutex_waiter *cur;
333
334 ww_mutex_lock_acquired(lock, ctx);
335
336 lock->ctx = ctx;
337
338 /*
339 * The lock->ctx update should be visible on all cores before
340 * the atomic read is done, otherwise contended waiters might be
341 * missed. The contended waiters will either see ww_ctx == NULL
342 * and keep spinning, or it will acquire wait_lock, add itself
343 * to waiter list and sleep.
344 */
345 smp_mb(); /* ^^^ */
346
347 /*
348 * Check if lock is contended, if not there is nobody to wake up
349 */
350 if (likely(atomic_read(&lock->base.count) == 0))
351 return;
352
353 /*
354 * Uh oh, we raced in fastpath, wake up everyone in this case,
355 * so they can see the new lock->ctx.
356 */
357 spin_lock_mutex(&lock->base.wait_lock, flags);
358 list_for_each_entry(cur, &lock->base.wait_list, list) {
359 debug_mutex_wake_waiter(&lock->base, cur);
360 wake_up_process(cur->task);
361 }
362 spin_unlock_mutex(&lock->base.wait_lock, flags);
363}
364
365/* 489/*
366 * Lock a mutex (possibly interruptible), slowpath: 490 * Lock a mutex (possibly interruptible), slowpath:
367 */ 491 */
@@ -378,104 +502,12 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass,
378 preempt_disable(); 502 preempt_disable();
379 mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip); 503 mutex_acquire_nest(&lock->dep_map, subclass, 0, nest_lock, ip);
380 504
381#ifdef CONFIG_MUTEX_SPIN_ON_OWNER 505 if (mutex_optimistic_spin(lock, ww_ctx, use_ww_ctx)) {
382 /* 506 /* got the lock, yay! */
383 * Optimistic spinning. 507 preempt_enable();
384 * 508 return 0;
385 * We try to spin for acquisition when we find that the lock owner
386 * is currently running on a (different) CPU and while we don't
387 * need to reschedule. The rationale is that if the lock owner is
388 * running, it is likely to release the lock soon.
389 *
390 * Since this needs the lock owner, and this mutex implementation
391 * doesn't track the owner atomically in the lock field, we need to
392 * track it non-atomically.
393 *
394 * We can't do this for DEBUG_MUTEXES because that relies on wait_lock
395 * to serialize everything.
396 *
397 * The mutex spinners are queued up using MCS lock so that only one
398 * spinner can compete for the mutex. However, if mutex spinning isn't
399 * going to happen, there is no point in going through the lock/unlock
400 * overhead.
401 */
402 if (!mutex_can_spin_on_owner(lock))
403 goto slowpath;
404
405 if (!osq_lock(&lock->osq))
406 goto slowpath;
407
408 for (;;) {
409 struct task_struct *owner;
410
411 if (use_ww_ctx && ww_ctx->acquired > 0) {
412 struct ww_mutex *ww;
413
414 ww = container_of(lock, struct ww_mutex, base);
415 /*
416 * If ww->ctx is set the contents are undefined, only
417 * by acquiring wait_lock there is a guarantee that
418 * they are not invalid when reading.
419 *
420 * As such, when deadlock detection needs to be
421 * performed the optimistic spinning cannot be done.
422 */
423 if (ACCESS_ONCE(ww->ctx))
424 break;
425 }
426
427 /*
428 * If there's an owner, wait for it to either
429 * release the lock or go to sleep.
430 */
431 owner = ACCESS_ONCE(lock->owner);
432 if (owner && !mutex_spin_on_owner(lock, owner))
433 break;
434
435 /* Try to acquire the mutex if it is unlocked. */
436 if (!mutex_is_locked(lock) &&
437 (atomic_cmpxchg(&lock->count, 1, 0) == 1)) {
438 lock_acquired(&lock->dep_map, ip);
439 if (use_ww_ctx) {
440 struct ww_mutex *ww;
441 ww = container_of(lock, struct ww_mutex, base);
442
443 ww_mutex_set_context_fastpath(ww, ww_ctx);
444 }
445
446 mutex_set_owner(lock);
447 osq_unlock(&lock->osq);
448 preempt_enable();
449 return 0;
450 }
451
452 /*
453 * When there's no owner, we might have preempted between the
454 * owner acquiring the lock and setting the owner field. If
455 * we're an RT task that will live-lock because we won't let
456 * the owner complete.
457 */
458 if (!owner && (need_resched() || rt_task(task)))
459 break;
460
461 /*
462 * The cpu_relax() call is a compiler barrier which forces
463 * everything in this loop to be re-loaded. We don't need
464 * memory barriers as we'll eventually observe the right
465 * values at the cost of a few extra spins.
466 */
467 cpu_relax_lowlatency();
468 } 509 }
469 osq_unlock(&lock->osq); 510
470slowpath:
471 /*
472 * If we fell out of the spin path because of need_resched(),
473 * reschedule now, before we try-lock the mutex. This avoids getting
474 * scheduled out right after we obtained the mutex.
475 */
476 if (need_resched())
477 schedule_preempt_disabled();
478#endif
479 spin_lock_mutex(&lock->wait_lock, flags); 511 spin_lock_mutex(&lock->wait_lock, flags);
480 512
481 /* 513 /*
@@ -679,15 +711,21 @@ EXPORT_SYMBOL_GPL(__ww_mutex_lock_interruptible);
679 * Release the lock, slowpath: 711 * Release the lock, slowpath:
680 */ 712 */
681static inline void 713static inline void
682__mutex_unlock_common_slowpath(atomic_t *lock_count, int nested) 714__mutex_unlock_common_slowpath(struct mutex *lock, int nested)
683{ 715{
684 struct mutex *lock = container_of(lock_count, struct mutex, count);
685 unsigned long flags; 716 unsigned long flags;
686 717
687 /* 718 /*
688 * some architectures leave the lock unlocked in the fastpath failure 719 * As a performance measurement, release the lock before doing other
720 * wakeup related duties to follow. This allows other tasks to acquire
721 * the lock sooner, while still handling cleanups in past unlock calls.
722 * This can be done as we do not enforce strict equivalence between the
723 * mutex counter and wait_list.
724 *
725 *
726 * Some architectures leave the lock unlocked in the fastpath failure
689 * case, others need to leave it locked. In the later case we have to 727 * case, others need to leave it locked. In the later case we have to
690 * unlock it here 728 * unlock it here - as the lock counter is currently 0 or negative.
691 */ 729 */
692 if (__mutex_slowpath_needs_to_unlock()) 730 if (__mutex_slowpath_needs_to_unlock())
693 atomic_set(&lock->count, 1); 731 atomic_set(&lock->count, 1);
@@ -716,7 +754,9 @@ __mutex_unlock_common_slowpath(atomic_t *lock_count, int nested)
716__visible void 754__visible void
717__mutex_unlock_slowpath(atomic_t *lock_count) 755__mutex_unlock_slowpath(atomic_t *lock_count)
718{ 756{
719 __mutex_unlock_common_slowpath(lock_count, 1); 757 struct mutex *lock = container_of(lock_count, struct mutex, count);
758
759 __mutex_unlock_common_slowpath(lock, 1);
720} 760}
721 761
722#ifndef CONFIG_DEBUG_LOCK_ALLOC 762#ifndef CONFIG_DEBUG_LOCK_ALLOC
diff --git a/kernel/locking/mutex.h b/kernel/locking/mutex.h
index 4115fbf83b12..5cda397607f2 100644
--- a/kernel/locking/mutex.h
+++ b/kernel/locking/mutex.h
@@ -16,7 +16,7 @@
16#define mutex_remove_waiter(lock, waiter, ti) \ 16#define mutex_remove_waiter(lock, waiter, ti) \
17 __list_del((waiter)->list.prev, (waiter)->list.next) 17 __list_del((waiter)->list.prev, (waiter)->list.next)
18 18
19#ifdef CONFIG_SMP 19#ifdef CONFIG_MUTEX_SPIN_ON_OWNER
20static inline void mutex_set_owner(struct mutex *lock) 20static inline void mutex_set_owner(struct mutex *lock)
21{ 21{
22 lock->owner = current; 22 lock->owner = current;
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index a0ea2a141b3b..7c98873a3077 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -8,7 +8,7 @@
8 * Copyright (C) 2005 Kihon Technologies Inc., Steven Rostedt 8 * Copyright (C) 2005 Kihon Technologies Inc., Steven Rostedt
9 * Copyright (C) 2006 Esben Nielsen 9 * Copyright (C) 2006 Esben Nielsen
10 * 10 *
11 * See Documentation/rt-mutex-design.txt for details. 11 * See Documentation/locking/rt-mutex-design.txt for details.
12 */ 12 */
13#include <linux/spinlock.h> 13#include <linux/spinlock.h>
14#include <linux/export.h> 14#include <linux/export.h>
diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
index d6203faf2eb1..7628c3fc37ca 100644
--- a/kernel/locking/rwsem-xadd.c
+++ b/kernel/locking/rwsem-xadd.c
@@ -246,19 +246,22 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)
246 246
247 return sem; 247 return sem;
248} 248}
249EXPORT_SYMBOL(rwsem_down_read_failed);
249 250
250static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem) 251static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem)
251{ 252{
252 if (!(count & RWSEM_ACTIVE_MASK)) { 253 /*
253 /* try acquiring the write lock */ 254 * Try acquiring the write lock. Check count first in order
254 if (sem->count == RWSEM_WAITING_BIAS && 255 * to reduce unnecessary expensive cmpxchg() operations.
255 cmpxchg(&sem->count, RWSEM_WAITING_BIAS, 256 */
256 RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) { 257 if (count == RWSEM_WAITING_BIAS &&
257 if (!list_is_singular(&sem->wait_list)) 258 cmpxchg(&sem->count, RWSEM_WAITING_BIAS,
258 rwsem_atomic_update(RWSEM_WAITING_BIAS, sem); 259 RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) {
259 return true; 260 if (!list_is_singular(&sem->wait_list))
260 } 261 rwsem_atomic_update(RWSEM_WAITING_BIAS, sem);
262 return true;
261 } 263 }
264
262 return false; 265 return false;
263} 266}
264 267
@@ -465,6 +468,7 @@ struct rw_semaphore __sched *rwsem_down_write_failed(struct rw_semaphore *sem)
465 468
466 return sem; 469 return sem;
467} 470}
471EXPORT_SYMBOL(rwsem_down_write_failed);
468 472
469/* 473/*
470 * handle waking up a waiter on the semaphore 474 * handle waking up a waiter on the semaphore
@@ -485,6 +489,7 @@ struct rw_semaphore *rwsem_wake(struct rw_semaphore *sem)
485 489
486 return sem; 490 return sem;
487} 491}
492EXPORT_SYMBOL(rwsem_wake);
488 493
489/* 494/*
490 * downgrade a write lock into a read lock 495 * downgrade a write lock into a read lock
@@ -506,8 +511,4 @@ struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem)
506 511
507 return sem; 512 return sem;
508} 513}
509
510EXPORT_SYMBOL(rwsem_down_read_failed);
511EXPORT_SYMBOL(rwsem_down_write_failed);
512EXPORT_SYMBOL(rwsem_wake);
513EXPORT_SYMBOL(rwsem_downgrade_wake); 514EXPORT_SYMBOL(rwsem_downgrade_wake);
diff --git a/kernel/locking/semaphore.c b/kernel/locking/semaphore.c
index 6815171a4fff..b8120abe594b 100644
--- a/kernel/locking/semaphore.c
+++ b/kernel/locking/semaphore.c
@@ -36,7 +36,7 @@
36static noinline void __down(struct semaphore *sem); 36static noinline void __down(struct semaphore *sem);
37static noinline int __down_interruptible(struct semaphore *sem); 37static noinline int __down_interruptible(struct semaphore *sem);
38static noinline int __down_killable(struct semaphore *sem); 38static noinline int __down_killable(struct semaphore *sem);
39static noinline int __down_timeout(struct semaphore *sem, long jiffies); 39static noinline int __down_timeout(struct semaphore *sem, long timeout);
40static noinline void __up(struct semaphore *sem); 40static noinline void __up(struct semaphore *sem);
41 41
42/** 42/**
@@ -145,14 +145,14 @@ EXPORT_SYMBOL(down_trylock);
145/** 145/**
146 * down_timeout - acquire the semaphore within a specified time 146 * down_timeout - acquire the semaphore within a specified time
147 * @sem: the semaphore to be acquired 147 * @sem: the semaphore to be acquired
148 * @jiffies: how long to wait before failing 148 * @timeout: how long to wait before failing
149 * 149 *
150 * Attempts to acquire the semaphore. If no more tasks are allowed to 150 * Attempts to acquire the semaphore. If no more tasks are allowed to
151 * acquire the semaphore, calling this function will put the task to sleep. 151 * acquire the semaphore, calling this function will put the task to sleep.
152 * If the semaphore is not released within the specified number of jiffies, 152 * If the semaphore is not released within the specified number of jiffies,
153 * this function returns -ETIME. It returns 0 if the semaphore was acquired. 153 * this function returns -ETIME. It returns 0 if the semaphore was acquired.
154 */ 154 */
155int down_timeout(struct semaphore *sem, long jiffies) 155int down_timeout(struct semaphore *sem, long timeout)
156{ 156{
157 unsigned long flags; 157 unsigned long flags;
158 int result = 0; 158 int result = 0;
@@ -161,7 +161,7 @@ int down_timeout(struct semaphore *sem, long jiffies)
161 if (likely(sem->count > 0)) 161 if (likely(sem->count > 0))
162 sem->count--; 162 sem->count--;
163 else 163 else
164 result = __down_timeout(sem, jiffies); 164 result = __down_timeout(sem, timeout);
165 raw_spin_unlock_irqrestore(&sem->lock, flags); 165 raw_spin_unlock_irqrestore(&sem->lock, flags);
166 166
167 return result; 167 return result;
@@ -248,9 +248,9 @@ static noinline int __sched __down_killable(struct semaphore *sem)
248 return __down_common(sem, TASK_KILLABLE, MAX_SCHEDULE_TIMEOUT); 248 return __down_common(sem, TASK_KILLABLE, MAX_SCHEDULE_TIMEOUT);
249} 249}
250 250
251static noinline int __sched __down_timeout(struct semaphore *sem, long jiffies) 251static noinline int __sched __down_timeout(struct semaphore *sem, long timeout)
252{ 252{
253 return __down_common(sem, TASK_UNINTERRUPTIBLE, jiffies); 253 return __down_common(sem, TASK_UNINTERRUPTIBLE, timeout);
254} 254}
255 255
256static noinline void __sched __up(struct semaphore *sem) 256static noinline void __sched __up(struct semaphore *sem)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 59965ec0b7de..f235c41a3532 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -90,22 +90,6 @@
90#define CREATE_TRACE_POINTS 90#define CREATE_TRACE_POINTS
91#include <trace/events/sched.h> 91#include <trace/events/sched.h>
92 92
93#ifdef smp_mb__before_atomic
94void __smp_mb__before_atomic(void)
95{
96 smp_mb__before_atomic();
97}
98EXPORT_SYMBOL(__smp_mb__before_atomic);
99#endif
100
101#ifdef smp_mb__after_atomic
102void __smp_mb__after_atomic(void)
103{
104 smp_mb__after_atomic();
105}
106EXPORT_SYMBOL(__smp_mb__after_atomic);
107#endif
108
109void start_bandwidth_timer(struct hrtimer *period_timer, ktime_t period) 93void start_bandwidth_timer(struct hrtimer *period_timer, ktime_t period)
110{ 94{
111 unsigned long delta; 95 unsigned long delta;
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 3ac43f34437b..49d5fb754e88 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -952,7 +952,7 @@ config PROVE_LOCKING
952 the proof of observed correctness is also maintained for an 952 the proof of observed correctness is also maintained for an
953 arbitrary combination of these separate locking variants. 953 arbitrary combination of these separate locking variants.
954 954
955 For more details, see Documentation/lockdep-design.txt. 955 For more details, see Documentation/locking/lockdep-design.txt.
956 956
957config LOCKDEP 957config LOCKDEP
958 bool 958 bool
@@ -973,7 +973,7 @@ config LOCK_STAT
973 help 973 help
974 This feature enables tracking lock contention points 974 This feature enables tracking lock contention points
975 975
976 For more details, see Documentation/lockstat.txt 976 For more details, see Documentation/locking/lockstat.txt
977 977
978 This also enables lock events required by "perf lock", 978 This also enables lock events required by "perf lock",
979 subcommand of perf. 979 subcommand of perf.
generated by cgit v1.2.2 (git 2.25.0) at 2024-11-22 15:08:28 -0500