aboutsummaryrefslogtreecommitdiffstats
Commit message (Collapse)AuthorAge
* An efficient binary heap implementation.wip-stage-binheapGlenn Elliott2012-05-26
| | | | | | | An efficient binary heap implementation coded in the style of Linux's list. This binary heap should be able to replace any partially sorted priority queue based upon Linux's list.
* Add kernel-style events for sched_trace_XXX() functionsAndrea Bastoni2012-03-30
| | | | | | | | | | | | | | | | | | Enable kernel-style events (tracepoint) for Litmus. Litmus events trace the same functions as the sched_trace_XXX(), but can be enabled independently. So, why another tracing infrastructure then: - Litmus tracepoints can be recorded and analyzed together (single time reference) with all other kernel tracing events (e.g., sched:sched_switch, etc.). It's easier to correlate the effects of kernel events on litmus tasks. - It enables a quick way to visualize and process schedule traces using trace-cmd utility and kernelshark visualizer. Kernelshark lacks unit-trace's schedule-correctness checks, but it enables a fast view of schedule traces and it has several filtering options (for all kernel events, not only Litmus').
* Typo in macroJonathan Herman2012-02-16
|
* Feather-Trace: keep track of interrupt-related interference.2012.1Bjoern B. Brandenburg2012-01-30
| | | | | | | Increment a processor-local counter whenever an interrupt is handled. This allows Feather-Trace to include a (truncated) counter and a flag to report interference from interrupts. This could be used to filter samples that were disturbed by interrupts.
* Litmus core: simplify np-section protocolwip-2011.2-bbbBjoern B. Brandenburg2011-11-24
| | | | | | User a 32-bit word for all non-preemptive section flags. Set the "please yield soon" flag atomically when accessing it on remotely-scheduled tasks.
* C-EDF: rename lock -> cluster_lockBjoern B. Brandenburg2011-11-24
| | | | The macro lock conflicts with locking protocols...
* locking: use correct timestampBjoern B. Brandenburg2011-11-24
|
* Feather-trace: let userspace add overhead eventsBjoern B. Brandenburg2011-11-24
| | | | | This is useful for measuring locking-related overheads that are partially recorded in userspace.
* ftdev: let bufffer-specific code handle writes from userspaceBjoern B. Brandenburg2011-11-24
| | | | | This allows us to splice in information into logs from events that were recorded in userspace.
* ftdev: remove event activation hackBjoern B. Brandenburg2011-11-24
| | | | | Instead of doing the hackisch 'write commands to device' thing, let's just use a real ioctl() interface.
* Feather-Trace: keep track of release latencyBjoern B. Brandenburg2011-11-24
|
* Feather-Trace: trace locking-related suspensionsBjoern B. Brandenburg2011-11-24
|
* Feather-Trace: start with the largest permissible rangeBjoern B. Brandenburg2011-11-24
| | | | MAX_ORDER is 11, but this is about number of records, not number of pages.
* bugfix: add processors in order of increasing indices to clustersBjoern B. Brandenburg2011-11-24
| | | | | Pfair expects to look at processors in order of increasing index. Without this patch, Pfair could deadlock in certain situations.
* Pfair: improve robustness of suspensionsBjoern B. Brandenburg2011-11-24
| | | | | | | | | | | | | | | This patch fixes two crash or hang bugs related to suspensions in Pfair. 1) When a job was not present at the end of its last subtask, then its linked_on field was not cleared. This confused the scheduler when it later resumed. Fix: clear the field. 2) Just testing for linked_on == NO_CPU is insufficient in the wake_up path to determine whether a task should be added to the ready queue. If the task remained linked and then was "preempted" at a later quantum boundary, then it already is in the ready queue and nothing is required. Fix: encode need to requeue in task_rt(t)->flags.
* Add option to turn off preemption state tracingBjoern B. Brandenburg2011-11-24
| | | | | | Preemption state tracing is only useful when debugging preemption- and IPI-related races. Since it creates a lot of clutter in the logs, this patch turns it off unless explicitly requested.
* Add unlikely() to rel master check (match pfair).Glenn Elliott2011-11-02
|
* bugfix: release master CPU must signal task was pickedBjoern B. Brandenburg2011-11-02
|
* Pfair: various fixes concerning release timersBjoern B. Brandenburg2011-08-27
|
* Pfair: add support for true sporadic releasesBjoern B. Brandenburg2011-08-27
| | | | | | This patch also converts Pfair to implement early releasing such that no timer wheel is required anymore. This removes the need for a maximum period restriction.
* Pfair: add release master support.Bjoern B. Brandenburg2011-08-27
| | | | | Merged in release master support for Pfair. Some merge conflicts had to be resolved.
* C-EDF: Make migration affinity work with Release MasterGlenn Elliott2011-08-27
| | | | | | | | | | Needed to update C-EDF to handle release master. Also updated get_nearest_available_cpu() to take NO_CPU instead of -1 to indicate that there is no release master. While NO_CPU is 0xffffffff (-1 in two's complement), we still translate this value to -1 in case NO_CPU changes. Signed-off-by: Andrea Bastoni <bastoni@cs.unc.edu>
* C-EDF: add release master supportBjoern B. Brandenburg2011-08-27
| | | | As with GSN-EDF, do not insert release master into CPU heap.
* PSN-EDF: add release master supportBjoern B. Brandenburg2011-08-27
| | | | We can give up a processor under partitioning, too.
* COMMENT: Correct comment on precise budget enforcementGlenn Elliott2011-08-27
| | | | | Original comment said that this feature wasn't supported, though it has been since around October 2010.
* Avoid needlessly costly migrations. CONFIG_SCHED_CPU_AFFINITYGlenn Elliott2011-08-27
| | | | | | | | | | | | | Given a choice between several available CPUs (unlinked) on which to schedule a task, let the scheduler select the CPU closest to where that task was previously scheduled. Hopefully, this will reduce cache migration penalties. Notes: SCHED_CPU_AFFINITY is dependent upon x86 (only x86 is supported at this time). Also PFair/PD^2 does not make use of this feature. Signed-off-by: Andrea Bastoni <bastoni@cs.unc.edu>
* Prevent Linux to send IPI and queue tasks on remote CPUs.wip-merge-v3.0Andrea Bastoni2011-08-27
| | | | | | Whether to send IPIs and enqueue tasks on remote runqueues is plugin-specific. The recent ttwu_queue() mechanism (by calling ttwu_queue_remote()) interferes with Litmus plugin decisions.
* Update PULL_TIMERS_VECTOR numberAndrea Bastoni2011-08-27
| | | | | | From 2.6.39 the "0xee" vector number that we used for pull_timers low-level management is is use by invalidate_tlb_X interrupts. Move the pull_timers vector below the max size of invalidate_tlb.
* Fix prototype mismatching and synch syscall numbersAndrea Bastoni2011-08-27
| | | | | | * Update prototypes for switched_to(), prio_changed(), select_task_rq(). * Fix missing pid field in printk output. * Synchronize syscall numbers for arm and x86.
* Merge 'Linux v3.0' into LitmusAndrea Bastoni2011-08-27
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some notes: * Litmus^RT scheduling class is the topmost scheduling class (above stop_sched_class). * scheduler_ipi() function (e.g., in smp_reschedule_interrupt()) may increase IPI latencies. * Added path into schedule() to quickly re-evaluate scheduling decision without becoming preemptive again. This used to be a standard path before the removal of BKL. Conflicts: Makefile arch/arm/kernel/calls.S arch/arm/kernel/smp.c arch/x86/include/asm/unistd_32.h arch/x86/kernel/smp.c arch/x86/kernel/syscall_table_32.S include/linux/hrtimer.h kernel/printk.c kernel/sched.c kernel/sched_fair.c
| * Linux 3.0Linus Torvalds2011-07-21
| |
| * Merge branch 'for_linus' of ↵Linus Torvalds2011-07-21
| |\ | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/linux-2.6-kgdb * 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/linux-2.6-kgdb: sparc,kgdbts: fix compile regression with kgdb test suite
| | * sparc,kgdbts: fix compile regression with kgdb test suiteJason Wessel2011-07-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit 63ab25ebbc (kgdbts: unify/generalize gdb breakpoint adjustment) introduced a compile regression on sparc. kgdbts.c: In function 'check_and_rewind_pc': kgdbts.c:307: error: implicit declaration of function 'instruction_pointer_set' Simply add the correct macro definition for instruction pointer on the Sparc architecture. Signed-off-by: Jason Wessel <jason.wessel@windriver.com> Acked-by: Mike Frysinger <vapier@gentoo.org> Acked-by: David S. Miller <davem@davemloft.net>
| * | Merge git://git.kernel.org/pub/scm/linux/kernel/git/sfrench/cifs-2.6Linus Torvalds2011-07-21
| |\ \ | | | | | | | | | | | | | | | | * git://git.kernel.org/pub/scm/linux/kernel/git/sfrench/cifs-2.6: CIFS: Fix wrong length in cifs_iovec_read
| | * | CIFS: Fix wrong length in cifs_iovec_readPavel Shilovsky2011-07-20
| | | | | | | | | | | | | | | | | | | | | | | | Signed-off-by: Pavel Shilovsky <piastryyy@gmail.com> Reviewed-by: Jeff Layton <jlayton@redhat.com> Signed-off-by: Steve French <sfrench@us.ibm.com>
| * | | Merge branch 'x86-urgent-for-linus' of ↵Linus Torvalds2011-07-21
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip * 'x86-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip: x86: Make Dell Latitude E6420 use reboot=pci x86: Make Dell Latitude E5420 use reboot=pci
| | * | | x86: Make Dell Latitude E6420 use reboot=pciH. Peter Anvin2011-07-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Yet another variant of the Dell Latitude series which requires reboot=pci. From the E5420 bug report by Daniel J Blueman: > The E6420 is affected also (same platform, different casing and > features), which provides an external confirmation of the issue; I can > submit a patch for that later or include it if you prefer: > http://linux.koolsolutions.com/2009/08/04/howto-fix-linux-hangfreeze-during-reboots-and-restarts/ Reported-by: Daniel J Blueman <daniel.blueman@gmail.com> Signed-off-by: H. Peter Anvin <hpa@zytor.com> Cc: <stable@kernel.org>
| | * | | x86: Make Dell Latitude E5420 use reboot=pciDaniel J Blueman2011-07-21
| | |/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rebooting on the Dell E5420 often hangs with the keyboard or ACPI methods, but is reliable via the PCI method. [ hpa: this was deferred because we believed for a long time that the recent reshuffling of the boot priorities in commit 660e34cebf0a11d54f2d5dd8838607452355f321 fixed this platform. Unfortunately that turned out to be incorrect. ] Signed-off-by: Daniel J Blueman <daniel.blueman@gmail.com> Link: http://lkml.kernel.org/r/1305248699-2347-1-git-send-email-daniel.blueman@gmail.com Signed-off-by: H. Peter Anvin <hpa@zytor.com> Cc: <stable@kernel.org>
| * | | Merge branch 'drm-intel-fixes' of ↵Linus Torvalds2011-07-21
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/keithp/linux-2.6 * 'drm-intel-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/keithp/linux-2.6: drm/i915: Fix unfenced alignment on pre-G33 hardware drm/i915: Add quirk to disable SSC on Lenovo U160 LVDS
| | * | | drm/i915: Fix unfenced alignment on pre-G33 hardwareChris Wilson2011-07-18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Align unfenced buffers on older hardware to the power-of-two object size. The docs suggest that it should be possible to align only to a power-of-two tile height, but using the already computed fence size is easier and always correct. We also have to make sure that we unbind misaligned buffers upon tiling changes. In order to prevent a repetition of this bug, we change the interface to the alignment computation routines to force the caller to provide the requested alignment and size of the GTT binding rather than assume the current values on the object. Reported-and-tested-by: Sitosfe Wheeler <sitsofe@yahoo.com> Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=36326 Signed-off-by: Chris Wilson <chris@chris-wilson.co.uk> Cc: stable@kernel.org Reviewed-by: Daniel Vetter <daniel.vetter@ffwll.ch> Signed-off-by: Keith Packard <keithp@keithp.com>
| | * | | drm/i915: Add quirk to disable SSC on Lenovo U160 LVDSKeith Packard2011-07-14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We've tried several times to make this machine 'just work', but every patch that does causes many other machines to fail. This adds a quirk which special cases this hardware and forces ssc to be disabled. There's no way to override this from the command line; that would be a significantly more invasive change. This patch fixes #36656 on fdo bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=36656 Signed-off-by: Keith Packard <keithp@keithp.com> References: https://bugs.freedesktop.org/show_bug.cgi?id=36656 Reviewed-by: Chris Wilson <chris@chris-wilson.co.uk>
| * | | | vfs: drop conditional inode prefetch in __do_lookup_rcuLinus Torvalds2011-07-21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It seems to hurt performance in real life. Yes, the inode will be used later, but the conditional doesn't seem to predict all that well (negative dentries are not uncommon) and it looks like the cost of prefetching is simply higher than depending on the cache doing the right thing. As usual. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
| * | | | FS-Cache: Fix __fscache_uncache_all_inode_pages()'s outer loopJan Beulich2011-07-21
| | |/ / | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The compiler, at least for ix86 and m68k, validly warns that the comparison: next <= (loff_t)-1 is always true (and it's always true also for x86-64 and probably all other arches - as long as pgoff_t isn't wider than loff_t). The intention appears to be to avoid wrapping of "next", so rather than eliminating the pointless comparison, fix the loop to indeed get exited when "next" would otherwise wrap. On m68k the following warning is observed: fs/fscache/page.c: In function '__fscache_uncache_all_inode_pages': fs/fscache/page.c:979: warning: comparison is always false due to limited range of data type Reported-by: Geert Uytterhoeven <geert@linux-m68k.org> Reported-by: Jan Beulich <jbeulich@novell.com> Signed-off-by: Jan Beulich <jbeulich@novell.com> Signed-off-by: David Howells <dhowells@redhat.com> Cc: Suresh Jayaraman <sjayaraman@suse.de> Cc: Geert Uytterhoeven <geert@linux-m68k.org> Cc: stable@kernel.org Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
| * | | Merge branch 'core-urgent-for-linus' of ↵Linus Torvalds2011-07-20
| |\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip * 'core-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip: signal: align __lock_task_sighand() irq disabling and RCU softirq,rcu: Inform RCU of irq_exit() activity sched: Add irq_{enter,exit}() to scheduler_ipi() rcu: protect __rcu_read_unlock() against scheduler-using irq handlers rcu: Streamline code produced by __rcu_read_unlock() rcu: Fix RCU_BOOST race handling current->rcu_read_unlock_special rcu: decrease rcu_report_exp_rnp coupling with scheduler
| | * \ \ Merge branch 'rcu/urgent' of ↵Ingo Molnar2011-07-20
| | |\ \ \ | | | | | | | | | | | | | | | | | | git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-2.6-rcu into core/urgent
| | | * | | signal: align __lock_task_sighand() irq disabling and RCUPaul E. McKenney2011-07-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The __lock_task_sighand() function calls rcu_read_lock() with interrupts and preemption enabled, but later calls rcu_read_unlock() with interrupts disabled. It is therefore possible that this RCU read-side critical section will be preempted and later RCU priority boosted, which means that rcu_read_unlock() will call rt_mutex_unlock() in order to deboost itself, but with interrupts disabled. This results in lockdep splats, so this commit nests the RCU read-side critical section within the interrupt-disabled region of code. This prevents the RCU read-side critical section from being preempted, and thus prevents the attempt to deboost with interrupts disabled. It is quite possible that a better long-term fix is to make rt_mutex_unlock() disable irqs when acquiring the rt_mutex structure's ->wait_lock. Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
| | | * | | softirq,rcu: Inform RCU of irq_exit() activityPeter Zijlstra2011-07-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The rcu_read_unlock_special() function relies on in_irq() to exclude scheduler activity from interrupt level. This fails because exit_irq() can invoke the scheduler after clearing the preempt_count() bits that in_irq() uses to determine that it is at interrupt level. This situation can result in failures as follows: $task IRQ SoftIRQ rcu_read_lock() /* do stuff */ <preempt> |= UNLOCK_BLOCKED rcu_read_unlock() --t->rcu_read_lock_nesting irq_enter(); /* do stuff, don't use RCU */ irq_exit(); sub_preempt_count(IRQ_EXIT_OFFSET); invoke_softirq() ttwu(); spin_lock_irq(&pi->lock) rcu_read_lock(); /* do stuff */ rcu_read_unlock(); rcu_read_unlock_special() rcu_report_exp_rnp() ttwu() spin_lock_irq(&pi->lock) /* deadlock */ rcu_read_unlock_special(t); Ed can simply trigger this 'easy' because invoke_softirq() immediately does a ttwu() of ksoftirqd/# instead of doing the in-place softirq stuff first, but even without that the above happens. Cure this by also excluding softirqs from the rcu_read_unlock_special() handler and ensuring the force_irqthreads ksoftirqd/# wakeup is done from full softirq context. [ Alternatively, delaying the ->rcu_read_lock_nesting decrement until after the special handling would make the thing more robust in the face of interrupts as well. And there is a separate patch for that. ] Cc: Thomas Gleixner <tglx@linutronix.de> Reported-and-tested-by: Ed Tomlinson <edt@aei.ca> Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
| | | * | | sched: Add irq_{enter,exit}() to scheduler_ipi()Peter Zijlstra2011-07-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Ensure scheduler_ipi() calls irq_{enter,exit} when it does some actual work. Traditionally we never did any actual work from the resched IPI and all magic happened in the return from interrupt path. Now that we do do some work, we need to ensure irq_{enter,exit} are called so that we don't confuse things. This affects things like timekeeping, NO_HZ and RCU, basically everything with a hook in irq_enter/exit. Explicit examples of things going wrong are: sched_clock_cpu() -- has a callback when leaving NO_HZ state to take a new reading from GTOD and TSC. Without this callback, time is stuck in the past. RCU -- needs in_irq() to work in order to avoid some nasty deadlocks Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
| | | * | | rcu: protect __rcu_read_unlock() against scheduler-using irq handlersPaul E. McKenney2011-07-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The addition of RCU read-side critical sections within runqueue and priority-inheritance lock critical sections introduced some deadlock cycles, for example, involving interrupts from __rcu_read_unlock() where the interrupt handlers call wake_up(). This situation can cause the instance of __rcu_read_unlock() invoked from interrupt to do some of the processing that would otherwise have been carried out by the task-level instance of __rcu_read_unlock(). When the interrupt-level instance of __rcu_read_unlock() is called with a scheduler lock held from interrupt-entry/exit situations where in_irq() returns false, deadlock can result. This commit resolves these deadlocks by using negative values of the per-task ->rcu_read_lock_nesting counter to indicate that an instance of __rcu_read_unlock() is in flight, which in turn prevents instances from interrupt handlers from doing any special processing. This patch is inspired by Steven Rostedt's earlier patch that similarly made __rcu_read_unlock() guard against interrupt-mediated recursion (see https://lkml.org/lkml/2011/7/15/326), but this commit refines Steven's approach to avoid the need for preemption disabling on the __rcu_read_unlock() fastpath and to also avoid the need for manipulating a separate per-CPU variable. This patch avoids need for preempt_disable() by instead using negative values of the per-task ->rcu_read_lock_nesting counter. Note that nested rcu_read_lock()/rcu_read_unlock() pairs are still permitted, but they will never see ->rcu_read_lock_nesting go to zero, and will therefore never invoke rcu_read_unlock_special(), thus preventing them from seeing the RCU_READ_UNLOCK_BLOCKED bit should it be set in ->rcu_read_unlock_special. This patch also adds a check for ->rcu_read_unlock_special being negative in rcu_check_callbacks(), thus preventing the RCU_READ_UNLOCK_NEED_QS bit from being set should a scheduling-clock interrupt occur while __rcu_read_unlock() is exiting from an outermost RCU read-side critical section. Of course, __rcu_read_unlock() can be preempted during the time that ->rcu_read_lock_nesting is negative. This could result in the setting of the RCU_READ_UNLOCK_BLOCKED bit after __rcu_read_unlock() checks it, and would also result it this task being queued on the corresponding rcu_node structure's blkd_tasks list. Therefore, some later RCU read-side critical section would enter rcu_read_unlock_special() to clean up -- which could result in deadlock if that critical section happened to be in the scheduler where the runqueue or priority-inheritance locks were held. This situation is dealt with by making rcu_preempt_note_context_switch() check for negative ->rcu_read_lock_nesting, thus refraining from queuing the task (and from setting RCU_READ_UNLOCK_BLOCKED) if we are already exiting from the outermost RCU read-side critical section (in other words, we really are no longer actually in that RCU read-side critical section). In addition, rcu_preempt_note_context_switch() invokes rcu_read_unlock_special() to carry out the cleanup in this case, which clears out the ->rcu_read_unlock_special bits and dequeues the task (if necessary), in turn avoiding needless delay of the current RCU grace period and needless RCU priority boosting. It is still illegal to call rcu_read_unlock() while holding a scheduler lock if the prior RCU read-side critical section has ever had either preemption or irqs enabled. However, the common use case is legal, namely where then entire RCU read-side critical section executes with irqs disabled, for example, when the scheduler lock is held across the entire lifetime of the RCU read-side critical section. Signed-off-by: Paul E. McKenney <paul.mckenney@linaro.org> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
| | | * | | rcu: Streamline code produced by __rcu_read_unlock()Paul E. McKenney2011-07-20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Given some common flag combinations, particularly -Os, gcc will inline rcu_read_unlock_special() despite its being in an unlikely() clause. Use noinline to prohibit this misoptimization. In addition, move the second barrier() in __rcu_read_unlock() so that it is not on the common-case code path. This will allow the compiler to generate better code for the common-case path through __rcu_read_unlock(). Suggested-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Acked-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
generated by cgit v1.2.2 (git 2.25.0) at 2024-11-22 07:21:35 -0500