x86: Keep current stack in NMI breakpoints

We want to allow NMI handlers to have breakpoints to be able to
remove stop_machine from ftrace, kprobes and jump_labels. But if
an NMI interrupts a current breakpoint, and then it triggers a
breakpoint itself, it will switch to the breakpoint stack and
corrupt the data on it for the breakpoint processing that it
interrupted.

Instead, have the NMI check if it interrupted breakpoint processing
by checking if the stack that is currently used is a breakpoint
stack. If it is, then load a special IDT that changes the IST
for the debug exception to keep the same stack in kernel context.
When the NMI is done, it puts it back.

This way, if the NMI does trigger a breakpoint, it will keep
using the same stack and not stomp on the breakpoint data for
the breakpoint it interrupted.

Suggested-by: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

1 files changed, 15 insertions, 0 deletions

diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c
index e88f37b58ddd..de8d4b333f40 100644
--- a/arch/x86/kernel/nmi.c
+++ b/arch/x86/kernel/nmi.c
@@ -408,6 +408,18 @@ static notrace __kprobes void default_do_nmi(struct pt_regs *regs)
 dotraplinkage notrace __kprobes void
 do_nmi(struct pt_regs *regs, long error_code)
 {
+        int update_debug_stack = 0;
+
+        /*
+         * If we interrupted a breakpoint, it is possible that
+         * the nmi handler will have breakpoints too. We need to
+         * change the IDT such that breakpoints that happen here
+         * continue to use the NMI stack.
+         */
+        if (unlikely(is_debug_stack(regs->sp))) {
+                debug_stack_set_zero();
+                update_debug_stack = 1;
+        }
         nmi_enter();
 
         inc_irq_stat(__nmi_count);
@@ -416,6 +428,9 @@ do_nmi(struct pt_regs *regs, long error_code)
                 default_do_nmi(regs);
 
         nmi_exit();
+
+        if (unlikely(update_debug_stack))
+                debug_stack_reset();
 }
 
 void stop_nmi(void)