Merge branch 'x86-bsp-hotplug-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

Pull x86 BSP hotplug changes from Ingo Molnar:
 "This tree enables CPU#0 (the boot processor) to be onlined/offlined on
  x86, just like any other CPU.  Enabled on Intel CPUs for now.

  Allowing this required the identification and fixing of latent CPU#0
  assumptions (such as CPU#0 initializations, etc.) in the x86
  architecture code, plus the identification of barriers to
  BSP-offlining, such as active PIC interrupts which can only be
  serviced on the BSP.

  It's behind a default-off option, and there's a debug option that
  allows the automatic testing of this feature.

  The motivation of this feature is to allow and prepare for true
  CPU-hotplug hardware support: recent changes to MCE support enable us
  to detect a deteriorating but not yet hard-failing L1/L2 cache on a
  CPU that could be soft-unplugged - or a failing L3 cache on a
  multi-socket system.

  Note that true hardware hot-plug is not yet fully enabled by this,
  because that requires a special platform wakeup sequence to be sent to
  the freshly powered up CPU#0.  Future patches for this are planned,
  once such a platform exists.  Chicken and egg"

* 'x86-bsp-hotplug-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip:
  x86, topology: Debug CPU0 hotplug
  x86/i387.c: Initialize thread xstate only on CPU0 only once
  x86, hotplug: Handle retrigger irq by the first available CPU
  x86, hotplug: The first online processor saves the MTRR state
  x86, hotplug: During CPU0 online, enable x2apic, set_numa_node.
  x86, hotplug: Wake up CPU0 via NMI instead of INIT, SIPI, SIPI
  x86-32, hotplug: Add start_cpu0() entry point to head_32.S
  x86-64, hotplug: Add start_cpu0() entry point to head_64.S
  kernel/cpu.c: Add comment for priority in cpu_hotplug_pm_callback
  x86, hotplug, suspend: Online CPU0 for suspend or hibernate
  x86, hotplug: Support functions for CPU0 online/offline
  x86, topology: Don't offline CPU0 if any PIC irq can not be migrated out of it
  x86, Kconfig: Add config switch for CPU0 hotplug
  doc: Add x86 CPU0 online/offline feature

15 files changed, 436 insertions, 41 deletions

diff --git a/Documentation/cpu-hotplug.txt b/Documentation/cpu-hotplug.txt
index 66ef8f35613d..9f401350f502 100644
--- a/Documentation/cpu-hotplug.txt
+++ b/Documentation/cpu-hotplug.txt
@@ -207,6 +207,30 @@ by making it not-removable.
 
 In such cases you will also notice that the online file is missing under cpu0.
 
+Q: Is CPU0 removable on X86?
+A: Yes. If kernel is compiled with CONFIG_BOOTPARAM_HOTPLUG_CPU0=y, CPU0 is
+removable by default. Otherwise, CPU0 is also removable by kernel option
+cpu0_hotplug.
+
+But some features depend on CPU0. Two known dependencies are:
+
+1. Resume from hibernate/suspend depends on CPU0. Hibernate/suspend will fail if
+CPU0 is offline and you need to online CPU0 before hibernate/suspend can
+continue.
+2. PIC interrupts also depend on CPU0. CPU0 can't be removed if a PIC interrupt
+is detected.
+
+It's said poweroff/reboot may depend on CPU0 on some machines although I haven't
+seen any poweroff/reboot failure so far after CPU0 is offline on a few tested
+machines.
+
+Please let me know if you know or see any other dependencies of CPU0.
+
+If the dependencies are under your control, you can turn on CPU0 hotplug feature
+either by CONFIG_BOOTPARAM_HOTPLUG_CPU0 or by kernel parameter cpu0_hotplug.
+
+--Fenghua Yu <fenghua.yu@intel.com>
+
 Q: How do i find out if a particular CPU is not removable?
 A: Depending on the implementation, some architectures may show this by the
 absence of the "online" file. This is done if it can be determined ahead of
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 03d1251a915e..5190f1706414 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -1984,6 +1984,20 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 
         nox2apic        [X86-64,APIC] Do not enable x2APIC mode.
 
+        cpu0_hotplug    [X86] Turn on CPU0 hotplug feature when
+                        CONFIG_BOOTPARAM_HOTPLUG_CPU0 is off.
+                        Some features depend on CPU0. Known dependencies are:
+                        1. Resume from suspend/hibernate depends on CPU0.
+                        Suspend/hibernate will fail if CPU0 is offline and you
+                        need to online CPU0 before suspend/hibernate.
+                        2. PIC interrupts also depend on CPU0. CPU0 can't be
+                        removed if a PIC interrupt is detected.
+                        It's said poweroff/reboot may depend on CPU0 on some
+                        machines although I haven't seen such issues so far
+                        after CPU0 is offline on a few tested machines.
+                        If the dependencies are under your control, you can
+                        turn on cpu0_hotplug.
+
         nptcg=          [IA-64] Override max number of concurrent global TLB
                         purges which is reported from either PAL_VM_SUMMARY or
                         SAL PALO.
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 6c304438b503..2d643255c40d 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1698,6 +1698,50 @@ config HOTPLUG_CPU
             automatically on SMP systems. )
           Say N if you want to disable CPU hotplug.
 
+config BOOTPARAM_HOTPLUG_CPU0
+        bool "Set default setting of cpu0_hotpluggable"
+        default n
+        depends on HOTPLUG_CPU && EXPERIMENTAL
+        ---help---
+          Set whether default state of cpu0_hotpluggable is on or off.
+
+          Say Y here to enable CPU0 hotplug by default. If this switch
+          is turned on, there is no need to give cpu0_hotplug kernel
+          parameter and the CPU0 hotplug feature is enabled by default.
+
+          Please note: there are two known CPU0 dependencies if you want
+          to enable the CPU0 hotplug feature either by this switch or by
+          cpu0_hotplug kernel parameter.
+
+          First, resume from hibernate or suspend always starts from CPU0.
+          So hibernate and suspend are prevented if CPU0 is offline.
+
+          Second dependency is PIC interrupts always go to CPU0. CPU0 can not
+          offline if any interrupt can not migrate out of CPU0. There may
+          be other CPU0 dependencies.
+
+          Please make sure the dependencies are under your control before
+          you enable this feature.
+
+          Say N if you don't want to enable CPU0 hotplug feature by default.
+          You still can enable the CPU0 hotplug feature at boot by kernel
+          parameter cpu0_hotplug.
+
+config DEBUG_HOTPLUG_CPU0
+        def_bool n
+        prompt "Debug CPU0 hotplug"
+        depends on HOTPLUG_CPU && EXPERIMENTAL
+        ---help---
+          Enabling this option offlines CPU0 (if CPU0 can be offlined) as
+          soon as possible and boots up userspace with CPU0 offlined. User
+          can online CPU0 back after boot time.
+
+          To debug CPU0 hotplug, you need to enable CPU0 offline/online
+          feature by either turning on CONFIG_BOOTPARAM_HOTPLUG_CPU0 during
+          compilation or giving cpu0_hotplug kernel parameter at boot.
+
+          If unsure, say N.
+
 config COMPAT_VDSO
         def_bool y
         prompt "Compat VDSO support"
diff --git a/arch/x86/include/asm/cpu.h b/arch/x86/include/asm/cpu.h
index 4564c8e28a33..5f9a1243190e 100644
--- a/arch/x86/include/asm/cpu.h
+++ b/arch/x86/include/asm/cpu.h
@@ -28,6 +28,10 @@ struct x86_cpu {
 #ifdef CONFIG_HOTPLUG_CPU
 extern int arch_register_cpu(int num);
 extern void arch_unregister_cpu(int);
+extern void __cpuinit start_cpu0(void);
+#ifdef CONFIG_DEBUG_HOTPLUG_CPU0
+extern int _debug_hotplug_cpu(int cpu, int action);
+#endif
 #endif
 
 DECLARE_PER_CPU(int, cpu_state);
diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
index 4f19a1526037..b073aaea747c 100644
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -166,6 +166,7 @@ void native_send_call_func_ipi(const struct cpumask *mask);
 void native_send_call_func_single_ipi(int cpu);
 void x86_idle_thread_init(unsigned int cpu, struct task_struct *idle);
 
+void smp_store_boot_cpu_info(void);
 void smp_store_cpu_info(int id);
 #define cpu_physical_id(cpu)    per_cpu(x86_cpu_to_apicid, cpu)
 
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index 1817fa911024..f78fc2b4deb0 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -2199,9 +2199,11 @@ static int ioapic_retrigger_irq(struct irq_data *data)
 {
         struct irq_cfg *cfg = data->chip_data;
         unsigned long flags;
+        int cpu;
 
         raw_spin_lock_irqsave(&vector_lock, flags);
-        apic->send_IPI_mask(cpumask_of(cpumask_first(cfg->domain)), cfg->vector);
+        cpu = cpumask_first_and(cfg->domain, cpu_online_mask);
+        apic->send_IPI_mask(cpumask_of(cpu), cfg->vector);
         raw_spin_unlock_irqrestore(&vector_lock, flags);