x86: vdso: fix pvclock races with task migration

If we were migrated right after __getcpu, but before reading the
migration_count, we wouldn't notice that we read TSC of a different
VCPU, nor that KVM's bug made pvti invalid, as only migration_count
on source VCPU is increased.

Change vdso instead of updating migration_count on destination.

Cc: stable@vger.kernel.org
Signed-off-by: Radim Krčmář <rkrcmar@redhat.com>
Fixes: 0a4e6be9ca17 ("x86: kvm: Revert "remove sched notifier for cross-cpu migrations"")
Message-Id: <1428000263-11892-1-git-send-email-rkrcmar@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

1 files changed, 12 insertions, 8 deletions

diff --git a/arch/x86/vdso/vclock_gettime.c b/arch/x86/vdso/vclock_gettime.c
index 30933760ee5f..40d2473836c9 100644
--- a/arch/x86/vdso/vclock_gettime.c
+++ b/arch/x86/vdso/vclock_gettime.c
@@ -99,21 +99,25 @@ static notrace cycle_t vread_pvclock(int *mode)
                  * __getcpu() calls (Gleb).
                  */
 
-                pvti = get_pvti(cpu);
+                /* Make sure migrate_count will change if we leave the VCPU. */
+                do {
+                        pvti = get_pvti(cpu);
+                        migrate_count = pvti->migrate_count;
 
-                migrate_count = pvti->migrate_count;
+                        cpu1 = cpu;
+                        cpu = __getcpu() & VGETCPU_CPU_MASK;
+                } while (unlikely(cpu != cpu1));
 
                 version = __pvclock_read_cycles(&pvti->pvti, &ret, &flags);
 
                 /*
                  * Test we're still on the cpu as well as the version.
-                 * We could have been migrated just after the first
-                 * vgetcpu but before fetching the version, so we
-                 * wouldn't notice a version change.
+                 * - We must read TSC of pvti's VCPU.
+                 * - KVM doesn't follow the versioning protocol, so data could
+                 *   change before version if we left the VCPU.
                  */
-                cpu1 = __getcpu() & VGETCPU_CPU_MASK;
-        } while (unlikely(cpu != cpu1 ||
-                          (pvti->pvti.version & 1) ||
+                smp_rmb();
+        } while (unlikely((pvti->pvti.version & 1) ||
                           pvti->pvti.version != version ||
                           pvti->migrate_count != migrate_count));