x86/mm/cpa: Move flush_tlb_all()

There is an atom errata, where we do a local TLB invalidate right
before we return and then do a global TLB invalidate.

Move the global invalidate up a little bit and avoid the local
invalidate entirely.

This does put the global invalidate under pgd_lock, but that shouldn't
matter.

Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Dave Hansen <dave.hansen@intel.com>
Cc: Bin Yang <bin.yang@intel.com>
Cc: Mark Gross <mark.gross@intel.com>
Link: https://lkml.kernel.org/r/20180919085947.882287392@infradead.org

1 files changed, 17 insertions, 27 deletions

diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index a22f6b71a308..b6a4c638f086 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -999,14 +999,24 @@ __split_large_page(struct cpa_data *cpa, pte_t *kpte, unsigned long address,
         __set_pmd_pte(kpte, address, mk_pte(base, __pgprot(_KERNPG_TABLE)));
 
         /*
-         * Intel Atom errata AAH41 workaround.
+         * Do a global flush tlb after splitting the large page
+         * and before we do the actual change page attribute in the PTE.
          *
-         * The real fix should be in hw or in a microcode update, but
-         * we also probabilistically try to reduce the window of having
-         * a large TLB mixed with 4K TLBs while instruction fetches are
-         * going on.
+         * Without this, we violate the TLB application note, that says:
+         * "The TLBs may contain both ordinary and large-page
+         *  translations for a 4-KByte range of linear addresses. This
+         *  may occur if software modifies the paging structures so that
+         *  the page size used for the address range changes. If the two
+         *  translations differ with respect to page frame or attributes
+         *  (e.g., permissions), processor behavior is undefined and may
+         *  be implementation-specific."
+         *
+         * We do this global tlb flush inside the cpa_lock, so that we
+         * don't allow any other cpu, with stale tlb entries change the
+         * page attribute in parallel, that also falls into the
+         * just split large page entry.
          */
-        __flush_tlb_all();
+        flush_tlb_all();
         spin_unlock(&pgd_lock);
 
         return 0;
@@ -1531,28 +1541,8 @@ repeat:
          * We have to split the large page:
          */
         err = split_large_page(cpa, kpte, address);
-        if (!err) {
-                /*
-                 * Do a global flush tlb after splitting the large page
-                 * and before we do the actual change page attribute in the PTE.
-                 *
-                 * With out this, we violate the TLB application note, that says
-                 * "The TLBs may contain both ordinary and large-page
-                 *  translations for a 4-KByte range of linear addresses. This
-                 *  may occur if software modifies the paging structures so that
-                 *  the page size used for the address range changes. If the two
-                 *  translations differ with respect to page frame or attributes
-                 *  (e.g., permissions), processor behavior is undefined and may
-                 *  be implementation-specific."
-                 *
-                 * We do this global tlb flush inside the cpa_lock, so that we
-                 * don't allow any other cpu, with stale tlb entries change the
-                 * page attribute in parallel, that also falls into the
-                 * just split large page entry.
-                 */
-                flush_tlb_all();
+        if (!err)
                 goto repeat;
-        }
 
         return err;
 }