sparc64: Niagara-4 bzero/memset, plus use MRU stores in page copy.

This adds optimized memset/bzero/page-clear routines for Niagara-4.

We basically can do what powerpc has been able to do for a decade (via
the "dcbz" instruction), which is use cache line clearing stores for
bzero and memsets with a 'c' argument of zero.

As long as we make the cache initializing store to each 32-byte
subblock of the L2 cache line, it works.

As with other Niagara-4 optimized routines, the key is to make sure to
avoid any usage of the %asi register, as reads and writes to it cost
at least 50 cycles.

For the user clear cases, we don't use these new routines, we use the
Niagara-1 variants instead.  Those have to use %asi in an unavoidable
way.

A Niagara-4 8K page clear costs just under 600 cycles.

Add definitions of the MRU variants of the cache initializing store
ASIs.  By default, cache initializing stores install the line as Least
Recently Used.  If we know we're going to use the data immediately
(which is true for page copies and clears) we can use the Most
Recently Used variant, to decrease the likelyhood of the lines being
evicted before they get used.

Signed-off-by: David S. Miller <davem@davemloft.net>

1 files changed, 13 insertions, 2 deletions

diff --git a/arch/sparc/lib/NG4patch.S b/arch/sparc/lib/NG4patch.S
index c21c34c61dda..a114cbcf2a48 100644
--- a/arch/sparc/lib/NG4patch.S
+++ b/arch/sparc/lib/NG4patch.S
@@ -32,12 +32,23 @@ niagara4_patch_copyops:
          nop
         .size   niagara4_patch_copyops,.-niagara4_patch_copyops
 
+        .globl  niagara4_patch_bzero
+        .type   niagara4_patch_bzero,#function
+niagara4_patch_bzero:
+        NG_DO_PATCH(memset, NG4memset)
+        NG_DO_PATCH(__bzero, NG4bzero)
+        NG_DO_PATCH(__clear_user, NGclear_user)
+        NG_DO_PATCH(tsb_init, NGtsb_init)
+        retl
+         nop
+        .size   niagara4_patch_bzero,.-niagara4_patch_bzero
+
         .globl  niagara4_patch_pageops
         .type   niagara4_patch_pageops,#function
 niagara4_patch_pageops:
         NG_DO_PATCH(copy_user_page, NG4copy_user_page)
-        NG_DO_PATCH(_clear_page, NGclear_page)
-        NG_DO_PATCH(clear_user_page, NGclear_user_page)
+        NG_DO_PATCH(_clear_page, NG4clear_page)
+        NG_DO_PATCH(clear_user_page, NG4clear_user_page)
         retl
          nop
         .size   niagara4_patch_pageops,.-niagara4_patch_pageops