x86: Fix instruction decoder to handle grouped AVX instructions

For reducing memory usage of attribute table, x86 instruction
decoder puts "Group" attribute only on "no-last-prefix"
attribute table (same as vex_p == 0 case).

Thus, the decoder should look no-last-prefix table first, and
then only if it is not a group, move on to "with-last-prefix"
table (vex_p != 0).

However, current implementation, inat_get_avx_attribute()
looks with-last-prefix directly. So, when decoding
a grouped AVX instruction, the decoder fails to find correct
group because there is no "Group" attribute on the table.
This ends up with the mis-decoding of instructions, as Ingo
reported in http://thread.gmane.org/gmane.linux.kernel/1214103

This patch fixes it to check no-last-prefix table first
even if that is an AVX instruction, and get an attribute from
"with last-prefix" table only if that is not a group.

Reported-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: yrl.pp-manager.tt@hitachi.com
Link: http://lkml.kernel.org/r/20111205120539.15475.91428.stgit@cloud
Signed-off-by: Ingo Molnar <mingo@elte.hu>

1 files changed, 3 insertions, 1 deletions

diff --git a/arch/x86/lib/insn.c b/arch/x86/lib/insn.c
index 374562ed670..5a1f9f3e3fb 100644
--- a/arch/x86/lib/insn.c
+++ b/arch/x86/lib/insn.c
@@ -202,7 +202,7 @@ void insn_get_opcode(struct insn *insn)
                 m = insn_vex_m_bits(insn);
                 p = insn_vex_p_bits(insn);
                 insn->attr = inat_get_avx_attribute(op, m, p);
-                if (!inat_accept_vex(insn->attr))
+                if (!inat_accept_vex(insn->attr) && !inat_is_group(insn->attr))
                         insn->attr = 0; /* This instruction is bad */
                 goto end;       /* VEX has only 1 byte for opcode */
         }
@@ -249,6 +249,8 @@ void insn_get_modrm(struct insn *insn)
                         pfx = insn_last_prefix(insn);
                         insn->attr = inat_get_group_attribute(mod, pfx,
                                                               insn->attr);
+                        if (insn_is_avx(insn) && !inat_accept_vex(insn->attr))
+                                insn->attr = 0; /* This is bad */
                 }
         }