Fix displaced-stepping RIP-relative VEX-encoded instructions (AVX) (PR gdb/22499)
authorPedro Alves <palves@redhat.com>
Mon, 4 Dec 2017 15:59:20 +0000 (15:59 +0000)
committerPedro Alves <palves@redhat.com>
Mon, 4 Dec 2017 15:59:20 +0000 (15:59 +0000)
commit50a1fdd59c1777672a9be0e81fe2301c2a115fce
treef6b8dbf15b61bcf1bc97953a8752a71dea00e91f
parent826c3f1edc2f9da4594aed86b57a7b447a46016d
Fix displaced-stepping RIP-relative VEX-encoded instructions (AVX) (PR gdb/22499)

PR gdb/22499 is about a latent bug exposed by the switch to "maint set
target-non-stop on" by default on x86-64 GNU/Linux, a while ago.  With
that on, GDB is also preferring to use displaced-stepping by default.

The testcase in the bug is failing because GDB ends up incorrectly
displaced-stepping over a RIP-relative VEX-encoded instruction, like
this:

 0x00000000004007f5 <+15>:    c5 fb 10 05 8b 01 00 00 vmovsd 0x18b(%rip),%xmm0        # 0x400988

While RIP-relative instructions need adjustment when relocated to the
scratch pad, GDB ends up just copying VEX-encoded instructions to the
scratch pad unmodified, with the end result that the inferior ends up
executing an instruction that fetches/writes memory from the wrong
address...

This patch teaches GDB about the VEX-encoding prefixes, fixing the
problem, and adds a testcase that fails without the GDB fix.

I think we may need a similar treatment for EVEX-encoded instructions,
but I didn't address that simply because I couldn't find any
EVEX-encoded RIP-relative instruction in the gas testsuite.  In any
case, this commit is forward progress as-is already.

gdb/ChangeLog:
2017-12-04  Pedro Alves  <palves@redhat.com>

PR gdb/22499
* amd64-tdep.c (amd64_insn::rex_offset): Rename to...
(amd64_insn::enc_prefix_offset): ... this, and tweak comment.
(vex2_prefix_p, vex3_prefix_p): New functions.
(amd64_get_insn_details): Adjust to rename.  Also skip VEX2 and
VEX3 prefixes.
(fixup_riprel): Set VEX3.!B.

gdb/testsuite/ChangeLog:
2017-12-04  Pedro Alves  <palves@redhat.com>

PR gdb/22499
* gdb.arch/amd64-disp-step-avx.S: New file.
* gdb.arch/amd64-disp-step-avx.exp: New file.
gdb/ChangeLog
gdb/amd64-tdep.c
gdb/testsuite/ChangeLog
gdb/testsuite/gdb.arch/amd64-disp-step-avx.S [new file with mode: 0644]
gdb/testsuite/gdb.arch/amd64-disp-step-avx.exp [new file with mode: 0644]