Fix floating conversion buffer overrun when host/target format matches
Running the testsuite with a gdb configured with --enable-libmcheck
reveals a problem:
(gdb) ptype 3 * 2.0
type = <12-byte float>
memory clobbered past end of allocated block
ERROR: Process no longer exists
UNRESOLVED: gdb.ada/ptype_arith_binop.exp: ptype 3 * 2.0
(gdb) PASS: gdb.dlang/expression.exp: ptype 0x1.FFFFFFFFFFFFFp1023
ptype 0x1p-52L
type = real
memory clobbered past end of allocated block
ERROR: Process no longer exists
UNRESOLVED: gdb.dlang/expression.exp: ptype 0x1p-52L
Even though this shows up with Ada and D, it's easy to reproduce in C
too. We just need to print a long double, when the current arch is
32-bit, which is the default when gdb starts up:
$ ./gdb -q -ex "ptype 1.0L"
type = long double
memory clobbered past end of allocated block
Aborted (core dumped)
Valgrind shows:
==22159== Invalid write of size 8
==22159== at 0x8464A9: floatformat_from_doublest (doublest.c:756)
==22159== by 0x846822: store_typed_floating (doublest.c:867)
==22159== by 0x6A7959: value_from_double (value.c:3662)
==22159== by 0x6A9F2D: evaluate_subexp_standard (eval.c:745)
==22159== by 0x7F31AF: evaluate_subexp_c (c-lang.c:716)
==22159== by 0x6A8986: evaluate_subexp (eval.c:79)
==22159== by 0x6A8BA3: evaluate_type (eval.c:174)
==22159== by 0x817CCF: whatis_exp (typeprint.c:456)
==22159== by 0x817EAA: ptype_command (typeprint.c:508)
==22159== by 0x5F267B: do_cfunc (cli-decode.c:105)
==22159== by 0x5F5618: cmd_func (cli-decode.c:1885)
==22159== by 0x83622A: execute_command (top.c:475)
==22159== Address 0x8c6cb28 is 8 bytes inside a block of size 12 alloc'd
==22159== at 0x4C2AA98: calloc (vg_replace_malloc.c:711)
==22159== by 0x87384A: xcalloc (common-utils.c:83)
==22159== by 0x873889: xzalloc (common-utils.c:93)
==22159== by 0x6A34CB: allocate_value_contents (value.c:1036)
==22159== by 0x6A3501: allocate_value (value.c:1047)
==22159== by 0x6A790A: value_from_double (value.c:3656)
==22159== by 0x6A9F2D: evaluate_subexp_standard (eval.c:745)
==22159== by 0x7F31AF: evaluate_subexp_c (c-lang.c:716)
==22159== by 0x6A8986: evaluate_subexp (eval.c:79)
==22159== by 0x6A8BA3: evaluate_type (eval.c:174)
==22159== by 0x817CCF: whatis_exp (typeprint.c:456)
==22159== by 0x817EAA: ptype_command (typeprint.c:508)
==22159==
type = long double
(gdb)
Even if the target and host floating-point formats match, the length
of the types might still be different. On x86, long double is the
80-bit extended precision type on both 32-bit and 64-bit ABIs, but by
default it is stored as 12 bytes on 32-bit, and 16 bytes on 64-bit,
for alignment reasons. Several places in doublest.c already consider
this, but floatformat_to_doublest and floatformat_from_doublest miss
it. E.g., convert_typed_floating and store_typed_floating,
Tested on x86-64 Fedora 23 with --enable-libmcheck, where it fixes the
crashed above.
gdb/ChangeLog:
2016-03-09 Pedro Alves <palves@redhat.com>
* doublest.c: Extend comments.
(floatformat_to_doublest, floatformat_from_doublest): Copy the
floatformat's total size, not the host type's size.