Avoid undefined behavior in extract_integer

-fsanitize=undefined showed that extract_integer could left-shift a
negative value, which is undefined.  This patch fixes the problem by
doing all the work in an unsigned type.  This relies on
implementation-defined behavior, but I tend to think we are on safe
ground there.  (Also, if need be, violations of this could probably be
detected, either by configure or by a static_assert.)

gdb/ChangeLog
2018-10-03  Tom Tromey  <tom@tromey.com>

	* findvar.c (extract_integer): Do work in an unsigned type.

diff --git a/gdb/ChangeLog b/gdb/ChangeLog
index 36c4493..5787d44 100644 (file)
--- a/gdb/ChangeLog
+++ b/gdb/ChangeLog
@@ -1,5 +1,9 @@
 2018-10-03  Tom Tromey  <tom@tromey.com>
 
+       * findvar.c (extract_integer): Do work in an unsigned type.
+
+2018-10-03  Tom Tromey  <tom@tromey.com>
+
        * common/enum-flags.h (enum_flags::operator~): Add static assert.
        * symfile-add-flags.h (enum symfile_add_flag): Use unsigned as
        base type.

diff --git a/gdb/findvar.c b/gdb/findvar.c
index 9256833..be6c9d6 100644 (file)
--- a/gdb/findvar.c
+++ b/gdb/findvar.c
@@ -50,7 +50,7 @@ template<typename T, typename>
 T
 extract_integer (const gdb_byte *addr, int len, enum bfd_endian byte_order)
 {
-  T retval = 0;
+  typename std::make_unsigned<T>::type retval = 0;
   const unsigned char *p;
   const unsigned char *startaddr = addr;
   const unsigned char *endaddr = startaddr + len;