btrfs-progs: Fix undefined behavior in radix-tree.c.

btrfs-progs: Fix undefined behavior in radix-tree.c.

When running with UndefinedBehaviorSanitizer, the tests produce the following
error:

  radix-tree.c:836:30: runtime error: shift exponent 18446744073709551613
  is too large for 64-bit type 'unsigned long'

(That's a negative shift exponent represented as an unsigned long.)

Even though the value is discarded in those cases, it's still undefined
behavior; see the C99 standard, section 6.5.7, paragraph three: "If the
value of the right operand is negative [...] the behavior is undefined."

Signed-off-by: Adam Buchbinder <abuchbinder@google.com>
Reviewed-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>