gdb-remote: Fix checksum verification for messages with escape chars
authorPavel Labath <labath@google.com>
Wed, 28 Mar 2018 10:19:10 +0000 (10:19 +0000)
committerPavel Labath <labath@google.com>
Wed, 28 Mar 2018 10:19:10 +0000 (10:19 +0000)
commit5a841234904b5138d1ce0bc7ad2ee69ea0d25792
tree5901b58554e742b16fc923a1badca33c8a7bdd2f
parente1d58778475e89d1b2c3a9de174716a2236e3570
gdb-remote: Fix checksum verification for messages with escape chars

Summary:
We've had a mismatch in the checksum computation between the sender and
receiver. The sender computed the payload checksum using the wire
encoding of the packet, while the receiver did this after expanding
un-escaping and expanding run-length-encoded sequences. This resulted in
communication breakdown if packets using these feature were sent in the
ack mode.

Normally, this did not cause any issues since the only packet we send in
the ack-mode is the QStartNoAckMode packet, but I ran into this when
debugging the lldb-server tests which (for better or worse) don't use
this mode.

According to the gdb-remote documentation "The two-digit checksum is computed as
the modulo 256 sum of all characters between the leading ‘$’ and the
trailing ‘#’", it seems that our sender is doing the right thing here.
Therefore, I fix the receiver the match the sender behavior and add a
test.

With this bug fixed, we can see that lldb-server is sending a stop-reply
after receiving the "k" in the same way as debugserver does (but we
weren't detecting this because at that point the connection was dead
already). I fix that expectation as well.

Reviewers: clayborg, jasonmolenda

Subscribers: mgorny, lldb-commits

Differential Revision: https://reviews.llvm.org/D44922

llvm-svn: 328693
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/unittests/Process/gdb-remote/CMakeLists.txt
lldb/unittests/Process/gdb-remote/GDBRemoteCommunicationTest.cpp [new file with mode: 0644]
lldb/unittests/tools/lldb-server/tests/TestClient.cpp