IB/hfi1: Remove atomic SDMA_REQ_SEND_DONE bit operation

IB/hfi1: Remove atomic SDMA_REQ_SEND_DONE bit operation

The atomic SDMA_REQ_SEND_DONE bit is set by the
process-level code, and then the same process-level
code uses the bit to test that all packets have been
submitted incurring a costly atomic read.

Use a bool type with a READ_ONCE/WRITE_ONCE
pairing for this bit, and use the same condition that
is used to set the bit to test that all packets have
been submitted.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: Sebastian Sanchez <sebastian.sanchez@intel.com>
Signed-off-by: Dennis Dalessandro <dennis.dalessandro@intel.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>