staging/rdma/hfi1: correctly check for post-interrupt packets

staging/rdma/hfi1: correctly check for post-interrupt packets

At the end of the packet processing interrupt and thread handler,
the RcvAvail interrupt is finally cleared down.  There is a window
between the last packet check (via DMA to memory) and interrupt
clear-down.  The code to recheck for a packet once the RcvAVail
interrupt is enabled must ultimately use a CSR read of RcvHdrTail
rather than depend on DMA'ed memory.

This change adds a CSR read of RcvHdrTail if the memory check does
not show a packet preset.  The memory check is retained as a quick
test before doing the more expensive, but always correct, CSR read.

In the ASIC, the CSR read used to force the RcvAvail clear-down write
to complete may bypass queued DMA writes to memory.  The only correct
way to decide if a packet has arrived without an interrupt to push DMA
to memory ahead of itself is to read the tail directly after RcvAvail
has been cleared down.  It is not sufficient to just read the tail and
skip pushing the clear-down.  Both must be done.  The tail read will not
push clear-down write due to it being in a different area of the chip.

At this point, it is OK to have packet data still being DMA'ed to
memory.  This is the end of packet processing for previous packets.
If the driver detects a new packet has arrived before interrputs were
re-enabled, it will force a new interrupt and the interrupt will push
the packet DMAs to memory, where the driver will then react to the
interrupt and do normal packet processing.

Reviewed-by: Mike Marciniszyn <mike.marciniszyn@intel.com>
Signed-off-by: Dean Luick <dean.luick@intel.com>
Signed-off-by: Jubin John <jubin.john@intel.com>
Signed-off-by: Doug Ledford <dledford@redhat.com>