server/shadow: avoid sending pointer updates when not active, fix remdesk leak