rtpjitterbuffer: Clear clock master before unreffing