tdm_server: erase unnecessary buffer ref