Free reference count on network when it goes away