clear_refs: introduce private struct for mm_walk