Update our internal state counters when adding/removing elements.