Fix a regression from an (old) optimization to avoid dirtying the scene too often