layers: Only validate access masks if layout changes