Unref enum type classes when we're done with them