Warn in CustomMarshallerAnalyzer if the MarshaMode is not valid (#87198)

Warn in CustomMarshallerAnalyzer if the MarshaMode is not valid (#87198)

I ran into an issue when making some tests where I tried to use MarshalMode as a flags enum, which lead to the generated code not being correct, but no warning was showing up. This PR creates a warning if you try to do an operation on the MarshalMode argument or if a casted int value is not defined in the MarshalMode enum.

Fixes #87139