Fix bad null check in pal_dsa.c

The code calls DSA_new() and then, only after assigning the value from
DSA_new to a pointer, tries to check that the pointer was not null.
Assigning to a null pointer leads to undefined behaviour and a compiler
can assume that assignment-to-a-null-pointer wont happen. Subsequently,
the compiler can remove the entire check because the pointer can't be
null.

Fix that by:

1. Moving the check for whether the pointer is null before dereferncing
   the pointer.

2. Checking the return value of DSA_new() spearately after it has been
   called.

Commit migrated from https://github.com/dotnet/corefx/commit/e7fd1669eafafd16de85cd90b558d54abb3a3343

diff --git a/src/libraries/Native/Unix/System.Security.Cryptography.Native/pal_dsa.c b/src/libraries/Native/Unix/System.Security.Cryptography.Native/pal_dsa.c
index a9a6123..70b40a0 100644 (file)
--- a/src/libraries/Native/Unix/System.Security.Cryptography.Native/pal_dsa.c
+++ b/src/libraries/Native/Unix/System.Security.Cryptography.Native/pal_dsa.c
@@ -20,13 +20,18 @@ void CryptoNative_DsaDestroy(DSA* dsa)
 
 int32_t CryptoNative_DsaGenerateKey(DSA** dsa, int32_t bits)
 {
-    *dsa = DSA_new();
     if (!dsa)
     {
         assert(false);
         return 0;
     }
 
+    *dsa = DSA_new();
+    if (!(*dsa))
+    {
+        return 0;
+    }
+
     if (!DSA_generate_parameters_ex(*dsa, bits, NULL, 0, NULL, NULL, NULL) ||
         !DSA_generate_key(*dsa))
     {