Do not try to use the encrypt-then-mac ciphers from OpenSSL 3.6.0

These ciphers claim to be CBC but since they are also include an HMAC
are more a mix of AEAD and CBC. Nevertheless, we do not support these
and also have no (good) reason to support them.

This patch defines the flag if the SSL library does not define the flag
to also work when the SSL library is upgraded after OpenVPN has been compiled.

Change-Id: Iafe3c94b952cd3fbecf6f3d05816e5859f425e7d
Signed-off-by: Arne Schwabe <[email protected]>
Acked-by: Frank Lichtenheld <[email protected]>
Gerrit URL: https://gerrit.openvpn.net/c/openvpn/+/1294
Message-Id: <[email protected]>
URL: https://www.mail-archive.com/[email protected]/msg33846.html
Signed-off-by: Gert Doering <[email protected]>

Author: schwabe

src/openvpn/crypto_openssl.c

@@ -789,7 +789,8 @@ cipher_kt_mode_cbc(const char *ciphername)
 #ifdef EVP_CIPH_FLAG_CTS
                    && !(EVP_CIPHER_flags(cipher) & EVP_CIPH_FLAG_CTS)
 #endif
-                   && !(EVP_CIPHER_flags(cipher) & EVP_CIPH_FLAG_AEAD_CIPHER));
+                   && !(EVP_CIPHER_flags(cipher) & EVP_CIPH_FLAG_AEAD_CIPHER)
+                   && !(EVP_CIPHER_flags(cipher) & EVP_CIPH_FLAG_ENC_THEN_MAC));
     EVP_CIPHER_free(cipher);
     return ret;
 }

src/openvpn/openssl_compat.h

@@ -211,4 +211,9 @@ SSL_get0_group_name(SSL *s)
 #endif
 #endif
 
+/* Introduced in OpenSSL 3.6.0 */
+#ifndef EVP_CIPH_FLAG_ENC_THEN_MAC
+#define EVP_CIPH_FLAG_ENC_THEN_MAC 0x10000000
+#endif
+
 #endif /* OPENSSL_COMPAT_H_ */