How to force a server to use a specific elliptic curve at the beginning of the handshake via openssl?

Question

Is there a possibility or command in openssl to force a server to use a specific elliptic curve that I am requesting with a certificate or key? Can I "ask" the process the curves it is able to use?

Answer 1

Your question is ambiguous and the answers are quite different.

Protocol. The supported-curves extension in the ClientHello message specifies the curves (standard/named, or 'explicit' GF(2^k) or GF(p)) the client is willing to have the server use. If it specifies more than one, the server can choose any of them; if it specifies one, that's the only one the server can use. If the server does not support the or any specified curve, the negotiation cannot select any EC suite, and unless other suites are supported by and acceptable to both ends the negotiation fails.

OpenSSL client to any server through 1.0.1 always sends this extension to allow all named curves (in rfc4492) and no explicit curves. Except RedHat (AFAIK still?) nobbles its OpenSSL builds to P-256, P-384 and P-521 only. 1.0.2 adds new API calls to control this extension, see https://www.openssl.org/docs/ssl/SSL_CTX_set1_curves_list.html . So either code and build your client program against 1.0.2* and not on RedHat, or get whoever maintains your client program to do so.

Any client to OpenSSL server for ECDHE. OpenSSL server through 1.0.1 can be configured in advance of handshake, either for all connections sharing an SSL_CTX or for each SSL connection separately, with ECDHE "temporary parameters" which is (are?) the curve. Only the configured curve is used for key-exchange. For ECDHE-ECDSA the publickey in the cert used (with its matching privatekey) to authenticate, can use a different curve; that key and curve was chosen before the cert was issued and "approved" by the CA and can't be changed now, whereas the "temporary" ECDHE choice is made by the server code or configuration. If either the "temp" curve, or the cert curve when the cert is ECC, is not in the list allowed by the client, then ECDHE or ECDHE-ECDSA can't be negotiated.

Instead of configuring in advance, the server can set a callback invoked by OpenSSL library to choose the ECDHE curve (or key) for a particular handshake. This should choose a curve acceptable to the client, unless that is unacceptable to the server. Again for ECDHE-ECDSA, the curve in the cert for authentication may be different, was already chosen before issue and can't be changed, and if not acceptable to the client can't be negotiated. Finally new in 1.0.2 there is an option for OpenSSL to automatically choose an ECDHE curve acceptable to the client while "matching" the strength of the authentication.

In server for static ECDH the curve (and key) used for key-exchange is the one in the cert, which as above was chosen before the cert was issued and cannot be changed. If this curve is not acceptable to the client, ECDH can't be negotiated.

Answer 2

Is there a possibility or command in openssl to force a server to use a specific elliptic curve

In general, I'm not sure how to do this through a CONF file. The SSL_CONF_* API calls (shown below) seem to indicate it can be part of a configuration file. But I've never done it.

---

How to force a server to use a specific elliptic curve at the beginning of the handshake via openssl?

You can, but you have to modify your sources. Here's from one of the OpenSSL devs in a private email. But I'm not sure it works in practice because I use the patching method described below.

Under 0.9.x:
         int nid = OBJ_sn2nid(named_curve);
         ecdh = EC_KEY_new_by_curve_name(nid);
         SSL_CTX_set_tmp_ecdh(ctx,ecdh);
         EC_KEY_free(ecdh);

 Under 1.x:
         SSL_CONF_CTX_set_ssl_ctx(cctx, ctx);
         SSL_CONF_cmd(cctx, "-named_curve", "P-256");
         SSL_CONF_CTX_finish(cctx);

---

For 1.0.1 and below, you can modify the sources to t1_lib.c. In particular, find ssl_prepare_clienthello_tlsext and pref_list and change it to what you like. Here's what it looks like by default:

static int pref_list[] =
 {
   NID_sect571r1, /* sect571r1 (14) */
   NID_sect571k1, /* sect571k1 (13) */
   NID_secp521r1, /* secp521r1 (25) */
   NID_sect409k1, /* sect409k1 (11) */
   NID_sect409r1, /* sect409r1 (12) */
   NID_secp384r1, /* secp384r1 (24) */
   NID_sect283k1, /* sect283k1 (9) */
   NID_sect283r1, /* sect283r1 (10) */
   NID_secp256k1, /* secp256k1 (22) */
   NID_X9_62_prime256v1, /* secp256r1 (23) */
   NID_sect239k1, /* sect239k1 (8) */
   NID_sect233k1, /* sect233k1 (6) */
   NID_sect233r1, /* sect233r1 (7) */
   NID_secp224k1, /* secp224k1 (20) */
   NID_secp224r1, /* secp224r1 (21) */
 };

Also see Issue #3179: Feature Request: Set Preference List for EC Curves in Client.

---

For OpenSSL 1.0.2 and above, you can use SSL_CTX_set1_curves. pref_list is no longer present when grepping the sources.