Debugging DataPower TLS / SSL Errors

SSL / TLS is a core requirement for a secure infrastructure. Often, the first time that two systems need to communicate will result in the handshake failing. In older firmware versions there was little information logged as to the specific cause of the problem. This caused both sides of the connection to extensively walk their respective configurations, usually after a round of blaming each other. In the newer firmware, this has gotten much better by allowing the SSL Library to emit errors into the DataPower log.

This article goes into depth about common handshaking errors through examples. It should help to reduce the amount of time spent debugging so that the project can move on to higher value business testing.

SSL / TLS is the most popular method of securing a connection between two endpoints. It allows the client and server to verify identity and then encrypt the subsequent data exchange. In an IBM DataPower Gateway environment, this is the most common integration point and also the most common location for trouble.

In the older firmware, you would have to walk a checklist of the entire SSL Proxy Profile to guess at what had gone wrong. It was a tedious process. Luckily, since firmwares 5.0.0.18, 6.0.0.10, 6.0.1.6, 7.0.0.3, 7.1.0.0 and 7.2.0.0 DataPower writes SSL library errors to the log. These errors occur at a lower abstraction level and therefore provide better granularity on the specific cause of the failure. This improves the debugging process substantially but still does not allieviate the DataPower administrator of a solid knowledge of TLS.

In this article, we issue deficient TLS requests to a DataPower endpoint and show what is captured in the log. To reproduce these tests, simply grab a copy of openssl and use the ‘s_client’ option.

In this scenario, the client forgot that we’re an HTTPS endpoint and hit it with plain ol’ HTTP. In the old firmware, you would see an error about writing a descriptor(8). Today, you clearly see that the SSL library identified the client as HTTP. This error can be common when network health checkers are misconfigured and don’t use half-open to determine service availability. Effort should be made to fix the extraneous errors from occurring in the log.

In this scenario, the client sent a TLS 1.0 connection but DataPower is configured to only accept TLS1.1 or TLS1.2 connections. This is common when integrating with legacy environments that may not support the latest TLS protocols. DataPower writes that an unknown protocol was encountered.

In this scenario, the client send a TLS 1.2 connection but DataPower is configured to only accept TLS 1.1. This error message occurs when a higher TLS version arrives than what the server supports. For completeness, the same error occurred when the client sent TLS 1.1 but DataPower only accepted 1.0.

In this scenario, the client sends an SSLv3 connection but DataPower is configured to only accept TLS 1.0, 1.1 and 1.2. The same error message occurs if the client sends SSLv2. SSLv3 and SSLv2 are ancient versions and any occurrence of this error in the DataPower environment should be forwarded to the enterprise security team to ensure that its still suitable for the client to be so backleveled.

In this scenario, the client allowable cipher list and the DataPower allowable cipher list have no matching entry. This should be an extremely rare scenario and the resolution will require the two administrators to talk, as it’s difficult to know the list of ciphers that the client supports.

In the scenario, the client legitimately did not present a certificate to the DataPower endpoint. We can tell that this is the root cause error as the SSL library error confirmed a lacking certificate.

In this scenario, the client has presented an expired certificate to DataPower. This will happen more often than it should, as certificate management is a critical component of a secure infrastructure but not enough attention is given to it in the enterprise.

In this scenario, DataPower was unable to validate the certificate presented by the client. This configuration is stored in the Validation Credentials associated with the SSL Server Profile. Unfortunately, this error will occur for a broad range of verification errors, such as:

• The client certificate is not in the certificate list when the certificate validation mode is ‘Exact Certificate’.
• An intermediate signer chain certificate is missing when the certificate validation mode is  ‘Full Certificate Chain Checking (PKIX)’ or ‘Match Exact Certificate or Immediate Issuer’
• The client certificate is self-signed and is not in the certificate list

Debugging this error will require a copy of the client certificate, either to inspect the signer chain or to add the certificate to the validation credentials.

In this scenario, the client certificate was successfully validated by DataPower. The log message includes the DN and the validation credential used. This message will only be seen in the logs when the log target is at the ‘info’ level for the ‘crypto’ object.

In this scenario, it’s the client who was unable to verify the certificate presented by DataPower via the identification credentials. Certificate verification is a two way street and if the client doesn’t verify the server cert then they will close the connection and this error will occur. In previous firmware levels, you would just see the connection close. The rule of thumb was that the side of the connection that closed the connection will also be the one that writes the error in the local logs. Luckily in the new firmware, we get a hint that the client didn’t like our cert.

At the practical-datapower GitHub page, you can download the OSC_PD_SSLError domain and install it into your own device. This will save you the time of creating the keys and certs to test the various scenarios. It contains an XML Firewall configured with SSL with all of the certs and keys uploaded to the domain.

The client certificates and corresponding private keys can be downloaded as well.

In this article, we ran real world scenarios of the common reasons that TLS handshakes fail. The TLS errors in the latest firmware levels has greatly improved the error messages and should be the first place to look to determine why a connection failed. If you are stuck on a lower firmware level, then the SSL library errors won’t be seen and you will have to inspect the config manually.