Practical DataPower: A Shared HTTPS Authorization Proxy

Improving endpoint security is a great justification for introducing a DataPower environment to the enterprise. Currently, HTTPS is the standard transport for communication between two endpoints. Therefore, a vital service that DataPower hosts is an endpoint that performs security services on client connections and then forwards the request to the backend for further processing. Pretty much every DataPower environment has a form of this service.

The problem is that this is usually the first service that a new team implements, is the most visible and is very difficult to adjust once put into production. A new team may not have the experience to understand the pain lurking down the road of their early decision. To help design this type of service, this article implements an HTTPS security service that is flexible, dynamic and secure. It validates client certificates for the endpoint they request while not falling into the trap of installing client certificates directly on the appliance. Not installing client certificates provides a substantial maintenance cost savings, as these certificates do not need to be renewed and don’t cause outages when they aren’t. The entire configuration is available for download on GitHub.

If your current shared proxy is the source of outages, project delays or has become a certificate maintenance nightmare, then read on.

The most common reason for bringing in an IBM DataPower Gateway environment is for security purposes. The device is top-notch in protocol support and the diversity of configuration options can support most requirements. In the security pattern, DataPower authenticates and authorizes an inbound connection and routes the validated connection to downstream services for further processing. The most common implementation of this pattern is as a shared HTTPS Proxy. Clients provide identification with an X509 certificate and requests access to a resource via the URL. It is then up to DataPower to determine if this client is allowed access. DataPower becomes the enterprise gatekeeper for endpoint security.

It is common for every DataPower customer to implement this system on their own. Worse, being a core platform requirement, it is the first service ever developed and hosted. It’s the first, it’s the most important, and it will live in the environment forever. This is a difficult task to give a new team and have them get it right the first time.

Part of this Practical DataPower series, this article implements an end-to-end HTTPS Authorization Proxy. This proxy is a shared service, used by all clients in the enterprise that want backend service access. It is a single authorization endpoint. We’ll discuss these benefits in the next section.

When DataPower becomes the sole authentication and authorization point for service access, many messes are avoided:

1. Some of the mess of enterprise certificate management is avoided. What cert is installed where? When does it expire? Who is using it?  Where are all the copies? Did we miss one? All of these questions need answers and they cost time and money every year. Expecting every platform to keep track of this information is naïve.
   • An exception is if your enterprise allows self-signed client certificates, but many enterprises don’t allow them.
   • Enterprises are starting to realize that certificates have an ongoing maintenance cost beyond development. The decisions of the past in placing all certificates directly into client and server validation keystores is taking a toll as they being to expire and cause outages.
     •  It’s not a problem with certificate technology. We just need to be smarter about how they are used.
2. The mess of enterprise service access control management. What clients can access what services? Where are the access list stored? If every technology is responsible for maintaining an independent list, then there is a risk of it not being completely secured and a critical endpoint is left open for attacks.
3. The mess of auditing the enterprise security policy. Your service whitelists, supported ciphers and allowed certificates are all in one spot.
4. The mess of ensuring all your technology subject matter experts understand security best practices. New exploits are being released faster than ever. When the next heartbleed attack is released, how do you ensure that your technologies are patched and safe?
   • It’s always easier to work with a single team than it is to try to hold parties accountable throughout the enterprise.

Being the single point in the enterprise for a security decision is not one that can be made lightly. If the process surrounding change is too slow, then every project will be impacted and projects will start looking for loopholes. Therefore, we need to ensure that this solution is stable, quick to update and reliable. Adding new services can not impact existing services and turnaround to modify endpoints cannot not take days, but minutes.

The HTTPS Authorization proxy service is built to meet the following requirements:

• The service authenticates all HTTPS service consumers.
• Clients must present an SSL certificate for identification.
  • Those that do not will be rejected.
• Clients are authorized by the following configuration tuple:  {certificate, [certificate issuer], [environment], URL}.
  • [issuer] and [environment] are optional, based on the specific enterprise requirement.
  • certificate and url are always required.
• A service’s whitelist should allow multiple certificates signed by different certificate authorities.
  • Endpoints exposed to vendors will need this as they always use different certificate authorities for their certs.
• The whitelist configuration should be portable between environments.
  • For example, the same physical file should be used in DEV as in QA as in pre-prod as in PROD.
• Clients can be authenticated by verifying certificate’s distinguished name (DN) or the certificate’s common name (CN).
  • DN provides more security at the cost of being susceptible to outages when the client certificate is renewed and some of the attributes change (usually by accident) (i.e.: city).
  •  CN validation ignores the country (CA), state (ST), locality/city (L), organization (O) and organization unit (OU) fields of the certificate.
• The endpoint should only accept TLS 1.1 or TLS 1.2 connections.
• The request payload can be either JSON or XML.
• The service should not inspect the payload of the incoming message to make routing or authorization decisions.
• Authorized requests are forwarded to a service specific URL for subsequent processing.
  • Unauthorized requests are rejected.
    • In non-production environments, a detailed error message should be logged and returned to the client.
      • This allows projects to self-heal bad requests without involving DataPower resources to check logs.
    • In production environments, a detailed error message should be logged but a generic error message should be returned to the client.
      • In production, it is important to contain information leaking. We shouldn’t tell a potential hacker what is wrong with their request message.
      •  An assigned resource and process should be available to check the logs and debug.
• When an error occurs, JSON clients should receive a JSON error message and XML clients should receive an XML error message.
• DataPower will define a platform standard URI format for all request messages.
  • Messages that do not meet this URI format will be rejected.
• Adding and Removing authorized certificates and servers should not require DataPower object configuration changes.
  • Changes should be contained to whitelist file updates.
• DataPower will provide two potential matches of an incoming URI against the whitelist
• The first match is for the entire URI. This is called the specific match. For example: /dp/EchoService/echo.
  • If a configuration exists for this match, then the next match’s result is ignored.
  • The second match is for the first two components of the URI. This is called the generic match. For example: an entry in the whitelist for /dp/EchoService that matches /dp/EchoService/reverse when the specific match does not exist.
    • Allowing a generic match helps to reduce the size of the whitelist when a group of URIs share the same allowable list of certificates. If the generic match didn’t exist, then a service with 100 endpoints would require 100 copy-paste entries becoming immediately unmaintainable.
• The whitelist for all services should not be contained in a single shared file.
  • A single location makes it too easy to typo an update and impact unrelated services during a change.
  • At the same time, requiring unique files becomes unmanageable, so a delicate balance needs to be achieved.

All of the project artifacts are available on GitHub as part of the ‘Practical DataPower’ repository. This repository hosts all the XSLT and DataPower configuration along with a domain export that can be easily imported.

The domain export can be downloaded from GitHub.

The OSC_PD_SSL_Service domain can be imported from the Control Panel, via the Import Configuration wizard. By default it listens on port 9193 of all interfaces.

The service is an XML Firewall but this does not restrict the incoming request to XML. A multi-protocol gateway could have been used, but unlike the stub service, this service doesn’t benefit from being provided over other transports. It is only relevant to HTTPS client authentication.

Requests to DataPower require the following URI format:

/dp/[service-config-id]/[0..N additional segments]/

• /dp/: This easily identifies that the URL is referring to a DataPower endpoint. This becomes helpful in mutli-technology debugging sessions when it is difficult to tell where clients are sending transactions and are sometimes given incorrect endpoints. If someone gives you a URL saying they having a connectivity error to DataPower and it doesn’t start with ‘/dp/’ you know it’s wrong.
• /[service-config-id]/ : This segment decides the configuration.xml file to load that holds the whitelist and routing information used to authorize the request. DataPower will load the file ‘local:///[service-config-id]/config.xml‘. This allows separation between unrelated services, but services that share the config-id can be grouped together.
  • service-config-id’s can be anything. They could be numeric or they could be random.
    • I suggest using either the domain that the request is forwarded to, or the name of the project the endpoint relates to. Hard to remember URIs make typos easy and the obfuscation justification is thin.
•  /[0..N additional segments]/ : These can be used to provide finer grain security over the incoming request, and to differentiate request types when forwarded on for subsequent processing.
  • For example: /dp/EchoService/echo or /dp/EchoService/echo/backwards.

The firewall will first attempt to load the configuration for the specific URI. For example ‘/dp/EchoService/echo/backwards’.  If a configuration can’t be found, then it will try the generic URI. The generic URI is the combination of the first two components. For example: ‘/dp/EchoService’. If the configruation still can’t be loaded, then an error is returned.

The following is the format for the XML whitelist. You can also view the sample config.xml included in the domain as a reference.

• <services>: The container element for the entire XML document.
  • <service>: The definition of a service configuration.
    • @url : The URI for the service configuration.
    • @env : The optional DataPower environment for the configuration (DEV/QA/PROD etc).
    • <allow>: A 0..N list of allowed issuer/certificate pairs
      • @issuer: The optional certificate authority that signed the client certificate.
      • <dn>: The complete distinguished name of the client certificate that is allowed access.
        •   Example: ‘<dn>/C=CA/ST=Ontario/L=Toronto/O=Orange Specs Incorporated/OU=Consulting/CN=client-1.ssltest.orangespecs.com</dn>’
          • The slash-delimited format of this string is provided by DataPower in an AAA.
      • <cn>: The common name of the client certificate allowed access to the service.
        • Example: <cn>client-1.ssltest.orangespecs.com</cn>’
    • <endpoint>: The container element for the backend hostname and port that an authorized request will be forwarded to. The intention is for this to be a port hosted on the DataPower device.
      • <hostname> : The hostname of the backend service.
      • <port>: The port of the backend service.

An XSD for the Whitelist XML format is also supplied.

This section describes the non-default configuration options needed on the XML Firewall object.

Firewall Type: Dynamic Backend

The backend service is dynamically set by the routing.xsl transformation based on the endpoint hostname and port from the config.xml.

Request Type: Non-XML

DataPower will treat the request message as binary, meaning that the appliance won’t attempt to validate or touch it in any way.

Response Type: Non-XML

DataPower will treat the response message as binary, meaning that the appliance won’t attempt to validate or touch it in any way.

Process messages whose body is empty: ON

This will allow HTTP GETs to flow through the proxy.

SSL Type: Server Profile

This uses the preferred ‘Server Profile’ object which superseded the SSL Proxy Profile object in firmware 7. If there was a requirement to host multiple SSL configurations from the same port, then we could have used an Server SNI Profile. The tradeoff would be that all clients would have to support SNI, which is not true in a large enterprise.

SSL Server Profile: SSL-Service-Server-Profile (see below)

This section describes the SSL Server Profile configuration.

Protocols:

• Enable SSL version 3 – OFF
• Enable TLS version 1.0 – OFF
• Enable TLS version 1.1 – ON
• Enable TLS version 1.2 – ON

The service should only allow TLS 1.1 & 1.2 handshakes, so the other options are disabled.

Ciphers: Left as defaults. This is where in the future, if a cipher is found to be vulnerable, you can remove it from the list. Direction over acceptable ciphers should be obtained from your enterprise security team. If not, the defaults are ok.

Identification Credentials: SSLService-Server-ID-Cred

This is the public certificate and private key pair the DataPower server sends to clients.

Request Client Authentication: ON

In the TLS standard, client authentication is not required. For our solution though, we need the client to identify themselves, so we turn this on.

Require Client Authentication: ON

From the above option, the client is merely requested to represent themselves. They can still choose to not present a certificate. With this option enabled, if the client doesn’t present one, DataPower will close the connection.

Validate Client Certificate: ON

DataPower will validate the presented client certificate against the validation credentials object specified later in the config. If this was turned off, then DataPower would accept any certificate. If the presented certificate is not in this validation credential object, then DataPower will close the connection.

Send client authentication CA list: OFF

When enabled, this will send a list of the certificate distinguished names from the validation credentials. In theory, this would allow a consumer to browse the list, identify a common signer of a certificate they have and respond with it. In the real world, clients are statically configured to present a single certificate and presenting the list is needless chatter. We’ll disable it.

Validation Credentials: SSLService-ValCred (see below)

This section describes validation credential object configuration.

Certificate List: This is the list of SSL certificates that the endpoint will accept during TLS negotiation.

It is very important to understand the types of certificates that should installed here:

• Self-Signed Certificates
• Certificate signer certificates (example: Verisign Certificate Authority, or intermediate issuer certificates

A specific client cert signed by a certificate authority should NOT be installed into the environment. This practice is known an SSL Certificate Pinning and it comes with a large avoidable cost. The cost is in maintaining and renewing these client certificates when they expire at inconvenient dates in the future. Each certificate has its own renewal process and production installation timing. If you pin and the client certificate changes, they won’t be able to connect and the result is an outage.

We mitigate the risk of accepting certificates at the signer level through the use of a DataPower AAA object and verifying the CN/DN in the whitelist contained in the config.xml.

Certificate Validation Mode: Full Certificate Chain Checking (PKIX)

DataPower will verify that the client certificate is either matched exactly in the certificate list, or that a complete signer chain from the incoming cert all the way to the root certificate authority can be established. Given that this endpoint is our primary method for verifying client identity, we should take every measure to ensure that it is fully validated.

This setting implies that we have a solid understanding of the signers that the clients use. We need to include all the intermediates and root signer certificates. If we don’t, DataPower will reject the connection.

Check Dates: ON

DataPower will verify that the current date is between the certificates notBefore and notAfter dates. If you turn this off, then you are susceptible to accepting expired certificates in your environment. An implied requirement of checking dates is that your data power appliance needs to be configured to use a reliable NTP server.

This completes the firewall configuration and TLS settings. Next we’ll talk about the Processing Policy Rules.

This rule services all requests for the proxy. It is a ‘Client To Server’ Rule with 4 configured actions:

1. A match action that accepts everything.
2. An AAA Action that authenticates and authorizes a client connection.
   • Input: NULL
   • AAA Policy: ‘SSLService_AAA‘
   • Output: NULL
3. An XSLT Transform action that routes the request to the appropriate backend.
   • Input: NULL
   • Transform File: ‘routing.xsl‘
   • Output: NULL
4. A Result action that sends the request to the backend.
   • Input: INPUT
   • Output: OUTPUT

The AAA object is broken down in 6 parts: Extract Identity (EI), Authenticate (AU), Map Credentials (MC), Map Resource (MR), Authorize (AZ) and Post Processing (PP). For our solution, the most important are the EI and AZ phases.

(EI) Define how to extract a user’s identity from an incoming request: Subject DN of SSL certificate from connection peer

DataPower will extract the distinguished name from the client certificate and use it as the client identity for the next stage of processing.

(AU) Define how to authenticate the user: Pass identity token to authorization phase

We’ve already authenticated that the user presented a valid certificate allowed by the validation credentials. We don’t need any additional authentication of the information at this point, so we tell the AAA to just pass along the extracted identity (the certificate DN).

(MC) Define how to map credentials: None

This stage would allow us to map the contents of the certificate into a different format if needed. For Example if the client presented ‘/C=CA/ST=Ontario/L=Toronto/O=Orange Specs Incorporated/OU=Consulting/CN=client-1.ssltest.orangespecs.com’ we could extract the CN as the subsequent identity of the client. We could also decide to map a list of allowed certificates into a credential that represents the service, ie ‘EchoService’. This ‘EchoService’ token would then be referenced elsewhere in the AAA.

We don’t use this option, as it would introduce an intermediary into the relationship between certificate DN and service access in our whitelist config.xml. We want the config.xml to be authoritative over the specific certificates it allows. We don’t want to have to look up this additional mapping in multiple locations.

(ER) Define how to extract the resources: URL sent by client

Our whitelist is dependent on the client URL, so we check that box.

(MR) Define how to map resources: None.

We want the pristine URL from the client, so we don’t map it.

(AZ) Define how to authorize a request: Custom template.

• URL: local://aaa-authorize.xsl

DataPower will run the aaa-authorize.xsl to determine if the client is authorized to access the service they requested.

(PP) Post Processing –  All post processing is not needed and disabled.

This is a small XSLT template that returns the current environment that represents the  DataPower appliance. Depending on the actual topology, it could be determined by source IP address or even included as a header by the requesting application. In our case, it’s hardcoded to return the string ‘DEV’. If the device was meant to represent production, then the XSLT on that production device would return ‘PROD’.