Practical DataPower: A Lightweight XML / JSON Stub Service

The number of service dependencies in a project is growing. Some of these dependencies are involved in their own internal development cycle. If the project waits for the new development to complete, progress would be slow. A robust solution to this problem is the introduction of stubs that impersonate the service. Stubs return pre-canned responses to the requests they receive. They use the same transport and the same message format, but just happen to not do the backend work that the real service would. The project uses these fake services to continue their own development without being affected by the dependent project. Once the real service is complete, switching over is as simple as updating the referenced URL inside the application.

DataPower is a solid platform for hosting this type of service. We go into detail on how to create the stub and provide ideas on how the pattern can be expanded for future projects.

The sheer volume of APIs woven together into new applications is increasing. These dependent services might be hosted locally or provided by a vendor. Many times, these services are new and part of a development cycle running in parallel to the consuming project. This can lead to progress stalling at these integration points. Without an endpoint to respond to a request, the application code path will always take an error route. This can prevent subsequent code paths from being exercised.

Some projects workaround this issue inside the application code by removing the service invocation entirely and hardcoding a response. This has drawbacks: The hack obfuscates the code, the service invocation code path is removed, and the developer is required to make code changes if different responses are needed.

A better solution is to let the code continue performing the outbound service call but change the outbound URL to a stub service hosted by DataPower. The stub service uses the same transports as the real service, but doesn’t invoke the real backend. Instead, it returns a predefined response message. 

The benefits are clear. To the developer, they can switch between the stub and live service by updating the URL, which is usually stored in an external location from the code. Developers are removed from updating the application if more responses are needed. The DataPower stub also allows the service invocation framework to be exercised. The magic of the stub is that to the consumer, the fake stub service is a real endpoint serving valid responses.

Once we’ve moved the stub into an external service, we have ability to increase its functionality. We can support les responses or we can place conditions on when different responses should be served. In the hardcoded case, we were stuck with a single response and having to build new versions of the client when a change was needed. In the stub service case, we can be as flexible as needed without having to change the client or trigger a build.

In this article, we show how to implement a simple stub service with DataPower. This service provides a solid foundation that future enhancements can be built upon.

The stub service is built to meet the following requirements:

• The stub will listen for HTTP GET and POST requests.
• The incoming request will provide an HTTP Header named ‘STUBID’. This will be used to select the stub response file.
• The STUBID is case insensitive.
• There is no security needed for consumers of the stub service.
• Requests to and responses from the stub are not validated.
• The stub will support XML and JSON request and response messages.
• Adding additional stub responses should not involve modifying the DataPower configuration.
• The stub responses are stored locally in the appliance filesystem.
• The stub should respond with JSON/XML Errors for JSON/XML Clients.
• If a requested stub does not exist or can’t be loaded, return an error.

All of the project artifacts are available on GitHub as part of the ‘Practical DataPower’ repository. This repository hosts all the XSLT and DataPower configuration along with a domain export that can be easily imported onto a DataPower appliance.

The domain export can be downloaded GitHub here.

The domain can be imported from the Control Panel, via the Import Configuration wizard. By default the service listens on port 9191 of all interfaces.

The service is implemented as an Multi-Protocol Gateway (MPGW). This gives easy flexibility to add new transports to the service expand functionality as needed. If we had chosen an XML Firewall we would limit supported client transports to a single HTTP(/S) port.

The HTTP Front Side Handler accepts both GET and POST verbs. The requested URL component of the request is ignored. The HTTP Content-Type header is used to determine if the request  response stub is asking for an XML or JSON stub. This is used to determine whether to send a XML or JSON response.

The stub response filenames follow the convention:

• <“STUBID”>.<“extension”>
  • STUBID is the content  from the corresponding HTTP Header
  • extension is either ‘.xml’ or ‘.json’.

The stub files are stored in the ‘local://responses’ folder on the device. To add additional stubs responses to the service, simply drop a new file into the directory and the filename will be the ‘STUBID’ that the client should pass in the header.

This is a link to the DataPower configuration file being discussed. 

HTTP Front Side Handler – Enable GET Method.

If this is not enabled, then HTTP GET requests will be rejected.

Process Messages whose Body is Empty – ON

This allows the processing policy to run for HTTP GETs. If we didn’t enable this setting, we would get an internal error returned to the client.

Type: Dynamic Backend

This is just to keep the UI clean as we’re going to skip the connection to a backside server in the processing policy.

Request Type: XML

I think this exposes a quirk in when DataPower validates incoming XML transactions. All of the actions in the processing policy are set with an INPUT context of NULL. They never parse the INPUT document. It appears that DataPower only validates XML when the context is used by an action in a policy. By setting this value to XML and never touching the input, we can actually support non-XML data formats without triggering parsing errors.

Response Type: Pass-through

We skip the backside connection in the gateway so this setting doesn’t matter. Just to be safe, let’s tell DataPower not to touch the response message.

Front Side Timeout: 5 seconds

The stubs are saved to the local file system so clients should never spend more than 5 seconds waiting for a file to load. If they are, then there’s bigger issues with the environment hosting the service.

This rule services all requests for the stub. It is a ‘Client To Server’ Rule with 6 configured actions:

• A match action that accepts everything.
• An XSLT Transform action that configures the internal DataPower variables used later in the processing.
  • Input: NULL
  • Transform File: processRequest.xslt
  • Output: NULL
• A Fetch action that retrieves the selected response document from the DataPower file system.
  • Input: NULL
  • Source: ‘var://context/stub/url’
  • Method: GET
  • Advanced Parameter Output Type: ‘binary’.
    • We don’t want DataPower to parse or validate the contents of the fetched file.
  • Output: A DataPower context variable named ‘response’
• An XSLT Transform that resets the ‘Content-Type’ header.
  • Input: NULL
  • Transform File: setResponseContentType.xslt
  • Output: NULL
• A Set Variable action that configures the MPGW to skip the backside connection.
  • Input: NULL
  • Variable Name: ‘var://service/mpgw/skip-backside’.
  • Variable Value: ‘1’.
• A Result action.
  • Input: The ‘response’ context variable from the fetch action.
  • Output: OUTPUT.

This is a link to the processRequest.xslt file on GitHub.

In this XSLT we:

1. Set an XSLT variable named ‘stubId’ that standardizes the STUBID HTTP Header to all uppercase characters.
2. Remove the STUBID HTTP Header from the response message.
3. Set an xslt variable named ‘type’ to either ‘json’ or ‘xml’.
   1. We inspect the Content-Type HTTP header.
      1. If it contains the word ‘json’ then we set the variable to ‘json’.
      2. Otherwise we set the variable to ‘xml’.
4. Set an xslt variable named ‘stubURL’ to the location of the stub in the filesystem
   1. Example: ‘local:///responses/<$stubId>.<$type>’
5. Set an XSLT variable named ‘stubFileResponseCode’ that holds the status of the ‘url-open’ call to the $stubURL.
   1. We use ‘responsecode-binary’ as we don’t need to work with the contents of the response and we don’t want DataPower to parse it.
6. Set a DataPower Content Variable named ‘var://context/stub/url’ with the value of string($stubURL).
7. Set a DataPower Context Variable named ‘var://context/stub/type’ with the value of string($type).
8. Set a DataPower Content Variable named ‘var:/context/stub/content-type’ with the value of string(dp:http-request-header(‘Content-Type’)).
   1. Using ‘string(..)’ when setting a variable ensures DataPower stores the value as as string and it will be easily viewed in the probes.
      1. If you don’t do this, you can potentially see ‘node-set’ instead of the actual value of the variable in the ‘context’ tab.
         1. Clicking the node-set usually doesn’t show the value either.
9. Emit a XSLT message at the debug level that outputs the value of the variables set in the stub.
   1. This is used for debugging via the system log file.
10. Reject the transaction if:
    1. There was no ‘STUBID’ Header sent in the request.
    2. The stub file couldn’t be loaded from the local filesystem
       1. The $stubFileResponseCode content is an XML document returned by the dp:url-open(..) containing the status of the invocation.
       2. If the file was found, the document will contain a result header named ‘x-dp-local-file’.
          1. If this header isn’t found then dp:reject the transaction.

We set the DataPower context variable (‘var://context/stub/url’) used in the next Fetch Action. If an error occurred, then the dp:reject statements would cause DataPower to flow into the error rule.

This is a link to the setResponseContentType.xslt file on GitHub. 

In our fetch action, we set the advanced option for output type to ‘binary’. This causes the HTTP Content-Type header to switch to ‘application/octect-stream’. This  header is used by  frameworks to determine how to decode the response message. If it’s wrong, then client parsing of the response can occur.

For that reason, we introduce a fix-up XSLT after the fetch action.

This XSLT:

1. Sets the HTTP Content-Type header back to the value stored in ‘var://context/stub/content-type’ from ‘processRequest.xslt‘.

If our request rule ended here, we would get an error from DataPower about being unable to contact the backend server. This is because the default nature of a MPGW is to process a request, call an external backend and then process the response message back to the client. In our rule, we’ve already loaded the desired response in the fetch action. We need a way to tell DataPower to skip calling the backend server and just return to the client.

We can do this by setting a special DataPower service variable ‘var://service/mpgw/skip-backside’ and setting it to a value of ‘1’.

This rule is used when a stub error occurs and the transaction is rejected due to either a missing STUBID header or the stub not being found in the local file system. It is a Error Rule with 4 configured actions:

• A match action that accepts all errors.
• A transform action that processes the internal error into an XML or JSONx error message.
  • Input: INPUT
  • Transform File: ‘processError.xslt‘
  • Output: ‘dpvar_1’
• A conditional action with two sub actions
  • When ‘dpvar_1′ contains a JSONX message (/*[namespace-uri()=’http://www.ibm.com/xmlns/prod/2009/jsonx’] is true):
    • Run a transform action that will convert the JSONX into JSON
    • Input: ‘dp_var1’
    • Transform File: ‘store:///jsonx2json.xsl’
    • Output: OUTPUT
  • Otherwise (1=1)
    • Run a result action and return the XML error back to the client
    • Input: ‘dp_var1’
    • Output: OUTPUT

Test cases that utilize the ABC and DEF sample stubs can be downloaded from GitHub. The test cases are provided as an import to the DHC Extension for Chrome.

To use the tests, simply update the URL to point at your DataPower environment.

Now that we have a simple stub serving requests, we can think about how to expand the capabilities for the future. The stub can be updated to:

• Add new Front Side Handlers to support additional service transports such as MQ or HTTPS.
• Remove the requirement for the STUBID HTTP Header by parsing the request, identifying a key in the message and using that key to select the response.
• Add the ability to map pieces of the request message into the response message.
• Load the stub response files from an externally managed HTTP server.
  • This would free the DataPower team from stub management tasks and give the project teams the ability to modify the stubs at will.
• Provide a set of scripts to automatically update the stub service as needed
• Add support for stubs that need Cobol Copy Book formatted responses.
• Support finer grained control over error message format. Today the stub will respond with a common XML/JSON format that might not suit all of the stub consumers.
• Expand the XML error handling to also respond with SOAP faults if the input document is SOAP.

With this simple lightweight DataPower stub service, we’ve:

• Improved Code Quality by removing hacks embedded in the code that would have been used to bypass the invocation
• Allowed the project to continue development progress in parallel to the development of dependent services.
• Introduced a new service that future projects can easily leverage by submitting their own response stubs without having to touch the DataPower processing logic.
• Shown that it didn’t take a ton of configuration to get DataPower up and running
• Provided ideas on how the service can be expanded in the future to become even more useful.

Not bad for under a hundred lines of code.