Executive Summary
Text snippet sharing services such as Pastebin and Github Gist are growing in popularity as an easy method of sharing structured data amongst multiple parties. A number of these services are readily available over the internet, but the equivalent service inside most corporate firewalls is missing. The lack of an internal snippet sharing service potentially pushes confidential information into these public repositories. This puts the enterprise at risk of data leaks. The alternative is to email snippets. When a snippet is emailed the email client can ruin formatting increasing debugging effort, the message is sent to a large distribution list who may not need the detailed technical information and the data contained in the snippet is now replicated into every inbox subtly increasing on-going email carrying costs.
The answer is to introduce an internal instance of one of these servers and reap the benefits of easy access and pretty formatting. This article outlines how to go about installing an open source server called HasteBin using another open source project called Docker.
Introduction
Previously, I’ve written about tools to improve the productivity of DataPower Developers. One of those tools was Alchemize, a chrome extension that formats XML & JSON to increase readability. This reformatted content is then usually sent via email to the tester(s). After thinking about this, I realized that emailing XML and JSON is an inelegant solution for transferring highly technical and structured data. The email client can mangle formatting and more emails steal focus and clutter inboxes. Wouldn’t it be better to host the data somewhere accessible from a browser and be able to forward a link? Luckily, these tools already exist.
On the internet there are hundreds of solutions like Pastebin and Github Gist which are growing in popularity for sharing snippets. These tools do exactly what we need, paste text and then reference it with an unique URL. They live outside the firewall so they can’t be used to paste internal enterprise information, lest it leak into the wrong hands. This means to enable this method of data sharing, we need to find a solution that can be installed privately and be managed inside the intranet.
In spite of the explosive growth in usage on the internet, inside the corporate firewall these tools have yet to see wide-spread adoption. I suspect the following reasons:
- The internal server provisioning process is high-touch, lengthy and overly expensive
- No team mandate exists for shared process improvement technologies
- The perception that there is no problem and not having a solution is fine
The worst part of not having an internal server is that sends your internal users off to the external services with your confidential information. These services may imply privacy, but once the data has breached the firewall any hope of guaranteeing confidentiality is lost.
In this article, we will go through the steps to configure a docker container with an open source snippet sharing server named Hastebin. At the end of the article, we’ll have a server up and running, ready for immediate consumption.
Introducing Hastebin
Hastebin consists of the haste-server and optional haste-client.
From the Server’s README.MD:
|
1 2 3 4 5 6 |
Haste is an open-source pastebin software written in node.js, which is easily installable in any network. It can be backed by either redis or filesystem, and has a very easy adapter interface for other stores. A publicly available version can be found at hastebin.com Major design objectives: Be really pretty. Be really simple. Be easy to set up and use. |
The reason that I chose hastebin is that it can be installed locally, offers a clean user interface and features a simple API to post snippets to the server. There are more complicated options available but in the DataPower usage scenario, we just want to be able to share XML and JSON quickly. This makes Hastebin a good fit.
Using the optional hastebin-client, posting to the server from a command line is as easy as piping the data to the haste command:
|
1 2 |
# echo "Hello World" | haste http://hastebin.com/awetacadex |
The command returns a new URL that can be shared as needed. Snippets are saved for 30 days before they are automatically deleted. This can be changed as needed.
Docker
Docker is a Linux container technology that automates the creation of software environments while not incurring the overhead of full-blown server virtualization such as VMWare or VirtualBox. This article assumes familiarity with Docker and that it is already installed on your server. If you need more information about the technology, check out the amazing 10 minute Docker Tutorial.
For even more information, I recommend purchasing The Docker Book by James Turnbull.
Hastebin – config.js
The hastebin server reads from a single JSON configuration file:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
{ "host": "0.0.0.0", "port": 7777, "keyLength": 10, "maxLength": 400000, "staticMaxAge": 86400, "recompressStaticAssets": true, "logging": [ { "level": "verbose", "type": "Console", "colorize": true } ], "keyGenerator": { "type": "phonetic" }, "storage": { <strong>"type": "file",</strong> <strong> "path": "./data"</strong> }, "documents": { "about": "./about.md" } } |
The only different between this configuration and the default is that I am defining the storage type to be the file system. I’m choosing the file system because the intention is that the number of users of the system is low and running a supporting database is overkill.
Docker – Dockerfile
The dockerfile defines how to build a Docker container. I prefer to build all my containers from a base Ubuntu image, rather than pulling from non-official repositories in DockerHub. If you prefer to the do the same, the docker file is here:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
FROM ubuntu:14.04 RUN apt-get update && apt-get install -y \ git \ nodejs \ npm RUN sudo ln -s /usr/bin/nodejs /usr/bin/node RUN git clone https://github.com/seejohnrun/haste-server.git /opt/haste ADD ./include/config.js /opt/haste/config.js WORKDIR /opt/haste RUN npm install EXPOSE 7777 CMD ["npm", "start"] |
You can then build a docker image by running:
|
1 2 |
docker build -t=hastebin-server . |
This will build a contained and name it ‘hastebin-server’. The docker file is based on ubuntu 14.04 and installs some dependencies: nodejs, npm and git. The server inside the container is running on port 7777, so we expose it.
When are building the container yourself, you need to include the config.js in a subdirectory named ‘include’ in order to have docker overwrite the default and enable file system storage (as per the ADD command).
Docker – Data Volume Container
By default in Docker, the data inside an instance of a container is only kept for as long as the container exists. If you delete the instance, then all of the corresponding data is deleted. This isn’t great as it would delete the snippets hosted on the server. To protect this data, we can leverage the Docker Data Container Pattern.
In this pattern, we create a container instance that holds a volume. This volume will store the contents of the /opt/haste/data directory, the directory that haste uses to store pastes.
|
1 2 3 |
docker create -v /opt/haste/data --name hastebin-data hastebin-server /bin/true |
Our new data container instance is named ‘hastebin-data’. We’ll reference it in the next command to start a new container from our hastebin-server image.
Starting the Hastebin Container
We have a docker image for hastebin and a data volume container where we can store the pastes, so lets start a new container and attach the volume.
|
1 2 3 |
docker run -d -p 7777:7777 --name hastebin-server-1 --volumes-from=hastebin-data hastebin-server |
In this command, we create an container from the hastebin-server image. We bind the containers’ exposed port 7777 to port 7777 on the docker host. We name the container instance hastebin-server-1. We use the volume defined by the hastebin-data data container.
When the command completes, the hastebin server will be available at http://localhost:7777. If you use boot2docker, replace localhost with the IP address from
|
1 |
boot2docker ip |
If you use docker-machine, replace localhost with the IP address for your environment from:
|
1 |
docker-machine ip <env> |
Using Hastebin
Hastebin has three methods for submitting content to the server. You can use:
- A Web Browser
- The haste-client binary
- Send cURL commands to the Web API
For a web browser, connect to the server’s URL and type the data that you’d like to save.
For the haste-client, you can read the documentation.
Hastebin Posting Script for OSX DataPower Developers
The common use case for data power developers is to copy unformatted data from a probe and then paste it into an email. I wrote a script to automate this process.
This script is for DataPower developers on OS X. It will take the contents of the clipboard and:
- Pretty-print JSON or XML
- Echo the contents to confirm the data to be pasted
- Submit the data to the server using cURL
- Copy the response URL into the clipboard
- Open the response URL into Safari
The command supports the following argument:
- -y: Bypass the confirmation prompt
In the script, you can change the HASTE_SERVER variable to point to the URL of your own server.
The command has the following dependencies:
- cURL – used to submit the request to the server.
- xmllint – used to format XML requests
- python – used to format JSON requests
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
# @author: Dan Zrobok # # Pastes Data from the clipboard into a HasteBin server. # The contents will be formatted if the payload is JSON or XML. # HASTE_SERVER=http://localhost:7777 xml=$(echo $(pbpaste) | xmllint --format - 2> /dev/null) if [ $? -eq 0 ]; then type="XML"; formatted=$xml; else json=$(pbpaste | python -m json.tool 2> /dev/null) if [ $? -eq 0 ]; then type="JSON" formatted=$json; else type="TEXT"; formatted=$(pbpaste); fi fi echo "" echo "=============$type START=============" echo -e "$formatted" echo "" echo "=============$type END===============" if [ $# -eq 0 ] && [ ]$1 != "-y" ]; then echo "" echo -n "Upload to $HASTE_SERVER [N/y] ? :" read UPLOAD else UPLOAD=Y fi case $UPLOAD in (y|Y|YES|yes|Yes) echo "" echo "=============TRANSFER=============" resp_json=$(curl -X POST -d "$formatted" $HASTE_SERVER/documents) echo "=================================" if [ $? -eq 0 ]; then echo "$resp_json" | awk -F '"' '{print "'"$HASTE_SERVER"/'"$4}' | pbcopy; echo "" echo "======== HASTE URL =========" pbpaste echo "=============================" echo "" echo "Opening Browser..." open $(pbpaste) echo "Success" echo "" exit else echo "Connection Error ($?) to $HASTE_SERVER. Nothing uploaded." exit fi esac echo "Aborted..." |
Given this script, I can now open a DataPower probe and copy the XML/JSON into my clipboard. I can then run the above script and the data will be uploaded to hastebin. My clipboard is now updated to hold the new URL, so I can paste it into an email or chat window. Easy as 1-2-3.
Conclusion
In this article, we went from a tedious information delivery process backed by email into a hastebin solution installed using Docker. Instead of emailing out large blocks of XML/JSON, they now reside cleanly in a third party tool that can manage them. In addition, we’re containing any confidential data from escaping to the internet-based services.
About the Author
Share this Post