Building Java-based Docker services Starting with Git and a Dockerfile

The TL;DR version: You can get pretty far with just bash, Git and a Dockerfile.

You have to start somewhere

DockerAs anyone working at a startup will tell you, it is critical to focus on what really matters: creating the right product at the right time that customers find great value in. This means that while you need to have a vision for the company, the long-term business and engineering execution has to be incredibly flexible.

For the engineering team, this means plotting an indirect path from nothing towards where we want to end up, while making critical trade-offs along the way. We believe in automated and complete development and tool chains (pull requests, continuous integration, artifact repositories, automated deployments and rollbacks, etc…), but very early on we had to triage our DevOps must-haves and nice-to-haves.

This post details our initial approach to building Java-based Docker services with nothing else than Git and a Dockerfile. Building refers to going from a Git repository to a Docker image ready to be pushed and deployed.

But first, a few caveats:

  • I’m not suggesting you don’t use CI tools and alike, we eventually will and you should too; this is merely one approach when your tool chain isn’t in place yet
  • There are different ways to achieve a similar result depending on which tooling you have available and choose to use. I’m not suggesting this is a particularly good approach, but definitely served us well thus far

Our day 1 must-haves and nice-to-haves

Our must-haves:

  • Prioritize our build approach to be simple, maintainable and reproducible over being quick and optimized
  • All code is in Git, all issues are in JIRA, use Maven
  • No service ever gets built and deployed on someone’s machine
  • Use Docker for all Java-based services
  • If it is not reproducible and documented for yourself and someone else then it is worthless

Our nice-to-haves:

  • Private Maven repositories
  • Continuous integration
  • Many others we will eventually get to (pull requests, auto-deployments, etc…)

This early decision setup the parameters for how we would build our Java-based Docker services initially:

  • Pull the code from Git
  • Do a Maven build
  • Configure the Docker image

We chose a Dockerfile-centric approach where the build process is encapsulated by the Dockerfile itself. Other approaches will also work and may be better suited for your use cases.

Annotated Dockerfile sections

After several iterations, this is the Dockerfile we settled on. We pick one of our services as an example and this would be run on our build master instance.

Note that for readability we didn’t combine commands that could be combined into a single RUN statement. Build optimization isn’t something we’ve gotten to just yet (apparently shipping features is more important!).

[code language=”text”]
# Setting up an image to host address-service
FROM pivotfreight/jvm-ubuntu:14.04.3.PF1
MAINTAINER Olivier Modica <olivier@pivotfreight.com>
[/code]

Boring stuff, our Ubuntu image adds hardening and base tools.

[code language=”text”]
# Run as the pivotfreight user.
USER pivotfreight
RUN mkdir /home/pivotfreight/.ssh
COPY ssh/pivotfreight-deployment /home/pivotfreight/.ssh/
COPY ssh/config /home/pivotfreight/.ssh/
RUN sudo chown -R pivotfreight /home/pivotfreight/.ssh
RUN chmod 600 /home/pivotfreight/.ssh/pivotfreight-deployment
RUN chmod 600 /home/pivotfreight/.ssh/config
RUN ssh-keyscan bitbucket.org &amp;amp;amp;amp;gt;&amp;amp;amp;amp;gt; /home/pivotfreight/.ssh/known_hosts
[/code]

We currently use Bitbucket, and because we want to pull the code(and do an initial clone) within the image, we deploy the repository private deployment key onto the image along with the required SSH config.

[code language=”text”]
# Pull down the Pivot Freight source code (devops-config, common-data and address-service).
RUN mkdir /home/pivotfreight/src
WORKDIR /home/pivotfreight/src

# Clone (this may be run more than once if –no-cache isn’t used).
RUN git clone git@bitbucket.org:pivotfreight/devops-maven.git
RUN git clone git@bitbucket.org:pivotfreight/common-data.git
RUN git clone git@bitbucket.org:pivotfreight/address-service.git

# Run an initial build that pulls the Maven dependencies that will be cached (new dependencies will still be pulled by the build).
RUN cd /home/pivotfreight/src/common-data &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp; mvn clean install clean
RUN cd /home/pivotfreight/src/address-service &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp; mvn clean verify clean

RUN mkdir -p /home/pivotfreight/scripts
RUN mkdir -p /etc/pivotfreight/address-service
RUN mkdir -p /var/log/pivotfreight/address-service

CMD [“sh”, “-c”, “/home/pivotfreight/src/devops-maven/supervisor/init/address-service.sh”]
EXPOSE 8035
[/code]

We didn’t use a private Maven repository at that time, so we were building two projects on our single shared Common Data project, which provides our main data model along with the service itself (in this case, Address Service). We’re working on addressing that now, but having a single shared project early on hasn’t been too painful.

We do an initial Maven build for both projects to pull the dependencies onto the (cached) image, so subsequent builds will use the cache dependencies.

We also pull common build scripts from Git and setup the Docker image CMD (which starts the service through Supervisor) and expose its well-known port.

Note: Up to this point all those commands will be cached by the Docker build and will actually run only once (unless –no-cache is passed).

[code language=”text”]
# Bust the cache from this point on, this depends on the build-datetime file being available.
COPY build-datetime.txt /home/pivotfreight/
COPY build-branchname.txt /home/pivotfreight/

# Pull the latest from devops-maven.
WORKDIR /home/pivotfreight/src/devops-maven
RUN git pull

RUN cp /home/pivotfreight/src/devops-maven/supervisor/conf.d/address-service.conf /etc/supervisor/conf.d

RUN cp /home/pivotfreight/src/devops-maven/scripts/address-service*.sh /home/pivotfreight/scripts
RUN cp /home/pivotfreight/src/devops-maven/scripts/common-data*.sh /home/pivotfreight/scripts

# Run the build and do the cleanup.
WORKDIR /home/pivotfreight/scripts
RUN sh common-data.sh &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp; sh address-service.sh `cat /home/pivotfreight/build-branchname.txt` &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp; sh common-data-cleanup.sh &amp;amp;amp;amp;amp;&amp;amp;amp;amp;amp; sh address-service-cleanup.sh
[/code]

From this point on we bust the cache to ensure that we pull the latest code. There is no easy way to specify when Docker stops using the statement cache within the Dockerfile itself, so the most reliable way is to COPY a new file into the image. We chose to generate the build datetime for that purpose.

Note: This great blog post explains in detail how Docker’s caching actually works and why you need to bust the cache to ensure that git clone or git pull will actually do what you intend.

We also support passing a branch name for the actual service project through a file. Note that this will now be easier by the newly announced Docker 1.9’s support for build-time arguments. Yeah!

Finally, we run the actual Maven builds (essentially: git checkout followed by mvn clean verify) and clean up the build artifacts afterwards, so that we push a smaller and cleaner Docker image.

At this point the Docker image is ready to be tagged, pushed and deployed.

Putting all together

Finally, putting it all together, here is the build script that gets run alongside the Dockerfile:

[code language=”bash”]
#!/bin/bash

# Create the current build datetime file, used to bust the Docker cache.
TZ=’America/Chicago’ date –iso-8601=s &amp;amp;amp;amp;amp;amp;gt; build-datetime.txt

# Optionally the branch can be specified, otherwise master will be used.
echo “$1″ &amp;amp;amp;amp;amp;amp;gt; build-branchname.txt

docker build –rm=true -t pivotfreight/address-service .
[/code]

Here is the full Dockerfile for one of our service:

[code language=”text”]
# Setting up an image to host address-service
FROM pivotfreight/jvm-ubuntu:14.04.3.PF1
MAINTAINER Olivier Modica <olivier@pivotfreight.com>

# Run as the pivotfreight user.
USER pivotfreight
RUN mkdir /home/pivotfreight/.ssh
COPY ssh/pivotfreight-deployment /home/pivotfreight/.ssh/
COPY ssh/config /home/pivotfreight/.ssh/
RUN sudo chown -R pivotfreight /home/pivotfreight/.ssh
RUN chmod 600 /home/pivotfreight/.ssh/pivotfreight-deployment
RUN chmod 600 /home/pivotfreight/.ssh/config
RUN ssh-keyscan bitbucket.org &amp;amp;amp;gt;&amp;amp;amp;gt; /home/pivotfreight/.ssh/known_hosts

# Pull down the Pivot Freight source code (devops-config, common-data and address-service).
RUN mkdir /home/pivotfreight/src
WORKDIR /home/pivotfreight/src

# Clone (this may be run more than once if –no-cache isn’t used).
RUN git clone git@bitbucket.org:pivotfreight/devops-maven.git
RUN git clone git@bitbucket.org:pivotfreight/common-data.git
RUN git clone git@bitbucket.org:pivotfreight/address-service.git

# Run an initial build that pulls the Maven dependencies that will be cached (new dependencies will still be pulled by the build).
RUN cd /home/pivotfreight/src/common-data &amp;amp;amp;amp;&amp;amp;amp;amp; mvn clean install clean
RUN cd /home/pivotfreight/src/address-service &amp;amp;amp;amp;&amp;amp;amp;amp; mvn clean verify clean

RUN mkdir -p /home/pivotfreight/scripts
RUN mkdir -p /etc/pivotfreight/address-service
RUN mkdir -p /var/log/pivotfreight/address-service

CMD [“sh”, “-c”, “/home/pivotfreight/src/devops-maven/supervisor/init/address-service.sh”]
EXPOSE 8035

# Bust the cache from this point on, this depends on the build-datetime file being available.
COPY build-datetime.txt /home/pivotfreight/
COPY build-branchname.txt /home/pivotfreight/

# Pull the latest from devops-maven.
WORKDIR /home/pivotfreight/src/devops-maven
RUN git pull

RUN cp /home/pivotfreight/src/devops-maven/supervisor/conf.d/address-service.conf /etc/supervisor/conf.d

RUN cp /home/pivotfreight/src/devops-maven/scripts/address-service*.sh /home/pivotfreight/scripts
RUN cp /home/pivotfreight/src/devops-maven/scripts/common-data*.sh /home/pivotfreight/scripts

# Run the build and do the cleanup.
WORKDIR /home/pivotfreight/scripts
RUN sh common-data.sh &amp;amp;amp;amp;&amp;amp;amp;amp; sh address-service.sh `cat /home/pivotfreight/build-branchname.txt` &amp;amp;amp;amp;&amp;amp;amp;amp; sh common-data-cleanup.sh &amp;amp;amp;amp;&amp;amp;amp;amp; sh address-service-cleanup.sh
[/code]

Hopefully, this gives you a good overview of one approach to building Java-based Docker services using the base building blocks you may start with on day 1.

Cheers,
Olivier.