This ticket had assigned some Differential requests:
D836
D879
D819
D881

The best tool I found so far is [[ https://github.com/tutumcloud/builder | tutumcloud builder ]].

What is it?
Excerpt from github: //"A docker image that builds, tests and pushes docker images from code repositories. It is used by the Tutum platform to automate build and tests."//

How it works?
The builder image consist of relatively simple bash script that builds, tests and eventually pushes the image under test. It utilizes either docker in docker or machine's docker via /var/run/docker.sock, the former had some weird behavior, which wasn't exactly clear to me (magic docker fog), the latter worked as expected. Behaviour of the script can be changed or expanded with hooks (10 hooks total, 3 implemented in script itself, can be overridden) .

How to use it for testing?
Testing part of script calls command docker-compose and passes it a docker compose file - a yaml file that tells what image to use, what volumes to mount, what command to run (the test itself). These test definition files are located in given directory, git repo or tarball and has to be named *.test.yml or *-test.yml. Output is exctracted with command docker logs and PASS/FAIL of given test depends on returncode of command in test definition file.

Example of usage:
Pull the builder image:

docker pull tutum/builder

Now, it depends if you want to build the image from provided Dockerfile and test it or just pull the image and test it. From IRC conversation I presume that pulling and testing is what we want to do.

Create a directory with this structure:

somedir/
  |- hooks/
  |     |- build
  |- some.test.yml

build is a hook file and overrides the default build behaviour, it has to be empty so we skip the build phase. Test definition file some.test.yml contains:

 sut:
    image: pullable/image
    command: echo "test outcome: PASS" 

Finally, from this directory run a command:

docker run --rm -it --privileged -v $HOME/.docker:/.docker:ro -v /var/run/docker.sock:/var/run/docker.sock:rw -v $(pwd):/app tutum/builder

The good?
Besides that this is tool fulfills "testing docker images" the most, at least from what I found so far, tutum was [[ https://blog.docker.com/2015/10/docker-acquires-tutum/ | acquired ]] by Docker. That means the code could keep pace in turbulent docker reality.

The bad?
It doesn't poke at services from outside of container, however, this could be achieved in some hook. If we want to run a script file instead of oneliner in yaml, or if we want the test output in file, some minor code changes are required (one or two lines changed) - this is the biggest drawback, it is not 100% out-of-the-box solution.

Can the tutumcloud runner use multiple images or things like kubernetes? From skimming the docs, it sounds like not so much?

@lbrabec, do you think that it'd be worth using something that would be pretty specific to docker images instead of using something that would be more general (python unittest, py.test, etc.), might take a bit more work but would likely be useable in more places?

I guess this is resolved.