Exposing dockerfile content to an EFS volume in Fargate 1.4.0

Semantic docker volume and Fargate EFS mount differences...

There is a page of Amazon ECS documentation that I tried to consume without success a couple of weeks ago. Possibly because of an AWS introduced regression in the Fargate platform. Possibly because my use-case is subtly different from the use case AWS tested. I reread the documentation several times, but ultimately failed to find a flaw in my configuration that explained why I was unable to elicit the desired behavior from Amazon ECS.

The use case in question was a deployment of OpenEMR to ECS with a stateful EFS volume mounted to the /var/www/localhost/htdocs/openemr/sites directory within the deployed container. Without this configuration, the state of my OpenEMR cluster would not persist beyond the lifecycle of each executing task, rendering my OpenEMR storage volatile and highly unsuitable for a production workload.

Running Fargate tasks is pretty routine at this point and so I was somewhat surprised¹ to receive a fatal error preventing the standard OpenEMR built container from starting in task logs upon container start-up:

2021-06-01T22:34:24.675-04:00	PHP Warning: require_once(/var/www/localhost/htdocs/openemr/sites/default/sqlconf.php): failed to open stream: No such file or directory in Command line code on line 1

2021-06-01T22:34:24.675-04:00	PHP Fatal error: require_once(): Failed opening required '/var/www/localhost/htdocs/openemr/sites/default/sqlconf.php' (include_path='.:/usr/share/php7') in Command line code on line 1

That’s not how this docker thing is supposed to work! If someone builds an image it’s meant to include all the dependencies necessary to get it to run. Why then is the file /var/www/localhost/htdocs/openemr/sites/default/sqlconf.php not included?

There’s no way a widely distributed docker image is this broken though. Especially after a quick google search yields nobody with the same problem as me. For kicks, though, I roll back to an older version of the docker image. And, still no discernible change.

You might have noticed the reason that the file was not available to read.² I notice that the file in question lives on the mounted volume and has been overridden by EFS.

Sure enough, upon removing the EFS mount point in the task definition the container starts as expected. Now, I have a working OpenEMR installation with no persistent volume attached. Not very useful, in and of itself, but at least I can be sure that my diagnosis of the problem is correct.

I am initially taken aback by this file masking behavior. In OpenEMR’s example docker-compose.yml configuration file, there is a volumes configuration specifying the same mount point as my EFS mount point. Why does EFS not work if the docker-compose.yml configuration does?

The first unlikely possibility that comes to me is that the docker-compose.yml is new and might just be broken. I quickly spin it up locally and rule that out as a possibility. Everything works as expected

Reluctantly, because I can already sense the rabbit hole here, I spawn several google searches and start reading through documentation. If this blog were a movie, a montage would probably start playing now showing me frantically reading AWS docs at like 50x speed³. It wouldn’t be very a very interesting montage⁴.

Here is some of what I uncovered down the rabbit hole:

1. If you start a container with docker and that docker container references a volume that does not yet exist, docker will create this container for you.⁵

2. Prior to mounting this new volume to the mount point, docker will first populate the new volume with the content found within the mounted directory.⁶

3. By default when you specify a docker volume in a docker-compose file, if the volume does not yet exist, docker-compose honors the docker specification and will pre-populate the volume with the content of the mounted directory prior to mounting the volume.⁷

In the case of OpenEMR’s docker-compose file, when docker-compose up is invoked, a volume is created, and the /var/www/localhost/htdocs/openemr/sites/default/sqlconf.php file is copied into the created volume as per the above specifications.

Apparently, when I specify a task definition in ECS and specify a volume and mount points, upon the initial task execution, Fargate does not copy the container content to the volume.

I speculate that this behavior is because Fargate creates the volume prior to launching the task definition. Pre-creating this volume prior to launching the task means that no data will be copied into the volume before it is mounted to the Fargate task. Because OpenEMR relies on content within a mounted directory to launch, and because Fargate does not ever copy this content to a mounted EFS directory, using the standard OpenEMR docker image with a volume at /var/www/localhost/htdocs/openemr/sites fails.

I do not know whether my hypothesis is correct, but this github issue seems to mirror my experience at least. I am not the only person confused as to how to get stuff to run on Fargate 1.4.0.

Note: After writing and publishing this, u/brunokktro points out that this behavior is most likely due to the docker engine runtime of Fargate being swapped out with containerd for Fargate 1.4.0. Thanks for the insight u/brunokktro.

Good news is AWS has seemingly issued a fix. So I follow the instructions in the linked documentation:

1. Create a Dockerfile… The VOLUME directive should specify an absolute path.

FROM openemr/openemr:6.0.0
VOLUME "/var/www/localhost/htdocs/openemr/sites"

2. In the task definition volumes section, define a volume.

volumes: [
        {
          name: 'site',
          efsVolumeConfiguration: [{
            fileSystemId: siteVolume.volume.id,
            rootDirectory: '/'
          }]
        }
      ]

3. In the containerDefinitions section, create the application container definitions so they mount the storage. The containerPath value must match the absolute path specified in the VOLUME directive from the Dockerfile.

        mountPoints: [
          {
            containerPath: '/var/www/localhost/htdocs/openemr/sites',
            sourceVolume: 'site'
          }
        ]

I deploy the changes, but no dice. Still seems that the container files aren’t being copied to the EFS directory. I’m not sure if this is AWS prematurely closing the above ticket or if my implemenation is flawed but it seems like I’m not the only one experiencing this.

Either way, I need this urgently and don’t have time to go through the AWS support turn-around time. So I add the following lines to my Dockerfile:

RUN mv /var/www/localhost/htdocs/openemr/sites /sites
CMD [ "./entrypoint.sh" ]

This moves all the assets from the mounted directory to a separate directory. Then I write the following ./entrypoint.sh:

#!/bin/sh
DIR="/var/www/localhost/htdocs/openemr/sites/"
if [ "$(ls -A $DIR)" ]; # Execute once 
then
    echo "Open EMR sites already initialized"
else
    mv /sites/* $DIR # Copy temp files to EFS volume
    chown -R apache /var/www/localhost/htdocs/openemr/sites
    chmod -R 755 /var/www/localhost/htdocs/openemr/sites 
fi

./run_openemr.sh # OpenEMR original entrypoint script

Similar steps can be used whenever you need hydrate an EFS volume once when you start a container for the first time.

First, you move the files you want to hydrate out of the mounted directory into a temporary directory. Then you create a new entrypoint to do the following steps:

1. If the EFS volume is not already hydrated, move the hydrated files from the temporary directory to the original directory that is now backed by EFS.

2. Ensure that ownership and permissions are adequately set for all files that you copied into your EFS volume

3. Run the original entrypoint.

I still don’t know whether Fargate was broken or my configuration was broken. Whatever the case, if you’re struggling to get Fargate to hydrate your EFS volume from a docker container, I’ve demonstrated a technique, somewhat inelegant though it may be, that I hope helps you achieve your goal.

1. This is a lie. I’m never actually surprised when things go wrong.

2. The title of the blog and aforementioned details give it away.

3. The video would play at 50x speed. I was reading pretty slowly.

4. Or movie for that matter.

5. https://docs.docker.com/storage/volumes/#start-a-container-with-a-volume

6. https://docs.docker.com/storage/volumes/#populate-a-volume-using-a-container

7. The docker-compose reference allows you to specify nocopy: true to circumvent this default behavior.