Why container size matter?

This is a good question: storage is cheap why should we care about it? Who cares if a ‘Hello world’ NodeJS app uses 2GB of image as long as it works? Let me be a grumpy sysadmin guy and ask some question my dear developer friend:

  • Is network usage free? No.
  • Will the application start slowly on a cold start? Yes. Will you blame the environment instead of reflecting your own setup? Yes.
  • Does a larger image mean it contains more packages? Yes. Do more packages means bigger attack surface? Yes. Basically do you create extra maintenance work for you? Yes.
  • Besides storage may cheap but not infinite.

Of course, it does not mean that every single image must be a pure SCRATCH container. Image still can contain a shell for debugging or monitoring. But there some practice that can help like multi-stage build. Today, I won’t discuss multi-stage builds; instead, I’ll explain an obvious concept that everyone learns – at least theoretically – during container education: how layers work.

What are layers?

Every single statement in a Containerfile (or Dockerfile) means one layer. Layers can be cached separately. For example, having a container where only the application has been changed, and makes a re-deploy, only the changed layers would be downloaded. Great concept, right?

How don’t do it?

I never blame anybody for what they are doing. I always wonder, what they were thinking – maybe they don’t know some basics or just don’t care? I never assume anything in advance; instead, I rather start by speaking with people about what they are doing and why. Maybe I am the person who is wrong and I have something to learn.

So what happened? Removing the uninterested part, this was in the Containerfile:

WORKDIR /tmp/app
RUN wget -nv "https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz" \
    && tar -xvf "trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz"

WORKDIR /app
RUN cp /tmp/app/trivy ./trivy \
    && chmod +x /app/trivy \
    rm -r /tmp/app

What is wrong with this file? It’s commendable that they considered removing the downloaded tarball. However, it does not really matter because the layer from the first RUN command remains and consumes space.

After build, let’s analyze the history of the image.

$ podman images
REPOSITORY                                       TAG               IMAGE ID      CREATED         SIZE
localhost/trivy-for-gitea                        latest            7d2c07fb0c2c  12 seconds ago  434 MB

$ podman history localhost/trivy-for-gitea:latest
ID            CREATED         CREATED BY                                     SIZE        COMMENT
79f494556b9d  31 seconds ago  /bin/sh -c #(nop) ENTRYPOINT [ "./scan.sh" ]   0B          FROM f636633dbc4d
<missing>     31 seconds ago  /bin/sh -c #(nop) LABEL summary="Trivy sca...  0B          FROM 79f494556b9d
<missing>     32 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  8.7kB       FROM 84962301ab7d
84962301ab7d  32 seconds ago  /bin/sh -c #(nop) COPY file:3e0dc7f6445563...  8.19kB      FROM 9db1ad5b3d33
9db1ad5b3d33  33 seconds ago  /bin/sh -c #(nop) COPY file:467d03d5b25988...  3.07kB      FROM b98538d538f3
175d35bb5f4f  33 seconds ago  /bin/sh -c #(nop) WORKDIR /app                 0B          FROM 175d35bb5f4f
<missing>     36 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  150MB       FROM 1229b642a3a4
b593e5530f94  38 seconds ago  /bin/sh -c #(nop) WORKDIR /app                 0B          FROM b593e5530f94
<missing>     42 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  195MB       FROM 715c1bb26193
9958fde18afb  58 seconds ago  /bin/sh -c #(nop) ARG TRIVY_VERSION dataSo...  0B          FROM 8b3fa6700c51
<missing>     58 seconds ago  /bin/sh -c #(nop) WORKDIR /tmp/app             0B          FROM 9958fde18afb
<missing>     58 seconds ago  /bin/sh -c mkdir -p /app /tmp/app              3.07kB      FROM e51ff272e7cb
e51ff272e7cb  59 seconds ago  /bin/sh -c apt-get update     && apt-get i...  11.2MB      FROM docker.io/library/debian:bookworm-slim
595d99e62673  5 weeks ago     # debian.sh --arch 'amd64' out/ 'bookworm'...  77.9MB      debuerreotype 0.15

Removing the uninterested lines, this two remains. Layer of first run:

<missing>     42 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  195MB       FROM 715c1bb26193

Layer of second run (copy of binary):

<missing>     36 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  150MB       FROM 1229b642a3a4

It is visible that, although the remove has been done, it does not reduce the size of previous layer.

How to fix it?

It is not difficult to fix it, just need to merge the two RUN statement.

WORKDIR /tmp/app
RUN wget -nv "https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz" \
    && tar -xvf "trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz" \
    && cp /tmp/app/trivy /app/trivy \
    && chmod +x /app/trivy \
    && rm -r /tmp/app

After its build, we can see that the image size has been reduced.

podman images
REPOSITORY                                       TAG               IMAGE ID      CREATED         SIZE
localhost/trivy-for-gitea                        latest            c6281027b217  6 seconds ago   239 MB

podman history localhost/trivy-for-gitea:latest
ID            CREATED         CREATED BY                                     SIZE        COMMENT
c6d0dd60d7c4  18 seconds ago  /bin/sh -c #(nop) ENTRYPOINT [ "./scan.sh" ]   0B          FROM c397c8d6547f
<missing>     19 seconds ago  /bin/sh -c #(nop) LABEL summary="Trivy sca...  0B          FROM c6d0dd60d7c4
<missing>     19 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  8.7kB       FROM 9316d745be98
9316d745be98  20 seconds ago  /bin/sh -c #(nop) COPY file:3e0dc7f6445563...  8.19kB      FROM 9d3530a4c297
9d3530a4c297  20 seconds ago  /bin/sh -c #(nop) COPY file:467d03d5b25988...  3.07kB      FROM 20ffb695bf5b
cec9cb87d99a  20 seconds ago  /bin/sh -c #(nop) WORKDIR /app                 0B          FROM cec9cb87d99a
<missing>     24 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  150MB       FROM 715c1bb26193
9958fde18afb  10 minutes ago  /bin/sh -c #(nop) ARG TRIVY_VERSION dataSo...  0B          FROM 8b3fa6700c51
<missing>     10 minutes ago  /bin/sh -c #(nop) WORKDIR /tmp/app             0B          FROM 9958fde18afb
<missing>     10 minutes ago  /bin/sh -c mkdir -p /app /tmp/app              3.07kB      FROM e51ff272e7cb
e51ff272e7cb  10 minutes ago  /bin/sh -c apt-get update     && apt-get i...  11.2MB      FROM docker.io/library/debian:bookworm-slim
595d99e62673  5 weeks ago     # debian.sh --arch 'amd64' out/ 'bookworm'...  77.9MB      debuerreotype 0.15

We can see that the 195MB big layer just disappeared. With this RUN statement we handle everything at one place, we only have just one 150MB layer (which is size of the binary).

<missing>     24 seconds ago  |3 TRIVY_VERSION=0.61.0 dataSource=gihub-r...  150MB       FROM 715c1bb26193

Another example for RUN:

RUN apt-get update \
    && apt-get install --no-install-recommends -y wget=* ca-certificates=* jq=* \
    && apt-get clean  \
    && rm -rf /var/lib/apt/lists/*

Final words

I always like to find out why the problem existed at first place. It is not judgement, not mocking or something, but I always like to hear and see feedback why something bad happened. It may help to improve learning materials or documentation.

In this special case, the root cause was, that they thought it works like a shell. So, they told, if they would do it in shell, like this:

$ cd /tmp/app
$ wget -nv "https://github.com/aquasecurity/trivy/releases/download/v${TRIVY_VERSION}/trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz" \
    && tar -xvf "trivy_${TRIVY_VERSION}_Linux-64bit.tar.gz"
$ cd /app
$ cp /tmp/app/trivy ./trivy
$ rm /tmp/app

And they just copied this method, by changing the cd with RUN. It is not bad that happened, or wrong. I believe that everybody in the IT learning until the very end. There are a lot of thing that I also don’t know, maybe doing things not perfectly, but always open to learn.