Securing Containers Supply Chain using Secure Base Layer Image

Generating CVE-less docker images

Long List of CVE Patching Backlogs

Supply Chain Security can be pain to deal with due to new CVEs popping everyday. With the rise of containerization and industries adapting it with other technologies such as Kubernetes for their dynamic workloads makes container security important.

Most of the public containers found on registries such as hub[dot]docker[dot]com will be vulnerable to CVEs so If someone reuses those vulnerable container images with several CVEs then it increases a supply chain threat risk.

For a long time, I've struggled with the same issue leading my containers to be vulnerable posing threat for its users. Mitigating CVEs in the containers introduce breaking changes and it's time consuming to fix all those manually.

The above image is only the list of CVEs affecting the container excluding CVEs from the software running inside the container.

I was searching for an another solution which could help me secure my container images.

Shifting Left

Instead of patching vulnerabilities in the docker images why not use a secure image layer which handles supply chain security for you.

Well it isn't that straight forward, I had to look whether does my software supports the base image, will the project be maintained in future if I'm relying on it, what if project gets discontinued? All such questions might pop up into your mind if you're going to rely on someone else's secure base images.

Creating a secure base image with minimal software installed and using it would also be time consuming and hard to maintain if you're alone going to maintain it.

So, I wanted to rely on some open source project and it's community which can handle security updates for me.

I stumbled upon wolfi-os which does exactly what I was looking for.

It maintains a base secure image with minimal software installed with security updates.

Wolfi (🐙 world's smallest octopus) is an interesting project which seems to be similar like alpine distro but it's distroless, it uses glibc instead of musl (used by alpine) and it doesn't have its own kernel rather it relies on the host (container runtime environment) for it. This project focuses on supply chain security.

It comes with couple of secure configurations such as it has root and nonroot pre-configured with predefined boundaries. (By default non root user doesn't have any permissions to dirs following least privilege principle)

Features

From organization's README file

  • Provides a high-quality, build-time SBOM as standard for all packages

  • Packages are designed to be granular and independent, to support minimal images

  • Uses the proven and reliable apk package format

  • Fully declarative and reproducible build system

  • Designed to support glibc

Building Secure Containers

Let's dive into building secure container images. I'll be using my OWASP OFFAT dockerfile as an example.

The Old Naive Way!

In my previous docker image I used to use python's bullseye slim base image from docker hub and install OWASP OFFAT in it using pip. Pretty straight forward!! But the compressed image size generated was around 100MB (due to several unused packages) and it was vulnerable due to several CVEs.

FROM python:3.11-slim-bullseye

VOLUME [ "/offat/data" ]

WORKDIR /offat/data
WORKDIR /offat

COPY ../ /offat/

RUN python3 -m pip install -U pip

RUN python -m pip install -e ".[api]"

The New Secure Way!

Previous docker image wasn't optimized and I'll be using python 3.12 (latest stable release while writing) this time.

I'll be also using 2 stages (builder and runtime) among which builder will be used to install dependencies and install necessary python packages inside the virtual environment using poetry.

Later I'll copy virtual environment directory into the runtime image. This will eliminate overhead/unused packages from the image.

Below is the docker commands for builder stage

FROM cgr.dev/chainguard/wolfi-base:latest as builder

WORKDIR /offat

# python version
ARG version=3.12

# Set the environment variable to ensure consistent UTF-8 encoding across different systems
ENV LANG=C.UTF-8

# Set the environment variable to prevent Python from writing byte code files (.pyc)
ENV PYTHONDONTWRITEBYTECODE=1

# Set the environment variable to ensure Python outputs are unbuffered, improving logging in Docker containers
ENV PYTHONUNBUFFERED=1

# add venv to path
ENV PATH="/offat/.venv/bin:$PATH"

# install python
RUN apk add python-${version} py${version}-pip && \
        chown -R nonroot.nonroot /offat

# install poetry and copy lock file
RUN python -m pip install poetry
COPY pyproject.toml poetry.lock README.md ./
COPY offat ./offat

# poetry config
ENV POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    POETRY_CACHE_DIR=/tmp/poetry_cache

# install offat dependencies
RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install -E api --without dev

In runtime, I'll install python, copy necessary files, virtual environment from builder stage and provide necessary permissions to nonroot user. Lastly I'll change my user to nonroot user and set entrypoint for starting offat. 

FROM cgr.dev/chainguard/wolfi-base:latest as runtime

WORKDIR /offat

ARG version=3.12

ENV LANG=C.UTF-8
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV PATH="/offat/.venv/bin:$PATH"
ENV VIRTUAL_ENV=/offat/.venv

# install python dependencies and provide offat directory access to nonroot user
RUN apk add python-${version} py${version}-pip && \
	    chown -R nonroot.nonroot /offat


# copy venv from builder image
COPY --from=builder ${VIRTUAL_ENV} ${VIRTUAL_ENV}

# copy necessary files
COPY offat ./offat
COPY README.md CODE_OF_CONDUCT.md DISCLAIMER.md pyproject.toml .

USER nonroot

ENTRYPOINT [ "offat" ]

Refer complete docker file below

############################
# Builder Stage
############################
FROM cgr.dev/chainguard/wolfi-base:latest as builder

WORKDIR /offat

ARG version=3.12

ENV LANG=C.UTF-8
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV PATH="/offat/.venv/bin:$PATH"


RUN apk add python-${version} py${version}-pip && \
        chown -R nonroot.nonroot /offat

# install poetry and copy lock file
RUN python -m pip install poetry
COPY pyproject.toml poetry.lock README.md ./
COPY offat ./offat

# poetry config
ENV POETRY_NO_INTERACTION=1 \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    POETRY_CACHE_DIR=/tmp/poetry_cache

RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install -E api --without dev

############################
# runtime stage
############################
FROM cgr.dev/chainguard/wolfi-base:latest as runtime

WORKDIR /offat

ARG version=3.12

ENV LANG=C.UTF-8
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
ENV PATH="/offat/.venv/bin:$PATH"
ENV VIRTUAL_ENV=/offat/.venv

RUN apk add python-${version} py${version}-pip && \
	    chown -R nonroot.nonroot /offat


# copy venv from builder image
COPY --from=builder ${VIRTUAL_ENV} ${VIRTUAL_ENV}

# copy necessary files
COPY offat ./offat
COPY README.md CODE_OF_CONDUCT.md DISCLAIMER.md pyproject.toml .

USER nonroot

ENTRYPOINT [ "offat" ]

Trivy image scan results after building container image using wolfi-os base image.

Pros

  • Building secure images

  • Less overhead packages

  • Easy to use

  • No manual CVE patching

  • Open Source Project with Apache License

Cons

  • You'll be responsible for installing packages that you need. (This can be pro as well as a con since installing packages sometimes can be tricky)

  • Some packages cannot be installed directly and we cannot use apks from apline package repository since it can introduce vulnerabilities into the image or can break it. Instead one can request a package by creating issue or use melange to create custom base image using declarative yaml files.

Conclusion

Wolfi-OS can help to speed up the container hardening process and provide secure software deployments for your organization and side projects.

Using Wolfi-OS as my base image, I was able to reduce my container compressed size to around 50MB and all CVEs from bullseye container image were mitigated.

PreviousValidating File Content Types to avoid Malicious File Hosting using ML Model

Last updated