5 Advanced Docker Caching Techniques

The most fundamental Docker caching technique is to order your Dockerfile layers from least to most frequently changing. This simple approach can yield significant performance improvements with minimal effort.

FROM node:18-alpine
WORKDIR /app
COPY . .
RUN npm install
EXPOSE 3000
CMD ["npm", "start"]

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD ["npm", "start"]

This pattern applies to any language/framework:

• Python: Copy requirements.txt first, then run pip install
• Ruby: Copy Gemfile and Gemfile.lock first, then run bundle install
• Go: Copy go.mod and go.sum first, then run go mod download
• Java: Copy pom.xml or build.gradle first, then resolve dependencies

Cache mounts are one of the most powerful features in BuildKit (Docker's modern build engine). They allow you to cache specific directories between builds without including them in the final image.

This technique is particularly valuable for package managers that maintain their own cache, such as npm, pip, apt, and Maven.

# syntax=docker/dockerfile:1.4
FROM node:18-alpine
WORKDIR /app
COPY package*.json .
RUN --mount=type=cache,target=/root/.npm \
    npm ci
COPY . .
CMD ["npm", "start"]

Here are examples for other package managers:

# Python pip cache
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

# APT package cache
RUN --mount=type=cache,target=/var/cache/apt \
    --mount=type=cache,target=/var/lib/apt \
    apt-get update && apt-get install -y some-package

# Maven cache
RUN --mount=type=cache,target=/root/.m2 \
    mvn package -DskipTests

To use this technique, you need to enable BuildKit by setting the DOCKER_BUILDKIT=1 environment variable or enabling it in your Docker configuration file.

In CI/CD environments, each build typically starts with a fresh environment, which means Docker can't reuse layer cache from previous builds. Remote caching solves this problem by storing the cache in a remote registry.

Docker BuildKit supports two methods for remote caching:

Inline Cache

This method embeds cache metadata directly in your Docker image:

# Building with inline cache metadata
DOCKER_BUILDKIT=1 docker build \
  --build-arg BUILDKIT_INLINE_CACHE=1 \
  -t myregistry.com/myapp:latest .

# Push image with cache metadata
docker push myregistry.com/myapp:latest

# In CI, pull and use cache
DOCKER_BUILDKIT=1 docker build \
  --cache-from myregistry.com/myapp:latest \
  -t myregistry.com/myapp:latest .

Dedicated Cache Storage

This method stores cache separately from your images, which is more efficient for CI/CD:

# Building with remote cache exporter
DOCKER_BUILDKIT=1 docker build \
  --cache-to type=registry,ref=myregistry.com/myapp:cache \
  -t myregistry.com/myapp:latest .

# In CI, import the cache
DOCKER_BUILDKIT=1 docker build \
  --cache-from type=registry,ref=myregistry.com/myapp:cache \
  -t myregistry.com/myapp:latest .

BuildKit can automatically parallelize independent build stages, significantly reducing build times for complex multi-stage builds. This feature is especially powerful when combined with other caching techniques.

# syntax=docker/dockerfile:1.4
FROM node:18 AS deps
WORKDIR /app
COPY package*.json .
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:18 AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:18-alpine AS runner
WORKDIR /app
COPY --from=builder /app/dist ./dist
RUN --mount=type=cache,target=/root/.npm npm ci --only=production
EXPOSE 3000
CMD ["npm", "start"]

To take full advantage of parallel building, it's important to structure your Dockerfile with independent stages that don't depend on each other. If you have a complex build with multiple components, consider separating them into independent stages:

# syntax=docker/dockerfile:1.4
# These stages can run in parallel
FROM node:18 AS frontend-builder
WORKDIR /app/frontend
COPY frontend/package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci
COPY frontend/ ./
RUN npm run build

FROM maven:3.8-openjdk-17 AS backend-builder
WORKDIR /app/backend
COPY backend/pom.xml ./
RUN --mount=type=cache,target=/root/.m2 mvn dependency:go-offline
COPY backend/ ./
RUN mvn package -DskipTests

# Final stage combines outputs from parallel stages
FROM nginx:alpine
COPY --from=frontend-builder /app/frontend/build /usr/share/nginx/html
COPY --from=backend-builder /app/backend/target/myapp.jar /app/
# ... rest of your Dockerfile

DOCKER_BUILDKIT=1 docker build --progress=plain .

For projects with complex dependency structures, a more sophisticated approach to dependency caching can yield additional performance improvements.

This technique leverages a combination of BuildKit features with careful stage design to optimize caching behavior for dependencies:

# syntax=docker/dockerfile:1.4
FROM node:18-alpine AS base
WORKDIR /app

# Stage 1: Install base dependencies (changes rarely)
FROM base AS deps-base
COPY package.json package-lock.json* ./
RUN --mount=type=cache,target=/root/.npm \
    npm ci --only=production && \
    cp -R node_modules prod_modules && \
    npm ci

# Stage 2: Install dev dependencies (changes occasionally)
FROM deps-base AS deps-dev
COPY tsconfig.json .eslintrc* jest.config* ./
RUN --mount=type=cache,target=/root/.npm \
    test -d node_modules || npm ci

# Stage 3: Build the application (changes frequently)
FROM deps-dev AS builder
COPY src ./src
COPY public ./public
RUN npm run build

# Stage 4: Production image (minimal)
FROM base AS runner
ENV NODE_ENV=production
COPY --from=deps-base /app/prod_modules ./node_modules
COPY --from=builder /app/dist ./dist
# ... rest of your configuration

You can adapt this pattern to other languages and frameworks:

# Python example with poetry
FROM python:3.10-slim AS base
WORKDIR /app

# Core dependencies (changes rarely)
FROM base AS deps-core
COPY pyproject.toml poetry.lock* ./
RUN --mount=type=cache,target=/root/.cache/pip \
    --mount=type=cache,target=/root/.cache/poetry \
    pip install poetry && \
    poetry export --without-hashes -f requirements.txt -o requirements.txt && \
    pip install -r requirements.txt

# Development dependencies (changes occasionally)
FROM deps-core AS deps-dev
RUN --mount=type=cache,target=/root/.cache/pip \
    --mount=type=cache,target=/root/.cache/poetry \
    poetry install --no-root

# Build stage
FROM deps-dev AS builder
COPY . .
RUN poetry build

# Production image
FROM base AS runner
COPY --from=deps-core /app/requirements.txt .
RUN pip install -r requirements.txt
COPY --from=builder /app/dist/*.whl .
RUN pip install *.whl