Menu

Docker in Production: Lessons from 50+ Deployments
May 12, 2025DevOps18 min read

Docker in Production: Lessons from 50+ Deployments

T
Tom Anderson

After deploying 50+ applications with Docker in production environments, here are the lessons that saved us from outages, security breaches, and 3 AM debugging sessions.

Lesson 1: Multi-Stage Builds Save Resources and Security

Our first Docker images were 2.4GB. Now they're 87MB. Here's how:

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Before: Single stage (2.4GB)
FROM node:18
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
# After: Multi-stage build (87MB)
# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production && npm cache clean --force

Benefits:

  • 96% smaller image size
  • No build tools in production image
  • Runs as non-root user
  • Faster deployments and scaling

Lesson 2: Health Checks Are Critical

Without proper health checks, Kubernetes kept routing traffic to broken containers.

text
1
2
3
4
5
6
# Dockerfile health check
HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \
CMD curl -f http://localhost:3000/health || exit 1
# Or for Node.js apps without curl
HEALTHCHECK --interval=30s --timeout=3s --start-period=40s --retries=3 \
CMD node health-check.js
text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// health-check.js
const http = require('http')
const options = {
hostname: 'localhost',
port: 3000,
path: '/health',
timeout: 2000
}
const req = http.request(options, (res) => {
if (res.statusCode === 200) {
process.exit(0)
} else {
process.exit(1)
}
})

Lesson 3: Secrets Management Done Right

Never put secrets in environment variables or Dockerfiles. Use Docker Secrets or external secret management.

text
1
2
3
4
5
6
7
8
# Bad: Secrets in environment variables
ENV DATABASE_PASSWORD=super_secret_password
# Good: Use Docker Secrets
docker service create \
--name myapp \
--secret db_password \
--env DATABASE_PASSWORD_FILE=/run/secrets/db_password \
myapp:latest
text
1
2
3
4
5
6
7
8
9
10
11
// Reading secrets in Node.js
const fs = require('fs')
function getSecret(secretName) {
try {
return fs.readFileSync(`/run/secrets/${secretName}`, 'utf8').trim()
} catch (error) {
// Fallback to environment variable for development
return process.env[secretName.toUpperCase()]
}
}
const dbPassword = getSecret('db_password')

Lesson 4: Resource Limits Prevent Cascading Failures

One container consumed all memory and killed our entire node. Solution: proper resource limits.

text
1
2
3
4
5
6
7
8
9
10
11
12
13
# Docker Compose
services:
app:
image: myapp:latest
deploy:
resources:
limits:
memory: 512M
cpus: '0.5'
reservations:
memory: 256M
cpus: '0.25'
restart: unless-stopped
text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# Kubernetes deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
template:
spec:
containers:
- name: myapp
image: myapp:latest
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"

Lesson 5: Logging Configuration

Default logging filled our disks. Configure log rotation and structured logging.

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# Docker daemon logging config
{
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
}
}
# Or in docker-compose.yml
services:
app:
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// Structured logging in Node.js
const winston = require('winston')
const logger = winston.createLogger({
level: process.env.LOG_LEVEL || 'info',
format: winston.format.combine(
winston.format.timestamp(),
winston.format.errors({ stack: true }),
winston.format.json()
),
transports: [
new winston.transports.Console()
]
})
// Usage
logger.info('User created', {
userId: user.id,
email: user.email,
ip: req.ip
})

Lesson 6: Security Scanning in CI/CD

We caught 23 critical vulnerabilities by scanning images before deployment.

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# GitHub Actions security scan
name: Docker Security Scan
on:
push:
branches: [main]
pull_request:
jobs:
security-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build Docker image
run: docker build -t myapp:${{ github.sha }} .
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master

Lesson 7: Monitoring and Observability

Essential metrics to monitor in production:

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
// Custom metrics endpoint
const promClient = require('prom-client')
const express = require('express')
// Create metrics
const httpRequestDuration = new promClient.Histogram({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'route', 'status']
})
const memoryUsage = new promClient.Gauge({
name: 'nodejs_memory_usage_bytes',
help: 'Memory usage in bytes',
collect() {
const memUsage = process.memoryUsage()
this.set({ type: 'rss' }, memUsage.rss)

Lesson 8: Blue-Green Deployments

Zero-downtime deployments with Docker Swarm:

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# deploy.sh
#!/bin/bash
set -e
NEW_VERSION=$1
SERVICE_NAME="myapp"
echo "Deploying $SERVICE_NAME version $NEW_VERSION"
# Update service with new image
docker service update \
--image myapp:$NEW_VERSION \
--update-parallelism 1 \
--update-delay 30s \
--update-failure-action rollback \
--update-monitor 60s \
$SERVICE_NAME
# Wait for deployment to complete

Production Dockerfile Template

Here's our battle-tested Dockerfile template:

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Multi-stage build for Node.js app
FROM node:18-alpine AS base
RUN apk add --no-cache curl
WORKDIR /app
# Dependencies stage
FROM base AS deps
COPY package*.json ./
RUN npm ci --only=production && npm cache clean --force
# Build stage
FROM base AS builder
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
# Production stage

Docker Compose for Development

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# docker-compose.yml
version: '3.8'
services:
app:
build: .
ports:
- "3000:3000"
environment:
- NODE_ENV=development
- DATABASE_URL=postgresql://user:pass@db:5432/myapp
volumes:
- .:/app
- /app/node_modules
depends_on:
db:

Key Takeaways

  • Use multi-stage builds to minimize image size and attack surface
  • Always implement health checks for proper orchestration
  • Never put secrets in images or environment variables
  • Set resource limits to prevent resource starvation
  • Configure log rotation to prevent disk space issues
  • Scan images for vulnerabilities in your CI/CD pipeline
  • Monitor container metrics and application performance
  • Use proper deployment strategies for zero-downtime updates

Docker in production requires attention to security, monitoring, and operational practices. Start with these patterns and adapt them to your specific needs.

Share this article