Logging Drivers and Debugging Containers

Knowing where logs go and finding out why a container misbehaves are the two skills that turn a Docker user into a Docker operator. This lesson covers logging drivers and rotation, how to ship logs elsewhere, and a repeatable procedure for debugging containers that crash, hang, or cannot reach the network.

How Container Logging Works

Docker captures the stdout and stderr of a container's main process and hands each line to a logging driver. The default, json-file, writes JSON lines under /var/lib/docker/containers/<id>/, and docker logs reads them back. Without limits that file grows forever, which is one of the most common causes of a full disk on a Docker host.

| Driver | Where logs go | docker logs works | |--------|---------------|---------------------| | json-file | Local JSON file (default) | Yes | | local | Local, compressed, rotated by default | Yes | | journald | systemd journal on the host | Yes | | syslog, gelf, fluentd | A remote collector | No | | awslogs, gcplogs | Cloud logging services | No | | none | Discarded | No |

Set rotation on the daemon so every container inherits it, then override per container only when needed. Changes in daemon.json require a daemon restart and apply to containers created afterwards. In Compose, the same options live under the service's logging: key:

yaml

services:
  api:
    image: my-api:1.0
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Shipping Logs Elsewhere

For more than one host, send logs to a central system. Docker can push directly with a driver, or a collector container can read the JSON files. The second approach keeps docker logs working and is what most stacks use: Promtail or Alloy for Grafana Loki, Filebeat for Elasticsearch, or Vector and Fluent Bit for anything. Whichever you choose, the application's job is unchanged: write structured lines (JSON is ideal) to stdout, one event per line, with the level and a timestamp.

bash

docker run -d --log-driver=fluentd --log-opt fluentd-address=localhost:24224 my-api:1.0
docker run -d --log-driver=journald my-api:1.0 && journalctl CONTAINER_NAME=api -f

Debugging Procedure: Container Exits Immediately

Work through these in order; each step narrows the cause:

docker ps -a shows the status and exit code, for example Exited (1) 3 seconds ago.
docker logs api almost always contains the reason: a missing environment variable, a port already in use, a file not found.
docker inspect --format '{{.State.Error}}' shows daemon-level failures such as exec: "node": executable file not found.
Exit code 0 with no logs means the process finished: a server that was never started (wrong CMD) or a shell that exited because no terminal was attached.
Bypass the entrypoint and look inside the image: docker run --rm -it --entrypoint sh my-api:1.0, then run the real command by hand and watch it fail with a full error.

bash

docker run --rm -it --entrypoint sh my-api:1.0
/app $ ls -la
/app $ node server.js
Error: Cannot find module '/app/dist/server.js'

docker events --since 10m shows the daemon's own timeline (create, start, die, oom) and is the quickest way to see whether the container is being killed by the OOM killer or by a restart policy loop.

Debugging a Running Container

When the container runs but misbehaves, combine the inspection commands from earlier lessons:

bash

docker exec -it api sh                       # look around
docker exec api env | sort                   # configuration it actually received
docker exec api getent hosts db              # can it resolve the database?
docker exec api wget -qO- http://db:5432 2>&1 | head -1   # can it reach it?
docker stats --no-stream api                 # CPU pinned? memory near the limit?
docker top api                               # unexpected child processes?

Minimal images (distroless, scratch) have no shell to exec into. Attach a debugging container to the same namespaces instead:

bash

docker run --rm -it --network container:api --pid container:api nicolaka/netshoot

netshoot ships curl, dig, tcpdump, ss and more; sharing the network namespace means localhost inside it is the application container, and sharing PID lets you see its processes. Docker Desktop also offers docker debug <container>, which attaches a toolbox shell to any container or image.

Common Mistakes

Running for months with the default json-file driver and no max-size, then losing the host to a full disk.
Logging to files inside the container where no driver and no docker logs can see them.

Key Takeaways

Docker captures stdout and stderr and hands them to a logging driver; json-file is the default and must be rotated.
Set max-size and max-file in daemon.json or Compose logging:; the local driver rotates by default.
Applications should write structured lines to stdout; a collector container ships them centrally without breaking docker logs.
For a crashing container: ps -a, logs, inspect .State.Error, then --entrypoint sh to reproduce by hand.
Use --network container: and --pid container: with a tool image to debug shell-less containers.

Next lesson: Image Optimization and Multi-Stage Builds — shrink images and speed up builds with staged Dockerfiles.