The Complete Guide to Docker Logging: An In-Depth Handbook for Container Logging Best Practices

Hi friend! Docker logging can seem deceptively simple at first, but solid logging is crucial for monitoring and troubleshooting containerized apps. In this handbook, I‘ll arm you with in-depth knowledge to level up your Docker logging skills.

Together, we‘ll learn:

Key logging concepts like stdout/stderr, JSON formats, storage drivers
How to access, filter, and manage logs with docker logs
Advanced techniques like log routing, structured logging, and analytics
How logging fits into overall monitoring and observability
Best practices for production-grade Docker logging

Let‘s get started!

An Introduction to Docker Logging Concepts

Docker captures stdout and stderr streams from containers and saves them in JSON-formatted log files. By default, this uses the json-file logging driver which stores logs in the host filesystem:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

This keeps container logs separated from the host logs under /var/log.

Capturing stdout vs stderr

Stdout is the standard output stream used by apps for regular logs.

Stderr logs error messages and diagnostics. They can provide crucial insights during crashes or boot failures.

JSON formatted logs

Docker JSON logs embed metadata like timestamps along with the log message.

Example log message:

{"log":"Listening on port 3000","stream":"stdout","time":"2019-07-03T23:30:22.936662772Z"}

This JSON format enables structured logging with searchable fields.

External logging services often require serialized log events instead of just raw text. JSON meets that need out of the box.

Storage drivers and plugins

Besides json-file, Docker supports pluggable logging drivers for forwarding logs external pipelines and analytics tools instead of local filesystem storage.

Some popular storage drivers include:

fluentd – Send logs to Fluentd log collector
splunk – Stream logs to Splunk
awslogs – Ship logs to CloudWatch
gcplogs – Integrate with Google Cloud logging
syslog – Forward to syslog collector

We‘ll see how to configure these drivers later. They enable advanced logging and monitoring setups.

Now that we‘ve covered some core concepts, let‘s see Docker‘s logging in action!

Accessing and Viewing Docker Logs

The docker logs command fetches stdout and stderr streams for a container.

For example, to access logs of a container named my_app:

$ docker logs my_app

This will print logs starting from the container creation.

You can also lookup logs using the container ID instead of name:

$ docker logs 38bef696cd5a

Some key options:

--tail N – Prints last N log lines
--follow – Continuously stream new logs (like tail -f)
--until TIME – Print logs before specified time
--since TIME – Print logs since specified time

These allow quick filtering without having to move files around!

For example, get logs from the last 5 minutes:

$ docker logs --since 5m my_app

You can directly access the JSON log files as well, but docker logs gives structured formatting out of the box.

Live Streaming Logs with Follow

The --follow or -f option streams real-time logs just like tail -f.

For example:

$ docker logs -f my_app

This continuously prints new logs generated until you hit Ctrl+C.

Live streaming with --follow is immensely helpful for monitoring deploys, incidents, or testing workflows.

Pro tip: You can combine -f with filters like --since to pinpoint issues:

$ docker logs -f --since=‘5 mins ago‘ my_app

This prints live logs in the 5 minute window where some problem may have started. Powerful stuff!

Finding Recent Logs with –tail

Often you don‘t need the full log history. The --tail option prints only the last N lines.

For example, to see only the latest 5 log items:

$ docker logs --tail=5 my_app

This avoids cluttering the output with irrelevant old logs.

--tail also works in combination with other filters like --since:

$ docker logs --tail=50 --since=2h my_app

Print last 50 logs from the past 2 hours.

The default is to print all logs, so --tail=5 and --tail 5 are equivalent.

Filtering Logs by Time

Two of the most used log filtering options are:

--since TIME – Prints logs generated after the given time
--until TIME – Prints logs before specified time

The time formats accepted are:

Absolute timestamps – e.g. 2018-01-02T13:23:37Z
Relative time offsets – e.g. 5m for 5 minutes ago

For example:

# Fetch logs from last 5 minutes
$ docker logs --since 5m my_app   

# Get logs before August 2020
$ docker logs --until 2020-08-01T00:00:00Z my_app

This allows slicing and dicing logs by time range super easily.

You can also combine time filters with other options like --tail and --follow for advanced analysis:

$ docker logs -f --tail=20 --since=2h my_app

Continuously stream last 20 logs from past 2 hours. This is incredibly helpful during incidents and outages!

More with –details and –timestamps

A couple more handy docker logs options:

--details – Shows extra metadata like log source
--timestamps – Prefixes each line with a timestamp

For example, --details adds the log source on each line:

2019-07-03T23:30:22.936662772Z stdout Listening on port 3000
2019-07-03T23:31:22.936662778Z stderr App crashed with signal 11

And --timestamps annotates every log:

2019-07-03T23:30:22.936662772Z Listening on port 3000
2019-07-03T23:31:22.936662778Z App crashed with signal 11

These provide extra context beyond just the message text.

Finding Relevant Logs with grep, sed, awk

The standard Linux text processing tools like grep, sed, awk etc. work nicely with Docker logs.

You can pipe docker logs into them for advanced analysis:

$ docker logs my_app | grep "error"

This prints logs containing "error".

Or use sed to filter logs:

$ docker logs my_app | sed -n ‘s/.*error.*/\0/p‘

Extract only lines with "error" text.

Linux pipes let you leverage the full power of the shell for searching and slicing logs!

Configuring Docker Logging Drivers

The default json-file driver just stores local logs.

For production, you‘ll want to enable more scalable logging pipelines.

Docker‘s logging plugins enable streaming logs to various storage platforms and analytics tools.

For example, to forward logs to Splunk:

$ docker run --log-driver=splunk my_app

Some other popular drivers:

fluentd: Send logs to Fluentd log collector
syslog: Forward to syslog service
gelf: Log to Graylog log management
awslogs: Ship logs to CloudWatch
gcplogs: Integrate with Google Cloud logging

This keeps logs centralized instead of scattered across hosts.

Refer to the Docker logging driver docs for more details on setup and configuration.

Now let‘s explore some best practices around structuring logs for easier analysis.

Structured Logging with Logstash, rsyslog, etc.

Raw text logs have limitations. It‘s harder to query and graph metrics and events effectively.

Structured logging formats each log event as a rich JSON object. For example:

{
  "time": "2019-07-07T12:34:56Z",
  "severity": "error",
  "service": "payment",
  "stack_trace": "...."
}

This models logs as discrete events with searchable fields instead of just a text blob.

Tools like Logstash and rsyslog have formats to handle structured logs.

For example, here is an rsyslog template for Docker logs:

template(name="json-template" 
  type="list") {    
  constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="time") 
      constant(value="\"\",\"message\":\"")
      property(name="log")    
      constant(value="\"\",\"stream\":\"")
      property(name="stream")
      constant(value="\",\"docker\":{\"id\":\"")
      property(name="$!docker_container_id")
      constant(value="\",\"name\":\"")
      property(name="$!docker_container_name")
      constant(value="\"}")
  constant(value="}\n")
}

This renders Docker logs in a consistent JSON structure.

Structured logging requires more upfront design but enables much easier long-term management and insights.

Log Routing with Logspout, Fluentd, Filebeat

Log routing involves shipping container logs to external storage and analytics services. This is crucial for production-grade logging.

Popular data pipelines like Fluentd, Logspout and Beats (Filebeat) are purpose-built for routing logs.

For example, Logspout can collect Docker logs and forward them to logstash, syslog, etc:

$ logspout syslog://logstash.example.com:5000

Filebeat similarly buffers and ships logs to Elasticsearch, Splunk, etc.

Centralizing logs is critical for analysis and avoids data gaps.

Monitoring and Alerting with ELK, Splunk, Datadog

Logging provides only one part of observability.

For holistic monitoring, solutions like the ELK stack, Splunk and Datadog give you:

Collection: They integrate with Docker via logging drivers
Search: Enable searching logs for failures, traffic spikes etc.
Alerting: Set up real-time alerts for PagerDuty, Slack etc.
Visualization: Graph trends and metrics via logs

For example, Datadog can graph application request latencies over time from Docker logs:

Datadog dashboard

Modern log management platforms bring tons of extra visibility. They are worth investing in.

This covers the major capabilities you need for robust Docker logging in production. Now let‘s wrap up with some key best practices.

Docker Logging Best Practices

Here are my recommended best practices for leveling up your Docker logging:

Store logs outside containers – Avoid in-container disk logs. Use external drivers like fluentd instead. This avoids losing logs on crashes or restarts.

Log to stdout/stderr – Route app logs to stdout/stderr streams. Docker captures them automatically unlike custom log files.

Use JSON formatting – Output logs in JSON format with metadata. This enables structured logging without lengthy parsing.

Ship logs centrally – Aggregate logs in a centralized pipeline with tools like fluentd. This avoids data silos.

Add context metadata – Enrich logs with contextual metadata like host, app details etc. Makes querying and filtering easier.

Monitor with analytics tools – Analyze trends, set alerts and create dashboards with tools like ELK stack, Datadog etc.

Finally, don‘t just log – act on the insights gleaned from logs! Logging achieves its true purpose when tied into workflows for incident response, capacity planning and product improvement.

Conclusion

And that‘s a wrap! We covered a ton of ground on how Docker logging works under the hood, viewing logs via docker logs, filtering by time and patterns, structuring logs, routing to external services, monitoring analytics, and production best practices.

Logging may seem like an afterthought, but truly effective logging requires foresight in design and architecture. With these skills, you can troubleshoot issues faster and build a robust observability foundation.

Now over to you – go forth and log like a pro! Let me know if you have any other Docker logging techniques. Happy to learn together.