Container Logging

Logging is an essential part of the system to identify any system/application issues and even for audit purposes. But, as the Docker container transiently exists, potentially the log files in the container will be removed together with the container itself. So we will need to store the log file in a safe place, and it will be usually integrated with other logging solution providers.

Logging Drivers (Logging tool is NOT a driver)

• none: no log for the container
• local: log stored locally
• json-file: log files stored in json format. (by default)
• syslog: Docker logs will be stored using syslogs.
• journald
• gelf: Send logs to Graylog Extended Log Format (Graylog, Logstash)
• fluentd
• awslogs: log sent to AWS Cloudwatch
• splunk
• etwlogs: Sends logs to Event Tracing for Windows, only applicable to windows host.
• gcplogs: Google Cloud Logging
• logentries: Rapid7 Logentries.

Mistakes in logging for the docker container

• Sometimes the engineers treat the containers as a virtual machine so, in the same logic, they misunderstand the virtual machine as the bare metal servers, which will be persistent after shutting down servers. But you should remember the virtual machine running your containers will be replaced anytime and will be not keepable forever.
• You can’t expect the log files to be in the container for a long time even in the virtual machine, some are doing the logging transfer like batch process once a day, but you can’t do it if you really want to keep logs safely. You should push the logs to the logging solutions right after getting them in.
• Don’t do couple container logs with containers, it should be separately managed and assumed the container will be deleted anytime, especially if you are using orchestrator tools like K8s, you even can’t guarantee your container will be in which node.

Delivery mode of log messages from container to log driver

Docker provides two modes for delivering messages from the container to the log driver.

• Blocking, delivery from container to driver: This is the default mode in docker, and the application will be interrupted every time it delivers a message to the driver. This will ensure all messages will be sent to the driver but will impact the application performance. Writing logs wouldn’t block the application as it’s in the same local file system but when it needs to open a connection to the other logging system, there will be a potential latency issue coming.
• non-blocking delivery that stores log messages in an intermediate per-container ring buffer for consumption by the driver: This mode is that the container will write logs in-memory (ring buffer), so this mode won’t impact the application performance even though there is a high volume of logs. But in order words, it can’t guarantee that all messages will get to the driver instead there will be potential loss where it is being sent. (*if the buffer is full)

What is the best combination?

What I can suggest is that you configure the json-file logging driver with the blocking mode. This may be the most common setting but it’s still the safest combination as in many scenarios, the log files are not loseable because of compliance, security, troubleshooting, and audit purposes. Of course, it will impact the performance of the application, but you should consider the hardware specs for the application including the logging capability (capacity)

Monitoring Tools for Docker

• Sematext
• Dynatrace
• Datadog
• Prometheus & Grafana
• Elasticsearch & Kibana
• SolarWinds Server & Application Monitor
• AppOptics Docker Monitoring with APM
• cAdvisor
• Sysdig
• ManageEngine Applications Manager
• Sumo Logic
• Splunk