Logging
Log Management
ELK is the golden trio for container log collection, processing, and searching:
Logstash (or Fluentd) is responsible for log collection
Elasticsearch stores logs and provides search capabilities
Kibana handles log querying and visualization
Note: Kubernetes by default uses fluentd (launched as a DaemonSet) to collect logs and then sends them to elasticsearch.
Pro Tip
When deploying a cluster with cluster/kube-up.sh, you can set the KUBE_LOGGING_DESTINATION environment variable to automatically deploy Elasticsearch and Kibana and use fluentd to collect logs (see configuration at addons/fluentd-elasticsearch):
If you're using GCE or GKE, you can also send logs to Google Cloud Logging and integrate with Google Cloud Storage and BigQuery.
For other logging solutions, you can customize the docker log driver to send logs to splunk, awslogs, and more.
Deployment Method
Since the Fluentd daemonset is only scheduled to run on Nodes with the label beta.kubernetes.io/fluentd-ds-ready=true, you need to label the Nodes accordingly:
Then download the manifest and deploy:
Note: The Kibana container might take a while during the first startup to optimize and cache bundles (Optimizing and caching bundles for kibana and statusPage. This may take a few minutes). Monitor the initialization via logs:
Accessing Kibana
You can find the Kibana service access URL from the output of kubectl cluster-info. Note that you'll need to import the apiserver certificate into your browser for authentication:
Alternatively, use the kubectl proxy to access without needing to import the certificate:
Then open http://<master-ip>:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana#. In the Settings -> Indices page, create an index, select Index contains time-based events, use the default logstash-* pattern, and click Create.
Filebeat
In addition to Fluentd and Logstash, you can use Filebeat for log collection:
Note: The default setup assumes Elasticsearch is accessible via elasticsearch:9200. If it's different, modify the details before deployment:
Reference Documents
The Golden Trio for Log Mastery
ELK stands out as the go-to combination for handling the entire lifecycle of log data in container environments:
Logstash (or its alternative, Fluentd) takes on the role of log collection.
Elasticsearch acts as the data storehouse, providing robust search capabilities.
Kibana serves up the user interface, streamlining log queries and visual presentation.
Quick heads-up: Kubernetes has a default set-up with fluentd, running as a DaemonSet, to scoop up logs and shuttle them off to Elasticsearch.
Handy Hint
Deploying your cluster with cluster/kube-up.sh gets even better when you use the KUBE_LOGGING_DESTINATION environment variable. This nifty trick sets up Elasticsearch and Kibana on autopilot, with fluentd gathering the logs for you (dive into the settings at addons/fluentd-elasticsearch):
Google's Cloud ecosystem users (think GCE or GKE) can funnel logs straight to Google Cloud Logging and also bring Google Cloud Storage and BigQuery into the mix.
Feeling adventurous with logging solutions? Swap out the docker log driver to integrate services like Splunk or awslogs.
Setting the Scene for Deployment
Fluentd daemonsets have a VIP list, and only nodes wearing the beta.kubernetes.io/fluentd-ds-ready=true badge get the daemonsets' attention. So start by tagging your nodes:
Next up, grab the manifest and deploy to your heart's content:
(Expect the usual "configured" symphony for roles, bindings, services, and deployments.)
Just a note: The first time Kibana boots up, it's going to take its sweet time optimizing and caching (imagine it humming "Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"). You can keep an eye on the warm-up process via logs:
Peeking into Kibana
Kibana's door can be found in the kubectl cluster-info output. Just a reminder to sync up with the apiserver certificate in your browser for a smooth handshake:
Or, bypass the whole certificate song and dance by delegating to kubectl proxy:
Go ahead and launch http://<master-ip>:8080/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana#. When you land, set up a new index. Go with the flow—choose Index contains time-based events, stick with the logstash-* default, and hit Create.
Enter Filebeat
For those who march to a different beat, Filebeat from the Elastic family offers another avenue for gathering logs with gusto:
Bear in mind, it presumes you can waltz straight into Elasticsearch at elasticsearch:9200. If your access route is different, adjust the settings and then deploy:
(and so on, with the port, username, and password)
Treasure Trove of References
If you're itching to explore deeper, check out these documents:
最后更新于
