Streamlining Logging and Monitoring in Amazon EKS with Fluentd and CloudWatch Agent
Introduction:
In today’s dynamic cloud environments, efficient logging and monitoring are essential for maintaining visibility and ensuring the health and performance of applications running on Amazon EKS (Elastic Kubernetes Service). This blog explores how to set up Fluentd and CloudWatch Agent on an Amazon EKS cluster to seamlessly stream logs, metrics, and container insights to Amazon CloudWatch for centralized monitoring and analysis.
Understanding Fluentd and CloudWatch Agent:
Fluentd is an open-source data collector that enables the unified logging layer by collecting, processing, and forwarding logs to various destinations. On the other hand, CloudWatch Agent is an Amazon-specific tool designed for easy integration with Amazon CloudWatch, allowing seamless collection and forwarding of metrics and logs from Amazon EKS clusters.
Why Use Both Fluentd and CloudWatch Agent?:
- Fluentd: Provides flexibility and customization for handling diverse log sources and formats within the EKS cluster.
- CloudWatch Agent: Offers native integration with Amazon CloudWatch, simplifying the setup and configuration for sending metrics and logs directly to CloudWatch.
Setting Up Fluentd and CloudWatch Agent on Amazon EKS:
1. Pre-requisites:
Ensure you have an Amazon EKS cluster set up and configured in your AWS account.
eksctl create cluster --name my-cluster --region us-west-1 --with-oidc --nodegroup-name ng-1 --nodes 2 --nodes-max 5 --nodes-min 2 --ssh-access --enable-ssm --node-private-networking --managed --instance-types t3.medium --asg-access --external-dns-access --full-ecr-access --version 1.25 
2. Installing Fluentd:
- Deploy Fluentd DaemonSet: Create a Fluentd DaemonSet to run Fluentd as a sidecar container on each node in the EKS cluster.
- Configure Fluentd Inputs and Outputs: Customize Fluentd configuration to collect logs from Kubernetes pods and forward them to CloudWatch Logs.
3. Installing CloudWatch Agent:
- Download CloudWatch Agent: Retrieve the CloudWatch Agent installer script from the AWS documentation.
- Configure CloudWatch Agent: Set up CloudWatch Agent to collect system metrics, application logs, and container insights from the EKS cluster.
The command you need to run for executing above two steps is:
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/quickstart/cwagent-fluentd-quickstart.yaml | sed "s/{{cluster_name}}/my-cluster/;s/{{region_name}}/us-west-1/" > EnableContainerInsights.yaml- Replace
my-clusterandus-west-1with your own cluster name and aws region.
kubectl apply -f EnableContainerInsights.yaml4. Verification and Testing:
- Validate Fluentd and CloudWatch Agent Deployment: Check the status of Fluentd DaemonSet and CloudWatch Agent installation to ensure successful deployment.
- Monitor CloudWatch Metrics and Logs: Use the CloudWatch console to monitor metrics and logs streamed from the EKS cluster, including container insights.
Benefits and Use Cases:
- Centralized Monitoring: Aggregate logs and metrics from EKS pods and nodes into a centralized location for easier monitoring and troubleshooting.
- Real-time Insights: Gain real-time visibility into application performance, resource utilization, and container health.
- Scalability and Flexibility: Scale logging and monitoring capabilities dynamically as the EKS cluster grows, adapting to changing workload demands.
Conclusion:
By leveraging Fluentd and CloudWatch Agent in conjunction with Amazon EKS, organizations can achieve comprehensive logging and monitoring capabilities that enable better visibility, troubleshooting, and optimization of Kubernetes-based applications. With the ability to stream logs, metrics, and container insights directly to Amazon CloudWatch, teams can effectively monitor and manage their EKS workloads with ease.
