Step-by-Step Guide To Using AWS CloudWatch Agent For Log Monitoring

AWS CloudWatch Logs is a powerful service that allows you to monitor, store, and access log files from various AWS resources. It plays a crucial role in maintaining the health and performance of your applications. In this blog, we’ll explore the key components of AWS CloudWatch Logs and provide a step-by-step guide to setting up log monitoring for an Apache web server.

Understanding AWS CloudWatch Logs

AWS CloudWatch Logs enables you to collect and monitor log data, helping you gain insights into your system’s operation and performance. It allows you to centralize logs from your applications, systems, and AWS services, making it easier to troubleshoot and debug issues.

Key Features of CloudWatch Logs:

• Log Collection: Aggregate logs from multiple sources into a single service.
• Log Monitoring: Set up alarms and triggers based on specific log patterns.
• Log Storage: Store logs for compliance and auditing purposes.
• Log Insights: Query and analyze log data to gain actionable insights.

Practical Steps to Set Up CloudWatch Logs for Apache Web Server With CloudWatch Agent

Step 1: Install and Configure Apache Web Server

First, install the Apache web server on your EC2 instance:

sudo yum install httpd -y
cd /var/www/html 

Create index file in DocumentRoot /var/www/html

Ensure that Apache is running:

sudo systemctl start httpd 
sudo systemctl enable httpd --now

Now hit to the server on your browser so that the server will generate logs.

Step 2: Configure Log File Locations

By default, Apache logs are stored in /var/log/httpd/access.log and /var/log/httpd/error.log. You can verify this in the Apache configuration file located at /etc/httpd/conf/httpd.conf or in individual virtual host files.

Step 3: Install and Configure CloudWatch Agent

To push logs to CloudWatch, install the CloudWatch agent on your EC2 instance:

sudo yum install amazon-cloudwatch-agent -y

Next, create a CloudWatch agent configuration file to specify which logs to collect. Store this configuration file in AWS Systems Manager (SSM) Parameter Store for easy retrieval.

Step 4: Store Configuration in SSM Parameter Store

Save the following CloudWatch agent configuration file as cloudwatch-config.json:

{
  "agent": {
           "metrics_collection_interval": 1,
           "run_as_user": "root"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/httpd/access_log",
            "log_group_class": "STANDARD",
            "log_group_name": "webserver_access_logs",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 3
          },
          {
            "file_path": "/var/log/httpd/error_log",
            "log_group_class": "STANDARD",
            "log_group_name": "webserver-error-logs",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 3
          }
        ]
      }
    }
  }
}

Store the configuration in SSM Parameter Store:

• For storing configuration this configuration file in ssm parameter, either use below command or manually go to SSM Console and create a parameter store.

aws ssm put-parameter --name "AmazonCloudWatch-logs" --type "String" --value file://cloudwatch-config.json

Step 5: Creating IAM Role

Before starting the Agent, we need to create a role that will allow ec2 to go and put metric data in the cloudwatch. This role will also perform get and put actions on ssm parameter store.

And attach this role to your ec2-instances.

Step 6: Start the CloudWatch Agent

Retrieve and start the CloudWatch agent using the configuration stored in SSM:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-logs -s

Let’s do the same Setup in other instances where apache server is installed:

Now we don’t have to create configuration file again and again. We will use our pre-configured file that is uploaded on SSM parameter store.

Hence by leveraging SSM Parameter Store, our cloudwatch agent setup process now reduced to just 2 steps. This will also help us in our automation game.

Step 7: Verify Logs in CloudWatch

Log in to the AWS Management Console, navigate to CloudWatch, and check the log groups. You should see webserver_access_logs and webserver-error-logs with log streams containing your Apache log data.

ACCESS_LOGS:

ERROR_LOGS

Conclusion

In this blog, we covered the essentials of AWS CloudWatch Logs, including setting up Apache web server log monitoring. By leveraging CloudWatch Logs, you can centralize your log data, gain valuable insights, and maintain the health of your applications effectively.

Key Takeaways:

• Centralized log collection and storage.
• Enhanced monitoring and troubleshooting capabilities.
• Configuration management with SSM Parameter Store.
• Real-time log analysis and insights.

Implementing CloudWatch Logs for your applications ensures that you have robust monitoring and logging in place, making it easier to maintain and troubleshoot your AWS infrastructure.