When and how to search with Amazon CloudWatch Logs – TechTarget

When and how to search with Amazon CloudWatch Logs – TechTarget

Getty Images
Amazon CloudWatch Logs delivers storage, processing, analysis and monitoring for a wide variety of log data from multiple sources — such as other AWS services. The service offers search functionality and integration capabilities with more sophisticated mechanisms for more detailed log analysis, at any scale.
Developers can configure their applications to export log data to CloudWatch Logs instead of keeping it in their own servers or EC2 instances. Once log data arrives to CloudWatch Logs, developers have the option to configure a retention period, export it to Amazon S3 or trigger custom actions on incoming log events. There is also the option to search and analyze log data directly in the CloudWatch Logs service.
Follow this step-by-step CloudWatch Logs tutorial, and explore other tools to work with log data.
There are three main CloudWatch Logs concepts users need to know.
Log event. When a monitored app or resource records an activity, it creates a log event. The event receives a timestamp and the raw event message.
Log groups. A log group is the top-level grouping related to log data.
Log streams. Streams are a way to group incoming log events within a log group.
For example, AWS sends log events related to a Lambda function into a log group dedicated to that particular function. Events are broken down into log streams that contain log events for specific executions.
For custom logs, such as Apache access logs generated in EC2 instances, it is a common pattern to create a log group for the application and log type — i.e., access and error — and then one log stream per EC2 instance. When developers configure custom application logs, they can specify how to group log events into log groups and log streams. For AWS built-in logging, the granularity of log groups and log streams are defined by each AWS service that sends data to CloudWatch Logs.
AWS has many options to execute searches on log data. The simplest one is through the CloudWatch Logs console.
Step 1. Select the applicable log group. There’s an option to select a particular log stream or to search all events within the whole log group.
When selecting a particular log group in the log groups screen, the console offers the menu shown in Figure 1.
Step 2. After clicking Search log group, the console displays Figure 2, which is also displayed when users click on a particular log stream.
This gives users the ability to choose a relative time range or to select a custom time range with specific start and end timestamps. This functionality executes a text-based search in the selected log data which allows for a simple way to find specific text at the log group or log stream level.
This option supports filter patterns in the search box, such as the following examples:
For space-delimited logs, such as Apache access logs, search filters can parse and filter specific fields. The following is a sample record in an Apache access log for a failed health check from Application Load Balancer (ALB):
The fields in this example can be parsed using the following pattern:
Records that failed ALB health checks could be searched using the following filter, which would return log events with an “ELB-HealthChecker” user_agent and status_code with value 500:
For automation purposes, developers can also use the AWS SDK or CLI to search CloudWatch Logs. The filter-log-events CLI command can be used in the following way:
The CLI allows for other fields, such as end-time, log-stream-name, log-stream-name-prefix and max-items for further refinement of search parameters.
CloudWatch Logs Insights is also a popular option to search logs. It offers a query syntax language that can be used to filter, parse and aggregate log records within a given time period. This tool allows for a more detailed search functionality compared to the standard text search.
Since log data can also be exported to S3, this allows for detailed analysis using Athena or other data analytics platforms available through AWS EMR. This is a feasible option for applications that produce a large volume of log data or require long-term log data retention. To analyze log data in S3 using tools such as Athena, ensure log records have a consistent pattern that can be parsed into table columns that will be used by Data Definition Language statements required by data analytics tools.
Subscription Filters is another useful feature that can be used to transform and export log data into multiple services and execute further search and analytics processes. Target services include the following:
If Congress approves the CHIPS Act, it will funnel billions into domestic chip production and provide companies like Intel and …
Blockchain is most famous for its cryptocurrency applications, but data centers can employ it for a variety of business-related …
Nvidia’s QODA platform bridges the chasm between quantum and classical environments. It could set the stage for quantum …
Patches are necessary for maintenance, but can often cause a rift between IT ops and security teams. Follow these best practices …
Discussions in the tech industry about how to overcome cloud security challenges for SBOMs include an early-stage CNCF idea that …
Docker and Podman offer similar capabilities to manage containers, but Docker’s security vulnerabilities might make Podman more …
Many organizations struggle to manage their vast collection of AWS accounts, but Control Tower can help. The service automates …
There are several important variables within the Amazon EKS pricing model. Dig into the numbers to ensure you deploy the service …
AWS users face a choice when deploying Kubernetes: run it themselves on EC2 or let Amazon do the heavy lifting with EKS. See …
Learn how container management products from VMware, Red Hat and Hewlett Packard Enterprise compare when it comes to their …
Submit your entry for the Best of VMware Explore 2022 Awards for a chance to win.
Read the updated rules, criteria and categories for the Best of VMware Explore 2022 Awards to find out what the requirements are …
All Rights Reserved, Copyright 2010 – 2022, TechTarget

Privacy Policy
Cookie Preferences
Do Not Sell My Personal Info


How do I find SEO services
Dominate search engine result pages, crush your competition, get more sales, and be the #1 provider in your service area. Work with Top SEO Agency in Dublin that also provides quality Digital Marketing Optimisation services. 
Our Search Sngine Optimisation Consultants will improve your site performance in all major search engines by implementing the Best SEO Solutions, Strategies and Techniques. Let us help you increase organic traffic, get more leads, more customers, and grow your revenue with a customised affordable SEO package.
Fully Managed WordPress Hosting


Get in touch

Globe Boss

Hampton Square
Dublin - Ireland
Tel: (+353)1 868 2345