Combining AWS Monitoring Services

Overview Workflow

In a previous article, we discussed the differences and usage of various Monitor services in AWS. In this example, we will combine CloudTrail, EventBridge, CloudWatch, X-Ray, and SNS together around a simple Lambda-S3 workflow.

Step-by-Step Instructions

1. Create a CloudTrail Trail

1. Open CloudTrail.
2. In the sidebar, choose Trails and then Create trail.
3. Name: monitor-demo-trail.
4. Event Type: Data events and Management events.
5. Edit General Details:
   • Data Event Type: S3.
   • Create a new Bucket for storage.
6. Additional Settings:
   • Enable SNS notification delivery.
7. Event Types:
   • Management events and Data events: to record All object-level API events (or we can restrict to just Put events if available).
8. Under Management events, ensure that Read/Write events is set to All.
9. Under S3 Data Event: choose resource type as s3, and log all events.
10. Ok, Create.

2. Create an SNS Topic

1. Go to the SNS console. Choose Topics on the left sidebar, then Create topic.
2. Select Topic type: Standard.
3. Name the topic: demo-event-sns.
4. Create topic.
5. Then click Create subscription, choose Email as the protocol. Remember to go to the email and confirm the subscription.

3. Create an EventBridge Rule to Trigger SNS on S3 “PutObject” Events

1. Navigate to the EventBridge console.
2. Click Rules on the left sidebar, then Create rule.
3. Name: demo-s3-put-obj-eb.
4. Event Bus: Use the default event bus.
5. Rule State: Ensure it is Enabled.
6. Creation Method: Choose Custom pattern (or “Use pattern form” and then switch to the JSON editor).

7. Define Event Pattern:

   {
   "source": ["aws.s3"],
   "detail-type": ["AWS API Call via CloudTrail"],
   "detail": {
   "eventName": ["PutObject"]
   }
   }

   4. Package the Function

8. Zip the project folder contents (ensure that index.js and the node_modules/ directory are at the root of the zip).
9. Under the Code tab, upload my-lambda-function.zip file.

6. Test

1. Upload a File to S3:
   • CloudTrail logs the S3 PutObject event.
   • EventBridge picks up the CloudTrail event and sends an SNS notification.
   • The S3 event also triggers the Lambda function.
2. Verify CloudTrail:
   • In the CloudTrail console, find Event history on the left side panel.
   • Look for a recent event with "eventName": "PutObject". This confirms CloudTrail is recording management events.
3. Check SNS and Email:
   • In the SNS console, check the topic for the recent event.
   • In the email, check for the notification.
4. View CloudWatch Logs and Metrics:
   • In the CloudWatch console, check the Logs group for our Lambda function to see the console output.
   • In the Metrics section, review Lambda invocation metrics.
5. Inspect X-Ray Traces:
   • Open the X-Ray console.
   • In the Service map or Trace list, we should see traces from our Lambda function.
   • Drill into a trace to view details (such as the custom “s3-getObject” subsegment), which shows the latency and any errors.

Summary

We have three workflows:

• Workflow A: S3 Upload → CloudTrail → EventBridge → SNS → Email
• Workflow B: S3 Upload → Lambda → X-Ray
• Workflow C: S3 Upload → Lambda → CloudWatch (Logs/Metrics)

FAQs

Q: What is the purpose of CloudTrail?
A: CloudTrail records management (control-plane) events such as S3 “PutObject” events.

Q: What is the purpose of EventBridge?
A: EventBridge watches for a CloudTrail event (when someone uploads an object to S3) and triggers an SNS notification.

Q: What is the purpose of X-Ray?
A: X-Ray collects trace segments (and subsegments) that show the end-to-end flow of the request (for example, how long it takes for the Lambda to perform a GetObject call on S3 or make other downstream calls).

Q: What is the purpose of CloudWatch?
A: CloudWatch collects logs and metrics from our Lambda function and provides insights into its performance and behavior.