Ingest AWS CloudTrail through Cloudwatch & Logstash into Elasticsearch

log flow

Problem statement

  1. Field explosion (way to many fields)
  2. An explosion of indices (oversharding as a result)

Out of the box solution

The solution

Completing the setup

  • Select a loggroup
  • Actions -> Subscription filters -> Create Lambda subscription filter
  • Select the Lambda function you created earlier
  • Add a subscription filter name
  • “Start streaming”

Victory

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

The Developers’ Burn Out Is Real. Here is how you can prevent it.

SubQuery Extends Invitation To Indexing Community

Dshell; build console apps in dart — part 2

Hack Reactor Week 3

If I Could Only Learn 6 Programming Languages in My Life, I’d Learn These

Leetcode — Car Fleet II

Terraform-tool over which the whole software industry going gaga!

Value at Risk, with Python

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Stijn Holzhauer

Stijn Holzhauer

More from Medium

SAP Monitoring using AWS Data Provider for SAP

AWS S3 Bucket and IAM User

Hands-On Approach to S3 Replication

AWS S3 Bucket and IAM User