Open Source Cloud Native Data Observability

With less than 48 hours until FluentCon 2021 Europe, I’m so excited to see lots of people signing up for my talk on May 4th: Fluent Bit - Swiss Army Tool of Observability Data Ingestion - Michael Marshall Prior to FluentCon, I wanted to announce an open source project I just released in hope that it would be of use to the wider community: https://github.com/neiman-marcus/fluent-bit-data-observability-platform In short, this project is a Docker-Compose based, miniature version of it’s much larger production equivalent we host in the cloud today. Continue reading

Observability Log Derived Metrics - Game Changer

Instrumenting cloud native applications for data observability can be challenging. Regardless of which cloud native design pattern you are using for your application, you likely have logs. We all have logs. Logs aid in troubleshooting, application management, and keeping track of what an application is doing. However, too many logs can quickly become unmanageable. To tame the logs and gain additional valuable insight, consider pushing them into a Data Observability Platform for monitoring the health of your application. Continue reading

How deploying AWS Lambda with Terraform is a Win-Win

In this blog post I will demonstrate how selecting the best tool for the job in the end can pay off resulting in both reduced complexity and lower costs. I was recently tasked with exporting CloudWatch logs from multiple AWS accounts into Grafana Loki, hosted in a central logging AWS account. My architecture leverages the event driven design pattern of subscribing a log sender Lambda function to a set of CloudWatch log groups. Continue reading

Terraform Learnings

A key feature of using Terraform is idempotency of Infrastructure as Code. Use of depends_on pointed at a data source can result resources being redeployed. Today I solved a problem where this principal was a bit less intuitive, had the same effect. The module I was using leveraged a null_resource to execute a local command when a config file changed. The triggers block referenced a external data source to calculate the md5 checksum of the config file. Continue reading

Managing the world through a keyhole

There are a few AWS services which seem to fall into the category of managing the world through a keyhole. I like to say, if I can see it, I can troubleshoot it. One example is AWS Fargate, a powerful container orchestration option of ECS (Amazon Elastic Container Service). With Fargate, you quickly realize that you have no console, no way to see what is going on realtime, basically running blind. Continue reading

CICD Blog Post

This content is being deployed on repo push! And here’s my AWS architecture:

My First Post

Launching my blog! AWS codebuild automatic updates. 1