Managing the world through a keyhole
There are a few AWS services which seem to fall into the category of managing the world through a keyhole. I like to say, if I can see it, I can troubleshoot it. One example is AWS Fargate, a powerful container orchestration option of ECS (Amazon Elastic Container Service). With Fargate, you quickly realize that you have no console, no way to see what is going on realtime, basically running blind. However, with a bit ingenuity, this to can be overcome. To be honest, although frustrating at first, one of the most attractive things about AWS is that they allow you to combine any of their 100+ services any way you can dream up to engineer solutions to even the trickiest problems.
I am going to author a series of blog posts detailing tricky technical roadblocks, some seemingly insurmountable, I encountered and solved while architecting a large scale installation of a popular log and metric aggregation system using AWS Fargate, AWS CloudFormation, and now Terrafrom.
As for lack of visibility when using Fargate containers, here are a couple of tools I use:
- Cloudwatch Logs + AWS Container Insights
- Build a customized docker container setup to do reconnaissance. This was extremely valuable. Send output to STDOUT and pick it up from Cloudwatch Logs. This takes the guesswork out of the equation.