Detect Infinite Loops in Amazon S3 Access Logging
Utilizing Python to catch misconfigured S3 logging.
Amazon S3 buckets have a feature called server access logging that provides details of the requests that are made to the bucket. This is useful for security audits and to analyze access patterns. Server access logging is enabled at the bucket level and can be done in the AWS Console:
A destination S3 bucket is chosen and all object-level operations are logged to the destination:
When enabling S3 access logging, it is important to not create an infinite loop. This occurs if an S3 bucket is logging to itself, or if BucketA is logging to BucketB, and BucketB is logging back to BucketA. This creates an endless stream of log events. Imagine if an object is written to BucketA. A log is then written to BucketB, which in turn generates another log in BucketA, and so on..
I’ve seen quite a few customers in this situation. It creates a mess of the logs and unnecessary API and storage costs. I’ve written a simple Python script to detect infinite loops with S3 access logging. The script uses the networkx Python package to create a directed graph and detect cycles. networkx uses Johnson’s algorithm under the hood. You can run the script as follows:
❯ pip install -r requirements.txt
❯ python s3_logs_detect_infinite_loops.py
--- LOGGING CONFIGURATIONS ---
BucketA --> BucketA
BucketB --> BucketC
BucketC --> BucketB
--- INFINITE LOOPS DETECTED ---
['BucketA']
['BucketB', 'BucketC']In addition to S3 access logging, you can use AWS CloudTrail to achieve a similar result, but there are important caveats to consider. For example, S3 access logging captures Object Size, Total Time, Turn-Around Time, HTTP Referer, and authentication failures, whereas these are not logged by AWS CloudTrail. I prefer AWS CloudTrail because you can enable it once at the account level for all current and future S3 buckets.
If anyone from AWS is reading this, feel free to steal the code and use it to generate a warning message in the AWS Console.
