Automating the AWS CloudWatch logs retention period for all log groups

CloudWatch logs cost a lot if you keep them forever. You have option of dumping them to S3 for further use. Still it may the case you don’t want to keep logs after certain amount of time. You can achieve it by individually changing the retention period of each log groups.

Workaround

You can use EventBrige Rules -> Lambda Setup to automate this process.

Steps:

1. Create a Lambda function with Python Run time. Add the following code in the code editor

import json
import logging

import boto3

logger = logging.getLogger()
logger.setLevel(logging.INFO)


def lambda_handler(event, context):
    client = boto3.client("logs")
    paginator = client.get_paginator("describe_log_groups")
    response_iterator = paginator.paginate()
    for page in response_iterator:
        for objects in page["logGroups"]:
            # print(objects)
            logs = objects["logGroupName"]
            retentionInDays = (
                objects["retentionInDays"] if "retentionInDays" in objects else "NA"
            )
            update_status = None
            if str(retentionInDays) not in ["3653"]:
                response = client.put_retention_policy(
                    logGroupName=logs, retentionInDays=3653
                )
                if response["ResponseMetadata"]["HTTPStatusCode"] == 200:
                    update_status = "Success"
                else:
                    update_status = "Failed"
                logger.info(
                    f"log group: {logs} old retention: {retentionInDays} response: {update_status}"
                )
            else:
                logger.info(f"log group: {logs}  retention: {retentionInDays}")
    # TODO implement
    return {"statusCode": 200, "body": json.dumps("Hello from Lambda!")}

2. Add arn:aws:iam::aws:policy/CloudWatchFullAccess Policy to your Lambda Execution Role

3. Add an Event Bridge trigger using AWS Lambda Console with Schedule expression: cron(0 18 ? * FRI *) and Event bus: default

This will trigger lambda every Friday at 6pm.

Limitations

The above code is region specific. If you want it to be updating all regions in your account then:

1. create a string array of all regions and pass it to boto3.client(‘logs’)

from boto3.session import Session
s = Session()
logs_regions = s.get_available_regions('logs')

1. Create a loop over the function as well as the string array and pass the region to the boto3 so it will able to update the retention period in all regions