Store Loki Logs in an S3 Bucket with Kube-Prometheus-Stack

Storing logs effectively is crucial for monitoring, troubleshooting, and ensuring the smooth operation of applications. Loki, a horizontally-scalable, highly-available log aggregation system inspired by Prometheus, offers an efficient way to collect and query logs.

However, storing these logs in a persistent and scalable manner can be a challenge. This is where AWS S3 comes in, providing a durable, scalable, and cost-effective storage solution. In this guide, we’ll walk you through the process of configuring Loki to store logs in an S3 bucket.

Why Store Logs in an S3 Bucket?

Storing logs in an S3 bucket provides several benefits:

Scalability: S3 can handle virtually unlimited amounts of data, making it ideal for storing large volumes of logs.
Durability: S3 is designed for 99.999999999% durability, ensuring that your logs are safe.
Cost-Effectiveness: S3 offers a pay-as-you-go pricing model, which can be more economical for long-term log storage compared to traditional storage solutions.
Integration: S3 integrates well with various AWS services and other tools, providing flexibility in how you manage and analyze your logs.

Prerequisites

Before you begin, ensure you have the following:

Kube-Prometheus-Stack Installed: This stack includes Prometheus, Grafana, and other monitoring tools. If you do not have it installed, you can install it from here Link.
AWS S3 Bucket: An S3 bucket where logs will be stored.

Step 1: Create IAM Role and Attach Customer Managed Policy

To allow Loki to write logs to your S3 bucket, you’ll need to create an IAM role with the necessary permissions.

Create a new IAM role:

Go to the IAM section in the AWS Management Console.
Click on “Roles” and then “Create role”.
Select “Custom trust policy” and then paste following trust policy.

				
					{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Principal": {
                "Service": "s3.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

4. Click “Next: Permissions”.

5. Create and attach a customer managed policy:

6. Click “Create policy”.

Choose the “JSON” tab and enter the following policy, replacing YOUR_S3_BUCKET_NAME with the name of your S3 bucket:

				
					{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::YOUR_S3_BUCKET_NAME/*"
   }
  ]
}

Click “Review policy”, give it a name, and then create the policy.
Attach this policy to the IAM role you created.

Step 2: Setup Loki-Distributed Values and Install Using Helm

Next, you’ll configure Loki to use your S3 bucket for log storage and deploy it using Helm.

Create a values.yaml file for Loki with the following configuration, replacing placeholders with your specific values: YOUR_S3_BUCKET_NAME,YOUR_AWS_REGION,IAM-ROLE-ARN

				
					USER-SUPPLIED VALUES:
loki:
  structuredConfig:
    ingester:
      chunk_idle_period: 1h
      chunk_target_size: 1536000
      max_chunk_age: 1h
      max_transfer_retries: 0
    schema_config:
      configs:
      - from: "2020-09-07"
        index:
          period: 24h
          prefix: loki_index_
        object_store: aws
        schema: v11
        store: boltdb-shipper
    storage_config:
      aws:
        bucketnames: YOUR_S3_BUCKET_NAME
        s3: s3://YOUR_AWS_REGION
        s3forcepathstyle: false
      boltdb_shipper:
        shared_store: s3
serviceAccount:
  annotations:
    iam.amazonaws.com/role: IAM-ROLE-ARN
  create: true
  name: loki-sa

Install Loki-Distributed using Following Commands

				
					helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki-distributed -f values.yaml

Step 3: Setup Promtail Values and Install Using Helm

Promtail is an agent that ships the logs to Loki. You’ll need to deploy Promtail to ensure logs are being collected and sent to your Loki instance.

Create a promtail-values.yamlfile with the following configuration, ensuring it points to your Loki instance:

				
					config:
  clients:
  - url: http://loki-loki-distributed-gateway/loki/api/v1/push
  enableTracing: false
  enabled: true
  logFormat: logfmt
  logLevel: info
  positions:
    filename: /run/promtail/positions.yaml
  serverPort: 3101
daemonset:
  enabled: true
deployment:
  enabled: false
fullnameOverride: promtail
serviceAccount:
  annotations: {}
  create: true
  name: promtail-sa
tolerations:
- effect: NoSchedule
  key: node-role.kubernetes.io/master
  operator: Exists
- effect: NoSchedule
  key: node-role.kubernetes.io/control-plane
  operator: Exists

2. Install Promtail using Helm:

				
					helm install promtail grafana/promtail -f promtail-values.yaml

Step 4: Connect Loki Data Source in Grafana

To visualize the logs stored in Loki, you need to configure Loki as a data source in Grafana.

Open Grafana:

Navigate to your Grafana instance in your web browser.

Add Loki as a Data Source:

Click on the Connection section.
Click on “Data Sources” and then “Add data source”.
Select “Loki” from the list of available data sources.
In the HTTP URL field, enter the URL of your Loki instance (e.g., http://LOKI-SERVICENAME).
Click “Save & Test” to ensure the connection is successful

Step 5: Verify and Adjust Settings

After deploying Loki and Promtail, you should verify that logs are being stored in your S3 bucket. Check the logs in Loki to ensure there are no errors and that log data is being written to S3 as expected. This process will create folders named “fake” and “index,” as well as a file named “loki_cluster_seed.json.” It can take up to 1 hour to push data into the S3 bucket, but this duration can be adjusted using the following configurations: chunk_idle_period: 1h, chunk_target_size: 1536000, and max_chunk_age: 1h under loki’s values.yaml file.

Conclusion

By following these steps, you’ve successfully configured Loki to store logs in an S3 bucket and deployed Promtail to collect and ship your logs. This setup will not only ensure your logs are stored securely but also provide you with the flexibility to scale your log storage as needed. Happy logging!

Store Loki Logs in an S3 Bucket with Kube-Prometheus-Stack