Here’s a conversation we hear constantly from SMB engineering teams: “Our monitoring bill is now higher than our database bill. Nobody can tell me what we’re paying for.” It’s the observability tax — and for lean teams it’s usually log ingestion and retention, not metrics, that quietly eats the budget. Every console.log, every debug-level framework line, every verbose nginx access log gets shipped to a vendor, stored for 30 days, and charged per gigabyte. Twice (ingestion + retention).
The good news: log costs are almost entirely controllable. This guide walks through the four levers that matter — measure, sample, filter, and tier — with real configs for Loki and Fluent Bit that you can deploy this week.
Step 1: Measure Before You Cut (Most Teams Skip This)
You can’t optimize what you don’t measure. Before touching a single config, find out which workloads generate the bulk of your log volume. If you run Loki, the logcli volume query does this in seconds:
# Top 10 log producers by namespace, last 24h
logcli query 'sum by (namespace) (count_over_time({job="kubernetes-pods"} [24h]))' \
--from "$(date -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" --limit 10
# Bytes ingested per app label
logcli query 'sum by (app) (bytes_over_time({job="kubernetes-pods"} [24h]))' \
--from "$(date -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" --limit 10In practice, we see the same pattern at almost every SMB: 20% of workloads produce 80% of logs, and the top offenders are usually (1) verbose application frameworks, (2) nginx/Ingress access logs, and (3) Kubernetes system components like kube-proxy and audit logs. List them, quantify them, and you now have a target list. For managed vendors, the equivalent is a per-service volume breakdown in their usage dashboard — pull the same top-10 list from there.
Step 2: Filter and Sample at the Edge
Never pay to ingest a log line you’re going to ignore. Edge filtering and sampling in your collector is the single biggest win — typically 50–80% volume reduction with zero visibility loss for real incidents.
With Fluent Bit, start by dropping noisy levels and known-chatty patterns, then sample debug lines:
[FILTER]
Name grep
Match *
Exclude log_level debug
[FILTER]
Name rewrite_tag
Match app.*
Rule $log_level ^info$ app.info.$TAG false
# Sample 10% of info logs, keep 100% of warn/error
[FILTER]
Name modify
Match app.info.*
Condition Key_value_matches log_level ^info$
Set __sampled__ true
[OUTPUT]
Name loki
Match app.info.*
Sampler tumbling
Sampling ratio 0.1If you’re on OpenTelemetry, the tail_sampling processor gives you policy-based control: keep 100% of errors and slow requests, sample the rest at 10%:
processors:
tail_sampling:
decision_wait: 10s
policies:
- name: keep-errors
type: status_code
status_code: { status_codes: [ERROR] }
- name: keep-slow
type: latency
latency: { threshold_ms: 2000 }
- name: sample-rest
type: probabilistic
probabilistic: { sampling_percentage: 10 }One caveat from real incident postmortems: never sample security-relevant logs (auth attempts, admin actions, payment events) and never sample before your alerting rules consume the data. Sampling belongs downstream of routing, not upstream of it.
Step 3: Convert Logs to Metrics — Pay for the Signal, Not the Noise
The most cost-effective observability move in 2026 is to stop storing every request as a log line and start storing the aggregate as a metric. Metrics are a fraction of the price of logs, and for 90% of operational questions (error rate, latency distribution, saturation) they’re more useful. Counters for nginx status codes are the classic example — one series instead of millions of lines:
# In your nginx log pipeline, emit a counter instead of shipping the raw line
[FILTER]
Name lua
Match nginx.*
call count_status
code |
function count_status(tag, timestamp, record)
local m = {}
m["nginx_http_requests_total"] = {
value = 1,
labels = { status = record["status"], host = record["host"] }
}
return 1, timestamp, record, m
end
[OUTPUT]
Name prometheus_remote_write
Match nginx.*
host prometheus.monitoring.svc
port 9090Same pattern applies to application logs: a payment_attempts_total counter plus an error counter replaces hundreds of thousands of lines per month, and your SLO dashboards get better data. If you’re building this from scratch, our OpenTelemetry adoption guide shows how to standardize on one pipeline that does logs, metrics, and traces together.
Step 4: Tier Your Storage — Hot, Warm, and Cold Retention
Nobody needs 30 days of raw debug logs in hot storage. The cheapest gigabyte is the one you delete — the second cheapest is the one on object storage. Loki has native tiering via the compactor; 15 days hot (SSD) plus 12 months cold (S3) costs roughly a tenth of keeping everything hot:
# Loki config: chunk retention in hot storage, then ship to S3
storage_config:
boltdb_shipper:
shared_store: s3
aws:
s3: s3://logs-bucket/loki
s3forcepathstyle: true
compactor:
working_directory: /data/compactor
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
limits_config:
retention_period: 4380h # 6 months, enforced per-tenant
# per-stream limits so one noisy app can't blow the budget
per_stream_rate_limit: 3MB
per_stream_rate_limit_burst: 15MB
max_streams_per_user: 5000
schema_config:
configs:
- from: 2026-01-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24hFor OpenSearch, the equivalent is an Index State Management (ISM) policy: roll over indices daily, move them to warm storage after 7 days, and delete after 90 — all automated:
{
"policy": {
"description": "SMB log lifecycle",
"default_state": "hot",
"states": [
{ "name": "hot", "actions": [{ "rollover": { "min_size": "50gb" } }],
"transitions": [{ "state_name": "warm", "conditions": { "min_index_age": "7d" } }] },
{ "name": "warm", "actions": [{ "warm_migration": {} }],
"transitions": [{ "state_name": "delete", "conditions": { "min_index_age": "90d" } }] },
{ "name": "delete", "actions": [{ "delete": {} }] }
],
"ism_template": { "index_patterns": ["logs-*"], "priority": 100 }
}
}While you’re at it, fix the two label mistakes that silently inflate every log system’s cost: high-cardinality labels (never put request IDs, user IDs, or IPs in labels — put them in the log body) and unbounded retention defaults. Both are covered in depth in our minimalist observability stack guide.
Step 5: Make the Savings Permanent — Budget Alerts and Review Cadence
Log costs creep back within a quarter unless someone owns the number. Set a simple monthly review: a dashboard with GB ingested per namespace and estimated cost per namespace, plus an alert when daily ingestion grows more than 20% week-over-week. In Grafana/Loki that’s one panel and one alert rule; in managed vendors, use their budget alerts. Tie it to the same FinOps cadence as your compute spend — our alerting hygiene guide shows how to keep the alert itself from becoming noise.
The pattern is simple: measure → filter → sample → convert to metrics → tier. In our client engagements this sequence routinely cuts observability spend by 50–70% within two months, and the same data gets more useful, because the signal-to-noise ratio of what reaches your dashboards goes up. If your log bill is growing faster than your revenue, book a free 30-minute consultation — we’ll audit your pipeline and show you exactly where the money is going.
Español 
English