Fluent Bit¶
Status: AI draft, 2026-11-16. Not yet verified by me. Written from the Fluent Bit documentation; Fluent Bit was not run for these notes. The log file format it reads is shown with real output on How Kubernetes logging works.
In one line¶
Fluent Bit is a small log agent, usually a DaemonSet, that tails log files, parses and enriches records with pod metadata, buffers them in memory or on disk, and sends them to outputs such as OpenSearch or S3. Buffer limits and retry settings decide whether logs survive outages.
The plain idea¶
Logs sit in files on every node (How Kubernetes logging works). Something on each node must read them as they grow, turn each line into a structured record, add who wrote it (namespace, pod, labels), and send it on, without losing lines when the destination is slow and without using much CPU or memory on a node that is running your applications. Fluent Bit is that agent.
How it works¶
flowchart TD
I["Input: tail /var/log/containers/*.log<br/>offsets kept in a DB file"] --> P["Parser: CRI format, then JSON;<br/>multiline joins stack traces"]
P --> F["Filters: kubernetes metadata,<br/>modify, grep, lua"]
F --> B["Buffer: memory, or filesystem chunks"]
B --> R["Router: match records by tag"]
R --> O1["Output: OpenSearch"]
R --> O2["Output: S3 archive"]
Inputs and the position database¶
tailreads files matching a path and emits one record per line, tagged (for Kubernetes typicallykube.*, which the kubernetes filter uses to find the pod).dbnames a small database file that records each file's inode and offset. After a restart Fluent Bit continues where it stopped instead of re-sending or skipping. Keep it on the host (ahostPath), not in the pod's own filesystem.- Other inputs:
systemd(the node's journal),forward, metrics inputs.
Parsers¶
| Need | Parser |
|---|---|
Container runtime lines (<time> stdout F <message>) |
Built-in cri multiline parser (reassembles P partial lines) |
| Docker JSON log files | Built-in docker parser |
| App writes JSON | json parser, or the kubernetes filter's merge_log |
| Fixed text format | A regex parser with named groups |
| Stack traces | Multiline parsers (built-in ones for Java, Python, Go, or custom rules) |
The kubernetes filter¶
Reads the pod name, namespace and container from the tag (the file name), asks the API server (or the kubelet) for the pod's metadata, and adds kubernetes.namespace_name, pod_name, labels, container_image and more to each record. Useful options: merge_log (parse a JSON log field into top-level fields), k8s-logging.parser (let pods suggest a parser by annotation) and k8s-logging.exclude (let pods opt out). Its service account needs RBAC to get pods.
Buffering and backpressure¶
- Memory buffering (default): fast, but lost on restart. When an input reaches its
mem_buf_limit, Fluent Bit pauses that input (loggingpaused (mem buf overlimit)) until chunks are flushed; meanwhile files keep growing and may rotate away unread. - Filesystem buffering (
storage.type filesystemwith astorage.path): chunks are written to disk, surviving restarts and absorbing longer outages, at the cost of disk I/O.
Retries¶
When an output fails, the chunk is scheduled for a retry with exponential backoff. Retry_Limit is set per output and defaults to 1: one retry, then the data is discarded. Set it higher, or to no_limits, for destinations that can be down for minutes, and pair that with filesystem buffering.
Outputs¶
| Output | Notes |
|---|---|
opensearch |
Bulk API to OpenSearch; supports AWS SigV4 auth for Amazon OpenSearch Service; watch for bulk rejections (OpenSearch) |
s3 |
Batches into objects; cheap archive; query later with Athena (S3) |
cloudwatch_logs |
Log groups and streams in CloudWatch (CloudWatch) |
stdout |
Debugging the pipeline |
On EKS, Fluent Bit gets AWS permissions through IRSA or Pod Identity (Cluster access, IRSA and Pod Identity).
A concrete example¶
A YAML configuration for the DaemonSet:
service:
flush: 5
http_server: on # metrics on port 2020
storage.path: /var/fluent-bit/state/buffer
pipeline:
inputs:
- name: tail
tag: kube.*
path: /var/log/containers/*.log
multiline.parser: cri
db: /var/fluent-bit/state/tail.db
mem_buf_limit: 50MB
storage.type: filesystem
filters:
- name: kubernetes
match: kube.*
merge_log: on
keep_log: off
k8s-logging.parser: on
k8s-logging.exclude: on
outputs:
- name: opensearch
match: kube.*
host: logs.example.com
port: 443
tls: on
aws_auth: on
aws_region: ap-south-1
logstash_format: on
logstash_prefix: k8s-logs
retry_limit: no_limits
/var/fluent-bit/state is a hostPath volume, so the offsets and buffered chunks survive pod restarts.
Why not just...?¶
- ...use memory buffering because it is simpler? A restart, or a backend outage longer than the memory buffer, loses logs, and paused inputs fall behind rotation. Use filesystem buffering for anything important.
- ...keep the default
Retry_Limit? One retry means a backend restart of a minute drops data. Raise it deliberately. - ...parse everything with regexes in the agent? Regex parsing costs CPU on every node and breaks when formats change. Have applications log JSON.
Commands to know¶
| Command | What it does |
|---|---|
kubectl logs -n logging ds/fluent-bit |
Agent errors: output failures, paused inputs, retries. |
kubectl exec -n logging <pod> -- curl -s localhost:2020/api/v1/metrics/prometheus |
Records in and out, retries and errors per plugin (with http_server on). |
kubectl exec -n logging <pod> -- curl -s localhost:2020/api/v1/storage |
Buffer chunk counts (with storage metrics enabled). |
Add an output name: stdout, match: kube.* |
See exactly what records look like after filters. |
fluent-bit --dry-run -c fluent-bit.yaml |
Check a configuration without running it. |
How it breaks¶
| Symptom | Layer | Check | Likely cause |
|---|---|---|---|
| Gaps in logs during a backend outage | Retries | retry_limit, output errors |
Default one retry, then dropped |
paused (mem buf overlimit) in agent logs |
Backpressure | mem_buf_limit, output speed |
Output slower than input; use filesystem buffering |
| Duplicate logs after a restart | Offsets | db location |
Position database not persisted on the host |
| Records missing pod metadata | Kubernetes filter | Tag, RBAC | Tag does not match the file naming, or no permission to get pods |
| Stack traces split | Parsing | Multiline settings | No multiline parser for that language |
| OpenSearch rejects records | Mapping | Rejection errors | Field type conflicts or mapping explosion (OpenSearch) |
Interview questions¶
1. Describe the Fluent Bit pipeline.
Answer
- Input (such as tail), parser, filters, buffer, router, outputs.
- Records carry tags; outputs match tags.
- Each stage can be configured per plugin.
2. How does Fluent Bit avoid re-reading or skipping logs after a restart?
Answer
- The tail input's
dboption stores each file's inode and offset. - On restart it resumes from those offsets.
- The database must live on the host (hostPath), not inside the container.
3. What does the kubernetes filter do?
Answer
- Derives the pod from the record's tag (file name) and fetches its metadata from the API server or kubelet.
- Adds namespace, pod, container, labels and more to each record.
- Can merge JSON log fields and honour pod annotations for parsers or exclusion.
4. Memory vs filesystem buffering?
Answer
- Memory: fast, lost on restart; inputs pause at
mem_buf_limit. - Filesystem: chunks on disk survive restarts and longer outages.
- Use filesystem buffering for logs you cannot afford to lose.
5. Logs went missing during a 10-minute OpenSearch outage. Why, and how do you prevent it?
Answer
- With the default
Retry_Limitof 1, failed chunks were retried once and discarded; memory buffers may also have paused inputs while files rotated. - Set a higher or unlimited retry limit and filesystem buffering with enough disk.
- Monitor retry and drop metrics.
6. How do you handle multiline stack traces?
Answer
- Use the CRI multiline parser first, then a language-specific multiline parser (Java, Python, Go) or custom rules.
- Or log JSON with the stack trace as one escaped field.
- Test with real traces before rolling out.
Try it¶
- Using the CRI lines on How Kubernetes logging works, write down the record Fluent Bit would produce for the JSON line after
merge_log. - Change the example configuration to also send everything to S3 with a different tag match.
- List the metrics you would alert on for the Fluent Bit DaemonSet itself.
Zoom out¶
Fluent Bit is the shipping step between node log files and a searchable store (How Kubernetes logging works, OpenSearch). Its reliability settings (buffering, retries) decide whether logs survive incidents, exactly when you need them most.
Sources¶
- Fluent Bit documentation: data pipeline
- Fluent Bit documentation: tail input
- Fluent Bit documentation: kubernetes filter
- Fluent Bit documentation: multiline parsing
- Fluent Bit documentation: buffering
- Fluent Bit documentation: backpressure
- Fluent Bit documentation: scheduling and retries
- Fluent Bit documentation: OpenSearch output