---
title: "Filter flow logs"
description: "Filter Calico Cloud flow logs through Fluent Bit to drop low-significance traffic and reduce managed Elasticsearch volume and cost."
product: "Calico Cloud"
version: "v20.4.0"
section: "Observability"
canonical_url: "https://docs.tigera.io/calico-cloud/next/observability/elastic/flow/filtering"
---

# Filter flow logs

## Big picture

Filter Calico Cloud flow logs.

## Value

Filter Calico Cloud flow logs to suppress logs of low significance, and troubleshoot threats.

## Concepts

### Container monitoring tools versus flow logs

Container monitoring tools are good for monitoring Kubernetes and orchestrated workloads for CPU usage, network usage, and log aggregation. For example, a data monitoring tool can tell if a pod has turned into a bitcoin miner based on it using more than normal CPU.

Calico Cloud flow logs provide continuous records of every single packet sent/received by all pods in your Kubernetes cluster. Note that flow logs do not contain all packet data; only the number of packets/bytes that were sent between specific IP/ports, and when. In the previous monitoring tool example, Calico Cloud flow logs could see the packets running to/from the bitcoin mining network.

Calico Cloud flow logs tell you when a pod is compromised, specifically:

- Where a pod is sending data to
- If the pod is talking to a known command-and-control server
- Other pods that the compromised pod has been talking to (so you can see if they're compromised too)

### Flow log format

A flow log contains these space-delimited fields (unless filtered out).

```text
startTime endTime srcType srcNamespace srcName srcLabels dstType dstNamespace dstName
dstLabels srcIP dstIP proto srcPort dstPort numFlows numFlowsStarted numFlowsCompleted
reporter packetsIn packetsOut bytesIn bytesOut action
```

**Example**

```text
1528842551 1528842851 wep dev rails-81531* - wep dev memcached-38456* - - - 6 - 3000 7 3 4 out 154 61 70111 49404 allow
```

- Fields that are not enabled or are aggregated, are noted by `-`
- Aggregated names (such as “pod prefix”), are noted by `*` at the end of the name
- If `srcName` or `dstName` fields contain only a `*`, aggregation was performed using other means (such as specific labels), and no unique prefix was present.

## Before you begin

**Required**

If you already filter flow logs, migrate your filters before you upgrade. Earlier releases collected logs with Fluentd and read filters written in Fluentd `<filter>` syntax from a ConfigMap named `fluentd-filters`. Calico Cloud now collects logs with Fluent Bit, which does not read that ConfigMap and cannot translate Fluentd syntax. Rewrite each filter as a Fluent Bit filter list using the steps on this page, and create it in a ConfigMap named `fluent-bit-filters`.

Until you do, Calico Cloud ships flow logs unfiltered, which increases your log volume and storage costs. If a key in `fluent-bit-filters` is not valid Fluent Bit YAML, Calico Cloud skips that key, keeps shipping unfiltered logs, and reports a warning naming the key:

```bash
kubectl get tigerastatus log-collector -o yaml
```

**Limitations**

Filters apply to logs collected from Linux nodes. The Windows log collector does not read them.

## How to

- [Create flow log filters](#create-flow-log-filters)
- [Add filters to ConfigMap file](#add-filters-to-configmap-file)

### Create flow log filters

Filters are written as a YAML list of [Fluent Bit filter](https://docs.fluentbit.io/manual/data-pipeline/filters/) entries. The `calico-fluent-bit` log collector ships four Fluent Bit filters — [`grep`](https://docs.fluentbit.io/manual/data-pipeline/filters/grep), [`record_modifier`](https://docs.fluentbit.io/manual/data-pipeline/filters/record-modifier), [`parser`](https://docs.fluentbit.io/manual/data-pipeline/filters/parser), and [`lua`](https://docs.fluentbit.io/manual/data-pipeline/filters/lua) — and no others. Filters you add under the `flow` key are applied to flow logs automatically; you do not need to set a `match` on each entry.

**Example: filter out a specific namespace**

This example filters out all flow logs whose source or destination namespace is `dev`. A record is dropped when it matches any of the `exclude` rules; additional namespaces could be filtered by adjusting the regular expressions, or by adding more `exclude` rules.

```yaml
- name: grep
  exclude:
    - source_namespace dev
    - dest_namespace dev
```

**Example: filter out internet traffic to a specific deployment**

This example filters inbound internet traffic to the deployment with pods named, `nginx-internet-*`. Note the use of `logical_op: and` to filter out only the traffic that is both to the deployment, and from the internet (source `pub`).

```yaml
- name: grep
  logical_op: and
  exclude:
    - dest_name_aggr ^nginx-internet
    - source_name_aggr pub
```

### Add filters to ConfigMap file

1. Create a `filters` directory with a file called `flow` with your desired filters. If you are also adding [DNS filters](https://docs.tigera.io/calico-cloud/next/observability/elastic/dns/filtering-dns.md), add the `dns` file to the directory.

2. Create the `fluent-bit-filters` ConfigMap in the `tigera-operator` namespace with the following command.

   ```bash
   kubectl create configmap fluent-bit-filters -n tigera-operator --from-file=filters
   ```

The operator inserts the filters inline into the log collector configuration and rolls the `calico-fluent-bit` DaemonSet automatically.

## Additional resources

- [Flow log aggregation](https://docs.tigera.io/calico-cloud/next/observability/elastic/flow/aggregation.md)
- [Archive logs to storage](https://docs.tigera.io/calico-cloud/next/observability/elastic/archive-storage.md)