Skip to main content
Calico Cloud documentation

Filter DNS logs

Calico Cloud supports filtering out DNS logs based on user provided configuration. Use filtering to suppress logs of low significance.

Before you begin​

Required

If you already filter DNS logs, migrate your filters before you upgrade. Earlier releases collected logs with Fluentd and read filters written in Fluentd <filter> syntax from a ConfigMap named fluentd-filters. Calico Cloud now collects logs with Fluent Bit, which does not read that ConfigMap and cannot translate Fluentd syntax. Rewrite each filter as a Fluent Bit filter list using the steps on this page, and create it in a ConfigMap named fluent-bit-filters.

Until you do, Calico Cloud ships DNS logs unfiltered, which increases your log volume and storage costs. If a key in fluent-bit-filters is not valid Fluent Bit YAML, Calico Cloud skips that key, keeps shipping unfiltered logs, and reports a warning naming the key:

kubectl get tigerastatus log-collector -o yaml

Limitations

Filters apply to logs collected from Linux nodes. The Windows log collector does not read them.

Configure DNS filtering​

DNS log filtering is configured through a ConfigMap in the tigera-operator namespace.

To enable DNS log filtering, follow these steps:

  1. Create a filters directory with a file named dns with the contents of your desired filter using Filter configuration files. If you are also adding flow filters also add the flow file to the directory.
  2. Create the fluent-bit-filters ConfigMap in the tigera-operator namespace with the following command. 
    kubectl create configmap fluent-bit-filters -n tigera-operator --from-file=filters

The operator inserts the filters inline into the log collector configuration and rolls the calico-fluent-bit DaemonSet automatically.

Filter configuration files​

Each file holds a YAML list of Fluent Bit filter entries. The calico-fluent-bit log collector ships four Fluent Bit filters — grep, record_modifier, parser, and lua — and no others. The examples below use grep. Filters in the dns file are applied to DNS logs automatically; you do not need to set a match on each entry. For the fields you can filter on, see the DNS log schema.

Example 1: filter out cluster-internal lookups​

This example filters out lookups for domain names ending with ".cluster.local". More logs could be filtered by adjusting the regular expression, or by adding additional exclude rules.

- name: grep
  exclude: qname \.cluster\.local$

Example 2: keep logs only for particular domain names​

This example will filter out all logs except those for domain names ending .co.uk.

- name: grep
  regex: qname \.co\.uk$