force filebeat to log to a file

Question

Using filebeat-8.7.1, I have it configured to push several sources to a remote elastisearch. I cannot get it to log its own logging to a local file /var/log/filebeat/filebeat.log, and I cannot prevent it from dumping its logs into /var/log/syslog (which is also going to elastisearch).

The relevant configuration from /etc/filebeat/filebeat.yml:

filebeat.inputs:
- type: filestream
  id: rsc-server
  enabled: true
  paths:
    - /var/log/rstudio/rstudio-connect/rstudio*.log
  fields:
    log_type: rsc_server
- type: filestream
  enabled: true
  ignore_older: 5m
  id: rsc-jobs
  paths:
    - /srv/R/RSC/jobs/[0-9]*/*/*
  fields:
    log_type: rsc
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 1
setup.kibana:
  host: "http://my.elasti.host:30002"
output.elasticsearch:
  hosts: ["http://my.elasti.host:30003"]
  allow_older_versions: true
  username: "myuser"
  password: "mypass"
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
logging:
  - level: warning
  - to_syslog: false
  - to_stderr: false
  - json: true
  - files:
    - path: /var/log/filebeat
    - name: filebeat
    - keepfile: 3
    - permissions: 0644

And from the /etc/filebeat/modules.d/ directory, the resulting enabled configs are:

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-nginx.html
- module: nginx
  # Access logs
  access:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
  # Error logs
  error:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
  # Ingress-nginx controller logs. This is disabled by default. It could be used in Kubernetes environments to parse ingress-nginx logs
  ingress_controller:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
# Module: redis
# Docs: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-redis.html
- module: redis
  # Main logs
  log:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths: ["/var/log/redis/redis-server.log*"]
  # Slow logs, retrieved via the Redis API (SLOWLOG)
  slowlog:
    enabled: false
    # The Redis hosts to connect to.
    #var.hosts: ["localhost:6379"]
    # Optional, the password to use when connecting to Redis.
    #var.password:
# Module: system
# Docs: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-system.html
- module: system
  # Syslog
  syslog:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:
  # Authorization logs
  auth:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

I believe that the system module use of syslog is to pull from /var/log/syslog, not push to it. (And when I disable that module, the behavior does not change.)

How do I fix the logging.* section to send logs to /var/log/filebeat/, and how to I prevent it from sending to syslog and /var/log/syslog?

(This is on an ubuntu-22.04 VM.)

---

Edit: /lib/systemd/system/filebeat.service has an empty BEAT_LOG_OPTS arg (this is the default file, not edited by me):

[Unit]
Description=Filebeat sends log files to Logstash or directly to Elasticsearch.
Documentation=https://www.elastic.co/beats/filebeat
Wants=network-online.target
After=network-online.target
[Service]
UMask=0027
Environment="GODEBUG='madvdontneed=1'"
Environment="BEAT_LOG_OPTS="
Environment="BEAT_CONFIG_OPTS=-c /etc/filebeat/filebeat.yml"
Environment="BEAT_PATH_OPTS=--path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat"
ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
Restart=always
[Install]
WantedBy=multi-user.target

With a reload (and no change in performance):

# systemctl daemon-reload
# systemctl restart filebeat.service
# systemctl status filebeat.service

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-05-04 14:49:12 UTC; 20s ago
       Docs: https://www.elastic.co/beats/filebeat
   Main PID: 2080223 (filebeat)
      Tasks: 25 (limit: 76964)
     Memory: 114.4M
        CPU: 5.315s
     CGroup: /system.slice/filebeat.service
             └─2080223 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

The process is running as root, and the /var/log/filebeat directory is owned by root with normal perms:

# ps faxu | grep [/]filebeat
root     2080223  9.4  0.1 2635592 169112 ?      Ssl  14:49   0:36 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

# ll -d /var/log/filebeat
drwxr-xr-x 2 root root 4096 May  4 12:54 /var/log/filebeat/
# ll  /var/log/filebeat
total 8
drwxr-xr-x  2 root root   4096 May  4 12:54 ./
drwxr-xr-x 18 root syslog 4096 May  4 12:54 ../

I see --environment systemd that per filebeat command-line-options means

If systemd or container is specified, Filebeat will log to stdout and stderr by default.

This suggests that logging is going to /var/log/syslog due to the way systemd is handling stdout/stderr from the process. That seems like progress, but we still don't have logging going to /var/log/filebeat/, which is ultimately where both stdout/stderr should be going (which would resolve the problem of logging to /var/log/syslog).

Answer 1

See the post Filebeat refuses to log to file.

The explanation there, quoted from the release notes is:

On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value

And the solution itself you may be able to adapt to your environment :

To override these variables, create a drop-in unit file in the /etc/systemd/system/filebeat.service.d directory.

To use [the logging] settings from the Filebeat file, empty the environment variable. For example:

[Service]
Environment="BEAT_LOG_OPTS="

To apply your changes, reload the systemd configuration and restart the service:

systemctl daemon-reload
systemctl restart filebeat

It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use systemctl edit filebeat.service .

Reference for the command line of filebeat to understand what the unit file does and what it can do: https://www.elastic.co/guide/en/beats/filebeat/7.0/command-line-options.html#global-flags 4

To see the default filebeat unit file that came with the installation do:

systemctl status filebeat

and check the content of the unit file. (The path to the unit file is in the output of the above command.)

Answer 2

harrymc helped identify the culprit, here are some final steps plus an alternative workaround.

The issue with filebeat logging to /var/log/syslog was with systemd services, not filebeat itself: the use of --environment systemd on the filebeat command line (which is the default on ubuntu, perhaps part of the problem) is causing filebeat to force logging to stdout. This is counter to the documentation suggesting that this bypass only occurs when logging is not otherwise defined:

 -environment
    For logging purposes, specifies the environment that Filebeat is
    running in. This setting is used to select a default log output
    when no log output is configured. Supported values are: systemd,
    container, macos_service, and windows_service. If systemd or
    container is specified, Filebeat will log to stdout and stderr
    by default. 

Option 1: remove --environment systemd

Run systemctl edit filebeat.service, and remove the command-line arguments from ExecStart (after first clearing that variable). The edited text should contain

### /lib/systemd/system/filebeat.service
[Service]
ExecStart=
ExecStart=/usr/share/filebeat/bin/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS

The first empty ExecStart= is intentional: without it, systemd will complain about Service has more than one ExecStart= setting.

By doing this, we now see /var/log/filebeat/filebeat-20230504.ndjson.

Many thanks to @user1686 for helping me to find the resolution for multiple ExecStart= settings.

Option 2: redirect stdout/stderr via systemd

Another option is to keep filebeat dumping to its stdout/stderr and using its internal mechanisms for redirecting to files. (If you made the change to /lib/systemd/system/filebeat.service in option 1, then undo that change for this option.)

Call systemctl edit filebeat.service and include this,

[Service]
StandardOutput=file:/var/log/filebeat/filebeat_stdout.log
StandardError=file:/var/log/filebeat/filebeat_stderr.log

(which is stored within /etc/systemd/system/filebeat.service.d/override.conf). Then restart the service, we now see /var/log/filebeat/filebeat_stdout.log (and _stderr).