can't find the shipped logs from filebeats to kibana

[wael-sudo2]

Problem description

i'am trying to ship logs from certain log files using filebeat with the configratrion filebeat.inputs

Stack configuration

i just adeed this :
filebeat.inputs:

• type: log
  enabled: true
  paths:
  • /test_logs/test_logs.log

and commented the filebeat.autodiscover config in the filebeat.yml

Docker setup

Client:
Cloud integration: v1.0.35+desktop.4
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:32:48 2023
OS/Arch: windows/amd64
Context: default

Server: Docker Desktop 4.23.0 (120376)
Engine:
Version: 24.0.6
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 1a79695
Built: Mon Sep 4 12:32:16 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.22
GitCommit: 8165feabfdfe38c65b599c4993d227328c231fca
runc:
Version: 1.1.8
GitCommit: v1.1.8-0-g82f18fe
docker-init:
Version: 0.19.0
GitCommit: de40ad0

docker-compose

Docker Compose version v2.21.0-desktop.1

Container logs

file beat logs :
2023-10-07 16:04:02 {"log.level":"info","@timestamp":"2023-10-07T15:04:02.993Z","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":174},"message":"Configured paths: [/usr/share/filebeat/test_logs/test_logs.log]","service.name":"filebeat","input_id":"58e2b0a5-caee-48f7-bdab-9881211d8ea0","ecs.version":"1.6.0"}
2023-10-07 16:04:02 {"log.level":"info","@timestamp":"2023-10-07T15:04:02.993Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 2937918885526542731)","service.name":"filebeat","ecs.version":"1.6.0"}
2023-10-07 16:04:02 {"log.level":"info","@timestamp":"2023-10-07T15:04:02.995Z","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
harvester 0
2023-10-07 15:26:41 {"log.level":"info","@timestamp":"2023-10-07T14:26:41.569Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":20746000}},"memory":{"mem":{"usage":{"bytes":53133312}}}},"cpu":{"system":{"ticks":190,"time":{"ms":10}},"total":{"ticks":640,"time":{"ms":20},"value":640},"user":{"ticks":450,"time":{"ms":10}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"22236f1d-29dc-406b-9bb8-a8f916451138","uptime":{"ms":330162},"version":"8.10.2"},"memstats":{"gc_next":35810696,"memory_alloc":21205840,"memory_total":73898088,"rss":105918464},"runtime":{"goroutines":37}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.1,"15":0.48,"5":0.48,"norm":{"1":0.0063,"15":0.03,"5":0.03}}}},"ecs.version":"1.6.0"}}

[antoineco]

Is /test_logs/test_logs.log mounted inside your filebeat container using a volume in the Compose file?

[wael-sudo2]

i think no this is how my filebeat compose look like :

version: '3.7'

services:
  filebeat:
    build:
      context: extensions/filebeat/
      args:
        ELASTIC_VERSION: ${ELASTIC_VERSION}
    # Run as 'root' instead of 'filebeat' (uid 1000) to allow reading
    # 'docker.sock' and the host's filesystem.
    user: root
    command:
      # Log to stderr.
      - -e
      # Disable config file permissions checks. Allows mounting
      # 'config/filebeat.yml' even if it's not owned by root.
      # see: https://www.elastic.co/guide/en/beats/libbeat/current/config-file-permissions.html
      - --strict.perms=false
    volumes:
      - ./extensions/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro,Z
      - type: bind
        source: /var/lib/docker/containers
        target: /var/lib/docker/containers
        read_only: true
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
        read_only: true
    environment:
      FILEBEAT_INTERNAL_PASSWORD: ${FILEBEAT_INTERNAL_PASSWORD:-}
      BEATS_SYSTEM_PASSWORD: ${BEATS_SYSTEM_PASSWORD:-}
    networks:
      - elk
    depends_on:
      - elasticsearch

[antoineco]

Right, so you need to add either your directory or the individual file under volumes:, following the same syntax as the existing entries.

[wael-sudo2]

works after addding - ./extensions/filebeat/config/test_logs:/usr/share/filebeat/test_logs:ro,Z to volumes