Fluent Bit encountering errors after detecting log files in FSx file shares on Windows Server 2019 (v2.0.9)

[evolart]

This is a continuation from #7185. Original file paths and other environment details/metadata has been stripped.

Problem:

After resolving the issue of Fluent Bit not detecting log files in FSx file shares on Windows Server 2019, we are now encountering errors when adding log files to bulk ingestion. It's worth noting that the message lines with "adding log to the bulk" show the file path with the backslashes escaped out. We had to remove the backslash escaping from our INPUT stanza to get Fluent Bit to detect the log files in the first place.

Environment:

• Fluent Bit Version: 2.0.9
• Operating System: Windows Server 2019
• FSx File Share Path: \\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs

Steps to Reproduce:

• Configure Fluent Bit with the following input stanza:

   [INPUT]
   Name tail
   Path \\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\*.txt
   Path_Key On
   Tag app_logs
   Buffer_Max_Size 1024k
   Read_from_Head On

• Start Fluent Bit with trace log level.

• Create or modify log files in the specified FSx file share path.

• Observe that Fluent Bit does detect log files but begins throwing errors:

2023/04/18 17:07:36] [ info] [sp] stream processor started
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734757 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt promote to TAIL_EVENT
[2023/04/18 17:07:36] [ info] [output:logzio:logzio.0] worker #0 started
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=3659174697240457 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.1 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=6755399441059732 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.2 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734381 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.3 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734386 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.4 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:36] [debug] [input:tail:tail.0] inode=281474976734391 file=\\fsx-example-host.example.net\share\staging\if\dir\example-server\Logs\example.txt.5 promote to TAIL_EVENT
[2023/04/18 17:07:36] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501512-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","ec2_instance_type":"","fluentbit_tag":"app_logs","host":"","hostname":"","log":"2023-04-18 17:06:54,191 [18] INFO  IFLog [(null)] - Server Info request from ","output_id":"logzio_output_1","private_ip":"","type":"logzio-fluent-bit","vpc_id":""}
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501547-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","ec2_instance_type":"","fluentbit_tag":"app_logs","host":"","hostname":"","log":"2023-04-18 17:06:54,644 [5] INFO  IFLog [(null)] - Server Info request from ","output_id":"logzio_output_1","private_ip":"","type":"logzio-fluent-bit","vpc_id":""}
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501551-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","ec2_instance_type":"","fluentbit_tag":"app_logs","host":"","hostname":"","log":"2023-04-18 17:07:24,226 [11] INFO  IFLog [(null)] - Server Info request from ","output_id":"logzio_output_1","private_ip":"","type":"logzio-fluent-bit","vpc_id":""}
2023/04/18 17:07:37 [logzio] adding log to the bulk: {"@timestamp":"2023-04-18T17:07:36.7501555-06:00","On":"\\\\fsx-example-host.example.net\\share\\staging\\if\\dir\\example-server\\Logs\\example.txt","account_id":"","ami_id":"","az":"us-east-1b","ec2_instance_id":"","vpc_id":""}
2023/04/18 17:07:38 [logzio] successfully sent bulk to logz.io
[2023/04/18 17:07:38] [debug] [out flush] cb_destroy coro_id=0
[2023/04/18 17:07:38] [debug] [task] destroy task=000001BBFAB2D750 (task_id=0)
[2023/04/18 17:07:38] [debug] [input chunk] update output instances with new chunk size diff=448
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory
[2023/04/18 17:07:39] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1441 errno=2] No such file or directory

Expected Behavior
Fluent Bit should detect and send logs to output without errors.

Actual Behavior
Fluent Bit does detect file changes in FSx file shares on Windows Server 2019 but throws many errors.

Additional Context
We have validated that the FSx file share path is accessible from the Windows Server 2019 system, and we are able to read the content of the files without any issues.

Please let me know if you need any further information or if there are any workarounds or solutions to this issue.

[edsiper]

@leonardo-albertovich @cosmo0920 this is an issue reported at KubeCon, can you take a loop pls ?

cc: @afishler

[leonardo-albertovich]

It seems like we might have a problem in flb_tail_file_name which is used to get the path that's passed to stat. I'll take a look on Tuesday and get back to you.

I'll try to reproduce it with a UNC path in my local network and go from there.

[evolart]

Great, I did notice that I was incorrect and logs are being sent to our end point despite the errors. So this is more so an annoyance with the errors vs not working entirely it seems.

[evolart]

@leonardo-albertovich @edsiper Any chance of an update here? The errors do flood our logs quite a bit.

[leonardo-albertovich]

Sorry @evolart, I've been swamped and haven't been able to put any time on this so far and can't foresee anyone from the team being able to dedicate time to it in within this week either.

I have this one on my queue and intend to give it closure but I'd be lying if I offered an ETA.

[tummalah]

Hi , I can confirm that I am seeing the same behavior with UNC paths. The shared folder is from another windows 2019 VM.
"\\ABCDSERVER\D$\JD\E920_OR\log\jde_5112.log"

[tummalah]

Sorry @evolart, I've been swamped and haven't been able to put any time on this so far and can't foresee anyone from the team being able to dedicate time to it in within this week either.

I have this one on my queue and intend to give it closure but I'd be lying if I offered an ETA.

Hi @leonardo-albertovich any chance of update on ETA. We have the same problem with the logs flooded with the error messages but the logs successfully streaming. Thanks!!

[leonardo-albertovich]

I'm wrapping up a set of changes that could potentially fix this as a side effect, I'll send an update by the end of the week with a link to the PR and would really appreciate it if tested it and helped with feedback so we can include that fix in the PR (if it doesn't address it).

[leonardo-albertovich]

As promised, this is the PR with the new file access abstraction layer #7564

Keep in mind that even though the code is complete and it was internally tested there is still a discussion and we might make changes so it would be good if you managed to test it and give us some feedback.

I have just started the test container creation process so if everything works as expected (and I can't see any reasons for it not to) there should be container images container images here ghcr.io/fluent/fluent-bit/test/leonardo-master-filesystem-access-abstraction-layer

Although I haven't ever used windows containers so I have no idea of how that works so you might be better off getting the build artifacts from appveyor or building fluent-bit yourself.

[tummalah]

As promised, this is the PR with the new file access abstraction layer #7564

Keep in mind that even though the code is complete and it was internally tested there is still a discussion and we might make changes so it would be good if you managed to test it and give us some feedback.

I have just started the test container creation process so if everything works as expected (and I can't see any reasons for it not to) there should be container images container images here ghcr.io/fluent/fluent-bit/test/leonardo-master-filesystem-access-abstraction-layer

Although I haven't ever used windows containers so I have no idea of how that works so you might be better off getting the build artifacts from appveyor or building fluent-bit yourself.

Thank you @leonardo-albertovich will check on this pr and see if we can find a way to build a windows version.

[tummalah]

ghcr.io/fluent/fluent-bit/test/leonardo-master-filesystem-access-abstraction-layer

@leonardo-albertovich tried to generate windows build on this branch but failed on "C:\fluent-bit\include\fluent-bit/flb_hmac.h(22): fatal error C1083: Cannot open include file: 'openssl/err.h': No such file or directory" . Any ideas?

[cosmo0920]

You need to specify the place for installed/compiled OpenSSL directory. We use OpenSSL installer from chocotaley. Also, you can install it from https://slproweb.com/products/Win32OpenSSL.html. And then, specifying the place with -DOPENSSL_ROOT_DIR=/path/to/openssl as cmake arguments:

fluent-bit/.github/workflows/call-build-windows.yaml

Lines 113 to 143 in 9ecb8b6

	- name: Get dependencies w/ chocolatey
	if: ${{ matrix.config.arch != 'amd64_arm64' }}
	uses: crazy-max/ghaction-chocolatey@v2
	with:
	args: install ${{ matrix.config.chocolatey_opt }} openssl -y
	- name: Set up with Developer Command Prompt for Microsoft Visual C++
	uses: ilammy/msvc-dev-cmd@v1
	with:
	arch: ${{ matrix.config.arch }}
	- name: Build openssl with vcpkg
	if: ${{ matrix.config.arch == 'amd64_arm64' }}
	run: |
	C:\vcpkg\vcpkg install --recurse openssl --triplet ${{ matrix.config.vcpkg_triplet }}
	shell: cmd
	- name: Build libyaml with vcpkg
	run: |
	C:\vcpkg\vcpkg install --recurse libyaml --triplet ${{ matrix.config.vcpkg_triplet }}
	shell: cmd
	- name: Build Fluent Bit packages
	# If we are using 2.0.* or earlier we need to exclude the ARM64 build as the dependencies fail to compile.
	# Trying to do via an exclude for the job triggers linting errors.
	# This is only supposed to be a workaround for now so can be easily removed later.
	if: ${{ matrix.config.arch != 'amd64_arm64' || needs.call-build-windows-get-meta.outputs.armSupported == 'true' }}
	run: |
	cmake -G "NMake Makefiles" -DFLB_NIGHTLY_BUILD='${{ inputs.unstable }}' -DOPENSSL_ROOT_DIR='${{ matrix.config.openssl_dir }}' ${{ matrix.config.cmake_additional_opt }} -DFLB_LIBYAML_DIR=C:\vcpkg\packages\libyaml_${{ matrix.config.vcpkg_triplet }} ../
	cmake --build .
	cpack

Also, if you want to create the installers, you have to install WiX toolset or NSIS.

[tummalah]

You need to specify the place for installed/compiled OpenSSL directory. We use OpenSSL installer from chocotaley. Also, you can install it from https://slproweb.com/products/Win32OpenSSL.html. And then, specifying the place with -DOPENSSL_ROOT_DIR=/path/to/openssl as cmake arguments:

fluent-bit/.github/workflows/call-build-windows.yaml

Lines 113 to 143 in 9ecb8b6

	- name: Get dependencies w/ chocolatey
	if: ${{ matrix.config.arch != 'amd64_arm64' }}
	uses: crazy-max/ghaction-chocolatey@v2
	with:
	args: install ${{ matrix.config.chocolatey_opt }} openssl -y
	- name: Set up with Developer Command Prompt for Microsoft Visual C++
	uses: ilammy/msvc-dev-cmd@v1
	with:
	arch: ${{ matrix.config.arch }}
	- name: Build openssl with vcpkg
	if: ${{ matrix.config.arch == 'amd64_arm64' }}
	run: |
	C:\vcpkg\vcpkg install --recurse openssl --triplet ${{ matrix.config.vcpkg_triplet }}
	shell: cmd
	- name: Build libyaml with vcpkg
	run: |
	C:\vcpkg\vcpkg install --recurse libyaml --triplet ${{ matrix.config.vcpkg_triplet }}
	shell: cmd
	- name: Build Fluent Bit packages
	# If we are using 2.0.* or earlier we need to exclude the ARM64 build as the dependencies fail to compile.
	# Trying to do via an exclude for the job triggers linting errors.
	# This is only supposed to be a workaround for now so can be easily removed later.
	if: ${{ matrix.config.arch != 'amd64_arm64' || needs.call-build-windows-get-meta.outputs.armSupported == 'true' }}
	run: |
	cmake -G "NMake Makefiles" -DFLB_NIGHTLY_BUILD='${{ inputs.unstable }}' -DOPENSSL_ROOT_DIR='${{ matrix.config.openssl_dir }}' ${{ matrix.config.cmake_additional_opt }} -DFLB_LIBYAML_DIR=C:\vcpkg\packages\libyaml_${{ matrix.config.vcpkg_triplet }} ../
	cmake --build .
	cpack

Also, if you want to create the installers, you have to install WiX toolset or NSIS.

Hi @leonardo-albertovich @cosmo0920 I Just tested the branch and found the results to be same as before.
This is what I have done to reproduce:

c:\fluent-bit>git status
On branch leonardo-master-filesystem-access-abstraction-later
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        deleted:    build/.empty

no changes added to commit (use "git add" and/or "git commit -a")

config file:

[SERVICE]
    # setting application logs level to debug
    #Log_Level info
    flush 5
    # setting location for the parsers file
    #Parsers_File jde_parser.conf
[INPUT]
    # using tail input to tail log files
    Name        tail
    # setting the unc path for the file to tail
    Path \\myshare\mydir\test.txt	 #works as before log read but throws error
    #Path c:\fluentbittest\test.txt      # works as before
    # setting the network mapped drive to unc path for the file to tail
    #Path H:\test.txt                    # works as before log read but throws error
    Skip_Empty_Lines On
    Ignore_Older 1d
    Path_Key jde_log_file
    Read_from_Head ON

STDOUT:

c:\>C:\fluent-bit\build\bin\fluent-bit.exe -c C:\fluentbittest\test.conf -o stdout
Fluent Bit v2.1.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/06/16 15:05:41] [ info] [fluent bit] version=2.1.5, commit=1d366594a8, pid=4620
[2023/06/16 15:05:41] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/16 15:05:41] [ info] [cmetrics] version=0.6.2
[2023/06/16 15:05:41] [ info] [ctraces ] version=0.3.1
[2023/06/16 15:05:41] [ info] [input:tail:tail.0] initializing
[2023/06/16 15:05:41] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/06/16 15:05:41] [ info] [sp] stream processor started
[2023/06/16 15:05:41] [ info] [output:stdout:stdout.0] worker #0 started
[2023/06/16 15:05:42] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1513 errno=2] No such file or directory
[2023/06/16 15:05:44] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1513 errno=2] No such file or directory
[0] tail.0: [[1686942342.009459500, {}], {"jde_log_file"=>"\\myshare\mydir\test.txt", "log"=>"test1"}]
[1] tail.0: [[1686942342.009516100, {}], {"jde_log_file"=>"\\myshare\mydir\test.txt", "log"=>"test2"}]
[2] tail.0: [[1686942342.009520000, {}], {"jde_log_file"=>"\\myshare\mydir\test.txt", "log"=>"test3"}]
[2023/06/16 15:05:47] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1513 errno=2] No such file or directory
[2023/06/16 15:05:48] [engine] caught signal (SIGINT)
[2023/06/16 15:05:49] [ warn] [engine] service will shutdown in max 5 seconds
[2023/06/16 15:05:49] [ info] [input] pausing tail.0
[2023/06/16 15:05:50] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/16 15:05:50] [ info] [input] pausing tail.0
[2023/06/16 15:05:50] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/06/16 15:05:50] [ info] [output:stdout:stdout.0] thread worker #0 stopped

[leonardo-albertovich]

Thanks for reporting your results, we will take a look as soon as possible.

One thing that confuses me in your log though is that the error message reports the source line as 1513 whereas in the code line 1513 is not a flb_errno call.

[tummalah]

Thanks for reporting your results, we will take a look as soon as possible.

One thing that confuses me in your log though is that the error message reports the source line as 1513 whereas in the code line 1513 is not a flb_errno call.

Hi @leonardo-albertovich I pulled the branch again and tested. Looks like the earlier test I did probably is against master branch. Here are the results :
config:

[SERVICE]
    # setting application logs level to debug
    #Log_Level info
    flush 5
    # setting location for the parsers file
    #Parsers_File jde_parser.conf
[INPUT]
    # using tail input to tail log files
    Name        tail
    # setting the unc path for the file to tail
    #Path \\myshare\mydir\test.txt	 # throws error but stream log
    #Path c:\fluentbittest\test.txt      # works
    #Path H:\test.txt                    #  throws error but stream log
    Skip_Empty_Lines On
    #Ignore_Older 1d
    Path_Key jde_log_file
    Read_from_Head ON

Log:

c:\fluent-bit>C:\fluent-bit\build\bin\fluent-bit.exe -c C:\fluentbittest\test.conf -o stdout
Fluent Bit v2.1.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/06/21 13:44:02] [ info] [fluent bit] version=2.1.5, commit=5006b264fd, pid=23132
[2023/06/21 13:44:02] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/06/21 13:44:02] [ info] [cmetrics] version=0.6.2
[2023/06/21 13:44:02] [ info] [ctraces ] version=0.3.1
[2023/06/21 13:44:02] [ info] [input:tail:tail.0] initializing
[2023/06/21 13:44:02] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/06/21 13:44:03] [ info] [sp] stream processor started
[2023/06/21 13:44:03] [ info] [output:stdout:stdout.0] worker #0 started
[2023/06/21 13:44:04] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1504 errno=2] No such file or directory
[2023/06/21 13:44:06] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1504 errno=2] No such file or directory
[0] tail.0: [[1687369443.775920500, {}], {"jde_log_file"=>"\\myshare\mydir\test.txt", "log"=>"test1"}]
[1] tail.0: [[1687369443.775940800, {}], {"jde_log_file"=>"\\myshare\mydir\test.txt", "log"=>"test2"}]
[2] tail.0: [[1687369443.775953700, {}], {"jde_log_file"=>"\\myshare\mydir\test.txt", "log"=>"test3"}]
[2023/06/21 13:44:08] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1504 errno=2] No such file or directory
[2023/06/21 13:44:11] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1504 errno=2] No such file or directory
[2023/06/21 13:44:13] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1504 errno=2] No such file or directory
[2023/06/21 13:44:16] [error] [C:\fluent-bit\plugins\in_tail\tail_file.c:1504 errno=2] No such file or directory
[2023/06/21 13:44:17] [engine] caught signal (SIGINT)
[2023/06/21 13:44:17] [ warn] [engine] service will shutdown in max 5 seconds
[2023/06/21 13:44:17] [ info] [input] pausing tail.0
[2023/06/21 13:44:18] [ info] [engine] service has stopped (0 pending tasks)
[2023/06/21 13:44:18] [ info] [input] pausing tail.0
[2023/06/21 13:44:18] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/06/21 13:44:18] [ info] [output:stdout:stdout.0] thread worker #0 stopped

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[Shreshth1]

hey guys
any luck on this issue? even i am seeing errors like below and was wondering if there is a way to fix it
[2023/09/21 05:31:11] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1515 errno=2] No such file or directory
[2023/09/21 05:31:11] [error] [C:\projects\fluent-bit-2e87g\plugins\in_tail\tail_file.c:1515 errno=2] No such file or directory

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[Kenifx]

Hi All,

Just wondering if previous issue has been resolved.
I encountered a similar issue using fluent bit on Windows server 2019 to monitor files on UNC (map the network drive locally)
File can be found but getting [2024/07/03 22:42:17] [error] [D:\a\fluent-bit\fluent-bit\plugins\in_tail\tail_file.c:1639 errno=2] No such file or directory

[2024/07/03 22:42:17] [debug] [input:tail:tail.0] 1 new files found on path 'Z:\My Docs\Log_7_4_2024.txt'
[2024/07/03 22:42:17] [debug] [file:file.0] created event channels: read=844 write=848
[2024/07/03 22:42:17] [debug] [stdout:stdout.1] created event channels: read=880 write=884
[2024/07/03 22:42:17] [debug] [router] match rule tail.0:file.0
[2024/07/03 22:42:17] [debug] [router] match rule tail.0:stdout.1
[2024/07/03 22:42:17] [ info] [sp] stream processor started
[2024/07/03 22:42:17] [ info] [output:stdout:stdout.1] worker #0 started
[2024/07/03 22:42:17] [ info] [output:file:file.0] worker #0 started
[2024/07/03 22:42:17] [debug] [input:tail:tail.0] inode=9367503187713458176 file=Z:\My Docs\Log_7_4_2024.txt promote to TAIL_EVENT
[2024/07/03 22:42:17] [error] [D:\a\fluent-bit\fluent-bit\plugins\in_tail\tail_file.c:1639 errno=2] No such file or directory