EC2 Ubuntu instance health check failure: SSM get correct "Online" Output response from instance, but UI shows Offline

[owenCCY]

Describe the bug

Customer using CLO 2.1.1 （upgraded from CLO1.0.x）
They have more than 100 instances.
They are using Ubuntu 22.04, and installed flb agent, SSM get correct "Online" Output response from instance, but UI shows Offline.

Output:
{"fluent-bit":{"version":"1.9.10","edition":"Community","flags":["FLB_HAVE_IN_STORAGE_BACKLOG","FLB_HAVE_PARSER","FLB_HAVE_RECORD_ACCESSOR","FLB_HAVE_STREAM_PROCESSOR","FLB_HAVE_TLS","FLB_HAVE_OPENSSL","FLB_HAVE_METRICS","FLB_HAVE_AWS","FLB_HAVE_AWS_CREDENTIAL_PROCESS","FLB_HAVE_SIGNV4","FLB_HAVE_SQLDB","FLB_HAVE_METRICS","FLB_HAVE_HTTP_SERVER","FLB_HAVE_SYSTEMD","FLB_HAVE_VALGRIND","FLB_HAVE_FORK","FLB_HAVE_TIMESPEC_GET","FLB_HAVE_GMTOFF","FLB_HAVE_UNIX_SOCKET","FLB_HAVE_LIBYAML","FLB_HAVE_ATTRIBUTE_ALLOC_SIZE","FLB_HAVE_PROXY_GO","FLB_HAVE_JEMALLOC","FLB_HAVE_LIBBACKTRACE","FLB_HAVE_REGEX","FLB_HAVE_UTF8_ENCODER","FLB_HAVE_LUAJIT","FLB_HAVE_C_TLS","FLB_HAVE_ACCEPT4","FLB_HAVE_INOTIFY","FLB_HAVE_GETENTROPY","FLB_HAVE_GETENTROPY_SYS_RANDOM"]}}

Expected Behavior

Instance Online

Current Behavior

Instance Offline

Reproduction Steps

Use CLO 2.1.1, have more then 1 page of instances, install agent in Ubuntu 22.04 then click load more.

Possible Solution

No response

Additional Information/Context

No response

Solution Version

2.1.1

AWS Region. e.g., us-east-1

No response

Other information

No response

[owenCCY]

Base on customer environment check, their frontend sends duplicate instance ids into SSM client, causing api error:
An error occurred (DuplicateInstanceId) when calling the SendCommand operation:

Checked their UI, duplicate instances are listed in the frontend, the api call getInstanceAgentStatus sends duplicate instance ids, causing the above issue.

The solution applied for customer:
Add dedupe code in CentralizedLogging-APIInstanceAPIInstanceAgentStatus
from : instance_list = args.get("instanceIds", list())
to: instance_list = list(set(args.get("instanceIds", list())))

[owenCCY]

Base on reproduction test, we do not see the same issue in the release version (2.1.1 and above).

Will keep watching if upgraded customers have the same issue.

[Jiale-Fang]

I am still encountering this issue. In the AWS Solution OpenSearch UI, the status alternates between showing "online" and "offline". Actually, i do see ssm run the command and successfully get the fluent bit information. But i am not sure why UI can not fetch that information

CLO Version: 2.2.0
EC2 Operating System: Windows Server 2022

[James96315]

@Jiale-Fang , in v2.2.0, the auto-refresh interval is 3 minutes. You can upgrade to v2.2.1 and we have changed it to 10 seconds.

[Jiale-Fang]

Thank you @James96315 . Actually, my question is: I see Fluent Bit is running fine on my Windows 2022 machine, but in the AWS CLO console, it cannot correctly get the agent status. Sometimes it displays as online, and sometimes it displays as offline. And in systems manager command history, i can see commands' output are correctly getting fluent bit's version info.

[James96315]

@Jiale-Fang , can you upgrade the CLO to the latest version. Also, what is the instance type you are using? If the instance type is t2.micro, the detection will be slower. If you still have questions, we can set up a time to take a look at it remotely!

[Jiale-Fang]

Hi @James96315 , Thank you for your response. Unfortunately, I am unable to upgrade to version 2.2.1 at this time. I am currently using CLO version 2.2.0 and installing fluent-bit in t3a.large instances. When i fetch the agent status for windows instances, Sometimes it displays as online, and sometimes it displays as offline. However, this doesn't affect my normal usage. Please feel free to contact me if you need more information. Thanks!