How to configure Healthcheck #658

daNutzzzzz · 2024-02-22T15:05:06Z

daNutzzzzz
Feb 22, 2024

Hi,

Im getting the following error randomly, how can i configure the polling/timeout values?

Health check exceeded timeout (5s)

qdm12 · 2024-02-25T15:25:57Z

qdm12
Feb 25, 2024
Maintainer

Where do you get this message from?

6 replies

qdm12 Feb 29, 2024
Maintainer

Can you share your full logs? I'm unclear on what's the console you refer to. The healthcheck shouldn't time out, and its time values are configurable by your docker config system (not configurable by the running program within the container)

daNutzzzzz Mar 1, 2024
Author

sorry ive been away..

When i referred to console i meant from the console logs that portainer captures... im still getting the issue with the warnings but now im not seeing that error in the logs, portainer and netdata are still alerting me though.. its the only container getting this issue

qdm12 Mar 1, 2024
Maintainer

Thanks for sharing more info! What are the container logs or unhealthy reason (find it with docker inspect)

daNutzzzzz Mar 1, 2024
Author

                    "Start": "2024-03-01T12:40:39.902029651Z",
                    "End": "2024-03-01T12:40:45.095112254Z",
                    "ExitCode": -1,
                    "Output": "Health check exceeded timeout (5s)"

daNutzzzzz Mar 3, 2024
Author

docker inspect ddns-updater
[
    {
        "Id": "6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115",
        "Created": "2024-03-03T03:53:17.742918611Z",
        "Path": "/updater/app",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 5141,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-03-03T03:53:24.665510549Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "unhealthy",
                "FailingStreak": 2,
                "Log": [
                    {
                        "Start": "2024-03-03T12:29:54.827276771Z",
                        "End": "2024-03-03T12:29:55.268640642Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-03T12:30:55.280689499Z",
                        "End": "2024-03-03T12:30:55.696660506Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-03T12:31:55.708718214Z",
                        "End": "2024-03-03T12:31:56.035984353Z",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-03-03T12:32:56.056296554Z",
                        "End": "2024-03-03T12:33:01.227891476Z",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    },
                    {
                        "Start": "2024-03-03T12:34:01.244164763Z",
                        "End": "2024-03-03T12:34:06.435969489Z",
                        "ExitCode": -1,
                        "Output": "Health check exceeded timeout (5s)"
                    }
                ]
            }
        },
        "Image": "sha256:adc3d0c9a6562980102c65eabca4e754285946b1aa144da12d90add79d706539",
        "ResolvConfPath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/resolv.conf",
        "HostnamePath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/hostname",
        "HostsPath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/hosts",
        "LogPath": "/mnt/ssd1/docker/docker-images/containers/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115/6bdecf4d928fd3b1ee88c8edee09c4fec5703c2eea008a1a1e56237eb8cbc115-json.log",
        "Name": "/ddns-updater",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/mnt/ssd1/docker/docker-data/DDNS-Updater/data:/updater/data:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "bridge-new",
            "PortBindings": {
                "8000/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "8001"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "ConsoleSize": [
                0,
                0
            ],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": [],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc-init/diff:/mnt/ssd1/docker/docker-images/overlay2/0e4ffb4f67d46c421ee9b0f171d0b0e151b786f5b1d34e9b108f62a86f56f8da/diff",
                "MergedDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc/merged",
                "UpperDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc/diff",
                "WorkDir": "/mnt/ssd1/docker/docker-images/overlay2/a8118db8ad36111647f6ebc49ee9293ae95624651ba48de13465ac7a9791bacc/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mnt/ssd1/docker/docker-data/DDNS-Updater/data",
                "Destination": "/updater/data",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "ddns-updater",
            "Domainname": "",
            "User": "1000:1000",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "8000/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "cpus=1.0",
                "memory-swap=100m",
                "BACKUP_DIRECTORY=/updater/data/backups",
                "HTTP_TIMEOUT=10s",
                "user=0",
                "TZ=Europe/London",
                "PERIOD=5m",
                "memory=100m",
                "BACKUP_PERIOD=72h0m",
                "memory-reservation=75m",
                "cpus-shares=256",
                "LISTENING_ADDRESS=:8000",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "CONFIG=",
                "UPDATE_COOLDOWN_PERIOD=5m",
                "PUBLICIP_FETCHERS=all",
                "PUBLICIP_HTTP_PROVIDERS=all",
                "PUBLICIPV4_HTTP_PROVIDERS=all",
                "PUBLICIPV6_HTTP_PROVIDERS=all",
                "PUBLICIP_DNS_PROVIDERS=all",
                "PUBLICIP_DNS_TIMEOUT=3s",
                "DATADIR=/updater/data",
                "RESOLVER_ADDRESS=",
                "RESOLVER_TIMEOUT=5s",
                "LISTENING_PORT=8000",
                "ROOT_URL=/",
                "LOG_LEVEL=info",
                "LOG_CALLER=hidden",
                "SHOUTRRR_ADDRESSES="
            ],
            "Cmd": null,
            "Healthcheck": {
                "Test": [
                    "CMD",
                    "/updater/app",
                    "healthcheck"
                ],
                "Interval": 60000000000,
                "Timeout": 5000000000,
                "StartPeriod": 10000000000,
                "Retries": 2
            },
            "Image": "qmcgaw/ddns-updater:v2.5.0",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "/updater/app"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "0157295188eab693ed6e3329e29c5b39a2254976b23cb889af5c63e848c1c8f9",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:adc3d0c9a6562980102c65eabca4e754285946b1aa144da12d90add79d706539",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "docker-compose",
                "com.docker.compose.project.config_files": "/mnt/ssd1/docker/docker-compose/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/mnt/ssd1/docker/docker-compose",
                "com.docker.compose.replace": "623a622aa9bb8984d1959c0ac841361c2c1105a93a2f10955f593327eabe4eb6",
                "com.docker.compose.service": "ddns-updater",
                "com.docker.compose.version": "2.16.0",
                "org.opencontainers.image.authors": "[email protected]",
                "org.opencontainers.image.created": "2023-06-07T09:11:38Z",
                "org.opencontainers.image.description": "Universal DNS updater with WebUI",
                "org.opencontainers.image.documentation": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.revision": "a6f72d9",
                "org.opencontainers.image.source": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.title": "ddns-updater",
                "org.opencontainers.image.url": "https://github.com/qdm12/ddns-updater",
                "org.opencontainers.image.version": "v2.5.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "47ec3b7c6d1e300adddc2dfedc7cafec833f65a74f65c57322efcb87a77e8b58",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "8000/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "8001"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "8001"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/47ec3b7c6d1e",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge-new": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "6bdecf4d928f",
                        "ddns-updater"
                    ],
                    "NetworkID": "74c87577f031b6f754156b220d1823b9e2462cddfc30ea1f96b1d6c4ed67180d",
                    "EndpointID": "ea93174886be9da8d5c81d2d8440bba2d82fb28438b39cfaf8f616539e4ed482",
                    "Gateway": "172.15.0.1",
                    "IPAddress": "172.15.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:0f:00:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

daNutzzzzz · 2024-03-11T15:19:26Z

daNutzzzzz
Mar 11, 2024
Author

Any thoughts?

to note i realised i was using an older version and updated to the latest but the issue still persists.

0 replies

qdm12 · 2024-03-12T07:58:47Z

qdm12
Mar 12, 2024
Maintainer

Health check exceeded timeout seems to happen after a few successful health checks (exit code 0), and the message comes from your docker daemon and not from the ddns-updater program. Maybe your Machine is overloaded sometimes?
Try increasing the healthcheck timeout (check docker compose documentation online)?

0 replies

daNutzzzzz · 2024-03-13T23:08:45Z

daNutzzzzz
Mar 13, 2024
Author

I'll look at the docker compose documentation but this is the only container on this VM/Host that is reporting any issues. There are no bottlenecks or capacity issues.

0 replies

daNutzzzzz · 2024-03-16T17:39:01Z

daNutzzzzz
Mar 16, 2024
Author

its only been a day but since adding the default compose values for healthcheck im no longer getting these alerts.

1 reply

qdm12 Mar 18, 2024
Maintainer

adding the default compose values for healthcheck

What do you mean by that? Did you increase the healthcheck timeout? 🤔

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to configure Healthcheck #658

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

How to configure Healthcheck #658

daNutzzzzz Feb 22, 2024

Replies: 5 comments · 7 replies

qdm12 Feb 25, 2024 Maintainer

qdm12 Feb 29, 2024 Maintainer

daNutzzzzz Mar 1, 2024 Author

qdm12 Mar 1, 2024 Maintainer

daNutzzzzz Mar 1, 2024 Author

daNutzzzzz Mar 3, 2024 Author

daNutzzzzz Mar 11, 2024 Author

qdm12 Mar 12, 2024 Maintainer

daNutzzzzz Mar 13, 2024 Author

daNutzzzzz Mar 16, 2024 Author

qdm12 Mar 18, 2024 Maintainer

daNutzzzzz
Feb 22, 2024

Replies: 5 comments 7 replies

qdm12
Feb 25, 2024
Maintainer

qdm12 Feb 29, 2024
Maintainer

daNutzzzzz Mar 1, 2024
Author

qdm12 Mar 1, 2024
Maintainer

daNutzzzzz Mar 1, 2024
Author

daNutzzzzz Mar 3, 2024
Author

daNutzzzzz
Mar 11, 2024
Author

qdm12
Mar 12, 2024
Maintainer

daNutzzzzz
Mar 13, 2024
Author

daNutzzzzz
Mar 16, 2024
Author

qdm12 Mar 18, 2024
Maintainer