Activatior issue: The cold start time significantly increases when rw.podsAddressable is set to false.

[kinderyj]

Knative v1.11 on Kubernetes 1.27
Issue Description:
The cold start time significantly increases when rw.podsAddressable is set to false.

The podsAddressable flag is set to false here.

After restarting the Activator pod, the cold start time is quickly as expect. However, after a period of time ( 2 hours or maybe 10 hours), podsAddressable is set to false again, leading to slower cold start times.

For example, we initiate a cold start every 10 minutes:

From 9:00 AM to 12:00 PM: Cold start times are consistently around 800ms.
At 12:01 PM: rw.podsAddressable is set to false by the activator, causing cold start times to increase drastically (e.g., 2000ms, 5000ms, or even 10000ms).

[kinderyj]

Accourding to #5820 , my cluster does not enabled mesh, and can communicating with the individual pods.

[kinderyj]

/kind bug

[github-actions]

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Reopen the issue with /reopen. Mark the issue as
fresh by adding the comment /remove-lifecycle stale.