probe fails between secondary IPs of Loxi instances

[Rammurthy5]

Describe the bug
Loxi requires secondary IP to communicate with each other for probing and respond to the service requests from the client. Currently, there needs a manual intervention to get this going. ip route add <secondary ip> via <gateway>
every time we might need to maually intervene and amend the route table appropriately. otherwise service will fail to respond.

To Reproduce
Steps to reproduce the behavior:
Set up loxi master and backup instances.
Deploy kube-loxilb.
Deploy a simple service and expose it via Loxi.
Check the probing status, and service response time.

Expected behavior
we can try and avoid manual intervention on the Loxi set up as secondary ip changes dynamically when the failover happens.

Environment (please complete the following information):

• OS: Talos Linux v1.6.6 for k8s cluster, loxi set up on ubuntu 22.04
• Kernel Version: 6.5.0-1023-aws
• LoxiLB Version: latest
• Cloud Environment: AWS
• K8s Version 1.28.3

[TrekkieCoder]

Did some further testing on it. As long as source/dest check is disabled, it seems to work fine. Please check AWS doc related to "Source/destination checking".

[TrekkieCoder]

Even with source/dest check disabled, few issues were identified and fixed in latest image. Please double check !!