Ingress for Anthos
STATUS: Draft; TASK: normalize to the AHR-* conventions;
"Ingress for Anthos is designed to meet the load balancing needs of multi-cluster, multi-regional environments. It's a controller for the external HTTP(S) load balancer to provide ingress for traffic coming from the internet across one or more clusters." [Ingress for Anthos, Concepts]
"Ingress for Anthos builds on the architecture of the External HTTP(S) Load Balancing. HTTP(S) Load Balancing is a globally distributed load balancer with proxies deployed at 100+ Google points of presence (PoPs) around the world."
NOTE: Prerequisites for registering a cluster https://cloud.google.com/anthos/multicluster-management/connect/prerequisites
If not roles/owner, then those roles are required:
roles/gkehub.admin roles/iam.serviceAccountAdmin roles/iam.serviceAccountKeyAdmin roles/resourcemanager.projectIamAdminRequired APIs
gcloud services enable --project=$PROJECT \ container.googleapis.com \ gkeconnect.googleapis.com \ gkehub.googleapis.com \ cloudresourcemanager.googleapis.comGrant users the cluster-admin RBAC role. Check it
kubectl create clusterrolebinding [BINDING_NAME] --clusterrole cluster-admin --user [USER] # check role binding for both region clusters kubectl --context=dc1-cluster auth can-i '*' '*' --all-namespaces kubectl --context=dc2-cluster auth can-i '*' '*' --all-namespaces
?. Verify gkehub API status
gcloud services list --available | grep gkehub.googleapis.com
gkehub.googleapis.com GKE Hub
?. Enable gkehub API
gcloud services enable gkehub.googleapis.com
?. Enable the Ingress for Anthos API for your project:
gcloud services enable multiclusteringress.googleapis.com
export GKEHUB_ID=gkehub
export GKEHUB_SA=$SA_DIR/$ORG-gkehub.json
export GKEHUB_SA_ID=$GKEHUB_ID@$PROJECT.iam.gserviceaccount.com
gcloud iam service-accounts create $GKEHUB_ID --project=$PROJECT
gcloud projects add-iam-policy-binding $PROJECT --member="serviceAccount:$GKEHUB_SA_ID" --role="roles/gkehub.connect"
gcloud iam service-accounts keys create $GKEHUB_SA --iam-account=$GKEHUB_SA_ID --project=$PROJECT
Each cluster must be registered as a member of a Hub.
?. Find the URIs for your clusters
gcloud container clusters list --uri
export CLUSTER_URI=$(gcloud container clusters list --uri --filter="name=$CLUSTER")
export DC1_CLUSTER_URI=https://container.googleapis.com/v1/projects/emea-cs-hybrid-demo6/zones/us-east1-b/clusters/dc1-cluster
export DC2_CLUSTER_URI=https://container.googleapis.com/v1/projects/emea-cs-hybrid-demo6/zones/asia-east1-b/clusters/dc2-cluster?. Register the dc1-cluster
# dc1-cluster
(
export CLUSTER=dc1-cluster
export CLUSTER_URI=$(gcloud container clusters list --uri --filter="name=$CLUSTER")
gcloud container hub memberships register $CLUSTER --project=$PROJECT --gke-uri=$CLUSTER_URI --service-account-key-file=$GKEHUB_SA
)?. Register the dc2-cluster
# dc2-cluster
(
export CLUSTER=dc2-cluster
export CLUSTER_URI=$(gcloud container clusters list --uri --filter="name=$CLUSTER")
gcloud container hub memberships register $CLUSTER --project=$PROJECT --gke-uri=$CLUSTER_URI --service-account-key-file=$GKEHUB_SA
)?. Verify membership list
gcloud container hub memberships list
NAME EXTERNAL_ID
dc2-cluster 1d977520-6ae6-11ea-9951-42010af0007c
dc1-cluster a52d527d-6ae5-11ea-aa7b-42010a8e0fdb"Ingress for Anthos runs as a service outside of the cluster and is managed by Google Cloud. The ingress controller watches for MultiClusterIngress and MultiClusterService resources in GKE and configures load balancers and NEGs as a result.
Specifying a config cluster enables Ingress for Anthos." [IfA Concepts, Architecture]
?. Enable Ingress for Anthos and select dc1-cluster as the config cluster
gcloud alpha container hub features multiclusteringress enable --config-membership=projects/$PROJECT/locations/global/memberships/dc1-cluster
Waiting for Feature to be created...done.
WARNING: generic::failed_precondition: WARNING: multiclusteringress feature will eventually require enablement of anthos.googleapis.com
?. Verify config cluster status
gcloud alpha container hub features multiclusteringress describe
featureState:
details:
code: OK
description: Multicluster Ingress requires Anthos license enablement. Unlicensed
usage is unrestricted for the MCI Beta API. Note that licensing will be enforced
for use of the Generally Available MCI API.
detailsByMembership:
projects/755738281610/locations/global/memberships/dc1-cluster:
code: OK
description: CRDs are established
lifecycleState: ENABLED
multiclusteringressFeatureSpec:
configMembership: projects/emea-cs-hybrid-demo6/locations/global/memberships/dc1-cluster
name: projects/emea-cs-hybrid-demo6/locations/global/features/multiclusteringress
updateTime: '2020-03-24T21:22:55.881503252Z'
NOTE: While lifecycleState is marked as enabled, this feature is not ready to use until your config cluster has an OK status.
The Ingress provides a shared virtual IP (VIP) address for the app deployments.
To create the load balancer, you need two resources: a MultiClusterIngress and one or more MultiClusterServices. MultiClusterIngress and MultiClusterService objects are multi-cluster analogs to the existing Kubernetes Ingress and Service resources used in the single cluster context.
?. Create a static global IP
gcloud compute addresses create apigee-mci --global
Created [https://www.googleapis.com/compute/v1/projects/emea-cs-hybrid-demo6/global/addresses/apigee-mci].
?. Verify status of the IP address
gcloud compute addresses describe apigee-mci --global
**address: 35.190.60.112**
addressType: EXTERNAL
creationTimestamp: '2020-03-26T05:16:29.168-07:00'
description: ''
id: '2585767625150768114'
ipVersion: IPV4
kind: compute#address
name: apigee-mci
networkTier: PREMIUM
selfLink: https://www.googleapis.com/compute/v1/projects/emea-cs-hybrid-demo6/global/addresses/apigee-mci
status: RESERVED?. Copy TLS secret between namespaces
# TODO: Debug:
# kubectl get secret emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs --namespace=istio-system -o yaml | yq w - data.namespace apigee-mci| yq d - metadata | kubectl apply -f -
# ns: apigee
kubectl -n apigee create secret tls emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs --key $RUNTIME_SSL_KEY --cert $RUNTIME_SSL_CERT
# ns: apigee-mci
kubectl -n apigee-mci create secret tls emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs --key $RUNTIME_SSL_KEY --cert $RUNTIME_SSL_CERT
?. Deifine mcs manifest
cat <<EOT > apigee-mcs.yaml
apiVersion: networking.gke.io/v1beta1
kind: MultiClusterService
metadata:
name: apigee-runtime-emea-cs-hybrid-demo6-test-zone-svc
namespace: apigee
annotations:
networking.gke.io/app-protocols: '{"https-8443":"HTTPS"}'
beta.cloud.google.com/backend-config: '{"ports": {"8443":"zone-health-check-cfg"}}'
spec:
template:
spec:
selector:
com.apigee.apigeedeployment: apigee-runtime-emea-cs-hybrid-demo6-test
ports:
- name: https-8443
protocol: TCP
port: 8443
targetPort: 8443
clusters:
- link: "us-east1-b/dc1-cluster"
- link: "asia-east1-b/dc2-cluster"
EOT?. Apply manifest
kubectl apply -f apigee-mcs.yaml?. Verify mcs The derived Service resource created by MCS are of Headless type. Therefore, None is an expected value of the ClusterIP property.
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
...
mci-apigee-runtime-emea-cs-hybrid-demo6--svc-ybxkj03wo8mpnbtu ClusterIP None <none> 8443/TCP 5d2h
?. Define Manifest
cat <<EOT > apigee-bec.yaml
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
name: zone-health-check-cfg
namespace: apigee
spec:
healthCheck:
checkIntervalSec: 5
timeoutSec: 5
healthyThreshold: 2
unhealthyThreshold: 3
type: HTTPS
port: 8443
requestPath: /ping
EOT?. Apply manifest
kubectl apply -f apigee-mcs.yaml
To be exposed by the same Ingress, these characteristics must be the same across clusters:
- Be deployed into a namespace with the same name across clusters.
- Share the same set of labels so they can be selected as a unit across the clusters.
?. Define MultiClusterService object
cat <<EOT > apigee-mci.yaml
apiVersion: networking.gke.io/v1beta1
kind: MultiClusterIngress
metadata:
name: apigee-hybrid-mci
namespace: apigee
annotations:
networking.gke.io/static-ip: 35.190.60.112
spec:
template:
spec:
backend:
serviceName: apigee-runtime-emea-cs-hybrid-demo6-test-zone-svc
servicePort: 8443
tls:
- secretName: emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs
EOT
?. Verify MCI
kubectl -n apigee-mci describe mci apigee-hybrid-mci
?. Execute curl request to /ping API
export RUNTIME_HOST_ALIAS=mc-demo6-test.hybrid-apigee.net
export RUNTIME_IP=35.190.60.112
curl --resolve "$RUNTIME_HOST_ALIAS:443:$RUNTIME_IP" https://$RUNTIME_HOST_ALIAS/ping -v
...
> GET /ping HTTP/1.1
> Host: mc-demo6.hybrid-apigee.net
> User-Agent: curl/7.52.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200
< user-agent: curl/7.52.1
< accept: */*
...
?. Trace view of the request above
It helps to be able to identify manifests and devived resources to investigate and diagnose potential problems.
?. MultiClusterService events
kubectl describe mcs apigee-runtime-emea-cs-hybrid-demo6-test-zone-svc
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SYNC 22m (x1330 over 5d2h) multi-cluster-ingress-controller Derived Service was ensured in cluster us-east1-b/dc1-cluster
Normal SYNC 57s (x1334 over 5d2h) multi-cluster-ingress-controller Derived Service was ensured in cluster asia-east1-b/dc2-cluster
?. MultiCluster Ingress Events and Status (I.e., references to Firewalls, Forwarding Rules, NEGs, etc.)
kubectl describe mci apigee-hybrid-mci
...
Secret Name: emea-cs-hybrid-demo6-test-dc1.hybrid-apigee.net-ingressgateway-certs
Status:
Cloud Resources:
Backend Services:
mci-2dy6qj-8443--i6n413-apigee-apigee-runtime-emea
Firewalls:
mci-2dy6qj-default-l7
Forwarding Rules:
mci-2dy6qj-fw-apigee-apigee-hybrid-mci
mci-2dy6qj-fws-apigee-apigee-hybrid-mci
Health Checks:
mci-2dy6qj-8443--i6n413-apigee-apigee-runtime-emea
Network Endpoint Groups:
zones/us-east1-b/networkEndpointGroups/k8s1-3b2d76ce-apig-mci-apigee-runtime-emea-cs-hybri-84-2bb37757
zones/us-east1-c/networkEndpointGroups/k8s1-3b2d76ce-apig-mci-apigee-runtime-emea-cs-hybri-84-2bb37757
zones/us-east1-d/networkEndpointGroups/k8s1-3b2d76ce-apig-mci-apigee-runtime-emea-cs-hybri-84-2bb37757
zones/asia-east1-a/networkEndpointGroups/k8s1-7c3e1506-apig-mci-apigee-runtime-emea-cs-hybri-84-627b0d18
zones/asia-east1-b/networkEndpointGroups/k8s1-7c3e1506-apig-mci-apigee-runtime-emea-cs-hybri-84-627b0d18
zones/asia-east1-c/networkEndpointGroups/k8s1-7c3e1506-apig-mci-apigee-runtime-emea-cs-hybri-84-627b0d18
Target Proxies:
mci-2dy6qj-apigee-apigee-hybrid-mci
mci-2dy6qj-apigee-apigee-hybrid-mci
URL Map: mci-2dy6qj-apigee-apigee-hybrid-mci
VIP: 35.190.60.112
Events:
...
?. Backend services
gcloud compute backend-services list
?. Network Endpoint Groups
gcloud compute network-endpoint-groups list
NOTES:
- Have some patience. It takes minutes (5-7) for mcs/mci/bec objects to be created and become eventually consistent across regions. It happens... Eventually
- Current BEC manifest uses /ping proxy as the healthcheck. This is a temporary hack until IaF will add support for TCP probes (end-of-April). Right now you can use any existing http/https probes.
- As of now, your mcs/mci/bec object need to be located in the same namespace as the pods you're exposing.
REFERENCES:
- Anycast introduction https://www.usenix.org/legacy/events/lisa10/tech/full_papers/Weiden.pdf
AHR-*-CTL 
