- Introduction
- Architecture
- Prerequisites
- Deployment Steps
- Validation
- Verify the pod-to-service communication
- Tear Down
- Troubleshooting
- Deleting Resources Manually
- Relevant Materials
Google Cloud networking with Kubernetes Engine clusters can be complex. This demo strives to simplify the best practices for exposing cluster services to other clusters and establishing network links between Kubernetes Engine clusters running in separate projects.
This project contains a set of Deployment Manager templates that allows a user to create networks, subnets, and Kubernetes Engine clusters. This project demonstrates the following best practices.
- Network design of launching Kubernetes Engine clusters in custom networks.
- Assigning node CIDR, container CIDR and service CIDR for Kubernetes Engine clusters.
- IP range management.
- Exposing pods of Kubernetes Engine clusters over peered networks
This example also includes Kubernetes manifests for:
- Deploying the Nginx pods in clusters.
- Exposing the Nginx pods of the clusters with different types of services like cluster IP, nodeport, internal load balancer, Network Load Balancer and Ingress.
- Validating the pod-to-service communication over the peered networks.
The execution of this code in the GCP environment creates two custom GCP networks connected via VPC peering. Each network will have two subnets one in the us-west1 region and the other in the us-central1 region. Each of the subnets hosts a Kubernetes Engine cluster which has nginx pods and services to expose those pods across other clusters.
Below is the detailed overview of GCP resources which will be created.
- Subnet: subnet1-us-west1 (10.1.0.0/28)
| cluster-ipv4-cidr | service-ipv4-cidr | zone | Initial Node count | Node Image |
|---|---|---|---|---|
| 10.108.0.0/19 | 10.208.0.0/20 | us-west1-b | 3 | COS |
- Subnet: subnet1-us-central1 (10.2.0.0/28)
| cluster-ipv4-cidr | service-ipv4-cidr | zone | Initial Node count | Node Image |
|---|---|---|---|---|
| 10.118.0.0/19 | 10.218.0.0/20 | us-central1-b | 3 | COS |
- Cluster IP, Nodeport, ILB, LB and Ingress services to expose pods in each of those clusters.
- VPC Peering connection with network2.
- Subnet: subnet3-us-west1 (10.11.0.0/28)
| cluster-ipv4-cidr | service-ipv4-cidr | zone | Initial Node count | Node Image |
|---|---|---|---|---|
| 10.128.0.0/19 | 10.228.0.0/20 | us-west1-c | 3 | COS |
- Subnet: subnet4-us-central1 (10.12.0.0/28)
| cluster-ipv4-cidr | service-ipv4-cidr | zone | Initial Node count | Node Image |
|---|---|---|---|---|
| 10.138.0.0/19 | 10.238.0.0/20 | us-central1-c | 3 | COS |
- Cluster IP, Nodeport, ILB, LB and Ingress services to expose pods in each of those clusters.
- VPC Peering connection with network1.
- Region for subnets and Node CIDR can be customized in /network/network.yaml.
- Cluster attributes like zone, image, node count, cluster CIDR and service CIDR can be customized in clusters/cluster.yaml.
- To add additional custom attributes to network or clusters yaml files and deployment manager scripts at /network/.py or clusters/.py needs to be updated accordingly.
A Google Cloud account and project is required for this.
Access to an existing Google Cloud project with the Kubernetes Engine service enabled If you do not have a Google Cloud account please signup for a free trial here.
Click the button below to run the demo in a Google Cloud Shell.
All the tools for the demo are installed. When using Cloud Shell execute the following command in order to setup gcloud cli.
gcloud initThis project will run on macOS, or in a Google Cloud Shell.
When not using Cloud Shell, the following tools are required.
- gcloud cli ( >= Google Cloud SDK 200.0.0 )
- bash
- kubectl - ( >= v1.11.0-gke.0 )
- jq
- Kubernetes Engine >= 1.11.0-gke.0
- Pull the code from git repo.
- Optionally, customize the configuration in .yaml files under /network/ or /clusters/ or /manifests/, if needed.
- The root folder is the "gke-networking-demos" folder.
- The "network" folder contains the manifest files and Deployment Manager templates to setup networks.
- The "clusters" folder contains the manifest files and Deployment Manager templates to create Kubernetes Engine clusters.
- The "manifests" folder contains the manifest files to create Kubernetes Engine services.
- The "gke-to-gke-peering" folder contains scripts specific to this demo.
The following steps will allow a user to
- Change directory to
gke-to-gke-peering - Run
./install.sh
- Make sure that there are no errors in the install script execution.
- Login to GCP console.
- Verify that the CIDR ranges of subnet-us-west1 and subnet-us-central1 matches the specification.
- Click on the VM instances in the Compute Engine and verify that the node IP addresses are drawn from the subnet's CIDR.
- Verify the created clusters in Kubernetes Engine. Click on the cluster hyperlink and verify that "Container address range" matches the specified cluster-ipv4-cidr.
- Click on workloads and verify that the status is OK for nginx pods.
- Click on discovery & load balancing. Verify that the cluster ip, nodeport, ILB and LB are created for cluster1.
- Click on discovery & load balancing. Verify that the cluster ip, nodeport, LB and ingress services are created for cluster2.
- Verify that cluster IP address of all the services for a cluster are drawn from service-ipv4-cidr.
- Access the endpoint for URL for external load balancer to view the nginx pods.
- Change directory to
gke-to-gke-peering - Run
./validate.sh
- Clusters in the same region communicate through the internal load balancer.
- Clusters across the different regions communicate through the global load balancer.
- All the services created to expose pods in a cluster are accessible to pods within that cluster.
- Refer to validate-pod-to-service-communication.sh script to view the commands to verify pod to service communication.
- Change directory back to project root. Run
./validate-pod-to-service-communication.shlocated in the project root directory - The above script demonstrates how the pods in cluster1 can access the local Kubernetes Engine services and the other Kubernetes Engine Internal/External load balancer services from the same or different regions.
- Change directory to
gke-to-gke-peering - Run
./cleanup.sh - Enter 'y' when prompted to delete the resources.
- Verify that the script executed with no errors.
- Verify that all the resources created are deleted.
- Remember to enable API's as mentioned in deployment steps in the project where the resources are to be created. Otherwise, API not enabled error is thrown.
- Verify that the project is associated with a valid billing account.
- Make sure to have the right permissions for the GCP account to create above GCP/Kubernetes Engine resources in project. Otherwise, permission denied error is thrown.
- Make sure that the deployments created through install script are deleted before you try to re-install the resources. Otherwise, resources will not be installed properly.
- If there are any errors in cleanup script execution, refer to steps for deleting resources manually.
- Goto Kubernetes Engine -> services. Delete all the services created through install script.
- Goto Network Services -> Load Balancing and delete the load balancers along with associated heathchecks.
- Goto Compute Engine -> VM Instances and delete all the instances created through install script.
- Goto Compute Engine -> Instance Groups and delete all the instance groups created through install script.
- Goto VPC Networks -> Firewall Rules and delete the firewall rules created for network1.
- Goto Deployment Manager -> Deployments and delete cluster and network deployments.
- Delete the dependent resources if network deployment doesn't get deleted.