Kubernetes
defines the concept virtual cluster
via the namespace
resource type, we will take the advantage of that concept for our maintenance.
- Both
aranya
's deployment andEdgeDevice
s are namespaced (see Behaviors and Tips for more information). Node
objects inKubernetes
are not namespaced but those managed byaranya
will contain a taint fornamespace
name.- You need to create workloads with tolerations if you do expect the workload to run in edge devices.
-
aranya
watches thenamespace
it has been deployed to by default, reconcilesEdgeDevice
s created in the samenamespace
.- You can deploy
aranya
to watch somenamespace
other than the one it was deployed to, but is discouraged if you are not working on something stated in docs/Multi-tenancy.md.
- You can deploy
-
Only one
aranya
instance in the samenamespace
will work as leader to do the reconcile job, servingkubelet
servers and connectivity managers.- Deploy
aranya
to mutiplenamespace
s if you have quite a lot (tipically more than 500)EdgeDevice
s to deploy, you should group theseEdgeDevice
s into differentnamespace
s with multiplearanya
instance.
- Deploy
-
aranya
requires host network to work properly.- Deploy multiple
aranya
in the samenamespace
to differentNode
s to avoid single point failure, you can achieve this withanti-affinity
.
- Deploy multiple
-
aranya
will request node certifications for each one of theEdgeDevice
s you have deployed. The node certification includes theNode
's address(es) and hostname(s) (Here theNode
is the onearanya
deployed to).- Use
StatefulSet
ornodeAffinity
to avoid unexpected certification regenation whenaranya
is deployed to different nodes. - Changes to
Node
's address(es) or hostname(s) whenaranya
has been serving thekubelet
servers and connectivity managers (which should be the rare case) for edge devices would result in connectivity failure and remote management failure, you need to restartaranya
to solve this problem.
- Use
Kubernetes
doesn't allow users to directly control the node via kubectl
, but with the help of privileged Pod
, we can do some host maintenance work. (Check out my device plugin arhat-dev/kube-host-pty to enable host access via kubectl
if you are interested)
I would say, it's the best practice for cloud computing to isolate maintenance and deployment with such barrier, but it's not so good in the case of edge devices, because in this way:
If using ssh
(or ansible
with ssh
) for device management (which enables full management):
- Edge devices need to expose ssh service port for remote management
- Most attack to IoT network happens to ssh services
- SSH key management in remote devices is another big problem
- Connectivity through network with NAT and Firewall requires extra effort (such as
frp
service)
If using privileged Pod
for management:
- We need to maintain a set of management container images
- To download when needed
- data usage is a thing
- To store preflight
- storage may be a concern
- To download when needed
So what if we need to be able to access our edge device at any place and at any time, once the edge device has been online?
aranya
and arhat
solves this problem with the concept virtual pod
virtual pod
is a Pod
object only living in Kubernetes
cluster and never will be deployed to your edge devices, its phase is always Running
. Any kubectl
command to the virtual pod
such as logs
, exec
, attach
and port-forward
will be treated as works to do in device host.
No ssh service exposure, no actual Pod
deployment, maintenance work done right for edge devices ;)
Thanks to protobuf
, we are able to upgrade both aranya
and arhat
separately and smoothly, without any interruption caused by protocol inconsistency (except when we introduced significant changes to our proto files and it's no longer compatible with older versions.)
To upgrade aranya
, just use the strategy Kubernetes
defined for cloud native applications.
NOTICE: EdgeDevice
s using grpc
as connectivity method will always experience a temporary connection failure when upgrading aranya
.
To upgrade arhat
, we can do upgrade work with the help of virtual pod
and kubectl
(say we have a EdgeDevice
called my-edge-device
):
-
Download the new
arhat
-
Option 1: Download in the form of a docker image
- Package new
arhat
into a docker image (useFROM scratch
) - Pull the docker image for new
arhat
and savearhat
to local filekubectl exec my-edge-device -- docker pull arhatdev/arhat-docker-grpc:latest
(use sha256 tag if preferred)kubectl exec my-edge-device -- docker image save -o ~/new-arhat-docker-grpc.tar arhatdev/arhat-docker-grpc:latest
kubectl exec my-edge-device -- tar xf ~/new-arhat-docker-grpc.tar -C ~/new-arhat-docker-grpc
- Delete the image with
kubectl exec my-edge-device -- docker rmi arhatdev/arhat-docker-grpc:latest
- Package new
-
Option 2: Download binary release directly from Internet or your own file server (always check the checksum)
kubectl exec my-edge-device -- curl -O ~/new-arhat-docker-grpc http[:]//arhat-release-site
kubectl exec my-edge-device -- sha256sum ~/new-arhat-docker-grpc
-
-
Find the old
arhat
's PIDexport OLD_ARHAT_PID=$(kubectl exec my-edge-device -- ps -ax | grep arhat | awk '{ print $1 }')
echo "${OLD_ARHAT_PID}"
-
Start the new
arhat
with proper configuration and make sure it will always be runingkubectl exec my-edge-device -- nohup /path/to/new-arhat-docker-grpc -c /path/to/arhat-config.yaml &
kubectl exec my-edge-device -- ps -ax | grep arhat | wc -l
(should be a2
there)- Watch the new
arhat
process with tools like top
-
Stop the old
arhat
and the newarhat
will communicate withKubernetes
from now onkubectl exec my-edge-device -- kill ${OLD_ARHAT_PID}
Note: You can automate these steps using shell scripts or by extending ansible
to use kubectl
, since they are just commands and args.