Add host container persistent storage & persist admin ctr ssh host keys #450

etungsten · 2019-10-23T19:23:44Z

Issue #, if available: Fixes #324

Description of changes:
Creates modelled type ValidIdentifier for things like container names.
Adds persistent storage for host containers
Generates host SSH keys in admin container's persistent storage location.
Updated ssh_config and ssh script accordingly.

Testing:
Unit tests for modelled-type ValidIdentifier passes.
Launch Thar instance, admin container and control container comes up successfully

bash-5.0# systemctl status host-containers@admin
● [email protected] - Host container: admin
   Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-10-23 19:14:39 UTC; 3min 14s ago
  Process: 2490 ExecStartPre=/usr/bin/mkdir -m 777 -p ${LOCAL_DIR}/host-containers/admin (code=exited, status=0/SUCCESS)
 Main PID: 2548 (host-ctr)
    Tasks: 21
   Memory: 43.3M
   CGroup: /system.slice/system-host\x2dcontainers.slice/[email protected]
           └─2548 /usr/bin/host-ctr -ctr-id=admin -source=722737851570.dkr.ecr.us-west-2.amazonaws.com/thar-admin-container-image:latest -superpowered=true

Oct 23 19:14:40 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: time="2019-10-23T19:14:40Z" level=info msg=Unpacking... img="ecr.aws/arn:aws:ecr:u
s-west-2:722737851570:repository/thar-admin-container-image:latest"
Oct 23 19:14:40 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: time="2019-10-23T19:14:40Z" level=info msg="Tagging image" imageName="722737851570
.dkr.ecr.us-west-2.amazonaws.com/thar-admin-container-image:latest"
Oct 23 19:14:40 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: time="2019-10-23T19:14:40Z" level=info msg="No clean up necessary, proceeding" ctr
-id=admin
Oct 23 19:14:41 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: time="2019-10-23T19:14:41Z" level=info msg="Successfully started container task"
Oct 23 19:14:41 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: rsa key already exists, will use existing key.
Oct 23 19:14:41 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: ecdsa key already exists, will use existing key.
Oct 23 19:14:41 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: ed25519 key already exists, will use existing key.
Oct 23 19:14:41 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: Server listening on 0.0.0.0 port 22.
Oct 23 19:14:41 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: Server listening on :: port 22.
Oct 23 19:14:43 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2548]: Accepted publickey for ec2-user from 205.251.233.178 port 30284 ssh2: RSA SHA256:+
qAlN881Y/8Rza0+mOmjviCFvfvNWhvHX5q/F+myiSE
bash-5.0# systemctl status host-containers@control
● [email protected] - Host container: control
   Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/[email protected]; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-10-23 19:14:39 UTC; 3min 25s ago
  Process: 2508 ExecStartPre=/usr/bin/mkdir -m 777 -p ${LOCAL_DIR}/host-containers/control (code=exited, status=0/SUCCESS)
 Main PID: 2589 (host-ctr)
    Tasks: 20
   Memory: 36.3M
   CGroup: /system.slice/system-host\x2dcontainers.slice/[email protected]
           └─2589 /usr/bin/host-ctr -ctr-id=control -source=328549459982.dkr.ecr.us-west-2.amazonaws.com/thar-control:v0.1 -superpowered=false

Oct 23 19:15:26 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:26 INFO [StartupProcessor] Unable to open serial port /dev/ttyS0:
 open /dev/ttyS0: no such file or directory
Oct 23 19:15:26 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:26 INFO [StartupProcessor] Attempting to use different port (PV):
 /dev/hvc0
Oct 23 19:15:26 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:26 INFO [StartupProcessor] Unable to open serial port /dev/hvc0: 
open /dev/hvc0: no such file or directory
Oct 23 19:15:26 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:26 ERROR [StartupProcessor] Error opening serial port: open /dev/
hvc0: no such file or directory
Oct 23 19:15:26 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:26 ERROR [StartupProcessor] Error opening serial port: open /dev/
hvc0: no such file or directory. Retrying in 5 seconds...
Oct 23 19:15:31 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:31 ERROR [StartupProcessor] Error occurred while opening serial p
ort: Timeout: Serial port is in use or not available
Oct 23 19:15:31 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:31 ERROR [instanceID=i-01864340d34fc765d] error occurred trying t
o start core module. Plugin name: StartupProcessor. Error: Timeout: Serial port is in use or not available
Oct 23 19:15:48 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:15:48 INFO [MessagingDeliveryService] [Association] No associations 
on boot. Requerying for associations after 30 seconds.
Oct 23 19:16:18 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:16:18 INFO [MessagingDeliveryService] [Association] Schedule manager
 refreshed with 0 associations, 0 new associations associated
Oct 23 19:17:16 ip-192-168-52-67.us-west-2.compute.internal host-ctr[2589]: 2019-10-23 19:17:16 INFO [HealthCheck] HealthCheck reporting agent health.
bash-5.0#

Host keys are being generated once by the admin container in persistent storage and reused:

$ ssh [email protected]          
The authenticity of host 'ec2-34-220-167-196.us-west-2.compute.amazonaws.com (34.220.167.196)' can't be established.
ECDSA key fingerprint is SHA256:L+YAmQenNjVNdwS11cKOeuStZtlwpiy+vlz+jhN4sTU.       
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-34-220-167-196.us-west-2.compute.amazonaws.com,34.220.167.196' (ECDSA) to the list of known hosts.
Welcome to Thar's Handy Administrator Resources (the admin container)!                                                                          
                   
This container provides access to the Thar host filesystems (see
/.thar/rootfs) and contains common tools for inspection and troubleshooting.
It is based on Amazon Linux 2, and most things are in the same places you would
find them on an AL2 host.
              
To permit more intrusive troubleshooting, including actions that mutate the
running state of the Thar host, we provide a tool called "sheltie" (`sudo sheltie`).
When run, this tool drops you into a root shell in the Thar host's root filesystem.
[ec2-user@ip-192-168-52-67 ~]$ sudo sheltie
bash-5.0# ls
bin  boot  dev  etc  home  lib  lib64  local  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  x86_64-thar-linux-gnu
bash-5.0# cd local/host-containers/                                                                    
bash-5.0# ls                                                            
admin  control     
bash-5.0# cd admin/                                              
bash-5.0# ls                                                          
etc
bash-5.0# cd etc/ssh/        
bash-5.0# ls
ssh_host_ecdsa_key  ssh_host_ecdsa_key.pub  ssh_host_ed25519_key  ssh_host_ed25519_key.pub  ssh_host_rsa_key  ssh_host_rsa_key.pub
**************** Restarted instance *************************
bash-5.0# Connection to ec2-34-220-167-196.us-west-2.compute.amazonaws.com closed by remote host.
Connection to ec2-34-220-167-196.us-west-2.compute.amazonaws.com closed.           
etung in ~ took 35s                        
******************** Public keys accepted ****************
$ ssh [email protected]
Welcome to Thar's Handy Administrator Resources (the admin container)!
                                          
This container provides access to the Thar host filesystems (see
/.thar/rootfs) and contains common tools for inspection and troubleshooting.
It is based on Amazon Linux 2, and most things are in the same places you would
find them on an AL2 host.                                    
                                                                 
To permit more intrusive troubleshooting, including actions that mutate the
running state of the Thar host, we provide a tool called "sheltie" (`sudo sheltie`).
When run, this tool drops you into a root shell in the Thar host's root filesystem.
[ec2-user@ip-192-168-52-67 ~]$ sudo sheltie            
bash-5.0# ls -al /local/host-containers/admin/etc/ssh/                                                                              
total 32                                                                   
drwxr-xr-x 2 root root 4096 Oct 23 19:10 .                                                                              
drwxr-xr-x 3 root root 4096 Oct 23 19:10 ..
-rw------- 1 root root  227 Oct 23 19:10 ssh_host_ecdsa_key
-rw-r--r-- 1 root root  210 Oct 23 19:10 ssh_host_ecdsa_key.pub
-rw------- 1 root root  452 Oct 23 19:10 ssh_host_ed25519_key                         
-rw-r--r-- 1 root root  130 Oct 23 19:10 ssh_host_ed25519_key.pub                                                                                          
-rw------- 1 root root 1679 Oct 23 19:10 ssh_host_rsa_key
-rw-r--r-- 1 root root  430 Oct 23 19:10 ssh_host_rsa_key.pub

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

etungsten · 2019-10-23T20:06:04Z

I've created and uploaded a new thar-admin image with the changes in this PR and tagged it as experimental and v0.2

zmrow

Nice work!

💴

workspaces/api/apiserver/src/modeled_types.rs

packages/workspaces/[email protected]

workspaces/api/apiserver/src/modeled_types.rs

etungsten · 2019-10-23T22:54:58Z

Addresses subset of @tjkirch 's comments.
Changed ValidIdentifier to Identifier
Updated comments

packages/workspaces/[email protected]

iliana · 2019-10-23T23:25:36Z

workspaces/api/apiserver/src/model.rs

@@ -32,7 +32,7 @@ pub struct Settings {
    pub updates: Option<UpdatesSettings>,

    #[serde(skip_serializing_if = "Option::is_none")]
-    pub host_containers: Option<HashMap<SingleLineString, ContainerImage>>,
+    pub host_containers: Option<HashMap<Identifier, ContainerImage>>,


Does this require a migration?

I don't believe so? Both SingleLineString and Identifier will accept what we're passing currently for container names, e.g. admin and control.

They accept what we're currently passing, but Identifier is more restrictive than SingleLineString, and we don't know what our customers are passing to it.

I think you're right, a sufficiently interesting value here could fail to parse.

While technically right, I'm not sure it's worth writing a migration here; we don't expect this has been used yet, and we'd have to mangle the user's naming tremendously to have a backwards-capable migration.

workspaces/api/apiserver/src/modeled_types.rs

Creates a new modeled type named Identifier for string used to identify container names which might be used to create files/directories

Creates a persistent storage location for containers under /local/host-containers/CONTAINER-NAME and mapped into the container under /.thar/host-containers/CONTAINER-NAME

Generates host ssh keys in admin container persistent storage under `/.thar/host-containers/admin/etc/ssh`

etungsten · 2019-10-23T23:35:22Z

Addresses @iliana 's comments.

Creates host container storage directory with mode 1777 instead of 777
Removes an inapplicable comment.

etungsten requested review from tjkirch, bcressey and zmrow October 23, 2019 19:23

etungsten changed the title ~~Persist host keys~~ Add host container persistent storage & persist admin ctr ssh host keys Oct 23, 2019

etungsten requested a review from iliana October 23, 2019 19:27

etungsten force-pushed the persist-host-keys branch from 3808ab6 to 22bc60c Compare October 23, 2019 20:07

zmrow approved these changes Oct 23, 2019

View reviewed changes

tjkirch reviewed Oct 23, 2019

View reviewed changes

etungsten force-pushed the persist-host-keys branch from 22bc60c to 4bf81d9 Compare October 23, 2019 22:54

iliana reviewed Oct 23, 2019

View reviewed changes

etungsten added 3 commits October 23, 2019 16:33

api: create new modeled type Identifier for container names

7fe8223

Creates a new modeled type named Identifier for string used to identify container names which might be used to create files/directories

host-ctr: adds persistent storage for host containers

37ef128

Creates a persistent storage location for containers under /local/host-containers/CONTAINER-NAME and mapped into the container under /.thar/host-containers/CONTAINER-NAME

admin-ctr: generate the host ssh keys in container persistent storage

b13e2cb

Generates host ssh keys in admin container persistent storage under `/.thar/host-containers/admin/etc/ssh`

etungsten force-pushed the persist-host-keys branch from 4bf81d9 to b13e2cb Compare October 23, 2019 23:34

tjkirch approved these changes Oct 24, 2019

View reviewed changes

iliana approved these changes Oct 24, 2019

View reviewed changes

etungsten merged commit d38899d into develop Oct 24, 2019

etungsten deleted the persist-host-keys branch October 24, 2019 21:11

iliana added this to the v0.2.0 milestone Nov 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add host container persistent storage & persist admin ctr ssh host keys #450

Add host container persistent storage & persist admin ctr ssh host keys #450

etungsten commented Oct 23, 2019 •

edited

Loading

etungsten commented Oct 23, 2019 •

edited

Loading

zmrow left a comment

etungsten commented Oct 23, 2019 •

edited

Loading

iliana Oct 23, 2019

etungsten Oct 23, 2019

iliana Oct 23, 2019

sam-aws Oct 23, 2019

tjkirch Oct 24, 2019

etungsten commented Oct 23, 2019

Add host container persistent storage & persist admin ctr ssh host keys #450

Add host container persistent storage & persist admin ctr ssh host keys #450

Conversation

etungsten commented Oct 23, 2019 • edited Loading

etungsten commented Oct 23, 2019 • edited Loading

zmrow left a comment

Choose a reason for hiding this comment

etungsten commented Oct 23, 2019 • edited Loading

iliana Oct 23, 2019

Choose a reason for hiding this comment

etungsten Oct 23, 2019

Choose a reason for hiding this comment

iliana Oct 23, 2019

Choose a reason for hiding this comment

sam-aws Oct 23, 2019

Choose a reason for hiding this comment

tjkirch Oct 24, 2019

Choose a reason for hiding this comment

etungsten commented Oct 23, 2019

etungsten commented Oct 23, 2019 •

edited

Loading

etungsten commented Oct 23, 2019 •

edited

Loading

etungsten commented Oct 23, 2019 •

edited

Loading