From a48ce741fb4677b643a96dac1ed5b012113cc4ad Mon Sep 17 00:00:00 2001 From: Navraj Singh Chhina Date: Fri, 22 Mar 2019 17:37:48 -0400 Subject: [PATCH] KubeauditConfig feature (#193) * define schema for config file and add cobra flag to specify config * initialize root logic for kubeauditConfig * push logic to PodOverride, prior to testing * pre-testing build * add support for custom capabilities * finalize tests and update readme * cleanup the mess * add override for netpols * add netpol feature to auditconfig override and add suggestion * add from config to file names * set nill auditConfig for test * reset auditConfig flag after use in testing suite * reset back * get rid extra newline * temp commit to change filenames to lowecase * revert back * oops forgot about the README * increase test coverage * add headsup for unmarshling unreachability --- README.md | 54 +++++++++++-- cmd/allowPrivilegeEscalation_test.go | 9 +++ cmd/automountServiceAccountToken_test.go | 8 ++ cmd/capabilities.go | 62 ++++++++------ cmd/capabilities_test.go | 31 +++++-- cmd/capabilities_util.go | 81 +++++++++++++++++++ cmd/config.go | 76 +++++++++++++++++ cmd/config_test.go | 11 +++ cmd/networkPolicies_test.go | 23 ++++++ cmd/privileged_test.go | 8 ++ cmd/readOnlyRootFilesystem_test.go | 8 ++ cmd/root.go | 42 +++++++--- cmd/runAsNonRoot_test.go | 11 +++ cmd/util.go | 41 ++++++++++ configs/allow_audit_from_config.yml | 3 + ...ount_service_account_token_from_config.yml | 6 ++ ...efault_deny_egress_net_pol_from_config.yml | 6 ++ ...fault_deny_ingress_net_pol_from_config.yml | 6 ++ ...ssing_default_deny_net_pol_from_config.yml | 7 ++ ...allow_privilege_escalation_from_config.yml | 6 ++ configs/allow_privileged_from_config.yml | 6 ++ ...only_root_filesystem_false_from_config.yml | 6 ++ configs/allow_run_as_non_root_from_config.yml | 6 ++ .../custom_capabilities_to_be_dropped_v1.yml | 20 +++++ configs/drop_Cap_Config_Manifest_v1.yml | 16 ---- configs/kubeauditConfig.yaml | 30 +++++++ 26 files changed, 523 insertions(+), 60 deletions(-) create mode 100644 cmd/capabilities_util.go create mode 100644 cmd/config.go create mode 100644 cmd/config_test.go create mode 100644 configs/allow_audit_from_config.yml create mode 100644 configs/allow_automount_service_account_token_from_config.yml create mode 100644 configs/allow_namespace_missing_default_deny_egress_net_pol_from_config.yml create mode 100644 configs/allow_namespace_missing_default_deny_ingress_net_pol_from_config.yml create mode 100644 configs/allow_namespace_missing_default_deny_net_pol_from_config.yml create mode 100644 configs/allow_privilege_escalation_from_config.yml create mode 100644 configs/allow_privileged_from_config.yml create mode 100644 configs/allow_read_only_root_filesystem_false_from_config.yml create mode 100644 configs/allow_run_as_non_root_from_config.yml create mode 100644 configs/custom_capabilities_to_be_dropped_v1.yml delete mode 100644 configs/drop_Cap_Config_Manifest_v1.yml create mode 100644 configs/kubeauditConfig.yaml diff --git a/README.md b/README.md index 93a0ea71..23fa93ec 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,7 @@ privileged, ... You get the gist of it and more on that later. Just know: - [Autofix](#autofix) - [Audits](#audits) - [Override Labels](#labels) +- [Audit Configuration](#audit-configuration) - [Contribute!](#contribute) @@ -76,6 +77,10 @@ or 1. if run with `-j/--json` it will log output json formatted so that its output can be used by other programs easily +`kubeaudit` supports using manual audit configuration provided by the user, use the command +`kubeaudit -f/--manifest /path/to/manifest.yml -k/--auditConfig /path/to/config.yml` +For more details on audit config check out [Audit Configuration](#audit-configuration). + `kubeaudit` has four different log levels `INFO, WARN, ERROR` controlled by `-v/--verbose LEVEL` and for those who counted and want to work on `kubeaudit` `DEBUG` @@ -311,7 +316,7 @@ WARN[0000] Memory limit exceeded, it is set to 512Mi but it must not exceed 125M ## Audit AppArmor -It checks that AppArmor is enabled for all containers by making sure the following annotation exists on the pod. +It checks that AppArmor is enabled for all containers by making sure the following annotation exists on the pod. There must be an annotation for each container in the pod: ``` @@ -337,8 +342,8 @@ ERRO[0000] AppArmor disabled. Annotation=container.apparmor.security.beta.kubern ## Audit Seccomp -It checks that Seccomp is enabled for all containers by making sure one or both of the following annotations exists -on the pod. If no pod annotation is used, then there must be an annotation for each container. Container annotations +It checks that Seccomp is enabled for all containers by making sure one or both of the following annotations exists +on the pod. If no pod annotation is used, then there must be an annotation for each container. Container annotations override the pod annotation: ``` @@ -349,7 +354,7 @@ seccomp.security.alpha.kubernetes.io/pod: container.seccomp.security.alpha.kubernetes.io/: ``` -where profile can be "runtime/default" or start with "localhost/" to be considered valid. "docker/default" is +where profile can be "runtime/default" or start with "localhost/" to be considered valid. "docker/default" is deprecated and will show a warning. It should be replaced with "runtime/default". If the Seccomp annotation is missing: @@ -576,7 +581,46 @@ capabilitiesToBeDropped: - SETFCAP #Set file capabilities. ``` -This can be overridden by using `-d` flag and providing your own defaults in the yaml format as shown above. +This can be overridden by using `-k` flag and providing your own defaults in the yaml format as shown below. + + + +## Audit Configuration + +Allows configuring your own audit settings for kubeaudit. By default following configuration is used: + +``` +apiVersion: v1 +kind: kubeauditConfig +audit: true # Set to false if you want kubeaudit to not audit your k8s manifests +spec: + capabilities: # List of all supported capabilities + NET_ADMIN: drop # Set to `keep` to keep capability + SETPCAP: drop # Set to `keep` to keep capability + MKNOD: drop # Set to `keep` to keep capability + AUDIT_WRITE: drop # Set to `keep` to keep capability + CHOWN: drop # Set to `keep` to keep capability + NET_RAW: drop # Set to `keep` to keep capability + DAC_OVERRIDE: drop # Set to `keep` to keep capability + FOWNER: drop # Set to `keep` to keep capability + FSETID: drop # Set to `keep` to keep capability + KILL: drop # Set to `keep` to keep capability + SETGID: drop # Set to `keep` to keep capability + SETUID: drop # Set to `keep` to keep capability + NET_BIND_SERVICE: drop # Set to `keep` to keep capability + SYS_CHROOT: drop # Set to `keep` to keep capability + SETFCAP: drop # Set to `keep` to keep capability + overrides: # List of all supported overrides + privilege-escalation: deny # Set to `allow` to skip auditing potential vulnerability + privileged: deny # Set to `allow` to skip auditing potential vulnerability + run-as-root: deny # Set to `allow` to skip auditing potential vulnerability + automount-service-account-token: deny # Set to `allow` to skip auditing potential vulnerability + read-only-root-filesystem-false: deny # Set to `allow` to skip auditing potential vulnerability + non-default-deny-ingress-network-policy: deny # Set to `allow` to skip auditing potential vulnerability + non-default-deny-egress-network-policy: deny # Set to `allow` to skip auditing potential vulnerability +``` + + ## Contributing diff --git a/cmd/allowPrivilegeEscalation_test.go b/cmd/allowPrivilegeEscalation_test.go index f55e9a41..7e8e423c 100644 --- a/cmd/allowPrivilegeEscalation_test.go +++ b/cmd/allowPrivilegeEscalation_test.go @@ -51,3 +51,12 @@ func TestAllowPrivilegeEscalationMultipleAllowMultipleContainers(t *testing.T) { func TestAllowPrivilegeEscalationSingleAllowMultipleContainers(t *testing.T) { runAuditTest(t, "allow_privilege_escalation_true_single_allowed_multiple_containers_v1beta.yml", auditAllowPrivilegeEscalation, []int{ErrorAllowPrivilegeEscalationTrue, ErrorAllowPrivilegeEscalationTrueAllowed}) } + +func TestAllowPrivilegeEscalationFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_privilege_escalation_from_config.yml" + runAuditTest(t, "security_context_nil_v1.yml", auditAllowPrivilegeEscalation, []int{ErrorAllowPrivilegeEscalationTrueAllowed}) + runAuditTest(t, "allow_privilege_escalation_nil_v1.yml", auditAllowPrivilegeEscalation, []int{ErrorAllowPrivilegeEscalationTrueAllowed}) + runAuditTest(t, "allow_privilege_escalation_true_v1.yml", auditAllowPrivilegeEscalation, []int{ErrorAllowPrivilegeEscalationTrueAllowed}) + runAuditTest(t, "allow_privilege_escalation_true_single_allowed_multiple_containers_v1beta.yml", auditAllowPrivilegeEscalation, []int{ErrorAllowPrivilegeEscalationTrueAllowed}) + rootConfig.auditConfig = "" +} diff --git a/cmd/automountServiceAccountToken_test.go b/cmd/automountServiceAccountToken_test.go index fcb6ee35..089345ab 100644 --- a/cmd/automountServiceAccountToken_test.go +++ b/cmd/automountServiceAccountToken_test.go @@ -25,3 +25,11 @@ func TestServiceAccountTokenMisconfiguredAllowV1(t *testing.T) { func TestServiceAccountTokenTrueAndDefaultNameV1(t *testing.T) { runAuditTest(t, "service_account_token_true_and_default_name_v1.yml", auditAutomountServiceAccountToken, []int{ErrorAutomountServiceAccountTokenTrueAndNoName}) } + +func TestAutomountServiceAccountTokenFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_automount_service_account_token_from_config.yml" + runAuditTest(t, "service_account_token_deprecated_v1.yml", auditAutomountServiceAccountToken, []int{ErrorServiceAccountTokenDeprecated}) + runAuditTest(t, "service_account_token_true_and_no_name_v1.yml", auditAutomountServiceAccountToken, []int{ErrorAutomountServiceAccountTokenTrueAllowed}) + runAuditTest(t, "service_account_token_nil_and_no_name_v1.yml", auditAutomountServiceAccountToken, []int{ErrorMisconfiguredKubeauditAllow}) + rootConfig.auditConfig = "" +} diff --git a/cmd/capabilities.go b/cmd/capabilities.go index 36d92464..0e74245a 100644 --- a/cmd/capabilities.go +++ b/cmd/capabilities.go @@ -2,19 +2,15 @@ package cmd import ( "fmt" - "io/ioutil" - "os" "strings" + "io/ioutil" + "github.com/Shopify/yaml" log "github.com/sirupsen/logrus" "github.com/spf13/cobra" ) -type capsDropList struct { - Drop []string `yaml:"capabilitiesToBeDropped"` -} - const defaultDropCapConfig = ` # SANE DEFAULTS: capabilitiesToBeDropped: @@ -35,25 +31,44 @@ capabilitiesToBeDropped: - SETFCAP #Set file capabilities. ` +var defaultCapList = &KubeauditConfigCapabilities{ + // SANE DEFAULTS: + NetAdmin: "drop", + SetPCAP: "drop", + MKNOD: "drop", + AuditWrite: "drop", + Chown: "drop", + NetRaw: "drop", + DacOverride: "drop", + FOWNER: "drop", + FSetID: "drop", + Kill: "drop", + SetGID: "drop", + SetUID: "drop", + NetBindService: "drop", + SYSChroot: "drop", + SetFCAP: "drop", +} + func recommendedCapabilitiesToBeDropped() (dropCapSet CapSet, err error) { - yamlFile := []byte(defaultDropCapConfig) - if rootConfig.dropCapConfig != "" { - if _, err = os.Stat(rootConfig.dropCapConfig); err != nil { - return - } - yamlFile, err = ioutil.ReadFile(rootConfig.dropCapConfig) + var kubeauditConfig = &KubeauditConfig{} + if rootConfig.auditConfig != "" { + data, err := ioutil.ReadFile(rootConfig.auditConfig) if err != nil { - return + log.Println(err) + return dropCapSet, err } - } - caps := capsDropList{} - err = yaml.Unmarshal(yamlFile, &caps) - if err != nil { - return - } - dropCapSet = make(CapSet) - for _, drop := range caps.Drop { - dropCapSet[CapabilityV1(drop)] = true + + // err check for unmarshalling is not useful as Root Init crashes the program if Config is not well formed + yaml.Unmarshal(data, kubeauditConfig) + + if kubeauditConfig != nil && kubeauditConfig.Spec != nil && kubeauditConfig.Spec.Capabilities != nil { + dropCapSet = dropCapFromConfigList(kubeauditConfig.Spec.Capabilities) + } else { + dropCapSet = dropCapFromConfigList(defaultCapList) + } + } else { + dropCapSet = dropCapFromConfigList(defaultCapList) } return } @@ -172,8 +187,7 @@ An ERROR log is generated when a pod has a capability which is on the drop list. A WARN log is generated when a pod has a capability allowed which is on the drop list. Example usage: -kubeaudit caps -kubeaudit caps -d drop_v1.yml`, defaultDropCapConfig), +kubeaudit caps`, defaultDropCapConfig), Run: runAudit(auditCapabilities), } diff --git a/cmd/capabilities_test.go b/cmd/capabilities_test.go index 8264b12d..c42692ba 100644 --- a/cmd/capabilities_test.go +++ b/cmd/capabilities_test.go @@ -33,6 +33,11 @@ func TestCapabilitiesSomeDroppedV1Beta2(t *testing.T) { runAuditTest(t, "capabilities_some_dropped_v1beta2.yml", auditCapabilities, []int{ErrorCapabilityNotDropped}) } +func TestAllowAuditCapabilitiesSomeDroppedFromConfigV1Beta2(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_audit_from_config.yml" + runAuditTest(t, "capabilities_some_dropped_v1beta2.yml", auditCapabilities, []int{ErrorCapabilityNotDropped}) +} + func TestCapabilitiesMisconfiguredAllowV1Beta2(t *testing.T) { runAuditTest(t, "capabilities_misconfigured_allow_v1beta2.yml", auditCapabilities, []int{ErrorMisconfiguredKubeauditAllow}) } @@ -53,12 +58,26 @@ func TestCapabilitiesSomeAllowedMultiContainersMixLabelsV1Beta2(t *testing.T) { runAuditTest(t, "capabilities_some_allowed_multi_containers_mix_labels_v1beta2.yml", auditCapabilities, []int{ErrorCapabilityAllowed, ErrorCapabilityAllowed}) } -func TestCapabilitiesManualConfigV1(t *testing.T) { - rootConfig.dropCapConfig = "../configs/capSetConfig.yaml" - runAuditTest(t, "capabilities_some_dropped_v1beta2.yml", auditCapabilities, []int{}) -} - func TestCapabilitiesManualConfigV2(t *testing.T) { - rootConfig.dropCapConfig = "../fake/file/path" + rootConfig.auditConfig = "../fake/file/path" runAuditTest(t, "capabilities_some_dropped_v1beta2.yml", auditCapabilities, []int{KubeauditInternalError}) + rootConfig.auditConfig = "" +} + +func TestCustomCapabilitiesToBeDroppedV1(t *testing.T) { + assert := assert.New(t) + rootConfig.auditConfig = "../configs/custom_capabilities_to_be_dropped_v1.yml" + capabilities, err := recommendedCapabilitiesToBeDropped() + assert.Nil(err) + assert.Equal(NewCapSetFromArray([]CapabilityV1{"MKNOD", "CHOWN", "DAC_OVERRIDE", "FSETID", "SETGID", "NET_BIND_SERVICE", "SETFCAP"}), capabilities, "") + rootConfig.auditConfig = "" +} + +func TestCustomCapabilitiesToBeDroppedV2(t *testing.T) { + assert := assert.New(t) + rootConfig.auditConfig = "../configs/custom_capabilities_to_be_dropped_v1.yml" + capabilities, err := recommendedCapabilitiesToBeDropped() + assert.Nil(err) + assert.NotEqual(NewCapSetFromArray([]CapabilityV1{"MKNOD", "SYS_CHROOT", "KILL", "CHOWN", "DAC_OVERRIDE", "FSETID", "SETGID", "NET_BIND_SERVICE", "SETFCAP"}), capabilities, "") + rootConfig.auditConfig = "" } diff --git a/cmd/capabilities_util.go b/cmd/capabilities_util.go new file mode 100644 index 00000000..afb5b886 --- /dev/null +++ b/cmd/capabilities_util.go @@ -0,0 +1,81 @@ +package cmd + +import "reflect" + +func dropCapFromConfigList(capList *KubeauditConfigCapabilities) (dropCapSet CapSet) { + var configCapabilityValue reflect.Value + var r reflect.Value + dropCapSet = make(CapSet) + r = reflect.ValueOf(capList) + configCapabilityValue = reflect.Indirect(r).FieldByName("SetPCAP") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("SETPCAP")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("MKNOD") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("MKNOD")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("AuditWrite") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("AUDIT_WRITE")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("Chown") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("CHOWN")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("NetRaw") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("NET_RAW")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("DacOverride") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("DAC_OVERRIDE")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("FOWNER") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("FOWNER")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("FSetID") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("FSETID")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("Kill") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("KILL")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("SetGID") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("SETGID")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("SetUID") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("SETUID")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("NetBindService") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("NET_BIND_SERVICE")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("SYSChroot") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("SYS_CHROOT")] = true + } + + configCapabilityValue = reflect.Indirect(r).FieldByName("SetFCAP") + if configCapabilityValue.String() == "drop" { + dropCapSet[CapabilityV1("SETFCAP")] = true + } + + return dropCapSet +} diff --git a/cmd/config.go b/cmd/config.go new file mode 100644 index 00000000..9d82c1ea --- /dev/null +++ b/cmd/config.go @@ -0,0 +1,76 @@ +package cmd + +// KubeauditConfig sets up config for kubeaudit from flag `config` +type KubeauditConfig struct { + APIVersion string `yaml:"apiVersion"` + Kind string `yaml:"kind"` + Spec *KubeauditConfigSpec `yaml:"spec"` + Audit bool `yaml:"audit"` +} + +// KubeauditConfigSpec contains Config Spec +type KubeauditConfigSpec struct { + Manifest []*KubeauditConfigManifest `yaml:"manifest"` + Capabilities *KubeauditConfigCapabilities `yaml:"capabilities"` + Overrides *KubeauditConfigOverrides `yaml:"overrides"` +} + +// KubeauditConfigManifest contains path to the manifests to audit +type KubeauditConfigManifest struct { + Path string `yaml:"path"` +} + +// KubeauditConfigCapabilities contains list of capabilities supported +type KubeauditConfigCapabilities struct { + NetAdmin string `yaml:"NET_ADMIN"` + SetPCAP string `yaml:"SETPCAP"` + MKNOD string `yaml:"MKNOD"` + AuditWrite string `yaml:"AUDIT_WRITE"` + Chown string `yaml:"CHOWN"` + NetRaw string `yaml:"NET_RAW"` + DacOverride string `yaml:"DAC_OVERRIDE"` + FOWNER string `yaml:"FOWNER"` + FSetID string `yaml:"FSETID"` + Kill string `yaml:"KILL"` + SetGID string `yaml:"SETGID"` + SetUID string `yaml:"SETUID"` + NetBindService string `yaml:"NET_BIND_SERVICE"` + SYSChroot string `yaml:"SYS_CHROOT"` + SetFCAP string `yaml:"SETFCAP"` +} + +// KubeauditConfigOverrides contains list of available overrides +type KubeauditConfigOverrides struct { + PrivilegeEscalation string `yaml:"privilege-escalation"` + Privileged string `yaml:"privileged"` + RunAsRoot string `yaml:"run-as-root"` + AutomountServiceAccountToken string `yaml:"automount-service-account-token"` + ReadOnlyRootFilesystemFalse string `yaml:"read-only-root-filesystem-false"` + NonDefaultDenyIngressNetworkPolicy string `yaml:"non-default-deny-ingress-network-policy"` + NonDefaultDenyEgressNetworkPolicy string `yaml:"non-default-deny-egress-network-policy"` +} + +func mapOverridesToStructFields(label string) string { + if label == "allow-privilege-escalation" { + return "PrivilegeEscalation" + } + if label == "allow-privileged" { + return "Privileged" + } + if label == "allow-run-as-root" { + return "RunAsRoot" + } + if label == "allow-automount-service-account-token" { + return "AutomountServiceAccountToken" + } + if label == "allow-read-only-root-filesystem-false" { + return "ReadOnlyRootFilesystemFalse" + } + if label == "allow-non-default-deny-egress-network-policy" { + return "NonDefaultDenyEgressNetworkPolicy" + } + if label == "allow-non-default-deny-ingress-network-policy" { + return "NonDefaultDenyIngressNetworkPolicy" + } + return "" +} diff --git a/cmd/config_test.go b/cmd/config_test.go new file mode 100644 index 00000000..aa2ec703 --- /dev/null +++ b/cmd/config_test.go @@ -0,0 +1,11 @@ +package cmd + +import ( + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestMapOverridesToStructFields(t *testing.T) { + assert.Equal(t, "", mapOverridesToStructFields("something-random")) +} diff --git a/cmd/networkPolicies_test.go b/cmd/networkPolicies_test.go index 27ee7f09..a9a72bdc 100644 --- a/cmd/networkPolicies_test.go +++ b/cmd/networkPolicies_test.go @@ -6,6 +6,11 @@ func TestNamespaceMissingDefaulDenyNetPol(t *testing.T) { runAuditTest(t, "namespace_missing_default_deny_netpol.yml", auditNetworkPolicies, []int{ErrorMissingDefaultDenyIngressAndEgressNetworkPolicy}) } +func TestAllowAuditNamespaceMissingDefaulDenyNetPolFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_audit_from_config.yml" + runAuditTest(t, "namespace_missing_default_deny_netpol.yml", auditNetworkPolicies, []int{ErrorMissingDefaultDenyIngressAndEgressNetworkPolicy}) +} + func TestNamespaceMissingDefaultDenyEgressNetPol(t *testing.T) { runAuditTest(t, "namespace_missing_default_deny_egress_netpol.yml", auditNetworkPolicies, []int{ErrorMissingDefaultDenyEgressNetworkPolicy}) } @@ -33,3 +38,21 @@ func TestAllowedNamespaceMissingDefaultDenyEgressNetPol(t *testing.T) { func TestAllowedNamespaceMissingDefaultDenyIngressNetPol(t *testing.T) { runAuditTest(t, "allowed_namespace_missing_default_deny_ingress_netpol.yml", auditNetworkPolicies, []int{ErrorMissingDefaultDenyIngressNetworkPolicyAllowed}) } + +func TestAllowedNamespaceMissingDefaulDenyNetPolFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_namespace_missing_default_deny_net_pol_from_config.yml" + runAuditTest(t, "namespace_missing_default_deny_netpol.yml", auditNetworkPolicies, []int{ErrorMissingDefaultDenyIngressAndEgressNetworkPolicyAllowed}) + rootConfig.auditConfig = "" +} + +func TestAllowedNamespaceMissingDefaultDenyEgressNetPolFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_namespace_missing_default_deny_egress_net_pol_from_config.yml" + runAuditTest(t, "namespace_missing_default_deny_egress_netpol.yml", auditNetworkPolicies, []int{ErrorMissingDefaultDenyEgressNetworkPolicyAllowed}) + rootConfig.auditConfig = "" +} + +func TestAllowedNamespaceMissingDefaultDenyIngressNetPolFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_namespace_missing_default_deny_ingress_net_pol_from_config.yml" + runAuditTest(t, "namespace_missing_default_deny_ingress_netpol.yml", auditNetworkPolicies, []int{ErrorMissingDefaultDenyIngressNetworkPolicyAllowed}) + rootConfig.auditConfig = "" +} diff --git a/cmd/privileged_test.go b/cmd/privileged_test.go index ae800632..89881d94 100644 --- a/cmd/privileged_test.go +++ b/cmd/privileged_test.go @@ -29,3 +29,11 @@ func TestPrivilegedTrueAllowedMultiContainerMultiLabelsV1(t *testing.T) { func TestPrivilegedTrueAllowedMultiContainerSingleLabelV1(t *testing.T) { runAuditTest(t, "privileged_true_allowed_multi_containers_single_label_v1.yml", auditPrivileged, []int{ErrorPrivilegedTrueAllowed, ErrorPrivilegedTrue}) } + +func TestAllowPrivilegedFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_privileged_from_config.yml" + runAuditTest(t, "security_context_nil_v1.yml", auditPrivileged, []int{ErrorPrivilegedNil}) + runAuditTest(t, "privileged_nil_v1.yml", auditPrivileged, []int{ErrorPrivilegedNil}) + runAuditTest(t, "privileged_true_v1.yml", auditPrivileged, []int{ErrorPrivilegedTrueAllowed}) + rootConfig.auditConfig = "" +} diff --git a/cmd/readOnlyRootFilesystem_test.go b/cmd/readOnlyRootFilesystem_test.go index cabc17ec..72f83435 100644 --- a/cmd/readOnlyRootFilesystem_test.go +++ b/cmd/readOnlyRootFilesystem_test.go @@ -29,3 +29,11 @@ func TestReadOnlyRootFilesystemFalseAllowedMultContainerMultiLabelsV1(t *testing func TestReadOnlyRootFilesystemFalseAllowedMultContainerSingleLabelV1(t *testing.T) { runAuditTest(t, "read_only_root_filesystem_false_allowed_multi_container_single_label_v1.yml", auditReadOnlyRootFS, []int{ErrorReadOnlyRootFilesystemFalseAllowed, ErrorReadOnlyRootFilesystemFalse}) } + +func TestAllowReadOnlyRootFilesystemFalseFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_read_only_root_filesystem_false_from_config.yml" + runAuditTest(t, "security_context_nil_v1.yml", auditReadOnlyRootFS, []int{ErrorReadOnlyRootFilesystemFalseAllowed}) + runAuditTest(t, "read_only_root_filesystem_nil_v1.yml", auditReadOnlyRootFS, []int{ErrorReadOnlyRootFilesystemFalseAllowed}) + runAuditTest(t, "read_only_root_filesystem_false_v1.yml", auditReadOnlyRootFS, []int{ErrorReadOnlyRootFilesystemFalseAllowed}) + rootConfig.auditConfig = "" +} diff --git a/cmd/root.go b/cmd/root.go index 6be980ac..181877f0 100644 --- a/cmd/root.go +++ b/cmd/root.go @@ -5,24 +5,30 @@ import ( "os" "path/filepath" + "io/ioutil" + log "github.com/sirupsen/logrus" "github.com/spf13/cobra" apiv1 "k8s.io/api/core/v1" + + "github.com/Shopify/yaml" ) var rootConfig rootFlags type rootFlags struct { - allPods bool - json bool - kubeConfig string - localMode bool - manifest string - namespace string - verbose string - dropCapConfig string + allPods bool + json bool + kubeConfig string + localMode bool + manifest string + namespace string + verbose string + auditConfig string } +var kubeauditConfig = &KubeauditConfig{} + // RootCmd defines the shell command usage for kubeaudit. var RootCmd = &cobra.Command{ Use: "kubeaudit", @@ -49,7 +55,7 @@ func init() { RootCmd.PersistentFlags().BoolVarP(&rootConfig.allPods, "allPods", "a", false, "Audit againsts pods in all the phases (default Running Phase)") RootCmd.PersistentFlags().StringVarP(&rootConfig.namespace, "namespace", "n", apiv1.NamespaceAll, "Specify the namespace scope to audit") RootCmd.PersistentFlags().StringVarP(&rootConfig.manifest, "manifest", "f", "", "yaml configuration to audit") - RootCmd.PersistentFlags().StringVarP(&rootConfig.dropCapConfig, "dropCapConfig", "d", "", "filepath for process capabilities to drop") + RootCmd.PersistentFlags().StringVarP(&rootConfig.auditConfig, "auditconfig", "k", "", "filepath for kubeaudit config file") } func processFlags() { @@ -70,4 +76,22 @@ func processFlags() { } rootConfig.kubeConfig = filepath.Join(home, ".kube", "config") } + + if rootConfig.auditConfig != "" { + var kubeauditConfig = &KubeauditConfig{} + data, err := ioutil.ReadFile(rootConfig.auditConfig) + if err != nil { + log.Warn("Unable to find file at set auditConfig path, auditing without any config") + return + } + err = yaml.Unmarshal(data, kubeauditConfig) + if err != nil { + log.Fatal("Unable to parse given auditConfig file, please check the syntax of your config file") + } + if !kubeauditConfig.Audit { + log.Warn("kubeaudit set to no-audit mode in auditConfig!") + os.Exit(0) + } + } + } diff --git a/cmd/runAsNonRoot_test.go b/cmd/runAsNonRoot_test.go index f272deda..d1f92604 100644 --- a/cmd/runAsNonRoot_test.go +++ b/cmd/runAsNonRoot_test.go @@ -61,3 +61,14 @@ func TestPSCRunAsRootFalseAllowedMultiContainersV1(t *testing.T) { func TestPSCRunAsRootFalseAllowedMultiContainersV2(t *testing.T) { runAuditTest(t, "run_as_non_root_psc_false_allowed_multi_containers_single_label_v1.yml", auditRunAsNonRoot, []int{ErrorRunAsNonRootPSCTrueFalseCSCFalse, ErrorRunAsNonRootPSCTrueFalseCSCFalse}) } + +func TestAllowAuditPSCRunAsRootFalseAllowedMultiContainersFromConfigV2(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_audit_from_config.yml" + runAuditTest(t, "run_as_non_root_psc_false_allowed_multi_containers_single_label_v1.yml", auditRunAsNonRoot, []int{ErrorRunAsNonRootPSCTrueFalseCSCFalse, ErrorRunAsNonRootPSCTrueFalseCSCFalse}) +} +func TestAllowRunAsNonRootFromConfig(t *testing.T) { + rootConfig.auditConfig = "../configs/allow_run_as_non_root_from_config.yml" + runAuditTest(t, "security_context_nil_v1.yml", auditRunAsNonRoot, []int{ErrorRunAsNonRootFalseAllowed}) + runAuditTest(t, "run_as_non_root_nil_v1.yml", auditRunAsNonRoot, []int{ErrorRunAsNonRootFalseAllowed}) + runAuditTest(t, "run_as_non_root_false_v1.yml", auditRunAsNonRoot, []int{ErrorRunAsNonRootFalseAllowed}) +} diff --git a/cmd/util.go b/cmd/util.go index 952ddaa1..47b2b455 100644 --- a/cmd/util.go +++ b/cmd/util.go @@ -12,6 +12,7 @@ import ( "sync" "github.com/Shopify/kubeaudit/scheme" + "github.com/Shopify/yaml" log "github.com/sirupsen/logrus" "github.com/spf13/cobra" apiv1 "k8s.io/api/core/v1" @@ -384,20 +385,60 @@ func getPodOverrideLabelReason(result *Result, overrideLabel string) (bool, stri if reason := result.Labels[podOverrideLabel]; reason != "" { return true, reason } + if rootConfig.auditConfig != "" { + var kubeauditConfig = &KubeauditConfig{} + + data, _ := ioutil.ReadFile(rootConfig.auditConfig) + + // err check for unmarshalling is not useful as Root Init crashes the program if Config is not well formed + yaml.Unmarshal(data, kubeauditConfig) + + tempLabel := mapOverridesToStructFields(overrideLabel) + if kubeauditConfig == nil || kubeauditConfig.Spec == nil || kubeauditConfig.Spec.Overrides == nil { + return false, "" + } + r := reflect.ValueOf(kubeauditConfig.Spec.Overrides) + configOverrideVal := reflect.Indirect(r).FieldByName(tempLabel) + if configOverrideVal.String() == "allow" { + return true, "Allowed " + overrideLabel + " in kubeauditConfig" + } + } return false, "" } func getNamespaceOverrideLabelReason(result *Result, nsName string, policyType string) (bool, string) { var namespaceOverrideLabel string + var tempLabel string if policyType == "egress" { namespaceOverrideLabel = "audit.kubernetes.io/" + nsName + "/" + "allow-non-default-deny-egress-network-policy" + tempLabel = "allow-non-default-deny-egress-network-policy" } if policyType == "ingress" { namespaceOverrideLabel = "audit.kubernetes.io/" + nsName + "/" + "allow-non-default-deny-ingress-network-policy" + tempLabel = "allow-non-default-deny-ingress-network-policy" } if reason := result.Labels[namespaceOverrideLabel]; reason != "" { return true, reason } + if rootConfig.auditConfig != "" { + var kubeauditConfig = &KubeauditConfig{} + + data, _ := ioutil.ReadFile(rootConfig.auditConfig) + + // err check for unmarshalling is not useful as Root Init crashes the program if Config is not well formed + yaml.Unmarshal(data, kubeauditConfig) + + tempOverrideField := mapOverridesToStructFields(tempLabel) + if kubeauditConfig == nil || kubeauditConfig.Spec == nil || kubeauditConfig.Spec.Overrides == nil { + return false, "" + } + r := reflect.ValueOf(kubeauditConfig.Spec.Overrides) + configOverrideVal := reflect.Indirect(r).FieldByName(tempOverrideField) + if configOverrideVal.String() == "allow" { + return true, "Allowed " + tempLabel + " in kubeauditConfig" + } + } + return false, "" } diff --git a/configs/allow_audit_from_config.yml b/configs/allow_audit_from_config.yml new file mode 100644 index 00000000..a9c98eb9 --- /dev/null +++ b/configs/allow_audit_from_config.yml @@ -0,0 +1,3 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true diff --git a/configs/allow_automount_service_account_token_from_config.yml b/configs/allow_automount_service_account_token_from_config.yml new file mode 100644 index 00000000..875cf53b --- /dev/null +++ b/configs/allow_automount_service_account_token_from_config.yml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + automount-service-account-token: allow diff --git a/configs/allow_namespace_missing_default_deny_egress_net_pol_from_config.yml b/configs/allow_namespace_missing_default_deny_egress_net_pol_from_config.yml new file mode 100644 index 00000000..da41edff --- /dev/null +++ b/configs/allow_namespace_missing_default_deny_egress_net_pol_from_config.yml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + non-default-deny-egress-network-policy: allow diff --git a/configs/allow_namespace_missing_default_deny_ingress_net_pol_from_config.yml b/configs/allow_namespace_missing_default_deny_ingress_net_pol_from_config.yml new file mode 100644 index 00000000..cfb58f3b --- /dev/null +++ b/configs/allow_namespace_missing_default_deny_ingress_net_pol_from_config.yml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + non-default-deny-ingress-network-policy: allow diff --git a/configs/allow_namespace_missing_default_deny_net_pol_from_config.yml b/configs/allow_namespace_missing_default_deny_net_pol_from_config.yml new file mode 100644 index 00000000..493a9c96 --- /dev/null +++ b/configs/allow_namespace_missing_default_deny_net_pol_from_config.yml @@ -0,0 +1,7 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + non-default-deny-egress-network-policy: allow + non-default-deny-ingress-network-policy: allow diff --git a/configs/allow_privilege_escalation_from_config.yml b/configs/allow_privilege_escalation_from_config.yml new file mode 100644 index 00000000..af5fc45c --- /dev/null +++ b/configs/allow_privilege_escalation_from_config.yml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + privilege-escalation: allow diff --git a/configs/allow_privileged_from_config.yml b/configs/allow_privileged_from_config.yml new file mode 100644 index 00000000..23bbd494 --- /dev/null +++ b/configs/allow_privileged_from_config.yml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + privileged: allow diff --git a/configs/allow_read_only_root_filesystem_false_from_config.yml b/configs/allow_read_only_root_filesystem_false_from_config.yml new file mode 100644 index 00000000..5d5b0aa2 --- /dev/null +++ b/configs/allow_read_only_root_filesystem_false_from_config.yml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + read-only-root-filesystem-false: allow diff --git a/configs/allow_run_as_non_root_from_config.yml b/configs/allow_run_as_non_root_from_config.yml new file mode 100644 index 00000000..44c5def6 --- /dev/null +++ b/configs/allow_run_as_non_root_from_config.yml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + overrides: + run-as-root: allow diff --git a/configs/custom_capabilities_to_be_dropped_v1.yml b/configs/custom_capabilities_to_be_dropped_v1.yml new file mode 100644 index 00000000..e2b51903 --- /dev/null +++ b/configs/custom_capabilities_to_be_dropped_v1.yml @@ -0,0 +1,20 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + capabilities: + NET_ADMIN: keep + SETPCAP: keep + MKNOD: drop + AUDIT_WRITE: keep + CHOWN: drop + NET_RAW: keep + DAC_OVERRIDE: drop + FOWNER: keep + FSETID: drop + KILL: keep + SETGID: drop + SETUID: keep + NET_BIND_SERVICE: drop + SYS_CHROOT: keep + SETFCAP: drop diff --git a/configs/drop_Cap_Config_Manifest_v1.yml b/configs/drop_Cap_Config_Manifest_v1.yml deleted file mode 100644 index 4a65627a..00000000 --- a/configs/drop_Cap_Config_Manifest_v1.yml +++ /dev/null @@ -1,16 +0,0 @@ -capabilitiesToBeDropped: - # https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities - - SETPCAP #Modify process capabilities. - - MKNOD #Create special files using mknod(2). - - AUDIT_WRITE #Write records to kernel auditing log. - - CHOWN #Make arbitrary changes to file UIDs and GIDs (see chown(2)). - - NET_RAW #Use RAW and PACKET sockets. - - DAC_OVERRIDE #Bypass file read, write, and execute permission checks. - - FOWNER #Bypass permission checks on operations that normally require the file system UID of the process to match the UID of the file. - - FSETID #Don’t clear set-user-ID and set-group-ID permission bits when a file is modified. - - KILL #Bypass permission checks for sending signals. - - SETGID #Make arbitrary manipulations of process GIDs and supplementary GID list. - - SETUID #Make arbitrary manipulations of process UIDs. - - NET_BIND_SERVICE #Bind a socket to internet domain privileged ports (port numbers less than 1024). - - SYS_CHROOT #Use chroot(2), change root directory. - - SETFCAP #Set file capabilities diff --git a/configs/kubeauditConfig.yaml b/configs/kubeauditConfig.yaml new file mode 100644 index 00000000..964d7626 --- /dev/null +++ b/configs/kubeauditConfig.yaml @@ -0,0 +1,30 @@ +apiVersion: v1 +kind: kubeauditConfig +audit: true +spec: + manifest: + - path: config/kubernetes/default/*/*.yaml + capabilities: + NET_ADMIN: drop + SETPCAP: drop + MKNOD: drop + AUDIT_WRITE: drop + CHOWN: drop + NET_RAW: drop + DAC_OVERRIDE: drop + FOWNER: drop + FSETID: drop + KILL: drop + SETGID: drop + SETUID: drop + NET_BIND_SERVICE: drop + SYS_CHROOT: drop + SETFCAP: drop + overrides: + privilege-escalation: deny + privileged: deny + run-as-root: deny + automount-service-account-token: deny + read-only-root-filesystem-false: deny + non-default-deny-egress-network-policy: allow + non-default-deny-ingress-network-policy: allow