Merge pull request #54 from sdslabs/docs

Add Katana Docs

.gitignore

@@ -10,8 +10,11 @@ katana
 test.go
 vendor/*
 *.log
+docs/themes/*
+!docs/content/Katana
 teams/*
 teamcreds.txt
 lib/harbor/certs/*
 peer_configs/*
-work.txt
+work.txt
+

.gitmodules

@@ -1,5 +1,9 @@
+[submodule "docs/themes/hugo-geekdoc"]
+	path = docs/themes/hugo-geekdoc
+	url = https://github.com/thegeeklab/hugo-geekdoc.git
 [submodule "[email protected]:sdslabs/katana-services.git"]
 	branch = main
 [submodule "katana-services"]
 	path = katana-services
 	url = [email protected]:sdslabs/katana-services.git
+

Makefile

@@ -97,6 +97,12 @@ build:
 run : build
 	sudo ./bin/katana run
 
+setup-docs:
+	git submodule update --init --recursive
+	cp ./docs/config.sample.toml ./docs/config.toml
+	npm install --prefix ./docs/themes/hugo-geekdoc
+	npm run build --prefix ./docs/themes/hugo-geekdoc
+
 # Prints help message
 help:
 	@echo "KATANA"

docs/archetypes/default.md

@@ -0,0 +1,6 @@
+---
+title: "{{ replace .Name "-" " " | title }}"
+date: {{ .Date }}
+draft: true
+---
+

docs/config.sample.toml

@@ -0,0 +1,25 @@
+baseURL = "http://localhost"
+title = "Geekdocs"
+theme = "hugo-geekdoc"
+
+pluralizeListTitles = false
+
+# Geekdoc required configuration
+pygmentsUseClasses = true
+pygmentsCodeFences = true
+disablePathToLower = true
+
+# Required if you want to render robots.txt template
+enableRobotsTXT = true
+
+# Needed for mermaid shortcodes
+[markup]
+  [markup.goldmark.renderer]
+    # Needed for mermaid shortcode
+    unsafe = true
+  [markup.tableOfContents]
+    startLevel = 1
+    endLevel = 9
+
+[taxonomies]
+   tag = "tags"

docs/content/Architecture/design.md

@@ -0,0 +1,19 @@
+---
+title: "Design"
+resources:
+  - name: arch
+    src: "../../resources/_gen/images/arch.svg"
+    title: Architecture
+---
+
+Katana uses a namespace-per-team model. Each team is assigned a namespace, and all of the team's resources are deployed into that namespace. This model allows Katana to provide a secure environment for each team, while also allowing teams to interact with each other.
+
+Every team starts with a team pod. The team pod is a pod that is deployed into the team's namespace. The team pod is used to provide the team with a persistent environment. The team pod is also used to provide the team with a persistent storage volume. The team pod is deployed using a [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/), which ensures that the pod is always deployed to the same node. This ensures that the team's persistent storage volume is always available to the team.
+
+The teams are give SSH access to the team pod. Each teams is given a user-password pair that can be used to SSH into the team pod. The team pod is given a public IP address, which can be used to SSH into the team pod from outside of the cluster.
+
+Challenges are pods that are deployed into the team's namespace. On patching, the pod is redeployed.
+
+Katana has its own namespace. This namespace is used to deploy Katana components. These components include flag handler service, challenge checker service, logging service, git server. We will discuss these components in more detail in the next section.
+
+![Image not found](/arch.png)

docs/content/ChallengeChecker/main.md

@@ -0,0 +1,9 @@
+---
+title: "Challenge Checker"
+---
+
+# WIP
+
+The challenge checker will be responsible for running the checks against the challenges. The challenge checker will be deployed as a Kubernetes CronJob/Service. The CronJob/Service will run at every tick and will check the status of the challenges. The challenge checker will be responsible for checking the status of the challenges and updating the status of the challenges in the database.
+
+It has been decided to use a Pod in the master namespace to routinely run a knative service which would start a new instantaneous pod and run the checks. The instantaneous pod will return a success or a failure response for its respective request. There will be (no. of challenges x no. of teams) instantaneous pods possible at any given time.

docs/content/DatabaseSetup/index.md

@@ -0,0 +1,52 @@
+---
+title: "Database"
+---
+
+# Introduction
+
+Katana uses a mongoDB to store its data. The database is used to store information like challenge data, user data, and more. This page will walk you through the process of setting up a mongoDB database. The database will run in the master namespace.
+
+# Setup
+
+Simply setup the database by changing the config variables in `config.toml` to the following:
+
+```toml
+[mongo]
+username = "[YOUR USERNAME HERE]"
+password = "[YOUR PASSWORD HERE]"
+port = "32000"
+mongosh_version = "1.6.1"
+```
+
+Default yaml files are written in the `manifests` folder for delpoying mongoDB pods in the master namespace during infraset. To deploy the database, you first need to set up the infrastructure with the help of the `/api/v2/admin/infraSet` endpoint. Then you need to hit the `/api/v2/admin/db` endpoint to setup the database.
+
+# Go Code For Database Setup
+
+The following code is responsible for setting up the database.
+
+In `connection.go`:
+
+```Golang
+var client, err = mongo.Connect(ctx, options.Client().ApplyURI("mongodb://"+configs.MongoConfig.Username+":"+configs.MongoConfig.Password+"@"+configs.ServicesConfig.ChallengeDeployer.Host+":"+configs.MongoConfig.Port+"/?directConnection=true&appName=mongosh+"+configs.MongoConfig.Version))
+```
+
+In `db.go`:
+
+```Golang
+func DB(c *fiber.Ctx) error {
+	client, err := utils.GetKubeClient()
+	if err != nil {
+		log.Println(err)
+	}
+	service, err := client.CoreV1().Services("default").Get(context.TODO(), "mongo-nodeport-svc", metav1.GetOptions{})
+	if err != nil {
+		log.Println(err)
+	}
+
+	// Print the IP address of the service
+	fmt.Println(service.Spec.ClusterIP)
+	mongo.Init()
+
+	return c.SendString("Database setup completed")
+}
+```

docs/content/Deployment/main.md

@@ -0,0 +1,50 @@
+---
+title: "Deployment"
+---
+
+
+## Introduction
+
+
+
+The deployment service is responsible for creating namespaces and deploying the ctf challenges for each team under their specific namespace along with the service attached with them.
+
+
+
+{{<  toc  >}}
+
+## Initialisation
+
+- ### Create Teams
+We need to make sure teams' namespaces are created and 
+This creates the namespaces, the master pod for each pod. It also creates the user in Gogs database for each  team.
+
+- ### Challenge Type
+Currently we support web challenges for the ctf. An example of the web challenge can be found here.
+https://github.com/dicegang/dicectf-2022-challenges/tree/master/web/notekeeper
+
+- ### Challenge Zip
+As an input to deployment service, we ask the zip file for the challenge.
+TIP : make sure to create a zip without the path input flag for extra information.
+
+## Sending Request
+A post request to /deploy route under 'admin' Group with the type mulipart-form , key as challenge and the zip file as the load is required to be sent.
+Here is an example of such a requst via postman.
+
+![Image not found](/deploy-postreq.png)
+
+You can also however send the request using frontend by setting up the saya frontend.
+
+## Flow 
+
+- ### Build Folders
+Assuming the name of challenge is notekeeper for illustartion purposes. The file gets unzipped in ..../katana/chall/notekeeper along with a copy of the zip file in order to use for copying inside the pod. Read patching service for more info. 
+
+- ### Build Image
+Assuming that the docker file is in /notekeeper root base. Image is build outside in the docker registry or the volume mounted and pushed isnide minikube registry as of now. It is to be updated to use the docker client.
+
+- ### Create Deployment
+The deployment for each team with 1 replica of each challenge pod is created under each namespace for the teams using the k8's client. Check out deploy.go for the code file. 
+
+- ### Create Service
+Next, under each namespace, a Nodeport service is also attached to the deployments exposing the web challenge.Furthur exposing via minikube service command is also done.

docs/content/Katana/getting-started.md

@@ -0,0 +1,11 @@
+---
+title: "Getting Started"
+---
+
+Katana is a cutting-edge Attack-Defense Capture the Flag (CTF) platform built. CTF is a popular cybersecurity competition where participants compete against each other by solving complex challenges related to cybersecurity. Katana makes it possible for users to easily host and play CTF competitions in a secure and scalable environment.
+
+One of the most notable features of Katana is its use of [Kubernetes](https://kubernetes.io/). Kubernetes is an open-source container orchestration system that simplifies the deployment, scaling, and management of containerized applications. By leveraging Kubernetes, Katana is able to provide a highly scalable and resilient platform that can handle a large number of users and high volumes of traffic.
+
+The platform also provides a dashboard for managing competitions. The dashboard allows users to manage the challenges, teams, and scoring for the competition. It also provides real-time statistics and monitoring of the competition, enabling the organizers to make adjustments as needed.
+
+Katana's use of Kubernetes also ensures that the platform is highly secure. Kubernetes provides a variety of security features that are designed to protect against attacks such as denial-of-service (DoS) attacks and data breaches. These features include role-based access control (RBAC), network policies, and pod security policies. Additionally, Kubernetes provides automated security updates, ensuring that the platform is always up-to-date with the latest security patches.

docs/content/Katana/setup.md

@@ -0,0 +1,19 @@
+---
+title: "Setup"
+---
+
+Setting up Katana involves of the following steps: 
+
+{{<  toc  >}}
+
+### Infraset 
+
+This is setting up the basic pods of katana in katana namespace. These include
+- #### MYSQL
+  MYSQL picks up the admin config from Config.Toml and is exposed using a NodePort type service on port 32001. This is used to store data for Gogs 
+- #### MongoDB
+  MongoDB is also a NodePort type service exposed on port 32000. This is used to store team credentials for [SSH Service](/Services/ssh/). This will also be used to store the flags and points of each team. 
+- #### GOGS
+  This is the locally running GitServer which is used for Patching Service. This has its database in MYSQL and is exposed using Cluster IP. 
+
+

docs/content/Patching/main.md

@@ -0,0 +1,49 @@
+---
+title: "Patching Service"
+---
+
+
+## Introduction
+
+
+
+The patching service of katana makes use of a locally run git service called Gogs running in the admin namespace. The decision to not use github was to decrease latency to pull stuff over internet. Following are the steps
+
+
+
+{{<  toc  >}}
+
+## Initialisation
+
+- ### Infraset
+During this time we estbalish MySQL, MongoDB and Gogs pods are created. 
+
+- ### Database Setup
+We establish connection with Mongo and MySQL. A mongoDB admin is established with team credentials. 
+
+- ### GitServer
+We hit the gogsIP/install which creates the gogs tables in the MySQL pod. If using a non-cloud based cluster (like minikube), establish a connection with LoadBalancer [```minikube tunnel```]. As of now you have to hit the Database setup one more time after GitServer to establish the admin user in Git 
+
+- ### Create Teams
+This creates the namespaces, the master pod for each pod. It also creates the user in Gogs database for each  team.
+
+## Setting up a challenge
+
+Whenever a challenge is setup, the broadcasting service is ivoked which creates a private repository for that challenge for each team along with applying a yaml file for that challenge in each team's namespace. Now the said broadcast service sends a zip file of the challenge to each and every pod, where it's unzipped and initialised dynamically w.r.t. each and every team repository. We pulll once to make sure the histories of both the local copy and the repository is in sync. 
+
+## Patch Challenge
+in /usr/bin/ we have a patch_challenge bash file which essentially runs a simple git add to git push command. It takes the commit message as it's argument so teams can have their own commit messages so they find it easier to point if they wish to backtrack to a previous patch. 
+As soon as a push is made, a github webhook sends a post request upon which the updates are pulled, an image is created and pushed into the K8s registry. The challenge pod of that particular team is killed, and when it restarts, it pulls the latest image from the registry. 
+
+## Inside/Out
+
+To reduce latency, the following architectural decisions were taken. MySQL server and Gogs Service will be running within the K8s cluster. Pulling the changes and creation of image happens outside the cluster. 
+
+## Image within
+
+We "briefly" considered the concept of maybe creation of images within the pod and pushing it into the registry from within the pod itself. This would lead to the changes never having to leave the cluster, thus increasing speed by a lot. However, this was scrapped as to make an image you have to either: 
+
+ - Create  a DIND (Docker in Docker), which would have secuirity impact. 
+ - Using Kaniko (A Go based library which allows image creation within the pod). However pushing it into the registry recquired to provide sudo privilege to team pods, which essentially left the entire cluster vulnerable to attacks.
+
+Thus the final decision was to run the patch service in the aforementioned manner to provide the team a seamlesss A&D  CTF exeperience and at the same time make it easier for the admin to host one

docs/content/TeamNamespaces/main.md

@@ -0,0 +1,62 @@
+---
+title: "Team Namespaces"
+---
+
+# Introduction
+
+As seen in the image below, each team has its own namespace. This is to ensure that each team has its own set of resources and does not interfere with other teams. This also allows for a more secure environment as each team can only access its own namespace.
+
+![Image Not Found](/team-namespaces-architecture.png)
+
+# Setup
+
+To deploy the database, you first need to set up the infrastructure with the help of the `/api/v2/admin/createTeams/:number` endpoint where `/:number` is the number of teams you want to initialise.
+
+# Go Code For Team Namespaces
+
+The following code in the `createTeams.go` controller is responsible for creating the namespaces for the teams and deploying the required resources into the namespaces.
+
+This function can be found in the create teams controller, in the CreateTeams function.
+```Golang
+	for i := 0; i < noOfTeams; i++ {
+		// Create a directory named katana-team-i in the teams directory
+		if _, err := os.Stat("teams/katana-team-" + strconv.Itoa(i)); os.IsNotExist(err) {
+			errDir := os.Mkdir("teams/katana-team-"+strconv.Itoa(i), 0755)
+			if errDir != nil {
+				log.Fatal(err)
+			}
+		}
+
+		log.Println("Creating Team: " + strconv.Itoa(i))
+		namespace := "katana-team-" + strconv.Itoa(i) + "-ns"
+		nsName := &coreV1.Namespace{
+			ObjectMeta: metav1.ObjectMeta{
+				Name: namespace,
+			},
+		}
+		_, err = client.CoreV1().Namespaces().Create(c.Context(), nsName, metav1.CreateOptions{})
+		if err != nil {
+			log.Fatal(err)
+		}
+		manifest := &bytes.Buffer{}
+		tmpl, err := template.ParseFiles(filepath.Join(clusterConfig.ManifestDir, "teams.yml"))
+		if err != nil {
+			return err
+		}
+		deploymentConfig := utils.DeploymentConfig()
+
+		if err = tmpl.Execute(manifest, deploymentConfig); err != nil {
+			return err
+		}
+		pathToCfg := filepath.Join(
+			os.Getenv("HOME"), ".kube", "config",
+		)
+		config, err := clientcmd.BuildConfigFromFlags("", pathToCfg)
+		if err != nil {
+			log.Fatal(err)
+		}
+		if err = deployment.ApplyManifest(config, client, manifest.Bytes(), namespace); err != nil {
+			return err
+		}
+	}
+```

docs/content/Tsuka/main.md

@@ -0,0 +1,38 @@
+---
+title: "Tsuka"
+---
+
+Each team under it's own namespace has its own master pod. This master pod is Tsuka. This pod acts as a dedicated virtual machine for each team. It is a ubuntu based pod with ssh server running inside it.
+
+All teams have access to their own master pod. They can ssh into it using the password provided to them. The password is stored in a file called teamcreds.txt which is generated when the team is created. NOTE: The teams have to be connected to Katana's VPN to access their master pod.
+
+Tsuka contains source code of all the challenges. A team is expected to patch the challenge and push it to the git server. The patching service will then build the image and push it to the registry. The challenge pod will then pull the latest image and run it.
+
+//UPDATE DIAGRAM OF A MASTER PODS WITH THINGS INSIDE IT.
+![Image Not Found](/team-pods-architecture.png)
+
+## Setup
+
+During setup, a `setup-script.sh` is run which does the following:
+- Does `apt-get update` and `apt-get upgrade` followed by `apt-get install git curl nano vim`.
+- It then sets up the ssh server by installing openssh-server and setting up the password for the team. The password is the root password for the master pod as well.
+- After this, it moves 2 binaries to `/usr/bin/` which are `patch-challenge` and `setup`. These binaries are used by the teams to patch their challenges and by the deployment service to setup challenges respectively.
+- Lastly, it sets up git config for the team and runs a flask server as a daemon process.
+
+## Working
+
+- Tsuka contains a flask server which helps in setting up challenges when they are deployed. It runs as a daemon process and runs a `setup` script, which is present in `/usr/bin/`, when a challenge is deployed.
+
+- There is also a `patch-challenge` binary present in `/usr/bin` that teams can use to patch their challenges. This binary is a wrapper around a bash script that runs a git add and git push command. The commit message is passed as an argument to the binary.
+
+## General Flow
+
+- Challenge gets copied to master pod
+- Flask server unzips the challenge in the master pod
+- Team makes changes to the challenge and uses `patch-challenge` binary to push the changes to the git server
+
+For patching we looked at few options before finalising on the [current one](/Patching/). The other options were:
+
+- [Challenge-Containers v1](/Tsuka/v1/)
+
+- [Challenge-Containers v2](/Tsuka/v2)