Merge branch 'master' of https://github.com/Shuffle/Shuffle-docs

docs/API.md

@@ -177,7 +177,8 @@ curl -XPOST https://shuffler.io/api/v1/workflows/{workflow_id}/execute -H "Autho
 
 Additional info:
 - If you don't send JSON to the API, but a random string, we will take the entire string as the execution argument.
-- You can add dynamic app authentication when starting a workflow by using the following header: 'appauth'. Example: 'appauth: jira=auth for jira;elasticsearch=elasticsearch auth'. This works both with the name of the auth, and the ID. 
+- You can add dynamic app authentication when starting a workflow by using the following header: 'appauth'. Example: 'appauth: jira=auth for jira;elasticsearch=elasticsearch auth'. This works both with the name of the auth, and the ID.
+- If you want to run the workflow with a different environment, use the "environment" header with the name. Example: 'environment: cloud'. The environment needs to be the name of an existing environment.
 
 **Success response** 
 ```
@@ -769,6 +770,22 @@ curl -XDELETE https://shuffler.io/api/v1/hooks/{webhook_id} -H "Authorization: B
 ## Notifications 
 Below are the API's associated with Notifications in Shuffle. These can be listed, marked as read, and cleared.
 
+### Create a notification
+Notifications can be manually created, and will show up on the /admin?tab=priorities tab. This API is automatically utilized if you are running onprem with the Worker. This WILL trigger the notification workflow if it has been set up, and they will be grouped according to normal Notification control rules.
+
+Methods: POST 
+
+```
+curl -XPOST https://shuffler.io/api/v1/notifications -H "Authorization: Bearer APIKEY" -d '{"org_id": "YOUR ORGID", "title": "The title", "description": "The description of the notification", "reference_url": "URL for where to go when the user clicks explore"}'
+```
+
+
+**Success response** 
+```
+{"success":true}
+```
+
+
 ### Get all notifications
 Get all notifications assigned to your user from your organizations 
 

docs/app_creation.md

@@ -1,4 +1,4 @@
-# Shuffle App Creation in Python
+# Shuffle App Creation
 Documentation for app development with Python (as opposed to creating them with the app editor, e.g. with an OpenAPI specification).
 
 ## Table of contents
@@ -26,13 +26,6 @@ Documentation for app development with Python (as opposed to creating them with
 * [Debugging apps](#debugging)
 * [FAQ](#faq)
 
-## App development process
-This page is about how to develop a new Shuffle app. Want to know how to decide whether to make something new? Checkout the [process document](https://github.com/frikky/shuffle-docs/blob/master/handbook/engineering/app_development.md) and then come back here.
-
-## Prerequisites
-* App creator: Understanding of HTTP
-* Python app: Base understanding of Python 3.x
-
 ## Introduction
 Apps are how work is done in Shuffle. They receive a piece of data, whether JSON or string, performs some action, then returns data back to the user. Apps are mostly community-made, and we aim to support that the best way we can. We allow apps to be made as Swagger/OpenAPI specifications using the app creator OR directly with Python. Each app contains multiple actions, which again can have multiple parameters. More about apps and [how they work here](/docs/apps).
 
@@ -44,20 +37,27 @@ The underlying design of Apps in Shuffle are based on [WALKOFF](https://walkoff.
 
 [More about apps](/docs/apps)
 
-## Why create a custom app?
+### App development process
+This page is about how to develop a new Shuffle app. Want to know how to decide whether to make something new? Checkout the [process document](https://github.com/frikky/shuffle-docs/blob/master/handbook/engineering/app_development.md) and then come back here.
+
+### Prerequisites
+* App creator: Understanding of HTTP
+* Python app: Base understanding of Python 3.x
+
+### Why create a custom app?
 There are many prebuilt apps in Shuffle, all performing unique actions. There may however be an integration you need that doesn't exist yet. You may either make this yourself or commission this to be made [through Shuffle](https://shuffler.io/contact) - we're always looking to expand our repository of apps!
 
-## Python or the App Creator
+### Python or the App Creator
 A normal question we get asked all the time - should I use the app creator or Python directly? Find out by answering these questions (will be expanded):
 ```
 1. Is the app a HTTP API? 	Use the App Creator.
 2. Does the app need to perform custom actions? Use Python.
 ```
 
-## How apps are built
+### How apps are built
 When you build an app in Shuffle, it goes through a validation process before building the app. This process ends in a fully working Docker Image containing your apps, pointing to the original app specification to ensure users can use it in the front-end as well.
 
-## App Creator Instructions
+## App Creator in the Shuffle UI
 The app creator in Shuffle is built to handle any integration for HTTP apps you may think of. It's based on OpenAPI, and will generate the app using the Python SDK in the background. 
 
 [Here's an example from a workshop](https://youtu.be/PNuXCixYwDc?t=7822)
@@ -92,7 +92,7 @@ Creating or editing an app in Shuffle is made to be as simple and fast as possib
 * Categories: The category the app belongs to. Nothing fitting? Set it to "Other"
 * Tags: Add 1-5 tags that seem to fit. 
 
-## Authentication
+### Authentication
 There are many ways to authenticate Shuffle apps. The most important thing is to understand how authentication is reflected in a Workflow. Authentication fields become "required" fields, and can be stored in the App Authentication, utilized in a workflow. These are the authentication Options found in the App Creator:
 
 - No Authentication
@@ -203,7 +203,7 @@ After filling these in, you can now safely proceed to a Workflow to use the app.
 The former two will have to be found by the USER, as the point of this authentication type is to run as the user itself. This has to be documented well, either in the description of the app itself, or in the [OpenAPI documentation folder on Github]() to make it easily available to users. An example [for Microsoft app registration can be found here](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app), which includes how to get a client ID and secret. 
 
 
-## Actions
+### Actions
 Actions are the meat of how apps actually work. The action part of the app creator provides the ability to control each individual endpoint very specifically. 
 
 Action List information:
@@ -252,7 +252,7 @@ When building is done, you may want to test the app that you've built. This can
 You may want to change an app later. This can be done by using the /apps UI to find the app, selecting the app, then clicking the "Edit" button. 
 ![Apps view 7](https://github.com/frikky/shuffle-docs/blob/master/assets/apps-view-7.png?raw=true)
 
-## Python Instructions
+## Python - Building apps manually
 Apps using Python can do pretty much anything you can do on a computer. As an example, most utility functions of Shuffle itself are written with as functions of Python in the app "Shuffle-Tools"
 
 One of the first things you have to do is select an SDK. There are three images currently in Shuffle, Alpine, Kali, and Blackarch. Alpine is your standard slim docker image. Kali allows you access to Kali tooling, and Blackarch is arch, with a kitchen sink approach to tools.
@@ -397,7 +397,7 @@ contact_info:
 ```
 
 
-#### Authentication
+### Authentication
 
 Not all apps need this step, but if yours requires a key, secret, or password, it is recommended to use a specialized "authentication" declaration in api.yaml, so those credentials are stored safely and not displayed in cleartext in your Shuffle workflow.
 
@@ -443,7 +443,7 @@ authentication:
 
 ```
 
-#### Actions
+### Actions
 
 Actions define the functions of your app, the args it accepts and the options available (static list of values for a given arg).
 * name: the python function name
@@ -483,7 +483,7 @@ actions:
         type: string
 ```
 
-#### Logos
+### Logos
 The app's logo can be encoded here using a base64 encoded representation of the image, listed after large_image, see example below, but omitted for brevity:
 ```
 large_image: data:image/jpg;base64,/9j/4AAQSkZJR
@@ -587,15 +587,15 @@ I have the function below, as specified in my api.yaml, that accepts the args I
     def run_o365poller(self, planType,tenantID,clientID,clientSecret, PollInterval,json_data):
 ```
 
-## Hotloading your app
+### Hotloading your app
 Once your app has been tested, you've created the necessary metadata files and directory structure under your shuffle-apps folder, there is no need to restart your containers!
 
 Shuffle has an app hot reloading feature in the GUI. Go to Apps on the top left hand tool bar of the GUI, and then look for a Double Arrow refresh.
 Give the system 20 or so seconds, and you'll see a pop up saying it was successful.
 
 Once complete, any workflows you have that use the existing app, you'll have to delete old versions of the app and re-add them in the workflow.
 
-## Publishing your app
+### Publishing your app
 Just made an app and want others to get access to it? Here's how to get it in the hands of everyone:
 
 **OpenAPI:**
@@ -618,7 +618,7 @@ After making an app in Python, here's the steps to get it to everyone
 
 Both of these methods makes your app highlighted on https://shuffler.io to be used by hundreds if not thousands of organizations.
 
-## Debugging
+## Debugging Apps
 As Shuffle has a lot of individual parts, debugging can be quite tricky. To get started, here's a list of the different parts, with the latter three being modular / location-independent.
 
 | Type     | Container name    | Technology       | Note |
@@ -713,7 +713,7 @@ As you will notice, app logs can be quite verbose (optional in a later build). I
 
 Please [notify me](https://twitter.com/frikkylikeme) if you need help debugging app executions ASAP, as I've done a lot of it, but it's more tricky than the other steps.
 
-## FAQ
+### FAQ
 ```
 Q:  What is multiline? how is that represented when filling out the values?
 A: Multiline means the field will be larger and easier to use in the UI. E.g. you can use newlines.

docs/configuration.md

@@ -969,9 +969,6 @@ With the change from 0.8 to 0.9 we're changing databases from Google's Datastore
 - trigger_auth
 - workflowqueue
 
-  ```
-  
-  ```
 
 Before you start:
 If you have data of the same kind in the same index within Opensearch, these will be overwritten.
@@ -990,9 +987,6 @@ Example: you have the user "admin" in the index "users" within Opensearch and Da
 
 - 3. Set it to false; SHUFFLE_ELASTIC=false
 
-     ```
-     
-     ```
 
 ### 2. Set up Datastore and Opensearch
 
@@ -1087,7 +1081,7 @@ As previously stated, a lot can go wrong. Here's the most common issues:
 * Badly formed apps.
 * Bad environment
 
-#### General debugging
+### General debugging
 
 This part is mean to describe how to go about finding the issue you're having with executions. In most cases, you should start from the top of the list previously described in the following way:
 

docs/extensions.md

@@ -32,12 +32,12 @@ From the start, Shuffle has been a platform about integrations. We've focused on
 
 These integrations will typically entail third party services connecting to Shuffle with inbound Webhooks as triggers in a workflow.
 
-## Single Signon SSO
+## OpenID & SAML/SSO - Single Signon
 Shuffle added Single Signon (SAML) from version 0.9.16 & OpenID since 1.0.0. This allows you to log into Shuffle from other sources, entirely controlled by your external environment. SSO is available for **onprem**, even without the Enterprise version of Shuffle cloud. It works by setting an Entrypoint (IdP) and X509 Certificate, both used to validate the requests. This can be added under /admin, and **only works for your PRIMARY organization**.
 
 PS: We suggest setting up SSO for a sub-organization first to see if it works, avoiding the potential problem of locking yourself out. We are currently building a system to make it possible to use SSO in an optional way (version 1.4.0~) 
 
-### Note 
+### Using ANY SSO platform 
 **ONPREM ONLY:** You will have to change the SSO_REDIRECT_URL variable in the .env file to match your front end server link i.e `SSO_REDIRECT_URL=http://<URL>:<PORT>` 
 
 ![Single Signon button](https://github.com/frikky/shuffle-docs/blob/master/assets/sso-3.png?raw=true)
@@ -175,7 +175,7 @@ Finally go back to shuffle and use SSO button to login.
 
 ![shuffle SSO](https://user-images.githubusercontent.com/31187099/162689445-8db0766c-6f18-4463-8a92-f6ae62213918.png?raw=true)
 
-## Azure AD
+### Azure AD
 To use OpenID with Azure AD, Shuffle supports OpenID connect with the use of Client IDs and Client secrets. To set up OpenID Connect with Azure, we use "ID_token" authentication. This entails a few normal steps regarding app creation in Azure App Registration.
 
 1. Set up an app in Azure AD with ID tokens enabled
@@ -208,30 +208,59 @@ Shuffle by default allows you to store authentication tokens within Shuffle itse
 
 The Shuffle KMS system is built as a third party Key:Value provider. You can reference keys from the KMS in any field marked as "Authentication" in the UI, or from within a Shuffle authentication itself (meaning you can authenticate the authentication..). The way you reference the keys is path-based, starting with `kms/`. Requirements:
 
-* Have an Authentication called **kms shuffle storage** in Shuffle
+* Have an Authentication called **kms shuffle storage** in Shuffle. On the [Auth page](/admin?tab=app_auth), it should clearly stand out as a different type of authentication.
 * The Authentication needs to be associated with an App in the IAM category
 * The App needs to have an action labeled as "Get KMS key"
-* If it's the FIRST translation, it may fail out without internet access to github.com.
+* If it's the FIRST translation, it may fail out without internet access to github.com
 
 When these requirements are fullfilled, you can do the following to use the KMS system:
 
 * Find the required parameters for the action. The first image below shows the parameters IN ORDER for Hashicorp Cloud Platform Vault.
 * Use the following format: `kms/field1/field2/field3/field4/...`. This NEEDS to start with `kms/`
-* Example referencing the "username" in the app name "Jira": `kms/998067a9-33f2-4c4d-bbb6-4a997d784def/2e9a877f-1a89-4394-a242-f2c6d9dd2420/jira/username`
+* Make sure the Environment is correct. It uses your default environment to connect to the KMS if not otherwise specified on the [Auth page](/admin?tab=app_auth). 
+<img width="919" alt="image" src="https://github.com/Shuffle/Shuffle-docs/assets/5719530/ba69ec4a-2206-4f30-9ee2-0d94390a9dde">
 
+* Example referencing the "username" in the app name "Jira": `kms/998067a9-33f2-4c4d-bbb6-4a997d784def/2e9a877f-1a89-4394-a242-f2c6d9dd2420/jira/username`
 <img width="617" alt="image" src="https://github.com/Shuffle/Shuffle-docs/assets/5719530/0b354dc6-1d8e-4366-9f38-2cff2fe94486">
 
 If all of this is fulfilled, you can run the workflow, and Shuffle will automatically reference the KMS correctly. **If it fails to authenticate**, you should see a Notification show up like in the following image.
 
 <img width="989" alt="image" src="https://github.com/Shuffle/Shuffle-docs/assets/5719530/8a24c8f1-c00d-4481-bf56-ece2b7749fed">
 
-### KMS failure handling
+### Failure handling and Debugging
+During the initial configuration of KMS, there are many things that CAN go wrong. After the setup is initially completed and it works, it should not fail again, meaning spending time to set it up properly the first time is well worth it.
+
+If you are using Shuffle onprem and have trouble with KMS translations, we suggest setting the environment variable `SHUFFLE_KMS_DEBUG=true` on the shuffle-backend container. This will allow you to test individual keys in ANY field, meaning you can e.g. print out the KMS value that it should translate to. This further means we will not be deleting the KMS execution, and you can see the execution on the [Workflow Runtime Debugger page.](/workflows/debug). **KMS debug mode is NOT enabled on shuffler.io**.
+
+<img width="275" alt="image" src="https://github.com/Shuffle/Shuffle-docs/assets/5719530/b93fd28d-cb92-4a56-a9f3-9b2cd7a96d32">
+
 If your KMS translation fails, it is most likely due to network connectivity OR translation errors [for the standard](https://github.com/Shuffle/standards/blob/main/translation_standards/get_kms_key.json) in use. After running a translation, you should see the following three file categories in the admin panel:
 - translation standards 	- The available translation standards
 - translation input		- The data FROM the kms used for translation. Does NOT contain values, only keys.
 - translation output		- The translation OUTPUT. Includes JSON paths to the correct keys
 
-IF your translation fails, the first area to look at is the "translation output", as this is where the translation for the schema happens. The hash will match the data you have sent in, which can be found in the "translation input" folder. Documentation about schemaless explains [how translations happen](https://github.com/frikky/schemaless?tab=readme-ov-file#example), and how they can be fixed. If you need further help, contact [email protected]
+IF your translation fails, the first area to look at is the "[translation output](/admin?tab=files&category=translation_output)" file category, as this is where the translation for the schema happens. The hash will match the data you have sent in, which can be found in the "translation input" folder. Documentation about schemaless explains [how translations happen](https://github.com/frikky/schemaless?tab=readme-ov-file#example), and how they can be fixed. If you need further help, contact [email protected]
+
+A valid KMS translation file should be in the following format:
+```
+{
+   "kms_value": "path.in.kms.json"
+}
+```
+
+**The "kms_key" field is NOT relevant - the ONLY thing that is important is that the kms_value path is correct.**
+
+### Using any KMS
+KMS is supported for any system as long as the sections above are covered. It has been tested for the following:
+
+- Microsoft Azure KMS 
+- Hashicorp Vault
+- Google Cloud Key Management
+- AWS Key Management Service
+- Hashicorp Cloud Platform
+- Github
+- AliCloud
+- ... and more! Ask if you need help.
 
 ## Native Actions
 Native Actions are a new way Shuffle interacts with data, built brick by brick since introducing Shuffle's Integration Layer API in late 2022. The goal of Native Actions is to enable ourselves and others to be able to perform actions towards a specific API, without necessarily know how to do it specifically for that system. 
@@ -247,7 +276,9 @@ As of early 2024, this system is in active development, and we will implement fe
 
 The Native Actions system is based on [generative AI for automatic mapping of fields (Schemaless)](https://github.com/frikky/schemaless), uses Github to [store configurations (Standards)](https://github.com/shuffle/standards), and uses [Shuffle's Integration Layer API](https://shuffler.io/docs/API#integration-layer) to run the actions. 
 
-## Webhooks
+## Inbound Webhooks
+This section describes inbound webhooks to Shuffle, and how to set them up in many commonly used third-party systems. If your system support outbound Webhooks, it can also forward to Shuffle as a GET or POST request. [More about webhook triggers](/triggers/#webhooks)
+
 ### Wazuh
 Wazuh is a SIEM platform for security operations. We've used it through their API multiple ways, but were missing an important component; alerting. That's why we've developed a simple alert forwarder from Wazuh to Shuffle.