Doc changes (#173)

* Update step-by-step.md Replace cmf-server-ip with cmf-server-url * Update cmf-server.md Remove 2nd line which is not required * Update step-by-step.md Replaced Test-env to pipeline name and added bucket name as "dvc-art" with a note for the user * Update cmf_client.md Updated 'exec_name' to 'exec_id' and 'Test-env' to 'pipeline-name' and included a generic path for defining or utilizing the mlmd file. * Update step-by-step.md Replaced pipeline-name with quotes for easier understanding * Update Getting Started with cmf.md Replaced cmf-server-ip with cmf-server-url * Update cmf_client.md Added "dev-art" as a default bucket name * Update cmf_client.md Updated the description of optional parameter -a(artifact_name). * Update step-by-step.md Made changes inside cmf init minios3 command by providing appropriate localhost and IP address. * Update cmf-server.md Sample example is given * Update index.md Updated Installation and Configuration of CMF. * Update index.md spelling mistake in python * Update step-by-step.md Made changes inside client and server configuration * Update step-by-step.md update MinIOS3 to minioS3 * Update index.md Added Prerequisite part inside installation step * Update step-by-step.md Replaced "here" with "web page name". * Update mkdocs.yaml Replaced "Getting Started" to "Getting started with cmf-server" and "Quick start with cmf-client". * Update cmf-server.md Added github repository clone as a prerequisite. * Update step-by-step.md Replaced Getting started to Getting started with cmf-server * Update index.md Replaced "Getting started" with "Quick start with cmf-client" * Update cmf-server.md Replaced title name * Update step-by-step.md Replaced title name * Update step-by-step.md Made changes in Initialize cmf step * Update minio-server.md Made changes inside initialization of minioS3. * Update cmf_client.md Made changes inside minioS3 initialization step. * Update cmf-server.md Made some changes inside description of an error and added number bullet point in right way * Update index.md Replaced getting started of example section with appropriate sentence. * Update index.md Made changes in minioS3 initialization * Update minio-server.md added space * Update cmf-server.md Made changes in number point * Update cmf-server.md made changes in formatting of page * Update cmf-server.md Formatted document * Update cmf-server.md Document formatted * Update cmf-server.md Document Formatted * Update cmf-server.md Document Formatted * Update cmf-server.md Document Formatted * Update cmf-server.md Document formatted * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md changed image * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md * Update cmf-server.md * Update minio-server.md * Update cmf_client.md Updated cmf-server-url with x.x.x.x:8080 Added --session-token inside amazons3 section * Update cmf_client.md Added Temporary Security Credentials part inside amazonS3 configuration part * Update cmf_client.md * Added python handler --------- Co-authored-by: varkha-d-sharma <[email protected]> Co-authored-by: first second <[email protected]>
HewlettPackard · May 30, 2024 · e852960 · e852960
1 parent c56ed92
commit e852960
Show file tree

Hide file tree

Showing 7 changed files with 171 additions and 112 deletions.
diff --git a/docs/cmf_client/Getting Started with cmf.md b/docs/cmf_client/Getting Started with cmf.md
@@ -44,7 +44,7 @@ mkdir example-folder
 
 CMF initialization is the first and foremost to use cmf-client commads. This command in one go complete initialization process making cmf-client user friendly.     Execute `cmf init` in the `example-folder` directory created in the [above](#create-a-folder) step.
 ```
-cmf init minioS3 --url s3://bucket-name --endpoint-url http://localhost:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-ip http://x.x.x.x:8080  --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://X.X.X.X:7687
+cmf init minioS3 --url s3://bucket-name --endpoint-url http://localhost:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080  --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://X.X.X.X:7687
 ```
 Check [here](./cmf_client.md) for more details.
 

diff --git a/docs/cmf_client/cmf_client.md b/docs/cmf_client/cmf_client.md
@@ -24,8 +24,10 @@ Usage: cmf init minioS3 [-h] --url [url]
 ```
 `cmf init minioS3` configures Minio S3 bucket as a cmf artifact repository. Refer [minio-server.md](./minio-server.md#steps-to-set-up-a-minio-server) to set up a minio server.
 ```
-cmf init minioS3 --url s3://bucket-name --endpoint-url http://localhost:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://127.0.0.1:80 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
+cmf init minioS3 --url s3://dvc-art --endpoint-url http://x.x.x.x:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
 ```
+> Here, "dvc-art" is provided as an example bucket name. However, users can change it as needed, if the user chooses to change it, they will need to update the Dockerfile for MinIOS3 accordingly.
+
 Required Arguments
 ```
   --url [url]                           Specify bucket url.
@@ -55,8 +57,11 @@ Usage: cmf init local [-h] --path [path] -
 ```
 `cmf init local` initialises local directory as a cmf artifact repository.
 ```
-cmf init local --path /home/user/local-storage --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://127.0.0.1:80 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
+cmf init local --path /home/XXXX/local-storage --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
 ```
+
+> Replace 'XXXX' with your system username in the following path: /home/XXXX/local-storage
+
 Required Arguments
 ```
   --path [path]                         Specify local directory path.
@@ -71,20 +76,59 @@ Optional Arguments
   --neo4j-uri <neo4j_uri>             Specify neo4j uri. Eg bolt://localhost:7687 (default: None)
 ```
 ### cmf init amazonS3
+Before setting up, obtain AWS temporary security credentials using the AWS Security Token Service (STS). These credentials are short-term and can last from minutes to hours. They are dynamically generated and provided to trusted users upon request, and expire after use. Users with appropriate permissions can request new credentials before or upon expiration. For further information, refer to the  [Temporary security credentials in IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html) page.
+
+To retrieve temporary security credentials using multi-factor authentication (MFA) for an IAM user, you can use the below command.
+```
+aws sts get-session-token --duration-seconds <duration> --serial-number <MFA_device_serial_number> --token-code <MFA_token_code>
+```
+Required Arguments
+```
+  --serial-number                Specifies the serial number of the MFA device associated with the IAM user.
+  --token-code                   Specifies the one-time code generated by the MFA device.
+```
+
+Optional Arguments
+```
+  --duration-seconds             Specifies the duration for which the temporary credentials will be valid, in seconds.
+```
+
+Example
+```
+aws sts get-session-token --duration-seconds 3600 --serial-number arn:aws:iam::123456789012:mfa/user --token-code 123456
+```
+
+This will return output like 
+
+``` 
+{
+    "Credentials": {
+        "AccessKeyId": "ABCDEFGHIJKLMNO123456",
+        "SecretAccessKey": "PQRSTUVWXYZ789101112131415",
+        "SessionToken": "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijlmnopqrstuvwxyz12345678910ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijlmnopqrstuvwxyz12345678910ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijlmnopqrstuvwxyz12345678910",
+        "Expiration": "2021-05-10T15:31:08+00:00"
+    }
+}
+```
+Initialization of amazonS3
 ```
 Usage: cmf init amazonS3 [-h] --url [url] 
                               --access-key-id [access_key_id]
-                              --secret-key [secret_key] 
-                              --git-remote-url [git_remote_url] 
+                              --secret-key [secret_key]
+                              --session-token [session_token]
+                              --git-remote-url [git_remote_url]
                               --cmf-server-url [cmf_server_url]
                               --neo4j-user [neo4j_user]
                               --neo4j-password neo4j_password]
                               --neo4j-uri [neo4j_uri]
 ```
 `cmf init amazonS3` initialises Amazon S3 bucket as a CMF artifact repository.
 ```
-cmf init amazonS3 --url s3://bucket-name --access-key-id XXXXXXXXXXXXX --secret-key XXXXXXXXXXXXX --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://127.0.0.1:80 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687 
+cmf init amazonS3 --url s3://bucket-name --access-key-id XXXXXXXXXXXXX --secret-key XXXXXXXXXXXXX --session-token XXXXXXXXXXXXX --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687 
 ```
+> Here, use the --access-key-id, --secret-key and --session-token generated from the `aws sts` command which is mentioned above.
+
+> The bucket-name must exist within Amazon S3 before executing the `cmf artifact push` command.
 
 Required Arguments
 ```
@@ -96,6 +140,7 @@ Required Arguments
 Optional Arguments
 ```
   -h, --help                          show this help message and exit
+  --session-token                     Specify session token. (default: )
   --cmf-server-url [cmf_server_url]   Specify cmf-server url. (default: http://127.0.0.1:80)
   --neo4j-user [neo4j_user]           Specify neo4j user. (default: None)
   --neo4j-password [neo4j_password]   Specify neo4j password. (default: None)
@@ -114,7 +159,7 @@ Usage: cmf init sshremote [-h] --path [path]
 ```
 `cmf init sshremote` command initialises remote ssh directory as a cmf artifact repository.
 ```
-cmf init sshremote --path ssh://127.0.0.1/home/user/ssh-storage --user XXXXX --port 22 --password example@123 --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://127.0.0.1:80 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
+cmf init sshremote --path ssh://127.0.0.1/home/user/ssh-storage --user XXXXX --port 22 --password example@123 --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080 --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
 ```
 Required Arguments
 ```
@@ -144,7 +189,7 @@ Usage: cmf artifact pull [-h] -p [pipeline_name] -f [file_name] [-a <artifact_na
 ```
 `cmf artifact pull` command pull artifacts from the user configured repository to the user's local machine.
 ```
-cmf artifact pull -p 'Test-env' 
+cmf artifact pull -p 'pipeline-name' 
 ```
 Required Arguments
 ```
@@ -153,7 +198,7 @@ Required Arguments
 Optional Arguments
 ```
   -h, --help                                            show this help message and exit
-  -a <artifact_name>, --artifact_name <artifact_name>   Specify artifact name.
+  -a <artifact_name>, --artifact_name <artifact_name>   Specify artifact name only; don't use folder name or absolute path.
   -f [file_name],--file-name [file_name]                Specify mlmd file name.
 ```
 ### cmf artifact push
@@ -175,7 +220,7 @@ Usage: cmf metadata pull [-h] -p [pipeline_name] -f [file_name]  -e [exec_id]
 ```
 `cmf metadata pull` command pulls the metadata file from the cmf-server to the user's local machine.
 ```
-cmf metadata pull -p 'Test-env' -f "/home/user/example/name_of_file"
+cmf metadata pull -p 'pipeline-name' -f "/path/to/mlmd-file-name"
 ```
 Required Arguments
 ```
@@ -184,7 +229,7 @@ Required Arguments
 Optional Arguments
 ```
 -h, --help                                  show this help message and exit
--e [exec_id], --execution [exec_name]       Specify execution id
+-e [exec_id], --execution [exec_id]         Specify execution id
 -f [file_name], --file_name [file_name]     Specify mlmd file name with full path(either relative or absolute).
 ```
 ### cmf metadata push
@@ -193,7 +238,7 @@ Usage: cmf metadata push [-h] -p [pipeline_name] -f [file_name]  -e [exec_id]
 ```
 `cmf metadata push` command pushes the metadata file from the local machine to the cmf-server.
 ```
-cmf metadata push -p 'Test-env' -f "/home/user/example/name_of_file"
+cmf metadata push -p 'pipeline-name' -f "/path/to/mlmd-file-name"
 ```
 Required Arguments
 ```
@@ -204,5 +249,5 @@ Optional Arguments
 ```
   -h, --help                                    show this help message and exit
   -f [file_name], --file_name [file_name]       Specify mlmd file name.
-  -e [exec_name], --execution [exec_name]       Specify execution id.
+  -e [exec_id], --execution [exec_id]           Specify execution id.
 ```
diff --git a/docs/cmf_client/minio-server.md b/docs/cmf_client/minio-server.md
@@ -21,8 +21,10 @@ Follow the below mentioned steps to set up a MinIO server:
 
 3.  Execute the following command to initialize the MinIO S3 bucket as a CMF artifact repository.
     ```
-    cmf init minioS3 --url s3://bucket-name --endpoint-url http://localhost:9000 --access-key-id minioadmin --secret-key   minioadmin --git-remote-url https://github.com/user/experiment-repo.git
+    cmf init minioS3 --url s3://dvc-art --endpoint-url http://x.x.x.x:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080  --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
     ```
+
+   > Here, "dvc-art" is provided as an example bucket name. However, users can change it as needed, if the user chooses to change it, they will need to update the Dockerfile for minioS3 accordingly.
 
 4. Execute `cmf init show` to check the CMF configuration. The sample output looks as follows:
    ```

diff --git a/docs/cmf_client/step-by-step.md b/docs/cmf_client/step-by-step.md
@@ -1,35 +1,30 @@
-# Getting started with cmf
+# Quick start with cmf-client
 Common metadata framework (cmf) has the following components:
 
 - **Metadata Library** exposes API’s to track the pipeline metadata. It also provides API’s to query the stored metadata. 
 - **cmf-client** interacts with the server to pull or push metadata from or to the cmf-server.
 - **cmf-server** interacts with all the remote clients and is responsible to merge the metadata transferred by the cmf-client and manage the consolidated metadata. 
 - **Central Artifact Repositories** hosts the code and data. 
 
+## Install cmf library i.e. cmflib
+Before proceeding, ensure that the CMF library is installed on your system. If not, follow the installation instructions provided inside the [CMF in a nutshell](../index.md) page. 
+
+## Install cmf-server
+cmf-server is a key interface for the user to explore and track their ML training runs. It allows users to store the metadata file on the cmf-server. The user can retrieve the saved metadata file and can view the content of the saved metadata file using the UI provided by the cmf-server.
+
+Follow the instructions on the [Getting started with cmf-server](../cmf_server/cmf-server.md) page for details on how to setup a cmf-server.
+
 ## Setup a cmf-client 
 cmf-client is a tool that facilitates metadata collaboration between different teams or two team members. It allows users to pull or push metadata from or to the cmf-server.
 
 Follow the below-mentioned steps for the end-to-end setup of cmf-client:-
 
-**Pre-Requisites**
-
-- Python 3.8+
-- Git latest version
+**Configuration**
 
-**Install cmf library i.e. cmflib**
-```
-pip install git+https://github.com/HewlettPackard/cmf
-```
-**OR**
-```
-pip install cmflib
-```
-Check [here](https://hewlettpackard.github.io/cmf/) for more details.
+1. Create working directory `mkdir <workdir>`
+2. Execute `cmf init` to configure dvc remote directory, git remote url, cmf server and neo4j. Follow the [Overview](./cmf_client.md) page for more details.
 
-## Install cmf-server
-cmf-server is a key interface for the user to explore and track their ML training runs. It allows users to store the metadata file on the cmf-server. The user can retrieve the saved metadata file and can view the content of the saved metadata file using the UI provided by the cmf-server.
 
-Follow [here](../cmf_server/cmf-server.md) for details on how to setup a cmf-server.
 
 ## How to effectively use cmf-client?
 
@@ -42,37 +37,43 @@ mkdir example-folder
 
 **Initialize cmf**
 
-CMF initialization is the first and foremost to use cmf-client commads. This command in one go complete initialization process making cmf-client user friendly.     Execute `cmf init` in the `example-folder` directory created in the [above](#create-a-folder) step.
+CMF initialization is the first and foremost to use cmf-client commads. This command in one go complete initialization process making cmf-client user friendly.     Execute `cmf init` in the `example-folder` directory created in the above step.
 ```
-cmf init minioS3 --url s3://bucket-name --endpoint-url http://localhost:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-ip http://x.x.x.x:8080  --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://X.X.X.X:7687
+cmf init minioS3 --url s3://dvc-art --endpoint-url http://x.x.x.x:9000 --access-key-id minioadmin --secret-key minioadmin --git-remote-url https://github.com/user/experiment-repo.git --cmf-server-url http://x.x.x.x:8080  --neo4j-user neo4j --neo4j-password password --neo4j-uri bolt://localhost:7687
 ```
-Check [here](./cmf_client.md) for more details.
+> Here, "dvc-art" is provided as an example bucket name. However, users can change it as needed, if the user chooses to change it, they will need to update the Dockerfile for minioS3 accordingly.
+
+Check [Overview](./cmf_client.md) page for more details.
 
 **Check status of CMF initialization (Optional)**
 ```
 cmf init show
 ```
-Check [here](./cmf_client.md) for more details.
+Check [Overview](./cmf_client.md) page for more details.
 
 **Track metadata using cmflib**
 
-Use [Sample projects](https://github.com/HewlettPackard/cmf/tree/master/examples) as a reference to create a new project to track metadata for ML pipelines.
+Use [Sample projects](https://github.com/HewlettPackard/cmf/blob/master/examples/example-get-started/README.md) as a reference to create a new project to track metadata for ML pipelines.
+
+More information is available inside [Getting Started](https://hewlettpackard.github.io/cmf/examples/getting_started/).
+
+
+> Before pushing artifacts or metadata, ensure that the cmf server and minioS3 are up and running.
 
-More info is available [here](https://hewlettpackard.github.io/cmf/examples/getting_started/).
 
 **Push artifacts**
 
 Push artifacts in the artifact repo initialised in the [Initialize cmf](#initialize-cmf) step.
 ```
 cmf artifact push 
 ```
-Check [here](./cmf_client.md) for more details.
+Check [Overview](./cmf_client.md) page for more details.
 
 **Push metadata to cmf-server**
 ```
 cmf metadata push -p 'Test-env'
 ```
-Check [here](./cmf_client.md) for more details.
+Check [Overview](./cmf_client.md) page for more details.
 
 ### cmf-client with collaborative development
 In the case of collaborative development, in addition to the above commands, users can follow the commands below to pull metadata and artifacts from a common cmf server and a central artifact repository.
@@ -83,15 +84,15 @@ Execute `cmf metadata` command in the `example_folder`.
 ```
 cmf metadata pull -p 'Test-env'
 ```
-Check [here](./cmf_client.md) for more details.
+Check [Overview](./cmf_client.md) page for more details.
 
 **Pull artifacts from the central artifact repo**
 
 Execute `cmf artifact` command in the `example_folder`.
 ```
-cmf artifact pull -p "Test-env"
+cmf artifact pull -p 'Test-env'
 ```
-Check [here](./cmf_client.md) for more details.
+Check [Overview](./cmf_client.md) page for more details.
 
 ## Flow Chart for cmf
 <img src="./../../assets/flow_chart_cmf.jpg" alt="Flow chart for cmf" style="display: block; margin: 0 auto" />