Skip to content

Commit

Permalink
add documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
Aditya Abhishek committed Aug 17, 2023
1 parent b65c31a commit 155d480
Show file tree
Hide file tree
Showing 2 changed files with 27 additions and 31 deletions.
1 change: 1 addition & 0 deletions website/docs/guides/0010-command-line.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ on the system.
| --a, --agentId, --clientId=\<id\> | No | string/text | An identifier that can be used to uniquely identify the instance of the Virtual Client in telemetry separate from other instances. The default value is the name of the system if this option is not explicitly defined (i.e. the name as defined by the operating system). |
| --port, --api-port=\<port\> | No | integer | The port to use for hosting the Virtual Client REST API service for profiles that allow multi-system, client/server operations (e.g. networking). Additionally, a port may be defined for each role associated with the profile operations using the format {Port}/{Role} with each port/role combination delimited by a comma (e.g. 4501/Client,4502/Server). |
| --cs, --content, --contentStore=\<authtoken\> | No | string/connection string/SAS | A full connection string or SAS URI to an Azure storage account blob store where files/content can be uploaded as part of background monitoring processes. Contact the VC Team to get a SAS URI for your team. See [Azure Storage Account Integration](./0600-integration-blob-storage.md). |
| --cspt, --contentPath, --contentPathPattern=\<folderPattern\> | No | string/text | The content path format/structure to use when uploading content to target storage resources. When not defined the 'Default' structure is used. Default: "{experimentId}/{agentId}/{toolName}/{role}/{scenario}" |
| --eh, --eventHub, --eventHubConnectionString=\<accesspolicy\> | No | string/connection string | A full connection string/access policy for the Azure Event Hub namespace where telemetry should be written. Contact the VC Team to get an access policy for your team. See [Azure Event Hub Integration](./0610-integration-event-hub.md). |
| --e, --experimentId=\<guid\> | No | guid | A unique identifier that defines the ID of the experiment for which the Virtual Client workload is associated. |
| --lp, --layoutPath=\<path\> | No | string/path | A path to a environment layout file that provides additional metadata about the system/hardware on which the Virtual Client will run and information required to support client/server advanced topologies. See [Client/Server Support](./0020-client-server.md). |
Expand Down
57 changes: 26 additions & 31 deletions website/docs/guides/0600-integration-blob-storage.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -89,24 +89,22 @@ are granted to the Virtual Client application for uploading and downloading blob


### Blob Store Folder/File Naming Conventions
In order to ensure that files associated with Virtual Client executors or monitors are easy to find in the blob stores, Virtual Client supports a flexible blob path template.
At a high level, all files associated with a given experiment are contained together in the blob store. The following pieces of information are included
in the virtual path and file name that is used to ensure that files associated with that experiment are organized in the blob store:
In order to ensure that files associated with Virtual Client executors or monitors are easy to find in the blob stores, Virtual Client supports a flexible blob path pattern.
At a high level, all files associated with a given experiment are contained together in the blob store.

The virtual path of uploaded logs in blob storage is controlled by a VirtualClient Component parameter "ContentPathTemplate".
The virtual path of uploaded logs in blob storage is controlled by a VirtualClient Command Line Option parameter "--contentPathPattern".

Example: ContentPathTemplate="stringValue1/{parameter1}abc{parameter2}/{standardProperty1}/stringValue2/{parameter3}/{standardProperty2}".
Example: --contentPathPattern="any-value1/{standardProperty1}any-value2{standardProperty2}/{standardProperty3}/any-value3/{standardProperty4}".

In above example, the virtual blob folder structure will have sub-folders corresponding to each element separated by a '/' in
ContentPathTemplate. The inlined values that are enclosed within brackets "{}", like "parameter1" and "standaradProperty1",
needs to be either a standard property of Virtual Client (ExperimentId, AgentId, ToolName, Role, Scenario) or needs to be provided
as a parameter to Virtual Client commandline or in Profiles.
In above example, the virtual blob folder structure will have sub-folders corresponding to each element separated by a '/' in the
ContentPathPattern. The inlined values that are enclosed within brackets "{}", like "standaradProperty1" and "standaradProperty2",
needs to be one among the 5 defined standard properties of Virtual Client (ExperimentId, AgentId, ToolName, Role, Scenario).

The first component of ContentPathTemplate (stringValue1 in above example) will be taken up as the name of Blob storage Container where all files will be uploaded.
The next component ({parameter1} in above example) will be the root folder within the container and so on for the complete virtual folder structure within the blob storage.
The first component of ContentPathPattern (any-value1 in above example) will be taken up as the name of Blob storage Container where all files will be uploaded.
The next component ({standardProperty1} in above example) will be the root folder within the container and so on for the complete virtual folder structure within the blob storage.

The default value of "ContentPathTemplate" is "{experimentId}/{agentId}/{toolName}/{role}/{scenario}". In the default template, each element
is a standard used in Virtual Client.
The default value of "ContentPathPattern" is "{experimentId}/{agentId}/{toolName}/{role}/{scenario}". In the default template, each element
is a standard property identified by Virtual Client.

* **Experiment ID**
The ID of the experiment is used to ensure all files associated with any executors or monitors are in the same virtual folder within the blob store.
Expand All @@ -126,12 +124,12 @@ is a standard used in Virtual Client.
* **ScenarioName**
It is an indicator of the scenario being tested by the workload. It is defined for each action/monitor in an execution profile.

All files will be uploaded in a virtual folder structure as defined by the ContentPathTemplate. For each file uploaded, a timestamp will be
All files will be uploaded in a virtual folder structure as defined by the ContentPathPattern. For each file uploaded, a timestamp will be
prefixed to it so as to provide unique names.

* **File Name**
This is the name of the file as it should be reflected in the blob store. A timestamp (round-trip date/time format) will be added to the beginning of the file name
(e.g. 2022-03-07T01:32:27.1237655Z-monitor.log). The addition of the timestamp ensures that the files within a given virtual folder in the blob store will all
(e.g. 2023-08-17t0637583781z-monitor.log). The addition of the timestamp ensures that the files within a given virtual folder in the blob store will all
have unique names in order to avoid collisions. Round-trip date/time formats in addition to being a valid timestamp are naturally sortable in UX experiences
such as a web browser.

Expand All @@ -142,35 +140,32 @@ Given the pieces of information noted above, the format for virtual paths and fi


``` csharp
// ContentPathTemplate and Parameters as defined in Virtual Client CommandLine:
// ContentPathPattern and Parameters as defined in Virtual Client CommandLine:
// -----------------------------------------------
VirtualClient.exe --profile=........ --parameters="ContentPathTemplate={Parameter1}/stringvalue1/{ExperimentID}_expt/stringvalue2/{AgentID}/{Parameter2},results_{Parameter3}/{ToolName},,,Parameter1=value1,,,Parameter2=value2,,,Parameter3=value3"
VirtualClient.exe --profile=........ --contentPathPattern="value1/expt_{experimentId}_agent_{agentId}/{toolName}/value-2"

// Examples:
// -----------------------------------------------
// Given the Following Information:
// - Experiment ID = 24149a49-66c9-4bd1-9332-18370c7c70e1
// - Agent ID = cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01
// - ToolName - a Background Monitor = ExampleMonitor
// - ToolName - a Background Monitor = exampleMonitor
// - Files Produced = monitor.log
//
// The blob virtual folder paths/names would like the following:
value1/stringvalue1/24149a49-66c9-4bd1-9332-18370c7c70e1_expt/stringvalue2/cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01/value2,results_value3/examplemonitor/2022-03-07T01:32:27.1237655Z-monitor.log
value1/stringvalue1/24149a49-66c9-4bd1-9332-18370c7c70e1_expt/stringvalue2/cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01/value2,results_value3/examplemonitor/2022-03-07T01:34:32.6483092Z-monitor.log
value1/stringvalue1/24149a49-66c9-4bd1-9332-18370c7c70e1_expt/stringvalue2/cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01/value2,results_value3/examplemonitor/2022-03-07T01:36:30.2645013Z-monitor.log
value1/expt_24149a49-66c9-4bd1-9332-18370c7c70e1_expt_agent_cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01/examplemonitor/value-2/2022-03-07T01:32:27.1237655Z-monitor.log
value1/expt_24149a49-66c9-4bd1-9332-18370c7c70e1_expt_agent_cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01/examplemonitor/value-2/2022-03-07T01:34:32.6483092Z-monitor.log
value1/expt_24149a49-66c9-4bd1-9332-18370c7c70e1_expt_agent_cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01/examplemonitor/value-2/2022-03-07T01:36:30.2645013Z-monitor.log

// Structure Within the Blob Store:
// -----------------------------------------------
// (container) value1
// -> (virtual folder) /stringvalue1
// -> (virtual folder) /24149a49-66c9-4bd1-9332-18370c7c70e1_expt
// -> (virtual folder) /stringvalue2
// -> (virtual folder) /cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01
// -> (virtual folder) /value2,results_value3
// -> (virtual folder) /examplemonitor
// -> (blob) 2022-03-07T01:32:27.1237655Z-monitor.log
// -> (blob) 2022-03-07T01:34:32.6483092Z-monitor.log
// -> (blob) 2022-03-07T01:36:30.2645013Z-monitor.log
// (container) value1
// -> (virtual folder) /expt_24149a49-66c9-4bd1-9332-18370c7c70e1_agent_cluster01,cc296787-aee6-4ce4-b814-180627508d12,anyvm-01
// -> (virtual folder) /examplemonitor
// -> (virtual folder) /value-2
// -> (blob) 2022-03-07T01:32:27.1237655Z-monitor.log
// -> (blob) 2022-03-07T01:34:32.6483092Z-monitor.log
// -> (blob) 2022-03-07T01:36:30.2645013Z-monitor.log
```

---------
Expand Down

0 comments on commit 155d480

Please sign in to comment.