From 205963b15a077f557f0de1b729f0cd18c3cefc62 Mon Sep 17 00:00:00 2001 From: Andrie de Vries Date: Fri, 16 Jun 2017 13:05:11 +0100 Subject: [PATCH 1/7] Fix errors and typos in vignettes #75 --- inst/doc/Authentication.Rmd | 57 +++++++++++++--------- inst/doc/Authentication.html | 74 +++++++++++++++++----------- inst/doc/tutorial.R | 21 ++++---- inst/doc/tutorial.Rmd | 95 +++++++++++++++++++++--------------- inst/doc/tutorial.html | 86 ++++++++++++++++++-------------- vignettes/Authentication.Rmd | 57 +++++++++++++--------- vignettes/tutorial.Rmd | 95 +++++++++++++++++++++--------------- 7 files changed, 291 insertions(+), 194 deletions(-) diff --git a/inst/doc/Authentication.Rmd b/inst/doc/Authentication.Rmd index 1deae5f..5129784 100644 --- a/inst/doc/Authentication.Rmd +++ b/inst/doc/Authentication.Rmd @@ -1,75 +1,88 @@ --- -title: "Getting Authorised for the AzureSMR Package" +title: "Authenticating AzureSMR to call the Azure API" author: "Alan Weaver and Andrie de Vries" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > - %\VignetteIndexEntry{AzureSMR tutorial} + %\VignetteIndexEntry{AzureSMR authentication} %\VignetteEngine{knitr::rmarkdown} %\usepackage[utf8]{inputenc} --- ## Configuration instructions -Follow these instructions to create an active directory. You will need to collect the tenant ID (`tenantID`), client ID (`clientID`) and authentication key (`authKey`) in order to authenticate with the `createAzureContect()` function. +To use the `AzureSMR` package, you must create an Azure Active Directory application with permisssions. This vignette contains instructions to do this. -### Create an Active Directory App. +You must collect three pieces of information to authenticate with the `createAzureContect()` function: -1. Login to the Classic (i.e., the old) Portal https://manage.windowsazure.com/. +* tenant ID (`tenantID`) +* client ID (`clientID`) +* authentication key (`authKey`) -2. On the left hand menu you should see amongst all the items one called ACTIVE DIRECTORY. Click the item and an active directory DIRECTORY will be listed. +## Create an Active Directory application. + +1. Login to the Classic (the old) Portal https://manage.windowsazure.com/. + +2. On the left hand menu you should see amongst all the items one called `ACTIVE DIRECTORY`. Click the item and an active directory DIRECTORY will be listed. 3. Click on an entry under the Name column (if there is only one entry the choice is easy!) to take you to a page of options to get started with some text like I WANT TO. -4. Along the top menu click APPLICATIONS. +4. Along the top menu click `APPLICATIONS`. -5. You probably want to create a new application so type a name for it in the Search box (I called mine AzureSMR). The search result will come back with no results and a button that says `ADD AN APPLICATION` -> which you should click. +5. You probably want to create a new application so type a name for it in the Search box (I called mine `AzureSMR`). The search result will come back with no results and a button that says `ADD AN APPLICATION` -> which you should click. 6. Give the application a name and choose `WEB APPLICATION AND/OR WEB API`. Then go to the next page `->`. 7. Provide some dummy URLs. They are not used but they must be valid URLs. Click on the tick to continue to create the application. -8. Under the Configure menu button take note of the client ID.9. Under the Keys section choose a 1 year duration (or 2) and click the Save button azToken the bottom of the screen. A key is generated which you should copy now and save it somewhere. +8. Under the Configure menu button take note of the **client ID**. + +9. Under the `Keys` section choose a 1 year duration (or 2) and click the Save button at the bottom of the screen. An **authenticatio key** is generated which you should copy now and save it somewhere. + +10. You also need the **tenant ID**. Click the `VIEW ENDPOINTS` button on the bottom of the screen and find a list of endpoints all including the tenant ID as a sequence of hexadecimals. -10. We also need the *tenant ID*. Click the `VIEW ENDPOINTS` button on the bottom of the screen and find a list of endpoints all including the tenant ID as a sequence of hexadecimals. +11. Now set up the applications permissions. Click the `Add application` button. In the resulting window scroll to `Windows Azure Service Management API` and select it. Then click the Tick icon. -11. Now we need to set up the applications permissions. Click the `Add application` button. In the resulting window scroll to `Windows Azure Service Management API` and select it. Then click the Tick icon. +12. Under the resulting "permissions to other applications" section, for the Windows Azure Service Management API entry, from the `Delegated Permissions` drop down tick the Access Azure Service Management as organization. -12. Under the resulting "permissions to other applications" section, for the Windows Azure Service Management API entry, from the "Delegated Permissions:0" drop down tick the Access Azure Service Management as organization. +13. Click on the Save icon at the bottom of the window again. -13. Click on the Save icon azToken the bottom of the window again. +14. Now assign the application a role and to do so you go to the (new) Azure portal. https://portal.azure.com/ -14. Now we need to assign the application a role and to do so we go to the (new) Azure portal. https://portal.azure.com/ +## Access control -To apply access control azToken Resource Group +You can apply access control at either the resource group level or the subscription level + +### To apply access control at Resource Group 15. Click on Resource Groups menu item on the left in the portal. 16. Identify the resource group you will associate with this application. -17. Choose the Access Control(IAM) menu item from the Resource scope. +17. Choose the `Access Control (IAM)` menu item from the Resource scope. 18. In the resulting scope click the `+ Add` button. -19. Choose the role as Owner and under the user search box enter the name of the App, e.g., AzureSMR. +19. Choose the role as Owner and under the user search box enter the name of the App, e.g. `AzureSMR`. -20. Select the resulting list item for that App then click Select in that scope then OK in the "Add access" scope. The user will be added to the list. +20. Select the resulting list item for that App then click Select in that scope then OK in the `Add access` scope. The user will be added to the list. -Alternatively you can access control azToken Subscription Level +### Alternatively you can access control at Subscription Level 15. Click on Subscriptions on the left menu item in the portal. 16. Identify the Subscription you will associate with this application. -17. Choose the Access Control(IAM) menu item. +17. Choose the `Access Control (IAM)` menu item. -18. In the resulting scope click the + Add button. +18. In the resulting scope click the `+ Add` button. -19. Choose the role as Owner and under the user search box enter the name of the App, e.g., AzureSMR. +19. Choose the role as Owner and under the user search box enter the name of the App, e.g. `AzureSMR`. 20. Select the resulting list item for that App then click Select in that scope then OK in the "Add access" scope. The user will be added to the list. +## Conclusion That is all. You can test this by trying: diff --git a/inst/doc/Authentication.html b/inst/doc/Authentication.html index 188416d..6d867a2 100644 --- a/inst/doc/Authentication.html +++ b/inst/doc/Authentication.html @@ -4,7 +4,7 @@ - + @@ -12,9 +12,9 @@ - + -Getting Authorised for the AzureSMR Package +Authenticating AzureSMR to call the Azure API @@ -68,57 +68,75 @@ -

Getting Authorised for the AzureSMR Package

+

Authenticating AzureSMR to call the Azure API

Alan Weaver and Andrie de Vries

-

2017-06-15

+

2017-06-16

Configuration instructions

-

Follow these instructions to create an active directory. You will need to collect the tenant ID (tenantID), client ID (clientID) and authentication key (authKey) in order to authenticate with the createAzureContect() function.

-
-

Create an Active Directory App.

+

To use the AzureSMR package, you must create an Azure Active Directory application with permisssions. This vignette contains instructions to do this.

+

You must collect three pieces of information to authenticate with the createAzureContect() function:

+
    +
  • tenant ID (tenantID)
  • +
  • client ID (clientID)
  • +
  • authentication key (authKey)
  • +
+
+
+

Create an Active Directory application.

    -
  1. Login to the Classic (i.e., the old) Portal https://manage.windowsazure.com/.

  2. -
  3. On the left hand menu you should see amongst all the items one called ACTIVE DIRECTORY. Click the item and an active directory DIRECTORY will be listed.

  4. +
  5. Login to the Classic (the old) Portal https://manage.windowsazure.com/.

  6. +
  7. On the left hand menu you should see amongst all the items one called ACTIVE DIRECTORY. Click the item and an active directory DIRECTORY will be listed.

  8. Click on an entry under the Name column (if there is only one entry the choice is easy!) to take you to a page of options to get started with some text like I WANT TO.

  9. -
  10. Along the top menu click APPLICATIONS.

  11. -
  12. You probably want to create a new application so type a name for it in the Search box (I called mine AzureSMR). The search result will come back with no results and a button that says ADD AN APPLICATION -> which you should click.

  13. +
  14. Along the top menu click APPLICATIONS.

  15. +
  16. You probably want to create a new application so type a name for it in the Search box (I called mine AzureSMR). The search result will come back with no results and a button that says ADD AN APPLICATION -> which you should click.

  17. Give the application a name and choose WEB APPLICATION AND/OR WEB API. Then go to the next page ->.

  18. Provide some dummy URLs. They are not used but they must be valid URLs. Click on the tick to continue to create the application.

  19. -
  20. Under the Configure menu button take note of the client ID.9. Under the Keys section choose a 1 year duration (or 2) and click the Save button azToken the bottom of the screen. A key is generated which you should copy now and save it somewhere.

  21. -
  22. We also need the tenant ID. Click the VIEW ENDPOINTS button on the bottom of the screen and find a list of endpoints all including the tenant ID as a sequence of hexadecimals.

  23. -
  24. Now we need to set up the applications permissions. Click the Add application button. In the resulting window scroll to Windows Azure Service Management API and select it. Then click the Tick icon.

  25. -
  26. Under the resulting “permissions to other applications” section, for the Windows Azure Service Management API entry, from the “Delegated Permissions:0” drop down tick the Access Azure Service Management as organization.

  27. -
  28. Click on the Save icon azToken the bottom of the window again.

  29. -
  30. Now we need to assign the application a role and to do so we go to the (new) Azure portal. https://portal.azure.com/

  31. +
  32. Under the Configure menu button take note of the client ID.

  33. +
  34. Under the Keys section choose a 1 year duration (or 2) and click the Save button at the bottom of the screen. An authenticatio key is generated which you should copy now and save it somewhere.

  35. +
  36. You also need the tenant ID. Click the VIEW ENDPOINTS button on the bottom of the screen and find a list of endpoints all including the tenant ID as a sequence of hexadecimals.

  37. +
  38. Now set up the applications permissions. Click the Add application button. In the resulting window scroll to Windows Azure Service Management API and select it. Then click the Tick icon.

  39. +
  40. Under the resulting “permissions to other applications” section, for the Windows Azure Service Management API entry, from the Delegated Permissions drop down tick the Access Azure Service Management as organization.

  41. +
  42. Click on the Save icon at the bottom of the window again.

  43. +
  44. Now assign the application a role and to do so you go to the (new) Azure portal. https://portal.azure.com/

-

To apply access control azToken Resource Group

+
+
+

Access control

+

You can apply access control at either the resource group level or the subscription level

+
+

To apply access control at Resource Group

  1. Click on Resource Groups menu item on the left in the portal.

  2. Identify the resource group you will associate with this application.

  3. -
  4. Choose the Access Control(IAM) menu item from the Resource scope.

  5. +
  6. Choose the Access Control (IAM) menu item from the Resource scope.

  7. In the resulting scope click the + Add button.

  8. -
  9. Choose the role as Owner and under the user search box enter the name of the App, e.g., AzureSMR.

  10. -
  11. Select the resulting list item for that App then click Select in that scope then OK in the “Add access” scope. The user will be added to the list.

  12. +
  13. Choose the role as Owner and under the user search box enter the name of the App, e.g. AzureSMR.

  14. +
  15. Select the resulting list item for that App then click Select in that scope then OK in the Add access scope. The user will be added to the list.

-

Alternatively you can access control azToken Subscription Level

+
+
+

Alternatively you can access control at Subscription Level

  1. Click on Subscriptions on the left menu item in the portal.

  2. Identify the Subscription you will associate with this application.

  3. -
  4. Choose the Access Control(IAM) menu item.

  5. -
  6. In the resulting scope click the + Add button.

  7. -
  8. Choose the role as Owner and under the user search box enter the name of the App, e.g., AzureSMR.

  9. +
  10. Choose the Access Control (IAM) menu item.

  11. +
  12. In the resulting scope click the + Add button.

  13. +
  14. Choose the role as Owner and under the user search box enter the name of the App, e.g. AzureSMR.

  15. Select the resulting list item for that App then click Select in that scope then OK in the “Add access” scope. The user will be added to the list.

+
+
+
+

Conclusion

That is all. You can test this by trying:

sc <- createAzureContext(tenantID = "{TID}", clientID = "{CID}", authKey= "{KEY}")
 rgs <- azureListRG(sc)
 rgs

For more information see the tutorial Use portal to create Active Directory application and service principal that can access resources

-
@@ -127,7 +145,7 @@

Create an Active Directory App.

(function () { var script = document.createElement("script"); script.type = "text/javascript"; - script.src = "https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; + script.src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; document.getElementsByTagName("head")[0].appendChild(script); })(); diff --git a/inst/doc/tutorial.R b/inst/doc/tutorial.R index affd947..ae9eda1 100644 --- a/inst/doc/tutorial.R +++ b/inst/doc/tutorial.R @@ -4,13 +4,15 @@ # devtools::install_github("Microsoft/AzureSMR") # library(AzureSMR) +## ---- eval=FALSE--------------------------------------------------------- +# library(AzureSMR) + ## ---- eval=FALSE--------------------------------------------------------- # sc <- createAzureContext(tenantID = "{TID}", clientID = "{CID}", authKey= "{KEY}") # sc ## ---- eval=FALSE--------------------------------------------------------- # azureListSubscriptions(sc) -# ## ---- eval=FALSE--------------------------------------------------------- # # list resource groups @@ -23,16 +25,14 @@ # # azureListAllResources(sc, type = "Microsoft.Sql/servers", location = "northeurope") # -# # azureCreateResourceGroup(sc, resourceGroup = "testme", location = "northeurope") # # azureCreateStorageAccount(sc,storageAccount="testmystorage1",resourceGroup = "testme") # # azureListAllResources(sc, resourceGroup = "testme") # -# # When finished, to delete a Resource Group use azureDeleteResourceGroup +# # When finished, to delete a Resource Group use azureDeleteResourceGroup() # azureDeleteResourceGroup(sc, resourceGroup = "testme") -# ## ---- eval=FALSE--------------------------------------------------------- # ## List VMs in a ResourceGroup @@ -45,10 +45,10 @@ # azureStopVM(sc, vmName = "DSVM1") ## ---- eval=FALSE--------------------------------------------------------- -# sKey <- azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") +# azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") ## ---- eval=FALSE--------------------------------------------------------- -# azureCreateStorageContainer(sc,"opendata",storageAccount = "testmystorage1", resourceGroup = "testme") +# azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme") ## ---- eval=FALSE--------------------------------------------------------- # azureListStorageContainers(sc, storageAccount = "testmystorage1", resourceGroup = "testme") @@ -66,6 +66,9 @@ # blob="HELLO", # type="text") +## ---- eval=FALSE--------------------------------------------------------- +# azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "opendata") + ## ---- eval=FALSE--------------------------------------------------------- # azureCreateHDI(sc, # resourceGroup = "testme", @@ -79,7 +82,6 @@ ## ---- eval=FALSE--------------------------------------------------------- # azureListHDI(sc, resourceGroup ="testme") -# ## ---- eval=FALSE--------------------------------------------------------- # azureResizeHDI(sc, resourceGroup = "testme", clustername = "smrhdi", role="workernode",size=3) @@ -108,6 +110,7 @@ # azureHiveStatus(sc, clusterName = "smrhdi", # hdiAdmin = "hdiadmin", # hdiPassword = "AzureSMR_password123") +# # azureHiveSQL(sc, # CMD = "select * from hivesampletable", # path = "wasb://opendata@testmystorage1.blob.core.windows.net/") @@ -116,8 +119,7 @@ # # stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout") # -# read.delim(text=stdout, header=TRUE, fill=TRUE) -# +# read.delim(text = stdout, header = TRUE, fill = TRUE) ## ---- eval=FALSE--------------------------------------------------------- # azureSparkNewSession(sc, clustername = "smrhdi", @@ -162,5 +164,4 @@ # kind = "sparkr") # azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2") # azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2") -# diff --git a/inst/doc/tutorial.Rmd b/inst/doc/tutorial.Rmd index a6ba176..0a54a27 100644 --- a/inst/doc/tutorial.Rmd +++ b/inst/doc/tutorial.Rmd @@ -9,7 +9,7 @@ vignette: > %\usepackage[utf8]{inputenc} --- -Use this package to manage Azure Resources from within an R Session. This is not a full SDK just a collection of functions that should prove useful for a Data Scientist who needs to access and manage Azure Resources. +Use this package to manage Azure Resources from within an R Session. This package does not expose the complete Azure API, but is meant as a collection of functions that a typical data scientist may use to access and manage Azure Resources. ## Installation instructions @@ -25,7 +25,7 @@ library(AzureSMR) ## Overview -AzureSMR provides an interface to manage resources on Microsoft Azure. The main functions address the following Azure Services: +`AzureSMR` provides an interface to manage resources on Microsoft Azure. The main functions address the following Azure Services: - Azure Blob: List, Read and Write to Blob Services - Azure Resources: List, Create and Delete Azure Resource @@ -35,35 +35,42 @@ AzureSMR provides an interface to manage resources on Microsoft Azure. The main - Azure Spark: List and create Spark jobs/Sessions against a HDInsight Cluster(Livy) - EXPERIMENTAL -For a detailed list of AzureSM functions and their syntax please refer to the Help pages. +For a detailed list of `AzureSMR` functions and their syntax please refer to the Help pages. -## Getting Authorisation configured +## Configuring authorisation in Azure Active Directory -To get started, please refer to the Authorisation tutorial. https://github.com/Microsoft/AzureSMR/blob/master/vignettes/Authentication.Rmd +To get started, please refer to the [authorisation tutorial](http://htmlpreview.github.io/?https://github.com/Microsoft/AzureSMR/blob/master/inst/doc/Authentication.html) -## Authenticating against the service +## Load the package -The AzureAPIs require lots of parameters to be managed. Rather than supplying all the paramters for each function call AzureSMR implements an AzureContext Variable which caches the last time a paramters is used so that it doesnt need to be repeatedly supplied. +```{r, eval=FALSE} +library(AzureSMR) +``` + +## Authenticating against the Azure service -To create an AzureContext object and attempt to authenticate against the Azure service, use: +The Azure APIs require many parameters to be managed. Rather than supplying all the arguments to every function call, `AzureSMR` uses an `azureActiveContext` object that caches arguments so you don't have to supply . + +To create an `azureActiveContext` object and attempt to authenticate against the Azure service, use: ```{r, eval=FALSE} sc <- createAzureContext(tenantID = "{TID}", clientID = "{CID}", authKey= "{KEY}") sc ``` -If you provide autentication paramters to createAzureContext() the function will automatically authenticate. -To manually get an authorisation token use `azureAuthenticate()`. -Note this token will time our after a period and therefore you need to run it again occasionally. TIP: Use AzureAuthenticate before a long running task. +If you provide autentication paramters to `createAzureContext()` the function will automatically authenticate. To manually get an authorisation token use `azureAuthenticate()`. Note this token will time out after a period and therefore you need to run it again occasionally. TIP: Use `azureAuthenticate()` before a long running task. -The `azureListSubscriptions()` function lists all the available subscriptions. If you only have one it sets the default Subscription in the `azureActiveContext` to that subscription ID. +## Subscriptions + +The `azureListSubscriptions()` function lists all the available subscriptions. If you only have one it sets the default Subscription in the `azureActiveContext` to that subscription ID. ```{r, eval=FALSE} azureListSubscriptions(sc) - ``` + + ## Manage resource Groups ```{r, eval=FALSE} @@ -77,24 +84,22 @@ azureListAllResources(sc, location = "northeurope") azureListAllResources(sc, type = "Microsoft.Sql/servers", location = "northeurope") - azureCreateResourceGroup(sc, resourceGroup = "testme", location = "northeurope") azureCreateStorageAccount(sc,storageAccount="testmystorage1",resourceGroup = "testme") azureListAllResources(sc, resourceGroup = "testme") -# When finished, to delete a Resource Group use azureDeleteResourceGroup +# When finished, to delete a Resource Group use azureDeleteResourceGroup() azureDeleteResourceGroup(sc, resourceGroup = "testme") - ``` ## Manage Virtual Machines -Use these functions to list, start and stop existing Virtual Machines (see templates for Creation). +Use these functions to list, start and stop existing Virtual Machines (see templates for creation). -To Create VMs please refer to Resource Templates below. +To create VMs please refer to Resource Templates below. ```{r, eval=FALSE} ## List VMs in a ResourceGroup @@ -107,19 +112,19 @@ azureStartVM(sc, vmName = "DSVM1") azureStopVM(sc, vmName = "DSVM1") ``` -## Access Storage Blobs +## Accessing storage blobs using the `azureActiveContext` -In order to access Storage Blobs you need to have a key. Use `azureSAGetKey()` to get a Key or alternatively supply your own key. When you provide your own key you no longer need to use `azureAuthenticate()` since the API uses a diferent authentication approach. +To access storage blobs you need to have a key. Use `azureSAGetKey()` to automatically retrieve your key or alternatively supply your own key. ```{r, eval=FALSE} -sKey <- azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") +azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") ``` To create containers in a storage account use`azureCreateStorageContainer()` ```{r, eval=FALSE} -azureCreateStorageContainer(sc,"opendata",storageAccount = "testmystorage1", resourceGroup = "testme") +azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme") ``` @@ -129,7 +134,7 @@ To list containers in a storage account use `azureListContainers()` azureListStorageContainers(sc, storageAccount = "testmystorage1", resourceGroup = "testme") ``` -To Write a Blobs use `azurePutBlob()` +To write a blob use `azurePutBlob()` ```{r, eval=FALSE} azurePutBlob(sc, storageAccount = "testmystorage1", container = "opendata", @@ -151,11 +156,25 @@ azureGetBlob(sc, storageAccount = "testmystorage1", container = "opendata", type="text") ``` + +## Accessing storage blobs without an `azureActiveContext` + +It is also possible to access the storage API without having an Azure Active Directory application. + +In this case, you should use the argument `azureActiveContect = NULL` to the storage functions. + +For example: + +```{r, eval=FALSE} +azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "opendata") +``` + + ## Manage HDInsight Clusters -You can use `AzureSMR` to manage Azure HDInsight clusters. To create clusters use azureCreateHDI or for advanced configurations use Resource Templates (See below). +You can use `AzureSMR` to manage HDInsight clusters. To create a cluster use `azureCreateHDI()`. -Also see functions for submitting Hive and Spark jobs. +For advanced configurations use Resource Templates (See below). ```{r, eval=FALSE} azureCreateHDI(sc, @@ -169,11 +188,10 @@ azureCreateHDI(sc, kind = "rserver") ``` -Use `azureListHDI()` to list available Clusters. +Use `azureListHDI()` to list available clusters. ```{r, eval=FALSE} azureListHDI(sc, resourceGroup ="testme") - ``` Use `azureResizeHDI()` to resize a cluster @@ -191,11 +209,12 @@ azureResizeHDI(sc, resourceGroup = "testme", clustername = "smrhdi", role="worke ## " headnode ( 2 * Standard_D3_v2 ) workernode ( 5 * Standard_D3_v2 ) zookeepernode ( 3 * Medium ) edgenode0 ( 1 * Standard_D4_v2 )" ``` + ## Resource Templates - Create Azure Resources -The easiest way to create resources on Azure is to use Azure Templates. To create Azure Resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into Settings > Automation scripts. Example templates can be found azToken this URL https://github.com/Azure/AzureStack-QuickStart-Templates. +The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates. -To create a resource using a template in AzureSM use AzureDeployTemplate. The Template and Paramters must be available in a public URL (Azure Blob). It may be worth getting the Azure Administrator to build a working template. +To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example Azure blob). ```{r, eval=FALSE} azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", @@ -209,15 +228,18 @@ azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", ## Finished: 2016-06-23 19:04:43 ``` -ADMIN TIP: If a deployment fails. Go to the Azure Portal and look azToken Actvity logs and look for failed deployments which should explain why the deployment failed. +ADMIN TIP: If a deployment fails, go to the Azure Portal and look at `Activity logs` and look for failed deployments - this should explain why the deployment failed. + ## Hive Functions -These functions facilitate the use of hive jobs on a HDInsight Cluster + +These functions facilitate the use of hive jobs on an HDInsight Cluster ```{r, eval=FALSE} azureHiveStatus(sc, clusterName = "smrhdi", hdiAdmin = "hdiadmin", hdiPassword = "AzureSMR_password123") + azureHiveSQL(sc, CMD = "select * from hivesampletable", path = "wasb://opendata@testmystorage1.blob.core.windows.net/") @@ -226,16 +248,15 @@ azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "openda stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout") -read.delim(text=stdout, header=TRUE, fill=TRUE) - +read.delim(text = stdout, header = TRUE, fill = TRUE) ``` ## Spark functions (experimental) -`AzureSMR` provides some functions that allow HDInsight Spark Sessions and jobs to be managed within an R Session +`AzureSMR` provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session. -To Create a new Spark Session (Via Livy) use `azureSparkNewSession()` +To create a new Spark session (via Livy) use `azureSparkNewSession()` ```{r, eval=FALSE} azureSparkNewSession(sc, clustername = "smrhdi", @@ -244,8 +265,7 @@ azureSparkNewSession(sc, clustername = "smrhdi", kind = "pyspark") ``` -To view the status of sessions use AzureSparkListSessions -Wait for status to be Idle +To view the status of sessions use `azureSparkListSessions()`. Wait for status to be idle. ```{r, eval=FALSE} azureSparkListSessions(sc, clustername = "smrhdi") @@ -293,6 +313,5 @@ azureSparkNewSession(sc, clustername = "smrhdi", kind = "sparkr") azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2") azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2") - ``` diff --git a/inst/doc/tutorial.html b/inst/doc/tutorial.html index 46f06ac..0a4135d 100644 --- a/inst/doc/tutorial.html +++ b/inst/doc/tutorial.html @@ -4,7 +4,7 @@ - + @@ -12,7 +12,7 @@ - + AzureSMR tutorial @@ -70,11 +70,11 @@

AzureSMR tutorial

Alan Weaver and Andrie de Vries

-

2017-06-15

+

2017-06-16

-

Use this package to manage Azure Resources from within an R Session. This is not a full SDK just a collection of functions that should prove useful for a Data Scientist who needs to access and manage Azure Resources.

+

Use this package to manage Azure Resources from within an R Session. This package does not expose the complete Azure API, but is meant as a collection of functions that a typical data scientist may use to access and manage Azure Resources.

Installation instructions

Install the development version of the package directly from GitHub with:

@@ -85,7 +85,7 @@

Installation instructions

Overview

-

AzureSMR provides an interface to manage resources on Microsoft Azure. The main functions address the following Azure Services:

+

AzureSMR provides an interface to manage resources on Microsoft Azure. The main functions address the following Azure Services:

-

For a detailed list of AzureSM functions and their syntax please refer to the Help pages.

+

For a detailed list of AzureSMR functions and their syntax please refer to the Help pages.

-
-

Getting Authorisation configured

-

To get started, please refer to the Authorisation tutorial. https://github.com/Microsoft/AzureSMR/blob/master/vignettes/Authentication.Rmd

+
+

Configuring authorisation in Azure Active Directory

+

To get started, please refer to the authorisation tutorial

-
-

Authenticating against the service

-

The AzureAPIs require lots of parameters to be managed. Rather than supplying all the paramters for each function call AzureSMR implements an AzureContext Variable which caches the last time a paramters is used so that it doesnt need to be repeatedly supplied.

-

To create an AzureContext object and attempt to authenticate against the Azure service, use:

+
+

Load the package

+
library(AzureSMR)
+
+
+

Authenticating against the Azure service

+

The Azure APIs require many parameters to be managed. Rather than supplying all the arguments to every function call, AzureSMR uses an azureActiveContext object that caches arguments so you don’t have to supply .

+

To create an azureActiveContext object and attempt to authenticate against the Azure service, use:

sc <- createAzureContext(tenantID = "{TID}", clientID = "{CID}", authKey= "{KEY}")
 sc
-

If you provide autentication paramters to createAzureContext() the function will automatically authenticate. To manually get an authorisation token use azureAuthenticate(). Note this token will time our after a period and therefore you need to run it again occasionally. TIP: Use AzureAuthenticate before a long running task.

+

If you provide autentication paramters to createAzureContext() the function will automatically authenticate. To manually get an authorisation token use azureAuthenticate(). Note this token will time out after a period and therefore you need to run it again occasionally. TIP: Use azureAuthenticate() before a long running task.

+
+
+

Subscriptions

The azureListSubscriptions() function lists all the available subscriptions. If you only have one it sets the default Subscription in the azureActiveContext to that subscription ID.

azureListSubscriptions(sc)
@@ -122,20 +129,19 @@

Manage resource Groups

azureListAllResources(sc, type = "Microsoft.Sql/servers", location = "northeurope") - azureCreateResourceGroup(sc, resourceGroup = "testme", location = "northeurope") azureCreateStorageAccount(sc,storageAccount="testmystorage1",resourceGroup = "testme") azureListAllResources(sc, resourceGroup = "testme") -# When finished, to delete a Resource Group use azureDeleteResourceGroup +# When finished, to delete a Resource Group use azureDeleteResourceGroup() azureDeleteResourceGroup(sc, resourceGroup = "testme")

Manage Virtual Machines

-

Use these functions to list, start and stop existing Virtual Machines (see templates for Creation).

-

To Create VMs please refer to Resource Templates below.

+

Use these functions to list, start and stop existing Virtual Machines (see templates for creation).

+

To create VMs please refer to Resource Templates below.

## List VMs in a ResourceGroup
 azureListVM(sc, resourceGroup = "testme")
 
@@ -145,15 +151,15 @@ 

Manage Virtual Machines

azureStartVM(sc, vmName = "DSVM1") azureStopVM(sc, vmName = "DSVM1")
-
-

Access Storage Blobs

-

In order to access Storage Blobs you need to have a key. Use azureSAGetKey() to get a Key or alternatively supply your own key. When you provide your own key you no longer need to use azureAuthenticate() since the API uses a diferent authentication approach.

-
sKey <- azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1")
+
+

Accessing storage blobs using the azureActiveContext

+

To access storage blobs you need to have a key. Use azureSAGetKey() to automatically retrieve your key or alternatively supply your own key.

+
azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1")

To create containers in a storage account useazureCreateStorageContainer()

-
azureCreateStorageContainer(sc,"opendata",storageAccount = "testmystorage1", resourceGroup = "testme")
+
azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme")

To list containers in a storage account use azureListContainers()

azureListStorageContainers(sc, storageAccount = "testmystorage1", resourceGroup = "testme")
-

To Write a Blobs use azurePutBlob()

+

To write a blob use azurePutBlob()

azurePutBlob(sc, storageAccount = "testmystorage1", container = "opendata", 
              contents = "Hello World",
              blob = "HELLO") 
@@ -164,10 +170,17 @@

Access Storage Blobs

blob="HELLO", type="text")
+
+

Accessing storage blobs without an azureActiveContext

+

It is also possible to access the storage API without having an Azure Active Directory application.

+

In this case, you should use the argument azureActiveContect = NULL to the storage functions.

+

For example:

+
azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "opendata")
+

Manage HDInsight Clusters

-

You can use AzureSMR to manage Azure HDInsight clusters. To create clusters use azureCreateHDI or for advanced configurations use Resource Templates (See below).

-

Also see functions for submitting Hive and Spark jobs.

+

You can use AzureSMR to manage HDInsight clusters. To create a cluster use azureCreateHDI().

+

For advanced configurations use Resource Templates (See below).

azureCreateHDI(sc,
                  resourceGroup = "testme",
                  clustername = "smrhdi", # only low case letters, digit, and dash.
@@ -177,7 +190,7 @@ 

Manage HDInsight Clusters

sshUser = "hdisshuser", sshPassword = "AzureSMR_password123", kind = "rserver")
-

Use azureListHDI() to list available Clusters.

+

Use azureListHDI() to list available clusters.

azureListHDI(sc, resourceGroup ="testme")

Use azureResizeHDI() to resize a cluster

azureResizeHDI(sc, resourceGroup = "testme", clustername = "smrhdi", role="workernode",size=3)
@@ -193,8 +206,8 @@ 

Manage HDInsight Clusters

Resource Templates - Create Azure Resources

-

The easiest way to create resources on Azure is to use Azure Templates. To create Azure Resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into Settings > Automation scripts. Example templates can be found azToken this URL https://github.com/Azure/AzureStack-QuickStart-Templates.

-

To create a resource using a template in AzureSM use AzureDeployTemplate. The Template and Paramters must be available in a public URL (Azure Blob). It may be worth getting the Azure Administrator to build a working template.

+

The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into Settings > Automation scripts. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates.

+

To create a resource using a template in AzureSMR use azureDeployTemplate(). The template and paramaters must be available in a public URL (for example Azure blob).

azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", 
                     templateURL = "{TEMPLATEURL}", paramURL = "{PARAMURL}")
 
@@ -204,14 +217,15 @@ 

Resource Templates - Create Azure Resources

## RRRRRRRRRRRRRRRRRRS ## Finished Deployed Sucessfully: 2016-06-23 19:04:43 ## Finished: 2016-06-23 19:04:43
-

ADMIN TIP: If a deployment fails. Go to the Azure Portal and look azToken Actvity logs and look for failed deployments which should explain why the deployment failed.

+

ADMIN TIP: If a deployment fails, go to the Azure Portal and look at Activity logs and look for failed deployments - this should explain why the deployment failed.

Hive Functions

-

These functions facilitate the use of hive jobs on a HDInsight Cluster

+

These functions facilitate the use of hive jobs on an HDInsight Cluster

azureHiveStatus(sc, clusterName = "smrhdi", 
                 hdiAdmin = "hdiadmin", 
                 hdiPassword = "AzureSMR_password123")
+
 azureHiveSQL(sc, 
              CMD = "select * from hivesampletable", 
              path = "wasb://opendata@testmystorage1.blob.core.windows.net/")
@@ -220,17 +234,17 @@ 

Hive Functions

stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout") -read.delim(text=stdout, header=TRUE, fill=TRUE)
+read.delim(text = stdout, header = TRUE, fill = TRUE)

Spark functions (experimental)

-

AzureSMR provides some functions that allow HDInsight Spark Sessions and jobs to be managed within an R Session

-

To Create a new Spark Session (Via Livy) use azureSparkNewSession()

+

AzureSMR provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session.

+

To create a new Spark session (via Livy) use azureSparkNewSession()

azureSparkNewSession(sc, clustername = "smrhdi", 
                      hdiAdmin = "hdiadmin", 
                      hdiPassword = "AzureSMR_password123",
                      kind = "pyspark")
-

To view the status of sessions use AzureSparkListSessions Wait for status to be Idle

+

To view the status of sessions use azureSparkListSessions(). Wait for status to be idle.

azureSparkListSessions(sc, clustername = "smrhdi")

To send a command to the Spark Session use azureSparkCMD(). In this case it submits a Python routine. Ensure you preserve indents for Python.

# SAMPLE PYSPARK SCRIPT TO CALCULATE PI
@@ -273,7 +287,7 @@ 

Spark functions (experimental)

(function () { var script = document.createElement("script"); script.type = "text/javascript"; - script.src = "https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; + script.src = "https://mathjax.rstudio.com/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"; document.getElementsByTagName("head")[0].appendChild(script); })(); diff --git a/vignettes/Authentication.Rmd b/vignettes/Authentication.Rmd index 1deae5f..5129784 100644 --- a/vignettes/Authentication.Rmd +++ b/vignettes/Authentication.Rmd @@ -1,75 +1,88 @@ --- -title: "Getting Authorised for the AzureSMR Package" +title: "Authenticating AzureSMR to call the Azure API" author: "Alan Weaver and Andrie de Vries" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > - %\VignetteIndexEntry{AzureSMR tutorial} + %\VignetteIndexEntry{AzureSMR authentication} %\VignetteEngine{knitr::rmarkdown} %\usepackage[utf8]{inputenc} --- ## Configuration instructions -Follow these instructions to create an active directory. You will need to collect the tenant ID (`tenantID`), client ID (`clientID`) and authentication key (`authKey`) in order to authenticate with the `createAzureContect()` function. +To use the `AzureSMR` package, you must create an Azure Active Directory application with permisssions. This vignette contains instructions to do this. -### Create an Active Directory App. +You must collect three pieces of information to authenticate with the `createAzureContect()` function: -1. Login to the Classic (i.e., the old) Portal https://manage.windowsazure.com/. +* tenant ID (`tenantID`) +* client ID (`clientID`) +* authentication key (`authKey`) -2. On the left hand menu you should see amongst all the items one called ACTIVE DIRECTORY. Click the item and an active directory DIRECTORY will be listed. +## Create an Active Directory application. + +1. Login to the Classic (the old) Portal https://manage.windowsazure.com/. + +2. On the left hand menu you should see amongst all the items one called `ACTIVE DIRECTORY`. Click the item and an active directory DIRECTORY will be listed. 3. Click on an entry under the Name column (if there is only one entry the choice is easy!) to take you to a page of options to get started with some text like I WANT TO. -4. Along the top menu click APPLICATIONS. +4. Along the top menu click `APPLICATIONS`. -5. You probably want to create a new application so type a name for it in the Search box (I called mine AzureSMR). The search result will come back with no results and a button that says `ADD AN APPLICATION` -> which you should click. +5. You probably want to create a new application so type a name for it in the Search box (I called mine `AzureSMR`). The search result will come back with no results and a button that says `ADD AN APPLICATION` -> which you should click. 6. Give the application a name and choose `WEB APPLICATION AND/OR WEB API`. Then go to the next page `->`. 7. Provide some dummy URLs. They are not used but they must be valid URLs. Click on the tick to continue to create the application. -8. Under the Configure menu button take note of the client ID.9. Under the Keys section choose a 1 year duration (or 2) and click the Save button azToken the bottom of the screen. A key is generated which you should copy now and save it somewhere. +8. Under the Configure menu button take note of the **client ID**. + +9. Under the `Keys` section choose a 1 year duration (or 2) and click the Save button at the bottom of the screen. An **authenticatio key** is generated which you should copy now and save it somewhere. + +10. You also need the **tenant ID**. Click the `VIEW ENDPOINTS` button on the bottom of the screen and find a list of endpoints all including the tenant ID as a sequence of hexadecimals. -10. We also need the *tenant ID*. Click the `VIEW ENDPOINTS` button on the bottom of the screen and find a list of endpoints all including the tenant ID as a sequence of hexadecimals. +11. Now set up the applications permissions. Click the `Add application` button. In the resulting window scroll to `Windows Azure Service Management API` and select it. Then click the Tick icon. -11. Now we need to set up the applications permissions. Click the `Add application` button. In the resulting window scroll to `Windows Azure Service Management API` and select it. Then click the Tick icon. +12. Under the resulting "permissions to other applications" section, for the Windows Azure Service Management API entry, from the `Delegated Permissions` drop down tick the Access Azure Service Management as organization. -12. Under the resulting "permissions to other applications" section, for the Windows Azure Service Management API entry, from the "Delegated Permissions:0" drop down tick the Access Azure Service Management as organization. +13. Click on the Save icon at the bottom of the window again. -13. Click on the Save icon azToken the bottom of the window again. +14. Now assign the application a role and to do so you go to the (new) Azure portal. https://portal.azure.com/ -14. Now we need to assign the application a role and to do so we go to the (new) Azure portal. https://portal.azure.com/ +## Access control -To apply access control azToken Resource Group +You can apply access control at either the resource group level or the subscription level + +### To apply access control at Resource Group 15. Click on Resource Groups menu item on the left in the portal. 16. Identify the resource group you will associate with this application. -17. Choose the Access Control(IAM) menu item from the Resource scope. +17. Choose the `Access Control (IAM)` menu item from the Resource scope. 18. In the resulting scope click the `+ Add` button. -19. Choose the role as Owner and under the user search box enter the name of the App, e.g., AzureSMR. +19. Choose the role as Owner and under the user search box enter the name of the App, e.g. `AzureSMR`. -20. Select the resulting list item for that App then click Select in that scope then OK in the "Add access" scope. The user will be added to the list. +20. Select the resulting list item for that App then click Select in that scope then OK in the `Add access` scope. The user will be added to the list. -Alternatively you can access control azToken Subscription Level +### Alternatively you can access control at Subscription Level 15. Click on Subscriptions on the left menu item in the portal. 16. Identify the Subscription you will associate with this application. -17. Choose the Access Control(IAM) menu item. +17. Choose the `Access Control (IAM)` menu item. -18. In the resulting scope click the + Add button. +18. In the resulting scope click the `+ Add` button. -19. Choose the role as Owner and under the user search box enter the name of the App, e.g., AzureSMR. +19. Choose the role as Owner and under the user search box enter the name of the App, e.g. `AzureSMR`. 20. Select the resulting list item for that App then click Select in that scope then OK in the "Add access" scope. The user will be added to the list. +## Conclusion That is all. You can test this by trying: diff --git a/vignettes/tutorial.Rmd b/vignettes/tutorial.Rmd index a6ba176..0a54a27 100644 --- a/vignettes/tutorial.Rmd +++ b/vignettes/tutorial.Rmd @@ -9,7 +9,7 @@ vignette: > %\usepackage[utf8]{inputenc} --- -Use this package to manage Azure Resources from within an R Session. This is not a full SDK just a collection of functions that should prove useful for a Data Scientist who needs to access and manage Azure Resources. +Use this package to manage Azure Resources from within an R Session. This package does not expose the complete Azure API, but is meant as a collection of functions that a typical data scientist may use to access and manage Azure Resources. ## Installation instructions @@ -25,7 +25,7 @@ library(AzureSMR) ## Overview -AzureSMR provides an interface to manage resources on Microsoft Azure. The main functions address the following Azure Services: +`AzureSMR` provides an interface to manage resources on Microsoft Azure. The main functions address the following Azure Services: - Azure Blob: List, Read and Write to Blob Services - Azure Resources: List, Create and Delete Azure Resource @@ -35,35 +35,42 @@ AzureSMR provides an interface to manage resources on Microsoft Azure. The main - Azure Spark: List and create Spark jobs/Sessions against a HDInsight Cluster(Livy) - EXPERIMENTAL -For a detailed list of AzureSM functions and their syntax please refer to the Help pages. +For a detailed list of `AzureSMR` functions and their syntax please refer to the Help pages. -## Getting Authorisation configured +## Configuring authorisation in Azure Active Directory -To get started, please refer to the Authorisation tutorial. https://github.com/Microsoft/AzureSMR/blob/master/vignettes/Authentication.Rmd +To get started, please refer to the [authorisation tutorial](http://htmlpreview.github.io/?https://github.com/Microsoft/AzureSMR/blob/master/inst/doc/Authentication.html) -## Authenticating against the service +## Load the package -The AzureAPIs require lots of parameters to be managed. Rather than supplying all the paramters for each function call AzureSMR implements an AzureContext Variable which caches the last time a paramters is used so that it doesnt need to be repeatedly supplied. +```{r, eval=FALSE} +library(AzureSMR) +``` + +## Authenticating against the Azure service -To create an AzureContext object and attempt to authenticate against the Azure service, use: +The Azure APIs require many parameters to be managed. Rather than supplying all the arguments to every function call, `AzureSMR` uses an `azureActiveContext` object that caches arguments so you don't have to supply . + +To create an `azureActiveContext` object and attempt to authenticate against the Azure service, use: ```{r, eval=FALSE} sc <- createAzureContext(tenantID = "{TID}", clientID = "{CID}", authKey= "{KEY}") sc ``` -If you provide autentication paramters to createAzureContext() the function will automatically authenticate. -To manually get an authorisation token use `azureAuthenticate()`. -Note this token will time our after a period and therefore you need to run it again occasionally. TIP: Use AzureAuthenticate before a long running task. +If you provide autentication paramters to `createAzureContext()` the function will automatically authenticate. To manually get an authorisation token use `azureAuthenticate()`. Note this token will time out after a period and therefore you need to run it again occasionally. TIP: Use `azureAuthenticate()` before a long running task. -The `azureListSubscriptions()` function lists all the available subscriptions. If you only have one it sets the default Subscription in the `azureActiveContext` to that subscription ID. +## Subscriptions + +The `azureListSubscriptions()` function lists all the available subscriptions. If you only have one it sets the default Subscription in the `azureActiveContext` to that subscription ID. ```{r, eval=FALSE} azureListSubscriptions(sc) - ``` + + ## Manage resource Groups ```{r, eval=FALSE} @@ -77,24 +84,22 @@ azureListAllResources(sc, location = "northeurope") azureListAllResources(sc, type = "Microsoft.Sql/servers", location = "northeurope") - azureCreateResourceGroup(sc, resourceGroup = "testme", location = "northeurope") azureCreateStorageAccount(sc,storageAccount="testmystorage1",resourceGroup = "testme") azureListAllResources(sc, resourceGroup = "testme") -# When finished, to delete a Resource Group use azureDeleteResourceGroup +# When finished, to delete a Resource Group use azureDeleteResourceGroup() azureDeleteResourceGroup(sc, resourceGroup = "testme") - ``` ## Manage Virtual Machines -Use these functions to list, start and stop existing Virtual Machines (see templates for Creation). +Use these functions to list, start and stop existing Virtual Machines (see templates for creation). -To Create VMs please refer to Resource Templates below. +To create VMs please refer to Resource Templates below. ```{r, eval=FALSE} ## List VMs in a ResourceGroup @@ -107,19 +112,19 @@ azureStartVM(sc, vmName = "DSVM1") azureStopVM(sc, vmName = "DSVM1") ``` -## Access Storage Blobs +## Accessing storage blobs using the `azureActiveContext` -In order to access Storage Blobs you need to have a key. Use `azureSAGetKey()` to get a Key or alternatively supply your own key. When you provide your own key you no longer need to use `azureAuthenticate()` since the API uses a diferent authentication approach. +To access storage blobs you need to have a key. Use `azureSAGetKey()` to automatically retrieve your key or alternatively supply your own key. ```{r, eval=FALSE} -sKey <- azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") +azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") ``` To create containers in a storage account use`azureCreateStorageContainer()` ```{r, eval=FALSE} -azureCreateStorageContainer(sc,"opendata",storageAccount = "testmystorage1", resourceGroup = "testme") +azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme") ``` @@ -129,7 +134,7 @@ To list containers in a storage account use `azureListContainers()` azureListStorageContainers(sc, storageAccount = "testmystorage1", resourceGroup = "testme") ``` -To Write a Blobs use `azurePutBlob()` +To write a blob use `azurePutBlob()` ```{r, eval=FALSE} azurePutBlob(sc, storageAccount = "testmystorage1", container = "opendata", @@ -151,11 +156,25 @@ azureGetBlob(sc, storageAccount = "testmystorage1", container = "opendata", type="text") ``` + +## Accessing storage blobs without an `azureActiveContext` + +It is also possible to access the storage API without having an Azure Active Directory application. + +In this case, you should use the argument `azureActiveContect = NULL` to the storage functions. + +For example: + +```{r, eval=FALSE} +azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "opendata") +``` + + ## Manage HDInsight Clusters -You can use `AzureSMR` to manage Azure HDInsight clusters. To create clusters use azureCreateHDI or for advanced configurations use Resource Templates (See below). +You can use `AzureSMR` to manage HDInsight clusters. To create a cluster use `azureCreateHDI()`. -Also see functions for submitting Hive and Spark jobs. +For advanced configurations use Resource Templates (See below). ```{r, eval=FALSE} azureCreateHDI(sc, @@ -169,11 +188,10 @@ azureCreateHDI(sc, kind = "rserver") ``` -Use `azureListHDI()` to list available Clusters. +Use `azureListHDI()` to list available clusters. ```{r, eval=FALSE} azureListHDI(sc, resourceGroup ="testme") - ``` Use `azureResizeHDI()` to resize a cluster @@ -191,11 +209,12 @@ azureResizeHDI(sc, resourceGroup = "testme", clustername = "smrhdi", role="worke ## " headnode ( 2 * Standard_D3_v2 ) workernode ( 5 * Standard_D3_v2 ) zookeepernode ( 3 * Medium ) edgenode0 ( 1 * Standard_D4_v2 )" ``` + ## Resource Templates - Create Azure Resources -The easiest way to create resources on Azure is to use Azure Templates. To create Azure Resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into Settings > Automation scripts. Example templates can be found azToken this URL https://github.com/Azure/AzureStack-QuickStart-Templates. +The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates. -To create a resource using a template in AzureSM use AzureDeployTemplate. The Template and Paramters must be available in a public URL (Azure Blob). It may be worth getting the Azure Administrator to build a working template. +To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example Azure blob). ```{r, eval=FALSE} azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", @@ -209,15 +228,18 @@ azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", ## Finished: 2016-06-23 19:04:43 ``` -ADMIN TIP: If a deployment fails. Go to the Azure Portal and look azToken Actvity logs and look for failed deployments which should explain why the deployment failed. +ADMIN TIP: If a deployment fails, go to the Azure Portal and look at `Activity logs` and look for failed deployments - this should explain why the deployment failed. + ## Hive Functions -These functions facilitate the use of hive jobs on a HDInsight Cluster + +These functions facilitate the use of hive jobs on an HDInsight Cluster ```{r, eval=FALSE} azureHiveStatus(sc, clusterName = "smrhdi", hdiAdmin = "hdiadmin", hdiPassword = "AzureSMR_password123") + azureHiveSQL(sc, CMD = "select * from hivesampletable", path = "wasb://opendata@testmystorage1.blob.core.windows.net/") @@ -226,16 +248,15 @@ azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "openda stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout") -read.delim(text=stdout, header=TRUE, fill=TRUE) - +read.delim(text = stdout, header = TRUE, fill = TRUE) ``` ## Spark functions (experimental) -`AzureSMR` provides some functions that allow HDInsight Spark Sessions and jobs to be managed within an R Session +`AzureSMR` provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session. -To Create a new Spark Session (Via Livy) use `azureSparkNewSession()` +To create a new Spark session (via Livy) use `azureSparkNewSession()` ```{r, eval=FALSE} azureSparkNewSession(sc, clustername = "smrhdi", @@ -244,8 +265,7 @@ azureSparkNewSession(sc, clustername = "smrhdi", kind = "pyspark") ``` -To view the status of sessions use AzureSparkListSessions -Wait for status to be Idle +To view the status of sessions use `azureSparkListSessions()`. Wait for status to be idle. ```{r, eval=FALSE} azureSparkListSessions(sc, clustername = "smrhdi") @@ -293,6 +313,5 @@ azureSparkNewSession(sc, clustername = "smrhdi", kind = "sparkr") azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2") azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2") - ``` From 38eace04632d98417fbd6c7692d5a10afba346b2 Mon Sep 17 00:00:00 2001 From: Andrie de Vries Date: Fri, 16 Jun 2017 13:51:24 +0100 Subject: [PATCH 2/7] Document how to use blob store functions without authentication #76 --- R/AzureBlob.R | 22 +++++++++++++++++++--- R/AzureContextObject.R | 8 ++++---- R/AzureCost.R | 11 ++++------- man-roxygen/blob_no_authentication.R | 2 ++ man/azureAuthenticate.Rd | 8 ++++---- man/azureBatchGetKey.Rd | 2 +- man/azureBlobCD.Rd | 7 ++++++- man/azureBlobFind.Rd | 7 ++++++- man/azureBlobLS.Rd | 7 ++++++- man/azureCancelDeploy.Rd | 2 +- man/azureCheckToken.Rd | 2 +- man/azureCreateBatchAccount.Rd | 2 +- man/azureCreateHDI.Rd | 2 +- man/azureCreateResourceGroup.Rd | 2 +- man/azureCreateStorageAccount.Rd | 2 +- man/azureCreateStorageContainer.Rd | 2 +- man/azureDataConsumption.Rd | 6 ++---- man/azureDeleteBatchAccount.Rd | 2 +- man/azureDeleteBlob.Rd | 7 ++++++- man/azureDeleteDeploy.Rd | 2 +- man/azureDeleteHDI.Rd | 2 +- man/azureDeleteResourceGroup.Rd | 2 +- man/azureDeleteStorageContainer.Rd | 2 +- man/azureDeleteVM.Rd | 2 +- man/azureDeletestorageAccount.Rd | 2 +- man/azureDeployStatus.Rd | 2 +- man/azureDeployTemplate.Rd | 2 +- man/azureExpenseCalculator.Rd | 9 +++------ man/azureGetAllVMstatus.Rd | 2 +- man/azureGetBlob.Rd | 7 ++++++- man/azureHDIConf.Rd | 2 +- man/azureHiveSQL.Rd | 2 +- man/azureHiveStatus.Rd | 2 +- man/azureListBatchAccounts.Rd | 2 +- man/azureListHDI.Rd | 2 +- man/azureListRG.Rd | 2 +- man/azureListSA.Rd | 2 +- man/azureListScaleSetNetwork.Rd | 2 +- man/azureListScaleSetVM.Rd | 2 +- man/azureListScaleSets.Rd | 2 +- man/azureListStorageBlobs.Rd | 10 +++++++++- man/azureListStorageContainers.Rd | 2 +- man/azureListSubscriptions.Rd | 2 +- man/azureListVM.Rd | 2 +- man/azurePricingRates.Rd | 5 +---- man/azurePutBlob.Rd | 7 ++++++- man/azureResizeHDI.Rd | 8 ++++---- man/azureRunScriptAction.Rd | 2 +- man/azureSAGetKey.Rd | 2 +- man/azureScriptActionHistory.Rd | 2 +- man/azureSparkCMD.Rd | 2 +- man/azureSparkJob.Rd | 2 +- man/azureSparkListJobs.Rd | 2 +- man/azureSparkListSessions.Rd | 2 +- man/azureSparkNewSession.Rd | 2 +- man/azureSparkShowURL.Rd | 2 +- man/azureSparkStopSession.Rd | 2 +- man/azureStartVM.Rd | 2 +- man/azureStopVM.Rd | 2 +- man/azureVMStatus.Rd | 2 +- man/createAzureContext.Rd | 6 +++--- man/pkg-deprecated.Rd | 2 +- man/setAzureContext.Rd | 8 ++++---- 63 files changed, 140 insertions(+), 95 deletions(-) create mode 100644 man-roxygen/blob_no_authentication.R diff --git a/R/AzureBlob.R b/R/AzureBlob.R index 4fd0bdf..adee442 100644 --- a/R/AzureBlob.R +++ b/R/AzureBlob.R @@ -4,6 +4,7 @@ #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureSAGetKey +#' @param azureActiveContext Either an `azureActiveContext` object or `NULL`. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey`. #' #' @param maxresults Optional. Specifies the maximum number of blobs to return, including all BlobPrefix elements. If the request does not specify maxresults or specifies a value greater than 5,000, the server will return up to 5,000 items. Setting `maxresults` to a value less than or equal to zero results in error response code 400 (Bad Request). #' @param prefix Optional. Filters the results to return only blobs whose names begin with the specified prefix. @@ -11,6 +12,9 @@ #' @param marker Optional. A string value that identifies the portion of the list to be returned with the next list operation. The operation returns a marker value within the response body if the list returned was not complete. The marker value may then be used in a subsequent call to request the next set of list items. The marker value is opaque to the client. #' #' @return Returns a data frame. This data frame has an attribute called `marker` that can be used with the `marker` argument to return the next set of values. +#' +#' @template blob_no_authentication +#' @references https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs #' @family Blob store functions #' @export azureListStorageBlobs <- function(azureActiveContext, storageAccount, storageKey, @@ -92,13 +96,15 @@ azureListStorageBlobs <- function(azureActiveContext, storageAccount, storageKey #' List blob blobs in a storage account directory. #' +#' @inheritParams azureListStorageBlobs #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureSAGetKey - +#' #' @param directory Blob store directory to list for content #' @param recursive If TRUE, list blob store directories recursively #' +#' @template blob_no_authentication #' @family Blob store functions #' @export azureBlobLS <- function(azureActiveContext, directory, recursive = FALSE, @@ -181,13 +187,15 @@ azureBlobLS <- function(azureActiveContext, directory, recursive = FALSE, #' Get contents from a specifed storage blob. #' +#' @inheritParams azureListStorageBlobs #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureSAGetKey #' @inheritParams azureBlobLS - +#' #' @param type String, either "text" or "raw". Passed to [httr::content()] #' +#' @template blob_no_authentication #' @family Blob store functions #' @export @@ -257,6 +265,7 @@ azureGetBlob <- function(azureActiveContext, blob, directory, type = "text", #' Write contents to a specifed storage blob. #' +#' @inheritParams azureListStorageBlobs #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureSAGetKey @@ -265,6 +274,7 @@ azureGetBlob <- function(azureActiveContext, blob, directory, type = "text", #' @param contents - Object or value to store #' @param file - Local filename to store in Azure blob #' +#' @template blob_no_authentication #' @family Blob store functions #' @export azurePutBlob <- function(azureActiveContext, blob, contents = "", file = "", @@ -337,11 +347,13 @@ azurePutBlob <- function(azureActiveContext, blob, contents = "", file = "", #' Find file in a storage account directory. #' +#' @inheritParams azureListStorageBlobs #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureSAGetKey #' @inheritParams azurePutBlob #' +#' @template blob_no_authentication #' @family Blob store functions #' @export azureBlobFind <- function(azureActiveContext, file, storageAccount, storageKey, @@ -383,12 +395,14 @@ azureBlobFind <- function(azureActiveContext, file, storageAccount, storageKey, #' Azure blob change current directory. #' +#' @inheritParams azureListStorageBlobs #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureSAGetKey #' @inheritParams azureBlobLS #' @inheritParams azurePutBlob #' +#' @template blob_no_authentication #' @family Blob store functions #' @export azureBlobCD <- function(azureActiveContext, directory, container, file, @@ -430,7 +444,7 @@ azureBlobCD <- function(azureActiveContext, directory, container, file, assert_that(is_storage_key(storageKey)) if (directory == "../" || directory == "..") { - # Basic attempt azToken relative paths + # Basic attempt at relative paths directory <- gsub("/[a-zA-Z0-9]*$", "", azureActiveContext$directory) } @@ -458,11 +472,13 @@ azureBlobCD <- function(azureActiveContext, directory, container, file, #' Delete a specifed storage blob. #' +#' @inheritParams azureListStorageBlobs #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureSAGetKey #' @inheritParams azureBlobLS #' +#' @template blob_no_authentication #' @family Blob store functions #' @export diff --git a/R/AzureContextObject.R b/R/AzureContextObject.R index 7ebb732..94537eb 100644 --- a/R/AzureContextObject.R +++ b/R/AzureContextObject.R @@ -38,10 +38,10 @@ createAzureContext <- function(tenantID, clientID, authKey, configFile){ #' #' Updates the value of an `azureActiveContext` object, created by [createAzureContext()] #' -#' @param azureActiveContext A container used for caching variables used by `AzureSMR` -#' @param tenantID The Tenant ID provided during creation of the Active Directory application / service principal -#' @param clientID The Client ID provided during creation of the Active Directory application / service principal -#' @param authKey The Authentication Key provided during creation of the Active Directory application / service principal +#' @param azureActiveContext A container used for caching variables used by `AzureSMR`, created by [createAzureContext()] +#' @param tenantID The tenant ID provided during creation of the Active Directory application / service principal +#' @param clientID The client ID provided during creation of the Active Directory application / service principal +#' @param authKey The authentication key provided during creation of the Active Directory application / service principal #' @param subscriptionID Subscription ID. This is obtained automatically by [azureAuthenticate()] when only a single subscriptionID is available via Active Directory #' @param resourceGroup Name of the resource group #' @param vmName Name of the virtual machine diff --git a/R/AzureCost.R b/R/AzureCost.R index 33e4d0e..68a4f47 100644 --- a/R/AzureCost.R +++ b/R/AzureCost.R @@ -10,9 +10,7 @@ #' #' @inheritParams setAzureContext #' -#' @param instance Instance name that one would like to check expe -#' nse. It is by default empty, which returns data consumption for -#' all instances under subscription. +#' @param instance Instance name that one would like to check expense. It is by default empty, which returns data consumption for all instances under subscription. #' #' @param timeStart Start time. #' @param timeEnd End time. @@ -228,6 +226,8 @@ azureDataConsumption <- function(azureActiveContext, #' Get pricing details of resources under a subscription. #' +#' The pricing rates function wraps API calls to Azure RateCard and currently the API supports only the Pay-As-You-Go offer scheme. +#' #' @inheritParams setAzureContext #' #' @param currency Currency in which price rating is measured. @@ -236,8 +236,6 @@ azureDataConsumption <- function(azureActiveContext, #' #' @param region region information about the subscription. #' -#' @note The pricing rates function wraps API calls to Azure RateCard and currently the API supports only the Pay-As-You-Go offer scheme. -#' #' @family Cost functions #' @export azurePricingRates <- function(azureActiveContext, @@ -313,8 +311,7 @@ azurePricingRates <- function(azureActiveContext, #' @inheritParams azureDataConsumption #' @inheritParams azurePricingRates #' -#' @return Total cost measured in the given currency of the specified Azure -#' instance in the period. +#' @return Total cost measured in the given currency of the specified Azure instance in the period. #' #' @family Cost functions #' @export diff --git a/man-roxygen/blob_no_authentication.R b/man-roxygen/blob_no_authentication.R new file mode 100644 index 0000000..6e2e10e --- /dev/null +++ b/man-roxygen/blob_no_authentication.R @@ -0,0 +1,2 @@ +#' @section Using blob store functions without authentication: +#' The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . diff --git a/man/azureAuthenticate.Rd b/man/azureAuthenticate.Rd index 97fc9ce..c56fafc 100644 --- a/man/azureAuthenticate.Rd +++ b/man/azureAuthenticate.Rd @@ -8,13 +8,13 @@ azureAuthenticate(azureActiveContext, tenantID, clientID, authKey, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} -\item{tenantID}{The Tenant ID provided during creation of the Active Directory application / service principal} +\item{tenantID}{The tenant ID provided during creation of the Active Directory application / service principal} -\item{clientID}{The Client ID provided during creation of the Active Directory application / service principal} +\item{clientID}{The client ID provided during creation of the Active Directory application / service principal} -\item{authKey}{The Authentication Key provided during creation of the Active Directory application / service principal} +\item{authKey}{The authentication key provided during creation of the Active Directory application / service principal} \item{verbose}{Print Tracing information (Default False)} } diff --git a/man/azureBatchGetKey.Rd b/man/azureBatchGetKey.Rd index e8fc7d5..f747860 100644 --- a/man/azureBatchGetKey.Rd +++ b/man/azureBatchGetKey.Rd @@ -8,7 +8,7 @@ azureBatchGetKey(azureActiveContext, batchAccount, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureBlobCD.Rd b/man/azureBlobCD.Rd index a049310..80e9dac 100644 --- a/man/azureBlobCD.Rd +++ b/man/azureBlobCD.Rd @@ -8,7 +8,7 @@ azureBlobCD(azureActiveContext, directory, container, file, storageAccount, storageKey, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{Either an \code{azureActiveContext} object or \code{NULL}. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is \code{NULL}, you must provide the \code{storageKey}.} \item{directory}{Blob store directory to list for content} @@ -29,6 +29,11 @@ azureBlobCD(azureActiveContext, directory, container, file, storageAccount, \description{ Azure blob change current directory. } +\section{Using blob store functions without authentication}{ + +The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . +} + \seealso{ Other Blob store functions: \code{\link{azureBlobFind}}, \code{\link{azureBlobLS}}, \code{\link{azureDeleteBlob}}, diff --git a/man/azureBlobFind.Rd b/man/azureBlobFind.Rd index 58a6766..754cf10 100644 --- a/man/azureBlobFind.Rd +++ b/man/azureBlobFind.Rd @@ -8,7 +8,7 @@ azureBlobFind(azureActiveContext, file, storageAccount, storageKey, container, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{Either an \code{azureActiveContext} object or \code{NULL}. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is \code{NULL}, you must provide the \code{storageKey}.} \item{file}{\itemize{ \item Local filename to store in Azure blob @@ -27,6 +27,11 @@ azureBlobFind(azureActiveContext, file, storageAccount, storageKey, container, \description{ Find file in a storage account directory. } +\section{Using blob store functions without authentication}{ + +The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . +} + \seealso{ Other Blob store functions: \code{\link{azureBlobCD}}, \code{\link{azureBlobLS}}, \code{\link{azureDeleteBlob}}, diff --git a/man/azureBlobLS.Rd b/man/azureBlobLS.Rd index 1f63bf3..debda91 100644 --- a/man/azureBlobLS.Rd +++ b/man/azureBlobLS.Rd @@ -8,7 +8,7 @@ azureBlobLS(azureActiveContext, directory, recursive = FALSE, storageAccount, storageKey, container, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{Either an \code{azureActiveContext} object or \code{NULL}. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is \code{NULL}, you must provide the \code{storageKey}.} \item{directory}{Blob store directory to list for content} @@ -27,6 +27,11 @@ azureBlobLS(azureActiveContext, directory, recursive = FALSE, storageAccount, \description{ List blob blobs in a storage account directory. } +\section{Using blob store functions without authentication}{ + +The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . +} + \seealso{ Other Blob store functions: \code{\link{azureBlobCD}}, \code{\link{azureBlobFind}}, diff --git a/man/azureCancelDeploy.Rd b/man/azureCancelDeploy.Rd index f2010e4..885ca51 100644 --- a/man/azureCancelDeploy.Rd +++ b/man/azureCancelDeploy.Rd @@ -8,7 +8,7 @@ azureCancelDeploy(azureActiveContext, deplname, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{deplname}{Deployment name} diff --git a/man/azureCheckToken.Rd b/man/azureCheckToken.Rd index fc7c308..d5f8ffe 100644 --- a/man/azureCheckToken.Rd +++ b/man/azureCheckToken.Rd @@ -7,7 +7,7 @@ azureCheckToken(azureActiveContext) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} } \description{ Check the timestamp of a token and renew if needed. diff --git a/man/azureCreateBatchAccount.Rd b/man/azureCreateBatchAccount.Rd index c027f3d..701d2e6 100644 --- a/man/azureCreateBatchAccount.Rd +++ b/man/azureCreateBatchAccount.Rd @@ -9,7 +9,7 @@ azureCreateBatchAccount(azureActiveContext, batchAccount, asynchronous = FALSE, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{location}{A string for the location to create batch account} diff --git a/man/azureCreateHDI.Rd b/man/azureCreateHDI.Rd index dde1fd1..dff204a 100644 --- a/man/azureCreateHDI.Rd +++ b/man/azureCreateHDI.Rd @@ -12,7 +12,7 @@ azureCreateHDI(azureActiveContext, resourceGroup, location, clustername, verbose = FALSE, debug = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureCreateResourceGroup.Rd b/man/azureCreateResourceGroup.Rd index ae729b8..2576703 100644 --- a/man/azureCreateResourceGroup.Rd +++ b/man/azureCreateResourceGroup.Rd @@ -8,7 +8,7 @@ azureCreateResourceGroup(azureActiveContext, resourceGroup, location, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureCreateStorageAccount.Rd b/man/azureCreateStorageAccount.Rd index 7a59191..d85694d 100644 --- a/man/azureCreateStorageAccount.Rd +++ b/man/azureCreateStorageAccount.Rd @@ -9,7 +9,7 @@ azureCreateStorageAccount(azureActiveContext, storageAccount, asynchronous = FALSE, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{storageAccount}{Name of the azure storage account. Storage account names must be between 3 and 24 characters in length and use numbers and lower-case letters only.} diff --git a/man/azureCreateStorageContainer.Rd b/man/azureCreateStorageContainer.Rd index c92cfe7..9819447 100644 --- a/man/azureCreateStorageContainer.Rd +++ b/man/azureCreateStorageContainer.Rd @@ -8,7 +8,7 @@ azureCreateStorageContainer(azureActiveContext, container, storageAccount, storageKey, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{container}{Storage container name. See \code{\link[=azureListStorageContainers]{azureListStorageContainers()}}} diff --git a/man/azureDataConsumption.Rd b/man/azureDataConsumption.Rd index 9a5afb2..cedcfd7 100644 --- a/man/azureDataConsumption.Rd +++ b/man/azureDataConsumption.Rd @@ -9,11 +9,9 @@ azureDataConsumption(azureActiveContext, instance = "", timeStart, timeEnd, granularity = "Hourly", verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} -\item{instance}{Instance name that one would like to check expe -nse. It is by default empty, which returns data consumption for -all instances under subscription.} +\item{instance}{Instance name that one would like to check expense. It is by default empty, which returns data consumption for all instances under subscription.} \item{timeStart}{Start time.} diff --git a/man/azureDeleteBatchAccount.Rd b/man/azureDeleteBatchAccount.Rd index 5ebcb17..3bd3ce5 100644 --- a/man/azureDeleteBatchAccount.Rd +++ b/man/azureDeleteBatchAccount.Rd @@ -8,7 +8,7 @@ azureDeleteBatchAccount(azureActiveContext, batchAccount, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureDeleteBlob.Rd b/man/azureDeleteBlob.Rd index dde2dde..6801ab0 100644 --- a/man/azureDeleteBlob.Rd +++ b/man/azureDeleteBlob.Rd @@ -8,7 +8,7 @@ azureDeleteBlob(azureActiveContext, blob, directory, storageAccount, storageKey, container, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{Either an \code{azureActiveContext} object or \code{NULL}. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is \code{NULL}, you must provide the \code{storageKey}.} \item{blob}{Blob name} @@ -27,6 +27,11 @@ azureDeleteBlob(azureActiveContext, blob, directory, storageAccount, storageKey, \description{ Delete a specifed storage blob. } +\section{Using blob store functions without authentication}{ + +The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . +} + \seealso{ Other Blob store functions: \code{\link{azureBlobCD}}, \code{\link{azureBlobFind}}, \code{\link{azureBlobLS}}, diff --git a/man/azureDeleteDeploy.Rd b/man/azureDeleteDeploy.Rd index 1819283..9517d72 100644 --- a/man/azureDeleteDeploy.Rd +++ b/man/azureDeleteDeploy.Rd @@ -8,7 +8,7 @@ azureDeleteDeploy(azureActiveContext, deplname, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{deplname}{Deployment name} diff --git a/man/azureDeleteHDI.Rd b/man/azureDeleteHDI.Rd index 0130994..b7f8cdf 100644 --- a/man/azureDeleteHDI.Rd +++ b/man/azureDeleteHDI.Rd @@ -8,7 +8,7 @@ azureDeleteHDI(azureActiveContext, clustername, subscriptionID, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureDeleteResourceGroup.Rd b/man/azureDeleteResourceGroup.Rd index f08b27c..12973c3 100644 --- a/man/azureDeleteResourceGroup.Rd +++ b/man/azureDeleteResourceGroup.Rd @@ -8,7 +8,7 @@ azureDeleteResourceGroup(azureActiveContext, resourceGroup, subscriptionID, type, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureDeleteStorageContainer.Rd b/man/azureDeleteStorageContainer.Rd index b546b70..ba500eb 100644 --- a/man/azureDeleteStorageContainer.Rd +++ b/man/azureDeleteStorageContainer.Rd @@ -8,7 +8,7 @@ azureDeleteStorageContainer(azureActiveContext, container, storageAccount, storageKey, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{container}{Storage container name. See \code{\link[=azureListStorageContainers]{azureListStorageContainers()}}} diff --git a/man/azureDeleteVM.Rd b/man/azureDeleteVM.Rd index 58985a8..9158d69 100644 --- a/man/azureDeleteVM.Rd +++ b/man/azureDeleteVM.Rd @@ -8,7 +8,7 @@ azureDeleteVM(azureActiveContext, resourceGroup, vmName, subscriptionID, mode = "Sync", verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureDeletestorageAccount.Rd b/man/azureDeletestorageAccount.Rd index 6fd43ee..fce8d68 100644 --- a/man/azureDeletestorageAccount.Rd +++ b/man/azureDeletestorageAccount.Rd @@ -8,7 +8,7 @@ azureDeletestorageAccount(azureActiveContext, storageAccount, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{storageAccount}{Name of the azure storage account. Storage account names must be between 3 and 24 characters in length and use numbers and lower-case letters only.} diff --git a/man/azureDeployStatus.Rd b/man/azureDeployStatus.Rd index 0bcba39..d309734 100644 --- a/man/azureDeployStatus.Rd +++ b/man/azureDeployStatus.Rd @@ -8,7 +8,7 @@ azureDeployStatus(azureActiveContext, deplname, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{deplname}{Deployment name} diff --git a/man/azureDeployTemplate.Rd b/man/azureDeployTemplate.Rd index 9b4a5ee..e4ff4dc 100644 --- a/man/azureDeployTemplate.Rd +++ b/man/azureDeployTemplate.Rd @@ -9,7 +9,7 @@ azureDeployTemplate(azureActiveContext, deplname, templateURL, paramURL, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{deplname}{Deployment name} diff --git a/man/azureExpenseCalculator.Rd b/man/azureExpenseCalculator.Rd index 432e1e4..f43e82a 100644 --- a/man/azureExpenseCalculator.Rd +++ b/man/azureExpenseCalculator.Rd @@ -8,11 +8,9 @@ azureExpenseCalculator(azureActiveContext, instance = "", timeStart, timeEnd, granularity, currency, locale, offerId, region, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} -\item{instance}{Instance name that one would like to check expe -nse. It is by default empty, which returns data consumption for -all instances under subscription.} +\item{instance}{Instance name that one would like to check expense. It is by default empty, which returns data consumption for all instances under subscription.} \item{timeStart}{Start time.} @@ -29,8 +27,7 @@ all instances under subscription.} \item{region}{region information about the subscription.} } \value{ -Total cost measured in the given currency of the specified Azure -instance in the period. +Total cost measured in the given currency of the specified Azure instance in the period. } \description{ Note if difference between \code{timeStart} and \code{timeEnd} is less than the finest granularity, e.g., "Hourly" (we notice this is a usual case when one needs to be aware of the charges of a job that takes less than an hour), the expense will be estimated based solely on computation hour. That is, the total expense is the multiplication of computation hour and pricing rate of the requested instance. diff --git a/man/azureGetAllVMstatus.Rd b/man/azureGetAllVMstatus.Rd index 6527fb6..b0069d9 100644 --- a/man/azureGetAllVMstatus.Rd +++ b/man/azureGetAllVMstatus.Rd @@ -7,7 +7,7 @@ azureGetAllVMstatus(azureActiveContext) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} } \description{ First queries the azure active context for all visible resources, then sequentially queries the status of all virtuam machines. diff --git a/man/azureGetBlob.Rd b/man/azureGetBlob.Rd index e1bf690..2f6a416 100644 --- a/man/azureGetBlob.Rd +++ b/man/azureGetBlob.Rd @@ -8,7 +8,7 @@ azureGetBlob(azureActiveContext, blob, directory, type = "text", storageAccount, storageKey, container, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{Either an \code{azureActiveContext} object or \code{NULL}. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is \code{NULL}, you must provide the \code{storageKey}.} \item{blob}{Blob name} @@ -29,6 +29,11 @@ azureGetBlob(azureActiveContext, blob, directory, type = "text", \description{ Get contents from a specifed storage blob. } +\section{Using blob store functions without authentication}{ + +The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . +} + \seealso{ Other Blob store functions: \code{\link{azureBlobCD}}, \code{\link{azureBlobFind}}, \code{\link{azureBlobLS}}, diff --git a/man/azureHDIConf.Rd b/man/azureHDIConf.Rd index e20a2d2..3eeffa7 100644 --- a/man/azureHDIConf.Rd +++ b/man/azureHDIConf.Rd @@ -8,7 +8,7 @@ azureHDIConf(azureActiveContext, clustername, resourceGroup, subscriptionID, name, type, location, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureHiveSQL.Rd b/man/azureHiveSQL.Rd index 37760cd..59b0bf8 100644 --- a/man/azureHiveSQL.Rd +++ b/man/azureHiveSQL.Rd @@ -8,7 +8,7 @@ azureHiveSQL(azureActiveContext, CMD, clustername, hdiAdmin, hdiPassword, path = "wasb:///tmp/", verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{CMD}{SQl COmmand String} diff --git a/man/azureHiveStatus.Rd b/man/azureHiveStatus.Rd index 3ef46b6..9592fa3 100644 --- a/man/azureHiveStatus.Rd +++ b/man/azureHiveStatus.Rd @@ -8,7 +8,7 @@ azureHiveStatus(azureActiveContext, clustername, hdiAdmin, hdiPassword, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureListBatchAccounts.Rd b/man/azureListBatchAccounts.Rd index 181f31f..f49598f 100644 --- a/man/azureListBatchAccounts.Rd +++ b/man/azureListBatchAccounts.Rd @@ -8,7 +8,7 @@ azureListBatchAccounts(azureActiveContext, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureListHDI.Rd b/man/azureListHDI.Rd index 5ed9b4b..3e7497f 100644 --- a/man/azureListHDI.Rd +++ b/man/azureListHDI.Rd @@ -8,7 +8,7 @@ azureListHDI(azureActiveContext, resourceGroup, clustername = "*", subscriptionID, name, type, location, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureListRG.Rd b/man/azureListRG.Rd index 30d129a..7d4a366 100644 --- a/man/azureListRG.Rd +++ b/man/azureListRG.Rd @@ -7,7 +7,7 @@ azureListRG(azureActiveContext, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{subscriptionID}{Subscription ID. This is obtained automatically by \code{\link[=azureAuthenticate]{azureAuthenticate()}} when only a single subscriptionID is available via Active Directory} diff --git a/man/azureListSA.Rd b/man/azureListSA.Rd index 356e2e0..117967c 100644 --- a/man/azureListSA.Rd +++ b/man/azureListSA.Rd @@ -8,7 +8,7 @@ azureListSA(azureActiveContext, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureListScaleSetNetwork.Rd b/man/azureListScaleSetNetwork.Rd index 8791fde..9c707a9 100644 --- a/man/azureListScaleSetNetwork.Rd +++ b/man/azureListScaleSetNetwork.Rd @@ -8,7 +8,7 @@ azureListScaleSetNetwork(azureActiveContext, resourceGroup, location, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureListScaleSetVM.Rd b/man/azureListScaleSetVM.Rd index 6f4f757..4c32fae 100644 --- a/man/azureListScaleSetVM.Rd +++ b/man/azureListScaleSetVM.Rd @@ -8,7 +8,7 @@ azureListScaleSetVM(azureActiveContext, scaleSet, resourceGroup, location, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{scaleSet}{Character vector with name of the scaleset (see also \code{\link[=azureListScaleSets]{azureListScaleSets()}})} diff --git a/man/azureListScaleSets.Rd b/man/azureListScaleSets.Rd index 08021e0..8c11ae9 100644 --- a/man/azureListScaleSets.Rd +++ b/man/azureListScaleSets.Rd @@ -8,7 +8,7 @@ azureListScaleSets(azureActiveContext, resourceGroup, location, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureListStorageBlobs.Rd b/man/azureListStorageBlobs.Rd index 7bbb82b..8cfea3c 100644 --- a/man/azureListStorageBlobs.Rd +++ b/man/azureListStorageBlobs.Rd @@ -8,7 +8,7 @@ azureListStorageBlobs(azureActiveContext, storageAccount, storageKey, container, maxresults, prefix, delimiter, marker, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{Either an \code{azureActiveContext} object or \code{NULL}. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is \code{NULL}, you must provide the \code{storageKey}.} \item{storageAccount}{Name of the azure storage account. Storage account names must be between 3 and 24 characters in length and use numbers and lower-case letters only.} @@ -32,6 +32,14 @@ Returns a data frame. This data frame has an attribute called \code{marker} that \description{ List storage blobs for specified storage account. } +\section{Using blob store functions without authentication}{ + +The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . +} + +\references{ +https://docs.microsoft.com/en-us/rest/api/storageservices/list-blobs +} \seealso{ Other Blob store functions: \code{\link{azureBlobCD}}, \code{\link{azureBlobFind}}, \code{\link{azureBlobLS}}, diff --git a/man/azureListStorageContainers.Rd b/man/azureListStorageContainers.Rd index 6d7bfc6..f146748 100644 --- a/man/azureListStorageContainers.Rd +++ b/man/azureListStorageContainers.Rd @@ -8,7 +8,7 @@ azureListStorageContainers(azureActiveContext, storageAccount, storageKey, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{storageAccount}{Name of the azure storage account. Storage account names must be between 3 and 24 characters in length and use numbers and lower-case letters only.} diff --git a/man/azureListSubscriptions.Rd b/man/azureListSubscriptions.Rd index 9a1c587..da120c9 100644 --- a/man/azureListSubscriptions.Rd +++ b/man/azureListSubscriptions.Rd @@ -7,7 +7,7 @@ azureListSubscriptions(azureActiveContext, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{verbose}{Print Tracing information (Default False)} } diff --git a/man/azureListVM.Rd b/man/azureListVM.Rd index 692534a..05ca228 100644 --- a/man/azureListVM.Rd +++ b/man/azureListVM.Rd @@ -8,7 +8,7 @@ azureListVM(azureActiveContext, resourceGroup, location, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azurePricingRates.Rd b/man/azurePricingRates.Rd index 383fcab..59aeaf0 100644 --- a/man/azurePricingRates.Rd +++ b/man/azurePricingRates.Rd @@ -8,7 +8,7 @@ azurePricingRates(azureActiveContext, currency, locale, offerId, region, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{currency}{Currency in which price rating is measured.} @@ -19,9 +19,6 @@ azurePricingRates(azureActiveContext, currency, locale, offerId, region, \item{region}{region information about the subscription.} } \description{ -Get pricing details of resources under a subscription. -} -\note{ The pricing rates function wraps API calls to Azure RateCard and currently the API supports only the Pay-As-You-Go offer scheme. } \seealso{ diff --git a/man/azurePutBlob.Rd b/man/azurePutBlob.Rd index 91d61d3..9b826d1 100644 --- a/man/azurePutBlob.Rd +++ b/man/azurePutBlob.Rd @@ -8,7 +8,7 @@ azurePutBlob(azureActiveContext, blob, contents = "", file = "", directory, storageAccount, storageKey, container, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{Either an \code{azureActiveContext} object or \code{NULL}. The blob store functions support two modes of connecting to the Azure API: authenticate using Active Directory or providing your own storage key. If this value is \code{NULL}, you must provide the \code{storageKey}.} \item{blob}{Blob name} @@ -35,6 +35,11 @@ azurePutBlob(azureActiveContext, blob, contents = "", file = "", directory, \description{ Write contents to a specifed storage blob. } +\section{Using blob store functions without authentication}{ + +The blob store functions support two modes of connecting to the Azure API:authenticate using Active Directory or providing your own storage key. If this value is `NULL`, you must provide the `storageKey` . +} + \seealso{ Other Blob store functions: \code{\link{azureBlobCD}}, \code{\link{azureBlobFind}}, \code{\link{azureBlobLS}}, diff --git a/man/azureResizeHDI.Rd b/man/azureResizeHDI.Rd index bbff30d..bd3da02 100644 --- a/man/azureResizeHDI.Rd +++ b/man/azureResizeHDI.Rd @@ -4,12 +4,12 @@ \alias{azureResizeHDI} \title{Resize a HDInsight cluster role.} \usage{ -azureResizeHDI(azureActiveContext, clustername, role = c("worker", "head", - "edge"), size = 2, mode = c("Sync", "Async"), subscriptionID, - resourceGroup, verbose = FALSE) +azureResizeHDI(azureActiveContext, clustername, role = c("workernode", + "headnode", "edgenode"), size = 2, mode = c("Sync", "Async"), + subscriptionID, resourceGroup, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureRunScriptAction.Rd b/man/azureRunScriptAction.Rd index 8d62722..ccd99c4 100644 --- a/man/azureRunScriptAction.Rd +++ b/man/azureRunScriptAction.Rd @@ -10,7 +10,7 @@ azureRunScriptAction(azureActiveContext, scriptname, scriptURL, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{scriptname}{Identifier for Custom action script operation} diff --git a/man/azureSAGetKey.Rd b/man/azureSAGetKey.Rd index fca79e0..64a06bb 100644 --- a/man/azureSAGetKey.Rd +++ b/man/azureSAGetKey.Rd @@ -8,7 +8,7 @@ azureSAGetKey(azureActiveContext, storageAccount, resourceGroup, subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{storageAccount}{Name of the azure storage account. Storage account names must be between 3 and 24 characters in length and use numbers and lower-case letters only.} diff --git a/man/azureScriptActionHistory.Rd b/man/azureScriptActionHistory.Rd index 50eb15e..f7ec168 100644 --- a/man/azureScriptActionHistory.Rd +++ b/man/azureScriptActionHistory.Rd @@ -11,7 +11,7 @@ azureScriptActionHistory(azureActiveContext, resourceGroup, clustername = "*", \method{summary}{azureScriptActionHistory}(object, ...) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureSparkCMD.Rd b/man/azureSparkCMD.Rd index e22f754..2553bf0 100644 --- a/man/azureSparkCMD.Rd +++ b/man/azureSparkCMD.Rd @@ -8,7 +8,7 @@ azureSparkCMD(azureActiveContext, CMD, clustername, hdiAdmin, hdiPassword, sessionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{CMD}{CMD} diff --git a/man/azureSparkJob.Rd b/man/azureSparkJob.Rd index 1da35c2..55eb05b 100644 --- a/man/azureSparkJob.Rd +++ b/man/azureSparkJob.Rd @@ -8,7 +8,7 @@ azureSparkJob(azureActiveContext, FILE, clustername, hdiAdmin, hdiPassword, log = "URL", verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{FILE}{file} diff --git a/man/azureSparkListJobs.Rd b/man/azureSparkListJobs.Rd index 7af149a..94d6145 100644 --- a/man/azureSparkListJobs.Rd +++ b/man/azureSparkListJobs.Rd @@ -8,7 +8,7 @@ azureSparkListJobs(azureActiveContext, clustername, hdiAdmin, hdiPassword, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureSparkListSessions.Rd b/man/azureSparkListSessions.Rd index 3089519..76a9e9e 100644 --- a/man/azureSparkListSessions.Rd +++ b/man/azureSparkListSessions.Rd @@ -8,7 +8,7 @@ azureSparkListSessions(azureActiveContext, clustername, hdiAdmin, hdiPassword, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureSparkNewSession.Rd b/man/azureSparkNewSession.Rd index 4ff5096..627ba74 100644 --- a/man/azureSparkNewSession.Rd +++ b/man/azureSparkNewSession.Rd @@ -8,7 +8,7 @@ azureSparkNewSession(azureActiveContext, clustername, hdiAdmin, hdiPassword, kind = "spark", verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureSparkShowURL.Rd b/man/azureSparkShowURL.Rd index 3a415a8..dbc14b7 100644 --- a/man/azureSparkShowURL.Rd +++ b/man/azureSparkShowURL.Rd @@ -7,7 +7,7 @@ azureSparkShowURL(azureActiveContext, URL) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{URL}{URL} } diff --git a/man/azureSparkStopSession.Rd b/man/azureSparkStopSession.Rd index 2496ed6..600d061 100644 --- a/man/azureSparkStopSession.Rd +++ b/man/azureSparkStopSession.Rd @@ -8,7 +8,7 @@ azureSparkStopSession(azureActiveContext, clustername, hdiAdmin, hdiPassword, sessionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{clustername}{Cluster name, used for HDI and Spark clusters. See \code{\link[=azureCreateHDI]{azureCreateHDI()}}} diff --git a/man/azureStartVM.Rd b/man/azureStartVM.Rd index 19a72b8..7826bc6 100644 --- a/man/azureStartVM.Rd +++ b/man/azureStartVM.Rd @@ -8,7 +8,7 @@ azureStartVM(azureActiveContext, resourceGroup, vmName, mode = "Sync", subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureStopVM.Rd b/man/azureStopVM.Rd index 78885ce..507af4a 100644 --- a/man/azureStopVM.Rd +++ b/man/azureStopVM.Rd @@ -8,7 +8,7 @@ azureStopVM(azureActiveContext, resourceGroup, vmName, mode = "Sync", subscriptionID, verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/azureVMStatus.Rd b/man/azureVMStatus.Rd index aa55c7a..1102bc3 100644 --- a/man/azureVMStatus.Rd +++ b/man/azureVMStatus.Rd @@ -8,7 +8,7 @@ azureVMStatus(azureActiveContext, resourceGroup, vmName, subscriptionID, ignore = "N", verbose = FALSE) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{resourceGroup}{Name of the resource group} diff --git a/man/createAzureContext.Rd b/man/createAzureContext.Rd index c124465..43f6b1b 100644 --- a/man/createAzureContext.Rd +++ b/man/createAzureContext.Rd @@ -7,11 +7,11 @@ createAzureContext(tenantID, clientID, authKey, configFile) } \arguments{ -\item{tenantID}{The Tenant ID provided during creation of the Active Directory application / service principal} +\item{tenantID}{The tenant ID provided during creation of the Active Directory application / service principal} -\item{clientID}{The Client ID provided during creation of the Active Directory application / service principal} +\item{clientID}{The client ID provided during creation of the Active Directory application / service principal} -\item{authKey}{The Authentication Key provided during creation of the Active Directory application / service principal} +\item{authKey}{The authentication key provided during creation of the Active Directory application / service principal} } \value{ An \code{azureActiveContext} object diff --git a/man/pkg-deprecated.Rd b/man/pkg-deprecated.Rd index 19f6de9..a42f06d 100644 --- a/man/pkg-deprecated.Rd +++ b/man/pkg-deprecated.Rd @@ -10,7 +10,7 @@ dumpAzureContext(azureActiveContext) AzureListRG(...) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} \item{...}{passed to \code{\link[=azureListRG]{azureListRG()}}} } diff --git a/man/setAzureContext.Rd b/man/setAzureContext.Rd index dbf933f..5a7f663 100644 --- a/man/setAzureContext.Rd +++ b/man/setAzureContext.Rd @@ -9,13 +9,13 @@ setAzureContext(azureActiveContext, tenantID, clientID, authKey, subscriptionID, hdiPassword, clustername, kind, sessionID) } \arguments{ -\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}} +\item{azureActiveContext}{A container used for caching variables used by \code{AzureSMR}, created by \code{\link[=createAzureContext]{createAzureContext()}}} -\item{tenantID}{The Tenant ID provided during creation of the Active Directory application / service principal} +\item{tenantID}{The tenant ID provided during creation of the Active Directory application / service principal} -\item{clientID}{The Client ID provided during creation of the Active Directory application / service principal} +\item{clientID}{The client ID provided during creation of the Active Directory application / service principal} -\item{authKey}{The Authentication Key provided during creation of the Active Directory application / service principal} +\item{authKey}{The authentication key provided during creation of the Active Directory application / service principal} \item{subscriptionID}{Subscription ID. This is obtained automatically by \code{\link[=azureAuthenticate]{azureAuthenticate()}} when only a single subscriptionID is available via Active Directory} From 6d95552325871a32efbe434b626151c64558d49f Mon Sep 17 00:00:00 2001 From: Andrie de Vries Date: Fri, 16 Jun 2017 14:05:13 +0100 Subject: [PATCH 3/7] More fixes to vignettes #76 --- inst/doc/tutorial.R | 14 ++++---------- inst/doc/tutorial.Rmd | 32 +++++++++++++------------------- inst/doc/tutorial.html | 34 ++++++++++++++-------------------- vignettes/tutorial.Rmd | 32 +++++++++++++------------------- 4 files changed, 44 insertions(+), 68 deletions(-) diff --git a/inst/doc/tutorial.R b/inst/doc/tutorial.R index ae9eda1..ed4d228 100644 --- a/inst/doc/tutorial.R +++ b/inst/doc/tutorial.R @@ -48,7 +48,7 @@ # azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") ## ---- eval=FALSE--------------------------------------------------------- -# azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme") +# azureCreateStorageContainer(sc, "opendata", storageAccount = "testmystorage1", resourceGroup = "testme") ## ---- eval=FALSE--------------------------------------------------------- # azureListStorageContainers(sc, storageAccount = "testmystorage1", resourceGroup = "testme") @@ -114,12 +114,6 @@ # azureHiveSQL(sc, # CMD = "select * from hivesampletable", # path = "wasb://opendata@testmystorage1.blob.core.windows.net/") -# -# azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "opendata") -# -# stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout") -# -# read.delim(text = stdout, header = TRUE, fill = TRUE) ## ---- eval=FALSE--------------------------------------------------------- # azureSparkNewSession(sc, clustername = "smrhdi", @@ -153,7 +147,7 @@ # ## [1] "Pi is roughly 3.140285" ## ---- eval=FALSE--------------------------------------------------------- -# azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID="0") +# azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID = "0") # # #[1] "3.1422" @@ -162,6 +156,6 @@ # hdiAdmin = "hdiadmin", # hdiPassword = "AzureSMR_password123", # kind = "sparkr") -# azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2") -# azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2") +# azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID = "2") +# azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID = "2") diff --git a/inst/doc/tutorial.Rmd b/inst/doc/tutorial.Rmd index 0a54a27..d2b1bca 100644 --- a/inst/doc/tutorial.Rmd +++ b/inst/doc/tutorial.Rmd @@ -114,7 +114,7 @@ azureStopVM(sc, vmName = "DSVM1") ## Accessing storage blobs using the `azureActiveContext` -To access storage blobs you need to have a key. Use `azureSAGetKey()` to automatically retrieve your key or alternatively supply your own key. +To access storage blobs you need to have a key. You can use `azureSAGetKey()` to automatically retrieve your key. ```{r, eval=FALSE} @@ -124,7 +124,7 @@ azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") To create containers in a storage account use`azureCreateStorageContainer()` ```{r, eval=FALSE} -azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme") +azureCreateStorageContainer(sc, "opendata", storageAccount = "testmystorage1", resourceGroup = "testme") ``` @@ -159,7 +159,7 @@ azureGetBlob(sc, storageAccount = "testmystorage1", container = "opendata", ## Accessing storage blobs without an `azureActiveContext` -It is also possible to access the storage API without having an Azure Active Directory application. +It is also possible to access the blob functionswithout having an Azure Active Directory application. In this case, you should use the argument `azureActiveContect = NULL` to the storage functions. @@ -170,9 +170,9 @@ azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "open ``` -## Manage HDInsight Clusters +## Manage HDInsight clusters -You can use `AzureSMR` to manage HDInsight clusters. To create a cluster use `azureCreateHDI()`. +You can use `AzureSMR` to manage [HDInsight](https://azure.microsoft.com/en-gb/services/hdinsight/) clusters. To create a cluster use `azureCreateHDI()`. For advanced configurations use Resource Templates (See below). @@ -210,11 +210,11 @@ azureResizeHDI(sc, resourceGroup = "testme", clustername = "smrhdi", role="worke ``` -## Resource Templates - Create Azure Resources +## Resource templates - create Azure resources -The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates. +The easiest way to create resources on Azure is to use Azure Resource Manager (ARM) templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates. -To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example Azure blob). +To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example in Azure blob store), or you can supply these as JSON strings. ```{r, eval=FALSE} azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", @@ -233,7 +233,7 @@ ADMIN TIP: If a deployment fails, go to the Azure Portal and look at `Activity l ## Hive Functions -These functions facilitate the use of hive jobs on an HDInsight Cluster +You can use these functions to run and manage hive jobs on an HDInsight Cluster. ```{r, eval=FALSE} azureHiveStatus(sc, clusterName = "smrhdi", @@ -243,12 +243,6 @@ azureHiveStatus(sc, clusterName = "smrhdi", azureHiveSQL(sc, CMD = "select * from hivesampletable", path = "wasb://opendata@testmystorage1.blob.core.windows.net/") - -azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "opendata") - -stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout") - -read.delim(text = stdout, header = TRUE, fill = TRUE) ``` @@ -256,7 +250,7 @@ read.delim(text = stdout, header = TRUE, fill = TRUE) `AzureSMR` provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session. -To create a new Spark session (via Livy) use `azureSparkNewSession()` +To create a new Spark session (via [Livy](https://github.com/cloudera/hue/tree/master/apps/spark/java#welcome-to-livy-the-rest-spark-server)) use `azureSparkNewSession()` ```{r, eval=FALSE} azureSparkNewSession(sc, clustername = "smrhdi", @@ -299,7 +293,7 @@ azureSparkCMD(sc, CMD = pythonCmd, sessionID = "0") Check Session variables are retained ```{r, eval=FALSE} -azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID="0") +azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID = "0") #[1] "3.1422" ``` @@ -311,7 +305,7 @@ azureSparkNewSession(sc, clustername = "smrhdi", hdiAdmin = "hdiadmin", hdiPassword = "AzureSMR_password123", kind = "sparkr") -azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2") -azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2") +azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID = "2") +azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID = "2") ``` diff --git a/inst/doc/tutorial.html b/inst/doc/tutorial.html index 0a4135d..fc5f47e 100644 --- a/inst/doc/tutorial.html +++ b/inst/doc/tutorial.html @@ -153,10 +153,10 @@

Manage Virtual Machines

Accessing storage blobs using the azureActiveContext

-

To access storage blobs you need to have a key. Use azureSAGetKey() to automatically retrieve your key or alternatively supply your own key.

+

To access storage blobs you need to have a key. You can use azureSAGetKey() to automatically retrieve your key.

azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1")

To create containers in a storage account useazureCreateStorageContainer()

-
azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme")
+
azureCreateStorageContainer(sc, "opendata", storageAccount = "testmystorage1", resourceGroup = "testme")

To list containers in a storage account use azureListContainers()

azureListStorageContainers(sc, storageAccount = "testmystorage1", resourceGroup = "testme")

To write a blob use azurePutBlob()

@@ -172,14 +172,14 @@

Accessing storage blobs using the azureActiveContext

Accessing storage blobs without an azureActiveContext

-

It is also possible to access the storage API without having an Azure Active Directory application.

+

It is also possible to access the blob functionswithout having an Azure Active Directory application.

In this case, you should use the argument azureActiveContect = NULL to the storage functions.

For example:

azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "opendata")
-

Manage HDInsight Clusters

-

You can use AzureSMR to manage HDInsight clusters. To create a cluster use azureCreateHDI().

+

Manage HDInsight clusters

+

You can use AzureSMR to manage HDInsight clusters. To create a cluster use azureCreateHDI().

For advanced configurations use Resource Templates (See below).

azureCreateHDI(sc,
                  resourceGroup = "testme",
@@ -205,9 +205,9 @@ 

Manage HDInsight Clusters

## " headnode ( 2 * Standard_D3_v2 ) workernode ( 5 * Standard_D3_v2 ) zookeepernode ( 3 * Medium ) edgenode0 ( 1 * Standard_D4_v2 )"
-

Resource Templates - Create Azure Resources

-

The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into Settings > Automation scripts. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates.

-

To create a resource using a template in AzureSMR use azureDeployTemplate(). The template and paramaters must be available in a public URL (for example Azure blob).

+

Resource templates - create Azure resources

+

The easiest way to create resources on Azure is to use Azure Resource Manager (ARM) templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into Settings > Automation scripts. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates.

+

To create a resource using a template in AzureSMR use azureDeployTemplate(). The template and paramaters must be available in a public URL (for example in Azure blob store), or you can supply these as JSON strings.

azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", 
                     templateURL = "{TEMPLATEURL}", paramURL = "{PARAMURL}")
 
@@ -221,25 +221,19 @@ 

Resource Templates - Create Azure Resources

Hive Functions

-

These functions facilitate the use of hive jobs on an HDInsight Cluster

+

You can use these functions to run and manage hive jobs on an HDInsight Cluster.

azureHiveStatus(sc, clusterName = "smrhdi", 
                 hdiAdmin = "hdiadmin", 
                 hdiPassword = "AzureSMR_password123")
 
 azureHiveSQL(sc, 
              CMD = "select * from hivesampletable", 
-             path = "wasb://opendata@testmystorage1.blob.core.windows.net/")
-
-azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "opendata")
-
-stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout")
- 
-read.delim(text = stdout,  header = TRUE, fill = TRUE)
+ path = "wasb://opendata@testmystorage1.blob.core.windows.net/")

Spark functions (experimental)

AzureSMR provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session.

-

To create a new Spark session (via Livy) use azureSparkNewSession()

+

To create a new Spark session (via Livy) use azureSparkNewSession()

azureSparkNewSession(sc, clustername = "smrhdi", 
                      hdiAdmin = "hdiadmin", 
                      hdiPassword = "AzureSMR_password123",
@@ -268,7 +262,7 @@ 

Spark functions (experimental)

## [1] "Pi is roughly 3.140285"

Check Session variables are retained

-
azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID="0")
+
azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID = "0")
 
 #[1] "3.1422"

You can also run SparkR sessions

@@ -276,8 +270,8 @@

Spark functions (experimental)

hdiAdmin = "hdiadmin", hdiPassword = "AzureSMR_password123", kind = "sparkr") -azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2") -azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2")
+azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID = "2") +azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID = "2")
diff --git a/vignettes/tutorial.Rmd b/vignettes/tutorial.Rmd index 0a54a27..d2b1bca 100644 --- a/vignettes/tutorial.Rmd +++ b/vignettes/tutorial.Rmd @@ -114,7 +114,7 @@ azureStopVM(sc, vmName = "DSVM1") ## Accessing storage blobs using the `azureActiveContext` -To access storage blobs you need to have a key. Use `azureSAGetKey()` to automatically retrieve your key or alternatively supply your own key. +To access storage blobs you need to have a key. You can use `azureSAGetKey()` to automatically retrieve your key. ```{r, eval=FALSE} @@ -124,7 +124,7 @@ azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1") To create containers in a storage account use`azureCreateStorageContainer()` ```{r, eval=FALSE} -azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme") +azureCreateStorageContainer(sc, "opendata", storageAccount = "testmystorage1", resourceGroup = "testme") ``` @@ -159,7 +159,7 @@ azureGetBlob(sc, storageAccount = "testmystorage1", container = "opendata", ## Accessing storage blobs without an `azureActiveContext` -It is also possible to access the storage API without having an Azure Active Directory application. +It is also possible to access the blob functionswithout having an Azure Active Directory application. In this case, you should use the argument `azureActiveContect = NULL` to the storage functions. @@ -170,9 +170,9 @@ azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "open ``` -## Manage HDInsight Clusters +## Manage HDInsight clusters -You can use `AzureSMR` to manage HDInsight clusters. To create a cluster use `azureCreateHDI()`. +You can use `AzureSMR` to manage [HDInsight](https://azure.microsoft.com/en-gb/services/hdinsight/) clusters. To create a cluster use `azureCreateHDI()`. For advanced configurations use Resource Templates (See below). @@ -210,11 +210,11 @@ azureResizeHDI(sc, resourceGroup = "testme", clustername = "smrhdi", role="worke ``` -## Resource Templates - Create Azure Resources +## Resource templates - create Azure resources -The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates. +The easiest way to create resources on Azure is to use Azure Resource Manager (ARM) templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates. -To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example Azure blob). +To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example in Azure blob store), or you can supply these as JSON strings. ```{r, eval=FALSE} azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1", @@ -233,7 +233,7 @@ ADMIN TIP: If a deployment fails, go to the Azure Portal and look at `Activity l ## Hive Functions -These functions facilitate the use of hive jobs on an HDInsight Cluster +You can use these functions to run and manage hive jobs on an HDInsight Cluster. ```{r, eval=FALSE} azureHiveStatus(sc, clusterName = "smrhdi", @@ -243,12 +243,6 @@ azureHiveStatus(sc, clusterName = "smrhdi", azureHiveSQL(sc, CMD = "select * from hivesampletable", path = "wasb://opendata@testmystorage1.blob.core.windows.net/") - -azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "opendata") - -stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout") - -read.delim(text = stdout, header = TRUE, fill = TRUE) ``` @@ -256,7 +250,7 @@ read.delim(text = stdout, header = TRUE, fill = TRUE) `AzureSMR` provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session. -To create a new Spark session (via Livy) use `azureSparkNewSession()` +To create a new Spark session (via [Livy](https://github.com/cloudera/hue/tree/master/apps/spark/java#welcome-to-livy-the-rest-spark-server)) use `azureSparkNewSession()` ```{r, eval=FALSE} azureSparkNewSession(sc, clustername = "smrhdi", @@ -299,7 +293,7 @@ azureSparkCMD(sc, CMD = pythonCmd, sessionID = "0") Check Session variables are retained ```{r, eval=FALSE} -azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID="0") +azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID = "0") #[1] "3.1422" ``` @@ -311,7 +305,7 @@ azureSparkNewSession(sc, clustername = "smrhdi", hdiAdmin = "hdiadmin", hdiPassword = "AzureSMR_password123", kind = "sparkr") -azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2") -azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2") +azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID = "2") +azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID = "2") ``` From b61f806234f50c365265e012d77c9a9d487809c4 Mon Sep 17 00:00:00 2001 From: Andrie de Vries Date: Fri, 16 Jun 2017 16:55:34 +0100 Subject: [PATCH 4/7] Fix bug in argument naming --- R/internal.R | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/R/internal.R b/R/internal.R index aed841d..c4ee918 100644 --- a/R/internal.R +++ b/R/internal.R @@ -165,7 +165,7 @@ refreshStorageKey <- function(azureActiveContext, storageAccount, resourceGroup) updateAzureActiveContext <- function(x, storageAccount, storageKey, resourceGroup, container, blob, directory) { # updates the active azure context in place - assert_that(is.azureActiveContext(azureActiveContext)) + assert_that(is.azureActiveContext(x)) if (!missing(storageAccount)) x$storageAccount <- storageAccount if (!missing(resourceGroup)) x$resourceGroup <- resourceGroup if (!missing(storageKey)) x$storageKey <- storageKey @@ -174,4 +174,3 @@ updateAzureActiveContext <- function(x, storageAccount, storageKey, resourceGrou if (!missing(directory)) x$directory <- directory TRUE } - From 94524370a3eaf754334ea4d19afd23416c42ce63 Mon Sep 17 00:00:00 2001 From: Andrie de Vries Date: Fri, 16 Jun 2017 16:56:03 +0100 Subject: [PATCH 5/7] Split message into multiple lines. --- R/AzureStorageAccount.R | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/R/AzureStorageAccount.R b/R/AzureStorageAccount.R index 2f5a1d4..7146775 100644 --- a/R/AzureStorageAccount.R +++ b/R/AzureStorageAccount.R @@ -123,7 +123,8 @@ azureCreateStorageAccount <- function(azureActiveContext, storageAccount, rl <- content(r, "text", encoding = "UTF-8") azureActiveContext$storageAccount <- storageAccount azureActiveContext$resourceGroup <- resourceGroup - message("Create request Accepted. It can take a few moments to provision the storage account") + message("Create request accepted") + message("It can take a few moments to provision the storage account") if (!asynchronous) { wait_for_azure( From da5184a101b5166fc09a92b039c56130b6076a8a Mon Sep 17 00:00:00 2001 From: Andrie de Vries Date: Fri, 16 Jun 2017 17:07:15 +0100 Subject: [PATCH 6/7] Small changes in Azure Batch functions --- R/AzureBatch.R | 24 +++++++++--------------- man/azureCreateBatchAccount.Rd | 3 +++ man/azureDeleteBatchAccount.Rd | 3 +++ tests/testthat/test-batch.R | 4 ++-- 4 files changed, 17 insertions(+), 17 deletions(-) diff --git a/R/AzureBatch.R b/R/AzureBatch.R index 0d24165..69b71e9 100644 --- a/R/AzureBatch.R +++ b/R/AzureBatch.R @@ -31,8 +31,11 @@ azureListBatchAccounts <- function(azureActiveContext, resourceGroup, subscripti #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureBatchGetKey +#' #' @param location A string for the location to create batch account #' @param asynchronous If TRUE, submits asynchronous request to Azure. Otherwise waits until batch account is created. +#' +#' @references https://docs.microsoft.com/en-us/rest/api/batchmanagement/batchaccount #' @family Batch account functions #' @export azureCreateBatchAccount <- function(azureActiveContext, batchAccount, @@ -47,9 +50,7 @@ azureCreateBatchAccount <- function(azureActiveContext, batchAccount, assert_that(is_subscription_id(subscriptionID)) assert_that(is_storage_account(batchAccount)) - body <- paste0('{ - "location":"', location, '", - }' + body <- paste0('{"location":"', location, '"}' ) uri <- paste0("https://management.azure.com/subscriptions/", subscriptionID, @@ -59,24 +60,16 @@ azureCreateBatchAccount <- function(azureActiveContext, batchAccount, r <- call_azure_sm(azureActiveContext, uri = uri, body = body, verb = "PUT", verbose = verbose) - if (status_code(r) == 409) { - message("409: Conflict : Account already exists with the same name") - return(TRUE) - } - - if (status_code(r) == 200) { - message("Account already exists with the same properties") - } stopWithAzureError(r) - rl <- content(r, "text", encoding = "UTF-8") azureActiveContext$batchAccount <- batchAccount azureActiveContext$resourceGroup <- resourceGroup - message("Create request Accepted. It can take a few moments to provision the batch account") + message("Create request accepted") + message("It can take a few moments to provision the batch account") if (!asynchronous) { wait_for_azure( - batchAccount %in% azureListBatchAccounts(azureActiveContext, subscriptionID = subscriptionID)$name + batchAccount %in% azureListBatchAccounts(azureActiveContext)$name ) } TRUE @@ -88,7 +81,8 @@ azureCreateBatchAccount <- function(azureActiveContext, batchAccount, #' @inheritParams setAzureContext #' @inheritParams azureAuthenticate #' @inheritParams azureBatchGetKey - +#' +#' @references https://docs.microsoft.com/en-us/rest/api/batchmanagement/batchaccount#BatchAccount_Delete #' @family Batch account functions #' @export azureDeleteBatchAccount <- function(azureActiveContext, batchAccount, diff --git a/man/azureCreateBatchAccount.Rd b/man/azureCreateBatchAccount.Rd index 701d2e6..cc4df81 100644 --- a/man/azureCreateBatchAccount.Rd +++ b/man/azureCreateBatchAccount.Rd @@ -24,6 +24,9 @@ azureCreateBatchAccount(azureActiveContext, batchAccount, \description{ Create an azure batch account. } +\references{ +https://docs.microsoft.com/en-us/rest/api/batchmanagement/batchaccount +} \seealso{ Other Batch account functions: \code{\link{azureBatchGetKey}}, \code{\link{azureDeleteBatchAccount}}, diff --git a/man/azureDeleteBatchAccount.Rd b/man/azureDeleteBatchAccount.Rd index 3bd3ce5..61585d0 100644 --- a/man/azureDeleteBatchAccount.Rd +++ b/man/azureDeleteBatchAccount.Rd @@ -19,6 +19,9 @@ azureDeleteBatchAccount(azureActiveContext, batchAccount, resourceGroup, \description{ Delete an azure batch account. } +\references{ +https://docs.microsoft.com/en-us/rest/api/batchmanagement/batchaccount#BatchAccount_Delete +} \seealso{ Other Batch account functions: \code{\link{azureBatchGetKey}}, \code{\link{azureCreateBatchAccount}}, diff --git a/tests/testthat/test-batch.R b/tests/testthat/test-batch.R index 44d4513..c55ee03 100644 --- a/tests/testthat/test-batch.R +++ b/tests/testthat/test-batch.R @@ -17,12 +17,12 @@ azureAuthenticate(asc) timestamp <- format(Sys.time(), format = "%y%m%d%H%M") resourceGroup_name <- paste0("AzureSMtest_", timestamp) batch_account <- paste0("azuresmr", timestamp) -batch_location = "westeurope" +batch_location = "centralus" test_that("Can create resource group", { skip_if_missing_config(settingsfile) - res <- azureCreateResourceGroup(asc, location = "westeurope", resourceGroup = resourceGroup_name) + res <- azureCreateResourceGroup(asc, location = "centralus", resourceGroup = resourceGroup_name) expect_equal(res, TRUE) wait_for_azure( From c16651a6b8d2c7bf36685e23f65c54259dd338b8 Mon Sep 17 00:00:00 2001 From: Andrie de Vries Date: Fri, 16 Jun 2017 17:19:29 +0100 Subject: [PATCH 7/7] Bump version to 0.2.6 --- DESCRIPTION | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/DESCRIPTION b/DESCRIPTION index 14c79ed..d6284d1 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -5,12 +5,12 @@ Description: Helps users to manage Azure Services and objects from within an Machines and HDInsight (Spark, Hive). To use the package, you must configure an Azure Active Directory application and service principal in the Azure portal. Type: Package -Version: 0.2.5 -Date: 2017-06-06 +Version: 0.2.6 +Date: 2017-06-16 Authors@R: c( person(family="Microsoft Corporation", role="cph"), - person("Alan", "Weaver", role=c("aut", "cre"), email="alanwe@microsoft.com"), - person("Andrie", "de Vries", role=c("aut"), email="adevries@microsoft.com") + person("Alan", "Weaver", role=c("aut"), email="alanwe@microsoft.com"), + person("Andrie", "de Vries", role=c("aut", "cre"), email="adevries@microsoft.com") ) Copyright: Microsoft License: MIT + file LICENSE @@ -28,7 +28,7 @@ Imports: miniUI (>= 0.1.1), rstudioapi (>= 0.5), DT, - lubridate, + lubridate Depends: R(>= 3.0.0) Suggests: