Merge pull request #88 from cycloidio/sd-azurerm-provider

AzureRM provider implementation

CHANGELOG.md

@@ -10,6 +10,8 @@
   ([PR #78](https://github.com/cycloidio/terracognita/pull/78))
 - New flag `--target` to allow specific resource+id import
   ([Issue #40](https://github.com/cycloidio/terracognita/issues/40))
+- New AzureRM provider
+  ([PR #88](https://github.com/cycloidio/terracognita/pull/88))
 
 ## [0.3.0] _2020-01-02_
 

CONTRIBUTING.md

@@ -17,7 +17,7 @@ There are 6 types of labels, they can be used for issues or PRs:
 - `bug`: These track bugs with the code
 - `docs`: These track problems with the documentation (i.e. missing or incomplete)
 - `maintenance`: These tracks problems, update and migration for dependencies / third-party tools
-- `refactoring`: These tracks internal improvment with no direct impact on the product
+- `refactoring`: These tracks internal improvement with no direct impact on the product
 - `need review`: this status must be set when you feel confident with your submission
 - `in progress`: some important change has been requested on your submission, so you can toggle from `need review` to `in progress`
 - `under discussion`: it's time to take a break, think about this submission and try to figure out how we can implement this or this
@@ -33,7 +33,7 @@ $ git clone https://github.com/<your-username>/terracognita.git
 $ cd terracognita
 ```
 
-In order to stay updated with the upstream, it's highly recommended to add cycloidio/terracognita as a remote upstream.
+In order to stay updated with the upstream, it's highly recommended to add `cycloidio/terracognita` as a remote upstream.
 
 ```shell
 $ git remote add upstream https://github.com/cycloidio/terracognita.git
@@ -50,7 +50,7 @@ $ git rebase upstream/master
 
 #### Build from sources
 
-Since Terracognita is a Go project, Go must be installed and configured on your machine (really ?). We currently support Go1.12 and Go.13 and go `modules` as dependency manager. You can simply pull all necessaries dependencies by running an initial
+Since Terracognita is a Go project, Go must be installed and configured on your machine (really ?). We currently support Go1.12 and Go.13 and go `modules` as dependency manager. You can simply pull all necessaries dependencies by running an initial.
 
 ```shell
 $ make build
@@ -59,32 +59,45 @@ $ make build
 This basically builds `terracognita` with the current sources.
 
 You also need to install other code dependencies not mandatory in the runtime environment:
-  * [enumer](https://github.com/dmarkham/enumer) is used to play around AWS or GCP provider, it is used to generate some code
-  * [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) is used to format / organize code and imports. CI will perform a check on this, we highly recommend to run `$ goimports -w <your-modified-files>`
+  * [enumer](https://github.com/dmarkham/enumer) is used to generate some code.
+  * [goimports](https://godoc.org/golang.org/x/tools/cmd/goimports) is used to format / organize code and imports. CI will perform a check on this, we highly recommend to run `$ goimports -w <your-modified-files>`.
 
 #### Add a component to an existing provider (AWS or GCP)
 
-We currently support two providers: Amazon Web Services (AWS) and Google Cloud Platform (GCP). If you want to play around with one of this provider, you can follow the following guidelines.
+We currently support three providers: Amazon Web Services (AWS), Google Cloud Platform (GCP) and Azure Resource Manager (AzureRM). If you want to play around with one of this provider, you can follow the following guidelines.
 
 For both providers, you need to add your component to this places or equivalent:
 
-* As a `const` value [here](https://github.com/cycloidio/terracognita/blob/000789d3bd61b81cf10695d414f4f45346ccc25f/google/resources.go#L17) for Google and [here](https://github.com/cycloidio/terracognita/blob/000789d3bd61b81cf10695d414f4f45346ccc25f/aws/resources.go#L22) for AWS. :warning: your component name must exactly maps the name as in Terraform documentation. Example: `ComputeInstance` will be later used as `google_compute_instance` :warning:.
+* As a `const` value:
+  * [here](https://github.com/cycloidio/terracognita/blob/21f3387820bae5577c277acb634ebde5607c23ec/aws/resources.go#L23) for AWS.
+  * [here](https://github.com/cycloidio/terracognita/blob/21f3387820bae5577c277acb634ebde5607c23ec/azurerm/resources.go#L17) for AzureRM.
+  * [here](https://github.com/cycloidio/terracognita/blob/21f3387820bae5577c277acb634ebde5607c23ec/google/resources.go#L19) for Google.
+
+:warning: your component name must exactly map the name as in Terraform documentation. Example: `ComputeInstance` will be later used as `google_compute_instance` :warning:.
 
 ```go
 //go:generate enumer -type ResourceType -addprefix google_ -transform snake -linecomment
 ```
 
 This means you will need to generate some code.
 
-* As a `key/value`, in this [array](https://github.com/cycloidio/terracognita/blob/000789d3bd61b81cf10695d414f4f45346ccc25f/google/resources.go#L25)([here](https://github.com/cycloidio/terracognita/blob/000789d3bd61b81cf10695d414f4f45346ccc25f/aws/resources.go#L96) for AWS). This is where you map your component with a middleware function.
+* As a `key/value`, in this:
+  * [array](https://github.com/cycloidio/terracognita/blob/21f3387820bae5577c277acb634ebde5607c23ec/aws/resources.go#L111) for AWS.
+  * [array](https://github.com/cycloidio/terracognita/blob/21f3387820bae5577c277acb634ebde5607c23ec/azurerm/resources.go#L30) for AzureRM.
+  * [array](https://github.com/cycloidio/terracognita/blob/21f3387820bae5577c277acb634ebde5607c23ec/google/resources.go#L46) for Google.
+
+This is where you map your component with a middleware function.
 
 ```go
-...
-ComputeInstance: computeInstance,
-...
+var (
+	resources = map[ResourceType]rtFn{
+		...
+		ComputeInstance: computeInstances,
+	}
+)
 ```
 
-Now, you can write your `computeInstance` function. Check-out the other functions, you basically fetch resources from a middleware layer and you add them as Terraform resources.
+Now, you can write your `computeInstances` function. Check-out the other functions, you basically fetch resources from a middleware layer and you add them as Terraform resources.
 
 ##### AWS Middleware layer
 
@@ -96,14 +109,16 @@ We have an `aws/cmd` that generates the `aws/reader` interface, which is then us
 
 Functions are based on `https://github.com/aws/aws-sdk-go/tree/master/service`.
 For example with `aws_db_parameter_group` you should be able to find the Entity, Prefix, Service in https://github.com/aws/aws-sdk-go/blob/master/service/rds/rdsiface/interface.go#L313
-In our case we want to read data, the dedicated function is `DescribeDBParameterGroups`
+In our case we want to read data, the dedicated function is `DescribeDBParameterGroups`.
 
   * Entity: use the function name without prefix "DBParameterGroups"
-  * Prefix: use the function prefix (generaly Describe, List or Get)
+  * Prefix: use the function prefix (generally Describe, List or Get)
   * Service: SubDirectory from aws-sdk-go service `service/rds`
 
+```shell
+$ vim aws/cmd/functions.go
 ```
-vim aws/cmd/functions.go
+```go
 
 	Function{
 		Entity:  "DBParameterGroups",
@@ -119,35 +134,35 @@ vim aws/cmd/functions.go
 
 Functions are used to generate Get methods in `aws/reader/reader.go`. After a `make generate` execution, we should find a `GetDBParameterGroups` method at the end of the file corresponding to our previous example.
 
-
 2. Add your resource type
 
 The naming is based on terraform resources like `aws_db_parameter_group` but without the `aws` prefix and Snake case like `bla_foo` needs to be replaced by Camel case  `blaFoo`. The function should start with a `lowercase` and end with a `s`.
 The end result would be `dbParameterGroups`.
 
-
+```shell
+$ vim aws/resources.go
 ```
-vim aws/resources.go
-
-
+```go
 const (
-
-  DBParameterGroup
+	...
+	DBParameterGroup
+)
 
 ...
 
-resources = map[ResourceType]rtFn{
-  DBParameterGroup:   dbParameterGroups,
-
+var (
+	resources = map[ResourceType]rtFn{
+		DBParameterGroup: dbParameterGroups,
+        ...
+	}
+)
 ```
 
-const is used to generate resourcetype_enumer.go
-
+The const is used to generate `aws/resourcetype_enumer.go`.
 
 3. Add the associated function to generate terraform codes.
 
-
-```
+```go
 func dbParameterGroups(ctx context.Context, a *aws, resourceType string, tags []tag.Tag) ([]provider.Resource, error) {
 	dbParameterGroups, err := a.awsr.GetDBParameterGroups(ctx, nil)
 
@@ -174,21 +189,115 @@ func dbParameterGroups(ctx context.Context, a *aws, resourceType string, tags []
 
 Last step is to re-generate the following files with enumer and build/install.
 
-  * aws/reader/reader.go
-  * aws/resourcetype_enumer.go
+  * `aws/reader/reader.go`
+  * `aws/resourcetype_enumer.go`
 
-```
-make generate
-make test
+```shell
+$ make generate
+$ make test
 ```
 
 5. Update CHANGELOG
 
 Don't forget to update the `CHANGELOG.md`
 
+##### AzureRM Middleware layer
+
+We have an `azurerm/cmd` that generates the `azurerm/reader_generated.go` methods used to get the resources from the Azure SDK. To add a new call you have to add a new Function to the list in `aws/cmd/generate.go` (and the corresponding AzureAPI if necessary) and run `make generate`, you'll have the code fully generated for that function.
+
+###### Example with azurerm_virtual_machine
+
+1. Add your function
+
+Functions are based on `https://github.com/Azure/azure-sdk-for-go/tree/master/services`.
+For example with `azurerm_virtual_machine` you should be able to find the required information in https://github.com/Azure/azure-sdk-for-go/blob/master/services/compute/mgmt/2019-12-01/compute/virtualmachines.go
+The API to use is the `compute` service with the version `2019-12-01`.
+In our case we want to list data from the ResourceGroup used when executing `terracognita`, the corresponding function is `List`, sometimes another List function is needed as it's depends if it based on a ResourceGroup, a Location or will only list all resources without those delimiters.
+
+  * API: SubDirectory azure-sdk-for-go services `compute`
+  * APIVersion: SubDirectory azure-sdk-for-go mgmt `2019-12-01`
+  * Resource: resource name in singular PascalCase `VirtualMachine`
+  * ResourceGroup: `true` as the List function require the resource group name as a parameter
+
+```shell
+$ vim aws/cmd/generate.go
+```
+```go
+var azureAPIs = []AzureAPI{
+	...
+	AzureAPI{API: "compute", APIVersion: "2019-07-01"},
+}
+
+var functions = []Function{
+	...
+	Function{Resource: "VirtualMachine", API: "compute", ResourceGroup: true},
+}
+```
+
+Functions are used to generate List methods in `azurerm/reader_generated.go`. After a `make generate` execution, we should find a `ListVirtualMachines` method at the end of the file corresponding to our previous example.
+
+2. Add your resource type
+
+The naming is based on terraform resources like `azurerm_virtual_machine` but without the `azurerm` prefix and Snake case like `bla_foo` needs to be replaced by Camel case  `blaFoo`. The function should start with a `lowercase` and end with a `s`.
+The end result would be `virtualMachines`.
+
+```shell
+$ vim azurerm/resources.go
+```
+```go
+const (
+	...
+	VirtualMachine ResourceType = iota
+)
+
+...
+
+var (
+	resources = map[ResourceType]rtFn{
+		...
+		VirtualMachine: virtualMachines,
+	}
+)
+```
+
+The const is used to generate `azurerm/resourcetype_enumer.go`.
+
+3. Add the associated function to generate terraform codes.
+
+```go
+func virtualMachines(ctx context.Context, a *azurerm, resourceType string, tags []tag.Tag) ([]provider.Resource, error) {
+	virtualMachines, err := a.azurer.ListVirtualMachines(ctx)
+	if err != nil {
+		return nil, errors.Wrap(err, "unable to list virtual machines from reader")
+	}
+	resources := make([]provider.Resource, 0)
+	for _, virtualMachine := range virtualMachines {
+		r := provider.NewResource(*virtualMachine.ID, resourceType, a)
+		resources = append(resources, r)
+	}
+	return resources, nil
+}
+```
+
+4. Make generate
+
+Last step is to re-generate the following files with enumer and build/install.
+
+  * `azurerm/reader_generated.go`
+  * `azurerm/resourcetype_enumer.go`
+
+```
+$ make generate
+$ make test
+```
+
+5. Update CHANGELOG
+
+Don't forget to update the `CHANGELOG.md`.
+
 ##### GCP Middleware layer
 
-In `reader.go`, you can add your middleware function `ListInstances`. You will need to be equiped with this [documentation](https://godoc.org/google.golang.org/api/compute/v1). Google SDK is pretty standard, APIs are most of the time used in a similar way.
+In `reader.go`, you can add your middleware function `ListInstances`. You will need to be equipped with this [documentation](https://godoc.org/google.golang.org/api/compute/v1). Google SDK is pretty standard, APIs are most of the time used in a similar way.
 You only need to find out if your component belongs to a `project` or a `project` and a `zone`. It's highly recommended to base your code on the other functions (a method to generate this function will be provided soon).
 
 #### Build and test your component

Makefile

@@ -4,6 +4,7 @@ BIN_DIR := $(GOPATH)/bin
 
 GOLINT := $(BIN_DIR)/golinter
 GOIMPORTS := $(BIN_DIR)/goimports
+ENUMER := $(BIN_DIR)/enumer
 MOCKGEN := $(BIN_DIR)/mockgen
 
 VERSION= $(shell git describe --tags --always)
@@ -34,6 +35,9 @@ $(MOCKGEN):
 $(GOIMPORTS):
 	@go get -u golang.org/x/tools/cmd/goimports
 
+$(ENUMER):
+	@go get -u github.com/dmarkham/enumer
+
 $(GOLINT):
 	@go get -u golang.org/x/lint/golint
 
@@ -42,7 +46,7 @@ lint: $(GOLINT) $(GOIMPORTS) ## Runs the linter
 	@GO111MODULE=on golint -set_exit_status ./... && test -z "`go list -f {{.Dir}} ./... | xargs goimports -l | tee /dev/stderr`"
 
 .PHONY: generate
-generate: $(MOCKGEN) ## Generates the needed code
+generate: $(MOCKGEN) $(GOIMPORTS) $(ENUMER) ## Generates the needed code
 	@GO111MODULE=on rm -rf ./mock/a && \
 		go generate ./... && \
 		goimports -w ./mock

README.md

@@ -11,11 +11,12 @@ We decided to opensource this tool as we believe that it will help people to ado
 
 ## Cloud providers
 
-Terracognita currently imports AWS and GCP cloud provider as terraform (0.12.7) resource/state.
+Terracognita currently imports AWS, GCP and AzureRM cloud provider as terraform (0.12.8) resource/state.
 Please see the following versions as follow:
 
 Providers:
  * AWS: 2.31.0
+ * AzureRM: 1.35.0
  * GCP: 2.16.0
 
 ## Installation

azurerm/cache.go

@@ -0,0 +1,45 @@
+package azurerm
+
+import (
+	"context"
+
+	"github.com/cycloidio/terracognita/errcode"
+	"github.com/cycloidio/terracognita/provider"
+	"github.com/cycloidio/terracognita/tag"
+	"github.com/pkg/errors"
+)
+
+func cacheVirtualNetworks(ctx context.Context, a *azurerm, rt string, tags []tag.Tag) ([]provider.Resource, error) {
+	rs, err := a.cache.Get(rt)
+	if err != nil {
+		if errors.Cause(err) != errcode.ErrCacheKeyNotFound {
+			return nil, errors.WithStack(err)
+		}
+
+		rs, err = virtualNetworks(ctx, a, rt, tags)
+		if err != nil {
+			return nil, errors.Wrap(err, "unable to get virtual networks")
+		}
+
+		err = a.cache.Set(rt, rs)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return rs, nil
+}
+
+func getVirtualNetworkNames(ctx context.Context, a *azurerm, rt string, tags []tag.Tag) ([]string, error) {
+	rs, err := cacheVirtualNetworks(ctx, a, rt, tags)
+	if err != nil {
+		return nil, err
+	}
+
+	names := make([]string, 0, len(rs))
+	for _, i := range rs {
+		names = append(names, i.Data().Get("name").(string))
+	}
+
+	return names, nil
+}