Skip to content

Commit

Permalink
Merge pull request #656 from Azure-Samples/anuj/fixing-linting-link-i…
Browse files Browse the repository at this point in the history
…ssue

Fixing linting issues with broken links
  • Loading branch information
promisinganuj authored Nov 21, 2023
2 parents 974bde8 + 33a8ff1 commit 47aea3f
Show file tree
Hide file tree
Showing 16 changed files with 22 additions and 25 deletions.
5 changes: 1 addition & 4 deletions .markdownlinkcheck.json
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,10 @@
"pattern": "^https://azure.microsoft.com/en-us/free/"
},
{
"pattern": "^https://azure.microsoft.com/en-us/products/data-factory/"
"pattern": "^https://azure.microsoft.com/en-us/products/"
},
{
"pattern": "^https://dev.azure.com"
},
{
"pattern": "^https://azure.microsoft.com/en-us/free/databricks/"
}
]
}
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ description: "Code samples showcasing how to apply DevOps concepts to the Modern

# DataOps for the Modern Data Warehouse

This repository contains numerous code samples and artifacts on how to apply DevOps principles to data pipelines built according to the [Modern Data Warehouse (MDW)](https://azure.microsoft.com/en-au/solutions/architecture/modern-data-warehouse/) architectural pattern on Microsoft Azure.
This repository contains numerous code samples and artifacts on how to apply DevOps principles to data pipelines built according to the [Modern Data Warehouse (MDW)](https://learn.microsoft.com/en-au/azure/architecture/solution-ideas/articles/enterprise-data-warehouse) architectural pattern on Microsoft Azure.

The samples are either focused on a single azure service (**Single Tech Samples**) or showcases an end to end data pipeline solution as a reference implementation (**End to End Samples**). Each sample contains code and artifacts relating one or more of the following

Expand Down
2 changes: 1 addition & 1 deletion e2e_samples/mdw_governance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ The following shows the architecture of the solution.
- [Azure Virtual Network](https://docs.microsoft.com/azure/virtual-network/virtual-networks-overview)
- [Private Endpoint](https://docs.microsoft.com/azure/private-link/private-endpoint-overview)
- [Azure Function](https://docs.microsoft.com/azure/azure-functions/)
- [Azure Logic App](https://azure.microsoft.com/products/logic-apps/)
- [Azure Logic App](https://azure.microsoft.com/en-us/products/logic-apps/)
- [Azure Private DNS](https://docs.microsoft.com/azure/dns/private-dns-overview)

## Key Learnings
Expand Down
8 changes: 4 additions & 4 deletions e2e_samples/parking_sensors/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DataOps - Parking Sensor Demo <!-- omit in toc -->

The sample demonstrate how DevOps principles can be applied end to end Data Pipeline Solution built according to the [Modern Data Warehouse (MDW)](https://azure.microsoft.com/en-au/solutions/architecture/modern-data-warehouse/) pattern.
The sample demonstrate how DevOps principles can be applied end to end Data Pipeline Solution built according to the [Modern Data Warehouse (MDW)](https://learn.microsoft.com/en-au/azure/architecture/solution-ideas/articles/enterprise-data-warehouse) pattern.

## Contents <!-- omit in toc -->

Expand Down Expand Up @@ -58,7 +58,7 @@ The sample demonstrate how DevOps principles can be applied end to end Data Pipe

## Solution Overview

The solution pulls near realtime [Melbourne Parking Sensor data](https://www.melbourne.vic.gov.au/about-council/governance-transparency/open-data/Pages/on-street-parking-data.aspx) from a publicly available REST api endpoint and saves this to [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction). It then validates, cleanses, and transforms the data to a known schema using [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/). A second Azure Databricks job then transforms these into a [Star Schema](https://en.wikipedia.org/wiki/Star_schema) which are then loaded into [Azure Synapse Analytics (formerly SQLDW)](https://azure.microsoft.com/products/synapse-analytics/) using [Polybase](https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-guide?view=sql-server-ver15). The entire pipeline is orchestrated with [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/).
The solution pulls near realtime [Melbourne Parking Sensor data](https://www.melbourne.vic.gov.au/about-council/governance-transparency/open-data/Pages/on-street-parking-data.aspx) from a publicly available REST api endpoint and saves this to [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction). It then validates, cleanses, and transforms the data to a known schema using [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/). A second Azure Databricks job then transforms these into a [Star Schema](https://en.wikipedia.org/wiki/Star_schema) which are then loaded into [Azure Synapse Analytics (formerly SQLDW)](https://azure.microsoft.com/en-us/products/synapse-analytics/) using [Polybase](https://docs.microsoft.com/en-us/sql/relational-databases/polybase/polybase-guide?view=sql-server-ver15). The entire pipeline is orchestrated with [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/).

### Architecture

Expand All @@ -85,7 +85,7 @@ It makes use of the following azure services:
- [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/)
- [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/)
- [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction)
- [Azure Synapse Analytics (formerly SQLDW)](https://azure.microsoft.com/products/synapse-analytics/)
- [Azure Synapse Analytics (formerly SQLDW)](https://azure.microsoft.com/en-us/products/synapse-analytics/)
- [Azure DevOps](https://azure.microsoft.com/en-us/products/devops/)
- [Application Insights](https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview)
- [PowerBI](https://www.microsoft.com/en-us/power-platform/products/power-bi/)
Expand Down Expand Up @@ -205,7 +205,7 @@ More resources:
#### Data Factory

- [Monitor Azure Data Factory with Azure Monitor](https://docs.microsoft.com/en-us/azure/data-factory/monitor-using-azure-monitor)
- [Alerting in Azure Data Factory](https://azure.microsoft.com/en-in/blog/create-alerts-to-proactively-monitor-your-data-factory-pipelines/)
- [Alerting in Azure Data Factory](https://azure.microsoft.com/en-us/blog/create-alerts-to-proactively-monitor-your-data-factory-pipelines/)

## How to use the sample

Expand Down
4 changes: 2 additions & 2 deletions e2e_samples/parking_sensors_synapse/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# DataOps - Parking Sensor (Synapse) <!-- omit in toc -->

The sample demonstrate how DevOps principles can be applied to an end to end Data Pipeline Solution built according to the [Modern Data Warehouse (MDW)](https://azure.microsoft.com/en-au/solutions/architecture/modern-data-warehouse/) pattern, implemented in Azure Synapse.
The sample demonstrate how DevOps principles can be applied to an end to end Data Pipeline Solution built according to the [Modern Data Warehouse (MDW)](https://learn.microsoft.com/en-au/azure/architecture/solution-ideas/articles/enterprise-data-warehouse) pattern, implemented in Azure Synapse.

## Contents <!-- omit in toc -->

Expand Down Expand Up @@ -59,7 +59,7 @@ See [here](#build-and-release-pipeline) for details.

It makes use of the following azure services:

- [Azure Synapse Analytics](https://azure.microsoft.com/products/synapse-analytics/)
- [Azure Synapse Analytics](https://azure.microsoft.com/en-us/products/synapse-analytics/)
- [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction)
- [Azure DevOps](https://azure.microsoft.com/en-us/products/devops/)
- [PowerBI](https://www.microsoft.com/en-us/power-platform/products/power-bi/)
Expand Down
2 changes: 1 addition & 1 deletion e2e_samples/temperature_events/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ time | timestamp of value
It makes use of the following azure services:

- [Azure Event Hubs](https://azure.microsoft.com/en-us/products/event-hubs/)
- [Azure Functions](https://azure.microsoft.com/en-us/services/functions/)
- [Azure Functions](https://azure.microsoft.com/en-us/products/functions/)
- [Azure IoT Device Telemetry Simulator](https://github.com/Azure-Samples/Iot-Telemetry-Simulator/)
- [Azure DevOps](https://azure.microsoft.com/en-us/products/devops/)
- [Application Insights](https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview)
Expand Down
2 changes: 1 addition & 1 deletion single_tech_samples/azuresql/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ The following are some sample [Azure DevOps](https://docs.microsoft.com/en-us/az
### Prerequisites

1. [Github account](https://github.com/)
2. [Azure Account](https://azure.microsoft.com/free)
2. [Azure Account](https://azure.microsoft.com/en-us/free/)
- *Permissions needed*: ability to create and deploy to an azure [resource group](https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/overview), a [service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals), and grant the [collaborator role](https://docs.microsoft.com/en-us/azure/role-based-access-control/overview) to the service principal over the resource group.
3. [Azure DevOps Account](https://azure.microsoft.com/en-us/products/devops/)
- *Permissions needed*: ability to create [service connections](https://docs.microsoft.com/en-us/azure/devops/pipelines/library/service-endpoints?view=azure-devops&tabs=yaml) and [pipelines](https://docs.microsoft.com/en-us/azure/devops/pipelines/get-started/pipelines-get-started?view=azure-devops&tabs=yaml).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -109,11 +109,11 @@ This sample implementation focuses on securing the Azure Databricks Environment

- Appropriately select the [Storage redundancy](https://docs.microsoft.com/en-us/azure/storage/common/storage-redundancy) option.

- Leverage [Azure Cost Management and Billing](https://azure.microsoft.com/en-us/services/cost-management/) to track usage cost of the Azure Databricks and Storage services.
- Leverage [Azure Cost Management and Billing](https://azure.microsoft.com/en-us/products/cost-management/) to track usage cost of the Azure Databricks and Storage services.

- Use [Azure Advisor](https://azure.microsoft.com/en-us/services/advisor/) to optimize deployments by leveraging the smart insights.
- Use [Azure Advisor](https://azure.microsoft.com/en-us/products/advisor/) to optimize deployments by leveraging the smart insights.

- Use [Azure Policies](https://azure.microsoft.com/en-us/services/azure-policy/) to define guardrails around deployment constraints to regulate the cost.
- Use [Azure Policies](https://azure.microsoft.com/en-us/products/azure-policy/) to define guardrails around deployment constraints to regulate the cost.

- Operational Excellence

Expand Down
2 changes: 1 addition & 1 deletion single_tech_samples/databricks/sample4_ci_cd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ The following are the prerequisites for deploying this template :
1. [Github account](https://github.com/)
2. [Azure DevOps account](https://dev.azure.com)
3. [Azure account](https://portal.azure.com)
4. [Azure Databricks Workspace](https://azure.microsoft.com/en-us/services/databricks/)
4. [Azure Databricks Workspace](https://azure.microsoft.com/en-us/products/databricks/)

### 2.2. Infrastructure as Code (IaC)

Expand Down
2 changes: 1 addition & 1 deletion single_tech_samples/datafactory/sample1_cicd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ This sample demonstrates how to apply DevOps with Azure Data Factory (ADF) by so

## Solution Overview

This solution sets up an [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) storage account, with a folder structure that enables data tiering (bronze, silver, gold), and an [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/)(ADF) instance with linked services connecting to the data lake, to a separate file share and a [key vault](https://azure.microsoft.com/en-us/services/key-vault/) for secrets.
This solution sets up an [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) storage account, with a folder structure that enables data tiering (bronze, silver, gold), and an [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/)(ADF) instance with linked services connecting to the data lake, to a separate file share and a [key vault](https://azure.microsoft.com/en-us/products/key-vault/) for secrets.

The Azure Data Factory contains a simple pipeline taking data from the file share and ingesting it to the bronze folder.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ The following technologies are used to build this sample:

- [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/)
- [Azure Batch](https://azure.microsoft.com/en-us/products/batch)
- [Azure Storage(ADLS)](https://azure.microsoft.com/services/storage/data-lake-storage/)
- [Azure Data Lake Storage](https://azure.microsoft.com/en-us/products/storage/data-lake-storage/)
- [NFS Mounts](https://learn.microsoft.com/azure/storage/blobs/network-file-system-protocol-support-how-to)

## How to use this sample
Expand Down
2 changes: 1 addition & 1 deletion single_tech_samples/datashare/Readme.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Data Share Automation

[Azure Data Share](https://azure.microsoft.com/en-us/services/data-share/) setup requires a number of steps to establish the connection between the source data and the destination. One of those steps is sending an invitation from a source data share account and accepting the invitation in a destination data share account.
[Azure Data Share](https://azure.microsoft.com/en-us/products/data-share/) setup requires a number of steps to establish the connection between the source data and the destination. One of those steps is sending an invitation from a source data share account and accepting the invitation in a destination data share account.

Through the portal UI, invitations can only be sent to email addresses and that requires the email recipient to perform some manual steps to accept the invitation and map the incoming data to the destination. However, the Azure Data Share SDK allows invitations to be sent to *service principals* as well, which opens up the opportunity to fully automate the process, even between different subscriptions and tenants.

Expand Down
2 changes: 1 addition & 1 deletion single_tech_samples/purview/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ Azure Purview is a managed data governance cloud service for centrally managing

## Solution Overview

This solution sets up an [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) storage account, with a folder structure that enables data tiering (bronze, silver, gold), an [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/)(ADF) instance with linked services connecting to the data lake, a [key vault](https://azure.microsoft.com/en-us/services/key-vault/) for any secrets, and an [Azure Purview](https://azure.microsoft.com/en-us/products/purview/) account for data governance.
This solution sets up an [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction) storage account, with a folder structure that enables data tiering (bronze, silver, gold), an [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/)(ADF) instance with linked services connecting to the data lake, a [key vault](https://azure.microsoft.com/en-us/products/key-vault/) for any secrets, and an [Azure Purview](https://azure.microsoft.com/en-us/products/purview/) account for data governance.

The Azure Data Factory contains a simple ADF Pipeline that is stored in a git repository, that is taking data from the Dropzone and ingesting it into the bronze folder.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Data lifecycle is one such feature on Azure using which you can optimize costs b

The following technologies are used to build this sample:

- [Azure Storage](https://azure.microsoft.com/products/storage/data-lake-storage)
- [Azure Storage](https://azure.microsoft.com/en-us/products/storage/data-lake-storage)
- [Terraform](https://www.terraform.io)

## 2. How to use this sample
Expand Down
2 changes: 1 addition & 1 deletion single_tech_samples/streamanalytics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

![introductory diagram](./docs/images/ASA-job.PNG)

[Azure Stream Analytics](https://azure.microsoft.com/products/stream-analytics/) is a serverless real-time analytics service. The goal of this sample is to demonstrate how to develop a streaming pipeline, with IaC and testability in mind.
[Azure Stream Analytics](https://azure.microsoft.com/en-us/products/stream-analytics/) is a serverless real-time analytics service. The goal of this sample is to demonstrate how to develop a streaming pipeline, with IaC and testability in mind.

## Prerequisites

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ The solution runs a flow triggered on a storage file upload. It then runs a samp

It makes use of the following azure services:

- [Azure Synapse Analytics](https://azure.microsoft.com/products/synapse-analytics/)
- [Azure Synapse Analytics](https://azure.microsoft.com/en-us/products/synapse-analytics/)
- [Azure Data Lake Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-introduction)

## Key Concepts
Expand Down

0 comments on commit 47aea3f

Please sign in to comment.