You are working with the New York City Taxi and Limousine Commission. They have several years of data about taxi trips in New York City, but have no way to analyze it.
You will build a modern data estate to enable analytic and real-time data analysis and visualization. Your task is to use Azure data technologies to build this data estate, including:
- Ingesting and storing raw data
- Cleaning and enriching data
- Loading data into a serving layer
- Consuming data in a BI layer
- Orchestrating the overall data flow for repeatability
- Challenge 1: Preparation - Ready, Set, GO!
- Prepare your workstation for this hack.
- Challenge 2: Data Ingestion to Blob Storage
- Ingest and Store Source Data in Blob Storage
- Challenge 3: Databricks
- Clean and Merge Data in Databricks
- Ingest cleaned/merged data from blob storage in Azure Data Warehouse using Polybase and Azure Data Factory
- Challenge 4: Model and Consume Data
- Create Data model in Azure Analysis Services using SQL DW as source
- Consume Data in Power BI from Databricks and Azure Analysis Services
- Challenge 5: Weather Data
- Access a Weather API, store weather data in Cosmos DB, join to transaction data in Power BI
NOTE: Some of the challenges can be worked on in parallel; others require preceding challenges to be completed. We suggest you read all the challenges before starting work and discuss/assign them as a team.
An end-to-end solution spanning the data lifecycle, with functional Power BI dashboard(s).
../Coach- Coach's Guide for each challenge
../Coach/Solutions- Solution code for each challenge
../Student- Guides for each challenge
../Student/Resources- Architecture diagram for reference during the hack.
- Patrick El-Azem
- Neelam Gupta