Skip to content

Commit

Permalink
feat: add example of data serverless pipeline with GCP workflows and …
Browse files Browse the repository at this point in the history
…Eventarc
  • Loading branch information
jackdbd committed Sep 12, 2022
1 parent 1a1c63a commit 62fd086
Show file tree
Hide file tree
Showing 5 changed files with 149 additions and 3 deletions.
49 changes: 49 additions & 0 deletions assets/bigquery-schemas/weather_data_table.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
[
{
"name": "sensorId",
"type": "STRING",
"mode": "NULLABLE"
},
{
"name": "timecollected",
"description": "UTC date string of when the sensor reading was collected",
"type": "TIMESTAMP",
"mode": "NULLABLE"
},
{
"name": "zipcode",
"type": "INTEGER",
"mode": "NULLABLE"
},
{
"name": "latitude",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "longitude",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "temperature",
"description": "temperature in Farhenheit",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "humidity",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "dewpoint",
"type": "FLOAT",
"mode": "NULLABLE"
},
{
"name": "pressure",
"type": "FLOAT",
"mode": "NULLABLE"
}
]
9 changes: 9 additions & 0 deletions docs/service-accounts.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,15 @@ gcloud iam service-accounts create sa-dash-earthquakes \
--display-name "dash-earthquakes SA"
```

### sa-dataflow-worker

Create a service account to run [Dataflow](https://cloud.google.com/dataflow/docs) jobs:

```sh
gcloud iam service-accounts create sa-dataflow-worker \
--display-name "SA Dataflow worker"
```

### sa-firestore-user-test

Service account that I use in [firestore-utils](../packages/firestore-utils/README.md) tests.
Expand Down
6 changes: 3 additions & 3 deletions iam-policies/project.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ bindings:
- members:
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
role: roles/clouddebugger.agent
- members:
- serviceAccount:[email protected]
Expand Down Expand Up @@ -146,9 +146,9 @@ bindings:
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
- serviceAccount:[email protected]
role: roles/secretmanager.secretAccessor
- members:
Expand Down
29 changes: 29 additions & 0 deletions workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,18 @@ gcloud workflows deploy lead-generation \
--labels customer=$CUSTOMER,environment=$ENVIRONMENT,resource=workflow
```

### Serverless data pipeline

```sh
gcloud workflows deploy serverless-data-pipeline \
--project $GCP_PROJECT_ID \
--location $WORKFLOW_LOCATION \
--description "Serverless data pipeline (variation of the codelab 'Building a Serverless Data Pipeline: IoT to Analytics')" \
--source workflows/serverless-data-pipeline.yaml \
--service-account $SA_WORKFLOWS_RUNNER \
--labels customer=$CUSTOMER,environment=$ENVIRONMENT,resource=workflow
```

### Web performance audit

```sh
Expand Down Expand Up @@ -156,3 +168,20 @@ You can find the list of available [Compute Engine images](https://cloud.google.
```sh
gcloud compute images list
```

### Publish message to PubSub topic

```sh
gcloud pubsub topics publish weather-data \
--message '{
"sensorId": "sensor-xyz",
"zipcode": 55049,
"temperature": 98.5,
"timecollected": "2022-08-15 15:29:35",
"latitude": 43.8657,
"longitude": 10.2513,
"humidity": 1.23,
"dewpoint": 4.56,
"pressure": 7.89
}'
```
59 changes: 59 additions & 0 deletions workflows/serverless-data-pipeline.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# https://codelabs.developers.google.com/codelabs/iot-data-pipeline
main:
params: [args]
steps:

- assign_variables:
assign:
- project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
- dataset_id: weather_data
- table_id: weather_data_table
- base64_decoded: "${base64.decode(args.data.data)}"
- json_data: "${json.decode(base64_decoded)}"

# - log_stuff:
# call: sys.log
# args:
# data: {
# "json_data": "${json_data}",
# "dataset_id": "${dataset_id}",
# "table_id": "${table_id}"
# }
# severity: "WARNING"

# - list_table_data:
# # https://cloud.google.com/workflows/docs/reference/googleapis/bigquery/v2/tabledata/list
# call: googleapis.bigquery.v2.tabledata.list
# args:
# projectId: ${project_id}
# datasetId: ${dataset_id}
# tableId: ${table_id}
# maxResults: 30
# result: tabledata_list_result

- insert_all_table_data:
# https://cloud.google.com/workflows/docs/reference/googleapis/bigquery/v2/tabledata/insertAll
call: googleapis.bigquery.v2.tabledata.insertAll
args:
projectId: ${project_id}
datasetId: ${dataset_id}
tableId: ${table_id}
# https://cloud.google.com/workflows/docs/reference/googleapis/bigquery/v2/Overview#TableDataInsertAllRequest
body: {
"rows": [
{"json": "${json_data}"}
]
}
# body: {
# "rows": [
# {"json": {"sensorId": "sensor-abc", "zipcode": 90210, "temperature": 80.5}},
# {"json": {"sensorId": "sensor-def", "zipcode": 10048, "temperature": 101.75}}
# ]
# }
result: tabledata_insertAll_result

- respond_to_caller:
return: {
"tabledata_insertAll_result": "${tabledata_insertAll_result}"
# "tabledata_list_result": "${tabledata_list_result}"
}

0 comments on commit 62fd086

Please sign in to comment.