Skip to content

Cloud Function event that triggers when a file is added to a Cloud Storage bucket and creates a BigQuery load job to write the data into a table.

Notifications You must be signed in to change notification settings

asaharland/functions-bq-load-job

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Cloud Functions: Cloud Storage To BigQuery Load Job

This code triggers a BigQuery load job when a file is written to a Google Cloud Storage bucket.

View the source code.

Deploy and Test

  1. Follow the Cloud Functions quickstart guide to setup Cloud Functions for your project.

  2. Clone this repository:

     git clone https://github.com/asaharland/functions-bq-load-job
    
  3. Create a Cloud Storage Bucket:

     gsutil mb gs://BUCKET_NAME
    
    • Replace BUCKET_NAME with the name of your Cloud Storage Bucket.
  4. Deploy the "loadFile" function with an HTTP trigger:

     gcloud functions deploy loadFile --runtime nodejs8 --trigger-resource gs://BUCKET_NAME --trigger-event google.storage.object.finalize
    
  5. Create a BigQuery dataset and a table with the following schema:

     user_id:STRING,amount:FLOAT
    
  6. Update the variables datasetId and tableId in index.js to reflect the dataset and table you created in the previous step.

  7. Upload 'sample.csv', located in the root of this repo, to your bucket:

     gsutil cp sample.csv gs://BUCKET_NAME
    
    • Replace YOUR_BUCKET_NAME with the name of your Cloud Storage Bucket.
  8. Query the BigQuery table to check that you can see that the data has been inserted successfully.

About

Cloud Function event that triggers when a file is added to a Cloud Storage bucket and creates a BigQuery load job to write the data into a table.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published