Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add fromServiceAccountJsonInputStream #7

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

vijaykramesh
Copy link

similar to the BigQuery.fromServiceAccountJson (that expects a file path), this new method allows you to pass in an InputStream. We have a use-case where all configuration is happening via ENV vars, and I'd prefer to not have to write the config to the local (container) filesystem.

I also added a test for both of the service account json methods I have added. Same as the existing test, it requires you change yourOwnProjectId and to have a file existing at $HOME/.bigquery/service_account.json:

> testOnly *ServiceAccountUsageExamplesSpec
[info] Formatting 1 Scala source {file:/Users/vramesh/oss/bigquery4s/}root(test) ...
[info] Formatting 1 Scala source {file:/Users/vramesh/oss/bigquery4s/}root(compile) ...
[info] Compiling 1 Scala source to /Users/vramesh/oss/bigquery4s/target/scala-2.12/classes...
[info] Compiling 1 Scala source to /Users/vramesh/oss/bigquery4s/target/scala-2.12/test-classes...
[info] ServiceAccountUsageExamplesSpec:
[info] Simple query example with publicdata using service account json file
Oct 23, 2017 10:48:10 AM com.google.api.client.googleapis.services.AbstractGoogleClient <init>
WARNING: Application name is not set. Call Builder#setApplicationName.
2017-10-23 10:48:11,608 INFO bigquery4s.UsageExamplesSpec WrappedDatasets({"datasetReference":{"datasetId":"samples","projectId":"publicdata"},"id":"publicdata:samples","kind":"bigquery#dataset"})
2017-10-23 10:48:12,719 INFO bigquery4s.UsageExamplesSpec JobId: JobId(ProjectId(REDACTED),job_-s7T-GAUeKvcYxyH9tHNAxp24KWf)
2017-10-23 10:48:12,991 INFO bigquery4s.UsageExamplesSpec Job: WrappedCompletedJob({"configuration":{"query":{"createDisposition":"CREATE_IF_NEEDED","destinationTable":{"datasetId":"_05fb90c360272d1beef3822267407d6db9668147","projectId":"REDACTED","tableId":"anon3fc6a91182188a91987aa4f1347569a2e9c8631b"},"query":"\n        SELECT weight_pounds,state,year,gestation_weeks FROM publicdata:samples.natality\n        ORDER BY weight_pounds DESC LIMIT 100;\n      ","writeDisposition":"WRITE_TRUNCATE"}},"etag":"\"SdUekaVqp4U9n5pPfq5AV4uIdyk/YuWvQ10O8vfZNFDTfivjFFZsmco\"","id":"REDACTED:job_-s7T-GAUeKvcYxyH9tHNAxp24KWf","jobReference":{"jobId":"job_-s7T-GAUeKvcYxyH9tHNAxp24KWf","projectId":"REDACTED"},"kind":"bigquery#job","selfLink":"https://www.googleapis.com/bigquery/v2/projects/REDACTED/jobs/job_-s7T-GAUeKvcYxyH9tHNAxp24KWf","statistics":{"creationTime":"1508780891716","endTime":"1508780892405","query":{"cacheHit":true,"totalBytesBilled":"0","totalBytesProcessed":"0"},"startTime":"1508780892210","totalBytesProcessed":"0"},"status":{"state":"DONE"},"user_email":"[email protected]"})
2017-10-23 10:48:13,279 INFO bigquery4s.UsageExamplesSpec
*** QueryResult ***

weight_pounds,state,year,gestation_weeks
18.0007436923,null,2008,null
18.0007436923,null,2008,32
18.0007436923,null,2007,38
18.0007436923,null,2006,null
18.0007436923,KY,2004,38
18.0007436923,null,2005,null
18.0007436923,null,2007,null
18.0007436923,null,2005,40
18.0007436923,null,2006,37
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,KY,2004,41
18.0007436923,TX,1969,null
18.0007436923,KY,2004,39
18.0007436923,KY,2004,47
18.0007436923,null,2006,39
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,null,2005,35
18.0007436923,KY,2004,47

[info] - runs
[info] Simple query example with publicdata using service account json string
Oct 23, 2017 10:48:13 AM com.google.api.client.googleapis.services.AbstractGoogleClient <init>
WARNING: Application name is not set. Call Builder#setApplicationName.
2017-10-23 10:48:13,582 INFO bigquery4s.UsageExamplesSpec WrappedDatasets({"datasetReference":{"datasetId":"samples","projectId":"publicdata"},"id":"publicdata:samples","kind":"bigquery#dataset"})
2017-10-23 10:48:13,939 INFO bigquery4s.UsageExamplesSpec JobId: JobId(ProjectId(REDACTED),job_eELt2uPdtORwxOIBApXFsa7mVyEn)
2017-10-23 10:48:14,123 INFO bigquery4s.UsageExamplesSpec Job: WrappedCompletedJob({"configuration":{"query":{"createDisposition":"CREATE_IF_NEEDED","destinationTable":{"datasetId":"_05fb90c360272d1beef3822267407d6db9668147","projectId":"REDACTED","tableId":"anon3fc6a91182188a91987aa4f1347569a2e9c8631b"},"query":"\n          SELECT weight_pounds,state,year,gestation_weeks FROM publicdata:samples.natality\n          ORDER BY weight_pounds DESC LIMIT 100;\n        ","writeDisposition":"WRITE_TRUNCATE"}},"etag":"\"SdUekaVqp4U9n5pPfq5AV4uIdyk/ePu_0fQijyMGuQmEJ3dUSmRHgao\"","id":"REDACTED:job_eELt2uPdtORwxOIBApXFsa7mVyEn","jobReference":{"jobId":"job_eELt2uPdtORwxOIBApXFsa7mVyEn","projectId":"REDACTED"},"kind":"bigquery#job","selfLink":"https://www.googleapis.com/bigquery/v2/projects/REDACTED/jobs/job_eELt2uPdtORwxOIBApXFsa7mVyEn","statistics":{"creationTime":"1508780893661","endTime":"1508780893800","query":{"cacheHit":true,"totalBytesBilled":"0","totalBytesProcessed":"0"},"startTime":"1508780893772","totalBytesProcessed":"0"},"status":{"state":"DONE"},"user_email":"[email protected]"})
2017-10-23 10:48:14,397 INFO bigquery4s.UsageExamplesSpec
*** QueryResult ***

weight_pounds,state,year,gestation_weeks
18.0007436923,null,2008,null
18.0007436923,null,2008,32
18.0007436923,null,2007,38
18.0007436923,null,2006,null
18.0007436923,KY,2004,38
18.0007436923,null,2005,null
18.0007436923,null,2007,null
18.0007436923,null,2005,40
18.0007436923,null,2006,37
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,KY,2004,41
18.0007436923,TX,1969,null
18.0007436923,KY,2004,39
18.0007436923,KY,2004,47
18.0007436923,null,2006,39
18.0007436923,null,2005,42
18.0007436923,KY,2004,47
18.0007436923,null,2005,35
18.0007436923,KY,2004,47

[info] - runs
[info] Run completed in 4 seconds, 148 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.

@vijaykramesh
Copy link
Author

@seratch any thoughts?

@@ -155,7 +155,17 @@ object BigQuery {
scopes: Seq[String] = Seq(BigqueryScopes.BIGQUERY)
): BigQuery = {

val credential = using(new FileInputStream(serviceAccountJsonFilePath)) { in =>
BigQuery.fromServiceAccountJsonInputStream(new FileInputStream(serviceAccountJsonFilePath), transport, jsonFactory, scopes)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the FileInputStream resource must be closed.

scopes: Seq[String] = Seq(BigqueryScopes.BIGQUERY)
): BigQuery = {

val credential = using(serviceAccountJsonInputStream) { in =>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, the passed resource is closed here. I don't think it's a good approach. Closing a resource near its creation is easy to maintain.

@vijaykramesh
Copy link
Author

hey @seratch I was on holiday and missed your comments. I switched things back a bit so the existing fromServiceAccountJson function opens and closes its own input stream (versus that function calling to the new fromServiceAccountJsonInputStream). It makes the code slightly less dry but I agree it makes the opening and closing of the resource much clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants