Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix gcs_client in order to load data using gcs_bucket parameter #164

Merged

Conversation

kashira202111
Copy link

@kashira202111 kashira202111 commented Jul 10, 2024

Why does this PR need

When I use gcs_bucket parameter, embulk-output-bigquery raise error.

Package version: embulk-output-bigquery (0.7.1), google-apis-storage_v1 (0.40.0)
JRuby version: JRuby runtime 9.4.8.0

error log
2024-07-10 23:24:55.290 +0900 [INFO] (0030:task-0000): embulk-output-bigquery: create /home/kashira/bq-batch/temp/embulk/embulk_output_bigquery_20240710-623778-j7h8sg.623778.4016.csv.gz
2024-07-10 23:24:55.366 +0900 [INFO] (0001:transaction): {done:  1 / 1, running: 0}
2024-07-10 23:24:55.372 +0900 [INFO] (0001:transaction): embulk-output-bigquery: Insert object... /home/kashira/bq-batch/temp/embulk/embulk_output_bigquery_20240710-623778-j7h8sg.623778.4016.csv.gz => pixiv-1:gs://pixiv_temp_embulk_upload/77a33aeb-b0e0-4db1-b869-7b47c560520f
2024-07-10 23:24:55.375 +0900 [WARN] (0001:transaction): embulk-output-bigquery: timeout_sec is deprecated in google-api-ruby-client >= v0.11.0. Use read_timeout_sec instead
2024-07-10 23:24:55.386 +0900 [INFO] (0001:transaction): embulk-output-bigquery: Delete table... pixiv-1:0_temp.LOAD_TEMP_6f272e03_0e00_4f37_b047_0ab4210a18bb_0_temp__example_upload
2024-07-10 23:24:55.749 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete /home/kashira/bq-batch/temp/embulk/embulk_output_bigquery_20240710-623778-j7h8sg.623778.4016.csv.gz
2024-07-10 23:24:57.897 +0900 [INFO] (0001:cleanup): BUNDLE_GEMFILE is being set: "/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/Gemfile"
2024-07-10 23:24:57.902 +0900 [INFO] (0001:cleanup): Gem's home and path are being cleared.
2024-07-10 23:24:59.662 +0900 [INFO] (0001:cleanup): Loaded JRuby runtime 9.4.8.0
2024-07-10 23:24:59.755 +0900 [INFO] (0001:cleanup): Loaded plugin embulk-input-mysql (0.13.2)
/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/googleauth-1.11.0/lib/googleauth/helpers/connection.rb:27: warning: attribute accessor as module_function
2024-07-10 23:25:03.596 +0900 [INFO] (0001:cleanup): Loaded plugin embulk-output-bigquery (0.7.1)
org.embulk.exec.PartialExecutionException: org.jruby.exceptions.ArgumentError: (ArgumentError) wrong number of arguments (given 3, expected 1..2)
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:580)
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:36)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
at org.embulk.spi.ExecInternal.doWith(ExecInternal.java:26)
at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:278)
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:288)
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:153)
at org.embulk.cli.EmbulkRun.runInternal(EmbulkRun.java:115)
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:24)
at org.embulk.cli.Main.main(Main.java:53)
Caused by: org.jruby.exceptions.ArgumentError: (ArgumentError) wrong number of arguments (given 3, expected 1..2)
at RUBY.insert_object(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/google-apis-storage_v1-0.40.0/lib/google/apis/storage_v1/service.rb:2772)
at RUBY.insert_object(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/embulk-output-bigquery-0.7.1/lib/embulk/output/bigquery/gcs_client.rb:84)
at RUBY.with_network_retry(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/embulk-output-bigquery-0.7.1/lib/embulk/output/bigquery/google_client.rb:51)
at RUBY.insert_object(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/embulk-output-bigquery-0.7.1/lib/embulk/output/bigquery/gcs_client.rb:84)
at RUBY.insert_objects(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/embulk-output-bigquery-0.7.1/lib/embulk/output/bigquery/gcs_client.rb:103)
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1981)
at org.jruby.RubyEnumerable.each_with_index(org/jruby/RubyEnumerable.java:1214)
at RUBY.insert_objects(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/embulk-output-bigquery-0.7.1/lib/embulk/output/bigquery/gcs_client.rb:101)
at RUBY.transaction(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/embulk-output-bigquery-0.7.1/lib/embulk/output/bigquery.rb:375)
at RUBY.transaction(/mnt/ssd1/home/kashira/bq-batch/embulk/bundle/jruby/3.1.0/gems/embulk-0.11.4-java/lib/embulk/output_plugin.rb:64)

Error: org.jruby.exceptions.ArgumentError: (ArgumentError) wrong number of arguments (given 3, expected 1..2)
Traceback (most recent call last):
  File "/mnt/ssd1/home/kashira/bq-batch/./bq_batch/updates/pixiv-1/0_temp/example_upload.py", line 23, in <module>
    Update().run_from_cli()
  File "/home/kashira/bq-batch/bq_batch/common/task/base.py", line 47, in run_from_cli
    self._run(_param)
  File "/home/kashira/bq-batch/bq_batch/common/task/base.py", line 59, in _run
    self.main()
  File "/mnt/ssd1/home/kashira/bq-batch/./bq_batch/updates/pixiv-1/0_temp/example_upload.py", line 19, in main
    self.run_embulk_upload()
  File "/home/kashira/bq-batch/bq_batch/common/updates/__init__.py", line 57, in run_embulk_upload
    EmbulkClient().run_from_config(c, env)
  File "/home/kashira/bq-batch/bq_batch/common/embulk/client.py", line 134, in run_from_config
    self.run_from_str(self.convert_config_to_liquid(config), env)
  File "/home/kashira/bq-batch/bq_batch/common/embulk/client.py", line 145, in run_from_str
    self.run(f.name, env)
  File "/home/kashira/bq-batch/bq_batch/common/embulk/client.py", line 80, in run
    raise RuntimeError(
RuntimeError: Embulk command failed (code=1): (ArgumentError) wrong number of arguments (given 3, expected 1..2)

Copy link
Member

@hiroyuki-sato hiroyuki-sato left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kashira202111 Thanks. LGTM👍

@joker1007 Could you take a look at this PR when you get a chance?
This PR fixes https://www.ruby-lang.org/en/news/2019/12/12/separation-of-positional-and-keyword-arguments-in-ruby-3-0/

Due to other reasons, CI doesn't work properly. I'll fix it on another PR.
(And I think this part tests does not exist yet)

@hiroyuki-sato hiroyuki-sato requested a review from joker1007 July 11, 2024 12:55
@hiroyuki-sato
Copy link
Member

@kashira202111 Fix CI in #166. Could you rebase this PR to check CI?

@kashira202111 kashira202111 force-pushed the kashira/fix_bug_about_gcs_client branch from b63043b to 2c1c9f6 Compare July 19, 2024 07:11
@kashira202111
Copy link
Author

@hiroyuki-sato Thank you. I rebased this PR !

@hiroyuki-sato
Copy link
Member

@joker1007 CI passed. Please take a look when you have time.

@hiroyuki-sato hiroyuki-sato merged commit c197ecf into embulk:master Jul 21, 2024
2 checks passed
@hiroyuki-sato
Copy link
Member

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

3 participants