mysqlの読み込みについて

[vktm7]

目的：mysqlからinputしたデータをBigQueryにOutputしたい

embulk preview rds-mysql-to-bq/target.yaml.liquid を叩くと、以下のようなエラーがでます。

org.embulk.config.ConfigException: InputPlugin 'mysql' is not found.
Maven repository specified is not found at "/home/ec2-user/.embulk/lib/m2/repository" ("/home/ec2-user/.embulk/lib/m2/repository").
Failed in initializing JRuby runtime.

エラー含め全文

2024-02-25 11:27:49.416 +0900 [INFO] (main): embulk_home is set by the location of embulk.properties found in: /home/ec2-user/.embulk
2024-02-25 11:27:49.423 +0900 [INFO] (main): m2_repo is set as a sub directory of embulk_home: /home/ec2-user/.embulk/lib/m2/repository
2024-02-25 11:27:49.423 +0900 [INFO] (main): gem_home is set as a sub directory of embulk_home: /home/ec2-user/.embulk/lib/gems
2024-02-25 11:27:49.423 +0900 [INFO] (main): gem_path is set empty.
2024-02-25 11:27:49.423 +0900 [DEBUG] (main): Embulk system property "default_guess_plugin" is set to: "gzip,bzip2,json,csv"
2024-02-25 11:27:52.206 +0900 [INFO] (main): Gem's home and path are set by system configs "gem_home": "/home/ec2-user/.embulk/lib/gems", "gem_path": ""
2024-02-25 11:27:52.210 +0900 [INFO] (main): Loaded JRuby runtime 9.4.5.0
2024-02-25 11:27:53.385 +0900 [INFO] (main): Started Embulk v0.11.1
2024-02-25 11:27:55.737 +0900 [INFO] (0001:preview): Gem's home and path are set by system configs "gem_home": "/home/ec2-user/.embulk/lib/gems", "gem_path": ""

org.embulk.config.ConfigException: InputPlugin 'mysql' is not found.
Maven repository specified is not found at "/home/ec2-user/.embulk/lib/m2/repository" ("/home/ec2-user/.embulk/lib/m2/repository").
Failed in initializing JRuby runtime.
	at org.embulk.plugin.PluginManager.buildPluginNotFoundException(PluginManager.java:155)
	at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:99)
	at org.embulk.spi.ExecSessionInternal.newPlugin(ExecSessionInternal.java:358)
	at org.embulk.spi.ExecInternal.newPlugin(ExecInternal.java:50)
	at org.embulk.exec.PreviewExecutor.newInputPlugin(PreviewExecutor.java:82)
	at org.embulk.exec.PreviewExecutor.doPreview(PreviewExecutor.java:91)
	at org.embulk.exec.PreviewExecutor.access$000(PreviewExecutor.java:31)
	at org.embulk.exec.PreviewExecutor$1.run(PreviewExecutor.java:69)
	at org.embulk.exec.PreviewExecutor$1.run(PreviewExecutor.java:66)
	at org.embulk.spi.ExecInternal.doWith(ExecInternal.java:26)
	at org.embulk.exec.PreviewExecutor.preview(PreviewExecutor.java:66)
	at org.embulk.EmbulkEmbed.preview(EmbulkEmbed.java:267)
	at org.embulk.EmbulkRunner.previewInternal(EmbulkRunner.java:212)
	at org.embulk.EmbulkRunner.preview(EmbulkRunner.java:104)
	at org.embulk.cli.EmbulkRun.runInternal(EmbulkRun.java:105)
	at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:24)
	at org.embulk.cli.Main.main(Main.java:53)
	Suppressed: org.embulk.plugin.PluginSourceNotMatchException
		at org.embulk.plugin.BuiltinPluginSource.newPlugin(BuiltinPluginSource.java:244)
		at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:76)
		... 15 more
	Suppressed: org.embulk.plugin.PluginSourceNotMatchException: org.embulk.deps.maven.MavenRepositoryNotFoundException: Maven repository specified is not found at "/home/ec2-user/.embulk/lib/m2/repository" ("/home/ec2-user/.embulk/lib/m2/repository").
		at org.embulk.plugin.maven.MavenPluginRegistry.findPluginPaths(MavenPluginRegistry.java:100)
		at org.embulk.plugin.maven.MavenPluginRegistry.lookup(MavenPluginRegistry.java:63)
		at org.embulk.plugin.maven.MavenPluginSource.newPlugin(MavenPluginSource.java:33)
		at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:82)
		... 15 more
	Caused by: org.embulk.deps.maven.MavenRepositoryNotFoundException: Maven repository specified is not found at "/home/ec2-user/.embulk/lib/m2/repository" ("/home/ec2-user/.embulk/lib/m2/repository").
		at org.embulk.deps.maven.MavenArtifactFinderImpl.<init>(MavenArtifactFinderImpl.java:46)
		at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
		at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
		at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
		at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
		at org.embulk.plugin.maven.MavenArtifactFinder.create(MavenArtifactFinder.java:13)
		at org.embulk.plugin.maven.MavenPluginRegistry.findPluginPaths(MavenPluginRegistry.java:98)
		... 18 more
	Caused by: java.nio.file.NoSuchFileException: /home/ec2-user/.embulk/lib/m2/repository
		... 25 more
	Suppressed: org.embulk.plugin.PluginSourceNotMatchException
		at org.embulk.plugin.SelfContainedPluginRegistry.search(SelfContainedPluginRegistry.java:77)
		at org.embulk.plugin.SelfContainedPluginRegistry.lookup(SelfContainedPluginRegistry.java:64)
		at org.embulk.plugin.SelfContainedPluginSource.newPlugin(SelfContainedPluginSource.java:44)
		at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:88)
		... 15 more
	Suppressed: org.embulk.plugin.PluginSourceNotMatchException: Failed to set PluginClassLoaderFactory as $temporary_internal_plugin_class_loader_factory__ in JRuby.
		at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:76)
		at org.embulk.plugin.PluginManager.newPlugin(PluginManager.java:94)
		... 15 more
	Caused by: org.embulk.jruby.JRubyRuntimeException: Failed in initializing JRuby runtime.
		at org.embulk.jruby.JRubyInitializer.initialize(JRubyInitializer.java:171)
		at org.embulk.jruby.LazyScriptingContainerDelegate.getInitialized(LazyScriptingContainerDelegate.java:325)
		at org.embulk.jruby.LazyScriptingContainerDelegate.put(LazyScriptingContainerDelegate.java:295)
		at org.embulk.jruby.JRubyPluginSource.newPlugin(JRubyPluginSource.java:74)
		... 16 more
	Caused by: org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- embulk/version
		at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:134)
		at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1296)
		at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1289)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.embulk.jruby.ScriptingContainerDelegateImpl.runScriptlet(ScriptingContainerDelegateImpl.java:841)
		at org.embulk.jruby.JRubyInitializer.initialize(JRubyInitializer.java:139)
		... 19 more
	Caused by: org.jruby.exceptions.LoadError: (LoadError) no such file to load -- embulk/version
		at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1071)
		at RUBY.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85)
		at RUBY.<main>(<script>:1)

Error: InputPlugin 'mysql' is not found.
Maven repository specified is not found at "/home/ec2-user/.embulk/lib/m2/repository" ("/home/ec2-user/.embulk/lib/m2/repository").
Failed in initializing JRuby runtime.

target.yaml.liquid

in:
  type: mysql
  host: {{ env.MYSQL_HOST_NAME }}
  user: {{ env.MYSQL_USER }}
  password: {{ env.MYSQL_PASSWORD }}
  database: testdb
  table: test

out:
  type: bigquery
  auth_method: json_key
  json_keyfile: ./credentials/bq_project.json
  path_prefix: tmp
  file_ext: .csv.gz
  source_format: CSV
  project: testproject
  dataset: test_dataset
  auto_create_table: true
  table: test
  formatter: {type: csv, charset: UTF-8, delimiter: ',', header_line: false}
  encoders:
  - {type: gzip}

mysqlはembulk-input-mysql (0.13.2 java)が入っています。
Mavenに関してはこのエラーを見るまで何も設定していなかったため、こちらを確認しましたがどのように指定をすれば良いか分からない状態です。

embulk.propertiesは以下のように指定しています。

jruby=file:///home/ec2-user/jruby-complete-9.4.5.0.jar
plugins.output.sftp=maven:org.embulk:sftp:0.3.1 

環境

Linux
Embulk 0.11.1
openjdk version “1.8.0_382”

他に必要な情報があれば追加します。検索 & Tryをしましたが自力で解決できそうになく、お助けいただきたいです 🙏

参考：https://github.com/orgs/embulk/discussions/12

[vktm7]

再度、https://github.com/orgs/embulk/discussions/12 を参考に設定漏れがないか確認したところ、JRuby設定項目の

embulk gem install embulk -v 0.11.2 # Must be the same Embulk version
こちらの設定が漏れていました。

自身のEmbulkと同じバージョンをinstallして実行をしたらpreviewができるようになりました！

解決したコマンド：
embulk gem install embulk -v 0.11.1

過去の質問等大変参考になりました。ありがとうございます。

[hiroyuki-sato]

自己解決されたようでなによりです。

このメッセージが、embulk.gem がインストールされていないときに出るメッセージとなっております。
今後の参考になれば幸いです。

	Caused by: org.jruby.exceptions.LoadError: (LoadError) no such file to load -- embulk/version
		at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1071)
		at RUBY.require(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:85)
		at RUBY.<main>(<script>:1)

余談ですが、output-bigqueryは0.7.0以上をご利用ください。こちらがv0.11対応になっております。

[vktm7]

ありがとうございます！今後の参考にさせていただきます。

output-bigqueryは0.7.0以上をご利用ください

初心者のため、このような情報はとても助かります🙏

また別でエラーがでているのですが、もしご存知であれば教えていただきたいです。（現在自身でもEmbulk Google::Apis::ClientErrorで検索中です）

Error: org.jruby.exceptions.StandardError: (Error) failed to load tmp.3111919.4016.csv.gz to piyopiyoworld:piyopiyoinc_dump.LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions in us/eu,
response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415", :error_class=>Google::Apis::ClientError}

Details

embulk run subscription.yaml.liquidの結果

2024-02-25 15:04:55.323 +0900 [ERROR] (Ruby-0-Thread-2: /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:153): embulk-output-bigquery: insert_job(piyopiyoworld, {:job_reference=>{:project_id=>"piyopiyoworld", :job_id=>"embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415"}, :configuration=>{:load=>{:destination_table=>{:project_id=>"piyopiyoworld", :dataset_id=>"piyopiyoinc_dump", :table_id=>"LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions"}, :schema=>{:fields=>[{:name=>"id", :type=>"INTEGER"}, {:name=>"user_id", :type=>"INTEGER"}, {:name=>"name", :type=>"STRING"}, {:name=>"quantity", :type=>"INTEGER"}, {:name=>"price", :type=>"INTEGER"}, {:name=>"expired_at", :type=>"TIMESTAMP"}, {:name=>"status", :type=>"INTEGER"}, {:name=>"membership_type", :type=>"STRING"}, {:name=>"is_bank", :type=>"INTEGER"}, {:name=>"subscription_plan_master", :type=>"STRING"}, {:name=>"created_at", :type=>"TIMESTAMP"}, {:name=>"updated_at", :type=>"TIMESTAMP"}]}, :write_disposition=>"WRITE_APPEND", :source_format=>"CSV", :max_bad_records=>0, :field_delimiter=>",", :encoding=>"UTF-8", :ignore_unknown_values=>false, :allow_quoted_newlines=>false}}}, {:upload_source=>"tmp.3111919.4016.csv.gz", :content_type=>"application/octet-stream"}), response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415", :error_class=>Google::Apis::ClientError}
warning: thread "Ruby-0-Thread-2: /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:153" terminated with exception (report_on_exception is true):
Embulk::Output::Bigquery::Error: failed to load tmp.3111919.4016.csv.gz to piyopiyoworld:piyopiyoinc_dump.LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions in us/eu, response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415", :error_class=>Google::Apis::ClientError}
              load at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:236
    with_job_retry at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:59
              load at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:169
  load_in_parallel at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:157
2024-02-25 15:04:55.335 +0900 [INFO] (0001:transaction): embulk-output-bigquery: Delete table... piyopiyoworld:piyopiyoinc_dump.LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions
2024-02-25 15:04:55.437 +0900 [ERROR] (Ruby-0-Thread-3: /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:153): embulk-output-bigquery: insert_job(piyopiyoworld, {:job_reference=>{:project_id=>"piyopiyoworld", :job_id=>"embulk_load_job_813efeab-3d09-4cfa-bba3-d10e16340b93"}, :configuration=>{:load=>{:destination_table=>{:project_id=>"piyopiyoworld", :dataset_id=>"piyopiyoinc_dump", :table_id=>"LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions"}, :schema=>{:fields=>[{:name=>"id", :type=>"INTEGER"}, {:name=>"user_id", :type=>"INTEGER"}, {:name=>"name", :type=>"STRING"}, {:name=>"quantity", :type=>"INTEGER"}, {:name=>"price", :type=>"INTEGER"}, {:name=>"expired_at", :type=>"TIMESTAMP"}, {:name=>"status", :type=>"INTEGER"}, {:name=>"membership_type", :type=>"STRING"}, {:name=>"is_bank", :type=>"INTEGER"}, {:name=>"subscription_plan_master", :type=>"STRING"}, {:name=>"created_at", :type=>"TIMESTAMP"}, {:name=>"updated_at", :type=>"TIMESTAMP"}]}, :write_disposition=>"WRITE_APPEND", :source_format=>"CSV", :max_bad_records=>0, :field_delimiter=>",", :encoding=>"UTF-8", :ignore_unknown_values=>false, :allow_quoted_newlines=>false}}}, {:upload_source=>"tmp.3111919.4020.csv.gz", :content_type=>"application/octet-stream"}), response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_813efeab-3d09-4cfa-bba3-d10e16340b93", :error_class=>Google::Apis::ClientError}
warning: thread "Ruby-0-Thread-3: /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:153" terminated with exception (report_on_exception is true):
Embulk::Output::Bigquery::Error: failed to load tmp.3111919.4020.csv.gz to piyopiyoworld:piyopiyoinc_dump.LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions in us/eu, response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_813efeab-3d09-4cfa-bba3-d10e16340b93", :error_class=>Google::Apis::ClientError}
              load at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:236
    with_job_retry at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:59
              load at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:169
  load_in_parallel at /home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:157
2024-02-25 15:04:55.625 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete tmp.3111919.4016.csv.gz
2024-02-25 15:04:55.631 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete tmp.3111919.4020.csv.gz
2024-02-25 15:05:02.460 +0900 [INFO] (0001:cleanup): Gem's home and path are set by system configs "gem_home": "/home/ec2-user/.embulk/lib/gems", "gem_path": ""
2024-02-25 15:05:05.293 +0900 [INFO] (0001:cleanup): Loaded JRuby runtime 9.4.5.0
2024-02-25 15:05:05.492 +0900 [INFO] (0001:cleanup): Loaded plugin embulk-input-mysql (0.13.2)
/home/ec2-user/.embulk/lib/gems/gems/googleauth-1.11.0/lib/googleauth/helpers/connection.rb:27: warning: attribute accessor as module_function
/home/ec2-user/.embulk/lib/gems/gems/concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_int
/home/ec2-user/.embulk/lib/gems/gems/concurrent-ruby-1.2.3/lib/concurrent-ruby/concurrent/executor/java_thread_pool_executor.rb:13: warning: method redefined; discarding old to_f
2024-02-25 15:05:18.588 +0900 [INFO] (0001:cleanup): Loaded plugin embulk-output-bigquery (0.7.0)
org.embulk.exec.PartialExecutionException: org.jruby.exceptions.StandardError: (Error) failed to load tmp.3111919.4016.csv.gz to piyopiyoworld:piyopiyoinc_dump.LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions in us/eu, response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415", :error_class=>Google::Apis::ClientError}
	at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
	at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:580)
	at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:36)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
	at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
	at org.embulk.spi.ExecInternal.doWith(ExecInternal.java:26)
	at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
	at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:278)
	at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:288)
	at org.embulk.EmbulkRunner.run(EmbulkRunner.java:153)
	at org.embulk.cli.EmbulkRun.runInternal(EmbulkRun.java:108)
	at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:24)
	at org.embulk.cli.Main.main(Main.java:53)
Caused by: org.jruby.exceptions.StandardError: (Error) failed to load tmp.3111919.4016.csv.gz to piyopiyoworld:piyopiyoinc_dump.LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions in us/eu, response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415", :error_class=>Google::Apis::ClientError}
	at RUBY.load(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:236)
	at RUBY.with_job_retry(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:59)
	at RUBY.load(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:169)
	at RUBY.load_in_parallel(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:157)
Caused by: org.jruby.exceptions.StandardError: (ClientError) notFound: Not found: Job piyopiyoworld:embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415
	at RUBY.check_status(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:244)
	at RUBY.check_status(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/api_command.rb:135)
	at RUBY.process_response(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:207)
	at RUBY.execute_once(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:326)
	at RUBY.do_retry(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:131)
	at RUBY.retriable(/home/ec2-user/.embulk/lib/gems/gems/retriable-3.1.2/lib/retriable.rb:61)
	at org.jruby.RubyFixnum.times(org/jruby/RubyFixnum.java:308)
	at RUBY.retriable(/home/ec2-user/.embulk/lib/gems/gems/retriable-3.1.2/lib/retriable.rb:56)
	at RUBY.do_retry(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:128)
	at RUBY.retriable(/home/ec2-user/.embulk/lib/gems/gems/retriable-3.1.2/lib/retriable.rb:61)
	at org.jruby.RubyFixnum.times(org/jruby/RubyFixnum.java:308)
	at RUBY.retriable(/home/ec2-user/.embulk/lib/gems/gems/retriable-3.1.2/lib/retriable.rb:56)
	at RUBY.do_retry(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:118)
	at RUBY.execute(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/http_command.rb:109)
	at RUBY.execute_or_queue_command(/home/ec2-user/.embulk/lib/gems/gems/google-apis-core-0.14.0/lib/google/apis/core/base_service.rb:477)
	at RUBY.get_job(/home/ec2-user/.embulk/lib/gems/gems/google-api-client-0.53.0/generated/google/apis/bigquery_v2/service.rb:380)
	at RUBY.wait_load(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:322)
	at RUBY.with_network_retry(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/google_client.rb:51)
	at RUBY.wait_load(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:322)
	at RUBY.load(/home/ec2-user/.embulk/lib/gems/gems/embulk-output-bigquery-0.7.0/lib/embulk/output/bigquery/bigquery_client.rb:229)
	... 3 more

Error: org.jruby.exceptions.StandardError: (Error) failed to load tmp.3111919.4016.csv.gz to piyopiyoworld:piyopiyoinc_dump.LOAD_TEMP_752ce3d0_60e8_4e4d_9459_56dabb6e8e93_subscriptions in us/eu, response:{:status_code=>404, :message=>"notFound: Not found: Job piyopiyoworld:embulk_load_job_d4485a10-6fb5-461f-8ee3-c57af3ba2415", :error_class=>Google::Apis::ClientError}

[hiroyuki-sato]

bqのロケーションの設定がないとかありますか？ (ファイルが見つからないってエラーみたいですね。)未指定の場合、usになります。

  location: asia-northeast1

次回で結構ですので、件名と違う内容の時は新しいトピックを作成していただく方がよいように思います。
(後で同じ問題の方が探しやすくなると思いますので、)

[vktm7]

locationを入れたところ解決し、無事にBQにデータをexport完了させることができました！ありがとうございます。
お二人とも、大変助かりました。

次回で結構ですので、件名と違う内容の時は新しいトピックを作成していただいたほうがいいと思います。

たしかにその通りでした。解決コメントの選択も一つしかできないため、次回以降1件につき1discussionでトピックを立てさせていただきます。

本件に関しては、私からの質問は以上にさせていただきます。ご対応ありがとうございました🙏

[hiroyuki-sato]

解決できてなによりです。(bqの動作確認していただきありがとうございます。)
解決された場合は、下にClose discussionというのがあると思うのでそれも押していただけると助かります。
(そんなボタンないという時はおっしゃってください)