Add template_table config

[joker1007]

If template_table is set, plugin fetches table schema from it instead of schema_path.

the use case is like templateSuffix.
If base table schema is changed, next created table uses new schema.

And, I refactored some methods for this feature.
How about this?

[sakama]

This feature and refactored code looks good to me.
I had wanted to implement like this feature #1.

I'll test at my environment, so please give me a little time.

[frsyuki]

@joker1007 @sakama why don't you create BigTable's schema file from Embulk's schema? Output plugins can know schema because input plugin already decided it.

[frsyuki]

I mean that template_table seems a good option. Other outputs (such as JDBC) also can have it. But I wonder that's the default.

[joker1007]

@frsyuki
If both schema_path and template_table is empty, plugins guess bigquery schema from Input schema. is this right?
It is better. I think so.
If I have time to spare, I want to implement it and submit PR.

[sakama]

@frsyuki @joker1007
Plan had been exists. It requires some works.

• Need to change plugin type from FileOutputPlugin to OutputPlugin
  • In my understanding, FileOutputPlugin can't get input schema.
  • But this is helpfull to support data splitting that is needed to satisfy BigQuery's Quota Policy Split file each 4GB for BigQuery Quota Policy #6.
• Need to have CSVFormatter/JsonFormatter/GzipFileEncoder at plugin itself.
  • We can use CSVFormatter and GzipEncoder that is implemented at embulk-standards.
  • Current JsonFormatter(embulk-formatter-jsonl) is written by JRuby. Java based JsonFormatter should become faster than them.

[joker1007]

Fmm.. it is slightly hard work.

[sakama]

I merged. Thanks!