Road to v0.2

[matthewmturner]

This issue for setting the specific issues we would like completed before a v0.2 release. My hope is that there can be a release by end of Sept / Beginning of October.

• Querys that return large # of results can freeze / crash the app #137
• To investigate: Is DDL working correctly? #138
• Add --flightsql paramter for -c and -f #128
• (REQUIRED) Add tests for configuration system #144
• (REQUIRED) Implement select * from 'foo.parquet' support #125
• Add back DDL mode #153
• (REQUIRED) Create / add duckdb metadata functions #148
• (REQUIRED) Update to DataFusion 42.0.0 #158
• Add syntax highlighting to config #159
• Update from datafusionrc to ddl in config dir #166
• (REQUIRED) Figure out ddl mode executed / error UI #167
• (REQUIRED) DDL should optionally be available from CLI #170
• Add serve subcommand for flightsql feature #168
• Setup localstack for testing s3 #173
• Add ui to flightsql tab when unable to connect #175
• Test flightsql error cases #177
• (REQUIRED) Benchmark from CLI
• (REQUIRED) Analyze from CLI
• DDL mode isnt loading DDL file correctly now #213

[matthewmturner]

@alamb let me know if there is anything you would like to add

[alamb]

I think we should consider adding #148 and #158 as well

[matthewmturner]

So I think it turns out that the DDL issue I have is a bug with datafusion registering external tables to a schema.

I have this in my ~/.datafusion/.datafusionrc`

CREATE SCHEMA staging;

CREATE TABLE staging.foo AS VALUES (1);
CREATE EXTERNAL TABLE staging.min_aggs STORED AS PARQUET LOCATION 'ny2://atlas/sip/aggregates/minute_by_ticker_monthly_v2/year_month=2015-08/data.01.parquet';

But when I run SHOW TABLES I get the following:

Notice how the schema isnt picked up for the the external table. I dont see anything in the docs for CREATE EXTERNAL TABLE that would suggest this shouldnt work but maybe im misunderstanding.

@alamb do you have a view on this? If you agree its a bug in datafusion then i can create a ticket there.

[alamb]

I agree it is a datafusion bug. I will file a ticket

[alamb]

Filed apache/datafusion#12607