pgfutter to load csv gives: driver bad connection. db.Ping() not supported?

[spreeker]

I tried to use pgfutter to load some csv data which uses database.sql sql driver. after disabling ssl which was the first hurdle in connecting. I successfully started a dolgres server instance.

started an instance, connected using psql.

doltgresql -P 9999

tried to run a simple tool to quickly load some csv in de database.

pgfutter --port 9999 --dbname doltgres --username doltgres csv -d ',' v20230703_v2_csv.csv

I get the following error and now I don't know what to do to fix this. A different postgress database driver?

driver: bad connection

I'm super excited to discover dolt!

[timsehn]

Thanks for the issue! We haven't tested with pgfutter yet but @Hydrocharged will see what's up today and give you an explanation.

[Hydrocharged]

Thank you for the issue! I'll see if I can pinpoint exactly what's going wrong here.

[Hydrocharged]

I've looked into this, and I can't seem to replicate the error message you're experiencing. On top of that, I also did not need to disable SSL. I tested this against v0.1.0 (which is also the tip of main as of this comment). For me, it appears to complete successfully, however I assume that pgfutter is swallowing an error, as DoltgreSQL doesn't support schemas just yet.

To try and pinpoint why you're getting this particular error, my guess is either one of two things: your OS (due to the SSL issues), or the CSV you're attempting to load. Can you provide which operating system you're using? In addition, can you provide a CSV that I could test? It does not have to be the exact one in case it has sensitive data, any CSV that gives you the same error will suffice. This will allow us to find other potential errors lurking within our program.

[spreeker]

I can fork pgfutter and not make it use a schema..

pgfutter allows for any csv to be loaded into postgres. pgfutter is german for pg postgres futter is food. Food for postgres and is my goto tool to load up some data into postgres and start working on the raw data.

Here is an example csv with energy data from a city in the netherlands.

alkmaar csv

I'm using ubuntu 23.04 just checketout the dolt source code.

[spreeker]

database connecting error using user=doltgres dbname=doltgres password='doltgres' host=localhost port=9999
pq: SSL is not enabled on the server

attempt 2.

database connecting error using user=doltgres dbname=doltgres password='doltgres' host=localhost port=9999 sslmode=disable connect_timeout=5
driver: bad connection

The code triggering the error in pgfutter.

func connect(connStr string, importSchema string) (*sql.DB, error) {                  
          db, err := sql.Open("postgres", connStr)
          if err != nil {
                  return db, err 
          }

[spreeker]

I made minimal golang example and can reproduce the error. Could it be Ping is not supported yet?

package main

import (
	"database/sql"
	"fmt"

	_ "github.com/lib/pq"
)

func connect(connStr string) (*sql.DB, error) {
	fmt.Println(connStr)
	db, err := sql.Open("postgres", connStr)

	if err != nil {
		fmt.Println(err)
	}

	err = db.Ping()

	if err == nil {
		fmt.Println("successfully connected..")
	} else {
		fmt.Println(err)
	}

	fmt.Println("the end")

	return db, err
}

func main() {
	connStr := "user=doltgres dbname=doltgres password=doltgres host=localhost port=9999 sslmode=disable"
	connect(connStr)
}

./test
user=doltgres dbname=doltgres password='doltgres' host=localhost port=9999 sslmode=disable
driver: bad connection
the end

[Hydrocharged]

Your deduction is correct! It's not necessarily that we don't support Ping(), there's a bug where we always assume that a successful parse produces an AST. It looks like Ping() is sending ;, which is a "complete" statement, but it doesn't result in an AST, which then causes a panic. Great find! We'll get this fixed in the next release.

[Hydrocharged]

Upon merging the tagged PR, this issue will close. I've made a new issue tracking the next blocker, but since it's a bit more general than just for pgfutter, I decided to make it a separate issue and close this one.