🧪 Connecting to ClickHouse Cloud (#167)

* :test-tube: Connecting to ClickHouse Cloud Signed-off-by: Julio Jimenez <[email protected]> * testing complete Signed-off-by: Julio Jimenez <[email protected]> * link to nyc taxi dataset Signed-off-by: Julio Jimenez <[email protected]> --------- Signed-off-by: Julio Jimenez <[email protected]>
juliojimenez · Mar 11, 2023 · 8485869 · 8485869
1 parent 3bc412f
commit 8485869
Show file tree

Hide file tree

Showing 8 changed files with 162 additions and 26 deletions.
diff --git a/README.md b/README.md
@@ -27,6 +27,7 @@ Common Lisp ClickHouse Client Library
 - [Examples](#examples)
   - [Connection to a local database](#connection-to-a-local-database)
   - [Query](#query)
+  - [Connecting to ClickHouse Cloud](#connecting-to-clickhouse-cloud)
 - [Bugs, Features, and Vulnerabilities Reporting](#bugs-features-and-vulnerabilities-reporting)
 
 ## Install
@@ -36,9 +37,9 @@ Common Lisp ClickHouse Client Library
 clickhouse-cl is on [Ultralisp.org](https://ultralisp.org)!
 
 ```
-CL-USER> (ql-dist:install-dist "http://dist.ultralisp.org/" :prompt nil)
+> (ql-dist:install-dist "http://dist.ultralisp.org/" :prompt nil)
 ...
-CL-USER> (ql:quickload :clickhouse)
+> (ql:quickload :clickhouse)
 ...
 ```
 
@@ -54,22 +55,21 @@ Clone this repo wherever your quicklisp `local-projects` folder is configured.
 Some dependencies are on [Ultralisp.org](https://ultralisp.org/), make sure you have it...
 
 ```lisp
-CL-USER> (ql-dist:install-dist "http://dist.ultralisp.org/" :prompt nil)
+> (ql-dist:install-dist "http://dist.ultralisp.org/" :prompt nil)
 ...
 ```
 
 In the emacs SLIME REPL or SBCL, load clickhouse-cl with...
 
 ```lisp
-CL-USER> (ql:quickload :clickhouse)
+> (ql:quickload :clickhouse)
 To load "clickhouse":
   Load 1 ASDF system:
     clickhouse
 ; Loading "clickhouse"
 [package clickhouse]
 
 (:CLICKHOUSE)
-CL-USER>
 ```
 
 ### Releases
@@ -92,7 +92,7 @@ If that doesn't work, checkout this [StackExchange](https://superuser.com/questi
 | port | y | 8123 | Database port, i.e. 8443 or 8123 |
 | ssl | y | nil | SSL option, boolean, t or nil. |
 | username | y | default | Database username |
-| password | y | | Database password |
+| password | y | nil | Database password |
 
 ### Usage
 
@@ -113,11 +113,10 @@ Binding an instance of `database`.
 Reading and setting a slot.
 
 ```lisp
-CL-USER> (ch::password *db*)
+> (ch::password *db*)
 "1amAsecretPassWord"
-CL-USER> (setf (ch::password *db*) "chang3m3plea5e")
+> (setf (ch::password *db*) "chang3m3plea5e")
 "chang3m3plea5e"
-CL-USER>
 ```
 ### Methods
 
@@ -126,14 +125,14 @@ CL-USER>
 ch:ping *obj* :ping *bool* :console *bool*
 
 ```lisp
-CL-USER> (ch::ping *db*)
+> (ch::ping *db*)
 "Ok."
 ```
 
 The `:ping t` keyword parameter explicitly calls the instance `/ping` endpoint.
 
 ```lisp
-CL-USER>  (ch::ping *db* :ping t)
+> (ch::ping *db* :ping t)
 "Ok."
 ```
 
@@ -142,7 +141,7 @@ CL-USER>  (ch::ping *db* :ping t)
 ch:replicas-status *obj* :console *bool* :verbose *bool*
 
 ```lisp
-CL-USER> (ch::replicas-status *db*)
+> (ch::replicas-status *db*)
 "Ok."
 ```
 
@@ -151,7 +150,7 @@ CL-USER> (ch::replicas-status *db*)
 ch:query *obj* *query* :console *bool* :no-format *bool* :timeout *int*
 
 ```lisp
-CL-USER> (ch::query *db* "SELECT 1")
+> (ch::query *db* "SELECT 1")
 "1"
 ```
 
@@ -160,15 +159,15 @@ CL-USER> (ch::query *db* "SELECT 1")
 All methods can take the keyword parameter `:console t`, providing a cleaner output when interacting directly with the library in the REPL.
 
 ```lisp
-CL-USER> (ch:query *db* "SHOW DATABASES")
+> (ch:query *db* "SHOW DATABASES")
 "INFORMATION_SCHEMA
 default
 information_schema
 system"
 ```
 
 ```lisp
-CL-USER> (ch:query *db* "SHOW DATABASES" :console t)
+> (ch:query *db* "SHOW DATABASES" :console t)
 INFORMATION_SCHEMA
 default
 information_schema
@@ -258,13 +257,13 @@ Helper function used to access key values in formats that result in a `BOOST-JSO
 ch:jget *obj* *key*
 
 ```lisp
-CL-USER> (defparameter *db* (make-instance 'ch:database))
+> (defparameter *db* (make-instance 'ch:database))
 *DB*
-CL-USER> (defparameter *result* (ch:query *db* "SELECT trip_id, passenger_count FROM trips LIMIT 10 FORMAT JSON"))
+> (defparameter *result* (ch:query *db* "SELECT trip_id, passenger_count FROM trips LIMIT 10 FORMAT JSON"))
 *RESULT*
-CL-USER> *result*
+> *result*
 #<BOOST-JSON:JSON-OBJECT {"meta":#,"data":#,"rows":10,"rows_before_limit_at_least":10,"statistics":#}>
-CL-USER> (ch:jget *result* "rows")
+> (ch:jget *result* "rows")
 10
 T
 ```
@@ -283,7 +282,7 @@ ch:input-parameters query &rest input
 
 ## Examples
 
-### Connection to a local database
+### Connecting to a local database
 
 This would be applicable to a recently [installed](https://clickhouse.com/docs/en/getting-started/quick-start/) database, prior to applying a password and/or adding any users.
 
@@ -297,6 +296,44 @@ This would be applicable to a recently [installed](https://clickhouse.com/docs/e
 (ch:query *db* "SELECT 1")
 ```
 
+### Connecting to ClickHouse Cloud
+
+This example connects to a [ClickHouse Cloud](https://clickhouse.com/cloud) database loaded with the [NYC Taxi](https://clickhouse.com/docs/en/getting-started/example-datasets/nyc-taxi/) dataset.
+
+```
+> (ql:quickload :clickhouse)
+> (defparameter *db* (make-instance 'clickhouse:database
+				                            :host "iqr3flp7yf.us-east-1.aws.clickhouse.cloud"
+				                            :port 8443
+				                            :ssl t
+				                            :username "default"
+				                            :password ")UwB2oL|QQpi"))
+> (ch:query *db* "SELECT count()
+                  FROM nyc_taxi 
+                  FORMAT PrettySpaceNoEscapes" :console t)
+
+  count()
+
+ 20000000
+NIL
+> (ch:query *db* "SELECT 
+                    trip_id,
+                    total_amount,
+                    trip_distance
+                  FROM nyc_taxi
+                  LIMIT 5 
+                  FORMAT PrettySpaceNoEscapes" :console t)
+
+    trip_id   total_amount   trip_distance
+
+ 1199999902          19.56            2.59 
+ 1199999919           10.3             2.4 
+ 1199999944           24.3            5.13 
+ 1199999969           9.95             1.2 
+ 1199999990            9.8            2.17 
+NIL
+```
+
 # Bugs, Features, and Vulnerabilities Reporting
 
 To report bugs, request a feature, or report a security vulnerability, please submit a new [issue](https://github.com/juliojimenez/clickhouse-cl/issues/new/choose).
diff --git a/clickhouse-test.asd b/clickhouse-test.asd
@@ -2,7 +2,7 @@
   :description "clickhouse Tests"
   :author "[email protected]"
   :license  "Apache-2.0"
-  :version "0.0.26"
+  :version "0.0.27"
   :depends-on (#:clickhouse
 	             #:fiveam)
   :components ((:module "t"

diff --git a/clickhouse.asd b/clickhouse.asd
@@ -2,7 +2,7 @@
   :description "Common Lisp ClickHouse Client Library"
   :author "[email protected]"
   :license "Apache-2.0"
-  :version "0.41.0"
+  :version "0.42.0"
   :depends-on (#:boost-json
 	             #:dexador
 	             #:lexer

diff --git a/src/#clickhouse.lisp# b/src/#clickhouse.lisp#
@@ -0,0 +1,91 @@
+(defpackage :clickhouse
+  (:nicknames :ch)
+  (:use :cl)
+  (:import-from :clickhouse.ch-sql-parser
+                :make-query)
+  (:import-from :clickhouse.http
+                :http-get
+                :http-post)
+  (:import-from :clickhouse.utils
+                :prettify
+                :ver)
+  (:import-from :cl-ppcre
+                :regex-replace)
+  (:export :database
+           :input-parameters
+           :jget
+           :ping
+           :query
+           :replicas-status))
+
+(in-package :clickhouse)
+
+(defclass database ()
+  ((host
+    :initarg :host
+    :initform "localhost"
+    :accessor host
+    :documentation "ClickHouse database hostname.")
+   (port
+    :initarg :port
+    :initform 8123
+    :accessor port
+    :documentation "ClickHouse database port, i.e. 8443 or 8123.")
+   (ssl
+    :initarg :ssl
+    :initform nil
+    :accessor ssl
+    :documentation "SSL option, t or nil.")
+   (username
+    :initarg :username
+    :initform "default"
+    :accessor username
+    :documentation "ClickHouse database username, default username is default.")
+   (password
+    :initarg :password
+    :initform nil
+    :accessor password
+    :documentation "Clickhouse database password.")))
+
+(defgeneric ping (obj &key)
+  (:documentation "Pings the database server."))
+
+(defmethod ping ((obj database) &key ping console)
+  "Pings the database server."
+  (with-slots ((h host) (p port) (s ssl)) obj
+    (prettify
+      (if (ver ping)
+	 (http-get h p s "/ping")
+	 (http-get h p s "/"))
+     :console console)))
+
+(defgeneric replicas-status (obj &key)
+  (:documentation "Get replicas status."))
+
+(defmethod replicas-status ((obj database) &key console verbose)
+  "Get replicas status."
+  (with-slots ((h host) (p port) (s ssl)) obj
+    (prettify
+      (cond (verbose (http-get h p s "/replicas_status?verbose=1"))
+            (t       (http-get h p s "/replicas_status")))
+      :console console)))
+
+(defgeneric query (obj query &key)
+  (:documentation "Execute a query."))
+
+(defmethod query ((obj database) query &key console no-format timeout)
+  "Execute a query."
+  (with-slots ((h host) (p port) (s ssl) (u username) (w password)) obj
+    (prettify
+     (http-post h p s u w (make-query query) timeout)
+     :console console :formatting (if no-format nil clickhouse.ch-sql-parser:*format*))))
+
+(defmacro jget (obj key)
+  "Get JSON value."
+  `(boost-json:json-getf ,obj ,key))
+
+(defun input-parameters (query &rest input)
+  (let ((new-query query))
+    (dolist (in input)
+      (setf new-query (regex-replace "\\$i" new-query in)))
+    (values new-query)))
diff --git a/src/.#clickhouse.lisp b/src/.#clickhouse.lisp
@@ -0,0 +1 @@
+[email protected]:1678404309
diff --git a/src/clickhouse.lisp b/src/clickhouse.lisp
@@ -43,6 +43,7 @@
     :documentation "ClickHouse database username, default username is default.")
    (password
     :initarg :password
+    :initform nil
     :accessor password
     :documentation "Clickhouse database password.")))
 
@@ -74,9 +75,9 @@
 
 (defmethod query ((obj database) query &key console no-format timeout)
   "Execute a query."
-  (with-slots ((h host) (p port) (s ssl)) obj
+  (with-slots ((h host) (p port) (s ssl) (u username) (w password)) obj
     (prettify
-     (http-post h p s (make-query query) timeout)
+     (http-post h p s u w (make-query query) timeout)
      :console console :formatting (if no-format nil clickhouse.ch-sql-parser:*format*))))
 
 (defmacro jget (obj key)

diff --git a/src/http.lisp b/src/http.lisp
@@ -15,11 +15,17 @@
 		   :force-string t)
     (values body)))
 
-(defun http-post (host-slot port-slot ssl-slot content timeout)
+(defun http-post (host-slot port-slot ssl-slot user-slot pass-slot content timeout)
   "HTTP handler for POST endpoints."
   (multiple-value-bind (body status response-header uri stream)
       (post (clickhouse.utils:format-url host-slot port-slot ssl-slot "")
+        :basic-auth (user-pass user-slot pass-slot)
 		    :content content
 		    :force-string t
 		    :read-timeout (if timeout timeout 60))
     (values body)))
+
+(defun user-pass (user pass)
+  (if (and user pass)
+      (cons user pass)
+      nil))
diff --git a/t/clickhouse.lisp b/t/clickhouse.lisp
@@ -138,7 +138,7 @@
 
 (test http-post
   (is (string= "1
-" (clickhouse.http::http-post "localhost" 8123 nil "SELECT 1" nil))))
+" (clickhouse.http::http-post "localhost" 8123 nil "default" nil "SELECT 1" nil))))
 
 ;; database class and methods