Skip to content

Commit

Permalink
Update error handling
Browse files Browse the repository at this point in the history
  • Loading branch information
volcan01010 committed May 16, 2024
1 parent b727907 commit 21b6739
Show file tree
Hide file tree
Showing 6 changed files with 48 additions and 10 deletions.
File renamed without changes.
33 changes: 33 additions & 0 deletions docs/demo_on_conflict.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
"""ETL Helper script to load records to a database table."""
import sqlite3
import etlhelper as etl

db_file = "igneous_rocks.db"
create_sql = """
CREATE TABLE IF NOT EXISTS igneous_rock (
id INTEGER PRIMARY KEY,
name TEXT UNIQUE,
grain_size TEXT
)"""
insert_sql = """
INSERT INTO igneous_rock (id, name, grain_size)
VALUES (:id, :name, :grain_size)
ON CONFLICT DO NOTHING
"""

igneous_rocks = [
{"id": 1, "name": "basalt", "grain_size": "fine"},
{"id": 1, "name": "basalt", "grain_size": "fine"} # duplicate row
]


with sqlite3.connect(db_file) as conn:
# Create table
etl.execute(create_sql, conn)

# Insert rows
etl.executemany(insert_sql, conn, rows=igneous_rocks)

# Confirm selection
for row in etl.fetchall('SELECT * FROM igneous_rock', conn):
print(row)
13 changes: 12 additions & 1 deletion docs/etl_functions/error_handling.rst
Original file line number Diff line number Diff line change
Expand Up @@ -106,4 +106,15 @@ By customising an INSERT query (which can be programmatically generated with
:func:`generate_insert_query() <etlhelper.generate_insert_query>`) the database
can be instructed how to process such rows.

TODO: example script of ON CONFLICT ignore
The following example for SQLite will ignore duplicate rows.
Different databases have different syntax and capabilities, including
``upsert`` and ``merge``.

.. literalinclude:: ../demo_on_conflict.py
:language: python

The output is:

.. code:: bash
{'id': 1, 'name': 'basalt', 'grain_size': 'fine'}
2 changes: 1 addition & 1 deletion docs/etl_functions/extract.rst
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ Row factories control the output format of returned rows.
The default row factory for ETL Helper is a dictionary, but this can be
changed with the ``row_factory`` argument.

.. literalinclude:: demo_named_tuple.py
.. literalinclude:: ../demo_namedtuple.py
:language: python

The output is:
Expand Down
7 changes: 0 additions & 7 deletions etlhelper/db_helpers/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,6 @@
They are not normally called directly.
For more details, see the source code in each module.
.. autoclass:: DbHelper
.. autoclass:: SQLiteDbHelper
.. autoclass:: PostgresDbHelper
.. autoclass:: OracleDbHelper
.. autoclass:: MSSQLDbHelper
"""
# flake8: noqa
from etlhelper.db_helpers.db_helper import DbHelper
Expand Down
3 changes: 2 additions & 1 deletion etlhelper/exceptions.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""
ETL Helper Exception classes
ETLHelper exceptions are derived from the ETLHelperError base class.
"""


Expand Down

0 comments on commit 21b6739

Please sign in to comment.