Skip to content

Commit

Permalink
Update docs.
Browse files Browse the repository at this point in the history
  • Loading branch information
ejball committed Jun 4, 2024
1 parent f1ff461 commit aa92272
Show file tree
Hide file tree
Showing 11 changed files with 197 additions and 16 deletions.
51 changes: 51 additions & 0 deletions BulkInsert.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# BulkInsert

The [`BulkInsert`](Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md) (and [`BulkInsertAsync`](Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md) extension methods allow efficient insertion of many rows into a database table with a familiar Dapper-like API.

## Problem

Dapper already has a mechanism for "bulk insert". Calling `Execute` with an `IEnumerable` will execute the specified `INSERT` command once for each item in the sequence. Unfortunately, this can be an extremely slow way to insert a large number of rows into a database.

Each DBMS has its own preferred approaches for efficiently inserting many rows into a database, but the most portable way is to execute an `INSERT` command with multiple rows in the `VALUES` clause, like so:

```
INSERT INTO widgets (name, size) VALUES ('foo', 22), ('bar', 14), ('baz', 42)
```

Building a SQL statement for a large number of rows is straightforward, but runs the risk of SQL injection problems if the SQL isn't escaped propertly.

Using command parameters is safer, but building and executing the SQL is more complex. Furthermore, databases often have a limit on the maximum number of command parameters that can be used, so it can be necessary to execute multiple SQL statements, one for each batch of rows to insert.

## Solution

`BulkInsert` is a simple Dapper-like extension method that builds the SQL commands for each batch and leverages Dapper to inject the command parameters.

```csharp
var widgets = new[] { new { name = "foo", size = 22 }, new { name = "bar", size = 14 }, new { name = "baz", size = 42 } };
connection.BulkInsert("INSERT INTO widgets (name, size) VALUES (@name, @size) ...", widgets);
```

The `...` after the `VALUES` clause must be included. It is used by `BulkInsert` to find the end of the `VALUES` clause that will be transformed. The call above will build a SQL statement like so:

```
INSERT INTO widgets (name, size) VALUES (@name_0, @size_0), (@name_1, @size_1)
```

The actual SQL statement will have as many parameters as needed to insert all of the specified rows. If the total number of command parameters exceeds 999 (the maximum number that [SQLite](https://www.sqlite.org/) supports and an efficient number for [MySql.Data](https://www.nuget.org/packages/MySql.Data/)), it will execute multiple SQL commands until all of the rows are inserted.

All of the transformed SQL will be executed for each batch, so including additional statements before or after the `INSERT` statement is not recommended.

Execute the method within a transaction if it is important to avoid inserting only some of the rows if there is an error.

## Reference

The `BulkInsert` and `BulkInsertAsync` methods of the `BulkInsertUtility` static class are extension methods on `IDbConnection`. They support these arguments:

* `sql` – The SQL to execute, which must have a `VALUES` clause that is followed by `...`.
* `commonParam` – The values of any command parameters that are outside the `VALUES` clause, or are common to every row. (Optional.)
* `insertParams` – The values of the command parameters for each row to be inserted.
* `transaction` – The current transaction, if any (see `Dapper.Execute`).
* `batchSize` – If specified, indicates the number of rows to insert in each batch, even if doing so requires more than 999 command parameters.
* `cancellationToken` – The optional cancellation token (`BulkInsertAsync` only).

The method returns the total number of rows affected (or, more specifically, the sum of the numbers returned when executing the SQL commands for each batch).
11 changes: 11 additions & 0 deletions Faithlife.Utility.Dapper.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Faithlife.Utility.Dapper assembly

The assembly `Faithlife.Utility.Dapper.dll` has 1 public type in 1 namespace.

## Faithlife.Utility.Dapper namespace

| public type | description |
| --- | --- |
| static class [BulkInsertUtility](Faithlife.Utility.Dapper/BulkInsertUtility.md) | Methods for bulk insert with Dapper. |

<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll -->
25 changes: 25 additions & 0 deletions Faithlife.Utility.Dapper/BulkInsertUtility.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# BulkInsertUtility class

Methods for bulk insert with Dapper.

```csharp
public static class BulkInsertUtility
```

## Public Members

| name | description |
| --- | --- |
| static [BulkInsert&lt;TInsert&gt;](BulkInsertUtility/BulkInsert.md)(…) | Efficiently inserts multiple rows, in batches as necessary. |
| static [BulkInsert&lt;TCommon,TInsert&gt;](BulkInsertUtility/BulkInsert.md)(…) | Efficiently inserts multiple rows, in batches as necessary. |
| static [BulkInsertAsync&lt;TInsert&gt;](BulkInsertUtility/BulkInsertAsync.md)(…) | Efficiently inserts multiple rows, in batches as necessary. |
| static [BulkInsertAsync&lt;TCommon,TInsert&gt;](BulkInsertUtility/BulkInsertAsync.md)(…) | Efficiently inserts multiple rows, in batches as necessary. |
| static [GetBulkInsertCommands&lt;TInsert&gt;](BulkInsertUtility/GetBulkInsertCommands.md)(…) | Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. |
| static [GetBulkInsertCommands&lt;TCommon,TInsert&gt;](BulkInsertUtility/GetBulkInsertCommands.md)(…) | Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. |

## See Also

* namespace [Faithlife.Utility.Dapper](../Faithlife.Utility.Dapper.md)
* [BulkInsertUtility.cs](https://github.com/Faithlife/DapperUtility/tree/master/src/Faithlife.Utility.Dapper/BulkInsertUtility.cs)
<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll -->
31 changes: 31 additions & 0 deletions Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# BulkInsertUtility.BulkInsert&lt;TInsert&gt; method (1 of 2)

Efficiently inserts multiple rows, in batches as necessary.

```csharp
public static int BulkInsert<TInsert>(this IDbConnection connection, string sql,
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null)
```

## See Also

* class [BulkInsertUtility](../BulkInsertUtility.md)
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md)

---

# BulkInsertUtility.BulkInsert&lt;TCommon,TInsert&gt; method (2 of 2)

Efficiently inserts multiple rows, in batches as necessary.

```csharp
public static int BulkInsert<TCommon, TInsert>(this IDbConnection connection, string sql, TCommon commonParam,
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null)
```

## See Also

* class [BulkInsertUtility](../BulkInsertUtility.md)
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md)

<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll -->
32 changes: 32 additions & 0 deletions Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# BulkInsertUtility.BulkInsertAsync&lt;TInsert&gt; method (1 of 2)

Efficiently inserts multiple rows, in batches as necessary.

```csharp
public static Task<int> BulkInsertAsync<TInsert>(this IDbConnection connection, string sql,
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken))
```

## See Also

* class [BulkInsertUtility](../BulkInsertUtility.md)
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md)

---

# BulkInsertUtility.BulkInsertAsync&lt;TCommon,TInsert&gt; method (2 of 2)

Efficiently inserts multiple rows, in batches as necessary.

```csharp
public static Task<int> BulkInsertAsync<TCommon, TInsert>(this IDbConnection connection, string sql,
TCommon commonParam, IEnumerable<TInsert> insertParams, IDbTransaction transaction = null,
int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken))
```

## See Also

* class [BulkInsertUtility](../BulkInsertUtility.md)
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md)

<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll -->
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# BulkInsertUtility.GetBulkInsertCommands&lt;TInsert&gt; method (1 of 2)

Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`.

```csharp
public static IEnumerable<CommandDefinition> GetBulkInsertCommands<TInsert>(string sql,
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken))
```

## See Also

* class [BulkInsertUtility](../BulkInsertUtility.md)
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md)

---

# BulkInsertUtility.GetBulkInsertCommands&lt;TCommon,TInsert&gt; method (2 of 2)

Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`.

```csharp
public static IEnumerable<CommandDefinition> GetBulkInsertCommands<TCommon, TInsert>(string sql,
TCommon commonParam, IEnumerable<TInsert> insertParams, IDbTransaction transaction = null,
int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken))
```

## See Also

* class [BulkInsertUtility](../BulkInsertUtility.md)
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md)

<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll -->
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1 +1 @@
### https://faithlife.github.io/RepoName/
### https://faithlife.github.io/DapperUtility/
10 changes: 5 additions & 5 deletions _layouts/default.html
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="{{ "/" | relative_url }}">ProjectName</a>
<a class="navbar-brand" href="{{ "/" | relative_url }}">Faithlife.Utility.Dapper</a>
</div>
<div class="navbar-collapse collapse" id="navbar-main">
<ul class="nav navbar-nav">
<li><a href="{{ "/ProjectName" | relative_url }}">Reference</a></li>
<li><a href="{{ "/Faithlife.Utility.Dapper" | relative_url }}">Reference</a></li>
</ul>
<ul class="nav navbar-nav navbar-right">
<li><a href="/"><img src="{{ "/assets/faithlife.png" | relative_url }}" class="faithlife-logo"></a></li>
Expand All @@ -32,9 +32,9 @@
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown" role="button" aria-expanded="false"><img src="{{ "/assets/github.png" | relative_url }}" class="navbar-logo"> <span class="caret"></span></a>
<ul class="dropdown-menu" role="menu">
<li><a href="https://github.com/Faithlife/RepoName">View Source Code</a></li>
<li><a href="https://github.com/Faithlife/RepoName/blob/master/LICENSE">View License</a></li>
<li><a href="https://github.com/Faithlife/RepoName/blob/docs/{{ page.path }}">Edit This Page</a></li>
<li><a href="https://github.com/Faithlife/DapperUtility">View Source Code</a></li>
<li><a href="https://github.com/Faithlife/DapperUtility/blob/master/LICENSE">View License</a></li>
<li><a href="https://github.com/Faithlife/DapperUtility/blob/docs/{{ page.path }}">Edit This Page</a></li>
</ul>
</li>
</ul>
Expand Down
Binary file modified assets/github.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
7 changes: 1 addition & 6 deletions assets/page.css
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,10 @@ body {
font-size: 125%;
}

.navbar-logo {
.github-logo {
height: 24px;
}

.faithlife-logo {
height: 28px;
margin: -2px;
}

.page-container {
margin-top: 21px;
margin-bottom: 21px;
Expand Down
12 changes: 8 additions & 4 deletions index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
# ProjectName
# Faithlife.Utility.Dapper

[![NuGet](https://img.shields.io/nuget/v/ProjectName.svg)](https://www.nuget.org/packages/ProjectName)
**Faithlife.Utility.Dapper** is a utility library for Dapper.

## Usage
## Features

See the [reference documentation](ProjectName.md).
* [BulkInsert](BulkInsert.md)

## Installation

Faithlife.Utility.Dapper should be installed [via NuGet](https://www.nuget.org/packages/Faithlife.Utility.Dapper).

0 comments on commit aa92272

Please sign in to comment.