-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
11 changed files
with
197 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,51 @@ | ||
# BulkInsert | ||
|
||
The [`BulkInsert`](Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsert.md) (and [`BulkInsertAsync`](Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md) extension methods allow efficient insertion of many rows into a database table with a familiar Dapper-like API. | ||
|
||
## Problem | ||
|
||
Dapper already has a mechanism for "bulk insert". Calling `Execute` with an `IEnumerable` will execute the specified `INSERT` command once for each item in the sequence. Unfortunately, this can be an extremely slow way to insert a large number of rows into a database. | ||
|
||
Each DBMS has its own preferred approaches for efficiently inserting many rows into a database, but the most portable way is to execute an `INSERT` command with multiple rows in the `VALUES` clause, like so: | ||
|
||
``` | ||
INSERT INTO widgets (name, size) VALUES ('foo', 22), ('bar', 14), ('baz', 42) | ||
``` | ||
|
||
Building a SQL statement for a large number of rows is straightforward, but runs the risk of SQL injection problems if the SQL isn't escaped propertly. | ||
|
||
Using command parameters is safer, but building and executing the SQL is more complex. Furthermore, databases often have a limit on the maximum number of command parameters that can be used, so it can be necessary to execute multiple SQL statements, one for each batch of rows to insert. | ||
|
||
## Solution | ||
|
||
`BulkInsert` is a simple Dapper-like extension method that builds the SQL commands for each batch and leverages Dapper to inject the command parameters. | ||
|
||
```csharp | ||
var widgets = new[] { new { name = "foo", size = 22 }, new { name = "bar", size = 14 }, new { name = "baz", size = 42 } }; | ||
connection.BulkInsert("INSERT INTO widgets (name, size) VALUES (@name, @size) ...", widgets); | ||
``` | ||
|
||
The `...` after the `VALUES` clause must be included. It is used by `BulkInsert` to find the end of the `VALUES` clause that will be transformed. The call above will build a SQL statement like so: | ||
|
||
``` | ||
INSERT INTO widgets (name, size) VALUES (@name_0, @size_0), (@name_1, @size_1) | ||
``` | ||
|
||
The actual SQL statement will have as many parameters as needed to insert all of the specified rows. If the total number of command parameters exceeds 999 (the maximum number that [SQLite](https://www.sqlite.org/) supports and an efficient number for [MySql.Data](https://www.nuget.org/packages/MySql.Data/)), it will execute multiple SQL commands until all of the rows are inserted. | ||
|
||
All of the transformed SQL will be executed for each batch, so including additional statements before or after the `INSERT` statement is not recommended. | ||
|
||
Execute the method within a transaction if it is important to avoid inserting only some of the rows if there is an error. | ||
|
||
## Reference | ||
|
||
The `BulkInsert` and `BulkInsertAsync` methods of the `BulkInsertUtility` static class are extension methods on `IDbConnection`. They support these arguments: | ||
|
||
* `sql` – The SQL to execute, which must have a `VALUES` clause that is followed by `...`. | ||
* `commonParam` – The values of any command parameters that are outside the `VALUES` clause, or are common to every row. (Optional.) | ||
* `insertParams` – The values of the command parameters for each row to be inserted. | ||
* `transaction` – The current transaction, if any (see `Dapper.Execute`). | ||
* `batchSize` – If specified, indicates the number of rows to insert in each batch, even if doing so requires more than 999 command parameters. | ||
* `cancellationToken` – The optional cancellation token (`BulkInsertAsync` only). | ||
|
||
The method returns the total number of rows affected (or, more specifically, the sum of the numbers returned when executing the SQL commands for each batch). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Faithlife.Utility.Dapper assembly | ||
|
||
The assembly `Faithlife.Utility.Dapper.dll` has 1 public type in 1 namespace. | ||
|
||
## Faithlife.Utility.Dapper namespace | ||
|
||
| public type | description | | ||
| --- | --- | | ||
| static class [BulkInsertUtility](Faithlife.Utility.Dapper/BulkInsertUtility.md) | Methods for bulk insert with Dapper. | | ||
|
||
<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll --> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
# BulkInsertUtility class | ||
|
||
Methods for bulk insert with Dapper. | ||
|
||
```csharp | ||
public static class BulkInsertUtility | ||
``` | ||
|
||
## Public Members | ||
|
||
| name | description | | ||
| --- | --- | | ||
| static [BulkInsert<TInsert>](BulkInsertUtility/BulkInsert.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | | ||
| static [BulkInsert<TCommon,TInsert>](BulkInsertUtility/BulkInsert.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | | ||
| static [BulkInsertAsync<TInsert>](BulkInsertUtility/BulkInsertAsync.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | | ||
| static [BulkInsertAsync<TCommon,TInsert>](BulkInsertUtility/BulkInsertAsync.md)(…) | Efficiently inserts multiple rows, in batches as necessary. | | ||
| static [GetBulkInsertCommands<TInsert>](BulkInsertUtility/GetBulkInsertCommands.md)(…) | Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. | | ||
| static [GetBulkInsertCommands<TCommon,TInsert>](BulkInsertUtility/GetBulkInsertCommands.md)(…) | Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. | | ||
|
||
## See Also | ||
|
||
* namespace [Faithlife.Utility.Dapper](../Faithlife.Utility.Dapper.md) | ||
* [BulkInsertUtility.cs](https://github.com/Faithlife/DapperUtility/tree/master/src/Faithlife.Utility.Dapper/BulkInsertUtility.cs) | ||
<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll --> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
# BulkInsertUtility.BulkInsert<TInsert> method (1 of 2) | ||
|
||
Efficiently inserts multiple rows, in batches as necessary. | ||
|
||
```csharp | ||
public static int BulkInsert<TInsert>(this IDbConnection connection, string sql, | ||
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null) | ||
``` | ||
|
||
## See Also | ||
|
||
* class [BulkInsertUtility](../BulkInsertUtility.md) | ||
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) | ||
|
||
--- | ||
|
||
# BulkInsertUtility.BulkInsert<TCommon,TInsert> method (2 of 2) | ||
|
||
Efficiently inserts multiple rows, in batches as necessary. | ||
|
||
```csharp | ||
public static int BulkInsert<TCommon, TInsert>(this IDbConnection connection, string sql, TCommon commonParam, | ||
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null) | ||
``` | ||
|
||
## See Also | ||
|
||
* class [BulkInsertUtility](../BulkInsertUtility.md) | ||
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) | ||
|
||
<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll --> |
32 changes: 32 additions & 0 deletions
32
Faithlife.Utility.Dapper/BulkInsertUtility/BulkInsertAsync.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# BulkInsertUtility.BulkInsertAsync<TInsert> method (1 of 2) | ||
|
||
Efficiently inserts multiple rows, in batches as necessary. | ||
|
||
```csharp | ||
public static Task<int> BulkInsertAsync<TInsert>(this IDbConnection connection, string sql, | ||
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) | ||
``` | ||
|
||
## See Also | ||
|
||
* class [BulkInsertUtility](../BulkInsertUtility.md) | ||
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) | ||
|
||
--- | ||
|
||
# BulkInsertUtility.BulkInsertAsync<TCommon,TInsert> method (2 of 2) | ||
|
||
Efficiently inserts multiple rows, in batches as necessary. | ||
|
||
```csharp | ||
public static Task<int> BulkInsertAsync<TCommon, TInsert>(this IDbConnection connection, string sql, | ||
TCommon commonParam, IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, | ||
int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) | ||
``` | ||
|
||
## See Also | ||
|
||
* class [BulkInsertUtility](../BulkInsertUtility.md) | ||
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) | ||
|
||
<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll --> |
32 changes: 32 additions & 0 deletions
32
Faithlife.Utility.Dapper/BulkInsertUtility/GetBulkInsertCommands.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
# BulkInsertUtility.GetBulkInsertCommands<TInsert> method (1 of 2) | ||
|
||
Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. | ||
|
||
```csharp | ||
public static IEnumerable<CommandDefinition> GetBulkInsertCommands<TInsert>(string sql, | ||
IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) | ||
``` | ||
|
||
## See Also | ||
|
||
* class [BulkInsertUtility](../BulkInsertUtility.md) | ||
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) | ||
|
||
--- | ||
|
||
# BulkInsertUtility.GetBulkInsertCommands<TCommon,TInsert> method (2 of 2) | ||
|
||
Gets the Dapper `CommandDefinition`s used by `BulkInsert` and `BulkInsertAsync`. | ||
|
||
```csharp | ||
public static IEnumerable<CommandDefinition> GetBulkInsertCommands<TCommon, TInsert>(string sql, | ||
TCommon commonParam, IEnumerable<TInsert> insertParams, IDbTransaction transaction = null, | ||
int? batchSize = null, CancellationToken cancellationToken = default(CancellationToken)) | ||
``` | ||
|
||
## See Also | ||
|
||
* class [BulkInsertUtility](../BulkInsertUtility.md) | ||
* namespace [Faithlife.Utility.Dapper](../../Faithlife.Utility.Dapper.md) | ||
|
||
<!-- DO NOT EDIT: generated by xmldocmd for Faithlife.Utility.Dapper.dll --> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1 @@ | ||
### https://faithlife.github.io/RepoName/ | ||
### https://faithlife.github.io/DapperUtility/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,7 +1,11 @@ | ||
# ProjectName | ||
# Faithlife.Utility.Dapper | ||
|
||
[![NuGet](https://img.shields.io/nuget/v/ProjectName.svg)](https://www.nuget.org/packages/ProjectName) | ||
**Faithlife.Utility.Dapper** is a utility library for Dapper. | ||
|
||
## Usage | ||
## Features | ||
|
||
See the [reference documentation](ProjectName.md). | ||
* [BulkInsert](BulkInsert.md) | ||
|
||
## Installation | ||
|
||
Faithlife.Utility.Dapper should be installed [via NuGet](https://www.nuget.org/packages/Faithlife.Utility.Dapper). |