Skip to content

Commit

Permalink
core/migrations: Limit concurrent network requests
Browse files Browse the repository at this point in the history
  Mapping over a large number of documents in a migration that makes
  network requests for these documents can lead to exhausting available
  resources.
  This results in `net::ERR_INSUFFICIENT_RESOURCES` errors and failed
  migrations.

  Limiting the concurrency to only 10 documents at a time should prevent
  these errors.
  • Loading branch information
taratatach committed Apr 27, 2022
1 parent b441aeb commit 72d29a5
Showing 1 changed file with 8 additions and 3 deletions.
11 changes: 8 additions & 3 deletions core/migrations/migrations.js
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
*/

const path = require('path')
const { Promise } = require('bluebird')

const metadata = require('../metadata')
const { SCHEMA_INITIAL_VERSION } = require('./constants')
Expand Down Expand Up @@ -352,8 +353,9 @@ module.exports = ([
docs /*: SavedMetadata[] */,
{ remote } /*: InjectedDependencies */
) /*: Promise<SavedMetadata[]> */ => {
return Promise.all(
docs.map(async doc => {
return Promise.map(
docs,
async doc => {
const remoteDir = await remote.remoteCozy.findDir(doc.remote._id)

if (remoteDir != null) {
Expand Down Expand Up @@ -400,7 +402,10 @@ module.exports = ([
}

return doc
})
},
// XXX: Avoid exhausting network resources by limiting the number of
// concurrent requests.
{ concurrency: 10 }
)
}
}
Expand Down

0 comments on commit 72d29a5

Please sign in to comment.