Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add forked docker process #39

Merged
merged 17 commits into from
Jun 24, 2024
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions launchSearch.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/*!
* Copyright © 2023 United States Government as represented by the
* Administrator of the National Aeronautics and Space Administration.
* All Rights Reserved.
*
* SPDX-License-Identifier: Apache-2.0
*/

import Dockerode from 'dockerode'

const [, , argv] = process.argv
const { dataDir, logsDir, engine, port, options } = JSON.parse(argv)

async function launchDocker() {
const Image =
engine === 'elasticsearch'
? 'elastic/elasticsearch:8.6.2'
: 'opensearchproject/opensearch:2.11.0'
console.log('Launching Docker container', Image)
const docker = new Dockerode()

const container = await docker.createContainer({
Env: [...options, 'path.data=/var/lib/search', 'path.logs=/var/log/search'],
HostConfig: {
AutoRemove: true,
Mounts: [
{ Source: dataDir, Target: '/var/lib/search', Type: 'bind' },
{ Source: logsDir, Target: '/var/log/search', Type: 'bind' },
],
PortBindings: {
[`${port}/tcp`]: [{ HostIP: '127.0.0.1', HostPort: `${port}` }],
},
},
Image,
})
const stream = await container.attach({ stream: true, stderr: true })
stream.pipe(process.stderr)
await container.start()
return container
}

const dockerContainer = await launchDocker()

const signals = ['message', 'SIGTERM', 'SIGINT']
signals.forEach((signal) => {
process.on(signal, async () => {
await dockerContainer.kill()
process.exit(0)
})
})

await dockerContainer.wait()
5 changes: 3 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,12 @@
],
"type": "module",
"files": [
"index.js"
"index.js",
"launchSearch.js"
],
"scripts": {
"prepare:husky": "husky install",
"prepare:esbuild": "esbuild index.ts --bundle --packages=external --outfile=index.js --platform=node --format=esm --tree-shaking=true",
"prepare:esbuild": "esbuild index.ts launchSearch.ts --bundle --packages=external --outdir=. --platform=node --format=esm --tree-shaking=true",
"prepare": "run-p prepare:*"
},
"engines": {
Expand Down
48 changes: 22 additions & 26 deletions run.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ import { mkdirP, temp } from './paths.js'
import rimraf from 'rimraf'
import { spawn, untilTerminated } from './processes.js'
import type { SandboxEngine } from './engines.js'
import Dockerode from 'dockerode'
import { UnexpectedResolveError, neverResolve } from './promises.js'
import { fork } from 'child_process'
import path from 'path'
import { fileURLToPath } from 'url'

type SearchEngineLauncherFunction<T = object> = (
props: T & {
Expand Down Expand Up @@ -63,37 +65,31 @@ const launchDocker: SearchEngineLauncherFunction = async ({
port,
options,
}) => {
const Image =
engine === 'elasticsearch'
? 'elastic/elasticsearch:8.6.2'
: 'opensearchproject/opensearch:2.11.0'
console.log('Launching Docker container', Image)
const docker = new Dockerode()
const container = await docker.createContainer({
Env: [...options, 'path.data=/var/lib/search', 'path.logs=/var/log/search'],
HostConfig: {
AutoRemove: true,
Mounts: [
{ Source: dataDir, Target: '/var/lib/search', Type: 'bind' },
{ Source: logsDir, Target: '/var/log/search', Type: 'bind' },
],
PortBindings: {
[`${port}/tcp`]: [{ HostIP: '127.0.0.1', HostPort: `${port}` }],
},
},
Image,
})
const stream = await container.attach({ stream: true, stderr: true })
stream.pipe(process.stderr)
await container.start()
const argv = {
dataDir,
logsDir,
engine,
port,
options,
}
const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)
const subprocess = fork(`${__dirname}/launchSearch.js`, [
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const __filename = fileURLToPath(import.meta.url)
const __dirname = path.dirname(__filename)
const subprocess = fork(`${__dirname}/launchSearch.js`, [
join(import.meta.dirname, 'launchSearch.js`, [

And of course add import { join } from 'node:path'.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2024-06-13 at 3 27 50 PM

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, I was one version behind. so if your local is not using the proper node version it's undefined.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you know if most people are using nvm? because we can prevent it from being undefined because of the wrong version if we add an .nvmrc file. That tells nvm what version to use for the project. But that would only be effective if people are using nvm.
Because if I use import.meta.dirname and they are not at the proper version, doing it this way will break whereas the other way would work regardless of node version.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't think we can expect that everyone is using NVM. What are you trying to do? Is importlib.meta.dirname only supported in recent Node.js versions? What is the minimum version that you need?

Copy link
Contributor Author

@Courey Courey Jun 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

upon further testing, the way that requires a node version of 20.11 doesn't seem to be an option right now.
I just ran into something interesting. As soon as I updated my node version to be 20.11.0, in order to terminate the process I started having to hit control + c twice. I changed my architect-plugin-search branch back to main, re-ran npm prepare and ensured that the package had updated in gcn. I then ran GCN at main after an npm install. I had to control + c twice to exit the process.

Screenshot 2024-06-13 at 4 32 38 PM

when I roll my version back to v20.9.0 I can quit out of the process with one control + c.

Screenshot 2024-06-13 at 4 37 30 PM

To be clear, for the import.meta.dirname to work GCN would have to be run using node version >= 20.11 and it is when I am running GCN with version 20.11.0 that I encounter this issue.

I haven't looked into what is making the process stick around in version v20.11.0 yet, so I don't know why it's doing it. Just that it happens with both branches on main if my version is 20.11.0.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

upon further testing, the way that requires a node version of 20.11 doesn't seem to be an option right now.

Alright. import.meta.url may be a better option, then. That has been supported at least as far back as Node.js 18. Please use import.meta.url.

Copy link
Contributor Author

@Courey Courey Jun 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that brings us back to this part of my other comment (cutting and pasting it here since it's relevant in this thread too):

import.meta.url is architect-plugin-search/index.js so that can't be used directly. Here is what I'm doing there and why it needs to have multiple steps:

  1. import.meta.url: file:///Users/courey/dev/architect-plugin-search/index.js this is getting the name of the file that is currently running. in this case it's index.js. I don't want index.js, I want launchSearch.js.
  2. so we then pass that into fileURLToPath so __filename would then be: /Users/courey/dev/architect-plugin-search/index.js. This is preparing it so that we can use path.
  3. I don't want the file, just the directory. Since we converted that to a path with fileURLToPath we can use path.dirname() to get just the path of the directory instead of the file that is running. that makes __dirname: /Users/courey/dev/architect-plugin-search
  4. I add /launchSearch.js to the end of that path to have the proper path and proper file name.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make this a lot simpler. There is no need to compute the path. Just bundle a single entry point with esbuild, as we did originally. Then launch the subprocess like this:

const subprocess = fork(import.meta.url, 'launch-docker-subprocess', JSON.stringify({
    dataDir,
    logsDir,
    engine,
    port,
    options,
  })

At file scope, add this code:

const [, , command, jsonifiedArgs] = process.argv
if (command === 'launch-docker-subprocess') {
  // put everything that you want the subprocess to do in here:
  // launch the container, attach signal handlers, wait for container to exit
}

JSON.stringify(argv),
])

return {
async kill() {
console.log('Killing Docker container')
await container.kill()
subprocess.send({ action: 'kill' })
Courey marked this conversation as resolved.
Show resolved Hide resolved
},
async waitUntilStopped() {
await container.wait()
return new Promise<void>((resolve) => {
lpsinger marked this conversation as resolved.
Show resolved Hide resolved
subprocess.on('exit', () => {
console.log('Docker container exited')
resolve()
})
})
},
}
}
Expand Down
4 changes: 3 additions & 1 deletion tsconfig.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
{
"extends": "@tsconfig/node14/tsconfig.json",
"compilerOptions": {
"resolveJsonModule": true
"resolveJsonModule": true,
"module": "esnext",
"target": "es2022"
}
}
Loading