Tracking node-oracledb 2.0 Development

[cjbj]

node-oracledb 2.0.15 is out on GitHub and I've merged the dev-2.0. code to the master branch.
After some final release testing today, I'll publish to npm tomorrow.

Please test with:

  npm install oracle/node-oracledb.git#v2.0.15
  npm install https://github.com/oracle/node-oracledb/releases/download/v2.0.15/oracledb-2.0.15.tgz
  npm install https://github.com/oracle/node-oracledb/releases/download/v2.0.15/oracledb-src-2.0.15.tgz

Quick links:

• CHANGELOG
• INSTALLATION
• Documentation

Known issues:

1. When streaming in Node 8, the 'close' event is being emitted multiple
   times for error conditions. (Tests 81.1.19 & 82.1.19 - currently
   skipped) Fixed in 123d845

2. We're checking memory usage for ~~~VARCHAR2 columns ResultSets in stress tests.~~ Fixed in cc63cf7

3. We're triaging dumps seen in long stress tests with ROWID, UROWID columns, and LONGs, NCHAR/NCLOB. Fixed in ff3ea24

4. A crash reported in oracledb 2 core dumps #718 with RETURNING INTO LOB binds when the number of rows returned > number of row originally allocated. Fixed in 59b2524

5. DRCP connectionClass is ignored Fixed in a55ee24

6. Users should be aware of the new DPI-1054 message when connections can't be closed due to resources being open. We've tightened up the checking of in-use ResultSets and Lobs to avoid leaks and threading issues by making sure the application has closed resources before connections can be closed. The impact on streaming and stream error handling is under discussion.

7. Triaging an infrequent DPI-1002 in stress tests

8. Reviewing prefetchRows and maxRows behavior with respect to ODPI-C's optimizations. We have a handle on this but want to lay the groundwork for return results as one array when row length is unknown #158 so may make some API changes now in the major release. We've gone with the plan spelled out below in Tracking node-oracledb 2.0 Development #716 (comment) It looks good.

9. Create binaries and host on GitHub. The 2.0.14-dev binaries are available, but some process steps need sorting out (e.g. packaging for npm). The next release will include source code (like v1 does) in what will be the npm package so that users with no access to GitHub (yes, this is real) can build from source [the current 2.0.14 package on npm includes the source, instructions on building from this source will be forthcoming - you just need to delete the install script in package.json]. There are plans for improvements in runtime errors about dependencies, see https://gist.github.com/cjbj/78a587855a81edec3fb4dedc8aece522, and some potential ODPI nicenesses at Custom error message when Visual Studio distributable package is not installed on Windows odpi#48 and Custom error message when incorrect-architecture OCI.DLL is in PATH on Windows odpi#49. A problem installing the binaries via proxies will be fixed in 2.0.15, see Installation Problem oracledb@dev on Win 10 x64 #803

[cjbj]

I retagged 2.0.13-dev to pick up ODPI-C 2.0.0-rc1, which was the version we intended to be used. It has a memory optimization for server->client character expansion that matches DPI v1's behavior. This is related to issue 2 above.

[cjbj]

I've pushed some updates to dev-2.0. See https://github.com/oracle/node-oracledb/blob/dev-2.0/CHANGELOG.md

[cjbj]

An update since it's been a while. Node-oracledb 2.0 release is inching closer. ODPI-C is going to have an RC2 soon which is the next milestone.

[cjbj]

I've pushed the next batch of changes to the dev-2.0 branch. Please take a moment to try it out. Comments on the new DPI-1054 error message are welcome, as we are reviewing streaming behavior.

[cjbj]

I pushed some updates to dev-2.0 that improve stability and pick up the latest ODPI-C.

You might have seen we had a busy day yesterday with ODPI-C 2.0.0 and Python cx_Oracle 6.0 Production releases (and a PHP OCI8 patch release to PECL). Phew!

There are just a couple more things on the list before we can do the first node-oracledb 2.0 release. The big two are evaluating the DPI-1054 change impact on streaming, and pre-built binaries (more on that in #18)

[cjbj]

Now that the happy Oracle OpenWorld chaos is over, it's time to make some headway on node-oracledb v2. Oracle OpenWorld is a huge conference. We had great interest in Node.js and Python. The sessions of @dmcghan and @anthony-tuininga were well attended.

For node-oracledb v2, @dmcghan is helping out with the binary install code. @anthony-tuininga and @pvenkatraman are looking at the final issues intermittent and memory usage issues on the to-do list.

[cjbj]

I just pushed the next set of stabilization changes to the dev-2.0 branch. See the top of the CHANGELOG for details.

[cjbj]

I just pushed the next batch of updates to the dev-2.0 branch and created a tag v2.0.14-dev.

The significant thing is the new /package directory, which has scripts for handling pre-built binaries. Check out the readme there. I'll discuss this more in #18.

I'll update the tracking list at the top of this issue.

[cjbj]

I added a v2.0.14 tag to the same code base as v2.0.14-dev. This doesn't mean the code is production - it is still Development. I added the tag so the binary installer could find the files on GitHub. This is why we have dev releases: to iron out little issues like this.

[cjbj]

I just put pre-built node-oracledb binaries out for the current 2.0.14-development release. Warning: there will be further changes to node-oracledb before a production 2.0 release is made.

• Pre-built binaries for Node 4,6,8 and 9 are available on Github for macOS, Windows 64-bit and Linux 64-bit. The Linux binaries were built on Oracle Linux 6.

• Yes, you still need Oracle client libraries installed separately.

• Test installing by running:

npm install https://github.com//oracle/node-oracledb/releases/download/v2.0.14/oracledb-2.0.14.tgz

Node-oracledb v2 user documentation is still on the dev-2.0 branch

Please followup on binary install issues at #18.

If binaries are not available for your platform, the INSTALL doc shows the syntax to compile from source code: npm install oracle/node-oracledb.git#v2.0.14

[cjbj]

I updated 8 & 9 in the top comment.

[cjbj]

I've uploaded node-oracledb 2.0.14-dev to npmjs.com with the tag 'dev', so you can now install using the pre-built binaries with the command:

npm install oracledb@dev

I also replaced the base package on the GitHub release page to match that on npm. The real difference is the addition of the source files, which will be useful for people without GitHub access. The repackaging may not have strictly been necessary for 2.0.14; I was trying to resolve an npm publish glitch but that turned out to be due to npm/npm#16723. In the end, an older npm let me publish.

[cjbj]

Related to the prefetchRows evaluation in point 8 in the original post, our current thoughts on the fetch performance are around simplifying tuning of queries to just one attribute.

Yell out soon with any comments.

ODPI-C 2.1 has decided to use "array fetching" instead of prefetching for query fetch network buffer sizing. These are both underlying OCI concepts related to tuning.

Plan

In node-oracledb v2, oracledb.prefetchRows is likely to change to oracledb.fetchArraySize and will apply to all query methods. The tuning effect is the same. The default is the same at 100. Only the name changes. The equivalent execute() option will also be replaced. Setting oracledb.prefetchRows will have no effect.

The attribute name change is to be technically accurate, and to make Oracle driver doc about the two concepts relatable to all language APIs.

Impact

• On Direct fetches limited to maxRows (non-ResultSets)

  Network performance is now tuneable with fetchArraySize. Scripts with huge maxRows will no longer be allocating too much (often unnecessary) memory advance.

  maxRows is now only a limit, not also the internal fetch array size.

  In a future release, it should be straight-forward to allow an unlimited maxRows; I don't want to do this right now since it will delay the 2.0 release due to testing requirements.

• On getRow() and getRows(numRows) ResultSet fetches

  No interface change. fetchArraySize replaces use of prefetchRows for tuning.

• On queryStream()

  fetchArraySize replaces the overloaded use of maxRows to specify the underlying getRows() numRows parameter.

Upgrading scripts from v1

• search and replace 'prefetchRows' with 'fetchArraySize'

• tune fetchArraySize for direct fetches that return a large (or small) number of rows

• tune fetchArraySize instead of maxRows for queryStream()

[cjbj]

With the installer fix (#803) confirmed, we're getting close to releasing 2.0. The extra delay has let us refine the fetchArraySize behavior a bit - it's no longer used for getRows(), where the numRows parameter is already available and usable for buffer sizing. Also we will change the default of maxRows to be unlimited - yay! This will close #158

[cjbj]

I just pushed the 2.0.15 code to the dev-2.0 branch. Release is getting closer.

I'm planning on swapping the master (aka v1) and the dev-2.0 branches soon.

[cjbj]

node-oracledb 2.0.15 is out on GitHub and I've merged the dev-2.0. code to the master branch.
After some final release testing today, I'll publish to npm tomorrow. This will be a Production release.

For the eager, the changelog is here.

[cjbj]

"Tomorrow" came early. No sense waiting around. V2 is now production on npm: See:
https://blogs.oracle.com/opal/node-oracledb-20-with-pre-built-binaries-is-on-npm

Thanks everyone!