SNOW-1031704: The package is using an old version of @google-cloud/storage. #759

jainmohit2001 · 2024-01-31T15:29:10Z

Please answer these questions before submitting your issue.
In order to accurately debug the issue this information is required. Thanks!

What version of NodeJS driver are you using?
[email protected]
What operating system and processor architecture are you using?
WSL2 on Windows 10
What version of NodeJS are you using?
(node --version and npm --version)
v20.10.0
What are the component versions in the environment (npm list)?

5.Server version:* E.g. 1.90.1

What did you do?

Using snowflake-sdk with Next.js.

What did you expect to see?

No error while running npm run build.

I got the following error instead

./node_modules/@google-cloud/storage/node_modules/retry-request/index.js
Module not found: Can't resolve 'request' in '/path/to/node_modules/@google-cloud/storage/node_modules/retry-request'

Import trace for requested module:
./node_modules/@google-cloud/storage/node_modules/retry-request/index.js
./node_modules/@google-cloud/storage/build/src/nodejs-common/util.js
./node_modules/@google-cloud/storage/build/src/nodejs-common/index.js
./node_modules/@google-cloud/storage/build/src/bucket.js
./node_modules/@google-cloud/storage/build/src/index.js
./node_modules/snowflake-sdk/lib/file_transfer_agent/gcs_util.js
./node_modules/snowflake-sdk/lib/file_transfer_agent/remote_storage_util.js
./node_modules/snowflake-sdk/lib/file_transfer_agent/file_transfer_agent.js
./node_modules/snowflake-sdk/lib/connection/statement.js
./node_modules/snowflake-sdk/lib/connection/connection.js
./node_modules/snowflake-sdk/lib/core.js
./node_modules/snowflake-sdk/lib/snowflake.js
./node_modules/snowflake-sdk/index.js
./src/lib/db.ts
./src/components/navbar/actions.ts
./src/components/navbar/navbar.tsx
./src/app/(authorized)/layout.tsx

The text was updated successfully, but these errors were encountered:

sfc-gh-dszmolka · 2024-01-31T15:58:33Z

hi - thanks for submitting this issue !

can you please provide some details (like steps you execute) to get to this issue ?

issuing npm install on a node v20.10.0 container successfully installs @google-cloud/[email protected] per requirement in package.json. Thank you in advance for the details!

jainmohit2001 · 2024-02-01T08:40:28Z

Hi @sfc-gh-dszmolka
Here are the steps I used:

Install Next.js with basic minimal template https://nextjs.org/docs/getting-started/installation
Install snowflake-sdk using npm
Create a connection pool in a .ts file.

import { Options as PoolOptions } from 'generic-pool'
import * as snowflake from 'snowflake-sdk'

const connectionOptions: snowflake.ConnectionOptions = {
  account: process.env.SNOWFLAKE_ACCOUNT!,
  database: process.env.SNOWFLAKE_DB,
  schema: process.env.SNOWFLAKE_SCHEMA,
  username: process.env.SNOWFLAKE_USERNAME,
  password: process.env.SNOWFLAKE_PASSWORD,
  warehouse: process.env.SNOWFLAKE_WAREHOUSE,
}

const poolOptions: PoolOptions = {
  max: 10,
  min: 2,
}

export const connectionPool = snowflake.createPool(
  connectionOptions,
  poolOptions,
)

Build the project using the following:

npm run build

You shall see the mentioned warnings in the console directly.

On another note, the npm package for @google-cloud/storage is currently at version 7.7.0 https://www.npmjs.com/package/@google-cloud/storage

It is a major version upgrade and I think the package should also upgrade the deps accordingly.

sfc-gh-dszmolka · 2024-02-01T09:26:02Z

thank you - probably we'll need to bump @google-cloud/storage to the next major version eventually, but first I would like to see the issue happening to be able to see how the new version of the package fixes it.

Currently, I'm not able to see the issue following your steps

# npx create-next-app@latest 
✔ What is your project named? … my-app
✔ Would you like to use TypeScript? … No / Yes (Yes)
✔ Would you like to use ESLint? … No / Yes (No)
✔ Would you like to use Tailwind CSS? … No / Yes (No)
✔ Would you like to use `src/` directory? … No / Yes (No)
✔ Would you like to use App Router? (recommended) … No / Yes (No)
✔ Would you like to customize the default import alias (@/*)? … No / Yes (No)
Creating a new Next.js app in /test/my-app.
..[gets created]
# cd my-app/
/my-app# npm i snowflake-sdk
..[gets installed]
/my-app# cat index.ts 
import { Options as PoolOptions } from 'generic-pool'
import * as snowflake from 'snowflake-sdk'

const connectionOptions: snowflake.ConnectionOptions = {
  account: process.env.SNOWFLAKE_ACCOUNT!,
  username: process.env.SNOWFLAKE_USERNAME,
  password: process.env.SNOWFLAKE_PASSWORD,
}

const poolOptions: PoolOptions = {
  max: 10,
  min: 2,
}

export const connectionPool = snowflake.createPool(
  connectionOptions,
  poolOptions,
)
/my-app# export SNOWFLAKE_ACCOUNT=... # export the envvars
/my-app# npm run build

> [email protected] build
> next build
...
./index.ts:2:28
Type error: Could not find a declaration file for module 'snowflake-sdk'. '/test/my-app/node_modules/snowflake-sdk/index.js' implicitly has an 'any' type.
  Try `npm i --save-dev @types/snowflake-sdk` if it exists or add a new declaration (.d.ts) file containing `declare module 'snowflake-sdk';`
# oops need the types, lets install them
/my-app# npm i --save-dev @types/snowflake-sdk
..[gets installed]
/my-app# npm run build

> [email protected] build
> next build

   ▲ Next.js 14.1.0

 ✓ Linting and checking validity of types    
   Creating an optimized production build ...
 ✓ Compiled successfully
 ✓ Collecting page data    
 ✓ Generating static pages (3/3) 
 ✓ Collecting build traces    
 ✓ Finalizing page optimization    

Route (pages)                             Size     First Load JS
┌ ○ /                                     4.89 kB        82.9 kB
├   └ css/9bf468b453c348b9.css            1.73 kB
├   /_app                                 0 B              78 kB
├ ○ /404                                  181 B          78.2 kB
└ λ /api/hello                            0 B              78 kB
+ First Load JS shared by all             78.7 kB
  ├ chunks/framework-5429a50ba5373c56.js  45.2 kB
  ├ chunks/main-930135e47dff83e9.js       31.8 kB
  └ other shared chunks (total)           1.76 kB

○  (Static)   prerendered as static content
λ  (Dynamic)  server-rendered on demand using Node.js

build looks successful even with the default 6.12.0 GCP storage.

So i'm probably missing something, wondering what that could be. Appreciate the pointers !

jainmohit2001 · 2024-02-01T09:38:51Z

Additional steps that might help us understand the error I am seeing:

Create a simple input form.
Create a Next.js Server action. Make sure to do some stuff using snowflake-sdk here. Just a call to the Database should suffice.
Call the server action upon form submission.

... Error should appear in the console.

sfc-gh-dszmolka · 2024-02-01T12:45:02Z

okay, so this requires a little bit more setup to mimic your specific environment :) for which, do you think it would be possible taking one of the two paths:

if possible, please create a full reproduction which when launched, produces the issue you're seeing. This can be shared either via a github repo, or attached here as a tarball, or something similar.
I know this requires additional resources, however since this issue seems to be very specific to certain framework, environment, and use-case - it's still the way forward.

OR

i created SNOW-1031704 bump @google-cloud/storage to v7 #760 (draft) to test if the version bump would instantly fail any tests; which it did not seem. Can you try to check out the fix branch from https://github.com/snowflakedb/snowflake-connector-nodejs/tree/test-bump-gcp-storage-to-7, use it as snowflake-sdk and see if it resolves the issue for you, in the same environment you're seeing the problem?

jainmohit2001 · 2024-02-01T13:39:07Z

Hi @sfc-gh-dszmolka. I'll try to share the repro soon.
I'll also try the bumped-up version.

jainmohit2001 · 2024-02-01T17:25:43Z

Hi @sfc-gh-dszmolka.
Check this out googleapis/nodejs-storage#2191

Update

Temporary Fix: Manually install retry-request and request packages.

WARNING: request has been deprecated, see request/request#3142

jainmohit2001 · 2024-02-02T05:46:15Z

Hi @sfc-gh-dszmolka.

Please find the minimal repro here https://github.com/jainmohit2001/snowflake-google-cloud-storage-issue

Update

The issue seems to be resolved when using the bumped up version of google cloud storage from https://github.com/snowflakedb/snowflake-connector-nodejs/tree/test-bump-gcp-storage-to-7.

I have created another branch for the same: https://github.com/jainmohit2001/snowflake-google-cloud-storage-issue/tree/bump-google-cloud-storage-to-7

sfc-gh-dszmolka · 2024-02-02T12:18:29Z

this is fantastic, thank you so much @jainmohit2001 ! issue can be very nicely reproduced using your repro, its clearly visible.

also happy to hear the fix branch helped you to tackle the issue, so I think it can use used as a workaround until we fix the issue properly. will keep this thread posted

sfc-gh-dszmolka · 2024-02-05T07:41:03Z

PR #760 which bumps @google-cloud/storage to v7 is now merged, and will be part of the next driver release, expected around second half of February.

sfc-gh-dszmolka · 2024-02-28T08:43:57Z

fix released with snowflake-sdk 1.10.0, closing this issue

jainmohit2001 · 2024-02-28T19:06:26Z

Hi @sfc-gh-dszmolka. Thanks for the release.
I upgraded the package and the error is now gone.

But I am seeing these messages in the console. Can you help me understand them and what they mean? If possible please share a link to the document where I can read more about this.

Thanks in advance!

{"level":"INFO","message":"[12:27:55.693 AM]: Trying to initialize Easy Logging"}
{"level":"INFO","message":"[12:27:55.706 AM]: Trying to initialize Easy Logging"}
{"level":"INFO","message":"[12:27:55.899 AM]: No client config file found in default directories"}
{"level":"INFO","message":"[12:27:55.900 AM]: Easy Logging is disabled as no config has been found"}
{"level":"INFO","message":"[12:27:55.921 AM]: No client config file found in default directories"}
{"level":"INFO","message":"[12:27:55.922 AM]: Easy Logging is disabled as no config has been found"}

sfc-gh-dszmolka · 2024-02-29T09:45:03Z

hi @jainmohit2001 happy to hear the error is gone for you now, thank you for the feedback!

The INFO level messages you're referring are unrelated to the fix and part of the Easy Logging feature (#758 , #770 , #779 ) which will help with easier/unified setting logging across multiple Snowflake drivers. We will document it for every driver once it is fully released.
You can set the loglevel to WARN or higher if you don't want to see INFO level messages or you can just ignore them.

jainmohit2001 · 2024-02-29T10:35:41Z

Thanks for the info!

I'll ignore them for now. I'll wait for the documentation first.

jainmohit2001 added the bug Something isn't working label Jan 31, 2024

jainmohit2001 changed the title ~~The package is using an old version of @google-cloud/storage. This leads to the following error:~~ The package is using an old version of @google-cloud/storage. Jan 31, 2024

github-actions bot changed the title ~~The package is using an old version of @google-cloud/storage.~~ SNOW-1031704: The package is using an old version of @google-cloud/storage. This leads to the following error: Jan 31, 2024

jainmohit2001 changed the title ~~SNOW-1031704: The package is using an old version of @google-cloud/storage. This leads to the following error:~~ SNOW-1031704: The package is using an old version of @google-cloud/storage. Jan 31, 2024

sfc-gh-dszmolka removed the bug Something isn't working label Jan 31, 2024

sfc-gh-dszmolka self-assigned this Jan 31, 2024

sfc-gh-dszmolka added the status-triage Issue is under initial triage label Jan 31, 2024

sfc-gh-dszmolka added the status-information_needed Additional information is required from the reporter label Jan 31, 2024

sfc-gh-dszmolka removed the status-information_needed Additional information is required from the reporter label Feb 1, 2024

sfc-gh-dszmolka added the status-information_needed Additional information is required from the reporter label Feb 1, 2024

sfc-gh-dszmolka mentioned this issue Feb 2, 2024

SNOW-1031704 bump @google-cloud/storage to v7 #760

Merged

sfc-gh-dszmolka closed this as completed Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNOW-1031704: The package is using an old version of @google-cloud/storage. #759

SNOW-1031704: The package is using an old version of @google-cloud/storage. #759

jainmohit2001 commented Jan 31, 2024 •

edited

Loading

sfc-gh-dszmolka commented Jan 31, 2024

jainmohit2001 commented Feb 1, 2024

sfc-gh-dszmolka commented Feb 1, 2024

jainmohit2001 commented Feb 1, 2024 •

edited

Loading

sfc-gh-dszmolka commented Feb 1, 2024

jainmohit2001 commented Feb 1, 2024

jainmohit2001 commented Feb 1, 2024 •

edited

Loading

jainmohit2001 commented Feb 2, 2024 •

edited

Loading

sfc-gh-dszmolka commented Feb 2, 2024

sfc-gh-dszmolka commented Feb 5, 2024

sfc-gh-dszmolka commented Feb 28, 2024

jainmohit2001 commented Feb 28, 2024

sfc-gh-dszmolka commented Feb 29, 2024

jainmohit2001 commented Feb 29, 2024

SNOW-1031704: The package is using an old version of @google-cloud/storage. #759

SNOW-1031704: The package is using an old version of @google-cloud/storage. #759

Comments

jainmohit2001 commented Jan 31, 2024 • edited Loading

sfc-gh-dszmolka commented Jan 31, 2024

jainmohit2001 commented Feb 1, 2024

sfc-gh-dszmolka commented Feb 1, 2024

jainmohit2001 commented Feb 1, 2024 • edited Loading

sfc-gh-dszmolka commented Feb 1, 2024

jainmohit2001 commented Feb 1, 2024

jainmohit2001 commented Feb 1, 2024 • edited Loading

Update

jainmohit2001 commented Feb 2, 2024 • edited Loading

Update

sfc-gh-dszmolka commented Feb 2, 2024

sfc-gh-dszmolka commented Feb 5, 2024

sfc-gh-dszmolka commented Feb 28, 2024

jainmohit2001 commented Feb 28, 2024

sfc-gh-dszmolka commented Feb 29, 2024

jainmohit2001 commented Feb 29, 2024

jainmohit2001 commented Jan 31, 2024 •

edited

Loading

jainmohit2001 commented Feb 1, 2024 •

edited

Loading

jainmohit2001 commented Feb 1, 2024 •

edited

Loading

jainmohit2001 commented Feb 2, 2024 •

edited

Loading