Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-1405429: SnowflakeRestful.fetch loops (seemingly) indefinitely or returns non-descriptive "{}" in case of error #1944

Closed
sfc-gh-pczajka opened this issue May 13, 2024 · 2 comments
Closed

SNOW-1405429: SnowflakeRestful.fetch loops (seemingly) indefinitely or returns non-descriptive "{}" in case of error #1944

sfc-gh-pczajka opened this issue May 13, 2024 · 2 comments
Assignees
@sfc-gh-sghosh
Labels
status-triage_done Initial triage done, will be further handled by the driver team

Comments

@sfc-gh-pczajka
Copy link
Contributor

sfc-gh-pczajka commented May 13, 2024 •
edited
Loading

Python version

3.10.11

Operating system and processor architecture

macOS-14.4.1-arm64-arm-64bit

Installed packages

annotated-types==0.6.0
asn1crypto==1.5.1
atpublic==4.1.0
black==24.4.2
certifi==2024.2.2
cffi==1.16.0
cfgv==3.4.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
coverage==7.5.0
cryptography==42.0.7
distlib==0.3.8
exceptiongroup==1.2.1
filelock==3.14.0
gitdb==4.0.11
GitPython==3.1.43
identify==2.5.36
idna==3.7
importlib_metadata==7.1.0
iniconfig==2.0.0
jaraco.classes==3.4.0
Jinja2==3.1.4
keyring==24.3.1
markdown-it-py==3.0.0
MarkupSafe==2.1.5
mdurl==0.1.2
more-itertools==10.2.0
mypy-extensions==1.0.0
nodeenv==1.8.0
packaging==24.0
pathspec==0.12.1
platformdirs==4.2.1
pluggy==1.5.0
pre-commit==3.7.0
pycparser==2.22
pydantic==2.7.1
pydantic_core==2.18.2
Pygments==2.18.0
PyJWT==2.8.0
pyOpenSSL==24.1.0
pytest==8.2.0
pytest-randomly==3.15.0
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.31.0
requirements-parser==0.9.0
rich==13.7.1
shellingham==1.5.4
six==1.16.0
smmap==5.0.1
-e git+ssh://git@github.com/snowflakedb/snowflake-cli.git@f05fc87214f2f4cc44a28b52622bdbab5eb2ea20#egg=snowflake_cli_labs
snowflake-connector-python==3.10.0
snowflake-snowpark-python==1.15.0
snowflake.core==0.8.0
sortedcontainers==2.4.0
syrupy==4.6.1
tomli==2.0.1
tomlkit==0.12.3
typer==0.12.3
types-setuptools==69.5.0.20240423
typing_extensions==4.11.0
urllib3==2.2.1
virtualenv==20.26.1
zipp==3.18.1

What did you do?

I wanted to use connection.rest.fetch to connect with https://docs.snowflake.com/LIMITEDACCESS/rest-api/reference/ (use case: we want to add "create" commands to SnowflakeCLI).

Connecting to endpoints and creating object works without an issue, but error handling behaves weird:

  • if no_retry == False, the command seems to indefinitly retry connecting
  • if no_retry == True, the command drops error info (for some errors - 404 and 409 shown in the code below)
    (409 is an error code if the object already exists)
class ObjectManager:
    def __init__(self):
        self._conn = self._build_connection():

    def _build_connection(self):
        ...

    def _send_rest_request(
        self, url: str, method: str, data: Optional[Dict[str, Any]] = None
    ):
        # SnowflakeRestful.request assumes that API response is always a dict with "code" key.
        # This is not true in case of this API, so we need to do this workaround:
        from snowflake.connector.network import (
            CONTENT_TYPE_APPLICATION_JSON,
            HTTP_HEADER_ACCEPT,
            HTTP_HEADER_CONTENT_TYPE,
            HTTP_HEADER_USER_AGENT,
            PYTHON_CONNECTOR_USER_AGENT,
        )

        self._log.debug(f"Sending {method} request to {url}")
        rest = self._conn.rest
        full_url = f"{rest.server_url}{url}"
        headers = {
            HTTP_HEADER_CONTENT_TYPE: CONTENT_TYPE_APPLICATION_JSON,
            HTTP_HEADER_ACCEPT: CONTENT_TYPE_APPLICATION_JSON,
            HTTP_HEADER_USER_AGENT: PYTHON_CONNECTOR_USER_AGENT,
        }
        return self._conn.rest.fetch(
            method=method,
            full_url=full_url,
            headers=headers,
            token=rest.token,
            data=json.dumps(data if data else {}),
            no_retry=True,
        )

    def _url_exists(self, url):
        try:
            result = self._send_rest_request(url, method="get")
            return bool(result) or result == []
        except InterfaceError as err:
            if "404 Not Found" in str(err):
                return False
            raise err

    def _get_rest_api_create_url(self, object_type: str):
        plural_object_type = _get_object_names(object_type).sf_plural_name.replace(
            " ", "-"
        )
        url_prefix = "/api/v2"

        url = f"{url_prefix}/{plural_object_type}/"
        if self._url_exists(url):
            return url

        db = self._conn.database
        url = f"{url_prefix}/databases/{db}/{plural_object_type}/"
        if self._url_exists(url):
            return url

        schema = self._conn.schema
        url = f"{url_prefix}/databases/{db}/schemas/{schema}/{plural_object_type}/"
        if self._url_exists(url):
            return url

        return None

    def create(self, object_type: str, object_data: Dict[str, Any]) -> str:
        url = self._get_rest_api_create_url(object_type)
        if not url:
            return f"Create operation for type {object_type} is not supported. Try using `sql -q 'CREATE ...'` command"
        # TODO: nice exception handling (if possible) - "sth wrong" is not enough
        response = self._send_rest_request(url=url, method="post", data=object_data)
        if not response:
            return (
                "Something went wrong ¯\_(ツ)_/¯. Try again with --debug for more info."
            )
        return response["status"]

ObjectManager().create("schema", "test_schema")
ObjectManager().create("schema", "test_schema")


### What did you expect to see?

I'd expect that `no_retry` option always throws an exception when InterfaceError occurs (instead of "{}"). In our use case - we want to return readable error message to the user

### Can you set logging to DEBUG and collect the logs?

```bash
➜ snow object create schema name=test_again --debug           
2024-05-13 10:29:49 DEBUG [snowflake.cli.api.sql_execution] Sending get request to /api/v2/schemas/
2024-05-13 10:29:49 DEBUG [snowflake.cli.api.sql_execution] Sending get request to /api/v2/schemas/
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 1/1 active sessions
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] remaining request timeout: N/A ms, retry cnt: 1
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Request guid: 8407c147-d1b5-48df-91c1-7bfa88f3d1a7
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] socket timeout: 60
2024-05-13 10:29:49 DEBUG [snowflake.connector.vendored.urllib3.connectionpool] https://na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com:443 "GET /api/v2/schemas/?request_guid=8407c147-d1b5-48df-91c1-7bfa88f3d1a7 HTTP/1.1" 404 0
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Ignored error
Traceback (most recent call last):
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 884, in _request_exec_wrapper
    return_object = self._request_exec(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 1193, in _request_exec
    raise err
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 1135, in _request_exec
    raise_failed_request_error(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 238, in raise_failed_request_error
    Error.errorhandler_wrapper(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/errors.py", line 290, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/errors.py", line 348, in hand_to_other_handler
    connection.errorhandler(connection, cursor, error_class, error_value)
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/errors.py", line 221, in default_errorhandler
    raise error_class(
snowflake.connector.errors.InterfaceError: 250003 (08001): None: 404 Not Found: get https://na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com:443/api/v2/schemas/?request_guid=8407c147-d1b5-48df-91c1-7bfa88f3d1a7
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 0/1 active sessions
2024-05-13 10:29:49 DEBUG [snowflake.cli.api.sql_execution] Sending get request to /api/v2/databases/PCZAJKA_SNOWCLI_DEMO/schemas/
2024-05-13 10:29:49 DEBUG [snowflake.cli.api.sql_execution] Sending get request to /api/v2/databases/PCZAJKA_SNOWCLI_DEMO/schemas/
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 1/1 active sessions
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] remaining request timeout: N/A ms, retry cnt: 1
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Request guid: f5c9e8b5-45ea-459f-9251-b19f238e6bfe
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] socket timeout: 60
2024-05-13 10:29:49 DEBUG [snowflake.connector.vendored.urllib3.connectionpool] https://na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com:443 "GET /api/v2/databases/PCZAJKA_SNOWCLI_DEMO/schemas/?request_guid=f5c9e8b5-45ea-459f-9251-b19f238e6bfe HTTP/1.1" 200 None
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] SUCCESS
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 0/1 active sessions
2024-05-13 10:29:49 DEBUG [snowflake.cli.api.sql_execution] Sending post request to /api/v2/databases/PCZAJKA_SNOWCLI_DEMO/schemas/
2024-05-13 10:29:49 DEBUG [snowflake.cli.api.sql_execution] Sending post request to /api/v2/databases/PCZAJKA_SNOWCLI_DEMO/schemas/
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 1/1 active sessions
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] remaining request timeout: N/A ms, retry cnt: 1
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] Request guid: 2f5edb44-4764-4c56-a740-ccaa7fccb541
2024-05-13 10:29:49 DEBUG [snowflake.connector.network] socket timeout: 60
2024-05-13 10:29:50 DEBUG [snowflake.connector.vendored.urllib3.connectionpool] https://na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com:443 "POST /api/v2/databases/PCZAJKA_SNOWCLI_DEMO/schemas/?request_guid=2f5edb44-4764-4c56-a740-ccaa7fccb541 HTTP/1.1" 409 None
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Ignored error
Traceback (most recent call last):
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 884, in _request_exec_wrapper
    return_object = self._request_exec(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 1193, in _request_exec
    raise err
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 1135, in _request_exec
    raise_failed_request_error(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/network.py", line 238, in raise_failed_request_error
    Error.errorhandler_wrapper(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/errors.py", line 290, in errorhandler_wrapper
    handed_over = Error.hand_to_other_handler(
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/errors.py", line 348, in hand_to_other_handler
    connection.errorhandler(connection, cursor, error_class, error_value)
  File "/Users/pczajka/Library/Application Support/hatch/env/virtual/snowflake-cli-labs/NeTiQO6n/local.py3.10/lib/python3.10/site-packages/snowflake/connector/errors.py", line 221, in default_errorhandler
    raise error_class(
snowflake.connector.errors.InterfaceError: 250003 (08001): None: 409 Conflict: post https://na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com:443/api/v2/databases/PCZAJKA_SNOWCLI_DEMO/schemas/?request_guid=2f5edb44-4764-4c56-a740-ccaa7fccb541
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 0/1 active sessions
Something went wrong ¯\_(ツ)_/¯. Try again with --debug for more info.
2024-05-13 10:29:50 DEBUG [snowflake.cli.api.commands.snow_typer] Executing command post execution callback
2024-05-13 10:29:50 DEBUG [snowflake.cli.api.commands.snow_typer] Executing command post execution callback
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 1/1 active sessions
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] remaining request timeout: 5000 ms, retry cnt: 1
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Request guid: 839fcadc-666b-40be-ad4f-98e6e9874382
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] socket timeout: 60
2024-05-13 10:29:50 DEBUG [snowflake.connector.vendored.urllib3.connectionpool] https://na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com:443 "POST /telemetry/send?request_guid=839fcadc-666b-40be-ad4f-98e6e9874382 HTTP/1.1" 200 None
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] SUCCESS
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 0/1 active sessions
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] ret[code] = None, after post request
2024-05-13 10:29:50 INFO [snowflake.connector.connection] closed
2024-05-13 10:29:50 INFO [snowflake.connector.connection] No async queries seem to be running, deleting session
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 1/1 active sessions
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] remaining request timeout: 5000 ms, retry cnt: 1
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Request guid: a752aa62-e0fe-4ac1-b8ad-ece63b14a82c
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] socket timeout: 60
2024-05-13 10:29:50 DEBUG [snowflake.connector.vendored.urllib3.connectionpool] https://na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com:443 "POST /session?delete=true&request_guid=a752aa62-e0fe-4ac1-b8ad-ece63b14a82c HTTP/1.1" 200 None
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] SUCCESS
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] Session status for SessionPool 'na_consumer_qa6.qa6.us-west-2.aws.snowflakecomputing.com', SessionPool 0/1 active sessions
2024-05-13 10:29:50 DEBUG [snowflake.connector.network] ret[code] = None, after post request
2024-05-13 10:29:50 DEBUG [snowflake.connector.connection] Session is closed
@github-actions github-actions bot changed the title SnowflakeRestful.fetch loops (seemingly) indefinitely or returns non-descriptive "{}" in case of error SNOW-1405429: SnowflakeRestful.fetch loops (seemingly) indefinitely or returns non-descriptive "{}" in case of error May 13, 2024
@sfc-gh-sghosh sfc-gh-sghosh self-assigned this May 23, 2024
@sfc-gh-sghosh
Copy link

Hello @sfc-gh-pczajka ,

Could you clarify the issue? What kind of application are you using to reproduce the issue? Are you using the snowflakeCLI or Python application? could you paste the code snippet so we can reproduce the issue?

If you using snowflakeCLI, at present snowflakeCLI doesnt support REST API, its on road map and getting implemented.

Regards,
Sujan

@sfc-gh-sghosh sfc-gh-sghosh added the status-triage_needed This is a new issue, and initial triage is needed label May 23, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka added status-triage Issue is under initial triage status-information_needed Additional information is required from the reporter and removed status-triage_needed This is a new issue, and initial triage is needed labels May 23, 2024
@sfc-gh-sghosh sfc-gh-sghosh added need more information Need more information to continue triage/fix/implementation and removed status-information_needed Additional information is required from the reporter labels May 23, 2024
@sfc-gh-sghosh
Copy link

Hello @sfc-gh-pczajka,

I'm closing the ticket as we haven't received a reply or the required artifacts. Feel free to reopen it when you have it.

Regards,
Sujan

@sfc-gh-sghosh sfc-gh-sghosh added status-triage_done Initial triage done, will be further handled by the driver team and removed need more information Need more information to continue triage/fix/implementation status-triage Issue is under initial triage labels May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sfc-gh-sghosh sfc-gh-sghosh

Labels
status-triage_done Initial triage done, will be further handled by the driver team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sfc-gh-dszmolka @sfc-gh-sghosh @sfc-gh-pczajka