Bug: Error when querying data

[Skryabind]

I tested Parseable and sent full http error object to stream. It led to create many fields in the stream, and after that I can't fetch data from this stream. I see in logs (when run a query from the parseable console):

2023-08-27 21:12:18 thread 'actix-rt|system:0|arbiter:2' panicked at 'called `Result::unwrap()` on an `Err` value: ArrowError(ComputeError("concat requires input of at least one array"))', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/datafusion-common-27.0.0/src/scalar.rs:2835:18

I can't append full stream schema to due to limit of issue body, but it's realy huge.

[nitisht]

Thanks for reporting @Skryabind let us check and get back to you.

[trueleo]

@Skryabind I recognize this error because we have ran into this couple of times. I created an issue for this while back over here. apache/datafusion#5706

I need to check though if this is the same issue. The issue mostly stems from null array fields in data or very nested struct type.

Can i know few things about your setup.

• full http error object - Is this an object directly dumped from a program?

[Skryabind]

@trueleo yes, I just sent full error object to log. The code was like this (in nodejs):

try {
  axios.get(....)
} catch (e) {
  log.error(e)
}

So log was:

{
  "address": "127.0.0.1",
  "code": "ECONNREFUSED",
  "config": {
    "baseURL": "http://127.0.0.1:4522/v1",
    "env": {},
    "headers": {
      "Accept": "application/json, text/plain, */*",
      "User-Agent": "axios/0.27.2"
    },
    "maxBodyLength": 1000000000,
    "maxContentLength": 100000000,
    "method": "get",
    "params": {
      "integrationIds": []
    },
    "timeout": 0,
    "transformRequest": [
      null
    ],
    "transformResponse": [
      null
    ],
    "transitional": {
      "clarifyTimeoutError": false,
      "forcedJSONParsing": true,
      "silentJSONParsing": true
    },
    "url": "/company/53/messages/unread",
    "xsrfCookieName": "XSRF-TOKEN",
    "xsrfHeaderName": "X-XSRF-TOKEN"
  },
  "errno": -61,
  "level": "error",
  "message": "connect ECONNREFUSED 127.0.0.1:4522",
  "name": "Error",
  "port": 4522,
  "request": {
    "_currentRequest": {
      "_closed": false,
      "_contentLength": 0,
      "_defaultKeepAlive": true,
      "_ended": false,
      "_events": {},
      "_eventsCount": 7,
      "_hasBody": true,
      "_header": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
      "_headerSent": true,
      "_keepAliveTimeout": 0,
      "_last": true,
      "_redirectable": "[Circular]",
      "_removedConnection": false,
      "_removedContLen": false,
      "_removedTE": false,
      "_trailer": "",
      "aborted": false,
      "agent": {
        "_events": {},
        "_eventsCount": 2,
        "defaultPort": 80,
        "freeSockets": {},
        "keepAlive": false,
        "keepAliveMsecs": 1000,
        "maxFreeSockets": 256,
        "maxSockets": null,
        "maxTotalSockets": null,
        "options": {
          "path": null
        },
        "protocol": "http:",
        "requests": {},
        "scheduling": "lifo",
        "sockets": {
          "127.0.0.1:4522:": [
            {
              "_closeAfterHandlingError": false,
              "_events": {
                "close": [
                  null,
                  null
                ],
                "connect": [
                  null,
                  null,
                  null
                ]
              },
              "_eventsCount": 8,
              "_hadError": true,
              "_host": null,
              "_httpMessage": "[Circular]",
              "_parent": null,
              "_pendingData": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
              "_pendingEncoding": "latin1",
              "_readableState": {
                "autoDestroy": true,
                "awaitDrainWriters": null,
                "buffer": {
                  "head": null,
                  "length": 0,
                  "tail": null
                },
                "closeEmitted": true,
                "closed": true,
                "constructed": true,
                "dataEmitted": false,
                "decoder": null,
                "defaultEncoding": "utf8",
                "destroyed": true,
                "emitClose": false,
                "emittedReadable": false,
                "encoding": null,
                "endEmitted": false,
                "ended": false,
                "errorEmitted": true,
                "errored": {
                  "address": "127.0.0.1",
                  "code": "ECONNREFUSED",
                  "errno": -61,
                  "port": 4522,
                  "syscall": "connect"
                },
                "flowing": true,
                "highWaterMark": 16384,
                "length": 0,
                "multiAwaitDrain": false,
                "needReadable": true,
                "objectMode": false,
                "pipes": [],
                "readableListening": false,
                "reading": true,
                "readingMore": false,
                "resumeScheduled": false,
                "sync": false
              },
              "_server": null,
              "_sockname": null,
              "_writableState": {
                "afterWriteTickInfo": null,
                "allBuffers": true,
                "allNoop": true,
                "autoDestroy": true,
                "bufferProcessing": false,
                "buffered": [],
                "bufferedIndex": 0,
                "closeEmitted": true,
                "closed": true,
                "constructed": true,
                "corked": 0,
                "decodeStrings": false,
                "defaultEncoding": "utf8",
                "destroyed": true,
                "emitClose": false,
                "ended": false,
                "ending": false,
                "errorEmitted": true,
                "errored": {
                  "address": "127.0.0.1",
                  "code": "ECONNREFUSED",
                  "errno": -61,
                  "port": 4522,
                  "syscall": "connect"
                },
                "finalCalled": false,
                "finished": false,
                "highWaterMark": 16384,
                "length": 157,
                "needDrain": false,
                "objectMode": false,
                "pendingcb": 1,
                "prefinished": false,
                "sync": false,
                "writelen": 157,
                "writing": true
              },
              "allowHalfOpen": false,
              "connecting": false,
              "parser": null,
              "server": null
            }
          ]
        },
        "totalSocketCount": 1
      },
      "chunkedEncoding": false,
      "destroyed": false,
      "finished": true,
      "host": "127.0.0.1",
      "maxHeadersCount": null,
      "maxRequestsOnConnectionReached": false,
      "method": "GET",
      "outputData": [],
      "outputSize": 0,
      "parser": null,
      "path": "/v1/company/53/messages/unread",
      "protocol": "http:",
      "res": null,
      "reusedSocket": false,
      "sendDate": false,
      "shouldKeepAlive": false,
      "socket": {
        "_closeAfterHandlingError": false,
        "_events": {
          "close": [
            null,
            null
          ],
          "connect": [
            null,
            null,
            null
          ]
        },
        "_eventsCount": 8,
        "_hadError": true,
        "_host": null,
        "_httpMessage": "[Circular]",
        "_parent": null,
        "_pendingData": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
        "_pendingEncoding": "latin1",
        "_readableState": {
          "autoDestroy": true,
          "awaitDrainWriters": null,
          "buffer": {
            "head": null,
            "length": 0,
            "tail": null
          },
          "closeEmitted": true,
          "closed": true,
          "constructed": true,
          "dataEmitted": false,
          "decoder": null,
          "defaultEncoding": "utf8",
          "destroyed": true,
          "emitClose": false,
          "emittedReadable": false,
          "encoding": null,
          "endEmitted": false,
          "ended": false,
          "errorEmitted": true,
          "errored": {
            "address": "127.0.0.1",
            "code": "ECONNREFUSED",
            "errno": -61,
            "port": 4522,
            "syscall": "connect"
          },
          "flowing": true,
          "highWaterMark": 16384,
          "length": 0,
          "multiAwaitDrain": false,
          "needReadable": true,
          "objectMode": false,
          "pipes": [],
          "readableListening": false,
          "reading": true,
          "readingMore": false,
          "resumeScheduled": false,
          "sync": false
        },
        "_server": null,
        "_sockname": null,
        "_writableState": {
          "afterWriteTickInfo": null,
          "allBuffers": true,
          "allNoop": true,
          "autoDestroy": true,
          "bufferProcessing": false,
          "buffered": [],
          "bufferedIndex": 0,
          "closeEmitted": true,
          "closed": true,
          "constructed": true,
          "corked": 0,
          "decodeStrings": false,
          "defaultEncoding": "utf8",
          "destroyed": true,
          "emitClose": false,
          "ended": false,
          "ending": false,
          "errorEmitted": true,
          "errored": {
            "address": "127.0.0.1",
            "code": "ECONNREFUSED",
            "errno": -61,
            "port": 4522,
            "syscall": "connect"
          },
          "finalCalled": false,
          "finished": false,
          "highWaterMark": 16384,
          "length": 157,
          "needDrain": false,
          "objectMode": false,
          "pendingcb": 1,
          "prefinished": false,
          "sync": false,
          "writelen": 157,
          "writing": true
        },
        "allowHalfOpen": false,
        "connecting": false,
        "parser": null,
        "server": null
      },
      "strictContentLength": false,
      "timeoutCb": null,
      "upgradeOrConnect": false,
      "useChunkedEncodingByDefault": false,
      "writable": true
    },
    "_currentUrl": "http://127.0.0.1:4522/v1/company/53/messages/unread",
    "_ended": true,
    "_ending": true,
    "_events": {},
    "_eventsCount": 3,
    "_options": {
      "agents": {},
      "headers": {
        "Accept": "application/json, text/plain, */*",
        "User-Agent": "axios/0.27.2"
      },
      "hostname": "127.0.0.1",
      "maxBodyLength": 1000000000,
      "maxRedirects": 21,
      "method": "GET",
      "nativeProtocols": {
        "http:": {
          "METHODS": [
            "ACL",
            "BIND",
            "CHECKOUT",
            "CONNECT",
            "COPY",
            "DELETE",
            "GET",
            "HEAD",
            "LINK",
            "LOCK",
            "M-SEARCH",
            "MERGE",
            "MKACTIVITY",
            "MKCALENDAR",
            "MKCOL",
            "MOVE",
            "NOTIFY",
            "OPTIONS",
            "PATCH",
            "POST",
            "PROPFIND",
            "PROPPATCH",
            "PURGE",
            "PUT",
            "REBIND",
            "REPORT",
            "SEARCH",
            "SOURCE",
            "SUBSCRIBE",
            "TRACE",
            "UNBIND",
            "UNLINK",
            "UNLOCK",
            "UNSUBSCRIBE"
          ],
          "STATUS_CODES": {
            "100": "Continue",
            "101": "Switching Protocols",
            "102": "Processing",
            "103": "Early Hints",
            "200": "OK",
            "201": "Created",
            "202": "Accepted",
            "203": "Non-Authoritative Information",
            "204": "No Content",
            "205": "Reset Content",
            "206": "Partial Content",
            "207": "Multi-Status",
            "208": "Already Reported",
            "226": "IM Used",
            "300": "Multiple Choices",
            "301": "Moved Permanently",
            "302": "Found",
            "303": "See Other",
            "304": "Not Modified",
            "305": "Use Proxy",
            "307": "Temporary Redirect",
            "308": "Permanent Redirect",
            "400": "Bad Request",
            "401": "Unauthorized",
            "402": "Payment Required",
            "403": "Forbidden",
            "404": "Not Found",
            "405": "Method Not Allowed",
            "406": "Not Acceptable",
            "407": "Proxy Authentication Required",
            "408": "Request Timeout",
            "409": "Conflict",
            "410": "Gone",
            "411": "Length Required",
            "412": "Precondition Failed",
            "413": "Payload Too Large",
            "414": "URI Too Long",
            "415": "Unsupported Media Type",
            "416": "Range Not Satisfiable",
            "417": "Expectation Failed",
            "418": "I'm a Teapot",
            "421": "Misdirected Request",
            "422": "Unprocessable Entity",
            "423": "Locked",
            "424": "Failed Dependency",
            "425": "Too Early",
            "426": "Upgrade Required",
            "428": "Precondition Required",
            "429": "Too Many Requests",
            "431": "Request Header Fields Too Large",
            "451": "Unavailable For Legal Reasons",
            "500": "Internal Server Error",
            "501": "Not Implemented",
            "502": "Bad Gateway",
            "503": "Service Unavailable",
            "504": "Gateway Timeout",
            "505": "HTTP Version Not Supported",
            "506": "Variant Also Negotiates",
            "507": "Insufficient Storage",
            "508": "Loop Detected",
            "509": "Bandwidth Limit Exceeded",
            "510": "Not Extended",
            "511": "Network Authentication Required"
          },
          "globalAgent": {
            "_events": {},
            "_eventsCount": 2,
            "defaultPort": 80,
            "freeSockets": {},
            "keepAlive": false,
            "keepAliveMsecs": 1000,
            "maxFreeSockets": 256,
            "maxSockets": null,
            "maxTotalSockets": null,
            "options": {
              "path": null
            },
            "protocol": "http:",
            "requests": {},
            "scheduling": "lifo",
            "sockets": {
              "127.0.0.1:4522:": [
                {
                  "_closeAfterHandlingError": false,
                  "_events": {
                    "close": [
                      null,
                      null
                    ],
                    "connect": [
                      null,
                      null,
                      null
                    ]
                  },
                  "_eventsCount": 8,
                  "_hadError": true,
                  "_host": null,
                  "_httpMessage": {
                    "_closed": false,
                    "_contentLength": 0,
                    "_defaultKeepAlive": true,
                    "_ended": false,
                    "_events": {},
                    "_eventsCount": 7,
                    "_hasBody": true,
                    "_header": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
                    "_headerSent": true,
                    "_keepAliveTimeout": 0,
                    "_last": true,
                    "_redirectable": "[Circular]",
                    "_removedConnection": false,
                    "_removedContLen": false,
                    "_removedTE": false,
                    "_trailer": "",
                    "aborted": false,
                    "agent": "[Circular]",
                    "chunkedEncoding": false,
                    "destroyed": false,
                    "finished": true,
                    "host": "127.0.0.1",
                    "maxHeadersCount": null,
                    "maxRequestsOnConnectionReached": false,
                    "method": "GET",
                    "outputData": [],
                    "outputSize": 0,
                    "parser": null,
                    "path": "/v1/company/53/messages/unread",
                    "protocol": "http:",
                    "res": null,
                    "reusedSocket": false,
                    "sendDate": false,
                    "shouldKeepAlive": false,
                    "socket": "[Circular]",
                    "strictContentLength": false,
                    "timeoutCb": null,
                    "upgradeOrConnect": false,
                    "useChunkedEncodingByDefault": false,
                    "writable": true
                  },
                  "_parent": null,
                  "_pendingData": "GET /v1/company/53/messages/unread HTTP/1.1\r\nAccept: application/json, text/plain, */*\r\nUser-Agent: axios/0.27.2\r\nHost: 127.0.0.1:4522\r\nConnection: close\r\n\r\n",
                  "_pendingEncoding": "latin1",
                  "_readableState": {
                    "autoDestroy": true,
                    "awaitDrainWriters": null,
                    "buffer": {
                      "head": null,
                      "length": 0,
                      "tail": null
                    },
                    "closeEmitted": true,
                    "closed": true,
                    "constructed": true,
                    "dataEmitted": false,
                    "decoder": null,
                    "defaultEncoding": "utf8",
                    "destroyed": true,
                    "emitClose": false,
                    "emittedReadable": false,
                    "encoding": null,
                    "endEmitted": false,
                    "ended": false,
                    "errorEmitted": true,
                    "errored": {
                      "address": "127.0.0.1",
                      "code": "ECONNREFUSED",
                      "errno": -61,
                      "port": 4522,
                      "syscall": "connect"
                    },
                    "flowing": true,
                    "highWaterMark": 16384,
                    "length": 0,
                    "multiAwaitDrain": false,
                    "needReadable": true,
                    "objectMode": false,
                    "pipes": [],
                    "readableListening": false,
                    "reading": true,
                    "readingMore": false,
                    "resumeScheduled": false,
                    "sync": false
                  },
                  "_server": null,
                  "_sockname": null,
                  "_writableState": {
                    "afterWriteTickInfo": null,
                    "allBuffers": true,
                    "allNoop": true,
                    "autoDestroy": true,
                    "bufferProcessing": false,
                    "buffered": [],
                    "bufferedIndex": 0,
                    "closeEmitted": true,
                    "closed": true,
                    "constructed": true,
                    "corked": 0,
                    "decodeStrings": false,
                    "defaultEncoding": "utf8",
                    "destroyed": true,
                    "emitClose": false,
                    "ended": false,
                    "ending": false,
                    "errorEmitted": true,
                    "errored": {
                      "address": "127.0.0.1",
                      "code": "ECONNREFUSED",
                      "errno": -61,
                      "port": 4522,
                      "syscall": "connect"
                    },
                    "finalCalled": false,
                    "finished": false,
                    "highWaterMark": 16384,
                    "length": 157,
                    "needDrain": false,
                    "objectMode": false,
                    "pendingcb": 1,
                    "prefinished": false,
                    "sync": false,
                    "writelen": 157,
                    "writing": true
                  },
                  "allowHalfOpen": false,
                  "connecting": false,
                  "parser": null,
                  "server": null
                }
              ]
            },
            "totalSocketCount": 1
          },
          "maxHeaderSize": 16384
        },
        "https:": {
          "globalAgent": {
            "_events": {},
            "_eventsCount": 2,
            "_sessionCache": {
              "list": [],
              "map": {}
            },
            "defaultPort": 443,
            "freeSockets": {},
            "keepAlive": false,
            "keepAliveMsecs": 1000,
            "maxCachedSessions": 100,
            "maxFreeSockets": 256,
            "maxSockets": null,
            "maxTotalSockets": null,
            "options": {
              "path": null
            },
            "protocol": "https:",
            "requests": {},
            "scheduling": "lifo",
            "sockets": {},
            "totalSocketCount": 0
          }
        }
      },
      "path": "/v1/company/53/messages/unread",
      "pathname": "/v1/company/53/messages/unread",
      "port": "4522",
      "protocol": "http:"
    },
    "_redirectCount": 0,
    "_redirects": [],
    "_requestBodyBuffers": [],
    "_requestBodyLength": 0,
    "_writableState": {
      "afterWriteTickInfo": null,
      "allBuffers": true,
      "allNoop": true,
      "autoDestroy": true,
      "bufferProcessing": false,
      "buffered": [],
      "bufferedIndex": 0,
      "closeEmitted": false,
      "closed": false,
      "constructed": true,
      "corked": 0,
      "decodeStrings": true,
      "defaultEncoding": "utf8",
      "destroyed": false,
      "emitClose": true,
      "ended": false,
      "ending": false,
      "errorEmitted": false,
      "errored": null,
      "finalCalled": false,
      "finished": false,
      "highWaterMark": 16384,
      "length": 0,
      "needDrain": false,
      "objectMode": false,
      "pendingcb": 0,
      "prefinished": false,
      "sync": true,
      "writecb": null,
      "writelen": 0,
      "writing": false
    }
  },
  "service": "app",
  "stack": "Error: connect ECONNREFUSED 127.0.0.1:4522\n    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1278:16)",
  "syscall": "connect",
  "timestamp": "2023-08-27 19:03:35"
}

[Skryabind]

@trueleo I've just confirmed: If you send the object from my previous message to stream, You'll get this error.

[trueleo]

@Skryabind Yeah unfortunately this exact json won't work with Parseable as of today. The data is very nested with chain of struct of list of structs. We use parquet as our underlying format which is schema sensitive.

But I am assuming you want to log api errors and stuff which you can do in an alternative way. You just need to remove noise from the error you are throwing and have it how you want to look at it later on.

I will recommend looking at a logging library for node which will format and send the logs to Parseable. If the logging library does not support http / json or requires a specific connector logic to Parseable we can look into it.

We also have a basic example for sending logs in JS but you will need to hook this logic to your node server which i am unsure of how to do. https://www.parseable.io/docs/log-ingestion/applications/javascript

[Skryabind]

@trueleo I agree that this object is not realy sutable for log. In real projects I use winston logger and send more informative and short logs. Now I'm looking for a new logging system and testing Parseable in different ways. I think that the thing that sending "incorrect" log breaks access to the stream (include other normal logs) is not good. Maybe you can reject such logs in an insert stage if you know limitations to avoid latter problems with stream.

[nitisht]

Thanks @Skryabind I agree we should avoid ingesting logs that can potentially break the stream. We’ll add a fix