JSON Schema Inference

[cpard]

I was trying to use recap today with some JSON data. I wanted to infer the schema from a JSON file and then use that as a way to navigate the recap types.

See the JSON I used here:

  "anonymousId": "78c53c15-32a1-4b65-adac-bec2d7bb8fab",
  "channel": "web",
  "context": {
    "campaign": {
      "name": "sales campaign",
      "source": "google",
      "medium": "medium",
      "term": "event data",
      "content": "Make sense of the modern data stack"
    },
    "library": {
      "name": "RudderLabs JavaScript SDK",
      "version": "2.9.1"
    },
    "locale": "en-US",
    "page": {
      "path": "/best-seller/1",
      "initial_referrer": "https://www.google.com/search",
      "initial_referring_domain": "google.com",
      "referrer": "https://www.google.com/search?q=estore+bestseller",
      "referring_domain": "google.com",
      "search": "estore bestseller",
      "title": "The best sellers offered by EStore",
      "url": "https://www.estore.com/best-seller/1"
    },
    "screen": { 
      "density": 420, 
      "height": 1794, 
      "width": 1080, 
      "innerHeight": 200, 
      "innerWidth": 100 
      },
    "userAgent": "Dalvik/2.1.0 (Linux; U; Android 9; Android SDK built for x86 Build/PSR1.180720.075)"
  },
  "event": "Product Reviewed",
  "integrations": { 
    "All": true 
    },
  "messageId": "1578564113557-af022c68-429e-4af4-b99b-2b9174056383",
  "properties": {
    "review_id": "review_id_1",
    "product_id": "product_id_1",
    "rating": 2.0,
    "review_body": "Sample Review Body"
  },
  "originalTimestamp": "2020-01-09T10:01:53.558Z",
  "type": "track",
  "sentAt": "2020-01-09T10:02:03.257Z"
}

after doing a recap crawl in the directory the file was, I executed a recap schema on the file and got the following result:

{
  "fields": [
    {
      "name": "anonymousId",
      "type": "string",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "channel",
      "type": "string",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "context",
      "type": "object",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "event",
      "type": "string",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "integrations",
      "type": "object",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "messageId",
      "type": "string",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "properties",
      "type": "object",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "originalTimestamp",
      "type": "datetime",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "type",
      "type": "string",
      "default": null,
      "nullable": null,
      "comment": null
    },
    {
      "name": "sentAt",
      "type": "datetime",
      "default": null,
      "nullable": null,
      "comment": null
    }
  ]
}

There are fields missing obviously but I don't understand why.

Also, if the JSON was stored in the file with newlines, recap was failing to parse it with a colorful but not helpful error produced by frictionless.

[criccomini]

I believe things are working now. I've made a bunch of changes since you ran your command (notably, implementing Recap's type spec in the Python library). Can you try re-running?

One thing to clarify: Recap currently only handles newline delimited JSON (akak ndjson or jsonl), which requires that each JSON object be on a single line. I'll open a GH issue to infer whether to treat JSON as ndjson or a single multi-line JSON object.

Anyway, I just ran:

pdm run recap schema /tmp/recap-test/test-subdir/bar.json

Against:

% cat /tmp/recap-test/test-subdir/bar.json
{  "anonymousId": "78c53c15-32a1-4b65-adac-bec2d7bb8fab",  "channel": "web",  "context": {    "campaign": {      "name": "sales campaign",      "source": "google",      "medium": "medium",      "term": "event data",      "content": "Make sense of the modern data stack"    },    "library": {      "name": "RudderLabs JavaScript SDK",      "version": "2.9.1"    },    "locale": "en-US",    "page": {      "path": "/best-seller/1",      "initial_referrer": "https://www.google.com/search",      "initial_referring_domain": "google.com",      "referrer": "https://www.google.com/search?q=estore+bestseller",      "referring_domain": "google.com",      "search": "estore bestseller",      "title": "The best sellers offered by EStore",      "url": "https://www.estore.com/best-seller/1"    },    "screen": {       "density": 420,       "height": 1794,       "width": 1080,       "innerHeight": 200,       "innerWidth": 100       },    "userAgent": "Dalvik/2.1.0 (Linux; U; Android 9; Android SDK built for x86 Build/PSR1.180720.075)"  },  "event": "Product Reviewed",  "integrations": {     "All": true     },  "messageId": "1578564113557-af022c68-429e-4af4-b99b-2b9174056383",  "properties": {    "review_id": "review_id_1",    "product_id": "product_id_1",    "rating": 2.0,    "review_body": "Sample Review Body"  },  "originalTimestamp": "2020-01-09T10:01:53.558Z",  "type": "track",  "sentAt": "2020-01-09T10:02:03.257Z"}

And got:

{
  "type": "struct",
  "fields": [
    {
      "name": "anonymousId",
      "type": "string32"
    },
    {
      "name": "channel",
      "type": "string32"
    },
    {
      "name": "context",
      "type": "struct",
      "fields": [
        {
          "name": "campaign",
          "type": "struct",
          "fields": [
            {
              "name": "name",
              "type": "string32"
            },
            {
              "name": "source",
              "type": "string32"
            },
            {
              "name": "medium",
              "type": "string32"
            },
            {
              "name": "term",
              "type": "string32"
            },
            {
              "name": "content",
              "type": "string32"
            }
          ]
        },
        {
          "name": "library",
          "type": "struct",
          "fields": [
            {
              "name": "name",
              "type": "string32"
            },
            {
              "name": "version",
              "type": "string32"
            }
          ]
        },
        {
          "name": "locale",
          "type": "string32"
        },
        {
          "name": "page",
          "type": "struct",
          "fields": [
            {
              "name": "path",
              "type": "string32"
            },
            {
              "name": "initial_referrer",
              "type": "string32"
            },
            {
              "name": "initial_referring_domain",
              "type": "string32"
            },
            {
              "name": "referrer",
              "type": "string32"
            },
            {
              "name": "referring_domain",
              "type": "string32"
            },
            {
              "name": "search",
              "type": "string32"
            },
            {
              "name": "title",
              "type": "string32"
            },
            {
              "name": "url",
              "type": "string32"
            }
          ]
        },
        {
          "name": "screen",
          "type": "struct",
          "fields": [
            {
              "name": "density",
              "type": "int64"
            },
            {
              "name": "height",
              "type": "int64"
            },
            {
              "name": "width",
              "type": "int64"
            },
            {
              "name": "innerHeight",
              "type": "int64"
            },
            {
              "name": "innerWidth",
              "type": "int64"
            }
          ]
        },
        {
          "name": "userAgent",
          "type": "string32"
        }
      ]
    },
    {
      "name": "event",
      "type": "string32"
    },
    {
      "name": "integrations",
      "type": "struct",
      "fields": [
        {
          "name": "All",
          "type": "bool"
        }
      ]
    },
    {
      "name": "messageId",
      "type": "string32"
    },
    {
      "name": "properties",
      "type": "struct",
      "fields": [
        {
          "name": "review_id",
          "type": "string32"
        },
        {
          "name": "product_id",
          "type": "string32"
        },
        {
          "name": "rating",
          "type": "float64"
        },
        {
          "name": "review_body",
          "type": "string32"
        }
      ]
    },
    {
      "name": "originalTimestamp",
      "type": "string32"
    },
    {
      "name": "type",
      "type": "string32"
    },
    {
      "name": "sentAt",
      "type": "string32"
    }
  ]
}

[criccomini]

I've opened #226