JSON schema validation with references to other schemas

[nicomuns]

Hello,

I'm trying to perform JSON schema validation with references to other schemas ($ref) inside the main schema but cannot make it work. I think my URI resolver doesn't work but I don't know why. I have read this example as well as this issue.

Reading the example, I don't understand how you come up with the code of the resolver, namely this part:

std::string pathname = "./input/jsonschema/";
pathname += std::string(uri.path());

My main schema looks like this:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "schema title",
    "type": "object",
    "properties": {
        "property_one": {
            "$ref": "./other.schema.json"
        },
        "property_two": {
            "$ref": "../../../../foo/bar/baz/v1/main.schema.json"
        },
        "$version": {
            "type": "integer"
        }
    },
    "required": [],
    "additionalProperties": false
}

My URI resolver (not my actual implementation, just the logging part):

jsoncons::ojson resolver(const jsoncons::uri& p_uri) {
    std::cout << "p_uri.base: " << p_uri.base().string() << std::endl;
    std::cout << "p_uri.path: " << p_uri.path() << std::endl;
    
    std::string pathname = p_uri.path();
    // ...
}

Whatever I try, p_uri.base().string() and p_uri.path() always have the same value: /./other.schema.json.
While in the example they have different values. I also don't understand this additional leading /.

When creating the compiled schema with jsoncons::jsonschema::make_json_schema I always end up with the error Don't know how to load JSON Schema '/./other.schema.json'.

I am working on providing a minimal example for this post, but it will take me some more time before it is ready.

Thanks in advance

[danielaparker]

It's easier for me to investigate once I have your minimal example, thanks.

[nicomuns]

Hey,
Thanks for the quick reply.
Here is the code for the minimal example.
Just run this command from the root folder: mkdir build && cd build && cmake .. && make -j && ./main

Project:

CMakeLists.txt:

cmake_minimum_required(VERSION 3.10)

project(JsonSchemaValidatorMinimalExample)

set(CMAKE_CXX_STANDARD 17)

add_executable(main main.cpp validator.cpp)
include_directories(/usr/local/include)
target_link_libraries(main)

main.cpp:

#include "validator.hpp"

const char MAIN_SCHEMA_PATH[] = "../dir1/foo/bar/baz/main.schema.json";

int main() {
    JsonValidator validator(MAIN_SCHEMA_PATH);
    return 0;
}

validator.hpp:

#pragma once

#include <memory>
#include <string>
#include <jsoncons_ext/jsonschema/jsonschema.hpp>

class JsonValidator {
 public:
    explicit JsonValidator(const std::string& p_schemaFileName);

 private:
    static jsoncons::ojson resolver(const jsoncons::uri& p_uri);

    std::unique_ptr<jsoncons::jsonschema::json_schema<jsoncons::ojson>> m_validator;
};

validator.cpp:

#include "validator.hpp"
#include <exception>
#include <fstream>
#include <iostream>
#include <memory>
#include <string>

JsonValidator::JsonValidator(const std::string& p_schemaFileName) {
    std::cout << "p_schemaFileName  : " << p_schemaFileName << std::endl;

    jsoncons::ojson schema;
    try {
        std::ifstream is(p_schemaFileName.c_str());
        if (!is.is_open()) {
            std::cout << "Cannot open main JSON schema" << std::endl;
        }
        schema = jsoncons::ojson::parse(is);
    } catch (const std::exception& ex) {
        throw std::runtime_error("Cannot parse main JSON schema: " + std::string(ex.what()));
    }

    std::cout << "Main schema parsed" << std::endl;
    // std::cout << pretty_print(schema) << std::endl;

    // Create the validator from the schema
    try {
        m_validator = std::make_unique<jsoncons::jsonschema::json_schema<jsoncons::ojson>>(jsoncons::jsonschema::make_json_schema(std::move(schema), resolver));
        std::cout << "Validator created" << std::endl;
    } catch (const std::exception& ex) {
        throw std::runtime_error("Cannot create JSON schema validator: " + std::string(ex.what()));
    }
}

jsoncons::ojson JsonValidator::resolver(const jsoncons::uri& p_uri) {
    std::string pathname = p_uri.path();

    std::cout << "p_uri.base        : " << p_uri.base().string() << std::endl;
    std::cout << "p_uri.path        : " << p_uri.path() << std::endl;

    std::fstream is(pathname.c_str());
    if (!is) {
        std::cout << "Resolver cannot parse " << pathname << std::endl;
        return jsoncons::ojson::null();
    } else {
        std::cout << "Resolver parsed " << pathname << std::endl;
        return jsoncons::ojson::parse(is);
    }
}

dir1/foo/bar/baz/main.schema.json:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Title",
    "type": "object",
    "properties": {
        "my_prop": { "$ref": "./other.schema.json" },
        "my_other_prop": { "$ref": "../../../../dir2/foo/bar/baz/main.schema.json" },
        "$version": { "type": "integer" }
    },
    "additionalProperties": false
}

dir1/foo/bar/baz/other.schema.json:

{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Title",
    "type": "object",
    "properties": {
      "prop_1": { "type": "string" },
      "prop_2": { "type": "string" }
    },
    "required": ["prop_1", "prop_2"],
    "additionalProperties": false
}

Ouput:

p_schemaFileName  : ../dir1/foo/bar/baz/main.schema.json
Main schema parsed
p_uri.base        : /./other.schema.json
p_uri.path        : /./other.schema.json
Resolver cannot parse /./other.schema.json
terminate called after throwing an instance of 'std::runtime_error'
  what():  Cannot create JSON schema validator: Don't know how to load JSON Schema '/./other.schema.json'
Aborted (core dumped)

[danielaparker]

I think I need 'dir2/foo/bar/baz/main.schema.json'?

[nicomuns]

It's an empty schema. I didn't mention it because the resolver already fails with "my_prop": { "$ref": "./other.schema.json" } before encountering dir2/foo/bar/baz/main.schema.json.

[danielaparker]

References to other schemas via the $ref keyword are resolved relative to the base URI, usually provided via the $id keyword. In your case, you don't provide a base URI, and it defaults to empty. When jsoncons attempts to resolve "./other.schema.json" relative to the base URI "", it produces "/./other.schema.json". And yes, that's a bug, you can't resolve a relative URI against another relative URI. This has been fixed on master.

Note however that JSONSchema recommends including an $id property as a unique identifier for each schema. That makes it questionable to resolve everything to a relative URI and using that to locate resources. I would suggest something along the following lines, supplying the path to the root of the file directory as a parameter to the resolver,

{
    "$id" : "https://my_domain.com",
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Title",
    "type": "object",
    "properties": {
        "my_prop": { "$ref": "dir1/other.schema.json" },
        "my_other_prop": { "$ref": "dir2/foo/bar/baz/main.schema.json" },
        "$version": { "type": "integer" }
    },
    "additionalProperties": false
}

jsoncons::ojson resolver(const jsoncons::uri& p_uri) 
{
    std::cout << "p_uri.base        : " << p_uri.base().string() << std::endl;  //  https://my_domain.com/dir1/other.schema.json
    std::cout << "p_uri.path        : " << p_uri.path() << std::endl;           //  /dir1/other.schema.json

    std::string my_file_path = "/path_to_root" + p_uri.path();
    std::cout << "my_file_path: " << my_file_path << "\n";

    // Look up file at location 'my_file_path'

[nicomuns]

Hello,

Thanks for your answer.

I'm working on a code base which isn't mine, and indeed the $id keyword isn't provided in any of the schemas. I must admit I don't know much about JSON schemas and URIs in general. I'm migrating to your lib from this one and trying to fix the issues as they appear (the previous lib was resolving everything without the $id keyword).

And yes, that's a bug, you can't resolve a relative URI against another relative URI. This has been fixed on master.

Does this mean I wouldn't have to modify the schemas and it would work with an empty base URI (if I checkout master instead of 0.178.0)?
If yes, is there going to be a release including this fix?
EDIT: Just tested with master, I misunderstood that part. Providing the $id keyword is mandatory then right?

Because if I understand correctly, adding the $id keyword means I must add an hard-coded path in the resolver?
EDIT2: How can this work if the root schema changes?

[danielaparker]

You can use master. If doing do, use the latest version, which includes some fixes for resolving abnormal relative URI's (which includes ones with dot segments) against a base URI. Expect master to be promoted to a release in 8 weeks or so.

You have a couple of options for making a full file pathname available in the resolver, without hardcoding. I'll post an example in a bit.

[danielaparker]

Here's a minimalistic example that shows how to pass the root directory as a command line argument. It should work with both the latest release, 0.178.0, and master, which will be released as 1.0.0.

#include <jsoncons/json.hpp>
#include <jsoncons_ext/jsonschema/jsonschema.hpp>
#include <iostream>
#include <strstream>

std::string main_schema = R"(
{
    "$id" : "https://my_domain.com",
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Title",
    "type": "object",
    "properties": {
        "my_prop": { "$ref": "dir1/other.schema.json" },
        "my_other_prop": { "$ref": "dir2/foo/bar/baz/main.schema.json" },
        "$version": { "type": "integer" }
    },
    "additionalProperties": false
}
)";

jsoncons::ojson other_schema = jsoncons::ojson::parse(R"(
{
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Title",
    "type": "object",
    "properties": {
      "prop_1": { "type": "string" },
      "prop_2": { "type": "string" }
    },
    "required": ["prop_1", "prop_2"],
    "additionalProperties": false
}
)");

jsoncons::ojson empty_schema = jsoncons::ojson::parse(R"(
{
}
)");

int main(int argc, char* args[])
{
    if (argc < 2)
    {
        return 1;
    }
    const char* root_dir = args[1];

    jsoncons::ojson schema;
    try {
        std::stringstream is(main_schema.c_str());
        schema = jsoncons::ojson::parse(is);

        auto resolver = [root_dir](const jsoncons::uri& p_uri)
            {
                std::string pathname = p_uri.path();

                std::cout << "p_uri.base        : " << p_uri.base().string() << std::endl;
                std::cout << "p_uri.path        : " << p_uri.path() << std::endl;

                std::string my_file_path = root_dir + p_uri.path();
                std::cout << "my_file_path: " << my_file_path << "\n";

                // Mimics file retrieval
                if (my_file_path == "/path_to_root/dir1/other.schema.json")
                {
                    return other_schema;
                }
                else if (my_file_path == "/path_to_root/dir2/foo/bar/baz/main.schema.json")
                {
                    return empty_schema;
                }
                else
                {
                    std::cout << "Resolver cannot parse " << my_file_path << std::endl;
                    return jsoncons::ojson::null();
                }
            };
        auto validator = jsoncons::jsonschema::make_json_schema(schema, resolver);
    }
    catch (const std::exception& ex) {
        throw std::runtime_error("Cannot create JSON schema validator: " + std::string(ex.what()));
    }
    return 0;
}

When run with command line argument /path_to_root, it produces

p_uri.base        : https://my_domain.com/dir1/other.schema.json
p_uri.path        : /dir1/other.schema.json
my_file_path: /path_to_root/dir1/other.schema.json
p_uri.base        : https://my_domain.com/dir2/foo/bar/baz/main.schema.json
p_uri.path        : /dir2/foo/bar/baz/main.schema.json
my_file_path: /path_to_root/dir2/foo/bar/baz/main.schema.json

[nicomuns]

Thank you for taking the time to answer and provide an example. Unfortunately command line arguments is not an option an my particular use case, but this helps me understand a bit more.
One thing I didn't understand is if I must add the $id keyword in all my schemas or not. And if I do, should the URL be an actual valid one?

EDIT: Am I right to assume that root_dir is only used to identify the path to the folder container the root schema (in this case dir1/foo/bar/baz/)? And that all subsequent call to the resolver will use the same value every time?

[danielaparker]

Am I right to assume that root_dir is only used to identify the path to the folder container the root schema (in this case dir1/foo/bar/baz/)? And that all subsequent call to the resolver will use the same value every time?

The resolver is called every time jsoncons requires a resource that hasn't already been loaded. Within the resolver, you have root_dir and the passed URI identifying the resource. You combine them to produce a pathname to a file, as shown in the example.

[nicomuns]

Ok, this part is clear for me now thanks.
The challenge I'm facing is that I don't have the schemas stored in jsoncons::json variables. I'm using CMake Resource Compiler and working with paths to schema files. I tried adapting my code to update the root_dir when needed and be able to open the schema files and parse the content but cannot make it work so far.

[danielaparker]

The example I posted should work without the $id keyword in the schema using any release build, latest is 0.178.0. It works for the wrong reason (see earlier comments), but it works nonetheless. It wouldn't currently work on master but will once we add a default base uri option (should be soon.)

The URI provided with a $id keyword needs to be valid in the sense that it can be parsed, but not in the sense that it corresponds to a registered domain name or web address. In your case you're only interested in the path component. The main place you'll see this URI is in the 'schemaLocation' property of validation reports.

[nicomuns]

I think I finally got it to work, after adapting my code, all legacy tests are passing.
I went back to 0.178.0 and didn't modify the schemas.

Thanks again for all the help!

[nicomuns]

Just one last quick unrelated question, is there a main public header for the whole library, so that I can have only one #include?

[danielaparker]

jsoncons/json.hpp pulls in all of the core library that you'll want. In addition, each extension has a header file with the same name as the directory name that pulls in all of the names required by that extension, e.g. jsoncons_ext/jsonschema/jsonschema.hpp.

[nicomuns]

Thanks!