Skip to content

Commit

Permalink
rename to jch and add basic readme
Browse files Browse the repository at this point in the history
  • Loading branch information
djellemah committed Apr 27, 2024
1 parent 871bd80 commit a1163b6
Show file tree
Hide file tree
Showing 5 changed files with 59 additions and 28 deletions.
28 changes: 14 additions & 14 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "cln"
name = "jch"
version = "0.1.0"
edition = "2021"

Expand Down
15 changes: 15 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# jch

Parse really large json files, fast. Using a streaming parser so memory usage is limited.

For example:

- the schema for a 26Mb file is calculated in about 750ms using about 4Mb of RAM.

- the schema for a 432Mb file is calculated in about 20s using about 4Mb of RAM.

Currently only outputs the schema of a json file, in a non-standard format.

Is designed in a modular way so you can use it as a base for filtering json. Like `jq`.

Might grow some kind of path-filtering languages, like jsonpath or xpath.
16 changes: 16 additions & 0 deletions src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,3 +1,19 @@
/*!
Parse really large json files, fast. Using a streaming parser so memory usage is limited.
For example:
- the schema for a 26Mb file is calculated in about 750ms using about 4Mb of RAM.
- the schema for a 432Mb file is calculated in about 20s using about 4Mb of RAM.
Currently only outputs the schema of a json file, in a non-standard format.
Is designed in a modular way so you can use it as a base for filtering json. Like `jq`.
Might grow some kind of path-filtering languages, like jsonpath or xpath.
*/

// parser and traits
pub mod parser;
pub mod jsonpath;
Expand Down
26 changes: 13 additions & 13 deletions src/main.rs
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
use cln::plain;
use cln::valuer;
use cln::channel;
use cln::shredder;
use cln::parser;
use cln::jsonpath;
use cln::schema;
use cln::handler;
use cln::fn_snd;
use jch::plain;
use jch::valuer;
use jch::channel;
use jch::shredder;
use jch::parser;
use jch::jsonpath;
use jch::schema;
use jch::handler;
use jch::fn_snd;

use std::process::exit;

Expand All @@ -19,13 +19,13 @@ fn main() {
match &args[1..] {
["-s", "-z"] => schema::sizes(&mut std::io::stdout()).unwrap(),
["-s", rst @ ..] => {
let istream = cln::make_readable(rst);
let istream = jch::make_readable(rst);
let mut jevstream = parser::JsonEvents::new(istream);
schema::schema(&mut jevstream);
}
// This are POC to see that the rest of the handlers and visitors work.
["-p", rst @ ..] => {
let istream = cln::make_readable(rst);
let istream = jch::make_readable(rst);
let mut jevstream = parser::JsonEvents::new(istream);

// just use a (mostly) simple function wrapper
Expand All @@ -40,7 +40,7 @@ fn main() {
.unwrap_or_else(|err| eprintln!("ending event reading because {err:?}"));
}
["-v", rst @ ..] => {
let istream = cln::make_readable(rst);
let istream = jch::make_readable(rst);
let mut jevstream = parser::JsonEvents::new(istream);

// accept all paths, and convert leafs to serde_json::Value
Expand All @@ -55,7 +55,7 @@ fn main() {
.unwrap_or_else(|err| {eprintln!("ending event reading because {err:?}"); exit(1)})
}
["-c", rst @ ..] => {
let istream = cln::make_readable(rst);
let istream = jch::make_readable(rst);
let mut jevstream = parser::JsonEvents::new(istream);
// producer reads file and converts to serde_json events, consumer just receives them.
channel::channels(&mut jevstream)
Expand Down

0 comments on commit a1163b6

Please sign in to comment.