first commit

haydenbbickerton · Oct 25, 2023 · 9d9a0a7 · 9d9a0a7
commit 9d9a0a7
Show file tree

Hide file tree

Showing 73 changed files with 33,615 additions and 0 deletions.
diff --git a/Pipfile b/Pipfile
@@ -0,0 +1,17 @@
+[[source]]
+name = "pypi"
+url = "https://pypi.org/simple"
+verify_ssl = true
+
+[dev-packages]
+
+[packages]
+boltons = "*"
+pyaml = "*"
+pydantic = "*"
+jupyter = "*"
+ruamel-yaml = "*"
+parse = "*"
+
+[requires]
+python_version = "3.6"
diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/README.html b/README.html
@@ -0,0 +1,62 @@
+<!DOCTYPE html>
+<html>
+
+<head>
+  <meta charset="utf-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>test</title>
+  <link rel="stylesheet" href="https://stackedit.io/style.css" />
+</head>
+
+<body class="stackedit">
+  <div class="stackedit__html"><h1 id="cboe-pitch-viewer">CBOE PITCH Viewer</h1>
+<blockquote>
+<p>A tool for viewing/debugging PITCH messages</p>
+</blockquote>
+<p>DEMO: <a href="http://cboe.haydenbickerton.com/">cboe.haydenbickerton.com</a></p>
+<p>Here is a screenshot of the web interface:</p>
+<p><img src="https://drive.google.com/uc?export=view&id=15KgEKh51IWeQcE7jn7xWy1QARKsuy8bi" alt="viewer"></p>
+<h2 id="run-the-compiled-viewer">Run the compiled viewer</h2>
+<pre class=" language-sh"><code class="prism  language-sh">$ cd /client/dist
+$ python3 -m http.server 8080
+...
+</code></pre>
+<p>Then open your browser to <a href="http://localhost:8080/">http://localhost:8080/ </a> to see the viewer.</p>
+<h2 id="installation--development">Installation &amp; Development</h2>
+<p>OS X &amp; Linux:</p>
+<pre class=" language-sh"><code class="prism  language-sh">$ cd /client
+$ yarn install
+yarn install v1.7.0
+[1/4] Resolving packages...
+[2/4] Fetching packages...
+...
+$ npm run serve
+...
+  App running at:
+  - Local:   http://localhost:8080/
+  - Network: http://10.0.0.200:8080/
+</code></pre>
+<h1 id="notes">Notes</h1>
+<p>The code is in one of two steps:</p>
+<ol>
+<li>Converting the PITCH Message Specifications into usage python/javascript parsers/models
+<ul>
+<li><code>/structs/specs</code> - YAML’s extracted from PDF using <a href="%5Bhttps://tabula.technology/%5D(https://tabula.technology/)">tabula</a></li>
+<li><code>/notebooks/Building Structs.ipynb</code> - Notebook to build YAMLs into py/js models</li>
+</ul>
+</li>
+<li>Web interface/app/viewer to view uploaded message data
+<ul>
+<li><code>/client/</code> - Vue.js app (pictured above)</li>
+<li><code>/structs/compiled/Cboe.js</code> - Generated in  step 1 , imported by web app</li>
+</ul>
+</li>
+</ol>
+<p>Initially I went the route of making the python models and types for a backend parsing service, as I’m primarily a python developer. But copy-pasting each value into hardcoded classes when there’s clearly an existing spec <em>somewhere</em> isn’t what I’d normally do. So I did the YAMLs, and referenced those as the specs in the vue app. More about this in <code>Building Structs.ipynb</code>. I left code from initial run in <code>/cboe-sdk/</code> and <code>/notebooks/Modeling.ipynb</code>.</p>
+<h3 id="other">Other</h3>
+<p>This feels very similar to <a href="%5Bhttps://www.acord.org/standards-architecture/acord-data-standards/Property_Casualty_Data_Standards#AL3%5D(https://www.acord.org/standards-architecture/acord-data-standards/Property_Casualty_Data_Standards#AL3)">ACORD AL3</a>, which is a popular <a href="%5Bhttps://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ad09530_.htm%5D(https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ad09530_.htm)">fixed length messaging standard</a> used by insurance software for the transmission of policies and claims. I created the AL3 implementation during my time at <a href="%5Bhttps://www.britecore.com/%5D(https://www.britecore.com/)">BriteCore</a> .</p>
+<p>The idea for the viewer comes from working with AL3, as there are thousands of different data/message types and no straightforward tool for displaying a feed of messages and their fields.</p>
+</div>
+</body>
+
+</html>
diff --git a/README.md b/README.md
@@ -0,0 +1,60 @@
+# CBOE PITCH Viewer
+> A tool for viewing/debugging PITCH messages
+
+DEMO: [cboe.haydenbickerton.com](http://cboe.haydenbickerton.com/)
+
+Here is a screenshot of the web interface:
+
+![viewer](https://drive.google.com/uc?export=view&id=15KgEKh51IWeQcE7jn7xWy1QARKsuy8bi)
+
+## Run the compiled viewer
+```sh
+$ cd /client/dist
+$ python3 -m http.server 8080
+...
+```
+Then open your browser to [http://localhost:8080/ ](http://localhost:8080/ ) to see the viewer.
+
+## Installation & Development
+
+OS X & Linux:
+
+```sh
+$ cd /client
+$ yarn install
+yarn install v1.7.0
+[1/4] Resolving packages...
+[2/4] Fetching packages...
+...
+$ npm run serve
+...
+  App running at:
+  - Local:   http://localhost:8080/
+  - Network: http://10.0.0.200:8080/
+```
+
+# Notes
+The code is in one of two steps:
+
+ 1. Converting the PITCH Message Specifications into usage python/javascript parsers/models
+	 - `/structs/specs` - YAML's extracted from PDF using [tabula]([https://tabula.technology/](https://tabula.technology/))
+	 -  `/notebooks/Building Structs.ipynb` - Notebook to build YAMLs into py/js models
+ 2. Web interface/app/viewer to view uploaded message data
+	 - `/client/` - Vue.js app (pictured above)
+	 - `/structs/compiled/Cboe.js` - Generated in  step 1 , imported by web app
+
+
+Initially I went the route of making the python models and types for a backend parsing service, as I'm primarily a python developer. But copy-pasting each value into hardcoded classes when there's clearly an existing spec *somewhere* isn't what I'd normally do. So I did the YAMLs, and referenced those as the specs in the vue app. More about this in `Building Structs.ipynb`. I left code from initial run in `/cboe-sdk/` and `/notebooks/Modeling.ipynb`.
+
+
+### Other
+This feels very similar to [ACORD AL3]([https://www.acord.org/standards-architecture/acord-data-standards/Property_Casualty_Data_Standards#AL3](https://www.acord.org/standards-architecture/acord-data-standards/Property_Casualty_Data_Standards#AL3)), which is a popular [fixed length messaging standard]([https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ad09530_.htm](https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ad09530_.htm)) used by insurance software for the transmission of policies and claims. I created the AL3 implementation during my time at [BriteCore]([https://www.britecore.com/](https://www.britecore.com/)) .
+
+The idea for the viewer comes from working with AL3, as there are thousands of different data/message types and no straightforward tool for displaying a feed of messages and their fields.
+
+
+
+
+
+
+
diff --git a/cboe-sdk/cboe/pitch/__about__.py b/cboe-sdk/cboe/pitch/__about__.py
@@ -0,0 +1,5 @@
+# Package data
+
+__version_info__ = ("0", "0", "1")
+__version__ = ".".join(__version_info__)
+VERSION = __version__
diff --git a/cboe-sdk/cboe/pitch/datatypes.py b/cboe-sdk/cboe/pitch/datatypes.py
@@ -0,0 +1,27 @@
+from functools import partial
+
+from pydantic import Schema, constr
+
+import cboe.pitch.validators as validators
+
+# MsgType = constr(regex=ALPHA_RE)
+PrintableAscii = constr(regex=validators.PRINTABLE_RE)
+Alpha = constr(regex=validators.ALPHA_RE)
+Numeric = constr(regex=validators.NUMERIC_RE)
+Base36Numeric = constr(regex=validators.BASE36_NUMERIC_RE)
+Price = constr(regex=validators.PRICE_RE)
+Timestamp = constr(regex=validators.TIMESTAMP_RE)
+
+
+def MsgType(name, **kwargs):
+    # Camelcase because these return constructed classes
+    return constr(regex=f"^{name}$", **kwargs)
+
+
+def OneOf(*names, **kwargs):
+    # Also allow a space, for empty field value
+    names = "|".join(["\x20", *names])
+    return constr(regex=f"^{names}$", **kwargs)
+
+
+OrderSide = OneOf("B", "S")
diff --git a/cboe-sdk/cboe/pitch/models.py b/cboe-sdk/cboe/pitch/models.py
@@ -0,0 +1,32 @@
+from functools import partial
+from boltons.cacheutils import LRI, cached
+
+from boltons.iterutils import flatten, flatten_iter
+from boltons.strutils import iter_splitlines
+from boltons.iterutils import bucketize
+
+import parse
+from typing import Optional, TypeVar
+
+
+def parse_line(val: str):
+    value = trim_starting_s(val)
+    msgtypecode = value[8]
+    try:
+        return MESSAGE_TYPES[msgtypecode].parse_line(value)
+    except KeyError as e:
+        raise KeyError(f"No Message type found for: {msgtypecode}") from e
+
+
+def parse_lines(lines):
+    for line in filter(str.strip, lines):  # Filter out empty lines
+        yield parse_line(line)
+
+
+def parse_text(text):
+    message_lines = iter_splitlines(text)
+    yield from parse_lines(message_lines)
+
+
+def group_messages(messages):
+    return bucketize(messages, lambda x: x.message_type)
diff --git a/cboe-sdk/cboe/pitch/specs.py b/cboe-sdk/cboe/pitch/specs.py
@@ -0,0 +1,50 @@
+from pathlib import Path
+from typing import Dict, List, NewType
+
+from boltons.iterutils import flatten, flatten_iter
+from pydantic.dataclasses import dataclass
+from boltons.strutils import iter_splitlines
+from boltons.iterutils import bucketize
+
+from cboe.pitch import utils, yaml
+
+
+def RPath(path):
+    return Path(path).resolve()
+
+
+def Directory(path):
+    path = RPath(path)
+    return path if path.is_dir() else path.parent
+
+
+def collect_files(search_path, exts: list):
+    search_dir = Directory(search_path)
+    for ext in flatten([exts]):
+        yield from search_dir.glob(f"**/*{ext}")
+
+
+@dataclass
+class SpecLoader:
+    search_dir: Path
+    extensions = [".yml", ".yaml"]
+
+    def read_specs(self):
+        for file_path in collect_files(self.search_dir, self.extensions):
+            data = yaml.load(file_path.read_text())
+            yield from utils.ensure_list(data)
+
+    def load_specs(self, loader):
+        for spec in self.read_specs():
+            data = loader(spec)
+            yield data
+
+
+def load_specs_in(search_dir, loader):
+    specs = SpecLoader(search_dir)
+    return specs.load_specs(loader)
+
+
+def read_specs_in(search_dir):
+    specs = SpecLoader(search_dir)
+    return specs.read_specs()
diff --git a/cboe-sdk/cboe/pitch/utils.py b/cboe-sdk/cboe/pitch/utils.py
@@ -0,0 +1,59 @@
+import re
+from boltons.formatutils import (
+    get_format_args,
+    tokenize_format_str,
+    BaseFormatField,
+    construct_format_field_str,
+)
+
+
+from boltons.cacheutils import LRI, cached
+import parse
+
+
+def range2chars(start: int, stop: int) -> set:
+    """Get string representation of characters in given range
+
+    Example:
+        >>> # These produce identical output
+        >>> range2chars(65, 90)         # Dec
+        >>> range2chars(0o101, 0o132)   # Oct
+        >>> range2chars(0x41, 0x5a)     # Hex
+        'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
+    """
+    stop += 1  # include end character
+    chars = [chr(x) for x in range(start, stop)]
+    return "".join(chars)
+
+
+def ensure_list(v):
+    return v if isinstance(v, list) else list(v)
+
+
+def trim_starting_s(model, val):
+    if val.startswith("S") and len(val) == (model.total_width() + 1):
+        val = val[1:]
+    return val
+
+
+def make_fmt_str(fill=None, align=None, width=None):
+    """Create a string used in the format mini specs"""
+    res = ""
+
+    if fill:
+        res += fill
+
+    if align:
+        char = {"left": "<", "center": "^", "right": ">"}.get(align, align)
+
+        res += char
+
+    if width:
+        res += str(width)
+
+    return res
+
+
+def make_named_fmt_str(name, *args, **kwargs):
+    fmt_str = make_fmt_str(*args, **kwargs)
+    return construct_format_field_str(name, fmt_str, None)
diff --git a/cboe-sdk/cboe/pitch/validators.py b/cboe-sdk/cboe/pitch/validators.py
@@ -0,0 +1,40 @@
+import re
+
+
+from cboe.pitch.utils import range2chars
+
+# Regex's for use with our data types
+# Also, x_CHARS for those that want to "if x in PRINTABLE_CHARS"
+
+
+# Printable ASCII
+# Printable ASCII values in the range of 0x20 – 0x7e
+PRINTABLE_CHARS = range2chars(0x20, 0x7E)
+PRINTABLE_RE = re.compile(r"^[\x20-\x7e]*$", flags=re.ASCII)
+
+# Alpha
+# Uppercase A-Z, and space(0x20)
+ALPHA_CHARS = chr(0x20) + range2chars(0x41, 0x5A)
+ALPHA_RE = re.compile(r"^[\x20A-Z]*$", flags=re.ASCII)
+
+# Numeric
+# ASCII numbers 0-9
+NUMERIC_CHARS = range2chars(0x30, 0x39)
+NUMERIC_RE = re.compile(r"^[0-9]*$", flags=re.ASCII)
+
+# Base36 Numeric
+# ASCII numbers 0-9, and Uppercase A-Z
+BASE36_NUMERIC_CHARS = range2chars(0x30, 0x39) + range2chars(0x41, 0x5A)
+BASE36_NUMERIC_RE = re.compile(r"^[0-9A-Z]*$", flags=re.ASCII)
+
+# Price
+# 10 ASCII numbers 0-9
+PRICE_CHARS = range2chars(0x30, 0x39)
+PRICE_RE = re.compile(r"^[0-9]{10}$", flags=re.ASCII)
+# PRICE_RE = re.compile(r"^(?'integer'[0-9]{6})(?'fractional'[0-9]{4})$", flags=re.ASCII)
+
+# Timestamp
+# 8 ASCII numbers 0-9
+TIMESTAMP_CHARS = range2chars(0x30, 0x39)
+TIMESTAMP_RE = re.compile(r"^[0-9]{8}$", flags=re.ASCII)
+
diff --git a/cboe-sdk/cboe/pitch/yaml.py b/cboe-sdk/cboe/pitch/yaml.py
@@ -0,0 +1,22 @@
+# Some YAML helpers that maintain order and don't dump weird like pyaml
+import ruamel.yaml
+
+
+def load(inp, **kwargs):
+    return ruamel.yaml.load(inp, Loader=ruamel.yaml.RoundTripLoader, **kwargs)
+
+
+def load_all(inp, **kwargs):
+    return ruamel.yaml.load_all(inp, Loader=ruamel.yaml.RoundTripLoader, **kwargs)
+
+
+def dump(data, stream, **kwargs):
+    return ruamel.yaml.round_trip_dump(
+        data,
+        stream,
+        indent=4,
+        block_seq_indent=2,
+        explicit_start=True,
+        width=1000,
+        **kwargs,
+    )
diff --git a/cboe-sdk/cboe_pitch.egg-info/PKG-INFO b/cboe-sdk/cboe_pitch.egg-info/PKG-INFO
@@ -0,0 +1,10 @@
+Metadata-Version: 1.0
+Name: cboe-pitch
+Version: 0.0.1
+Summary: UNKNOWN
+Home-page: UNKNOWN
+Author: UNKNOWN
+Author-email: UNKNOWN
+License: UNKNOWN
+Description: UNKNOWN
+Platform: UNKNOWN