README.md

betamark

An experimental benchmark meant for portability and finite compute.

Installation Instructions

pip install git+https://github.com/exobyte-labs/betamark.git

To update the installation:

pip install git+https://github.com/exobyte-labs/betamark.git -U

For HackOS 1, the current code base uses train or dev datasets. For the final benchmarking near demo time, this repository will be updated to use validation datasets.

External Datasets of Interest:

• ARC AGI: https://github.com/fchollet/ARC-AGI
• Genomic Benchmarks (with dataset for OCR): root GitHub repo, OCR
• MS COCO: https://cocodataset.org/#home
• MMLU: https://paperswithcode.com/dataset/mmlu

ARC-AGI

from betamark import arc_agi

def placeholder(x):
    """
    Params:
    -------
    x: dict of ARC AGI input

    Returns:
    --------
    y_pred: list, a n x n matrix of the predicted answer
    """
    return [[0,0],[0,0]]

results = arc_agi.run_eval(user_func=placeholder)

TinyMMLU Eval

from betamark import tiny_mmlu

def placeholder(x):
    """
    Params:
    -------
    x: string that is a text prompt

    Returns:
    --------
    y_pred: str, ideally a single character {A, B, C, D} corresponding to a multiple choice answer from MMLU or tinyMMLU
    """
    return "A"

tiny_mmlu.run_eval(user_func=placeholder)

OCR Binary Classification

from betamark import ocr

def placeholder(x):
    """
    Params:
    -------
    x: string representing a genomic sequence

    Returns:
    --------
    y_pred: int where 0 is negative (not an OCR) or 1 (is an OCR)
    """

    return 0

ocr.run_eval(user_func=placeholder)

Bike Detection on MS Coco Subsample

from betamark import bicycle

def placeholder(x):
    """
    Params:
    -------
    x: NumPy array representation of an image (dimensions are non-fixed)

    Returns:
    --------
    y_pred: int where 0 is negative (no bicycle) or 1 (there is a bicycle)
    """
    return 0

bicycle.run_eval(user_func=placeholder)

Example of Running Validation Code

When the validation code is pushed at around 2:00 PM ET (Saturday, Oct. 26), please do the following:

(1) Run the following command to update betamark

pip install git+https://github.com/exobyte-labs/betamark.git -U

(2) Replace run_eval with run_validation, for example, the bicycle case becomes:

bicycle.run_validation(user_func=placeholder)

The function static types are the same, so you don't need to worry about integration. The same replacement applies for ocr and tiny_mmlu.