Code

[WIP] Codebase Refactor#22

Code

Draft

maddox-j wants to merge 120 commits intomainIST-DASLab/PanzaMail:mainfrom jen/eval-refactorIST-DASLab/PanzaMail:jen/eval-refactorCopy head branch name to clipboard

+3,196-2,370

Commits

Commits on Aug 16, 2024

• Restructure as python package (#19)

  Show description for 52099b6
  

  shawseanyang

  authored and

  ArmandNM

  committed
  52099b6

  Copy full SHA for 52099b6

Commits on Aug 20, 2024

• Add Ollama inference

  ArmandNM

  committed
  053dc09

  Copy full SHA for 053dc09

• Panza Web Server (#12)

  Show description for 0e8eb89
  

  shawseanyang

  and

  ArmandNM

  authored
  0e8eb89

  Copy full SHA for 0e8eb89

• Add black line length configuration

  ArmandNM

  committed
  498fdc1

  Copy full SHA for 498fdc1

• Add new Panza src path

  ArmandNM

  committed
  aec8d50

  Copy full SHA for aec8d50

Commits on Aug 22, 2024

• Create class interfaces

  ArmandNM

  committed
  39d37e2

  Copy full SHA for 39d37e2

Commits on Aug 23, 2024

• Set up unit testing

  ArmandNM

  committed
  769277c

  Copy full SHA for 769277c

• web hosting

  shawseanyang

  committed
  42acc78

  Copy full SHA for 42acc78

• implement ollama llm

  shawseanyang

  committed
  081bf94

  Copy full SHA for 081bf94

Commits on Aug 24, 2024

• Add FAISS retriever and update Document interface to preepare for indexing

  ArmandNM

  committed
  de2fe12

  Copy full SHA for de2fe12

• Fix missing method in retriever interface

  ArmandNM

  committed
  32a8a39

  Copy full SHA for 32a8a39

• Make thread and past_messages optional in EmailInstruction

  ArmandNM

  committed
  094fa24

  Copy full SHA for 094fa24

• Add email prompt builder

  ArmandNM

  committed
  c9ea456

  Copy full SHA for c9ea456

Commits on Aug 25, 2024

• Add local transformers inference

  ArmandNM

  committed
  6a9897e

  Copy full SHA for 6a9897e

Commits on Aug 26, 2024

• Add Peft models and conditional imports

  ArmandNM

  committed
  30b3dc1

  Copy full SHA for 30b3dc1

• Add Panza Writer

  ArmandNM

  committed
  1d73aff

  Copy full SHA for 1d73aff

Commits on Aug 27, 2024

• Add support to return full prompt from writer

  ArmandNM

  committed
  c648f98

  Copy full SHA for c648f98

• Set corresponding retriever document type in prompt builder

  ArmandNM

  committed
  09f08e4

  Copy full SHA for 09f08e4

• Remove debugging print from prompting utils

  ArmandNM

  committed
  a3c6ac6

  Copy full SHA for a3c6ac6

• Add Hydra config-based runner for Panza writer

  ArmandNM

  committed
  6b67156

  Copy full SHA for 6b67156

Commits on Aug 28, 2024

• rename ollama_llm.py to ollama.py

  shawseanyang

  committed
  cd9b041

  Copy full SHA for cd9b041

• add type annotations to OllamaLLM

  shawseanyang

  committed
  7e3ee43

  Copy full SHA for 7e3ee43

• add some more type annotations to OllamaLLM

  shawseanyang

  committed
  8b1ef60

  Copy full SHA for 8b1ef60

• check installation for OllamaLLM

  shawseanyang

  committed
  73b37be

  Copy full SHA for 73b37be

• rename test_llm.py to test_local_llm.py

  shawseanyang

  committed
  90d59d1

  Copy full SHA for 90d59d1

• add pytest to dev dependencies

  shawseanyang

  committed
  f9ddb8f

  Copy full SHA for f9ddb8f

• add sampling_params to super() init call

  shawseanyang

  committed
  af88a28

  Copy full SHA for af88a28

• add unit tests for ollama_llm.py

  shawseanyang

  committed
  28dce17

  Copy full SHA for 28dce17

• black formatting

  shawseanyang

  committed
  b8fba39

  Copy full SHA for b8fba39

• fix types

  shawseanyang

  committed
  5f05228

  Copy full SHA for 5f05228

• add to gitignore

  shawseanyang

  committed
  91ef882

  Copy full SHA for 91ef882

• black formatting

  shawseanyang

  committed
  d4d0a95

  Copy full SHA for d4d0a95

• add omegaconf to dependencies

  shawseanyang

  committed
  a9f9152

  Copy full SHA for a9f9152

Commits on Sep 1, 2024

• Add FFT runner

  ArmandNM

  committed
  49dfb1d

  Copy full SHA for 49dfb1d

Commits on Sep 2, 2024

• Merge branch 'refactor' of github.com:IST-DASLab/PanzaMail into refactor

  shawseanyang

  committed
  c1775ca

  Copy full SHA for c1775ca

• comment out the running examples

  shawseanyang

  committed
  48ba916

  Copy full SHA for 48ba916

• split dependencies into base and training and add documentation in README_panza3 for that

  shawseanyang

  committed
  b662f80

  Copy full SHA for b662f80

• add hydra to dependencies

  shawseanyang

  committed
  4b2fa76

  Copy full SHA for 4b2fa76

Commits on Sep 3, 2024

• Fix unused num_thread_emails parameter

  ArmandNM

  committed
  d4bd1aa

  Copy full SHA for d4bd1aa

• Add RoSA runner

  ArmandNM

  committed
  8df8f11

  Copy full SHA for 8df8f11

• Temporarily rename serialized document in vector db metadata

  ArmandNM

  committed
  2c221cc

  Copy full SHA for 2c221cc

• move some dependencies from training into base

  shawseanyang

  committed
  e72d13c

  Copy full SHA for e72d13c

• add outputs folder to gitignore

  shawseanyang

  committed
  7447aac

  Copy full SHA for 7447aac

• add writer to the constructor arguments of the web service

  shawseanyang

  committed
  b151fc3

  Copy full SHA for b151fc3

• delete run_panza.py bc its just a test file

  shawseanyang

  committed
  77c1867

  Copy full SHA for 77c1867

• rename constructor argument for the Ollama LLM class to match the local LLM classes

  shawseanyang

  committed
  fc7db3f

  Copy full SHA for fc7db3f

• add none retriever to allow running without RAG

  shawseanyang

  committed
  b6f9921

  Copy full SHA for b6f9921

• add config for Ollama LLM

  shawseanyang

  committed
  d70f512

  Copy full SHA for d70f512

Commits on Sep 4, 2024

• Merge branch 'refactor' of github.com:IST-DASLab/PanzaMail into refactor

  shawseanyang

  committed
  5d79649

  Copy full SHA for 5d79649

• remove DEFAULT_PORT and add integer type hint

  shawseanyang

  committed
  01b4052

  Copy full SHA for 01b4052

Commits on Sep 5, 2024

• add interfaces

  shawseanyang

  committed
  1d05864

  Copy full SHA for 1d05864

• add hydra yaml overrides to training script (works for full training only atm)

  

  Eugenia Iofinova

  committed
  a522768

  Copy full SHA for a522768

• remove my user from configs and add comments to show how to enable other interfaces

  shawseanyang

  committed
  3a2c70b

  Copy full SHA for 3a2c70b

• add temporary inference instructions

  shawseanyang

  committed
  3801351

  Copy full SHA for 3801351

• use command line config specifications in the inference instructions

  shawseanyang

  committed
  f1bf461

  Copy full SHA for f1bf461

• add example to inference instructions

  shawseanyang

  committed
  9c821b3

  Copy full SHA for 9c821b3

Commits on Sep 6, 2024

• update training script for RoSA

  

  Eugenia Iofinova

  committed
  46f222d

  Copy full SHA for 46f222d

• Merge branch 'refactor' into jen/train-refactor

  

  Eugenia Iofinova

  committed
  a4a38d5

  Copy full SHA for a4a38d5

• remove redundant, unnecessary and problematic use_rag and use_thread arguments from prompt building

  

  Eugenia Iofinova

  committed
  62d3b27

  Copy full SHA for 62d3b27

• minor training cleanups

  

  Eugenia Iofinova

  committed
  bd42248

  Copy full SHA for bd42248

Commits on Sep 10, 2024

• add config for peft writer

  

  Eugenia Iofinova

  committed
  7e8f343

  Copy full SHA for 7e8f343

Commits on Sep 11, 2024

• deprecate panza_finetuning.yaml

  

  Eugenia Iofinova

  committed
  090c8f1

  Copy full SHA for 090c8f1

• small config fixes

  

  Eugenia Iofinova

  committed
  aa6a586

  Copy full SHA for aa6a586

• Refactor data summarization

  ArmandNM

  committed
  19e5b54

  Copy full SHA for 19e5b54

• add sampling parameters to ollama LLM

  shawseanyang

  committed
  4fb5c8d

  Copy full SHA for 4fb5c8d

• Merge branch 'refactor' of github.com:IST-DASLab/PanzaMail into refactor

  shawseanyang

  committed
  422a2a5

  Copy full SHA for 422a2a5

Commits on Sep 12, 2024

• refactor configs to get unnecessary params out of configs/base

  

  Eugenia Iofinova

  committed
  21a3fcf

  Copy full SHA for 21a3fcf

• allow code execution during model loading to allow phi3.5

  

  Eugenia Iofinova

  committed
  9a1c9c2

  Copy full SHA for 9a1c9c2

• greatly simplify the .sh training script to take advantage of the configs

  

  Eugenia Iofinova

  committed
  3c44172

  Copy full SHA for 3c44172

• add streaming to cli

  shawseanyang

  committed
  6849903

  Copy full SHA for 6849903

• add streaming to transformers

  shawseanyang

  committed
  764f980

  Copy full SHA for 764f980

Commits on Sep 13, 2024

• update training scripts to process arguments correctly

  

  Eugenia Iofinova

  committed
  50dcde2

  Copy full SHA for 50dcde2

• minor fix for train_fft

  

  Eugenia Iofinova

  committed
  6937018

  Copy full SHA for 6937018

• write the full checkpoint to the expected location

  

  Eugenia Iofinova

  committed
  0dc314a

  Copy full SHA for 0dc314a

• add sampling parameters to ollama LLM

  

  shawseanyang

  authored and

  ArmandNM

  committed
  9fa93f0

  Copy full SHA for 9fa93f0

• Refactor data summarization

  ArmandNM

  committed
  ddd6041

  Copy full SHA for ddd6041

• add streaming to cli

  

  shawseanyang

  authored and

  ArmandNM

  committed
  91a64f0

  Copy full SHA for 91a64f0

• add streaming to transformers

  

  shawseanyang

  authored and

  ArmandNM

  committed
  41d27c3

  Copy full SHA for 41d27c3

• update web.py to match LLM interface

  

  shawseanyang

  authored and

  ArmandNM

  committed
  66cf2d4

  Copy full SHA for 66cf2d4

Commits on Sep 16, 2024

• Merge branch 'refactor' of github.com:IST-DASLab/PanzaMail into refactor

  

  Eugenia Iofinova

  committed
  098ef32

  Copy full SHA for 098ef32

• first pass at porting evaluation to new framework

  

  Eugenia Iofinova

  committed
  0c456b5

  Copy full SHA for 0c456b5

• do NOT split test and train data by default

  

  Eugenia Iofinova

  committed
  2e20f12

  Copy full SHA for 2e20f12

Commits on Sep 17, 2024

• make the json interface more robust to json file format

  

  Eugenia Iofinova

  committed
  298d023

  Copy full SHA for 298d023

Commits on Sep 19, 2024

• fix bug where RoSA FFT still tries to move folder over

  

  Eugenia Iofinova

  committed
  02e0322

  Copy full SHA for 02e0322

• emergency bugfix for streaming to cli

  

  Eugenia Iofinova

  committed
  d4b70f8

  Copy full SHA for d4b70f8

Commits on Oct 18, 2024

• Merge branch 'refactor' into jen/eval-refactor

  

  Eugenia Iofinova

  committed
  226082e

  Copy full SHA for 226082e

• add creating RAG DB to data preparation script

  

  Eugenia Iofinova

  committed
  3314abe

  Copy full SHA for 3314abe

• add test-train splitting to data preparation

  

  Eugenia Iofinova

  committed
  ab96fea

  Copy full SHA for ab96fea

• undo accidental commenting out

  

  Eugenia Iofinova

  committed
  e3f12e0

  Copy full SHA for e3f12e0

• bug fix

  

  Eugenia Iofinova

  committed
  dbcc5d0

  Copy full SHA for dbcc5d0

• add tqdm to json interface

  

  Eugenia Iofinova

  committed
  dfbde00

  Copy full SHA for dfbde00

• let panza writer load latest checkpoint

  

  Eugenia Iofinova

  committed
  eb720d1

  Copy full SHA for eb720d1

Commits on Oct 21, 2024

• add email extraction to data preparation

  

  Eugenia Iofinova

  committed
  d7b298f

  Copy full SHA for d7b298f

• update panza readme

  

  Eugenia Iofinova

  committed
  924033e

  Copy full SHA for 924033e

• update env preparation script

  

  Eugenia Iofinova

  committed
  ea5ffea

  Copy full SHA for ea5ffea

Commits on Oct 28, 2024

• slight refactor of runner.py

  

  Eugenia Iofinova

  committed
  ca4b690

  Copy full SHA for ca4b690

• remove some unused .sh files

  

  Eugenia Iofinova

  committed
  233083e

  Copy full SHA for 233083e

• qq

  Show description for 6f94379
  

  Eugenia Iofinova

  committed
  6f94379

  Copy full SHA for 6f94379

Commits on Oct 29, 2024

• make the first part of the data preparation script optional (in case of manual modifications

  

  Eugenia Iofinova

  committed
  8ce3c4a

  Copy full SHA for 8ce3c4a

Commits on Oct 31, 2024

• Edits and Bug Fixes

  Show description for c1f36a8
  maddox-j

  committed
  c1f36a8

  Copy full SHA for c1f36a8

Commits on Nov 4, 2024

• Add additional clarification on username importance

  maddox-j

  committed
  bacbdf7

  Copy full SHA for bacbdf7

• update data preparation

  

  Eugenia Iofinova

  committed
  6be5813

  Copy full SHA for 6be5813

Commits on Nov 8, 2024

• Miscellanous updates

  Show description for 302bae6
  

  Andrej Jovanovic

  committed
  302bae6

  Copy full SHA for 302bae6

• Fix function address

  

  Andrej Jovanovic

  committed
  c13aa07

  Copy full SHA for c13aa07

Commits on Nov 12, 2024

• Clean up code TODOs and revert to defaults

  

  Andrej Jovanovic

  committed
  fee7cbd

  Copy full SHA for fee7cbd

• move top-level README to default location

  

  Eugenia Iofinova

  committed
  99f5c7c

  Copy full SHA for 99f5c7c

• update the scripts/ readme

  

  Eugenia Iofinova

  committed
  1238d39

  Copy full SHA for 1238d39

• remove useless assert

  

  Eugenia Iofinova

  committed
  c0a94a3

  Copy full SHA for c0a94a3

• Merge branch 'jen/eval-refactor' of github.com:IST-DASLab/PanzaMail into jen/eval-refactor

  

  Eugenia Iofinova

  committed
  99a0b68

  Copy full SHA for 99a0b68

• Merge changes.

  

  Eugenia Iofinova

  committed
  3e2203c

  Copy full SHA for 3e2203c

• Once again, try to centralize the main README.

  

  Eugenia Iofinova

  committed
  1e62597

  Copy full SHA for 1e62597

• Update the README

  

  Eugenia Iofinova

  committed
  71a85c9

  Copy full SHA for 71a85c9

Commits on Nov 13, 2024

• Update README.md remove resolved TODO

  ohaijen

  authored
  73308a0

  Copy full SHA for 73308a0

• update hyperparameter tuning guide

  

  Eugenia Iofinova

  committed
  e5a9e44

  Copy full SHA for e5a9e44

• Merge branch 'jen/eval-refactor' of github.com:IST-DASLab/PanzaMail into jen/eval-refactor

  

  Eugenia Iofinova

  committed
  49258b9

  Copy full SHA for 49258b9

• Refactor panza3 -> panza

  

  Andrej Jovanovic

  committed
  b3bc00f

  Copy full SHA for b3bc00f

Commits on Nov 14, 2024

• Clear ollama and web use-case

  

  Andrej Jovanovic

  committed
  be76c39

  Copy full SHA for be76c39

Commits on Nov 18, 2024

• Update README.md remove confusing period.

  ohaijen

  authored
  677dd8a

  Copy full SHA for 677dd8a

• Update README.md Add instructions for quantized training

  

  ohaijen

  authored and

  Eugenia Iofinova

  committed
  e1e8e6d

  Copy full SHA for e1e8e6d

• correct README for quantized training

  

  Eugenia Iofinova

  committed
  525d0e3

  Copy full SHA for 525d0e3