Code

Code Refactor for Final Release#28

Code

Merged

pomonam merged 129 commits intodevpomonam/kronfluence:devfrom openwebtextpomonam/kronfluence:openwebtextCopy head branch name to clipboardJul 11, 2024

+7,164-3,837

Commits

Commits on Jun 27, 2024

• Modify gitignore

  pomonam

  committed
  37e5d32

  Copy full SHA for 37e5d32

• Modify openwebtext dataset

  pomonam

  committed
  db7222a

  Copy full SHA for db7222a

• Format the code

  pomonam

  committed
  0ee47f9

  Copy full SHA for 0ee47f9

• Add pipeline

  pomonam

  committed
  58c7006

  Copy full SHA for 58c7006

• Fix tests

  pomonam

  committed
  40115e3

  Copy full SHA for 40115e3

• Add valid modules

  pomonam

  committed
  55cdb96

  Copy full SHA for 55cdb96

• Fix minor typos

  pomonam

  committed
  5fce5e2

  Copy full SHA for 5fce5e2

• Use more module partitions

  pomonam

  committed
  d545d50

  Copy full SHA for d545d50

• Change extreme reduce

  pomonam

  committed
  e423e49

  Copy full SHA for e423e49

• Fix tests

  pomonam

  committed
  d112dbf

  Copy full SHA for d112dbf

• Change pipeline

  pomonam

  committed
  2707a8b

  Copy full SHA for 2707a8b

• Fix tests

  pomonam

  committed
  78e3e3f

  Copy full SHA for 78e3e3f

• Fix tests

  pomonam

  committed
  798406e

  Copy full SHA for 798406e

• Put the whole model into half

  pomonam

  committed
  6213c91

  Copy full SHA for 6213c91

• minor

  pomonam

  committed
  b6eaddd

  Copy full SHA for b6eaddd

Commits on Jun 28, 2024

• Load bfloat16

  pomonam

  committed
  d9df4ed

  Copy full SHA for d9df4ed

• Load bfloat16

  pomonam

  committed
  b2cc4e1

  Copy full SHA for b2cc4e1

• Revert to older version for tests

  pomonam

  committed
  1c48e35

  Copy full SHA for 1c48e35

• Add post process func

  pomonam

  committed
  de8ba8e

  Copy full SHA for de8ba8e

Commits on Jun 29, 2024

• Add dailymail code

  pomonam

  committed
  b472ccd

  Copy full SHA for b472ccd

• Load to right device

  pomonam

  committed
  6aeed66

  Copy full SHA for 6aeed66

• Change to t5

  pomonam

  committed
  60a16d5

  Copy full SHA for 60a16d5

• Lint fix & Simplfiy pytest codes

  pomonam

  committed
  2569fbb

  Copy full SHA for 2569fbb

• Clean up tests

  pomonam

  committed
  db1c89f

  Copy full SHA for db1c89f

• Lint fix

  pomonam

  committed
  98c174d

  Copy full SHA for 98c174d

• Use smaller models

  pomonam

  committed
  8e5eae3

  Copy full SHA for 8e5eae3

• Lint fix

  pomonam

  committed
  d1aecdc

  Copy full SHA for d1aecdc

• Lint fix

  pomonam

  committed
  7fad0d7

  Copy full SHA for 7fad0d7

• Add debug points

  pomonam

  committed
  74ad18e

  Copy full SHA for 74ad18e

Commits on Jun 30, 2024

• minor

  pomonam

  committed
  2c26f58

  Copy full SHA for 2c26f58

• Add analyze scripts

  pomonam

  committed
  46df33a

  Copy full SHA for 46df33a

Commits on Jul 1, 2024

• Add exact modules to track

  pomonam

  committed
  50961a8

  Copy full SHA for 50961a8

• Reload dataset

  pomonam

  committed
  f403704

  Copy full SHA for f403704

• Clean up factor arguments

  pomonam

  committed
  312f43e

  Copy full SHA for 312f43e

Commits on Jul 3, 2024

• Initial commit for final planned optimization

  pomonam

  committed
  2828847

  Copy full SHA for 2828847

Commits on Jul 4, 2024

• Various optimizations done

  pomonam

  committed
  427f6ce

  Copy full SHA for 427f6ce

• Fix device mismatch problem

  pomonam

  committed
  a2d3318

  Copy full SHA for a2d3318

• minor

  pomonam

  committed
  67e2847

  Copy full SHA for 67e2847

Commits on Jul 5, 2024

• Remove CPU requirements

  pomonam

  committed
  87ae3c5

  Copy full SHA for 87ae3c5

• Factors code cleanup

  pomonam

  committed
  8573276

  Copy full SHA for 8573276

• Debug code to track memory

  pomonam

  committed
  346d4fd

  Copy full SHA for 346d4fd

• Add cuda condition

  pomonam

  committed
  3dde1de

  Copy full SHA for 3dde1de

• add debug code

  pomonam

  committed
  27aa722

  Copy full SHA for 27aa722

• Print device

  pomonam

  committed
  57442ed

  Copy full SHA for 57442ed

• Remove reset memory

  pomonam

  committed
  c56e285

  Copy full SHA for c56e285

• minor

  pomonam

  committed
  c78c092

  Copy full SHA for c78c092

• Memory cleanup

  pomonam

  committed
  c135ecd

  Copy full SHA for c135ecd

• m

  pomonam

  committed
  288427e

  Copy full SHA for 288427e

Commits on Jul 6, 2024

• Final covariance cleanup

  pomonam

  committed
  d0154f1

  Copy full SHA for d0154f1

• Finalize factor computations

  pomonam

  committed
  a80595e

  Copy full SHA for a80595e

• Add score computations

  pomonam

  committed
  42e539b

  Copy full SHA for 42e539b

• Release memory after prepare

  pomonam

  committed
  1c603fd

  Copy full SHA for 1c603fd

Commits on Jul 7, 2024

• Do GPU tests

  pomonam

  committed
  743070a

  Copy full SHA for 743070a

• minor

  pomonam

  committed
  1f26c88

  Copy full SHA for 1f26c88

• Set condition

  pomonam

  committed
  9ece91c

  Copy full SHA for 9ece91c

• Use default names

  pomonam

  committed
  614ccb0

  Copy full SHA for 614ccb0

• Modify fsdp

  pomonam

  committed
  2df875b

  Copy full SHA for 2df875b

• Only initialize when necessary

  pomonam

  committed
  c918176

  Copy full SHA for c918176

• Modify all other tests

  pomonam

  committed
  398503c

  Copy full SHA for 398503c

Commits on Jul 8, 2024

• Finalize refactor

  pomonam

  committed
  133f53c

  Copy full SHA for 133f53c

• Add measurement score CPU tests

  pomonam

  committed
  7ec811f

  Copy full SHA for 7ec811f

• Remove state initialization

  pomonam

  committed
  207457b

  Copy full SHA for 207457b

• Add DDP tests

  pomonam

  committed
  b665723

  Copy full SHA for b665723

• Disable nccl initialization

  pomonam

  committed
  4b2f9bb

  Copy full SHA for 4b2f9bb

• Reduce logging level

  pomonam

  committed
  ea1f03d

  Copy full SHA for ea1f03d

• Change contiguous tensor

  pomonam

  committed
  7d9c794

  Copy full SHA for 7d9c794

• Fix DDP test

  pomonam

  committed
  0a4cf37

  Copy full SHA for 0a4cf37

• Add FSDP tests

  pomonam

  committed
  9119228

  Copy full SHA for 9119228

• Add AMP tests

  pomonam

  committed
  c1e19fa

  Copy full SHA for c1e19fa

• Add compile tests

  pomonam

  committed
  7c6e0a3

  Copy full SHA for 7c6e0a3

• Add debug code

  pomonam

  committed
  e0bc934

  Copy full SHA for e0bc934

• Add normal lines

  pomonam

  committed
  f5aa709

  Copy full SHA for f5aa709

• Add reset compiler

  pomonam

  committed
  3dce62d

  Copy full SHA for 3dce62d

• Remove debug lines

  pomonam

  committed
  1dc7d85

  Copy full SHA for 1dc7d85

• Add debug line for wiki

  pomonam

  committed
  1c42025

  Copy full SHA for 1c42025

• minor

  pomonam

  committed
  85f99a5

  Copy full SHA for 85f99a5

• minor

  pomonam

  committed
  3f83568

  Copy full SHA for 3f83568

• Remove debug lines

  pomonam

  committed
  7556c56

  Copy full SHA for 7556c56

• Remove reference to score

  pomonam

  committed
  8f5e80f

  Copy full SHA for 8f5e80f

• Add debug lines

  pomonam

  committed
  37aacd8

  Copy full SHA for 37aacd8

• Check mem leak

  pomonam

  committed
  4d9dad1

  Copy full SHA for 4d9dad1

• Explicitly remove

  pomonam

  committed
  132ac9c

  Copy full SHA for 132ac9c

• More debug lines

  pomonam

  committed
  3cfcc48

  Copy full SHA for 3cfcc48

• Reduce size

  pomonam

  committed
  386b335

  Copy full SHA for 386b335

• Explicit deletion

  pomonam

  committed
  75962ae

  Copy full SHA for 75962ae

• Remove processed lambda count

  pomonam

  committed
  0535cf4

  Copy full SHA for 0535cf4

Commits on Jul 9, 2024

• Minimize size for dot product

  pomonam

  committed
  6645ca9

  Copy full SHA for 6645ca9

• Remove self cache

  pomonam

  committed
  9744252

  Copy full SHA for 9744252

• Change to torch einsum

  pomonam

  committed
  c757145

  Copy full SHA for c757145

• Use einsum

  pomonam

  committed
  77c0e3a

  Copy full SHA for 77c0e3a

• Remove einsum

  pomonam

  committed
  a2efa47

  Copy full SHA for a2efa47

• Fix linting

  pomonam

  committed
  1919097

  Copy full SHA for 1919097

• Fix linting in tests

  pomonam

  committed
  23eda1c

  Copy full SHA for 23eda1c

• Start wikitext add

  pomonam

  committed
  62123a4

  Copy full SHA for 62123a4

• Improve the examples

  pomonam

  committed
  eb11cf0

  Copy full SHA for eb11cf0

• Modify score_args name

  pomonam

  committed
  6de4afb

  Copy full SHA for 6de4afb

• Fix depreciated commands

  pomonam

  committed
  2e2802d

  Copy full SHA for 2e2802d

• Modify default batch_size

  pomonam

  committed
  1868e86

  Copy full SHA for 1868e86

• Fix incorrect paths

  pomonam

  committed
  2a6e1d2

  Copy full SHA for 2a6e1d2

• Fix examples

  pomonam

  committed
  4a7ed69

  Copy full SHA for 4a7ed69

• Make two modules split

  pomonam

  committed
  94c22a9

  Copy full SHA for 94c22a9

• Load 32 models

  pomonam

  committed
  c15ece6

  Copy full SHA for c15ece6

• Try out lambda batch size

  pomonam

  committed
  51f64b5

  Copy full SHA for 51f64b5

• Test out lambda

  pomonam

  committed
  6f5c2af

  Copy full SHA for 6f5c2af

Commits on Jul 10, 2024

• Remove contiguous calls

  pomonam

  committed
  5cf4864

  Copy full SHA for 5cf4864

• Let minimze flops

  pomonam

  committed
  0ad5a3c

  Copy full SHA for 0ad5a3c

• Add einsum

  pomonam

  committed
  8ca4e86

  Copy full SHA for 8ca4e86

• Remove inspect arguments

  pomonam

  committed
  9544c36

  Copy full SHA for 9544c36

• Remove contract operation

  pomonam

  committed
  afff397

  Copy full SHA for afff397

• Remove cpu flag

  pomonam

  committed
  9a5626d

  Copy full SHA for 9a5626d

• fix tests

  pomonam

  committed
  3aa89bc

  Copy full SHA for 3aa89bc

• Remove contract path dependency

  pomonam

  committed
  9c3f89c

  Copy full SHA for 9c3f89c

• Fix task formulation

  pomonam

  committed
  395bbc1

  Copy full SHA for 395bbc1

• Finish dailymail example

  pomonam

  committed
  7209dac

  Copy full SHA for 7209dac

• add openwebtext

  pomonam

  committed
  7374ddd

  Copy full SHA for 7374ddd

• Ignore Attn computation

  pomonam

  committed
  c6efd69

  Copy full SHA for c6efd69

• Remove blank einsum calls with >3 operands

  pomonam

  committed
  ab16f3f

  Copy full SHA for ab16f3f

• Update documentations

  pomonam

  committed
  2ce91a6

  Copy full SHA for 2ce91a6

• Finalize all examples

  pomonam

  committed
  44764e3

  Copy full SHA for 44764e3

• Fix spacing issues

  pomonam

  committed
  3310f19

  Copy full SHA for 3310f19

• Add debug statement

  pomonam

  committed
  7ea5ca7

  Copy full SHA for 7ea5ca7

• Remove debug statement

  pomonam

  committed
  dd69813

  Copy full SHA for dd69813

• Try unscale

  pomonam

  committed
  c1eed9d

  Copy full SHA for c1eed9d

• Increase timeout

  pomonam

  committed
  496b8ed

  Copy full SHA for 496b8ed

Commits on Jul 11, 2024

• Add score computation script

  pomonam

  committed
  ad06d11

  Copy full SHA for ad06d11

• Fix typo

  pomonam

  committed
  37b37d1

  Copy full SHA for 37b37d1

• Update commands

  pomonam

  committed
  041a6e3

  Copy full SHA for 041a6e3

• Disable module parition

  pomonam

  committed
  c6be439

  Copy full SHA for c6be439

• Add factors name arguments

  pomonam

  committed
  05d06a0

  Copy full SHA for 05d06a0