* fix test files

* new file

* add new

* fix zero

* update moe tests for forward and backward

* remove useless test

* remove print

* moe

* code style

* code style

* rename

* rename

* remove useless func

* update param check

* update utils and config

* update optim

* update grad handler

* update moe param interface

* update doc

* move moe tensor

* polish code

* rename

* refactor code

* fix test

* refactor code

* update flash attention version

* Support TP (#6)

* add tp test

* update tp test

* update

* remove fa dependency

* update dependency

* update softmax

* update checkpointio

* update processgroupmesh

* update name

* update param

* add keep vars

* add local moe

* update moe layer

* init

* update moe ckpt

* update config

* support openmoe infernece

* update config

* remove pdb

* update ci

* update requirement

* add build ffn experts

* update requirement

* update ci

* update ci

* update require

* update ci

* init

* update moe ckpt

* update config

* support openmoe infernece

* update config

* remove pdb

* support train

* add ckpt download

* update ckpt loading

* use general ckpt

* init

* update moe ckpt

* update config

* support openmoe infernece

* update config

* remove pdb

* support train

* add ckpt download

* update ckpt loading

* use general ckpt

* add loss and optim

* update ci

* update require

* init

* update moe ckpt

* update config

* support openmoe infernece

* update config

* remove pdb

* support train

* add ckpt download

* update ckpt loading

* use general ckpt

* add loss and optim

* update ci

* update require

* move

* move

* remove legacy

* update file name and restore moe context

* update module

* update build_ffn_experts

* update init

* add ctx

* docs: add shape spec

* docs: add doc

* feat: add top_k router

* feat: update init

* test: add moe router tests

* fix: reorder return values

* feat: check z_loss and add doc

* style: rename misleading variable

* feat: modify auxiliary loss

* feat: add aux_loss in topk router and modify doc

* docs: add fn doc

* update triton

* update kernel

* add init

* add version check

* update precision

* update precision

* update kernel in experts

* update test arg

* update settings

* feat: add benchmark train

* perf: use flash_attn

* fix: modify benchmark config

* fix: check flash attn installation

* fix: update config with args

* perf: optimize top2 router

* init policy

* renam,e

* update pp

* finish pp

* update script

* update plugin

* finish pp

* update setup for different plugin

* update ci

* update ci

* update ci

* support ep inside or dp inside

* update arg for kernel

* disable ci

* update train script

* update plugin

* init policy

* renam,e

* update pp

* finish pp

* update script

* update plugin

* finish pp

* update setup for different plugin

* update ci

* update ci

* update ci

* support ep inside or dp inside

* update arg for kernel

* disable ci

* update train script

* fsdp

* update train

* update train

* fsdp benchmark

* rename

* update fsdp bench

* fix plugin

* update benchmark

* init policy

* renam,e

* update pp

* finish pp

* update script

* update plugin

* finish pp

* update setup for different plugin

* update ci

* update ci

* update ci

* support ep inside or dp inside

* update arg for kernel

* disable ci

* update train script

* fsdp

* update train

* update train

* fsdp benchmark

* rename

* update fsdp bench

* fix plugin

* update benchmark

* fix ci

* fix ci

* rename

* update ci

* update test

* update vocab

* update chunk head

* update benchmark script

* update pp strategy

* update plugin

* update bench script

* optimize

* update pp layers

* update zero ep

* ep

* update ckpt

* update test

* overlap comm

* fix typo

* update bench script

* add option

* update script

* update bench

* overlap comm

* fix typo

* update bench script

* add option

* update script

* update bench

* param init

* support dp zero

* fix zero dp

* fxi bug

* update pg bug

* update experts

* fix optim bug

* update config

* kaishen niubi

* fix bug

* embed

* Merge branch 'feature/MoE' of https://github.com/hpcaitech/ColossalAI into bench

* update bench

* update optim

* update doc

* update sync

* fix test

* fix arg

* update ckpt

* update test

* fix

* remove print

* polish code

* update hybrid zero optim

* update print

* add load balance

* update test

* update param exchange

* pass test

* update test

* update test

* update test

* update test

* fix ranks

* update

* test: add more ep/tp test case

* to: add TPOverlap fn

* fix: fix tp overlap

* fix: remove useless variables

* feat: add async all to all

* feat: add overlap ep

* fix: fix import error

* fix: fix ep/tp tests

* perf: optimize overlap

* fix: add world_size check

* doc

* update script

* update experts

* update optim in fsdp

* update kernel in sparse

* empty cache

* update script

* update bench

* update script

* remove epzero2

* fix

* update print

* update test script

* update script

* update manager

* update host

* update script

* update

* update ckpt

* update train

* update train

* dist bench

* update fsdp