-
+
*Figure 2: DeepSpeed sequence parallelism strong scalability evaluation
at different sequence length and GPU count.*
@@ -220,66 +231,89 @@ at different sequence length and GPU count.*
### Dense Attention Evaluation
-Next, we evaluate DeepSpeed sequence parallelism on 30 billion parameter
-dense attention model and benchmark against Megatron sequence
-parallelism on 64 A100 GPUs. The results of these evaluations are shown
-in Figures 3.
-
-We compare DeepSpeed sequence parallelism with Megatron-LM for a 30B
-model running various sequence lengths. For our evaluation we chose the
-sequence parallelism degree and global batch size that produced the best
-performance (measured as throughput or TFLOPs) for both DeepSpeed
-sequence parallelism and Megatron-LM, this we call optimal (batch
-size-sequence length) configurations. For DeepSpeed sequence
-parallelism, we always use a ZeRO parallelism degree of 64.
-
-Figure 3 shows that DeepSpeed sequence parallelism consistently
-outperforms Megatron-LM for the sequence length that can be run with
-both. In addition, DeepSpeed sequence parallelism can run longer
-sequence than Megatron-LM. DeepSpeed sequence parallelism performance
-advantages are two folds: (1) DeepSpeed sequence parallelism in
-combination with ZeRO-3 fits more sample than Megatron-LM because of
-memory optimization leading to higher throughput (2) DeepSpeed sequence
-parallelism benefits from efficient all-to-all communication relative to
-*all-gather* communication as applied in Megatron-LM sequence
-parallelism.
+Next, we evaluate Ulysses on 7 billion (7B) and 30 billion (30B) parameter
+GPT dense attention models and compare against Megatron-LM's sequence
+parallelism (Megatron LM) and Colosal AI sequence parallelism (ColAI-SP) on
+32 and 64 A100 GPUs respectively. The results of these evaluations are shown
+in Figures 3 and 4.
+
+We compare Ulysses with Megatron-LM and ColAI-SP for 7B and 30B models
+running various sequence lengths. We chose the sequence parallelism
+degree and micro-batch size that produced the best performance
+(measured as TFLOPs) for the three methods, this we call optimal
+(batch size-sequence length) configurations. For Ulysses, we always
+use a ZeRO-3 parallelism degrees of 32 and 64 for 7B and 30B models
+respectively.
+
+
+Figures 3 and 4 show that Ulysses consistently outperforms Megatron-LM
+and ColAI-SP for the sequence length that can be run with them. In addition,
+Ulysses can run longer sequence than the two existing methods. Ulysses
+performance advantages are two folds: (1) Ulysses in combination with ZeRO-3
+parameter sharding across both data and sequence parallel groups fits more
+samples than Megatron-LM and ColAI-SP because of the memory optimization
+leading to higher throughput (2) Ulysses benefits from efficient *all-to-all*
+communication relative to *all-gather* *reduce-scatter* and *ring-style* P2P
+communication as applied in Megatron-LM and ColAI-SP sequence parallelism.
+However, for dense attention at long sequence length, the throughput is
+primarily determined by local attention computation due to quadratic
+computation complexity of attention, therefore performance gap between Ulysses
+and the two existing methods closes for sequence length that can be run with them.
-
+
-*Figure 3: Evaluation of DeepSpeed and Megatron LM sequence parallelism on 30B
-parameter model with dense attention.*
+*Figure 3: Evaluation of Ulysses vs Megatron LM vs ColAI-SP on GPT-7B parameter
+ model with dense attention (32 GPUs).*
+
+
+
+
+
+*Figure 4: Evaluation of Ulysses vs Megatron LM vs ColAI-SP on GPT-30B parameter
+ model with dense attention (64 GPUs).*
### Sparse Attention Evaluation
-Similarly, we evaluate DeepSpeed sequence parallelism on 30 billion
-parameter sparse attention model and benchmark against Megatron sequence
-parallelism. Results of our evaluation are shown in Figure 4. We observe
-similar trends with sparse attention as dense attention experiments. We
-observe more than 2X throughput performance of DeepSpeed sequence
-parallelism compared to Megatron-LM. For memory saving, DeepSpeed
-sequence parallelism leveraging ZeRO-3 scales to 4X longer sequence
-lengths than Megatron-LM.
-
-DeepSpeed sequence parallelism outperforms Megatron-LM for sequence
-length that can be run with both. In fact, the current DeepSpeed
-throughput is bottlenecked by the local sparse attention implementation,
-and as a result DeepSpeed throughput decreases as the sequence length
-increases. We expect this gap in performance between DeepSpeed and
-Megatron to increase further for larger sequence lengths as we improve
-the performance of the local sparse attention implementation in future.
+Similarly, we evaluate Ulysses on 7 billion and 30 billion parameter sparse
+attention models and benchmark against Megatron-LM sequence parallelism.
+There is no public implementation of block sparse attention for ColAI-SP,
+therefore, evaluation of sparse attention is in comparison with Megatron-LM.
+Results of our evaluation are shown in Figures 5 and 6. We observe similar
+trends with sparse attention as dense attention experiments. We observe more
+than 2x throughput performance of Ulysses compared to Megatron-LM. For memory
+saving, Ulysses leveraging ZeRO-3 scales to 4x longer sequence lengths
+than Megatron-LM.
+
+Ulysses outperforms Megatron-LM for sequence length that can be run with both.
+In fact, the current Ulysses throughput is bottle-necked by the local sparse
+attention implementation, and as a result Ulysses throughput decreases as
+the sequence length increases. We expect this gap in performance between our
+method and Megatron-LM to increase further for larger sequence lengths as we
+improve the performance of the local sparse attention implementation in future.
+A noteworthy observation is that the decreasing performance gap between Ulysses
+and Megatron-LM observed in dense attention evaluation is less pronounced in
+sparse attention evaluation, because the attention computation in sparse attention
+is less dominant compared to dense attention.
+
+
+
+
+*Figure 5: Evaluation of Ulysses and Megatron LM sequence parallelism on GPT-7B
+parameter model with block sparse attention (32 GPUs).*
+
-
+
-*Figure 4: Evaluation of DeepSpeed and Megatron LM sequence parallelism on 30B
-parameter model with block sparse attention.*
+*Figure 6: Evaluation of Ulysses and Megatron LM sequence parallelism on GPT-30B
+parameter model with block sparse attention (64 GPUs).*
### Convergence Study
-Lastly, Figure 5 shows convergence of a 1.3 billion GPT model at 32K
+Lastly, Figure 7 shows convergence of a 1.3 billion GPT model at 32K
sequence length on 8 A100 GPUs with sequence parallelism degree set at 4
for both DeepSpeed and Megatron-LM sequence parallelism. For DeepSpeed
sequence parallelism, we evaluate convergence with different ZeRO
@@ -289,9 +323,9 @@ there is no (negative) impact on quality of trained models, this assertion is
validated through experiments and is shown in Figure 5.
-
+
-*Figure 5: Convergence evaluation of DeepSpeed sequence parallelism with different
+*Figure 7: Convergence evaluation of DeepSpeed sequence parallelism with different
ZeRO memory optimization stages.*
diff --git a/blogs/deepspeed-ulysses/media/convgZ.png b/blogs/deepspeed-ulysses/media/convgZ.png
new file mode 100644
index 0000000000000000000000000000000000000000..324f47cd61bd37d5a36b455c6c4d421d1e909f14
GIT binary patch
literal 38034
zcmdqJcQ}{*|2O_dg9>G2lOjTqT|@~L4Wg0=$<8P;GD{*+At5s|N
2c!;&
z3hgqrv$L^1EG%sKKfiFm`j(0CKHUL#T!hNzPvuU~Lcdf|yu3ePm-v-HF+GAyzh(Cp0m|Kv#zk!1-A~s{$)x-2+a^-GJeMnMGI?mId+6$>p&t>>wHNmJ
z#_qbmU4fGO^n%Vh-^=V({(t{6K6D3kbMLFf?-mztY<)?|EpQiKUeosF
z%a=)-0*~1boi~o`&}$Y5_RX;!dai9!6?zb#k55cYW!E0E9gI2uI{W6&FNKe#OGBq@_Q9o`!^!x278=%kgt7#s_Us^U>~4FX#L@
z-u59v)~P{%b+9^opODb2OQ~0tec#SkR#Y6crzG_c4t9r3;2CiX_2i}nBWDGlQ6pdfmR%``lR54XqE
zq+H1gJ!rV8prAnO^5q|cgUYE_^QG;_LkrH{xFK}ho2IO*ivv&2DtY7U*ML10RaKu1
zx8YJ^6YUu#U%ouJGdKFXQRCdrxzP)*i+s^8_f+EhrUxo{+Y9EJuV`s&2S!D`({mWE
z<6fGdcoY}6Co?m1!^Vy2?(1%z_F3LvYom@-4%f%MzLJF*vU~l@DO}8iCN+PamE`8;
zw)A6JHRQm^*jt^P^X@t|H8pq&cSc33_4_Sl1zl>OF)tCBIQ
zOUnyWPZAUPE?&GSZq*q|E5EYcU)iwEZLB#(r@-YhmuG6Ku-d$Y2jgN`kDNTy0UiC%
z_o<{09r`gm{I&EUy(&+VjQw~FZi>$gtgX)-J?gQ>e1Cgzd1osbCHG{idYp3ltpSyt
z2XvAOA6qZaPptg;`Q%1N=0w4S$xuz?>{!baWq;O#CN=ES)6;*ZD}_fobHvIYbA0Ec
ztCe4u#kw(R|7a%rtBB%Ne0;ifk6dhAoLNWaelIVt*^%$w_wJEjb@{E-+0~VEXHMWs
z<_#<^iW4WwMGYybsg3$dAIi?w9T(C{^T35>7KfrgU>F~B$v?(?mUz<+&9&&6bmiV&
zdNr1Ld#t%W&%t`MIc4i`V7S}rQvEw!8ROb0|FAF?ZfF6!xg>8e
zp9z;VT3cD<_dWl^R_glK?bohd)5^aiq553t<<WauMrp*>2Yb-3PQ>HT@Mq
zr_P)is1z=IbtTJc>1Xc#{rmlKmxH(ZPe$Dt-8b5jnkp|fKR^Fz`kP>DN5{-iwEJ|o
z^He}^a9KH*`?jLDmp^>1s_Oq1EGTsO{f3xM3q`>tu|mr~zkd8#Tuj$5CV$K!^Chdt
z?NN9*+oPTP$$P89#AMh0{QPL%Y+}a;QUI76WvS<0u3j)5e{OgOg?hMY(&b)klCs^d&K4Bj?l)KhM9l1jG
ztg*3CbnK(qcjcS5wi*o$4YOlsqhw!)hK23>`NjWxs;!-!X2RJ>?Lm7C_B+xyOr=YC
z_Sz)_gO`1Sg9);KdOcOHUX5~FeZKz+!%?TH{Ds-E!?#CYeY9w)iCO(o;>~#Ua{4u|
zSbpEwCiCEgGvNZROS9@9wNnbNT3Nm88yZTS9c@xAoSa`|kucZvx<*Y)D}2+B%Y7xC
zod5Ul-*l^nYBD$K{^Yc2>|C^5N+^cqE$x;q=SfCw?~Qop!@|R_T
zEyk@l@RVDh+dORNe(kz97X(+UcM$~*MWVF1dA?v`&%S*R>G{rz7<}H~X)tgxMY}%v
zlCazQs>AAGK0ZFIqS6eD7L~5%3C@DFw6wfjw+YmWSFc`)7=2^#48O9pFm>*Oadm6%
z?PRwupB;Oo@67Vwym@o!wnfXkvTxsf?^0|gx9KaTx@Ks|a^#Mt$W6k@mIULIN;Y0nVEt*#pxkqtYG
zN?CX7^mT6G76?IaXjF(h@njHp8~p6Gm0jA#qdx9LPs>D4LDYd=yJ8eC&kR=2J)|!>
z!R4x`smXBl>Q%epy1QrS9YFVLXliVgICf58;cdJJ%|H{ftEoukt=FOXrI&kbR`t$qe&D*ynw1_Kg*|J5Y
zw0~&m(%Qo%CJc}|gBMoNbDuqj55IW!jKOn>npM20q2Yb0eNDunlHOv^n|5}k#l>Wo
zE?pvaqQ@59D~snGX2c4sJ32agS0`T1ceLm{Wr*aFxOOJczA%t>ghU@$4RG
zInSu*_og|`7-w4d)S#YSPSYoIU!7yZm;E%vEPLb#G2$r2KPR$!22%^yc5`zdbKlPz
zbM7#)TTvJ_M(x*}Tszn2-**XXr|b+?DZ@?FM>(DJmXJ^N5qc!c!KG@vc
z%r0Z^C)Mp3_l@rD`^jAw*9xvSpd}q~p5M;NS<{?yr7hEv4NGjG{sfm1iuJ9{@3#k^
zB-$jmwYBwOhso25iW`JK{URS6Tv)i&RbIZ?V`iu}gkQ#Cl6`KpsidanY++4FiRV6z
z1Y#kdKY!j7wHkB(ozH``|4Qa7qsr%>W@==J1%lE1aHH+LC|~AU4>PChth(oWTx+b-
zn(g4V4H^ljf2=Idyw%OydF9HLP1~4bgst%nNHjDwv*T?uR%go#W6!@<(e25;X~>{d
zTw6=)?!NAxF#)uJYSfx$K!LLUJ@v_xC#g2<>vmc4Grpo?VpyV_v%~eCWsjJ|+}B);
zt6wNNIywfgu-_NCAi6n(hSm%pqNKPxi%c<#}+fTIDjwe&;FclP!$f=mgnd9iHV7EZXfC^d!*Va
zd(GJRONVt~HJ&5v#>+fST0F<2EwF>EDg+M33FPHvG0;O9$scT}W|
z16oH#Vzk?q;e@MUl9sGk)FnMV(S3c_+*W>{Xf&4q651~&xpnJSRLD|51*$Duu3?ib
znd!T@x;iZmgiw#><>b`oJ3H(YzFLLrG9SFA`0(e_e24W&ODflX?T`AXO5&z<9?6$d
z={PwbdwNn7uFiKHiL703LT^4Q#aM}Ln4XrVU*thXfEQrV>E9LmxUV<)^>yX-v5i^807GnCp@qkf}U2(rb+-be_kk$Y{Q)zv_At@92lz8E@N+}Fs#$B=58diy{`-HJ{2VCWZ;#z=
zb&f)#IWRQTRD}Ap)1*CXXv5A7?Ud9k+o7}ZwnH_({{H@by}dEpm_?2ra-M%%@F^zd
zsMAD7%QNgd!P}#a>f__%fBzq7s;MQ%@pr7KSp!czfV6K$Mn>QG_*=WFZr!#VTNAIE
z+V%{K;G&|V=mjb_?d>mq@neb8$+;EZm|Gble2EkfP`gQ^d?oc~`4hq854Nf6=;$y2
z0hE=OpJ0)5$$pPLPB)Yo`#t2DB(+~G|2JN#P1FtV|+m3{xN;j}u}5}cMMvQh1G
zNy*$CR&p@ffZ*A)XSc*e(Sc3ysmKV+HMqNaeHQEZu~S!BatR%nRG~6
zn)whb@no8c*!}qZrKy`5Jj9kq5`}afBf2r=?
z*tE3mDHk7wgfM|%e4#nvqk{JD>E*RCL|88YKLdj?^wyj>sUmcL`(EWc$M5fZTI;mf
zKmK+l`L`y@X9DhFD=Er9`BY?IO@#n!GBED@^)4Y-~zxOGD-Ba}n
zMJqugwid;!hP>gA&5Psr?TqBZY?f43K9aC#PP<+~cjLwl(%0JBAyl!?X!T#~gU_ln
zh_g};4t`nAotvHp`F$qUxJf0uM|;x
zeg#T(%a4{FJB5XXC2zFG=w%s{Z2FX+Pql4ZW9oPF3@{K9Fb-~L+LbE-E+W<1jCQVq
zK}*lfOmJS9T=`W(3wkOlDoQ%3s%nTriS0;2??Xqa!0qQI_)dowO?Kri0{N)F*2M3E
z|0$v!gM_1od7^_D7#WonxGwYa^Xo6oj%4TL7+6}K2|1waiAsGoKyjiLqaV;Q
zGM0U)m8%5VKjiX9gcR+*b_g}SFU%y08|eDgE1plEKFv<_xXhdZy*z1WC!V66(FaO5
z6r$&bp6%9)zZ5rXxG(3rWHU2(#&fmig~THQbnQ9uSl`O3=GCd7$tCvHp!j%R!!LdZ
zrL7@vEEsjf4!+4aUKDJAhyS#`wj5l*HP@f@)p3DB=R1RazN}BTo^|r9>lW_Q8dn1z
zZ0_DF_N}1St#0#`%DiRPJEN}?9mi6#vRxKMNubsy)h{G4Nu*r)#-ZBiT40_LbXSr;
ztHiaQ0#}!rYAJC0-_=q*+qZ8oB6!}>qen^V>grYDQs)Yz(Zxh;2T!jqGwHbmpsu1(
zobaaMumN6eXrRO50Pj^TT`bfVHkPn9CGzH6(GFP1^
zBwX{HW(IdKFl-nO{JOTf926bxUOUrz=7sn}C>+z+ceYRl0FekhGO#31zxRlT!;h4#
z?%iTy8ihQ2_g?(wwrP&-RDYZt$x&Hnup!z)&`2Ju{yT_nL!rC7!tvu&R8&NZY)rm%
z4&(oM<1Jg;k)~uOaM~rKGGE5hvNAH8wG}7MJHydQ_+c@ej4e>OMPGPAnSBW6FY6?Zj+-|w8;l0)&gZ(9TflT~`2pjZue$bXnx
zDf#GIe`_&+;kVoMUB;q8Iotc>KBTMJ&U?x!cPtbh#Bv62*rawX!FzoC1}W33O9iWS
z+qcQV^{FTFYoQ>u5w&v5e&Dd6#^ru2V*ElvZ`GV-Wb(F5jk#ae(i+A??odl^YDhCU
z?m6-;A~JI7PDjw*Gkx;wi?k-SQBAZ)TfbkhpfAGpLC#~`xwyD$K^txC?Z0$%gojI6
z2R?mT1Xx0vWopmO%L{3b)N5jGOgY}2-BR(QaC;Wp7WV?lXHj}q682gOKH8{?U2`ID
z)8{5SLOy?1KoNCaV&0odEBlKCeTRpKXW0Av`o3MeJbK*MyY#!(yPuNfz8nCToQ5Dp
zmdhiwckhGEoQ_BJ%CWrsm2Jx%Q5pQOJ+P929HeqJG&L2kU%&2^7;sQZD)Cg%UdA_9
zo}|g+q0Bs)BklxuEW$j_jLd&l8D=iBNoUa_9pcKcFse$ct{^i
zYTrq=DEe`TJ{Dd`iLK349G%KN`FS$>Hw~-tcp4)V4x*dC&nrF8QdTDCft(^fO=BtbfgZKW7VKjN9E^pSqoQMCT6FL#R5H
zz`8joamV`l$HppO9{0|+8==G6G@Ad`*4Fm>^H$+s?G~vcGTd@Y!+WrFot8!trxsoM
z2L^&2Lyw{EfI+1f=R!OSeD>^mT_ykNODShao}QkS`6V}?iCvI08}2$E{(5+|-8}Yy
z-9YiyElfR;$4hiQMS}FMa^HVwYh%OzVI0f7w5ch04`5F{%mpx8_$03`
zll08pmaBlYOHgnt5I-(mxmr7X)pf1JePn5b?#7n8ykchEQ{C&=7!*RIR#sg4e*XMD
zGm|%zr(zL~vXTxOr;?X*PBVX6ZtB9^s%~V3ZjnThZCv!;iZfC|I~<13?K!dM{&tBT
zOx;62zkW;wAp^Rg)?oU&TMekxjm#&u+gM(+e7W~Sl!Q)2TucP_CNvmEdN#I7Xb?G0
zGkd$cyFV?BCKD|C3yrFez#b}nYIV&X>TJKcxo=KOKG5m|c~zeio!!uz`+oh`?Kac~
zU5ZkNl4ks6&K>$|2dHzNB3bw->pmdweMWuxlA%AlKY;?xMTw6u?(eWIQD0c9iXQW0
z^ZWa`I6kKow3RSBkjvFF7c@=DMIgQv=Ri{PuC_V5+UJ^dsjP?Shz}dD6CbkHJ|MNm
z_p|)>XEggcH^p3ik=a?^NyUBt^$QULaugC#d*(_ZsojSEi6^%D4DhjlQWNkf_*WSH2GelrjAt67gvxJZZ)vsU$bNtgEF
zYOk$KTA4Rk^6V#K!7DmHIcexg@$q>B(xib|EZhHc_tL75>1GHIu2o~}HHM8JA!Y3Vv}LpT)VMd*MmBKjn({j!>x
z$6%qqmZu8GvgjEZmB4b-va%9^6#Y14o}dv+)^Zt*4eI`rkTz@qk`LCJ%c^`?JNj_1ap
z(LiMsL{}o$hm9kCqxI?6uO}aK9{Cn-UGNZmAV&2p6g5InOgtBDTpz0hb(-R7KtOTa
z6FC)q{aq&>vm10|TIv+KMMAixc>3&FX_9(esJIzjeSJN!kI40}PeoIEXtr!=;8XLB
zO>>yKin7<5Wz9v(bC^mv=D7(N?UiB&?n(lDJgK76P`KIW(IZ2EV-yxI$VO?-3s!)5
zT0d<642E0Zf;S>*-F+H0d3G%EO!${#&&|*XsWJc8p!~qU32AL@;L>~=7FO{>!h)3c
z@#BVGxI*{6ybeN?0tnwWnwOV{*@y7*4#VOEw@ad9yTeU)e(^=~hWkzPnz
zzOjx*)2@4Y+^p?AFM;$LLR{yXE^XMfNfesXYA)b0Cujovg0D3-3EA19+{YjE{`^T1
z({t7gY-e#YZ>r8zTv18Mz|5=y(k~W8jCD@|Mshm?gQocvEGQBN5DKmoKd8n12Mr`zf1=zh&My4oApL{wm1QBqMIl;)#m8#PYN
z?wVOFr9U@ZdmG-BZ*1Ksr1%#T-KlRSL%=zCVAoaDkUaKKthqq&YjN@2s;VlwF)>k5z;K}ty4V5WD#ie~vy)x$;%d@v
zk6msmWvjfzng|ctv+qD|
z3)6bLK_?c4M6^mg+XIO&(+@q@<)))7LaGjja9;u!QK!*J`LO$KGV+Z`!FvuD7EG(}J_gX^HL&4es
zLV&$ZNlV)jawhU{Ferhjc_aVsJ$taR_MpXrEyUKF66*ru`4IF_RHHPE+<6U+%1_4ST=^g?J&MNGdE`h9gK&MFGQx%g%6dm7aZIL>K(xS(};*0X$(7e
z5_yC-`R*=hYl_3aiyNpC@p**7WSH?Vr*<2JSla(
zJ?>kZhw-=PZnPIi+`P6+T!Be=YkvGPudr}!>&q2EZ(6R4@XzBxIt3m-eyjq8Cm6HN
z+DQ15kP?gNEb&n?$fV$3OAj#N8^(kYORWr&7?d$hP`U(pdGrwh13Jn}OZm~zFQURSK-G*#WBCSV$B=R{IpT=(#gXrcPx?kj-w=Pdwzl>S`||SS
z@A&xmeFDKMELaq7T`+s#US{41JEjYWWmFN!yR=Zawv^oP>J)uhXQvMQNp%J#b=V9k
zAFaCDW4ZuZX9wcRzUe|VWmHms2Ie-9ztF7#nVb==v8Z8rU7%Y^@86ic|p19QZcNW~-
zJkHLQW#!}~`4(0wX72k*U&tva!T^oo
zknP^JtElw{B(<;NA`(yigEvZzzP72kc=B*2*Mt1)<4@6jv~%r70@IJui1@mvj_=(6O!D8;b5z(KCyiK20x4vP0Q&nQt>0XbwYG!?OM3
z>p(}vYJ$EbSoTtT=$c0n0L|wcBRhK))|$#&T59UDii+3YejdyODvENQzr5-UYY$6z
z(kTxYOVWHJ8dAv^aYtzP?$0ri$;pCGj?6Oy;(szea`0dPc=->Avf%A#&S
zHo`L=g$IaE5vTNM=gQjh)Y1e%6_jClW%|%FBn#MP0}QgL_byuU#v>th%->EVMy(p3H7{pZ>Qzsd{!>6#zvXRA)EMs2>{#?PM>M>TNV;n
zepA-DlPC3u>t24m(MIQi{L@Hf!P2N8Q2^!L3*5S3^ge+x10U?ViHU}eP8FchuiwAP
z>Z}vR!8lwNx*SZ!SpiQ#ZLXL4>;N~Y%d*x38l>E^MIa*u1%U)Cmz|f#_RK6p8(P?0
zMF%!oTZRP_pith=#Vd#^5vT;1o3}>wpIvRHmBHU_%XgMKcI+5}Z)46IVM?W()1G>enwom7HWKi9-@bhgHyp-W4@S8ynGyB@
zGAG-FMeHgFmsfStHF)c2z>*q(VuCy$z*0xV5EgeghLD<3VgV2BjZk
zsf7i}nSJZ0X!WTR#cyouEc&5b%rUlwyHvg{w8O@9mOE}DFBUNj#J7af<^Ij5b!^qrb3ePLl>Py|1P@6QdV*Tlr+g_Ly-
z=<%;#zxWrV_X`Puqz^=S`|=A2^i}XhmsVGwaeuqcz|8EU9LLSe8(*5fZcDl3PJs5L
zB>vSyL~B7N6CR(@%J@CP)klN(6;KpLbVCc}(
zhf3evjjn(k$3X}ekknAv$O$SSA(2rv)`i@aKCS`)y7Z$TMGmIS?ecDUJv}{GkxI92
z-O4#$?LbIa=+PB~FQu%!siLA{ld6FMHztv&wum;QgR4MJCJa7MVUc0oRX+GyQBH#_
z1VHARUqvf}p&0`%Pf$8!S&8z5#I0#v%-uEZvW|x%0eenpVI*1ozwFxkTbRF3ud=F2
zlf)|~_TBU#1B2N!J!i>FHJijW)*@u;O>Y886n^?7JAkVZ5ybT4i>M1y*gFCs1L8?y@GXpbi|+EaY@b9~>($m`SLl4soA
z2yTHOidOpj_vdwVb%U{J1*EJx>$09ae5j5+^cJ*zpQNOu3byn!Xu15_DHn+-NJHW|
zuCk`43&h{8tgZ$le(}EZr>d%|hNeKU=(SSH)m%Vm0gv}L+C)8b0qVXHoM8${u}kr2
ztG+ooIT%A&+_8{Xu9$V@W*p0d5uJlu
z#pPZey7u~0t@G_aYm$<|v6|Dae=@X?g#a@{t7V-#d_Q=P_V>x_+)SMB%024xl^-3a
zRL>t68lvaodIGrtd9*Jyi7InQWD)8tBoJm926d%x%bzZVOj#QSh95P*%YKPz7uBOz
z+YG&T-{0P(a_7!rENAq)H1Kig;))sERQmg4rp;)0e%Qse5>a)<3_51!llaB|CK7<{
zTeCloY+iY2NYk%-t~8;bg`)$&eIHz)e`!RXhsE8^+IHM?Q)X!gCYH;xiwR&K+a9zb
zZKrAfMQ^qPxlV!aIArwuP5)AXE@0-ZT4Bi*cFomE(M;}bU+e$=K`#E_`PbFuV{Oe9
zZOV@oi4U71c|0=ihgq|cBCJiPmWta6LL#ysK$q^_T?&3lig1Om(t6WQVZ`U#vv6fq
z*}6Z+-pPzz<@+jj*K7M;b68BDCwunYW~6yXGm`*uenY)wJLI
z#Dk#8@-RBrZ^YA-uKxd|6d{84aR!Qt_m9ayW#n`oKZu5?8*7CC4D{j8-@nsiO(G)e
zI1(R%l!9226mr{;SRX`2azJK}2Xsc48UB}0sA^qF${Bd&+-z{HPvWlu6f2sOn|sZ|
z!nr4ljfLgZ`SXFrbQ~Q1$lg$F+E&!jvHUL60S++`ALZ7qaR8Zcsx`E6InInP#at378Zt31N#u79Ph}Y#SqluzUKrM(wD4jWm}MmLS+%rl82r4g@}q&1o4e@=gT091(05*|ifr9I$5+81xiGc#KY=}8V0b)1Yn!X2Hgg0|&
zaPUE1o(vkE5pX;#{9BWqhg=u2--#Xv1cBtikBNyh@Iv4Q@}V~w0VOQuA|#oBi4-?)
z^uex7Lw|upAbnd!Ie(2BZrG0am-3{?QWC_-0a6hTBH(u3$d>__epff{+df``i0aA
z+=qm33<@J}yRxz}Ipl8zEVZ>GM|E`A5g@t-0znuHu#7$+L$oEj1kV(NHkXd%By5&A
zJ$cJ^aT-q9uUOjBw?`R3f#RURmwx&3JU6nc>NJLl!UK$qgq6nw4Gw8neT1jfdSKfe
zH2ji=m2XbVe;VbM7Mmf&k>*f<5JQl{=N)Ys@LaT;Lb`4l#6VoO^-_Phr
z1V90evYYG-LpbOKx(TcBuycQ_EM@D%<1`Gf67?^%RJU?dalbjlEBs`}vV+B_hphZ@
z?3?=rRUKfjgy0NqbsLh>9-yjym|Pnh5fY&Nr}=gZaf{|{MZGe+xVi5_25tc1C9z7F
zzY<-!mc
z>yACj+js0Zh9CTnLtkkWmp<7?D24C!2;|!J#58Mb+{}IV!ln!`LFj4R0ePcOCx@ii$dG
zZZ2TuUVA&wsB)+BWA;D;}o>i?e^j#X1gEt(b<{|A%@98aMo;?~$_%(9LmL44;Rn#&Q?2gRO{
zn&nn!H3+zYBq?F#qN59>pF;`{`~yZ0kQv2|o`Oigr9kMkH!-a%D=R7=Z{E5Ui!BF~
zU;`xpD?$nW5fu9kAk@SrZllnO>L_W;daWt&QQI=d5?VO~RuNPHBV-JDzkT@d0cXTe
ztdP^TRI_*XS&>9|544Grl5!X15v+TnaFJe!o52d+4NnBVPDtxH7~!~W(}E3oYg=1A
z=mmb^GXfu{p*9n?9uW-tCch*0J_cSZ3WsjM0Y=(BB)UbyF@}QE506FJZ@gQ&<8t6g
z(y~$QpJfFD?)&^mEPbwOZsxC?xSH*YXF#qkFsPGD#(o<@@3%UDufyu<)PT;1+dlAN
z_ih>hzW%f_T}ccM+(i_?SmBxJX(C=}f?`Ja5U|(SJy2k8A(sc;YL2R6;{S*XO|C9Y
z#m4TAa+yBW+1WW;zI5*H-MbJeU@xn%FUxz!thb=^JoNQ#h^Am-V#1`9Aq-McSt*tj
z{y&zAVoo$YC}-bZF7<LEmeoV7t(51@aPR%HlLh`JiDudOO7
zE03n03dfl~C_>24+Cml}0?bBGlm<|62=xj&EF>tv5<|oz@V$r=JPI{1Bl92yPN6`0!!agn-Zg
zEH4wLG&IR_P*!Buhp~kx7Gzfomh+pl8wYlL`Z-rorDbt
zVRXrY(e~CY$K{EvMQ9U55)lJn1iGt}f9Ha`Yhtzx#&+!r=oXYi^}>~z)UTs4Uo*%^
znP$^W0o!BeO&gPTfcO4D?uGy`;6CrIx(>UoEt;TuT(PO|xch}a~~NU-7vmFk1+hwGZa{9QAJ1$!8!qWTJZwsQ
zqoWSrtHI@nuS`5V7%6dB073gmxIlg5Z=FknwNnN$65DWW1CK?>%G%nju#n>+E3Yp(
zo(WTVJ-+-anJiev$cE_`VrV#ubr6oEl;i2M@FF4i|9}V~d|QUdtmsx>zI++hOh8!J
zLr>4+Mw|*>uFE}(c2`&Scn#K#HnM0M2maj3Z3mwD@X3>M=qi76*c$P7W`~cV{Qc>B
zB#el7(r&moAO1QwZ!h-rT&U*xK!Fkk@ffQt#7!3@3^`FET4R)nUxO-FtpD0dwqrJ~
zb(_$mAI5$+4~*}`3E18LyE&trlD`%H*%f)i!^2i9$Z&7KF;YioXE88&LY6}sux2&
z+?}7dwknwViJUmK1%xnw917GUZ!ZS;R#
z0KZXGjW0CEe9T`r@Abgf8aR94_4(Jt2`A#8HzIGonY@(z`84{^XFZ@kTDfpOw
zZ;lNZAUf17D|NQ)y(}N}ZgTWTn
z=pC5W-fW#eWi!{0(3;owJGe3C0dla5UKEF*;_X)#laay<;helhu-m4L4?N#1*aLkl
zeEbjEOXEu;l!Quj7lj2LodWa2*+A|Av;gY|L0{r?uzKB-k=EF8kXBG}i-thX*
za}er4b5%Y~%h;IJh8(By?0~qx4|tpSf$=-ZJs*x8C*3$;!c16@Fz?ww$!`s&ER**>
zf1~y^!2I9G=D=fDW{l4ANU;~4zT4VUD25RRRZrT42O4N!YX14co7JO~!kzWsBO#lK
z>tsG3?6h+rJ9?YD;$Lmx@Y4--8%$SZNWJzc%$Di_|2)n8^i$-f}+lzRd<)G6BC;rHRW)HP;X=>6~aXU59g
z`?=ZYvS6B*@*(LBDyWcZT^0x4Y+)0ch|CQg~q~C|7fir>my=q_#LM^
zO8-6W-Tq6ikz{r3{{I=QO}MMNHy_FLsn%&LmDqrPR_KKw%*4hczhzI=)>(ISTZ6^$@&C?p@F}{ZFE6n-Qj>Z9g3C>qB4ym5tbh-}|
zX6S7sPLbfI@P@v1=jV52|MNT#Pw{zxKp4YJC9=g4a;`6MlI1(>0XRFtzlddZGdk&6N6+-$y%5cD6rDJfi|lHuUJv4=gy9MZ
zFqQ59_N@>UV!YD<2ZmpYmZXR>ot@&dF1=orVbiAJb>Z&4Gn}nM$6~#Gd-n&lwzjj#
zoW_^$(@Xa6>2avM?o4|0=uy3?xYm^`Y;TD`A*geF=01LYB(E#=Ut7NTIDYzu$3~J|
z3q$AkqJvD0TmD?tpj@7G-E)F?JBg38(R(U=#2yLN3rBUhL8_0()u-Y83Q#Lf0&2gu
z+$p5x5f>Nd2BG#%UsCdylIWG9w&Jx?Kbrn;s;ggDO}Ern?oI#a|Nh~
z8N66LW*p9;A!mg~X)s>*CjP{OLghE~T8zba2c}s6S-C5!gI{QhLW!3+0FL`}DEk{E
zT(n)io14vfGO_0OcSv9=@uE0?fz6t&+Vp1gFF37Oe%kA9z`oFIeehO>aA@ayl=oWA
zaQzl1`5&;5zO$0%=IR#kK-jIT!o;FEw&}&V{{Swb=&zPSS413WK!l8R;OZyh9U|#S
zTMnGX-;7y-cbPK4BSxo3-#II_$<54MZBZCdn{#&>R)#`Uh7~P?Z2x*
zqq8)i@$Z_O)=b`@rp|zqlO)(=#H&^)ig1Ju{s^;}_-}dg^1^K%Q;hr1bi2REG_eaZ
z)JsmyT;
zEiccNp{+YHI3zY_LN$1Oui?^zzEpUiBybBy?p#O7jC}
z5zHr~zhm9qv7q6h75g#@zR{iL;}2LN`{*>PL~=>xzklMZ2!T}8IW8n`up#oo-iSm;
zM4Tqx5d)74%d^juR!-!WUF`0a!*VaVHWlRl7W;SkpT|&(2nhIsTawbixHw&Y9iw5F
zE<)ZAZ!?jwvWfj{l`+4eXkx`^%cB7+eLP!i?(2(G`s(UCaK6XGiZv918My~e>jvPA
zN>yE{E+xAd#~){A$Ats`_A&Q2_~{aKq7n-Yg|Os5
zE&GqRc`)gK_D3J