From b3b3198dcbd8402251297fb44e5d8b4691bc6226 Mon Sep 17 00:00:00 2001 From: Dewi Yokelson Date: Fri, 30 Aug 2024 16:46:52 -0700 Subject: [PATCH 01/10] first draft of workflow, lots of overlap with existing docs --- docs/workflow.rst | 73 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 docs/workflow.rst diff --git a/docs/workflow.rst b/docs/workflow.rst new file mode 100644 index 000000000..3f1482751 --- /dev/null +++ b/docs/workflow.rst @@ -0,0 +1,73 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +The Benchpark Workflow +============================== + +A benchpark workflow consists of the following steps: + +1. Initialize the system: build the correct system configuration files (i.e. for the hardware you want to run on) +2. Initialize the experiments: build the experiment configuration files +3. Setup the experiments: validate that the experiment and system configurations are compatible, create ramble input files and experiment root directory +4. Build the experiments: build the source code, and setup the experiment directory structure within the root directory (can this be combined with prev step?) +5. Run the experiments: run the code built in the previous step, output files are stored in proper directory structure +6. Analyze the experiments: ramble generates a summary of results + +Once you have cloned the repository and installed it (Getting Started page) you can either run an existing configuration as is, or create a new one for your system by following the steps below. + +1. ** Initialize the System ** + +To run benchpark on a system that does not have an existing configuration in var/sys_repo already, it is recommended to either start with one of the existing configurations that is similar, or start with a generic example (e.g., generic_x86). + +We provide an example of editing the generic_x86 system configurations. + +The main driver for configuring a system is done by defining a subclass for that system in a var/sys_repo//system.py file, which inherits from the System base class defined in /lib/benchpark/system.py. + +As is, the x86_64 system subclass should work for most x86_64 systems, but potential common changes might be to edit the number of cores per cpu, compiler locations, or adding external packages. + +TODO: Examples of making these changes... + +Once the system subclass is written with proper configurations run: +``./benchpark system init --dest x86_64`` + +This will generate the required yaml configurations for your system and you can move on to experiments. + +2. ** Initialize the Experiments ** + +Similar to systems, all experiments are configured by an Experiment subclass that inherits from the Experiment base class in /lib/benchpark/experiment.py. + +To initialize the saxpy experiments for a cpu only system you would run: + +To update it for a gpu system... + +3. ** Setup the experiments ** + +Once the experiments you want to run have been initialized you can run setup, this will validate that the system and experiment are compatible and create a root directory for your experiments. + +``benchpark setup `` + +4. ** Build the Experiments ** + +Now that the experiments are validated, ramble will build the source code for the experiments and further setup the directory structure for each experiment: + +``ramble -P -D . workspace setup`` + +5. ** Run the Experiments ** + +To run all of the experiments in a workspace: + +``ramble -P -D . on`` + +6. ** Analyze the Experiments ** + +Ramble will automatically analyze and create a summary of experiment results in a workspace. + +``ramble -P -D . workspace analyze`` + + + + + From 4a1fbe9b6853cbe1f7a12a40801664230e0bad5d Mon Sep 17 00:00:00 2001 From: dyokelson Date: Fri, 20 Sep 2024 15:16:36 -0700 Subject: [PATCH 02/10] initial checkin of new docs --- docs/_static/images/new-sys-workflow.png | Bin 0 -> 23750 bytes docs/customizing-system-hardware.rst | 10 ++++++++ docs/customizing-system-software.rst | 10 ++++++++ docs/identifying-system-hardware.rst | 31 +++++++++++++++++++++++ docs/identifying-system-software.rst | 10 ++++++++ docs/index.rst | 11 ++++++++ docs/run-benchpark-on-new-system.rst | 23 +++++++++++++++++ docs/validating-system.rst | 10 ++++++++ 8 files changed, 105 insertions(+) create mode 100644 docs/_static/images/new-sys-workflow.png create mode 100644 docs/customizing-system-hardware.rst create mode 100644 docs/customizing-system-software.rst create mode 100644 docs/identifying-system-hardware.rst create mode 100644 docs/identifying-system-software.rst create mode 100644 docs/run-benchpark-on-new-system.rst create mode 100644 docs/validating-system.rst diff --git a/docs/_static/images/new-sys-workflow.png b/docs/_static/images/new-sys-workflow.png new file mode 100644 index 0000000000000000000000000000000000000000..784f22a1b56d3a0bf72e1233216a4593148ccc54 GIT binary patch literal 23750 zcmeFZWpf-svo0vKMlfP#W@ct)28)@&VrH;pFGL`Q+0Rt*R^oM8HRYfPetX$x5n2KtLOS`(uDl;O}~NF*pba zNE%xS2~~Ru83|_xXIBjuQ*%o(3uiM+bs2FsK0aOu2-+lb6BBh=M%ocmbQ6=2Nd`Iu zS1Mk7(tlfKoS8goFVJy$WAd#xKQti>ar3^ ztx2Du6DKoZtT;kWcNL=QU_~8%c9D`EW+vySRzO^JMHZHnL{LWuk^)JzcaYUy`A`Y-V$G~k%>cou*%W=f+dZivA^Rxe z6U7-JrXk4C(C?bznOS7eVZp-C1KoV1B~q5E=D6Qu@o)i`kYDEHgF}hbG&;|w2|J{ce`#5+9@%J7ALKH$yQcTku z@+=q5+d!weCBwIsalB>S53~`luzRzKNEv<=B}!Tt`R7-n0jWWoC@DNDb2#!**P2|; z;KmnF*O-)RtL6Cd`%_C3&ECv}jTPrU;biN}2#4DkGf3FYsTy1 z_+sfUNo-jtO40w>OaQ7oyAl7}kD^d)xG-hniwe%@fb4gP!Ai$8F+$3MF052ppRdaM=BzfY-fo{Z+rM%pBGJfY1Iw|CT zw~z}!lTSotY0GENG{@8p%k1JN|4Kq(J!%gRt{t`A8OlTp75odf##vsSzv34sKZGPUY@&9|Gy z$KM6S<|Te*Z;qaUGygDa28K~ao*9f2&z_f#F=v#qbN$Qf9Vxewq3)hxzRmwCs!f9w zlI|$lA!|bSEOYeJ8Q%pqTE6cP{X<&YLRCH>1N$H&7?Z?4FNC}r$PJmXzor@T8OU8- z@iqj`ywe@U?i{&{4#>vcEWOiW8mAFP<= zE^=8|SQvY0t{sgc*y65vmag=#s;f6ZmofX6q*V*x+x6>He1SB@C8G;VWCUBE0KKk4mJgQk zpwZ2+wSB%W=lP#e-|m6KC^!Vm((ooshm!|EzXe14j|>G%5|VxXb;&3$*Uznu-6e_Y zQ?a_80@+P3B5Ka^?A=<41zU9T-nUUSTMYb;>USLDdONvlM=Gk_h&GsL^;M=HnbMeK z63_`UkYEylHzLmO>K6uzG~-7cG|jm^=-$8Sz_N~}xY&$2;F6oH8ssWS`yRe_+aI%Vif(uQVzNvDJqVpuE80 zMvNAI!lza@g`t3bVZgM>k;5zQpwWPq+3`fB*AY<2&~YXoaS|u>u3EXwwh3YAGg?36 z_#Hyqy{lpSOLOC~=L{<4row(2fDYmn!>;m~tUnpc>5XzEg``1!a2asulNi|`uTg8O z`x9{a>E4Rw%Q$SGXR;_KpqIAcy7v>Eic1bxe*y~8mI{Ru=chEwMA2Ay6SkMpIT>>U zg^OL4Ts|^}Rz86>y9;6$u&m%he1S{`4PRR9`A_g8?4y)7w>N%(gPpbr(U>YZ7w$qT zyW)`6xQ{#XhsqRXS`(xx1RB;q2BeH~7RH(EgVl!&O}Ei8WG~qr)~?MSvj*7iorX~+ zIxuoZxh)9=w3_h=2nZ0z_d5M6))*g4HUi%_EiXq(4kfFd^waN?c;eEzE~Z+C4?29J z=nMQj&;01n#`vD`7Z|Q~@tuApm*+PRB*Yk#U>^i{%4CSVTDR@Pns81?Jf#oKJtq&A zkw2x|%1M+42{dg8x>A#+HL9bNh5!J?Baw#ZBh<$T5hvUA>g6?j7=5A_nBYT_7|R;z znWGqsi8Bm&hcni%l&HcXhL+j=B6JAWr}S&M5$d`yABzESj=T*B7#32riSN*he8%so zH&+7BsYa2QoIHE8{pTkZp1PQaM>7%)QZyzCIphJFY-*H!D;x5c$+GJjK+9JcC=5}| z91UU#jJ7oL8r8P=97dEyxN{0fbQfC4bpQ&92->Xqm{)JA&rulyn2G-K?M8$a(uVZnklcQ4A*TGmCTKfH~yUjB`ElHGs1{L%yMaf(4A|)Po09HZXEC`|9 zcw%Dico;&N;CrNDAUuI2R3U~-UjqfbK_LoF0_Fc5hWuA2qz)W=BOv0SQ2*}-{(!@5 zgj}rupI;Hoz*8xD)vyM;OyKYK?nGg!`F@m~o<)4q>qPWCdI-ruNg>2=_+@5urFISA zxYcBJcCz4;7<w+pIKq^imkTE^r>0W=Q;-=h;Vi8SuC@8s3A}VYS=X@rR*S;Q%tp1@ zP>NhIj4Zg3ro;Sn@i{D&-KrU|IPJYIHf}ximQz<3cO#AT%_%D<>;BGtMUN+P=%saPSy``Q%Q*bdb=YhHxD-$#;9}pPvOq7N7X}qvG0tK$<Cr>7@h zCQ#9vX<5&e!>`TSi%1WfC8=iv(z0pILzrsK<7O0X;a#7Ra4%2#Q)WLD$HtCn-$}*k z^!T$5c`M&|TMBBwA2-(v0+7Nq$|QUz-nu(%-}wzaX7}iTgCntefqR)DWJ3_4qwL4Q zFdfWwJzRocgsB2y+OhAG>`Ma{*HK&obhLFcL6)`2Vy>^hj1%7{-YiGrMn@-7vaJH; zVi5SVPaJesr0JNps8{rUv0L^2@@4W!)o1C{d3!m2ovfKjWPo=m5E)PomMCiXxetj8 z_0ul^*=aI-`(hexhP&rf@BJTB3!#S6Y|9cAjpp@#2|g9*G}Lz1C(Uo;Z71^pEP-tx z&gv?T-rlZqKbBskn_YGQi3Qgy<$Ld1z%w!^aS8inq3G{=6$V4q`)YkqZX_}gofZ={ z97X8%+&R|k@4MlpvDH=NSxTgl!&|y@pvtDcL+5n}97uFi&Yh0o2SGd%SU zlOay^zU}ki(gSKQz9cRtGSL>z^n}l#0gr^A3g=dR4bjt2Hp*#`P`~jBbDnqhuc__F z0P!9QR(f5%W{;Znz|q#*XHD9H+vg^ijT8K#9Eyi2ViAuK|E<${fMYl$)DWRFyd5RL zWV!x;qUzVTO&k{y+BpzmEZZh}d6J3Oqm_B=&s+Lh($=oPB?3^?rbo^!sMG7KlR_jS z$n8=%u@H7k16LolWh41Zg%*;BV!QjHC6e7k?iKL++lY<(GH#$e|INJl!ePn1)Hs2I z$zTX1+`amL0aM^uMRmu{Ebbz=r~A_U8eYU`CiR@?dHgqquoJb8YMycqzga5Ko2|1X z<+D9Y{tF#Pp-kJBY3>BYM7{nO*vrX;>t_C+(rxqImCEx6=Xn9#tQHQ(+g)h#y1V z^@|D0%yeR0N5*#@(0d&=a12{e>%@zZ+o%StO{07REI3WDg^HWEy9VA^WOZXpM}JLJ z9!)Y`_dycwiT#Rnm666%z z8*|-!xvQO_)=5690H$Th8}y1dzCf7Xjeh zB}qnZ*aIr$0H4@6NE$B)csk<~ydvmzJ8n&JFrN*FVfNHb00h_@McwOknCr;X{#-P}eEZ}9n%^=H7x26)=w*^{_5SuCGo(llsO;!Ey740SL~xOS+8c-#IJ zOQk6ST?p#pZFgGVBqyYI54l7bQw*0`P$SXNZ|Ok@c8T$=O4mHeCYa#vM3O{cP!8A7EpH&|0kp>j?%pT8`M; z4tP~0AycPRfuHlBZj@Lh5YJ^!yqMV#bXfN^?QxCxk31Voago5fO9jg(RiS&wHG|%^ zbk4kHk!2sJ!kXGw0GA+wUe?>@es(jR{S$V*v7k5YXBpD{+@rm*(8=4uhI2Q~f5N-3 z%#id3Gy6xaoG*VQJ`gKvb?=14To1b=DRo2!RVdPccI31ZYyUo{yUzZ(FN#a}KWVh6vQV?xWEdibjUrRGsuo-l?@!{EYya(M4t1$}ZI?zN%S5d# zkRGF5KOxr^RI+|Bo&cE4`dYL|BjZx>2`HQ_9wq7Wk;K&HOqMw=%1 z_HG)BY=c9*;SwDw60coy@|wotn}rXg5c?IW_Cp92x2)h)Y#V;HcJtK*h{M}TB{BZZ z?fqwAHxC?_vJ$>0d<=q3EW#k_5RH9Ph!{dZaIZbDB#4P(q0JnNxl!U>zcC<${UM>- zp7p-dmV#j5HyTlhA5W}pe#a9e#;27M2~a^Kt3I{yf5oAnRe}+8PVicELvx#Yv)$9Z zeRS%CjLlE0mr=?5BfVgtV963mCn&QTeAoPoT>2^_?#vzGVT?$Fp!U&H7#6aq$5K%C2nd(kSLoB z#(M2&rjc>?vi5`GtNyh5s7*Nby>;hELzN~;aQ|T5Wd6&MVSydlk5D#mGFkSJhetCY zNI)Smw@(4pIbm<}^A$_8W*{bh7##Y*?B~F$nQ!|Z^0LnLtRe43>_zmxQeuhrCmDHWGZXU&Py?)Uh!_6jc#>qcd= zP9(ZRH0_pO?oy4fA>5L7l)d+`p?;yYq~_M9zV&L%0rxA@)okb!^M55T5(he7D@vDI z@#HAD44N!#Y)5&q=muWahgcn1FS&bnI8q(Lb3dgdblzGMlnbs+ZxU9~en1r9w9keL z>=#U-PUkE$;lHhPJ1dDgCpagKU`8|}X1Y+AlVIY6yNRM*VuugjfP~elk$>stXE#q+ zEU>S<&tCySXzZx{=I9kBO3*MTZRBWzO?VGwq~*ot{*K@0*aS)Y`dFZb-@{{H|9Fu`OY(&!f6iu8`-132#(6lduz0m*g3(3HrB<_#1Ky$Qw@3S+nom* z8blr@1{AJ2t^S1rncI3J=ygi)jWlO$dxrnV5_~L|hKv!}bl3Z{RT;%8aO+$D%Wfkk zv$t7E#ia|$bHWO4dQ$IrottKy4C=`S56O)PF1mv*%j$ZU?b-K@tyLZGg<)jDB~0jn zG_RK5vPd|ExSv0>D@%=`o<(f8_X-UoES}uX!-(iENr7P*uz+3QJ_H;;2L||)rtE}% zC;Ys(;MZk4@#q)v{Np1{CMHd<$d5X^ldWW5QffNPPaFQ1jEa}BQMKs;PJyjzZU>MT z1}nx(+R33>(8;0Stcl$av~mm2yI3Q2w}F6wm1Ksz0+ej-pngtOMWg{0+{8LwjV=8@WG`ZtCX*K_AP z;;8KQa9!A2uR;U}5s90jrPDbk_lKUs7ZHhN%GKc0ikW2b*C~UFXns;}m`E>LiWVue zHo)&cp`bwUw)hw6KHCBZ@Jg@(B2<}&Zpc+8bWIvPZkJes)%Zy6gJE*Oel;VXt-lzD zjDYTGI0_jw8Dbpr!G%u$yVl&gfaMw*6SV-A}~6i$IWdvmI~+u)%Th6TDVTz%tyQ-+6uNtH7Bo21b*!Mh`pzX=xc+CYDA=45kY=Z&%|o{e8amuu^tx-Rpv` zE(93`X^@D87_(%uFNpPiT_9!8Rp2riiHz&^&Kx{N^6NW@=G{Rkh8xe!&jECx6jBp} z%?PanzDga(g;rT=?_5n~Ms^jl`?_6Q?{v&%Z zlGtgNOmSBv`ts{ia%FDd`= z#ho`E0*hHhA{6dj_-Z{M@b2q`-dmMkc$_&De6r*N0{g!mlxR#lJ7^l`U@UC9ogLWm z+0;t_!R6pO57yIxPS9Wd&dcJLt5LtH)%ZV6JX}?uf;M;kgvFDIeg9Q%MrGdDNm9VF z70+~F@g9h zb8g+tEoF>P2ZRr50*|W9{m1$2%a)=X`Q?6-2wEPJq;eaVwB`#RAck5qk}=Nfi<^e; zDQK0ykDt7s1ys~ewYnQNr61^FT}e#9dy9w)FmTB9fJpEpReuFoi2an#;~UTCFe=*2 zIxLMU&5+?G84J$9nw|GNsyGA^ch<+;A zF)geptgC-of9Vah`h%bu1YR8yKL9dPhyRa6g#I|{ccP^77OwBw(6^J(YtDI(`;FwH zW*gGD?RowOg}OQeSKzqHOcX-8WsUg`L>n#UfrcfAq9eJy`X&d>iAX|?qS~^#s47IS z-#1K{qE873ITdI!#I6aTcz|D5fHFk|$gGIh*A@AQG_t5_brY|O4)=wHorWgGXkap} z1$viVp4Pg+5c>t0#cA9H{t{o>eM(tmw@5|`()<Y^?i0-^Pykt6Z$JA9E zUfJYibKwI79Kkb8z_u~N?u0p~0`CeGkoaZmymk&8A5@XS3<#{6hbvDdN8N2@_*@9a zhfO?q36T_;q3qsKiZt~9$rfQ^$j zz2PX&=_wn0xWe||8IG$djFX@45s2}_DG=~EQs9yGcppB_wFQ8O%qp~!`51Eh$Tve? z#~$?xTxs?N$8X7R=%f(nvSv`4?aAI3D<0^8T+!cO=nV_!c9X#MgKn72l0O*$5%7?b zxuQ~15e{qZOvn45=v~&Pa$W1@gGwi}D%u-W?A6;#j~Da8zD;iqtAMYggn;tRuajSb zZAH$=c~Nxz*oZ?CKdJ+GSSa9n!q?Q^#cKWin&|-H**Wtd;0X&GUNWlo@e4E3@jbFe-AK^%@vU4JocC3?wVa{N1scQE0Vm?(9Zy_ivU7re^&)K$Yc|pfl^9CrkfQ+p@_pPu-5;ylO zL49ZHp6;!noxEu<>4LptYd-kwf^dX~a@zzf96~`}2Ym2DCWKc#ilK1OmV$bN&*Am| z{7<-N^yN{6yiecohx;u(9%F`5yW#4-2Wu}rKEB0P;n~&dkDk$J4H@y!<<=7yJb?>? z>)6p@W50qGzJPY5v~Se@>kceJZOy-3r%C)$z~lxn+#WbU8}z@>WqL4vp3>9vIA^uL z20XIGE6rPN$A6sUFmBA2@SPlIPCd?&FE4kK@eOKi<|!827K?;fa=8@$PYFSxK<;SQ zt(_6I8(baYq}VQT=ey)HBtq;Mi#ymbPi3N|Q?YxqCt$`x!`2Pmjg*{fh z2Kbmvhuf>W;Gy?tV~%!Qk9zA@z)Aq-VyWoF_BQCE+epB4rZSgRAB9@eD-B%4YY`>~ zbv|zW$=_0oyvZss4e}E_=sn>CGeoX^UZ;Nl z!&>8SIwAL2fpvXSjG*@|5hDR~clbBb^|W*^YSaXzR2C-YHdo6ir;dKln)7sPTnb&k zQ`b~A!;}wK($*yA7`Cwfm*u%95fUlYG7Yk5DH5>~%X_4p6`PL0xv=>y#^-d$bq2Ty z-BUBlZNV!{Pt(HsH}aBm2NiKKc+}Vx1<0P57)hVgqFI>p9GX7>z1TFh8alz1YpHD7 zI%=32)gLV-VeGpeEAHu1aAL%`1q5`p1oRN=V-#ep6C!@#K4NK>;OEko4Uxv|l|Z7e z0(D8cty=Oy?D?pe>)+ShRju&{{3|94{UiPoK~LcBxX4FQllXxRKUW%EW{&k_KA=fSP(QIMH0q>3dz=R=icz0>8e2WO!rC`rW$xszVv&0;i_$E?@HgFN!d^S5krz{0mz)t zzm7RN092EZ4K9!YznA<1!8;Yk7Vh!-xGuI9x%X zLKwqN9s1xhfGa{Jg9_yhaLO8%2xE(MT9i6YkD}Z*0%#Sy74MoyWyIaUj`Xo1!|{oo zi^Hmw?%tP$aveBDW-uUJ6rNhSggl&AF3hwM^PF|of(%-^3}ay(L+_MP;OVBw)*KDT zxy;$gP^h6dzsM;mEgh>9P7y3i!@Ok&L%VVj6%b;2*It23yTa8@wU+SvHKJ*#~;1 zm3xA69Sk292H$)3U)zX9g-n;r8e2;IVB z5B=i(+b$j5PF43|#Y7q!uO^;F4{F!>TeZ%(B2Iz;U;L$qA!017d|TE%&QxHGBxq~i zeV%owZ_p7QBKDk>zzaM)mULjIe5E|b!=DSYH-+M=)TMha~?n-9m= z#;yve{@xG!+sWj9hT2i6)BY!i0L4;Ha#Eh9%Iz}(73yr+iE59WB-nYE4<9>3*sH8q zX_c1#OlH?T?|uui$Y^nPJCgt2xXV{wRl_Pj8Jn~u`^6OJXtglmRHR`%-1JAD2fizS zxNmCsO9*A^-KtwXGx8WVOaf%^0T5xQbD7dIV|f@sVb&c9CK zH6Xhi;s|l4y>m-;Fu{r%@6+t$nu|ECn`#T{53l{y>90V02>{WeTm-}7xA10Tt|tc; zhE1|4PXh7B3j-F0i__wgNbZu!B}oG9?U@GklA}E`Rn!2>T#6ta3x8h+oHXTA*wzZQ*3_uY+*;}C7|%65 z7pW@y2^yQrkTz9vc-}Tz{0BO~1m9ykJ~EiPM5xkhJGMB?{E&5XDyV9dbsv;hCumU- zW(|E(SsKjw< zj>Hn*5tkmmX&U;A8HYzmAkM$Fh$H9cXSYT-MZJxr5!;AY?*ZPd<-~)>w3CrIJLI6$ z$p2LLGz#i(Zs3=VsVQ6St&&waH%!m=^OFMvxp87URs>1$i0e!mx{tmaytc=&j*_8P zMBW-C{&<=ChZW6`n1njE_;`$VjaAGTwfskH>s7Wf-a~Dq!CGUA;%Go>PsDSvP)dnz zb0_I@_oGmwQ@Ob*&n=^p&MpSyql=u4|UhK49Y7BO?2i?DYmn2TxwhNi4Y*m1?h`9!kL zxYDjfWY7wt#?=P;h7v{BC=>JZSMasT_?fu0CYM-LR~o_} zz{MI=1jr%>!;>6V@wIF}=7L9>7r5vi2;W-*u9gOW!L$Qe(Y< z85{c}qwLedonp9T{WwVuyR&kw*+{`FpRpIMeGresMCHVPe6tfmi54pm^+oI_O4=o& zn-vLIx!1&7r(*HH$Q-r%daDTlO16848|@mr^nGi-Hz^`NGF)^r8~_*M%K!PLA~`9DV5Ye*Y25NYv+Q?>_$njB7dGrCV77w!4lv}~nwE%X5woEOOk#YZPtGr7 zFUW;~&diTxG)ymKx9(cnx$H5ST4qqUt{3X+L2J>w0P>UqHdahvS zL}-;{pN+t=yBdC>>`D~cUN{fVy9V=j?`%3m+tnH3;@?rb5f%jAq)mp;5sHDeEKePL zLmGN>*(lmwHnUrMJ{DaHk@zemg6bde5K@GjP!z=k`X=MZDiNa4nuJhK^VzLc^jM+j zC-g0IXrke>8M4G5=i0-yS$@7#19n8qn7PD_A4In{jJEsqD5c0MZ!N<3ekn@^qoRIa z*#b+b+gsoeble4#x6Lk<_u<(!=zXW^Se}O@y`<4Irx{3+Q|0i;jXgag{*|~N^kPXJ zzJ!P#S57LZe1X1G*j|m58Wx34BH}@7vi7I8ztfB_f|5XKCuV2y+pW6Z7$Am7P>~5f zm7`b>u2o(MX?NV+qQ~|)w_{RJ<`jh`aOry6R^LhCU~LboD{^g9bib+j%$u;+5ohb* z&}n91|8U2251g=?$$_n&@u6l+k_qkd}1&(Ky}5n zm$VL^a+@4## za$iwP2DEuS8u+#h(V_;f5w__SSC-M1m8)tn?|ZLSh}7C$2wxmXN4c%gdy9|}S6@?6 zwUGIFknGtrDIY@$Vz1tB&s6WV&k2vcydySqQDW(mbLd;tDs{Z`&*I%61I{6yw`%O8 zCaG zo>fA$QTJN1x8Fa%QFBsYz60rex2C7Y6O%V1*iGE6=@vl++2Ld9+0qPbj#80Q*-5Ny zOeWRJG~0F&_G=Xljo3}Kk!#g+pq*Y4-MR9NQ)L@Sdm3D%f zs`f7k!-;yH|N0g=6e8Fo0gCvE@ll|Z&}i4R(?@rCvP4d3iIo6l@w+vT(kt=`ZfBZw zIpF+jv5n8|^;gjjtq-I`js*f8#S8G)HoPRHj?=2h9zLZFVn&yc3 zfT33TKSo)wK3;ulYdU&cXkrZ{H1{) z7fDFxL_#e&V4O)4idm~o{ERr^f3O)pfS*CMg8nq#yQ=RWi~E0CK^#Uw|H2Sl=Enba zg)AYsU=wXVo%O%Sg8zNdD(^=j2kBxz!sU)Q!C<27}3#yO+tFusJLic1a1#IojW`aC=8HMy7qtOwsv zX+z$h?t7>NXofY@)6%F^UJgW(GxD8NYX_o`e|Mu*k(t72o`@a z;lPvXt1wWI@uu&!(v@8iOcff0d2D@rnZeJcO~?F)@W1=}qatr__ukrPG3`~5j*ktN z@oH^01z&!b3`m24ibT<6m}Kdl(2Sod`NHb!g<*t3zM9<5E94%{&8EHThxscwP51{oo<{r&wr_v_UX1swA(LC@378_f1H12o)3 z_wKG493Y3{vbJW92l4UHFPYm;4W;LTSdWv%Ny1j+@w$NV$;mY2)q}Ayc|QM# zx|J<-M8B!-fM-gXX%beG;HuBRX6GU|$Lj5uY8tGkiC6SLAFrG@@tDMP*Xp-2lOu{e zMYB~^mnWU^58=`Tg|eNO`&d^uZf#!ze{`=&MXr1~CjkGG7hLT8Z}f7#w_ zU!q={PuL|*L8OtokaaT2WF8aPu7+;QD5s#{M$u60zyaix<2ya&1j`2d&xoa0~@l<+;=7JD4(x*vPGJrP((E|a{Gst7$63%-m{l=}Of z4*>~I^{;*Jh3-daL~iMPb{-G*`|{uB9SB4MJ73=FhbNw$H@)&Zd934of`+X{81mmH ztp9>-CY?m&A~?K%H)-otS=#nJoym~id?-FGH5K!_;FlkIJ$FE+4vjM9QiqpT&q;jq zUt1kswbuGSwnWha&*FHB-43Qo=~OfG0-tPs4wrh6%WcorS~Io;U+R!ZgxB2!#~wO- z;aR1%sa3v{-){x=xZmtTB5%FkwOWS7ugpyg+Ye!ZA<(ntf1sVMH*MiB&u4jOxF|1< z>uqY0NO%eZRG{ zxJcWLC!*2^>1}Rl8Js-gYeyvxP{_o4D~E`|?Gvi*DLJN?R#0L0A<6RLs|zUhsAy?v zHAuwal+CyOr4+N-VYQ6%jPX(6t#g_?d)IBpy4ro8`D9G`~3}jw8e4l z@4DMGS4y69(0#|%kxC#tr@E1hlIs3m0lh=Ed$OSK1TKe9_upBC{r9&A8Mk>ueCAJ< zh!~k(TRt)foY?QPDHyJ}AK zKBE&NTApP3iW|>yVP@JGxnKUiS`h%H`tWr88~6HoVqestR+(N%`>av#)Trl^nKXaho#iUyV` z`>W*n?ZEC`wj2mO4q{wY8o$j3I;63|+^5%8Lc(JN0v2BhMFQ6~xT&vMdahzDjwN3a zx^Q;hU>}XENu;q1vReaQ_N)kMj&O|pXK;Vkv|q`Os{UqBho0?uyJdNNWCtOSx7y%DKTGn;zD!V)N-XB5=Nh}ev*7`}uU^oVOJ-j+3; z$MhU*GVC#H&K*jD1u3Zl^KPH;1e|iz{!PHay}(tn4dL%y$JGWICQ3rPzQN*z&-0)3 z4N%_PAD=zf_;OlLW?4=(oBNg1FKe&#FTdWpZrT(~I0tv%PA}s8!1F zZSucf3l&<_;g|4(d-H7q!{ETFv0bow41LLGIAu!ck9!fSlc6RLLE?$3uvbQ+Lu#s$W zd0d7*S2+is=;zG^U4)SU7U#8&+P00ZYVhYsm$qEO{Ix(DsOwABUce+A7F)@1u0Ib> zt(zk*!|d}X6IqR>SI=LxLsT_vN^7tfY)PNf(x3IB1$)EdLGk{#RSewTkow?f^_>BQemGzm$j}8N?I< z7;)U-fMGujURuMQ22l|v0)1@8{GPx=GL4%1pr0gTkr*&})z39Ymkzm!&B_GD7Y93Nr8hS+=`#d*ceofK9rbWJ`A&^Ld(*Y&^PW|7Kys&E2;lp zIEI@_&(Ie6o{#SZeqq;xUki&2bZf}EDp*w5I+xYC&r58!hK+!z*xOV3Iw%B0AVtDJ z^^RTS#LM0O-J?sA;u06R68NwaWfesJKJdD0#5F+*gSvTGrr9?ngt=*lYZxk*gs)R& zcGueTI=SxU((_OqNlDR(5qi-zUfb5+8MG}U!mft=0AYFJXBQi`3oOD$VgrDr2uuq4 zUU8r(;y|9^X{}a_5?xcB!x&57x&HX;1?`arxaXd5_%i7iWoraNEABmze3XfKzlvt- zv~5E^)%%iM3;;r0(I+u`H`yLM4M6jM8c#SqI$L3 zsll)IkvzeCf9`|(hm``Wn1_1zseW;OvF`A+Ss7lWMw>n>*>hPgY9nt%2NX@WYKq~f zJFTVQC;^}nL;)~`m*A({zL>IqNvMGxTqW>kB$e&mWm>#)z{vCl*&hW^n#ZtHbZxG4 zmNM0UyL)p($nE;sNiht|VD~oM&P3>dJ7FD5PC+XjH*TP)5Ec%2JU)7m_BXWE6;Nn?nk4ZaS&nmg|4T z$&3h$md)N$q_7j zI`k>ru5GY$}k}RYaSCF*!)L&$osAcAIQ)kS=*QrDaR`qH1?R zU7(&Dc+y(eUHgq*+uP=H*gXJ|`fJ_TyIM-hsnhNbQA>oYqwZo2YX026>Yvi=rPaUw zBkef7rhA!+!N^tQ@MRx`4*@6Etyxbd)$Tw>8`KHIFVUvfxXtUwS+HIhh^_QzO@6Uc z3~7L*GQ0jaSsI5{I+$rd}$+-KvnSt?y{3+dD;p~?J=*0Mah)3~|GL_+l zOb-oA38|6ST`E;oGTuWPlhU58$x?;+l;wh1BRkx{wWL>Y>HcG;x`b^n@ITorPsqHt z4ld{^v4P;KR2UBK%cW{dcj*pAH&ntWv-6;zWF)GZOy6zN4oKi=a;yepGVLFkJ>T{#Upshs^6C_k)Up9$6c1Zs-05OoS)02Qxp_NVBx7VF)AX` zjV5OY)k4VKx7|)fHwaS#n|w&)z#-PJk39`><&zbPm|xTu`l08LgQ>Th$Ui%$a}2R} zCV31blBg_&?5x9C3=USz9t&@u+sBAXr~gqpVkJx)C{g~W&r|NpShgf1*=rk+Eg478 z?OzhR*+Qt!JrC${4{m;Dc!pGZ;nrKydQd#+92qYb!61MSfJ{CrMdD@xjklaleeF6) z%sLK z0$9txJCH=Sf`1WPW`vO?v_ZL&*fLu+&FeZKTlf!~<+60j7y3Sx+neGuMbm98b<+@zZOEx@hI}!07tW6)6ihaa{x? zaGOTyDNNLa)`qW9W4p0hEP;@UtYoA!2%<`hZVQ66k_h#(xKREwG3-S{p?3VYeE&ej z7GgOzTn4;}2w-<7Y`m1@M>w6mI(-)eED*5o=|G(_kfIpQRLD3)%f6i@@k9I)7ng?9 zS$pjJTFhYCukM^#-vzbDu>~Jh4CuVeSaLGQEV#jFldeNL?pJ{5-`di=EV|oY7DJc*RrG{1662Z1xKFJq2AHJ$oRXJDk?!Y z>EOE|OV;4XAJhZSh&UKFZ@0nWTbD0_o@1#*wsb#(a2k9sbc1Cyv1VtOtv z4+xtCXN&`eLTw-iEk$^u=E6KkM@pVuyBY{V1_fcpP~B5hyR-K5>4!MtsZ*kkgi;0f zhkWeSYPJ%w|J>&6a*$>7oKlQ}2V+GziXrSY(=ZHJ zs_AcI5{P=Qk^I(^`BO>(lOIv$Y;}4vdvntB;R~8z+dseYt?3>VD#FH;Wo?d;>^kJ9 zcPRE)+c2|4>8OK45MH&xHs-aNXI8sH_$hNRemK}1Wxb-DZwA87Xk89531WCf)*M;A zqkbORL&jF6!Wu~Qt4NOblbMivi?ZmHVo70dfc&i0YR^Ngi!S9sUgs=$o3@#HC%CQu zVQC`S45oEeh|jSnr+og)sTZqwa2I)47*(fZT#(W8!`-9Zy;6qLgoSGu9B}WLb7hl? z88&I^yEed4S&CCN?a5?v^bk(Vl}PyXGwU>GM2t;PmSNpbpHtHU+a1J#(B0!mwupEH zw;rx%DebY^>nqHS?|KnfDvh%GT!C#A|5Om|*u^AO|IuCwx`%4@a@bB%pH#=-1KxLa zbjEz*z>*1lZQFwhmK!2V5{PUZeb^X^Uf`mr12XwZyW{_)XI-Dwp^d6wiemLs)@wbi zU3G?ru3nki-kRLzghI&^&lIT()i-K?-7xn>@<#9fzC#ogsnDo!K@J0{^jh}hRaJN& z{ibwIo&VVG`D0ijVz|BHU|(N1W_wA%g>E-Eax{kOfYx~iljp+P(RK|g)-rMQeMmvy z((*C0SFE_XP-br2j#oH8GqufC4eI8U9Y+L|LR|uV;isPCOLT*u1c=0S;rZgj5z;9| zh|EoBPe5EDuv1f~9u_{~2FnD}f%B*1#Lh8U;n>G-ZT{ZGGb#i8CUPj7RjSCn?w0P5 zKhoz_P*T!@^*V|FVNCb1KmgS0o09cf(w2ym>O9n*(drOe(3*0>Fc;wFkHj;)J(iZ? zoW4;OptMr%e^64Db5bDZ#ENZV;vB@OCRSe?C-cdo;%prux9Zeuu?EivC0cW)NI zGdSky!u7EAf%|A)^T9bvTLxc0@{-`RxTi}zeIAV>Q9gq zK;mN`;>EkgTW=9)wSNt(sJ<@Dov3n|5eeBOv0ikcyh^yCRYS#ZhO58V{+M)?nt2qd z#?5E?4)0*$y56At!>t&@f6L6#Q`4ra&T2O9tYUic{Oi|E=xoVk3dOcuwcO=qLXACC zqBV*)0c~>o|GK;sJb<)u^32lEEI=jxofQ51yDL6~ms+5!Rewe^w_suE=!37z$gh-c z{I?d{&3|W-S!BRv1`Smnjr)AfHa?Rhz%h{`L)R|sSoe6BZAL!Vss!iNXBHsfsYbe&b8^!ZuDiZNXywi?5FihqLejtz z_L2IUu|w2se313^F_fge!~0jyDT`7Bl?xg3C=1U!Yx`f81x@eOv#_wPZqW*rvCnTS zsy;C?XV}`#roa&Ms`f^Qq{RdYH8`kdRK#5(}qDi5E_hn_= zoTT=W=t;BXXes8t+JQ%)QpYkj+?3X8@BD?Qrs!yf*}(IrmHv>*cV73+RY0ogpG)R9 zeR?*8*Xhpq(F1K8viTP7vC!;caqC1;(Z%;@`;$f#p$nt$Y9r3L9 z2AkAjZ|Oqu4_8-<@FKAnCKZfs`oqHK&lb(*PzC~|v^j169*CS6q0H?o$d`$D5f z*7ITop&U%4o%%KTBg?PB7ukFzWSJh{LF;n4YLkL9ygk>mV^zKR5b1>bv-d~e>!}W| z-P+NJ;t?7)5AJa^sn0d=DV1lh6+g(!?ps<^207P}t3rm1gzR~xMU8i8bQRTDlgjjy zdzMW2H`)HGS^6w0MR+veWSKdA#5uG#7&gE@a#sAANBK3{U?8@|Is@LW z#o42d=IdeNEaJ6>!JAu>4agm0f`MX6!v{T|8m)A8XHyfh%G9ZRJ{aeXDkU-ZqAB$~O{;w}Fzm>X z3O|ew*@JiqY%DaZ9@q2&m-WOn%4>UESEIlF1jfI=PALzBsXGxMD91PH-*ExXfHSSY z_>=V%&R2UlfQWMqxN3fI@uT7gUxabYFY8o*RXDWvgYl=_2r#hvld=4WyiaD6vm?IZ<4DmS6f(3*va@=aMvKUqia?_Eo@cy9CXfY?N_ zAZ8liSg(zV(7Kp;)>os0UK&uK!Bl((_yue-k-{W*VgQB+wQfEx%Bp=gEJx>O(B2M?qbml6hf(H{|xluIFM_*a#%A;xQa9xWE zMKP$e)DhC#0SM`S-rJa#j{ulT!lhfuaN9{>2zZ#jm{hd|16k;p&wqnBhZ}ie@GKCW5CQX1_D(@iism1F& z4JOG@CyAsP1@ItCNX^hIt!C{Ovq2=WJ3zK_a7=jDOQE2gW{SYbfDqaM#H&RcsL1H# z-1y9$0F}Oag622wFYzNJVSm=XBc+;n)EF-eb+i##|$Q?}mTW^mL9!(ay z*FBs2d^T%O3rg$gfNkcK%R=!3_i01fmR{V?i5V5nQ|$HgnOdIi+#=S!Jnkh9rCJBv zFf3kUhYP{(fgps%?V7qn6DBt2Mb#g$@L&7fP z-IY9Az1K!qm+Lh`t8d=`18Tm1gL%{50s zfj~`gC3^@1%cLb1AGx*q0yjGYwKcJ->60_`H_rx&*#Sh%s^*#9<%F|=hoDINjM7a@ zvIGGAi$iRs4gRT)8_)DTuxl)6g?>FdX=vKnaUU3-UzU;PZ{gg9!sl;a{rJ0PX0bRBS_HKZuyC#ZfFJt0?3U0dCcB!Jhg zb1*o30yxUlu_N>{dTkd8kIqzh8U}APk~RAnX~8Dgbp=&N7fH&;K)+!>G3hS?$POMD z>aT;EUBDzYvS3Ix-8W2>)=lqPITkVe1d};X-~8ERH&fwCKCr5Fr6}Y%Kvkl%GfKw; z$R5AC{pxIt*56k1c|oGWFL*_FS|!1d_%Z!XpiI4H#)F0*)&_W^;ze=e6xVbFb#l+= zCot1|J|^5t1wX=E>-G;B*(t@3kX-%B7Sb`Ol^SYE3q#hu2`AY1_m(dtE@7uNcv~lS zS8pjMT+mAeVx!qoK1Sa8U8)sX^Jjo_NuVm{L$J51=C_q#Aaq=5GRP}@8DE#cC>P*(M#M$JShkRz0wWZChH^|pZ08%%wINN$hgBT}Qc z3KsH!dy;QX=HBE_Bwyis9U>|0G_aEkBd?(3WDL{G1;R?2Jhz3bwv*F)`-9)dRZlhp zB?BL}u4nsROq*h+Zo;FlUOCND*VKLm@X);g=3ZdtK8G|&9u}Fd>G-Z$2lyiyCGT_I z7Lms-4wDYJ9QPJsGZz;bamFvvYywbnRUje$#f*)%F?SYrZ zI`UKnCyn$P!8LAKDcE>GH_LkAl*06Dn{K!oPh`JOdO0vn*~zbm>Z5de0e#?vnm$ME zf%rmZpRIcC06ZjtQ&v_9CO-8be#{c!9KX6)b^fHCOz3pBvHpO%n!g~}KC{Ezv%{!fnzGeZbd<~PT8I4)lJi>z literal 0 HcmV?d00001 diff --git a/docs/customizing-system-hardware.rst b/docs/customizing-system-hardware.rst new file mode 100644 index 000000000..f7be542e9 --- /dev/null +++ b/docs/customizing-system-hardware.rst @@ -0,0 +1,10 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +Customizing a System - Hardware Specifications +============================== + +To run any of the benchpark \ No newline at end of file diff --git a/docs/customizing-system-software.rst b/docs/customizing-system-software.rst new file mode 100644 index 000000000..e8fd07e90 --- /dev/null +++ b/docs/customizing-system-software.rst @@ -0,0 +1,10 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +Customizing a System - Software Specifications +============================== + +To run any of the benchpark \ No newline at end of file diff --git a/docs/identifying-system-hardware.rst b/docs/identifying-system-hardware.rst new file mode 100644 index 000000000..75061b43c --- /dev/null +++ b/docs/identifying-system-hardware.rst @@ -0,0 +1,31 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +Identifying a System with Similar Hardware +============================== + +The easiest place to start when configuring a new system is to find the closest similar +one that has an existing configuration already. Existing system configurations are listed +in the table in :doc:`system-list`. + +From the perspective of setting up a benchpark system configuration, if the following +specs match the system would be considered similar: + +1. processor.name +2. processor.ISA +3. processor.uArch + +Optional additional matching can be determined by looking at: + +1. accelerator.name +2. interconnect.name + +If there is more than one that is similar from a hardware perspective, it can be further +narrowed down in the next step when looking at software similarities (:doc:`identifying-system-software`). + + + + diff --git a/docs/identifying-system-software.rst b/docs/identifying-system-software.rst new file mode 100644 index 000000000..327d56adf --- /dev/null +++ b/docs/identifying-system-software.rst @@ -0,0 +1,10 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +Identifying a System with Similar Software +============================== + +To run any of the benchpark \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index b4ac90aa6..8d0e45d73 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -18,6 +18,17 @@ 6-run-experiment 7-analyze-experiment +.. toctree:: + :maxdepth: 1 + :caption: New System Workflow + + run-benchpark-on-new-system + identifying-system-hardware + identifying-system-software + customizing-system-hardware + customizing-system-software + validating-system + .. toctree:: :maxdepth: 1 :caption: FAQ diff --git a/docs/run-benchpark-on-new-system.rst b/docs/run-benchpark-on-new-system.rst new file mode 100644 index 000000000..eb3d91fd8 --- /dev/null +++ b/docs/run-benchpark-on-new-system.rst @@ -0,0 +1,23 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +Running benchpark experiments on your system +============================== + +To run any of the benchpark experiments on your system you will need to follow this workflow. + +.. image:: /_static/images/new-sys-workflow.png + + +1. Identify a similar system based on: + a. hardware specs (:doc:`identifying-system-hardware`) + b. software specs (:doc:`identifying-system-software`) +2. Customize your system for new: + a. hardware specs (:doc:`customizing-system-hardware`) + b. software specs (:doc:`customizing-system-software`) +3. Validate your system: + a. run existing benchpark experiments (:doc:`validating-system`) + diff --git a/docs/validating-system.rst b/docs/validating-system.rst new file mode 100644 index 000000000..09ebded1d --- /dev/null +++ b/docs/validating-system.rst @@ -0,0 +1,10 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +Validate a New System +============================== + +To run any of the benchpark \ No newline at end of file From 3d57a3556d73c5aac222b88470f5be009d1e4f90 Mon Sep 17 00:00:00 2001 From: dyokelson Date: Thu, 26 Sep 2024 18:13:43 -0700 Subject: [PATCH 03/10] still needs updates but the overall flow is there --- docs/1-getting-started.rst | 7 +- docs/2-benchpark-list.rst | 10 +- docs/3-benchpark-workflow.rst | 70 +++++++++ docs/3-opt-edit-experiment.rst | 181 ----------------------- docs/5-build-experiment.rst | 2 + docs/6-run-experiment.rst | 2 + docs/_static/images/new-sys-workflow.png | Bin 23750 -> 0 bytes docs/_static/images/new-workflow.png | Bin 0 -> 40676 bytes docs/add-a-benchmark.rst | 11 +- docs/add-a-system-config.rst | 8 + docs/add-an-experiment.rst | 143 +++++++++++++++++- docs/customizing-system-hardware.rst | 10 -- docs/customizing-system-software.rst | 10 -- docs/identifying-similar-system.rst | 32 ++++ docs/identifying-system-hardware.rst | 31 ---- docs/identifying-system-software.rst | 10 -- docs/index.rst | 14 +- docs/run-benchpark-on-new-system.rst | 23 --- 18 files changed, 276 insertions(+), 288 deletions(-) create mode 100644 docs/3-benchpark-workflow.rst delete mode 100644 docs/3-opt-edit-experiment.rst delete mode 100644 docs/_static/images/new-sys-workflow.png create mode 100644 docs/_static/images/new-workflow.png delete mode 100644 docs/customizing-system-hardware.rst delete mode 100644 docs/customizing-system-software.rst create mode 100644 docs/identifying-similar-system.rst delete mode 100644 docs/identifying-system-hardware.rst delete mode 100644 docs/identifying-system-software.rst delete mode 100644 docs/run-benchpark-on-new-system.rst diff --git a/docs/1-getting-started.rst b/docs/1-getting-started.rst index b736cc6f4..e026f77eb 100644 --- a/docs/1-getting-started.rst +++ b/docs/1-getting-started.rst @@ -20,6 +20,9 @@ To install this, you can use:: pip install -r requirements.txt -Now you are ready to look at the benchmarks and systems available in Benchpark, -as described in :doc:`2-benchpark-list`. +The executable is in ``benchpark/bin``, to check the version you can run:: + ./bin/benchpark --v + +Now you are ready to look at the benchmarks and systems available in Benchpark, +and determine your workflow as described in :doc:`2-benchpark-list`. diff --git a/docs/2-benchpark-list.rst b/docs/2-benchpark-list.rst index 9cdd48b89..c9ce8dea6 100644 --- a/docs/2-benchpark-list.rst +++ b/docs/2-benchpark-list.rst @@ -7,7 +7,10 @@ Searching Benchpark =================== -The user can search for available system and experiment specifications in Benchpark. +The easiest way to get started is to run existing experiments one existing systems, or +to modify one that is similar. You can search through the existing experiments and benchmarks with the below commands. + +Search for available system and experiment specifications in Benchpark. .. list-table:: Searching for specifications in Benchpark :widths: 25 25 50 @@ -39,7 +42,6 @@ The user can search for available system and experiment specifications in Benchp - -Once you have decided on a ``system`` you will use, and the ``benchmark/ProgrammingModel`` -to run, you can proceed to :doc:`4-benchpark-setup`. - For a complete list of options, see the help menu in :doc:`benchpark-help`. + +Now that you know the existing benchmarks and systems, you can determine your necessary workflow in :doc:`3-benchpark-workflow`: diff --git a/docs/3-benchpark-workflow.rst b/docs/3-benchpark-workflow.rst new file mode 100644 index 000000000..c83be59b8 --- /dev/null +++ b/docs/3-benchpark-workflow.rst @@ -0,0 +1,70 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +================================= +Benchpark Workflow +================================= + +Follow the workflow diagram to determine how many new configurations are required. + +.. image:: /_static/images/new-workflow.png + +If you are running on an existing ``system``, and the ``benchmark`` and ``experiment`` are +already configured, you can proceed directly to :doc:`4-benchpark-setup`. + +Otherwise, you will be editing one or more of the files below. + +Benchpark configuration files are organized as follows:: + + $benchpark + ├── var + | ├── exp_repo + | | └── experiments + | | └── ${BENCHMARK1} + | | └── experiment.py + | └── sys_repo + | └── systems + | └── ${SYSTEM1} + | ├── system.py + | ├── compilers + | └── externals + └── repo + ├── ${BENCHMARK1} + │ ├── application.py + │ └── package.py + └── repo.yaml + + + +System Specification +-------------------- +Files under ``benchpark/var/sys_repo/systems/${SYSTEM}`` provide the specification +of the software stack on your system: + +* Find a similar ``system``: :doc:`identifying-similar-system` + +* Add or edit a ``system``: :doc:`add-a-system-config` + + +Benchmark Specification +----------------------- +If you would like to modify a specification of your benchmark, +you can do so by upstreaming changes to Spack and/or Ramble, +or working on your benchmark specification in ``benchpark/repo/${BENCHMARK}``: + +* Add a ``benchmark``: :doc:`add-a-benchmark` + + +Experiment Specification +------------------------ +Files under ``benchpark/var/exp_repo/experiments/${BENCHMARK}/${ProgrammingModel}`` +provide the specifications for the experiments. +If you would like to make changes to your experiments, such as enabling +specific tools to measure the performance of your experiments, +you can manually edit the specifications in ``ramble.yaml``: + +* Add/edit an ``experiment``: :doc:`add-an-experiment` + + diff --git a/docs/3-opt-edit-experiment.rst b/docs/3-opt-edit-experiment.rst deleted file mode 100644 index eeb39da4b..000000000 --- a/docs/3-opt-edit-experiment.rst +++ /dev/null @@ -1,181 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -================================= -Editing the experiment (optional) -================================= - -Benchpark configuration files are organized as follows:: - - $benchpark - ├── configs - │ ├── ${SYSTEM1} - │ │ ├── auxiliary_software_files - │ │ │ ├── compilers.yaml - │ │ │ └── packages.yaml - │ │ ├── software.yaml - │ │ └── variables.yaml - ├── experiments - │ ├── ${BENCHMARK1} - │ │ ├── ${ProgrammingModel1} - │ │ │ ├── execute_experiment.tpl - │ │ │ └── ramble.yaml - │ │ ├── ${ProgrammingModel1} - │ │ │ ├── execute_experiment.tpl - │ │ │ └── ramble.yaml - └── repo - ├── ${BENCHMARK1} - │ ├── application.py - │ └── package.py - └── repo.yaml - -You can edit these configuration files to change the behavior of your experiments. - -System Specification --------------------- -Files under ``benchpark/configs/${SYSTEM}`` provide the specification -of the software stack on your system -(see :doc:`add-a-system-config` for details). - -Benchmark Specification ------------------------ -If you would like to modify a specification of your benchmark, -you can do so by upstreaming changes to Spack and/or Ramble, -or working on your benchmark specification in ``benchpark/repo/${BENCHMARK}`` -(see :doc:`add-a-benchmark` for details). - -Experiment Specification ------------------------- -Files under ``benchpark/experiments/${BENCHMARK}/${ProgrammingModel}`` -provide the specifications for the experiments. -If you would like to make changes to your experiments, such as enabling -specific tools to measure the performance of your experiments, -you can manually edit the specifications in ``ramble.yaml`` -(see :doc:`add-an-experiment` for details). - -Benchpark Modifiers -------------------- -In Benchpark, a ``modifier`` follows the `Ramble Modifier -`_ -and is an abstract object that can be applied to a large set of reproducible -specifications. Modifiers are intended to encasulate reusable patterns that -perform a specific configuration of an experiment. This may include injecting -performance analysis or setting up system resources. - -Requesting resources with Allocation Modifier ---------------------------------------------- -Given: - - - an experiment that requests resources (nodes, cpus, gpus, etc.), and - - a specification of the resources available on the system (cores_per_node, gpus_per_node, etc.), - -the ``Allocation Modifier`` generates the appropriate scheduler request for these resources -(how many nodes are required to run a given experiment, etc.). - - -.. list-table:: Hardware resources as specified by the system, and requested for the experiment - :widths: 20 40 40 - :header-rows: 1 - - * - Resource - - Available on the System - - Requested for the Experiment - * - Total Nodes - - (opt) sys_nodes - - n_nodes - * - Total MPI Ranks - - - - n_ranks - * - CPU cores per node - - sys_cores_per_node - - (opt) n_cores_per_node - * - GPUs per node - - sys_gpus_per_node - - (opt) n_gpus_per_node - * - Memory per node - - sys_mem_per_node - - (opt) n_mem_per_node - - -The experiment is required to specify: - - - n_ranks it requires - - n_gpus (if using GPUs) - -If the experiment does not specify ``n_nodes``, the modifier will compute -the number of nodes to allocate to provide the ``n_ranks`` and/or ``n_gpus`` -required for the experiment. - -The system is required to specify: - - - sys_cores_per_node - - sys_gpus_per_node (if it has GPUs) - - sys_mem_per_node - -The modifier checks the resources requested by the experiment, -computes the values for the unspecified variables, and -checks that the request does not exceed the resources available on the system. - -To use the resource allocation modifier with your experiment, -add the following in your ramble.yaml:: - - ramble: - include: - - ... - - ./configs/modifier.yaml - config: - ... - modifiers: - - name: allocation - applications: - ... - software: - ... - environments: - - ... - - '{modifier_package_name}' - - -Profiling with Caliper Modifier -------------------------------- -We have implemented a Caliper modifier to enable profiling of Caliper-instrumented -benchmarks in Benchpark. More documentation on Caliper can be found `here -`_. - -To turn on profiling with Caliper, add ``--modifier=`` to the Benchpark -setup step:: - - ./benchpark setup --modifier= - -Valid values for ```` are found in the **Caliper Modifier** -column of the table below. Benchpark will link the experiment to Caliper, -and inject appropriate Caliper configuration at runtime. After the experiments -in the workspace have completed running, a ``.cali`` file -is created which contains the collected performance metrics. - -.. list-table:: Available caliper modifiers - :widths: 20 20 50 - :header-rows: 1 - - * - Caliper Modifier - - Where Applicable - - Metrics Collected - * - caliper - - Platform-independent - - | - Min/Max/Avg time/rank: Minimum/Maximum/Average time (in seconds) across all ranks - | - Total time: Aggregated time (in seconds) for all ranks - * - caliper-mpi - - Platform-independent - - | - Same as basic caliper modifier above - | - Profiles MPI functions - * - caliper-topdown - - x86 Intel CPUs - - | - Retiring - | - Bad speculation - | - Front end bound - | - Back end bound - * - caliper-cuda - - NVIDIA GPUs - - | - CUDA API functions (e.g., time.gpu) diff --git a/docs/5-build-experiment.rst b/docs/5-build-experiment.rst index cbc2204c4..529e7c317 100644 --- a/docs/5-build-experiment.rst +++ b/docs/5-build-experiment.rst @@ -33,3 +33,5 @@ Ramble will build the source code and set up the following workspace directory s If you edit any of the files, see :doc:`FAQ-what-to-rerun` to determine whether you need to re-do any of the previous steps. + +Once that is complete, you are ready for :doc:`6-run-experiment`. diff --git a/docs/6-run-experiment.rst b/docs/6-run-experiment.rst index 7af222991..909dd59e4 100644 --- a/docs/6-run-experiment.rst +++ b/docs/6-run-experiment.rst @@ -42,3 +42,5 @@ Further, if the benchmark has restart capability, existing output may alter the benchpark would run in the second run. Generally, we would advise the user to remove the ``$workspace/experiments`` directory before re-running the experiments using ``ramble -P -D . on``. + +Once you have run your experiment you can try :doc:`7-analyze-experiment`. \ No newline at end of file diff --git a/docs/_static/images/new-sys-workflow.png b/docs/_static/images/new-sys-workflow.png deleted file mode 100644 index 784f22a1b56d3a0bf72e1233216a4593148ccc54..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 23750 zcmeFZWpf-svo0vKMlfP#W@ct)28)@&VrH;pFGL`Q+0Rt*R^oM8HRYfPetX$x5n2KtLOS`(uDl;O}~NF*pba zNE%xS2~~Ru83|_xXIBjuQ*%o(3uiM+bs2FsK0aOu2-+lb6BBh=M%ocmbQ6=2Nd`Iu zS1Mk7(tlfKoS8goFVJy$WAd#xKQti>ar3^ ztx2Du6DKoZtT;kWcNL=QU_~8%c9D`EW+vySRzO^JMHZHnL{LWuk^)JzcaYUy`A`Y-V$G~k%>cou*%W=f+dZivA^Rxe z6U7-JrXk4C(C?bznOS7eVZp-C1KoV1B~q5E=D6Qu@o)i`kYDEHgF}hbG&;|w2|J{ce`#5+9@%J7ALKH$yQcTku z@+=q5+d!weCBwIsalB>S53~`luzRzKNEv<=B}!Tt`R7-n0jWWoC@DNDb2#!**P2|; z;KmnF*O-)RtL6Cd`%_C3&ECv}jTPrU;biN}2#4DkGf3FYsTy1 z_+sfUNo-jtO40w>OaQ7oyAl7}kD^d)xG-hniwe%@fb4gP!Ai$8F+$3MF052ppRdaM=BzfY-fo{Z+rM%pBGJfY1Iw|CT zw~z}!lTSotY0GENG{@8p%k1JN|4Kq(J!%gRt{t`A8OlTp75odf##vsSzv34sKZGPUY@&9|Gy z$KM6S<|Te*Z;qaUGygDa28K~ao*9f2&z_f#F=v#qbN$Qf9Vxewq3)hxzRmwCs!f9w zlI|$lA!|bSEOYeJ8Q%pqTE6cP{X<&YLRCH>1N$H&7?Z?4FNC}r$PJmXzor@T8OU8- z@iqj`ywe@U?i{&{4#>vcEWOiW8mAFP<= zE^=8|SQvY0t{sgc*y65vmag=#s;f6ZmofX6q*V*x+x6>He1SB@C8G;VWCUBE0KKk4mJgQk zpwZ2+wSB%W=lP#e-|m6KC^!Vm((ooshm!|EzXe14j|>G%5|VxXb;&3$*Uznu-6e_Y zQ?a_80@+P3B5Ka^?A=<41zU9T-nUUSTMYb;>USLDdONvlM=Gk_h&GsL^;M=HnbMeK z63_`UkYEylHzLmO>K6uzG~-7cG|jm^=-$8Sz_N~}xY&$2;F6oH8ssWS`yRe_+aI%Vif(uQVzNvDJqVpuE80 zMvNAI!lza@g`t3bVZgM>k;5zQpwWPq+3`fB*AY<2&~YXoaS|u>u3EXwwh3YAGg?36 z_#Hyqy{lpSOLOC~=L{<4row(2fDYmn!>;m~tUnpc>5XzEg``1!a2asulNi|`uTg8O z`x9{a>E4Rw%Q$SGXR;_KpqIAcy7v>Eic1bxe*y~8mI{Ru=chEwMA2Ay6SkMpIT>>U zg^OL4Ts|^}Rz86>y9;6$u&m%he1S{`4PRR9`A_g8?4y)7w>N%(gPpbr(U>YZ7w$qT zyW)`6xQ{#XhsqRXS`(xx1RB;q2BeH~7RH(EgVl!&O}Ei8WG~qr)~?MSvj*7iorX~+ zIxuoZxh)9=w3_h=2nZ0z_d5M6))*g4HUi%_EiXq(4kfFd^waN?c;eEzE~Z+C4?29J z=nMQj&;01n#`vD`7Z|Q~@tuApm*+PRB*Yk#U>^i{%4CSVTDR@Pns81?Jf#oKJtq&A zkw2x|%1M+42{dg8x>A#+HL9bNh5!J?Baw#ZBh<$T5hvUA>g6?j7=5A_nBYT_7|R;z znWGqsi8Bm&hcni%l&HcXhL+j=B6JAWr}S&M5$d`yABzESj=T*B7#32riSN*he8%so zH&+7BsYa2QoIHE8{pTkZp1PQaM>7%)QZyzCIphJFY-*H!D;x5c$+GJjK+9JcC=5}| z91UU#jJ7oL8r8P=97dEyxN{0fbQfC4bpQ&92->Xqm{)JA&rulyn2G-K?M8$a(uVZnklcQ4A*TGmCTKfH~yUjB`ElHGs1{L%yMaf(4A|)Po09HZXEC`|9 zcw%Dico;&N;CrNDAUuI2R3U~-UjqfbK_LoF0_Fc5hWuA2qz)W=BOv0SQ2*}-{(!@5 zgj}rupI;Hoz*8xD)vyM;OyKYK?nGg!`F@m~o<)4q>qPWCdI-ruNg>2=_+@5urFISA zxYcBJcCz4;7<w+pIKq^imkTE^r>0W=Q;-=h;Vi8SuC@8s3A}VYS=X@rR*S;Q%tp1@ zP>NhIj4Zg3ro;Sn@i{D&-KrU|IPJYIHf}ximQz<3cO#AT%_%D<>;BGtMUN+P=%saPSy``Q%Q*bdb=YhHxD-$#;9}pPvOq7N7X}qvG0tK$<Cr>7@h zCQ#9vX<5&e!>`TSi%1WfC8=iv(z0pILzrsK<7O0X;a#7Ra4%2#Q)WLD$HtCn-$}*k z^!T$5c`M&|TMBBwA2-(v0+7Nq$|QUz-nu(%-}wzaX7}iTgCntefqR)DWJ3_4qwL4Q zFdfWwJzRocgsB2y+OhAG>`Ma{*HK&obhLFcL6)`2Vy>^hj1%7{-YiGrMn@-7vaJH; zVi5SVPaJesr0JNps8{rUv0L^2@@4W!)o1C{d3!m2ovfKjWPo=m5E)PomMCiXxetj8 z_0ul^*=aI-`(hexhP&rf@BJTB3!#S6Y|9cAjpp@#2|g9*G}Lz1C(Uo;Z71^pEP-tx z&gv?T-rlZqKbBskn_YGQi3Qgy<$Ld1z%w!^aS8inq3G{=6$V4q`)YkqZX_}gofZ={ z97X8%+&R|k@4MlpvDH=NSxTgl!&|y@pvtDcL+5n}97uFi&Yh0o2SGd%SU zlOay^zU}ki(gSKQz9cRtGSL>z^n}l#0gr^A3g=dR4bjt2Hp*#`P`~jBbDnqhuc__F z0P!9QR(f5%W{;Znz|q#*XHD9H+vg^ijT8K#9Eyi2ViAuK|E<${fMYl$)DWRFyd5RL zWV!x;qUzVTO&k{y+BpzmEZZh}d6J3Oqm_B=&s+Lh($=oPB?3^?rbo^!sMG7KlR_jS z$n8=%u@H7k16LolWh41Zg%*;BV!QjHC6e7k?iKL++lY<(GH#$e|INJl!ePn1)Hs2I z$zTX1+`amL0aM^uMRmu{Ebbz=r~A_U8eYU`CiR@?dHgqquoJb8YMycqzga5Ko2|1X z<+D9Y{tF#Pp-kJBY3>BYM7{nO*vrX;>t_C+(rxqImCEx6=Xn9#tQHQ(+g)h#y1V z^@|D0%yeR0N5*#@(0d&=a12{e>%@zZ+o%StO{07REI3WDg^HWEy9VA^WOZXpM}JLJ z9!)Y`_dycwiT#Rnm666%z z8*|-!xvQO_)=5690H$Th8}y1dzCf7Xjeh zB}qnZ*aIr$0H4@6NE$B)csk<~ydvmzJ8n&JFrN*FVfNHb00h_@McwOknCr;X{#-P}eEZ}9n%^=H7x26)=w*^{_5SuCGo(llsO;!Ey740SL~xOS+8c-#IJ zOQk6ST?p#pZFgGVBqyYI54l7bQw*0`P$SXNZ|Ok@c8T$=O4mHeCYa#vM3O{cP!8A7EpH&|0kp>j?%pT8`M; z4tP~0AycPRfuHlBZj@Lh5YJ^!yqMV#bXfN^?QxCxk31Voago5fO9jg(RiS&wHG|%^ zbk4kHk!2sJ!kXGw0GA+wUe?>@es(jR{S$V*v7k5YXBpD{+@rm*(8=4uhI2Q~f5N-3 z%#id3Gy6xaoG*VQJ`gKvb?=14To1b=DRo2!RVdPccI31ZYyUo{yUzZ(FN#a}KWVh6vQV?xWEdibjUrRGsuo-l?@!{EYya(M4t1$}ZI?zN%S5d# zkRGF5KOxr^RI+|Bo&cE4`dYL|BjZx>2`HQ_9wq7Wk;K&HOqMw=%1 z_HG)BY=c9*;SwDw60coy@|wotn}rXg5c?IW_Cp92x2)h)Y#V;HcJtK*h{M}TB{BZZ z?fqwAHxC?_vJ$>0d<=q3EW#k_5RH9Ph!{dZaIZbDB#4P(q0JnNxl!U>zcC<${UM>- zp7p-dmV#j5HyTlhA5W}pe#a9e#;27M2~a^Kt3I{yf5oAnRe}+8PVicELvx#Yv)$9Z zeRS%CjLlE0mr=?5BfVgtV963mCn&QTeAoPoT>2^_?#vzGVT?$Fp!U&H7#6aq$5K%C2nd(kSLoB z#(M2&rjc>?vi5`GtNyh5s7*Nby>;hELzN~;aQ|T5Wd6&MVSydlk5D#mGFkSJhetCY zNI)Smw@(4pIbm<}^A$_8W*{bh7##Y*?B~F$nQ!|Z^0LnLtRe43>_zmxQeuhrCmDHWGZXU&Py?)Uh!_6jc#>qcd= zP9(ZRH0_pO?oy4fA>5L7l)d+`p?;yYq~_M9zV&L%0rxA@)okb!^M55T5(he7D@vDI z@#HAD44N!#Y)5&q=muWahgcn1FS&bnI8q(Lb3dgdblzGMlnbs+ZxU9~en1r9w9keL z>=#U-PUkE$;lHhPJ1dDgCpagKU`8|}X1Y+AlVIY6yNRM*VuugjfP~elk$>stXE#q+ zEU>S<&tCySXzZx{=I9kBO3*MTZRBWzO?VGwq~*ot{*K@0*aS)Y`dFZb-@{{H|9Fu`OY(&!f6iu8`-132#(6lduz0m*g3(3HrB<_#1Ky$Qw@3S+nom* z8blr@1{AJ2t^S1rncI3J=ygi)jWlO$dxrnV5_~L|hKv!}bl3Z{RT;%8aO+$D%Wfkk zv$t7E#ia|$bHWO4dQ$IrottKy4C=`S56O)PF1mv*%j$ZU?b-K@tyLZGg<)jDB~0jn zG_RK5vPd|ExSv0>D@%=`o<(f8_X-UoES}uX!-(iENr7P*uz+3QJ_H;;2L||)rtE}% zC;Ys(;MZk4@#q)v{Np1{CMHd<$d5X^ldWW5QffNPPaFQ1jEa}BQMKs;PJyjzZU>MT z1}nx(+R33>(8;0Stcl$av~mm2yI3Q2w}F6wm1Ksz0+ej-pngtOMWg{0+{8LwjV=8@WG`ZtCX*K_AP z;;8KQa9!A2uR;U}5s90jrPDbk_lKUs7ZHhN%GKc0ikW2b*C~UFXns;}m`E>LiWVue zHo)&cp`bwUw)hw6KHCBZ@Jg@(B2<}&Zpc+8bWIvPZkJes)%Zy6gJE*Oel;VXt-lzD zjDYTGI0_jw8Dbpr!G%u$yVl&gfaMw*6SV-A}~6i$IWdvmI~+u)%Th6TDVTz%tyQ-+6uNtH7Bo21b*!Mh`pzX=xc+CYDA=45kY=Z&%|o{e8amuu^tx-Rpv` zE(93`X^@D87_(%uFNpPiT_9!8Rp2riiHz&^&Kx{N^6NW@=G{Rkh8xe!&jECx6jBp} z%?PanzDga(g;rT=?_5n~Ms^jl`?_6Q?{v&%Z zlGtgNOmSBv`ts{ia%FDd`= z#ho`E0*hHhA{6dj_-Z{M@b2q`-dmMkc$_&De6r*N0{g!mlxR#lJ7^l`U@UC9ogLWm z+0;t_!R6pO57yIxPS9Wd&dcJLt5LtH)%ZV6JX}?uf;M;kgvFDIeg9Q%MrGdDNm9VF z70+~F@g9h zb8g+tEoF>P2ZRr50*|W9{m1$2%a)=X`Q?6-2wEPJq;eaVwB`#RAck5qk}=Nfi<^e; zDQK0ykDt7s1ys~ewYnQNr61^FT}e#9dy9w)FmTB9fJpEpReuFoi2an#;~UTCFe=*2 zIxLMU&5+?G84J$9nw|GNsyGA^ch<+;A zF)geptgC-of9Vah`h%bu1YR8yKL9dPhyRa6g#I|{ccP^77OwBw(6^J(YtDI(`;FwH zW*gGD?RowOg}OQeSKzqHOcX-8WsUg`L>n#UfrcfAq9eJy`X&d>iAX|?qS~^#s47IS z-#1K{qE873ITdI!#I6aTcz|D5fHFk|$gGIh*A@AQG_t5_brY|O4)=wHorWgGXkap} z1$viVp4Pg+5c>t0#cA9H{t{o>eM(tmw@5|`()<Y^?i0-^Pykt6Z$JA9E zUfJYibKwI79Kkb8z_u~N?u0p~0`CeGkoaZmymk&8A5@XS3<#{6hbvDdN8N2@_*@9a zhfO?q36T_;q3qsKiZt~9$rfQ^$j zz2PX&=_wn0xWe||8IG$djFX@45s2}_DG=~EQs9yGcppB_wFQ8O%qp~!`51Eh$Tve? z#~$?xTxs?N$8X7R=%f(nvSv`4?aAI3D<0^8T+!cO=nV_!c9X#MgKn72l0O*$5%7?b zxuQ~15e{qZOvn45=v~&Pa$W1@gGwi}D%u-W?A6;#j~Da8zD;iqtAMYggn;tRuajSb zZAH$=c~Nxz*oZ?CKdJ+GSSa9n!q?Q^#cKWin&|-H**Wtd;0X&GUNWlo@e4E3@jbFe-AK^%@vU4JocC3?wVa{N1scQE0Vm?(9Zy_ivU7re^&)K$Yc|pfl^9CrkfQ+p@_pPu-5;ylO zL49ZHp6;!noxEu<>4LptYd-kwf^dX~a@zzf96~`}2Ym2DCWKc#ilK1OmV$bN&*Am| z{7<-N^yN{6yiecohx;u(9%F`5yW#4-2Wu}rKEB0P;n~&dkDk$J4H@y!<<=7yJb?>? z>)6p@W50qGzJPY5v~Se@>kceJZOy-3r%C)$z~lxn+#WbU8}z@>WqL4vp3>9vIA^uL z20XIGE6rPN$A6sUFmBA2@SPlIPCd?&FE4kK@eOKi<|!827K?;fa=8@$PYFSxK<;SQ zt(_6I8(baYq}VQT=ey)HBtq;Mi#ymbPi3N|Q?YxqCt$`x!`2Pmjg*{fh z2Kbmvhuf>W;Gy?tV~%!Qk9zA@z)Aq-VyWoF_BQCE+epB4rZSgRAB9@eD-B%4YY`>~ zbv|zW$=_0oyvZss4e}E_=sn>CGeoX^UZ;Nl z!&>8SIwAL2fpvXSjG*@|5hDR~clbBb^|W*^YSaXzR2C-YHdo6ir;dKln)7sPTnb&k zQ`b~A!;}wK($*yA7`Cwfm*u%95fUlYG7Yk5DH5>~%X_4p6`PL0xv=>y#^-d$bq2Ty z-BUBlZNV!{Pt(HsH}aBm2NiKKc+}Vx1<0P57)hVgqFI>p9GX7>z1TFh8alz1YpHD7 zI%=32)gLV-VeGpeEAHu1aAL%`1q5`p1oRN=V-#ep6C!@#K4NK>;OEko4Uxv|l|Z7e z0(D8cty=Oy?D?pe>)+ShRju&{{3|94{UiPoK~LcBxX4FQllXxRKUW%EW{&k_KA=fSP(QIMH0q>3dz=R=icz0>8e2WO!rC`rW$xszVv&0;i_$E?@HgFN!d^S5krz{0mz)t zzm7RN092EZ4K9!YznA<1!8;Yk7Vh!-xGuI9x%X zLKwqN9s1xhfGa{Jg9_yhaLO8%2xE(MT9i6YkD}Z*0%#Sy74MoyWyIaUj`Xo1!|{oo zi^Hmw?%tP$aveBDW-uUJ6rNhSggl&AF3hwM^PF|of(%-^3}ay(L+_MP;OVBw)*KDT zxy;$gP^h6dzsM;mEgh>9P7y3i!@Ok&L%VVj6%b;2*It23yTa8@wU+SvHKJ*#~;1 zm3xA69Sk292H$)3U)zX9g-n;r8e2;IVB z5B=i(+b$j5PF43|#Y7q!uO^;F4{F!>TeZ%(B2Iz;U;L$qA!017d|TE%&QxHGBxq~i zeV%owZ_p7QBKDk>zzaM)mULjIe5E|b!=DSYH-+M=)TMha~?n-9m= z#;yve{@xG!+sWj9hT2i6)BY!i0L4;Ha#Eh9%Iz}(73yr+iE59WB-nYE4<9>3*sH8q zX_c1#OlH?T?|uui$Y^nPJCgt2xXV{wRl_Pj8Jn~u`^6OJXtglmRHR`%-1JAD2fizS zxNmCsO9*A^-KtwXGx8WVOaf%^0T5xQbD7dIV|f@sVb&c9CK zH6Xhi;s|l4y>m-;Fu{r%@6+t$nu|ECn`#T{53l{y>90V02>{WeTm-}7xA10Tt|tc; zhE1|4PXh7B3j-F0i__wgNbZu!B}oG9?U@GklA}E`Rn!2>T#6ta3x8h+oHXTA*wzZQ*3_uY+*;}C7|%65 z7pW@y2^yQrkTz9vc-}Tz{0BO~1m9ykJ~EiPM5xkhJGMB?{E&5XDyV9dbsv;hCumU- zW(|E(SsKjw< zj>Hn*5tkmmX&U;A8HYzmAkM$Fh$H9cXSYT-MZJxr5!;AY?*ZPd<-~)>w3CrIJLI6$ z$p2LLGz#i(Zs3=VsVQ6St&&waH%!m=^OFMvxp87URs>1$i0e!mx{tmaytc=&j*_8P zMBW-C{&<=ChZW6`n1njE_;`$VjaAGTwfskH>s7Wf-a~Dq!CGUA;%Go>PsDSvP)dnz zb0_I@_oGmwQ@Ob*&n=^p&MpSyql=u4|UhK49Y7BO?2i?DYmn2TxwhNi4Y*m1?h`9!kL zxYDjfWY7wt#?=P;h7v{BC=>JZSMasT_?fu0CYM-LR~o_} zz{MI=1jr%>!;>6V@wIF}=7L9>7r5vi2;W-*u9gOW!L$Qe(Y< z85{c}qwLedonp9T{WwVuyR&kw*+{`FpRpIMeGresMCHVPe6tfmi54pm^+oI_O4=o& zn-vLIx!1&7r(*HH$Q-r%daDTlO16848|@mr^nGi-Hz^`NGF)^r8~_*M%K!PLA~`9DV5Ye*Y25NYv+Q?>_$njB7dGrCV77w!4lv}~nwE%X5woEOOk#YZPtGr7 zFUW;~&diTxG)ymKx9(cnx$H5ST4qqUt{3X+L2J>w0P>UqHdahvS zL}-;{pN+t=yBdC>>`D~cUN{fVy9V=j?`%3m+tnH3;@?rb5f%jAq)mp;5sHDeEKePL zLmGN>*(lmwHnUrMJ{DaHk@zemg6bde5K@GjP!z=k`X=MZDiNa4nuJhK^VzLc^jM+j zC-g0IXrke>8M4G5=i0-yS$@7#19n8qn7PD_A4In{jJEsqD5c0MZ!N<3ekn@^qoRIa z*#b+b+gsoeble4#x6Lk<_u<(!=zXW^Se}O@y`<4Irx{3+Q|0i;jXgag{*|~N^kPXJ zzJ!P#S57LZe1X1G*j|m58Wx34BH}@7vi7I8ztfB_f|5XKCuV2y+pW6Z7$Am7P>~5f zm7`b>u2o(MX?NV+qQ~|)w_{RJ<`jh`aOry6R^LhCU~LboD{^g9bib+j%$u;+5ohb* z&}n91|8U2251g=?$$_n&@u6l+k_qkd}1&(Ky}5n zm$VL^a+@4## za$iwP2DEuS8u+#h(V_;f5w__SSC-M1m8)tn?|ZLSh}7C$2wxmXN4c%gdy9|}S6@?6 zwUGIFknGtrDIY@$Vz1tB&s6WV&k2vcydySqQDW(mbLd;tDs{Z`&*I%61I{6yw`%O8 zCaG zo>fA$QTJN1x8Fa%QFBsYz60rex2C7Y6O%V1*iGE6=@vl++2Ld9+0qPbj#80Q*-5Ny zOeWRJG~0F&_G=Xljo3}Kk!#g+pq*Y4-MR9NQ)L@Sdm3D%f zs`f7k!-;yH|N0g=6e8Fo0gCvE@ll|Z&}i4R(?@rCvP4d3iIo6l@w+vT(kt=`ZfBZw zIpF+jv5n8|^;gjjtq-I`js*f8#S8G)HoPRHj?=2h9zLZFVn&yc3 zfT33TKSo)wK3;ulYdU&cXkrZ{H1{) z7fDFxL_#e&V4O)4idm~o{ERr^f3O)pfS*CMg8nq#yQ=RWi~E0CK^#Uw|H2Sl=Enba zg)AYsU=wXVo%O%Sg8zNdD(^=j2kBxz!sU)Q!C<27}3#yO+tFusJLic1a1#IojW`aC=8HMy7qtOwsv zX+z$h?t7>NXofY@)6%F^UJgW(GxD8NYX_o`e|Mu*k(t72o`@a z;lPvXt1wWI@uu&!(v@8iOcff0d2D@rnZeJcO~?F)@W1=}qatr__ukrPG3`~5j*ktN z@oH^01z&!b3`m24ibT<6m}Kdl(2Sod`NHb!g<*t3zM9<5E94%{&8EHThxscwP51{oo<{r&wr_v_UX1swA(LC@378_f1H12o)3 z_wKG493Y3{vbJW92l4UHFPYm;4W;LTSdWv%Ny1j+@w$NV$;mY2)q}Ayc|QM# zx|J<-M8B!-fM-gXX%beG;HuBRX6GU|$Lj5uY8tGkiC6SLAFrG@@tDMP*Xp-2lOu{e zMYB~^mnWU^58=`Tg|eNO`&d^uZf#!ze{`=&MXr1~CjkGG7hLT8Z}f7#w_ zU!q={PuL|*L8OtokaaT2WF8aPu7+;QD5s#{M$u60zyaix<2ya&1j`2d&xoa0~@l<+;=7JD4(x*vPGJrP((E|a{Gst7$63%-m{l=}Of z4*>~I^{;*Jh3-daL~iMPb{-G*`|{uB9SB4MJ73=FhbNw$H@)&Zd934of`+X{81mmH ztp9>-CY?m&A~?K%H)-otS=#nJoym~id?-FGH5K!_;FlkIJ$FE+4vjM9QiqpT&q;jq zUt1kswbuGSwnWha&*FHB-43Qo=~OfG0-tPs4wrh6%WcorS~Io;U+R!ZgxB2!#~wO- z;aR1%sa3v{-){x=xZmtTB5%FkwOWS7ugpyg+Ye!ZA<(ntf1sVMH*MiB&u4jOxF|1< z>uqY0NO%eZRG{ zxJcWLC!*2^>1}Rl8Js-gYeyvxP{_o4D~E`|?Gvi*DLJN?R#0L0A<6RLs|zUhsAy?v zHAuwal+CyOr4+N-VYQ6%jPX(6t#g_?d)IBpy4ro8`D9G`~3}jw8e4l z@4DMGS4y69(0#|%kxC#tr@E1hlIs3m0lh=Ed$OSK1TKe9_upBC{r9&A8Mk>ueCAJ< zh!~k(TRt)foY?QPDHyJ}AK zKBE&NTApP3iW|>yVP@JGxnKUiS`h%H`tWr88~6HoVqestR+(N%`>av#)Trl^nKXaho#iUyV` z`>W*n?ZEC`wj2mO4q{wY8o$j3I;63|+^5%8Lc(JN0v2BhMFQ6~xT&vMdahzDjwN3a zx^Q;hU>}XENu;q1vReaQ_N)kMj&O|pXK;Vkv|q`Os{UqBho0?uyJdNNWCtOSx7y%DKTGn;zD!V)N-XB5=Nh}ev*7`}uU^oVOJ-j+3; z$MhU*GVC#H&K*jD1u3Zl^KPH;1e|iz{!PHay}(tn4dL%y$JGWICQ3rPzQN*z&-0)3 z4N%_PAD=zf_;OlLW?4=(oBNg1FKe&#FTdWpZrT(~I0tv%PA}s8!1F zZSucf3l&<_;g|4(d-H7q!{ETFv0bow41LLGIAu!ck9!fSlc6RLLE?$3uvbQ+Lu#s$W zd0d7*S2+is=;zG^U4)SU7U#8&+P00ZYVhYsm$qEO{Ix(DsOwABUce+A7F)@1u0Ib> zt(zk*!|d}X6IqR>SI=LxLsT_vN^7tfY)PNf(x3IB1$)EdLGk{#RSewTkow?f^_>BQemGzm$j}8N?I< z7;)U-fMGujURuMQ22l|v0)1@8{GPx=GL4%1pr0gTkr*&})z39Ymkzm!&B_GD7Y93Nr8hS+=`#d*ceofK9rbWJ`A&^Ld(*Y&^PW|7Kys&E2;lp zIEI@_&(Ie6o{#SZeqq;xUki&2bZf}EDp*w5I+xYC&r58!hK+!z*xOV3Iw%B0AVtDJ z^^RTS#LM0O-J?sA;u06R68NwaWfesJKJdD0#5F+*gSvTGrr9?ngt=*lYZxk*gs)R& zcGueTI=SxU((_OqNlDR(5qi-zUfb5+8MG}U!mft=0AYFJXBQi`3oOD$VgrDr2uuq4 zUU8r(;y|9^X{}a_5?xcB!x&57x&HX;1?`arxaXd5_%i7iWoraNEABmze3XfKzlvt- zv~5E^)%%iM3;;r0(I+u`H`yLM4M6jM8c#SqI$L3 zsll)IkvzeCf9`|(hm``Wn1_1zseW;OvF`A+Ss7lWMw>n>*>hPgY9nt%2NX@WYKq~f zJFTVQC;^}nL;)~`m*A({zL>IqNvMGxTqW>kB$e&mWm>#)z{vCl*&hW^n#ZtHbZxG4 zmNM0UyL)p($nE;sNiht|VD~oM&P3>dJ7FD5PC+XjH*TP)5Ec%2JU)7m_BXWE6;Nn?nk4ZaS&nmg|4T z$&3h$md)N$q_7j zI`k>ru5GY$}k}RYaSCF*!)L&$osAcAIQ)kS=*QrDaR`qH1?R zU7(&Dc+y(eUHgq*+uP=H*gXJ|`fJ_TyIM-hsnhNbQA>oYqwZo2YX026>Yvi=rPaUw zBkef7rhA!+!N^tQ@MRx`4*@6Etyxbd)$Tw>8`KHIFVUvfxXtUwS+HIhh^_QzO@6Uc z3~7L*GQ0jaSsI5{I+$rd}$+-KvnSt?y{3+dD;p~?J=*0Mah)3~|GL_+l zOb-oA38|6ST`E;oGTuWPlhU58$x?;+l;wh1BRkx{wWL>Y>HcG;x`b^n@ITorPsqHt z4ld{^v4P;KR2UBK%cW{dcj*pAH&ntWv-6;zWF)GZOy6zN4oKi=a;yepGVLFkJ>T{#Upshs^6C_k)Up9$6c1Zs-05OoS)02Qxp_NVBx7VF)AX` zjV5OY)k4VKx7|)fHwaS#n|w&)z#-PJk39`><&zbPm|xTu`l08LgQ>Th$Ui%$a}2R} zCV31blBg_&?5x9C3=USz9t&@u+sBAXr~gqpVkJx)C{g~W&r|NpShgf1*=rk+Eg478 z?OzhR*+Qt!JrC${4{m;Dc!pGZ;nrKydQd#+92qYb!61MSfJ{CrMdD@xjklaleeF6) z%sLK z0$9txJCH=Sf`1WPW`vO?v_ZL&*fLu+&FeZKTlf!~<+60j7y3Sx+neGuMbm98b<+@zZOEx@hI}!07tW6)6ihaa{x? zaGOTyDNNLa)`qW9W4p0hEP;@UtYoA!2%<`hZVQ66k_h#(xKREwG3-S{p?3VYeE&ej z7GgOzTn4;}2w-<7Y`m1@M>w6mI(-)eED*5o=|G(_kfIpQRLD3)%f6i@@k9I)7ng?9 zS$pjJTFhYCukM^#-vzbDu>~Jh4CuVeSaLGQEV#jFldeNL?pJ{5-`di=EV|oY7DJc*RrG{1662Z1xKFJq2AHJ$oRXJDk?!Y z>EOE|OV;4XAJhZSh&UKFZ@0nWTbD0_o@1#*wsb#(a2k9sbc1Cyv1VtOtv z4+xtCXN&`eLTw-iEk$^u=E6KkM@pVuyBY{V1_fcpP~B5hyR-K5>4!MtsZ*kkgi;0f zhkWeSYPJ%w|J>&6a*$>7oKlQ}2V+GziXrSY(=ZHJ zs_AcI5{P=Qk^I(^`BO>(lOIv$Y;}4vdvntB;R~8z+dseYt?3>VD#FH;Wo?d;>^kJ9 zcPRE)+c2|4>8OK45MH&xHs-aNXI8sH_$hNRemK}1Wxb-DZwA87Xk89531WCf)*M;A zqkbORL&jF6!Wu~Qt4NOblbMivi?ZmHVo70dfc&i0YR^Ngi!S9sUgs=$o3@#HC%CQu zVQC`S45oEeh|jSnr+og)sTZqwa2I)47*(fZT#(W8!`-9Zy;6qLgoSGu9B}WLb7hl? z88&I^yEed4S&CCN?a5?v^bk(Vl}PyXGwU>GM2t;PmSNpbpHtHU+a1J#(B0!mwupEH zw;rx%DebY^>nqHS?|KnfDvh%GT!C#A|5Om|*u^AO|IuCwx`%4@a@bB%pH#=-1KxLa zbjEz*z>*1lZQFwhmK!2V5{PUZeb^X^Uf`mr12XwZyW{_)XI-Dwp^d6wiemLs)@wbi zU3G?ru3nki-kRLzghI&^&lIT()i-K?-7xn>@<#9fzC#ogsnDo!K@J0{^jh}hRaJN& z{ibwIo&VVG`D0ijVz|BHU|(N1W_wA%g>E-Eax{kOfYx~iljp+P(RK|g)-rMQeMmvy z((*C0SFE_XP-br2j#oH8GqufC4eI8U9Y+L|LR|uV;isPCOLT*u1c=0S;rZgj5z;9| zh|EoBPe5EDuv1f~9u_{~2FnD}f%B*1#Lh8U;n>G-ZT{ZGGb#i8CUPj7RjSCn?w0P5 zKhoz_P*T!@^*V|FVNCb1KmgS0o09cf(w2ym>O9n*(drOe(3*0>Fc;wFkHj;)J(iZ? zoW4;OptMr%e^64Db5bDZ#ENZV;vB@OCRSe?C-cdo;%prux9Zeuu?EivC0cW)NI zGdSky!u7EAf%|A)^T9bvTLxc0@{-`RxTi}zeIAV>Q9gq zK;mN`;>EkgTW=9)wSNt(sJ<@Dov3n|5eeBOv0ikcyh^yCRYS#ZhO58V{+M)?nt2qd z#?5E?4)0*$y56At!>t&@f6L6#Q`4ra&T2O9tYUic{Oi|E=xoVk3dOcuwcO=qLXACC zqBV*)0c~>o|GK;sJb<)u^32lEEI=jxofQ51yDL6~ms+5!Rewe^w_suE=!37z$gh-c z{I?d{&3|W-S!BRv1`Smnjr)AfHa?Rhz%h{`L)R|sSoe6BZAL!Vss!iNXBHsfsYbe&b8^!ZuDiZNXywi?5FihqLejtz z_L2IUu|w2se313^F_fge!~0jyDT`7Bl?xg3C=1U!Yx`f81x@eOv#_wPZqW*rvCnTS zsy;C?XV}`#roa&Ms`f^Qq{RdYH8`kdRK#5(}qDi5E_hn_= zoTT=W=t;BXXes8t+JQ%)QpYkj+?3X8@BD?Qrs!yf*}(IrmHv>*cV73+RY0ogpG)R9 zeR?*8*Xhpq(F1K8viTP7vC!;caqC1;(Z%;@`;$f#p$nt$Y9r3L9 z2AkAjZ|Oqu4_8-<@FKAnCKZfs`oqHK&lb(*PzC~|v^j169*CS6q0H?o$d`$D5f z*7ITop&U%4o%%KTBg?PB7ukFzWSJh{LF;n4YLkL9ygk>mV^zKR5b1>bv-d~e>!}W| z-P+NJ;t?7)5AJa^sn0d=DV1lh6+g(!?ps<^207P}t3rm1gzR~xMU8i8bQRTDlgjjy zdzMW2H`)HGS^6w0MR+veWSKdA#5uG#7&gE@a#sAANBK3{U?8@|Is@LW z#o42d=IdeNEaJ6>!JAu>4agm0f`MX6!v{T|8m)A8XHyfh%G9ZRJ{aeXDkU-ZqAB$~O{;w}Fzm>X z3O|ew*@JiqY%DaZ9@q2&m-WOn%4>UESEIlF1jfI=PALzBsXGxMD91PH-*ExXfHSSY z_>=V%&R2UlfQWMqxN3fI@uT7gUxabYFY8o*RXDWvgYl=_2r#hvld=4WyiaD6vm?IZ<4DmS6f(3*va@=aMvKUqia?_Eo@cy9CXfY?N_ zAZ8liSg(zV(7Kp;)>os0UK&uK!Bl((_yue-k-{W*VgQB+wQfEx%Bp=gEJx>O(B2M?qbml6hf(H{|xluIFM_*a#%A;xQa9xWE zMKP$e)DhC#0SM`S-rJa#j{ulT!lhfuaN9{>2zZ#jm{hd|16k;p&wqnBhZ}ie@GKCW5CQX1_D(@iism1F& z4JOG@CyAsP1@ItCNX^hIt!C{Ovq2=WJ3zK_a7=jDOQE2gW{SYbfDqaM#H&RcsL1H# z-1y9$0F}Oag622wFYzNJVSm=XBc+;n)EF-eb+i##|$Q?}mTW^mL9!(ay z*FBs2d^T%O3rg$gfNkcK%R=!3_i01fmR{V?i5V5nQ|$HgnOdIi+#=S!Jnkh9rCJBv zFf3kUhYP{(fgps%?V7qn6DBt2Mb#g$@L&7fP z-IY9Az1K!qm+Lh`t8d=`18Tm1gL%{50s zfj~`gC3^@1%cLb1AGx*q0yjGYwKcJ->60_`H_rx&*#Sh%s^*#9<%F|=hoDINjM7a@ zvIGGAi$iRs4gRT)8_)DTuxl)6g?>FdX=vKnaUU3-UzU;PZ{gg9!sl;a{rJ0PX0bRBS_HKZuyC#ZfFJt0?3U0dCcB!Jhg zb1*o30yxUlu_N>{dTkd8kIqzh8U}APk~RAnX~8Dgbp=&N7fH&;K)+!>G3hS?$POMD z>aT;EUBDzYvS3Ix-8W2>)=lqPITkVe1d};X-~8ERH&fwCKCr5Fr6}Y%Kvkl%GfKw; z$R5AC{pxIt*56k1c|oGWFL*_FS|!1d_%Z!XpiI4H#)F0*)&_W^;ze=e6xVbFb#l+= zCot1|J|^5t1wX=E>-G;B*(t@3kX-%B7Sb`Ol^SYE3q#hu2`AY1_m(dtE@7uNcv~lS zS8pjMT+mAeVx!qoK1Sa8U8)sX^Jjo_NuVm{L$J51=C_q#Aaq=5GRP}@8DE#cC>P*(M#M$JShkRz0wWZChH^|pZ08%%wINN$hgBT}Qc z3KsH!dy;QX=HBE_Bwyis9U>|0G_aEkBd?(3WDL{G1;R?2Jhz3bwv*F)`-9)dRZlhp zB?BL}u4nsROq*h+Zo;FlUOCND*VKLm@X);g=3ZdtK8G|&9u}Fd>G-Z$2lyiyCGT_I z7Lms-4wDYJ9QPJsGZz;bamFvvYywbnRUje$#f*)%F?SYrZ zI`UKnCyn$P!8LAKDcE>GH_LkAl*06Dn{K!oPh`JOdO0vn*~zbm>Z5de0e#?vnm$ME zf%rmZpRIcC06ZjtQ&v_9CO-8be#{c!9KX6)b^fHCOz3pBvHpO%n!g~}KC{Ezv%{!fnzGeZbd<~PT8I4)lJi>z diff --git a/docs/_static/images/new-workflow.png b/docs/_static/images/new-workflow.png new file mode 100644 index 0000000000000000000000000000000000000000..6b06fc3a00f2109e60c73376b6a7a4ebd794dbae GIT binary patch literal 40676 zcmYJb19T-%v^^Z#w#`X0v2EM-1QXk~F>!8e+qP}ncK-Rj_1^d0UcIXBU4`zd?y7V4 z-sgrZ$ce+l;J^R@0l`a3h$sO8fy@8_0VhL&{QROJp&0&?06Qv)3jtM4;hp|`2{P4? zG?SGDqWQ@~0f7Uf0D=Bj-{9q_7a+oKbIo+-w7PW1atue zBmg8SBBaQ>m$Rb#ZI_AK_u3&D?cCb=cef(q)qL z^3<_7^2KMCzo=9SqhhN>o)O~GmF=}@qx)-+?}WdO#t2^0gQd&G3X}KQLdj-3FE6j% zc30*12z1}p))qB2H7F!J4yT9HRoT9ov47w-9;a|n6kelRhSsLY#SBuQ``7Bbxy;93 zSJ&8whu`#iAY#ZjcQ;b9(l!`dUBaDPV$eGDN-vbf^bB++6h;!b!r9(S+JvrStPV;{E!E%-*4SQa9rUJlPDfH@nXLHt% zj<{OM3aaULjzUTYP4hGwpR2o@e8nGaDpDLldeDt#F`ZpXV2BqpbH!Rq0&e6P1m+W< zU@a9)5WF}YkJRhU&0}1z(ndx`AP{*J+MF!!%TI0jEcK}3*_JNm-T#a*GD7*~$z9lw z5<2U<8iY%$Z6+i`2rruu?-EMa#*f#5YL6Y+88CE zWF$o*V7+6egNYIRi}bUuLvkc=1V(AQK(d#`7=uY3O{Caeid$4Zjw#;b82zU&t__8E zSDLX+aHLSS$a|!~1wVPW*=;D~^6%`xPE-J;di{ewFnXKe!&tLNH!fHP=w-9Red?0j zD*s9)lbrL2mh%sZeV9kTNaW!MG7f^evcn&)4xtVa$0_O|Wj``&FP(_sAm6BZAzS00 z>U9BDM-QiT7?WqEv;*}(9Q2bLivenw93q{&Fg)gqTvc1`e!=c)-ofRjRoaSPl$)Yf z^dH92AuiMy%)ka-UOd{$X+-vK;(yqgP47%t$}OG!`zIWrG-HtLsmsS2Gr}*(;z_-N zR(^wA==Wt?7FlK`A0qpf6K1*cx_{ExJEq4D z);9^~xo2FgjSx9$Bg%fPz%e|`5q^u{2mrfDnmX@Iy-pnd{c~M1T9DAl)J}Yx4U?&1 zp=7Bgw}RXGsRzZCG;)WBwlmZ7d2m&FT}tuNi*&3hf8QMvPlke|Nl~O4Qh(AK35k0v z{PQ5zD&4O5`CNV42miG^L=IW!iw+XGQK3U=l~ThSpnX{V|WI&H3M~s zZlOV<`87Z7=_qG-TU%S=!SOTKrIGi8A}ZcZ-n<9Hn+^(@6uQCrC;;DS zw)@^TkY5k!ab~H~s3z=n`EO+hd+_|aSS)-L?C}DV!eAUHd^ilCx#cYj!j9hS ze}uYEnOld&R%Y8s*9a#sPJLSU%Pr} zGG&}$!F%8W@EpFRCKkxAD<)SmdMs!j^$6|wvzOqlnd{CeA8CKj!;k_C_5?s}5L(lx zSu)h^c~+q=kultE)ZV{yt}+*Bq{D`rM`An?ZhGrDNF_9@?}9RMt6yv1h?J*-M4oVg zkoa!Ckedr5i3JhPdw+Dzuluflur{Lyl*4=2FVyLFQ*3suHSVGYcfg=x`(E}TS{Lhd z5q!(Et264h-|W9xil{NDmg`6Ee(T;3-oiq*94_Y?jfdhsg!(bO4?oTm+Ho57K7EU2 zuk-+_;Im7V+MZrs0cexKwC2ckC;CQv@*fR8T$}KsogVvAhKz)a(*eu2WC=GG7h68` z*~3}y8LBxrvd*NB9;pzq%hfZP*(Cd);9o0u(J&*d#ZU*q!EVd@m3n^XeDCMU3k}*M zk&I40{KV7Pm8(ehHsqu?eg0f-w}1cK7;G3Nqes0}>6)0b zDx-3@C%`|fy9@{y@za}=S}*wUA1TH=_@54sq zx@a1>rM;coJhEoAIp4p+uuv`QjQ?qTNE34UVlTzTPNQ{k0KB@fe6wEeb!euzxcEAD z^>%p4E+8_6VC!^q0zZuD{?DuR9TZdwjlK{^g_Zz^8S(jjJ?NxIyff~aEKFQSQTegN=du0I}V~g_n?PWwU!Mun|`mwx{?q zdBb;n-c@8wriHp$TxQtj_%UpBDcN<~FhYCsdi%VHj^@Jk*{7_u8oehS;M4qP$}^+5 zH}X9wWv6TT%_vENQ8Y(j#JC(!$p#*2ew#8pZ2lWnEN(KOvsJ82JfmfHr8{#zD_uIp zHf^f-d9}TqpPfHa^YX_x#2c$t!7T&zU@gOa&|lLMtesQ+hztb5E<|?}2+wvM6=i5= zg>OvDrUT2W_*))wc<*1~q`-sb67Rs7Gg;Ww;YzC-FlR0}x?*DCL!sM3&P+*9V8t-j(NJzcUjy5A3u621-_G;=M%e7AlXzt^%BS^BJUB>q_pcPjiGr!|U@f=k?) z$Ba#B!z|5-6{InZ7jzHoapL;oC_~o2ZcicoR6${b^yjIY$yU}jq)C>%oDsVyY@zgk z{dehk>3_2ds48v+-+jEW)oRrZ9}?U{Q+TIlp8bpwLM=#8@!2CS zJco-0ey^HLe{byi?lDs5)BCzz$3z6TyTGJl_NCsl)(XC@t!?vuMWVC0xwmqitMP+( z?fo0KKxk@5^BCf@Out0G@{hNb)8o+L`vM9xu75{8Qq%xhvO9gzo0aJ~e&nWjb`;gK zCRM*2Fl;)2e}osda8oI{Ygo$_{SOWb1fA19hKysj#yP5b#D6g6bpkxGN6z+ zevNWmUmD#_7}?}mHD^H)8oQb>&oODZ#y7{btl?#KTIFWBR|<}R8dZEH(l_aYg!R*A zN!axp=l64W@6u+hPAj|LZz`f@i|bhSVS)(h%EV_-|3$dFd(^kYU+ep_h`f1IWI;}g(1^odtkUW38>`2P??Ro*Z`Q)?lC_T1P`cePDvhtdXr^*8f*F@2Od8m5wvz(O_%Vi;+SEuDB}Q_vgd3^QB1R;Tz2As{}m9-yIVghG)EO7%%=Yn0_L@;Dz!Kih3@Xv z>9JH|wT@#-nsV_WnS-_ofzqGq>l+3@tjEXb+z$gF6BSTUQ1U1gvicN-PF;uvbta*S z%Zm~Qz(OrCJ&>3)iXN93=Pi@?vZTw>O2&YgAb7i-a$Q{2f9=$vMD|YM4w-$}*(l!{nrF*=AK+Zu}Nk&mj99NN1-jtFu>?jp4CZ2+Wh)diu zpX8q+7%>nC-&yECE9&IG(wNaxCY zh14Zz51Z*H4jw_vb=X6~B@DgQb)mY-*X7F>R-nfCoPFO`1g405Xi{3#toDWRB3jvN0B-B3YtirhY(|*Fg zL6T(RhuL@bB`2;_t{vi#6d zCK^>*WqqDe2e4erJp_F2CX3m8mD9P}+FD%pKQErE%<~WKzFH(4FNg7@-CjCvb2KA%(gulsZsB9}SOFvnDoW{HO&v~_YKiEI)v!mP*6)aXo~ zU1Lc4v&P4WzpJ;Pw|RNkzBbq}dxbpJKN|0)(t9_OBoGvOQbq?rvLMIxe zoNvWDRK%Ta0mRo`F3VIZ*J=#=1?72J-_NVvQ(v*Q;^t&aFX++$f@;^F%nL*mvCVWc z$GJkEdIF|lf4wb+n}5~SSo(j_=8997j>4S9D>rs@p~mvnLYV&ZD8uJzxba>JNU>Kqno55iU-wli$7rn~^l zCVytE_(H@&7rkhoC${ZpD#BafHK@vc!dcHQxcxTEgzVjxL*y~2btZ;B6Z+WluXL*o z0SfZzY?Y$pDX~u%?so^HJv}|a(kVJ!UajMuy?PsM-j0nFo^(i=izP7|^D3&efTj<&Xc%@VrERd-7 z7-TBMTkye+)U|V1RtT8I0#ES%n1`u0luh{Bf$ovKiWlRZQ3YCmD3cmn*6w0F@T=#97{P(Yu6&DX-3a!Q`r>iHYO^u!;+JFddjP`9j zOJPRDNbq_bT;}nL4u{Dv(e>CW!{>N`?l?JQZ(8n!t@SbWO`sM*#F!(}zF)PEBQ$~H z5OWanj{h^yaC$ax7*itqk(s?d`mR?6`-md1$=TCRLK(cQYmBv(P}|h^gbmE&uHRj+ zH)9Bo1daVZlF^5i>kZ+&zS+bdrT(DFmH?er2W9gKD0p~qv4RNop{Uc>5mH@dk)Xlx z6kL3B ztK@Q(bXLbAp8_AZ@uPRS-tDQ3(3bnVNJgQV4$;W)9N({x8pFf!RNu>J_80?E1H0Xj~g{s!ST;P z&4BQ)j)~gUE9m@UIWMJF0o-dE^pMUUo8js2ajwkRL#F#hM|D|~zZly1TOG+!H@%^D z2BvH*gtYP4m#JwQdsS$0wa_N7J2YPPP0*PJ4^M2lDoj5p zC!zNZut(Bxj#~iDNq3(gpLnwziTzG=9v-uUin}L~nYrrMMIKpFo?u}LcT6>}`>w*p zA(ofJYPj=pIQ+!*_r_Z#jbX0-FIsPAjCXP)M@?4)rcae=vA~Oh4|i&aMHAfSj)ORT zOQ!zXRw18G-S1AZVF&{j$)@6 zd=jAs)5~VdUx4EN&@rP<&qY8_t@k?H1eI!kf4$KAL^?}#41PvRN{GW=XpV1{Vto+> zJW;CC*#@W8v#kYO0s!-M2hcwB*3!apfDQ^0xXuI+!{>LIsFpwmLy#)(?DL~5{%?p5 z8rFpD(16+S28a=iVeXH(3}x-pOh;`MW|jK6yFhtq!%vimIq-ea%(*)LuHG5$#$2af zzHK{RIxh*nLW4jU3#&2H>I78-$q8*&X9VB^hZb_SH4w|%@q)pr@b`Q(4*8mNm>R_o0ynY8?psj)0(>~pOJsd}T6OzGCqgDN(#Uf$hQbB$gF+z5cV$D>?784z{C@D`&{@OjQk#~v zyfY{UH3-o#PF%x7$I64pCLsY_+^6iVBF6fzl|#@kF(;oKXAKG53;Y;u&|>tijYCow zzn(Pb!#3n+cI{-XE#qq_tOw~ou#wwCrIEuNY+;>BKU;R>YCQgvG>iH4*GD8HTFVGz z2HPoy^!xjJaZwS&_xrWD$HR`@+(7txv)yn%XbxPq(fCf82uwFzKTNZRc1syyRo06? zir8(B>UJqA0Gbn5*FBPy2G&SEi;MNNemafID3X}o?)LssbybN5z2;2yH=bJX@-4G@ zH0ag#AEZ7Pw504w(3^=hFSo(Fl5Gv~`8p4Cgt-A0WNMSQxo0ASvR)c=kULZ`G>|Cx zOc}8j^?maelKMT{sEIK>4mZ+1iRlJ!a7O=E?QYTZX$UMUb{u5wU|$&`gBeEWs_j%> zmu~%caJnD>ZF!PpMlE87_oK9S^Pk={X6LtuM(gDi8qK=z{a66Y3MzV|&AA#zkysAT z=n}B zUs@3HrxNDyYlUj|?5CE%^dh#LIVZl?mxr-O>E?N5&K@;d(Dl;WVJ@5vl;o`pc+^lA zH&_6*?d7U%W9i{O3~(EdSG#@aL{nXboYDNR%*)5-oKd!9VI-_5crjBo3>C))YDZ-? zqx!Qn(#eHZAp}LibM!f{D*J#c3h`e!57KUR#tFuXTpr9euI}6ysWnEqq85v7tUjp= z&o8(8`}|)oOO@Jo-LDir0D0bfIKWr(clhzO}NFdAhEuWfOW)h)NnMNeVi zBGvc%<1!37%}$*ZHUd76M!Q_r!TB<=FnGQUjTVYDxX8{3fn-dHsXv6Rlo&Wdm)8@^ z;Bo>fhODyLchN_YK>ehi*u~^n>2Ckg1cw>p zYpt}1$Hb|h{VfXqM%XMp*;I3rz^%?vM1Lu@u7&8{zpZLiw5T=UGgUaSj4u2G+8(d7 zd{pt>Y?0AWY;*(GG-h&37J2fP;qJVc92nx`$)$NS9! z{N;6+(D(Dzyn-?eR*MYS2^Tcm4MZQl=|Ld{^AT;d|D=dW0x=qwZDAVN2F91hY{m~5 z%a*EGtcPpmdn%*Yc68e!8|SYas0+P)F0HZJWS^qx#+U(P_c?nt2F(Z?RhJk$u0X&3 zLb2Oibom;!#f9WxNH*0UP5IuUzBcsB?=L1H`#BhV!egSFpv>0%>isdzzhQF~O<8RR z_*><3e-(t7wt?mc7fPB$ue)?#r}pK)EIHt9S9B zr=@SoF_z6cWz%j{hcmsvLz z*2!7y3_Y=;X@6ia7L%6(eIc4b>I7L3CUqO02llL3A9dsWNaML_;F=21y@D{#@2F7p z#y$>(9_Tl6m%A}Ij=NBb023c6xH=GiKx~{`)X*`BlK$F==p7z;t+GF;;{NCoW4U8J zR^=^?fj1b}$`6cw+yEOhyEku2jc+bi|2MAe*L}eczFTq+2shjx+-CQc_2q4Q=SV^g z?)1VEx%C=ooA`3V5&N?c)*9pu8cy6#xFljc_6SYfS~)iOL5yd7 z?6^8WvfRsYbe+ocVbKIeR@`o`hyNjxZxjmp-ley_;78xxsxj95U3yF^UkJVp!SCdF z1wRB19;lT=-_hx_bS9~4n|!7{7CG?5rUDW#`fTx8BaP0{uCb-Tl`beQ_O3ru zR`Yzadku(v*4=$vj;@Dn!j_~!hgBt9H|}G^HLGPJnc#7s&A_`#PA|7B&@*Rbh0}&J zE`z%`uSOrr26bU(+ffV~r=bkg0Hp=!MxSnl8(}%I{y}(GN=2O3WY#LM0nzt;MDN0Ba!+O5GAb@tA+X^}gKt{hJREkO=ScIL z%#cQ-AM}EPYB72)Ol8DWsa2xR%BXq^PAv^==tZ`5REyzv-~DwaPM&&`wAw$^KYnbN z71>qL+X*NPN4$Y8(w1+*7$q6`s_XQP5H=Q$Bivo{Fy zSMqs-+QI3QD?-KWc>2Mtmu)r7Lb*RbGE~n6`aiqer+k;adfRV;ROMNkQ&Y+;7oYxA z4K9)LSLIB_nA0^1h#58~Kw3M8&f}=rdhf^_%&vIPc)Fbgy+9;u5!(=H5TQE4X^i7! zb?ZK2=d%c%bu8vM?xh-;QAImA2k%%-nEtILTYMd7Dq(+!eVvgq4Fw*8;A0XS{-DSG z>^}&$tAQ+Z$!!21H*P3rT8mkeW^rXj7b>e<@8)O_F_k+zed|daZLX@0dK2D<5kV@5 z9zn%Dom??25q*sLVK>y_p0*qHE7snNmljL>1TjP33OoV8RC7m_*#oEaX!|lBl7Gh+ z|B=Q5yU4BC+n>Jhi~a4e7^;PIpJ0R>y@g(Ux)>8=ZYOROhll)UGIaL?b(eW}&HiTQ zJ{VWY{Y$2;v&ZR#;#DG3+@R~GX)#!045kwA{Q@SySNw=3Z{cD%dwGBm zNP~PSwOsQ)8n_cT4%(g6|W&=fuH&^nQ z3eeG_U$nl_39hH1-;Z1EhocW$QOnrw!7+TwhDligSzhtT_hCv$2V(6wqGmHOTGu8G z<*)AjaAq7U+!F8SLyV7|?r7gTc9GfW&1LxDOGEn6gCr9Pgl5WN2oSo_f|$b-F!MLN z^`1_U1e>jaZp6e!8 zUrNo(gby3HU#nb6XOu)*)q>(aU1cq>^7S{?Z~yyUtr;_j-gfnJ&FsIeh9kj z2ile(Oy(ptplw{q0=i6g){{NCF=Mroi%k|HSF5wvF@Vx$Wi;$!x!lf~gpyc{T?j|M z_}gU$dpbN|vqUI7FoYZyp17mqXqNTm(?Jma54JeP24Kl?W&T_bUZbnzrReK!Aqlr` zjOqGnavE>vrUD<7`~GVlHd1M$uJOzdC*DA8{vfs+#f-}|L7yCvWzK^GSMJNDmREp; zB43Hr^pL!7?aO3l!BlpNu`&|)zhKuM*ihDV`(UZW*3O0wv~rOR<=9>ts&d8+Zg7G` zY%crgG$79!j0d_fMCv9sv4Ld8drm~noF1Y-J&RVMUsLdJs@)1XOwi(}kR2*FQf48?c*n%@FIlA(r8-7aEsWpo^u)OKve|~<9 zzUkS1M|5Lq`f1R6vHvn&eIIdoq;0TJ1>6?|V{9c)iKrX@Og=5{rIi6310%07^)yA=&w=@-=9*hO=`p@n1{hd_+wfHv5XDH^rwqn<9?Vf0PL{>-r zEK6n1KUnjoV~UNC9K) zU60c>_rs}l^oO)*Y*2-S`TF`6LjLv&sz$dV+&(-xmzxU)kb@ot$az4t!tW6b@##(1 zmmbL)!9k3S*oqJD@nvLm?WXGWNN@DnFoCF65ppK)LpL_4mD4Z+lh0mL>nHkOvUYo= zT?Zx-L+s|kgoK=ohb;9%TOpb>iv8P+GdF_S*SD~iC|d}BU7qe@HCES~U+p_w>nG7S zA(T%4BQF#Jnhk~`zU!mQ8&sj=@zXtMqMIXn^J9-^%zZlpvM=H(rW-0|!&lj2c_#N! z4^@Ih8A~ll@J4RkR#Sdqg2l3pe&LRaHa*3GJ!2LTmH;Xgv80_@E5NFh^^Q||5qXF8 z*N;knsecGdzlQEnL{)+z606|`MOm5)!$_H(*{3iVbZ#I^binQSrWv?Psm{BaEX!o+6&8zQstw+Q{tY%gs&1v?pgC7P=`_!*y zNOirhT}JX2cwnJ#y|Dso7}=6A<|H&+montb43|SbTz-Do5 zVLD0-}X~@+&HZc~#h@P_$xC|a7Ex-#& z$2r9LRn8_w9WRF5qd6uu2T@(GqCQ_gt1NNzC%5^0Knx?lKTb(f>|aQ;v5rFhaEHlalP>oZJ9R;GFGOpt z>T4*W@5cu$;{k*DRLk{e+LVNO4!1n1U#eL98n5!alFudwY$fw86>;qf zcCbT#o~f7Ci(7NmtSI+LbPR!W{@Imea!U4?YdLwh3(Ku< zSTW_jB1HuU1hahKTs5;Gm!Rt~(p>wh;|~V}rmnD&{1?Q)JUGtC6g&Dszc{JMjIqlYHB)A^IEo=r#RRWkHGCZz9- zcYQH24)Vm=MZUlPY4zoO96sjd2Qs@P&wGQ6hKQeBhACY|qu~6w5r#wG3__?=#WrMZ zpmub6V^3!@de6FWFlR<;x@|ib?_cG`Y$+mNxx3I!9FCCMy~>Vbg>K9*@)I|579 zwSID(jwLcbVg-*M+By+WSEMiw$1Q(Ahl>EcGno(a5F3@OvV0cgOt&m@{1?Ir$w;PX z{fqq!w`IImDOp@M6c3A=eP9T@omr+D|Rn?^|erg?v5yNlcjaCYN^I z2Z{Z7sW)#ixc4>YX4&k<=So1_Eq1XYOCpQo;8`yLwAq}s2}fw68q2Qrtr)0R^Tluv zRl+=`ViUWE>`{;( z;%=B%;h;wGi$z<9A2ka9&$!KzdSEo7Q~@Nb<{(n5lQ8#0% zwuZa4CkC^wH7#!=#P>>3dEV*iznke_-H0k!bzqq7zi91l-61K`ZxBvmr7L4A9?6k{5aqrU&lGyQ|Tn9M7O!^$G8xQ=8Jvz# zHJn)HrS>aQKYY;-2~gzg2BRG*G+{=NiC6A&@nSto;zEJX?eTj68XB6WP9aa|;%d;W z2&}=?@;xM(Ms;j>ITepXOwRM)ynsnkUN>MC?_xGon6{*gO(^m=a<2AtDN?RO|LL?y z6GSYQkb*Oo$UnzOnqbxV5p2?SpMYvsF*S_pmN<+rM8KcKWsdo&O6KdR=t^!9mSLN<@$ZoeHVZ(Xi>2#;X-VOBeKXvdy z7u&LKhBBBn4%DUNn$=<77o@xoZBGDWUcvKuQQv5_yw&V#*Bf=WK3 zSSvz+GGK4b8!yO>DaC%><*ngZF!wCc3ar( zR#A-Pgm$CeCkT3dKc2KR*ld1nsK6tN`{!-38q5&IRDJG0Z?9INt;46kyuCeLF30jv zBbv`afI~#|9Y|4*+!o$stb<&IiB&j@R?*>z>+ax)nO(E|K_i9gzdd{7q*9YOVK?P6Zl>^wXI&##Zy;kPIHtLtu$MVf&yF6Xn3o4v@F z052{do`EpTOyTyk9N#UsOIBO$CaXchF?4!DYcP@icy$#`&%Qg|C5F8{4~nmWB|y2x zO3Jbfd8a6H3x+T0>0GqZ?sIcVv&2)g^1Klud5{W?-pueg3HF(JLhFRvcTF@_WW#De z7AW(=?$Dbg*+mPKPPDX~wCp`2fLmckihq}o7C2-6&pOWw(LXpm)(6x&D2&LVn9FtF zmoHEBZp(c!uo#BTw#$g9h2kGwQIdh=>1@4uPM3FWc!`@@kpY!^fqM0;??*3T7ybaF zP7P3iJcUdTYtX!g{V5TO@nln_w!7%VXyS=P2pld35>hU{cC|zO%i`u9Czo1?IwOTX zNA~-EuYw8h`jx5+oUwTIOzzm9+MX+fm|#EJq^E>2}uowa(&P$IP1g!Y$Xo)cJ`vGx?c2EOrag4JinP3&Opq;tsRKlWjS z0Mrm4gcpCK9gQ}VaE#HrWl%_j-scdcHIHVIH@$sr1qa1-c(hb z5EhT>ijxG@g5`KRsE(%(i$XqnhVW7;mDnO{oxV9H&2tUwr_knA=%RAEm2-=rRt<}n z<-(4Hn_gERyFm*Xmc*(NsFE4k{2s3(L-mDypVI^_14WLzm1D^jdx%VxzGnI#!N#eu zn2!i{s!bwkwsv>4$5ogNN6`uUN)xKxJ$;XDYIFzM3aR}eEac@7ncQxpoBfq$?RTZr zmNyfR((AVzocZPQp<3Flx@EE zB{W3B)QP778B-xDA?|y;Je%de8T*x~ENub!kxLth4^?f3TVHGVv}lZ{?LJ$4e|+s1 zq6d;3oC8mT5vsZS>EWUj`e>Okhp4tgR2ZtZyH(*{zM8nrYfdP7`}kXrz&xcba-U`x zw=Sx?ZC`pom#g$XPlpK~_gO+$*`xH2+^h#U0wsg1yr$<8O|6qH)~dPm_p5@3VbG~1 ze6x~r=MnOHI9lF*gZ5mJQ?6)qtR}L)WRre#xv)?-I?T`EJ*YRjr)wbwySug8z3mch z^6ERcu93Sia>dXIIhaJwdotWL3J*cmUfkn{u`X(*92)d_)5_OQV{yA0${riE7O%w^ zV*W>}X#ldwrgcc;+`hOV)Aac=R0`cKLq7Jkjgi4eY`*!6k+gs?$SZwa>jD3YJWh?TuK3`h|C39ubEL^DoYmzbamp~S0 zZ4g8X^p~rqN2We;H*(v)+lkZgDct41OCq#T3M2e2(f#sgNtM$&{gBrq<_861u7@%9 z5@^jAXUpvLrnMQb4z*~kYYfO-;&vb#o`x!#r|yZBLITBsd&>E0~s zza&0gFHiTdvie$I!r{d9iNZ1`o1h@+)M{KJ7^)jpzoR`rJvC!S!xV^h0JF;GJr5BJD zvYo0&#}bYiYa2zT-ieYB%XA~v%e!_&zWvR`Y)U?C_f-!K*osW~z9$3*3k%@OriWZK zaluj=%?Xj~h||e8ViNN>${mz?RT@A+k01s{^25p#VZE(9SchCbn>afu1Qm|VCnmi( z*N3WV$x2A^*k@xdAHppL?9G^z;0c|l2%w5#Q3&2yaMob#(K(EcULMb{7QCtb(>KUF zLrf9I4_Gg4FXmRdj<#;zX&8zPX5pvU4Y#RHdR?f3osGxDd4@-%sa7tzDOY66|CmYkvF8 z5R!ZuUfVF@e`VBanel-EBoAPNAzV<%zNxL>yFQFzw~sF)is|+a>(dP!Yw>J$UXS8q z;yhNt$W}fXzNd)WeVr2NGiQP1u)AQ0pf(N=eJ!Z+-HikfqyG-#h4r|>CQSuPns1F^ zkuG{KnUg6#JPOV3Z+`g%<+;Tg%w?Dh0U55U;t>8tMr2c>)QgKZ+rQ#h2uBO;OG99p zd>^7nuVSH~G)Sf|c}rb(|0x&N8j>X^%1mHDPK@WtA-E53ODq;Pnkg)>N^*k&_6u&g zZxDupvWVz5^TQ=)>UCcZ*aUOiPsknz2WG^0g(;;z-E zmHTgvIl9G-gb6A6w5VP*z?-L&C%1PyBfaDdvxU5WZt;q-{($cNoO=1L9R0A90x7pp zL4g*e&_Q6e=Ofm(bF-$MpzB347roo#7D6d(2&HX{-#DEFB%&@a4w?;Zg2ZG*Ihy^Z zO(4pTo%vXyDrE{)g8TuNW}?7xzAx~X_B--pyg!nw3nQmqK@g&{cEjJ17T54;@;p&- zOIHgU83POCA$s)g)0ypddE1Fie?c0^)+01vK!s3qz2_}-6wzpSb4{3`yno}I-F7du z@S*S5h$&XoEnj)eq5h;d8+0)f3W;mT6092M2^L*RjZt{P3og~rjts}7SJ&e8wn}y% zKg>^)m>>w({k=iX&wKmg?AfMO{Q#}O{X9?-# z^Wr3}XK4G&PG8J$LYOgnLw2B+#@i9fEvO!7PRH-dWN>{abLmk+aPYc73GjwKUtQ!7 z{0}(d-DTRXYlt8mhrg70>aR)mfuq~)On=*EV~x~A{o)-o^C$Bk^*XN(rDl=^61pefi4umT-WbWN#T$s+y^gBJ1>cU)tb{-mU)%O*OQ_ryi+iYK-bV|kX2JH>Mn{xiC~`9u1>%%yu; z#Ul%`4{{UvE_Al<#4(F+`|Q~chUmw6%rU13%-Jbe4tgA~e zH6o0;CGeN4f5P_R%J~Dh2#Q?P;_Cb2i{OqNaMC}^(^Dg5NVxYmT~1$oY&3R5o+nF^ zl%U^Ns1S%Xndmu_!sk;W^t_`Ws5-toFdt%#Y_e@miR5Hoh0aS2Z=1B2FZ(BFtBPTy z6}eS2zl@p7rg3G&q)#XST$^qoDuOf>EIQY*gtg1%jZ38iW8y)0n!XUo*vk0WG$7t` zfQdNb^IQbWnfkjTLPUrZ#w z?ozKHUo-@VNk*`ZicR;<=W&Jg1$uH$^l5u?q#LoL81tuL0L${if=GVK09~03nV57k zG26zhyipRJBN>i|T;}o2*+gsw+P&U3V`Y+~+G^EWdr$u+zI>lVp}Ftlv_ zSlOzgX&IyuRG>Uh2ge0TfB3%Mpu=YE#(2+-?|d**L6@tY<`=QLrvWhr^pqy2q0{r< zXDiG3?`T_F!+DgIALk8_KS3SFV!@%p+j(un7+v)ClH&NMc=n{$3Ia5$*T&56S4#Uf zE5O{xf@Vm`*ss)eq@IVU2WZDk5u_^=yz$@WAhP<34gGxqCGVI5M_=$V00|uEQwYM;EcWc`v*SgQU(->})fQI_d z^EMy{tJ$iX;_UfHnWs^DkWfivAPttOuw#Io_a!Byolkv@0tN<*DFRd+O(X04-Z478 zhy(=-8yOSh(x`>IfQbZWRM*z{Mw48#Q>NLez2_M1C!L3*ad0(+)fys=VkXw(tl*sW z&&<}edxQvKfI6As5u2x=2v`kv#Y|)W@F~6+R_<_c*eiYI{Kf8Y7hz}3;kww z23>bqle4wh4xWw(w6=ELws}(GpCDC={|Qpf4GkT>TyN9O(s>%zZ0r}0`c?n-6W}^Q z7aDg6<9qZ_IwniOkOR(=G*s=i(q9vet@O!XPv)xPG;=Mk8 z3_ELeW8!(B7`NL&6?Ihyh1+<5c*c>m_)QK2=uB_~Ok#ScdCmU~y=E&N{-`gHOZ1wMIH-^aJFH`p zIIW`og!j&v@;hsks78m@2TMT7kOGBNy^_OxrJ(F6aGI}><} zOq#^G^N5QVL#(tQ>jTHjyEf_8&0f!T83@$|h6Gi2={ZtSd>|E6X3>ax%`Z#`+_0yZ zbcFqV{G1@$Q<_x1h<4*Jqv`X-!^6YZ5m8Pt-V&?QN%T`IPC=$Ut*ESM3a>==C77fx zNMSMDc=)bl?iJLsG1FXUFEb;&EzW~@# z<#l;nen>6=R+}3cu3+79vMnYIyDPULak~{V_d(9RI3woW{a%R&H}Dv-#t0uc0{8!8 z?ybVA+`jNpfu$~x25IRKkdp2W5orMdX=$Xpk?xRA>F&;@NQxjKozk7sapuy!_wRph z&bc`k=Xur@%=yju#u)RR?}*P#)SXeP-73auf=^H3>V<@d_SyR8%bl`-T!A+fdO`dz z22yzKHPG7%tBcdWY7rxVr$h;zlefy%&3j&IwCu%pkVmsXCkKP5bc^X&9z!fcU^{PW zEHC(66aVm9=>9%X67(?*WmQoTOiZA{qA(>aOy5o=E%|;Xz_$G%-FPq&DQ$QZ4PPkT zNBF0iw4?|GUD}G5zTrQ5OavMXTM~(&k$en)4E$*%j}fqVNa|K|XaC{u{Ctoo$Pt5( zI1_MR{~b*P(?%5{v1l`v5NG!v#rOYZ>p&$XkI4pn5)P%?PB;oKqDH+5NuJ%6pCk5_gV_TAIuc4=syR&q$DIh zvgqFS(Paua@1@;5)|NHehnBvn$+unZ7QQ|Z2bAR2SkZA8zHTA5E{OAn`5;u{p`PAe z?JqRh_sm;RuHrxjq+l_S$fTsCA)%|Ct)C^C9!DXGi4(msR7GnUg6?joJ8xQ?b~|n_ z!tTd}uXiamuUiQU^1gISH__T|2ayFFiyYO-G!L%WI~6Tbbau6wiBvOkz1 z7?$L|J`)C==P+Wa0#CY+Ixaog8uOQ(>L#%)b{T+$au z*Ks_^pA3jA^2I`^3!mC&FnWJ@-1Do83lOP^zJ$a}#ldkhsvzuWZ*SP`Q@`A>;AJsZ z7-RVJr7YmfI|g^qnve!hLe|jpb~6BC6NA{UYC>E3}sejhL%;hC$}@ z4h}+dDTwdAIzLxz1zT43w&ipr?Y5Bs}`x!;rU*kqaiEcR$-XxFSWzdG9=qDK*9ff@zeOUozy?B3VcXEXOn zJm^bj=WC+?l*cG|&l*9+i_+Mat>n><5l=K_r+w~mwRqjQ&31+PbM9qxgM*D$sDy-C z2h-*mghXz!7nNDi%hIwGFG@6>U{?+3%{wf;Xpx-!DiRgt9r~xd&#v(LzH?$7oShZ? zN>Ru3U0l?&N?1)vk||3^MhL>^;d9u)BqR(mCO$?X)hMJnq;t&2i*24HMoi~-?(V{1 zm(|vu-5Aah7ZbA~t!@yLbtS>_jb2>z@$uoFcR?B#v*ObK;?MlHna4mfcfij_2}05r zBwmxMi1TtBll3=eq}utPD@_h_2_wM^ttqbOkIrVOcZ z|Fc8nz{LxKRjGVpwf{9ln2>5|5lG|?8#cNogDB=_qIS2(N zKo0od04rF)(j7swsPX@siSG`uqg3wU{$l?d;3yc(N{p3VtqvrwQQI1%5RT?kSSfy1 zRkB>EOnz*jJD74mkXMdj<&W%WJV2O1CV8&8(%fZKdyj9WtfJ$RVS+$bEZ`mi<=G>m zKaD0y5s2Z9G;+<}y4LDBy~jeO!Ksy?w1*PIPt;wkN>J-1Cm>*YoKrdsRCA&4KzsCQ zV`R#QTBm^-%~w?itndg40hDzC&!~L&*1i_Dq;$>otV@!-kBCgkP6I`uPa3otkbvR+ z-Ge}0BW51Wfi$7?y7V<4Dh^{xSg3&$8(SOz#qhJZ5{d{AtWj;H4F9IdFY_Ep506&O zvK$OQ|6{*<7GEdXnOJybH2=PrKgCZ7ne3*ir$$<)O{*B6Ki87;2hvk1ZXq8BZTlnj zuA;Cl_|UFq8Koo5W2T=;P2Vh$A2vyeD1PU3h*bo$vkQV(QQ4fZn_$gXK`zcej4Aj> z{Z0en|8Xy={0LvgKVZQUuA}um+VfBURfI+opnHwlaW4b@(}NkHdO^EviQFiE#T<&r z90a;CPpl~G|Cv2_7!0ma`tAd(5&voXH#q9}#@VE*@$Yz*g5cScr1q{kA1Hpn6Y8$d zm(|L&h?$&$*Vt9FOq26N0Z0mRznXJKqg++{(=8E>c@nMMiW-}s0^AN8oo}1r?pi4v zNc_{!qtVfr9}2Lru;!Ybxr0B#B@GjzjfbFcRoaxAI&;@{Csgop6W{umNoE8oyK&#I zu2gQTk*FaC0k@sW&LDIqo!Y02ewNFi_>t5L}%#KFmlXW-#!6z_?&#b-{Uw>j}x zi2+FwYm$o_lT9@(u82WiQ<@EC_LpPySUk0X_8a#+WG)u&I_|^~_omB;3&`xl2;X%l z>OQ!CA8S03H^b$q4`0`-N$;~CNo7!DT1rZ>?ionK5W3l6H&UVGy*us15YlM1`6T4) z|L4`(7(jMZ7Zrmm-n3nQpU|C!5a3GqQnGq32g74P)RkYxC54T!5`e4~>*pP?+%k$^#ttTWfofjah@@-h0U=F84W5NxQ=E*MD|} z;?o69+OG$n932_-yB~cG;dkE4k{#~P&!p22%v+3(etCF^&#bK&rMfhnetv#lxJ5u6 zqs1@k;)RHnV97|$b zH#A)Ly0YqvrO6}|n9RZh)w2B9^*0&S%jMEIHm5#+kX&T-k$DeR4hzAmwD9e`=RB|y zmdJGFF_OCq-r3uB0cPfMkjn{>NHfR`eYoGzp?~8>?bD}{&tWwl?jOGA<>mE&7nNAZ z;~<)%P$5{@d2iN!NT@{O;rnf1$-2gh!YTbLblNVrlGp$eyaMS$m^>M-TSe0DXI}7p zLNb1}DrFi-04QK(C>?dWF@-N3!8;U^pqxX>T#QJ=rxn{{6BBCg7r$cZ{K{h^{jVP!d6 z>8m7g>{=X7D91PKL2fdT#+Ku-IWnnTJH9*D!adSGIh-Tib`qPd8r6z|N3Hl{CbcA} zVBLG(+k46HKA_mf8}`+8y)OEs$>JH_5@m{(S!xDCH0#Vh)E@TicL;OZES_4;R6b-= zq?a{{+3&b|KR?@ad4gTf4f!Z|pVzT-?Yj&yE{3q8gcKwUhfcn5gi>S!(RuIVzyUF5 zxVi-4L}g`VL57#xbU2}S5JrV+Q}sL!@*gXR_LX zE=b$U7%zTHyxq-j-4<}Gchl_1s9rIy@P%p&!FYz9F zUibty*)9i~lOp!ZB7R==hvcIDIy_*tnfj3L`YoZ9N6+zIXU7PKDOz|l;(B;wI$zuE zTI#uzT(W@cx3%xnD!Y=9*toc+Ux?X~&F(IRm7nV>qcGkF-ekHUN~^yn*v;!pR^ukf zji2$5&az)0B-pgMeNBtqFaDt&+%K$V>N-nubZ#zzr+)t8c>N(NNg-o#>ddX@6)UVN z5DzOCq-~$0Z`+gWYm&`dOf@?K8i&ksM$K^1qoN4+?`nw2^(JWJ)5bIDjYagbH198} z=R$J=ywE``lE^d+Hry;3R`387c2a(a%Yv3BHr-%#_2qboj+WN=^n8?VkSp8-nL|7f zok_WNaqEZvWv$s%J~m>D*u977xex*CTII4{D7VEt2@}zF0c)ec%>cWLp@ z`!66iEXjp*^Jf@$w!6tf0Nq3MuNbhjS}{XmXn60KkxVT>{KeR(Tf<241RmMvvXa@4 zM6kLNdMhJhxOGD7g3O*LMo4m9hUEhNTfc=_n+@_nXVF7iI{4)GS1F;45xq zl_(y4##r!Xkh{UB> zp|9+k1Ro*i`d%hMJeMlxlxDZ>HY3d+yiU#nZDjLbr}3;et0UB}|Cm52Qso;;S(1_d zhF;un+L-E?*!I<{4*PXC<=8yEPt>J@uOqXQbrTbldJ79%sfQ$|U%TVGG`iQ8Pto9; z9mjq>efBYNgild(rY0K6H;zT}X5$0Kmy*}pa|H_Rt#hKfDM@LpCaeC#b3MGK7jKIv zor;%wUlJS9$Q24i49eC`RC$Oz(xUcRG(a=!q5_ z!>=lkOlmxUqP3|GUU!eXHpw$1!;hmXOq*g)9+F3H^korPdg7ND!na8`;!IaMgI_`R zZxj)?nZcCh>-TC5HJqUgu79kT9uat)+%lCy`a|E8I*Xb|La9a?^|B;ko%GcpwU{Zj ztpB_0yLu_5e_+2H9Ko4+tQ6r;!Zs~PXsi|4xTs8s6$f+F4@owPBhi^v^21R~PlBJN zWyD?Bkd4HC=xOuZt0U_B|9JgPwa~Hk?f5( z`%iFjjVis%z0|P5^1zBRurSHc-)>hDU-l~H@NgRL&$peOW@Cqojzl(rReu zF@RhRR^9C^Qiy*n6m{k4;SprA+B{mtMSA&Nm{FZnz(sL2iu8U|Q2*;=U&21FQ-Ab z55w!?4F##6t#KFcx@6-C*i|!DV6d!J7KAHlpL8W2I{E zfpHjqmPTG-E~-@{yaUdACy)x;4(z%8#@+ey+ZJa~faGCJ+OuHae2_vOO7NuH*?ztK z>U4%xV@Rj1mgID23W?1CE$BB)?9O$+P4W?5bZE`OkB>G+Lm|sOG3>^JNusWk$m&&i z$sTue;955L`T|5J%4PE6luNZ^nThLDsslS=!vjf^?NlmwKgW_U{~E=n{=m&~AZdw8 zik04Sq5MgKIxFMN@{q7X&sWQ-+7Pq6l-#|Aj?DC4;dGvu1VfA?CBtqGtFd6O%!s~t zqBMjL%d;DY$a3E15Gt1cehv8@_z5KenHE5b-l|Xi=7+SoIfLW{O%^cI^BEQ zYN`a-IPzxHMl{?`$`JWe7km5Lf-g(v5S{y|v2Su_O|1=WA8yUkt!Q@YX7-nj=4$9V~H zvu!rnIsl!6+>uqyKx4BYA{SfX(xZUCNK{Au5#qjU50s_8m7)hrhZ=ibry! zF*Ng3)(27PKpBT-T(Tk6>NCTrC%kq>vpr~_Y3WB>)efvvU73{FI9_KX60=kE;;-fFf#6T*dG4y+@{%b8;>c2i1nUA zOewb4-NGFu_N!E095y~iqL4qQ3~Ug~h9RyWpv(|!?;iPd)<_$%nU8&O4VLq*1{Zdg zWT+V7XrT6zo*%#4SYe`cq{J=E8tJpaeum&>MDEx$r^Qd@@&Hg$rq1!*@FIHXLhbCY-?CNAUOq7oCGSmAKc^68WA6Fpv8D=8J(Z}%c z?+VBlzzWbFdb!%#+V0NO=c%j(y9yk^&dwIR#NP%_zR`kj@oo?z1P<-VI?sWY=ANO~u?^og?%0#fVo*RrBe?MrycDQz__PqMe&22q> zGR8jCdNuDJ(AoyiUjGqF&#T;~ik$Lw;BFbCR=ZTk^?0|T-7-z$R+lqWvW@Mt~L@SXSXx{2H4r64Ft=G=CA z;ViKa&mF6Slcc1Iu~@dkR}QV@)V7+t&U#;cRNU!^-|crV?b z|00X@hkHSh9!sd@RUwQ9;r7XdAieBibaq<`@m4dcg)I67|Y;cV+nA?YX* zr3yn|pQs9!R5NW*1gQj3PhX$HS5r6o@hhQjX(`F6^WBnD<^!8E)3-HY5epzyqT2TU z@wQ{V3B^syNa=;vy_+vGleSnga`H(vc3(F3n&ShHYV09tAa=LX_#ryl3Akfr-kk~8 zu7c?E@m#cw?5|QmpJE8ndRs@IKEtN}msnasM4-dvIZ*=mcV~>HT`zvQ*!y-kDzWl^ zgzh9Z0v$fjK`p4z&4du|{12-NhgHU4JTnga-_%lnNTEMsw1WUZ(9WmFxv2k9PH}LY z6USYts_<{K%7=i$tqf9&ehw!A|8g;rfV`5QhbasGc?=w|kYWPw8>yyfKRnj`Pct4~ z0O@#Kk@!V_PquwWHrKa)jj6g@CYdYX=QB#B=jbbb$!T#A{qmD0aGDbV*J1`>TeV3L zirT>#lLQDQkKi<7j^tbO+C>qkLEE|q;moOY>yZ2R?>AZg&ie%h0WLy~w~NukUw-Xw zndbxq1wAbItw|!KJ{WBeCvBJdp)zxDe0chAvk1^k@*f{JjV$^jTWlG05{_Q640s!Kh3RG(}S zi5FZ}vp!Ho9$udI^M`(?cGIP@GuWG67KkNsfm|i4o2k#7^L! zZZV$A8;38W13D?Tm=y?TUX$N?)VkaT;D2nTxh3I!M;Ab#Ny$23rupfwEvAG{$}DD; z!in6^Mzue(44p^`AO8e?=3xE0NIVwbQcOk0P>ZqRfLh<#%*V-J|F9BzRuD@%8$}#A zIhyudlRy$RQ~E1eQcBT$w+JFoCrV`O+-Y7QhN~J;_7{BXxS%VY zUb5kM*Wc5G1cijW)QyZlO=_#+Ee&h^TD>6-CpP`JRj7}pW(LunV}lE|@L%+rn$al# z5gvSLaxw^dKw~%_^PsD;=RdNNCk`?_| z&;Cl!R8}gHMqIw-497K0{GZrTSL8@rQ?j zdcK^jtXs9Wx4M+b7BjSw9L%FpFXv@^S%8SCX<=QpQ@IDA;kL9N$4P z*kU3|>hnmekPG_ntL>=XRVAI{Y&wt3%}I>Z?T^QgAS25A`fM2N78B+zG?c8_2GGg=Q*GfAo#EW z11JEFj}+#Q$z!{ucbU$gSl%_)oPMNz#jWU7I^UqTnsHGlM*R2;%_g^4_6=P9^|3JD zmfmorLrG3Y@gmh@-Sl6n@7~zoe9kK~cM!dOx~W?ZKEv?M>1LWYTh~r@goU`nKg+Ht z<+#BsHN~mK$rN7SwaDl)ouj9u_#@nnL4t$u3V&O@@M-g;lcSq?vql5r`>_|g_${|E zWrwPHod7Mhed=b7Fz!Yu30}}x!1EI(Rq9rUIV2NGurIQ8kKEkOEH19Ip9ssnx#l{O z)(AOB;N-hm;I0aRSmfX@br~%ZYea?Gu^sLZs`xF+JPIWmi*QV6_t!GqRgISv3Q58_7 zmB$FfU2Bg;ALS*^FNldW>u^aMp6Ls|XvE{r`Y!Wf+TH!}^We{p--bHBkt@S#a^Sy0 zK+F!SL+D*=6@#GLD`FE-K^LD+7uuy(`#zqE?(8)Tfrp=o!h|rvHhb2J!;j-!3X%$f z&t0;|>n^Ja*z4Sg`|USQD}(7Dv)w1k%QWkVuC_`p-cq`Duv3vYYh0PV1$_#ba;g1% zUXm+`$jo?|E)nlhQ99{XKDWE)Lzzl6n*MbO!h3~1(t4FK9Zg2h5 zE0Xi1%0upDcmJO*)_My+J^8Yj&U06HcW+MjQ<&%nsYp^Qd4^BGr|wHaK~DSliFSae z$HqIT4NV#qBtQeRQUMap1-z|0z=8TSLxy;*RtY#;jexUNTw<xp1CRd<(x< zb371(9rp=rQ|78E!zqA3n1MOuUzf5&^P;zYE^N$`7 z1a&Ex6k-rTJku8=g801s8fffbO5ecd=$o0%(z$_| zlNQWw@Tq}nJx^}zr+-e-I;_fO_*BBUKY{k|rS^sc+ zD6b@i`;|-89s1t<@8~7D57dt&FE2OkKLCIPN5e?m z8Mtr%ZDxZD4rrT?39)wnArAC#065TRU6dXEg#<7qIFwM*lM@Iuj@*IQ~ z_HK27lB&BSbtRuiL269ZU!=%H09I~m>NC-cuFiJ!1(;n-Sr{dcl*$a zhrXU%&k@q>w0&xm$UKBj8or``@=lPjmXJSwduY4zJs24{&@6zSc< zyemxlb*X0(HKv#;0yZB6`l{Jyq$C@p9~9br4t)9)CD^SEZ?DTjJUYxrM=3P;ph!bR zLPR8#W7B`$9Swp={Rz^0MhytpKH6vc+*5)V>kS?6oSX*Jc!#FgTdmgGD#U{hm%G_o zK@gEAxJFl*lL&hW0RNrw0C0PIf70!M!9-5y*6(ip{UIv4dFDSu;~64uPgH=9@UwVe zt@#`uIXO8So9#{#o6#HDk_N!*{TN#642sA){YaC~BeBDvfbk3${Ho`|ZRZWszUne*CHR=}UlF=Co;AcH2cTO){Imzdz57T0G^V z7KG1Xr`N|jna%KYAQ{8}lql8Ef^vYrkV%8`UM3tYbkxmRfw(_ikIlR|I{DVKL4F)M zIh$5FDmJ*Q@hT^3ek%X_7@7CZLc@M>JY%NI(TkM_1#QWl_V*unU3B561xI;ic%H^7 z<&?fpZLkcfJzT~R7C4+Vba0#iY$)u(+Tbgs(r4g|I=y?9pdBq|3Ii_By=A2wJZ23&=4e?lHp+<#VTX-qHW?B= zmrA3)*pm>e98e5d-krVYs2z9{+uYBTMT%k5`aj8LJMOmo;u4?fkl^7toopy=h20Vw z9F2srPzLKMemY7p|5+-oiiv=A@Bp!WNx4WnPZsR|(rXY<`25{BQUM#7L6Fm9e0&p6 zj{Es9x6bw#_B9k<0o7`ksYg|bGUVa0p1-*|)o-m!$=k5Hs<%38upYW5P%gy8vf_8z zGkPuvSWgg=$@BF*scJPlw5z*YGksBVE$YR1vHn^_aBx;cGFh41V+j~P^5PZj?HBO| zqYFFsFD-jbCL7TdAb3F7bL*q+>w{Usi5G_}cl&2MQ-+#HK1pJoQW3!=F#1kuQto-O zjj=wDE_?M#JyG!1A8fX+Bcsy-he*SFrJkr%1dUOT9VP$X1#X5+~uX3F>K95OOs>q3^-)iBm zlhl_N0g*+DQ=7;gqdG`jZxy#X7GdZobI~Sk!OEA*|`|9i3_f&TfR0`@r#0|z%1*wp+ z=X8Y$xQ^texWo{1ejz=cCbnkXLAbe!BId`Nu-hIl`@ZCckFZbeljiCE;X#Rd*)Nmp zUG9a0%l#fQc2&ykgDA!Gp@Subla0fRB(`8gkJ(l;97F{9A>fbQTd?kk&=z}VV`KA8 zt;|cec0;?`rS1G|;Z62)qYXs~{|Vqe<$uFU{ngBx#Nq}-tK`pm!+e^5Z@r06dxT@S z33gBTG-)}IObrRH@vU(_7#ch&To!7DiWJ6z`K)U!*+apdi+5fe49{?Kq{4l6K#A{p z&pS_)$UP$=wcub=P{;N@`sYG^vtM7cvdGg~KzxA2*l`eALet~7-@&M+x1cy^5Z0=M z>PH$YMQe|Nlc=bv5!eCc{(Rjd_XmLq#SOE5y?%JXe8Im7CQzlZ__%KhGRuFLcuT;p zpqTGNaBRj0lE;5&Of-YIB5<^}Ki9kGi%)xP0Q~a#2n*-003*BYfSFG-syH$F(9;gIG?m2={Ci5%5Sx6!?er$ez=&BlV z;ctzMx726gjZWHErd5AnnyR7~%f|fpSZSY<9C2O>_7H-;tf0RxYH=)D|B%P_7sTE` zw!zlG0O$z`TQB5U1yy@El2SeC=14%Q=xR)7ka!rb?X=c|y>U(!yJ{9fmgja4(p4*z zkOIOH%PkV_8)s=fp+roU(wiPDYE@1v2RDQKwSAV3aDAi~mws%Fv=fvfm}Mcdiv$Cz z4Ng`)^1*bg0XAf0_(3+&^44vcuH7E)p!Pd+5oKK>i`IDc8+Ic%Lw+`<5Q>|vZ!E6} zucSj&Qr+6Ze4s{52-f&K!Q)^3MQX3d$h=r{n;mxhv8O)oxAoNez4@YA?HoIA{i-lwbz*>O*=KdteuZQn7WktTog4`g)oRK^sAjspRBRG z!quCExab+8kRCJ>may=*^l+#j#{C(VU-#RM zPAClLx^k=Rx=<(=Pj~0OrC?FtGnj?b&+_i}ty)kT|92otRg^q5@uL5Tjek1xNt9;C z`zJLAj1}#&NyJXL;T*v6=V+N#4V|S%Q1f-#9*-cKe07@B3GkT&ECV9XOJdD!)H3%p;!UwEoJcgHzHefF^r zf?*T|)%&;P9Cn#hi(%Q_XnB$+P5jviKx}zvqHPmphe!oW%UqP5Uq#az^bAceF>n(7 z3>K-U(;K0TMGZKxI{TuBK;#}-WjWtASIiBreW`TOw!RgRoiaGvFX5u>z93a7-jHJHepJ(X$lI8 zXY|t3dfYm3croRL)DG$I`;cJryvFPpT#J1;xGMxm-w6A>CFCV69!BP>^mJjdlIoO` zl*~@a(n$(QmgEh^O!!E`5+Lto7iGer^6#urqwz?7hWRL9%A`BkoCjm&-ghZfq4N*4 z?zoDa>zR7sFrLO6{3KV}n2GP(w-?l3HEjznu=+}_w=622Om=iwy0`$>qA(F}Y;CY5 z2H(+gREcU?XyIpOn1t2P^=hKs>bHq9t;FwgQ_t4}ggm3EA7BuQA!<2oN^%hL8>4O# zzc~L=l#`v!!uM)m@Cm+rhOor^Li}XWnQpng?c}I!GkRI+$ zwx{by;|-%e&<~L4x4ivE>(xt>#C$7g`XhXEw9o$Qn%7Rf^n>go;oIOOv3bgBrr46M zw3KHr0i-=&Pqn6Ws(cbT>JeJ5T`>(J){PJmqM#(ntnbe<{4nbq=Dk`#y2akET3d)u z6gGJoo0v#w?g#3RWXBil0D|Zl>Yn~$cU^CS-KBDb$o1x`rwMuTU455OEe)oC)IhZ6 z$O-joSue2AP)2AKE>34~HBNWtUxniFaqyqEA4Yjc6QxM};Gj|32{#vg0eCE@$|hJjE1SD1|E@_Tm*1{48*xL(oDd z0cD;@*~~vChl2#1tFuI%FQ-~n%6VMQv!5jbR}PnJ%^re~Q^Tq1j3I$e>G#NHdvxJs z!VZ(H{nV{qhk1!ab6)P|`;POAdL-SAD=)0Ku7mSmJ(wqy`wQc~sE`m5 z$p#bDC)8fDrQE|X3!MuZHLq|3iY-T_$CwU}tdk)IIP<_@q21DIKF34>yX=!$3+Ye0CBA}#IBwY>8B3LMXITfrhq683S?PYtPbudnt6 zx>>~06Eq32fzZb+LMtCI zXgv}RTQF;9BIF>4R8A#%T-#X?`Tu2qw$-N z`D+HhNiO)ZLlzPM9@Wu4WzR?=h`84RE__3fDLRuhxFw|arkmn^D7Aj;V9~_=NH^3O zCE%y;G}ar#8{2@Orx`AfUgLf6DX4NobKXK(KLC&3@5~Lv1dpS@p0&Q_tAn}_BkTlp z2BODH>{*>jE@OeKg&yOh9_h}Y{h5X+0JA!d(Gh!s!~+R7*i8m=Vc6a$tVUQ)PZ_jx zpUJw{w&%s zHOsBKoDwiRKjEWNdPxS@g8~tF~3U{A-6O(tMiz1&lmDZEUJZBCjMvTy_i|M?Qr z=!yGwVj;{BM-j34*>KfU*p;fkfq)AS{724?JgL1%K7vSQZ;be}>7T!fK$LMRU;i-O z`lb>AxSqd1Zp>)d5}LE9;XhACNX3bU=^qo9egh}LpTAj?llyiO@A-9;Bkobas-K-X zgNu<+A4`%iL#S3Ak*0*yW#&T2N4qm_##k|Vy67eSOxSU^M* zfy_!}S>N6A+E06ICeP?KjbqU#h>458wJ1>oo3M!$4T~Xt;z@_tE(vHYAFFduivb-W zO|E+$^$KJxWVwNKh(e;UWyGM_%#8CV!Nqii(^*4+nwcn9p-T)WdEcBE?Xh#%GdGsp zjODu{4{w>)pK%`2SG0p%e7aFH^oWJJw|6Pu9l7ro=|N)=yCW6iRJu>keRSUVxm%n# zm8o5s-7yNO$f2Hyn$gGamg-Kn=J+2Cm8*pLe22PdFGhnF9R#3pT=p*3V}1)0 z3#O2TleQ5z4+j#cQLdE}^C?CEy&0ifw0-ojAW?<)#@$AIp})vOys_Heu=cnvCPqr2 z3oS8SsnKQESke}-NYqx1GIHM~76YL!vtK@&hO1m87i2VliYcm6L9}Dw@5=74g<>A} zZjUMm@q*83HP>03?am~secRC6T_=T2Pjp9fHp6V?q$LG0A2q z;Ko+JuPy&>X2AgCZI)B)0j7o{BLKd9z5_NzNyCe(T+_VF$`d^W37D*3S&zRcib%lZ zAyVF#Md{4a6O@RQ(dfPdI6MUYY5kkC$OXD!_i@7*ti%_OhjXm#nVK0S0^B35L{v)OsSiP!=p%}KqityXb zpWm#E2x1Vielg>}N`bHrgpp@GTl2p^aAO1EmP*=^{Z}zyya>o7gH45M|NS8X^l8M( zZ^r&d+Ocea$OhRn7lVZBzbauv1^T@AFDm~}jnaUMETx5+|36Np9GT=1KC|VItj{Zb zaa#43Z%Wk4u&AVx)qel^7 zXrArOMUn8@wPl>N!Ikx#)Yqa(737Jkm230bt@L^U3Tol^{=!}R)gB{YjseRDG6UPM zkG>Ta7V5OQ(NI&fvR}@>HvdYwcps7`^bVv{?RVVWJ|y#CH6BR9rjbP@;@ALb%Q`j2 zuk`e?@KN80|5 zt8v!j;;5^u%WA4Rzz&p2MfE9uU9tyExf}oSkGc>aI^>TPOso;i3|&!z5R z4mdW+hHxWJ~5Z{FTyx%~=MKgMsOMj>RwHXu($@OJ6U2ihFX{O#KtA2euB? z#^KrPxbqT|C>zEp(eDTO*}83RoVLUuzkD$Hm}T>;d66FwfH262TR(JKPx2Cc%M1X| zr%IiYN#?X*E%p%YSPBm0%URE#KAw^H5ZjF0GFvQ&DnX}hh;p-%wy-kOf5IvOlSIVk z(T3z1!`JcW2fo_iOSQb?ko)&c_%jwz3#jb(|789-+>&=sZOZ(oT=>W`i#AvvQ$qex zQK(Ux4S&A)1RYBJYydEfAJhMu&%zmj{;x;_(S7d#PYnpqHD-1^Cg7>kFn z!m87RXvZ{t^m3WOYOi>dw=AI;9|5bE)pr1or`7erGffKWPLesVEMt)ds_*Zg9F@M( zV0{J?_IZ?8yVO6Y&=@OH^O%2~Wk#ss-R~>Sant>NC3BGJy-4XR_n3zRso6!n)Hnr! z3726O)Axed`_l899C{OFXE&PfRovv4@9YQSb>&ls52DoU!QsGxY^3w!DLeTk9?~#s z(%PKN@+0wZLd-xj4y8KiwWQuser81Np?bV5xBR`p4H<5)v`ApFY~;z~An4zH1`ikX z%8X&ekc3yMRH??JjLv88a;RRc*>J1=PV_`JN`QSL{PC)T?Yqx=M6;RqYo&MdU#%uj zQP>Z^kiQtSqRmh&z((N5=|%MwXU`(`7EM8zgN`P=#X5Lek=L^KjQY(o6;)7+OL3;9 zJp9-MkGYO9d*DDNcjZ{8x9l+PG`-(keGnR}hD)dO{R_gz7#uE=;=_00rXic~$F7Xk z=u*=olYxHP8EvHmT_|Jd6g$(mpicgR^MR+^(%@>gITB(s1k2_TLX`D&9^ntBGkQt7>=eXc zA!8xm?E5b{%iJ1xS-jG63NR%0v@*blR;8s4y)=}5R{hReq1KHawot=|dhwIT%vV6?V*reJF96^{0Lf}<#A&Ib z`{Vmyj(k{2b!(|*K@<1a25xMYbsoP|Wl_of(vHJ_uEC=_%&5)+igr9>V;fdPK_5_O zwO$HC#+UDsN=`|;<$biyvbMgC@40urx*|ZE->zq~G~47?=%BqooYHX)WFMjpOre~>}7RE_g6`UB;jYYLMhs&-%IL_&lHLUJj@r=9|@x{DrQ^Mtbedy z&s)3BYovcEyg*!6X478z{yQi+q5HA_JgiD3!7K3GkMhto%l+Il7o6L zJY?o68`Mn2JmZR;shq;oxs%?y%06aR&)Km8n?Yse(PDHm5hjd0KsZ*asg)u)S!r@B zKp|X_a)*JeBHrNLgBPlXhc5kVvqbwa z(O@{a-(SiGdOTfiWxsB9jx{X7kVqvSFS*zZnO+!APc*AHpP;-Z55}!pA0q$NqazR| zd|yA^+K?qJPfOZxGeKX-ik9eLbXM!Kg3Ury}4u~V3nPMNnm8?oxpWSIE*l@Wt`Gort1#k5Gf zP|f-0W_W)<#DsEFlFokN?|ZjvB)qwH7vn6tIgS{kjcv+|51eTZ`!^foF8olGU4QLhvCp1As_F}{u%*~*dhi>LmFXdtPi-2r?W}}J zkae^NtMCU)=YL;qk%TeRrb#2`C^M+%UmrMNr(#ap8c~VnjCYOEFA$g+e&6e2S5p{_ zXFu3K!&okvZ8{n&`SG!?RQQ^${(XS4mAP{<{_wNQ9osKuLf4y*2M91Yc7{u*1sR=g zr9N(qi(7Q)3MJ$)Bo@ADGGfZ;FHx`fwZT!)OMH@uG{#p1zm)FEXBOD5?8n^B3Jg z=H@w7*WYxitz^;mZH}p%DGv8r+)!?^k-vZu+~pkn@@?5z96%kL-lDvs19 z-M)-R*;;Efrr#)OabnsLH=MR9YS+`~Yt}zcY$As-%wYWt3V|_)De$`QKJ%JtYI9#N zL?dY6!nOVvElWW@&x7yRmtWvHu5tH{7(WZetj;A8{58Lk?1wjQjU22+lDSx^7@_4- z3dYvp(oDi0kP?^zdpSaR-Kj_)t0nC%0d_2?C8NYJ^e}AMxY#PQebq6Xd7@Y7ot-*| zF8=FtHq4*IbFWRwD?tQtq!L*dbR-|&SqV9nSKH5aPB&AMWJ%Y44Sjc-_i@eLUu@e% zECegLW^P(6tPGJ+Yl{W%=~uJeT^X~ctIboT(>M|;6e^YgbkHdl1~ooSMUf%(pGt;W zg7qido6U88$u!Z+;?E9oiwz~vrPB

2Fy$g<=zN2{gkk2vA2U389Faaw~Rdtd`%F z(C$$n;w$T|%mJHn2Cau_Rj?09SUjSW*nBrcnFwy65svMnNf<;;x9Je5{1sa5Hn&(p zVGAOpjc_5lX+(^hghNs1@K7i)L~f5&Fs9&=-ITKB)c`{`Zyn^gEnwA_6~1R`;TQCb~q40gRR`{d@zr|`$Z(GRyqD=WlvapsEQ+KxG!PD z9qRnJy)3M(e8)Lw@WN%fq}CSvEzUttV~1V2tURMKgD(SU{_we2?&~#SK}^r3O?a;v zHZ)zxZ%W@anY-wYXjr^`wZgUePS~i9MXZGy0CE6p5kNRR9JA2^iNFLliekr#t;s>g z2~Uy;sWpyv%T+F4iFy$t;X5g$LM^*5%Fy0PQuUi-;pALr-)mFmD{vUs&ptcHlP6oE ziJt9QX%xqEc=rB!8GmtI6{gOhnnStRCTVl^C$FS0=I~!I+c4O;TS~A;%J*o3QGnNV zf^3s$+iRs+%!EAH$)|BW&N(F-%zp%AC%t3z&~s#Ly`(W`OHf;zWaN?{z2Vh*6#6XW zInL{c!(x2 z&}{=v;97GfW5{;?gfc(MV3*@}GfUy38GUW{D`rr;T*)YrnHu?Or-i6Y1|NqKLZPt$ zZC4iOpv0#KqX3bU_I}E~EEw;UEocOH)|8>lutabj!DdX$Xff1L7gM_2^A+87=xXwW zCO*}>ZvKfQZRMyKXtPzbi>}tN#^Xelv+dG-ky?^YoKVFIKjG>-2bsC8Un(BX0>SPPWyl1j0zY=s?aq9*Z8!n$*d>-*8q?c%M>xtSn zRxun7ZDHKtblswl?H8=0FC%|*ksR`hhyP$oO$IZF%qF(mNGzNuD1hm)bJ15#ra)%@ zeZ)ef+&i2(1v-b*B)dp7^U39-$A~$&4Z83^iB_obK3RX`5wHA0mB<64(r9X{^O8s! z6?2=(nG|lFP~LG<^o(BbTK@qQgkMkS>Sxw1xHRl~ODEEAb=?s2dvw1Vxoze8kvyiY} z+;?M2VqISzWlkza>v#C&{F_>%n9aI!AHhb>%1FHCMQL)$dd}#yqX-r>g`>`Ccfr-7 zui$I1zenSoaQ%H6F||da7dj9<_*TAs@xol9)g<&$ z#rxg!%$tj+_rb)ENx1K!wKr_!jkg&25QSgqinx;M6?Q^Dfp6}RP}_`UfjzsQg1Lj( zdF~(by2<7k7q;N3V%cMgxHVNzr`Z)k(Wbv6LO_- zB$lIrb;}CEU~nD*Su%-Em2A(Nz(bkejYUA0=oQL3ag)qoUi8Ei2oF+w^rK*kgR+=Q zksMNg)}E%_OfeDdb5NapH58`3@^ z)hPi}j*9F^x>4n(yw$v^56E-#Ubr zG&k%n4(h(?i(R48vL9I^p=0azdUvP7cl{cvuu;6bh5dN^({8tor)6;P*aYj@UhQ7;P>XG0^n=fa&LOtBzpABzVzy=D#EX9ffqrbUlLx z7k>}f&hcm&?sE*EH8@U(Q|^{43z>7LSF1>W+D|c&Z14G{j17PUF2>cwA#+A6EeL)r z2c_qIt%`$OWTF=?R3D8P43(oq!k>ghOT6Yd=`O2VStXM;>Jsw<4+JIT=|7nl{Lr|w z>}zD8^{S6?DpgVoV*OGJ!qqxs`3K0LJZTp@?NdbUm$O_@Yf1SZ)%$J1$Zb`J^Vw@G zVgZ``nBUx~Hht;~JTb+ulLROCgPhB851O9*C4KkrouUKtUWB7^*aQnLz}eBhx?t(t zwY%;(`Ezt6^-qKBC`s2cJTsdT(n(}}QdiHWR(zYR$C$A^~yO0=fB+Tgb}bfqXL}Z6|z|ViBfy%(z3>%1}gP-(t_Z4M5W#MbMxDU zRi`r|>8l*(wF80g=#Rz+fSh7f*9mq%7N}w3#IpQ;fE3LKkUbGGLnM`fYZcr!tZz{8 z3xE<#vCF3KGA^rrL;ukcSfJ#DMTvLA_W z5h2Jv8jL~%EM=BxFgoU}6<%-5 zTf`RNd44hD=1w3JhQzaMGy3giD3vJbXYZZ%5$0KHb3HVSc;Mo2Blkt`3k$r}?OaGh zLl5ir-?0fl^04gcZHTo_9q()PtW8f;(_j;x`{y1HVZ6amw?VuM*RdSyc22(Zx`-#3 z-B)}f+YZ5k$;ajgT~=RTzYLosxX@8jq`+f)R=HfE9Ygr|rLfoa13ersnt3%L6IlIb zz)PS0`cr5D*OkLcByB(aH-q0+2pSGDh)G(tb&W-*oNNrMJ^>q*#;?*)`m2M!hP$ku z6I?<(^rb7BaIk;AB=F$4Sxse9@Hr?qabFjJeiT>pWx`B&Q8{8cF?zs`x{W-`H6vjb z`Rx4nYUJ#InaUk|;wI+QP zw^*&Lq;Wah&?ng`i{GOM3xCH%1rNs-~wTRh$^CH-+bW z`2`={czxp4s0UWnelJ&226oa7KT>P#*PJdtyH5ib5Ba8`>^m2?%0%3pLqHagQY8<; zq`*i=CdHHY(*_=F?5*4AAK0ckQ#xgu9oA3dx3ya4)XY7Gf*?Z@IT#qiK4!y3YC9B` zBnd?HQOjMBpEMrdHT-{$m|2(qguHV5m3QVrIJA&KJA&cMqtgqPe@I}tL+-v9>xdujD0Kw22 zpktI3N!vv>-^wjCq`H`Gcf9h^R-$H-_oDmu*sg1Qy!o~y3@C>AS!B2ZxqA9*n1o`M z(PhCZ=M$2VDTf5k%2?&!Sq=RpXi!IPdk>p|YD)6$>nr9bEKvmpNsYYjxI>@6 zCslDcs+#U*vH-Dd8eywbVQUo~sQCH1X=`q#MH)MOQwcn^-XSmkVabii38Yi3WFQP1E%SYLT4Zc@|f87DiAmGRToU0c_wx%}7=)CA6>=E?jgIZ7-z>6P z+&bh#&2N7}Sq%ai9=#H6T&nl9|J3c;k1>*}=^B_;MUR0ECAZf37vFCHZ?gapW~6$m z#uBwwg;C$n8ZAg3g;vE(-CkeEC6HJZ8O|>8>Oz#SGhi{yUb8+Ct95x! zmBu+SS`{GEW8P*GQ$=xRM?;|DMKo+OY&sRHxKPEBK>=7Bu4q{mB3&R-NzITR@Cz!9 z5txtHw~EdB=!=V%7~0kG9N(3x@YuO z9x`1s>z*b&1($?VWz8{YO%_AJ$ zJ>Ca0#yJ?~EB`AqF!+0ZkH%Zus7K!Jq3qh0_lkTZMcr=Qhm9zPm)3n;!gVSfm?zAD zU2xYD_`TESi3dOAQ$O$EliWn=aL*r3^4;VrTqK;aPCt@sufXDyxwRg8O%T>n&pK;g z>594TbU4qJ4%Kc4n*jiZfA_UWJl z7-7Z|{5^)Q^aOajei9#?&TPhR88`gg3jJVnAU4#u6M<^7y7h7zcbj4WHAHi7Sno!> zaU3=;TUKq+I#_(KDmPU?LiXDl_ROc`G=#nED)`=|WWmtK=UvkD>UV(8OF0$Uisz=m F{|6cN&tCul literal 0 HcmV?d00001 diff --git a/docs/add-a-benchmark.rst b/docs/add-a-benchmark.rst index 4badab154..79fb13981 100644 --- a/docs/add-a-benchmark.rst +++ b/docs/add-a-benchmark.rst @@ -7,7 +7,10 @@ Adding a Benchmark ================== -The following system-independent specification is required for each ${Benchmark1}: +Add a New Benchmark +------------------------ + +The following system-independent specification is required for each ${Benchmark1} in ``benchpark/repo/${Benchmark1}``: - ``package.py`` is a Spack specification that defines how to build and install ${Benchmark1}. - ``application.py`` is a Ramble specification that defines the ${Benchmark1} input and parameters. @@ -20,3 +23,9 @@ By default, Benchpark will use ${Benchmark1} specifications (``package.py`` and provided in the Spack and Ramble repos. It is possible to overwrite the benchmark specifications provided in the Spack and Ramble repos; see :doc:`FAQ-benchpark-repo` for details. + + +Validate a Benchmark (Needed?) +------------------------ + +Now that the benchmark has been created/updated/validated you need to configure at least one experiment, see :doc:`add-an-experiment`. diff --git a/docs/add-a-system-config.rst b/docs/add-a-system-config.rst index 5a6aacea9..06ba5a9ee 100644 --- a/docs/add-a-system-config.rst +++ b/docs/add-a-system-config.rst @@ -156,3 +156,11 @@ file If defining a specific system, one can be more specific with available software versions and packages, as demonstrated in :doc:`add-a-site-specific-system-config`. + + +Validate the System +------------------------ + +dryrun dynamic system static experiment + +Once the dryrun passes, the new system has been validated and you can continue your :doc:`3-benchpark-workflow`. \ No newline at end of file diff --git a/docs/add-an-experiment.rst b/docs/add-an-experiment.rst index 27dd7de74..9356a70f0 100644 --- a/docs/add-an-experiment.rst +++ b/docs/add-an-experiment.rst @@ -7,10 +7,12 @@ Adding an Experiment ==================== -Experiment Specifications are located in ``benchpark/experiments``. -They are organized by the *ProgrammingModel* used for on-node parallelization for the experiment, -e.g., ``benchpark/experiments/amg2023/cuda`` for an AMG2023 experiment using CUDA (on an NVIDIA GPU), -and ``benchpark/experiments/amg2023/openmp`` for an AMG2023 experiment using OpenMP (on a CPU). +Experiment Specifications are created with ``experiment.py`` files, each located in ``benchpark/var/exp_repo/experiments/${Benchmark1}``. + +Variants of the experiment can be added to utilize different *ProgrammingModels* used for on-node parallelization, +e.g., ``benchpark/var/exp_repo/experiments/amg2023/experiment.py`` has variant ``programming_model``, which can be +set to ``cuda`` for an AMG2023 experiment using CUDA (on an NVIDIA GPU), +or ``openmp`` for an AMG2023 experiment using OpenMP (on a CPU). These files, in conjunction with the system configuration files and package/application repositories, are used to generate a set of concrete Ramble experiments for the target system and programming model. @@ -18,3 +20,136 @@ are used to generate a set of concrete Ramble experiments for the target system - ``execution_template.tpl`` serves as a template for the final experiment script that will be concretized and executed. A detailed description of Ramble configuration files is available at `Ramble workspace_config `_. + +Benchpark Modifiers +------------------- +In Benchpark, a ``modifier`` follows the `Ramble Modifier +`_ +and is an abstract object that can be applied to a large set of reproducible +specifications. Modifiers are intended to encasulate reusable patterns that +perform a specific configuration of an experiment. This may include injecting +performance analysis or setting up system resources. + +Requesting resources with Allocation Modifier +--------------------------------------------- +Given: + + - an experiment that requests resources (nodes, cpus, gpus, etc.), and + - a specification of the resources available on the system (cores_per_node, gpus_per_node, etc.), + +the ``Allocation Modifier`` generates the appropriate scheduler request for these resources +(how many nodes are required to run a given experiment, etc.). + + +.. list-table:: Hardware resources as specified by the system, and requested for the experiment + :widths: 20 40 40 + :header-rows: 1 + + * - Resource + - Available on the System + - Requested for the Experiment + * - Total Nodes + - (opt) sys_nodes + - n_nodes + * - Total MPI Ranks + - + - n_ranks + * - CPU cores per node + - sys_cores_per_node + - (opt) n_cores_per_node + * - GPUs per node + - sys_gpus_per_node + - (opt) n_gpus_per_node + * - Memory per node + - sys_mem_per_node + - (opt) n_mem_per_node + + +The experiment is required to specify: + + - n_ranks it requires + - n_gpus (if using GPUs) + +If the experiment does not specify ``n_nodes``, the modifier will compute +the number of nodes to allocate to provide the ``n_ranks`` and/or ``n_gpus`` +required for the experiment. + +The system is required to specify: + + - sys_cores_per_node + - sys_gpus_per_node (if it has GPUs) + - sys_mem_per_node + +The modifier checks the resources requested by the experiment, +computes the values for the unspecified variables, and +checks that the request does not exceed the resources available on the system. + +To use the resource allocation modifier with your experiment, +add the following in your ramble.yaml:: + + ramble: + include: + - ... + - ./configs/modifier.yaml + config: + ... + modifiers: + - name: allocation + applications: + ... + software: + ... + environments: + - ... + - '{modifier_package_name}' + + +Profiling with Caliper Modifier +------------------------------- +We have implemented a Caliper modifier to enable profiling of Caliper-instrumented +benchmarks in Benchpark. More documentation on Caliper can be found `here +`_. + +To turn on profiling with Caliper, add ``--modifier=`` to the Benchpark +setup step:: + + ./benchpark setup --modifier= + +Valid values for ```` are found in the **Caliper Modifier** +column of the table below. Benchpark will link the experiment to Caliper, +and inject appropriate Caliper configuration at runtime. After the experiments +in the workspace have completed running, a ``.cali`` file +is created which contains the collected performance metrics. + +.. list-table:: Available caliper modifiers + :widths: 20 20 50 + :header-rows: 1 + + * - Caliper Modifier + - Where Applicable + - Metrics Collected + * - caliper + - Platform-independent + - | - Min/Max/Avg time/rank: Minimum/Maximum/Average time (in seconds) across all ranks + | - Total time: Aggregated time (in seconds) for all ranks + * - caliper-mpi + - Platform-independent + - | - Same as basic caliper modifier above + | - Profiles MPI functions + * - caliper-topdown + - x86 Intel CPUs + - | - Retiring + | - Bad speculation + | - Front end bound + | - Back end bound + * - caliper-cuda + - NVIDIA GPUs + - | - CUDA API functions (e.g., time.gpu) + + +Validate the Experiment +------------------------ + +To validate the new experiment run a dryrun with a static system. + +When the experiment passes the dryrun you are now ready to setup and run it, go to :doc:`4-benchpark-setup`. \ No newline at end of file diff --git a/docs/customizing-system-hardware.rst b/docs/customizing-system-hardware.rst deleted file mode 100644 index f7be542e9..000000000 --- a/docs/customizing-system-hardware.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -============================== -Customizing a System - Hardware Specifications -============================== - -To run any of the benchpark \ No newline at end of file diff --git a/docs/customizing-system-software.rst b/docs/customizing-system-software.rst deleted file mode 100644 index e8fd07e90..000000000 --- a/docs/customizing-system-software.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -============================== -Customizing a System - Software Specifications -============================== - -To run any of the benchpark \ No newline at end of file diff --git a/docs/identifying-similar-system.rst b/docs/identifying-similar-system.rst new file mode 100644 index 000000000..ca019bb24 --- /dev/null +++ b/docs/identifying-similar-system.rst @@ -0,0 +1,32 @@ +.. Copyright 2023 Lawrence Livermore National Security, LLC and other + Benchpark Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: Apache-2.0 + +============================== +Identifying a Similar System +============================== + +The easiest place to start when configuring a new system is to find the closest similar +one that has an existing configuration already. Existing system configurations are listed +in the table in :doc:`system-list`. + +If you are running on a system with an accelerator, find an existing system wih the same accelerator vendor, +and then secondarily, if you can, match the actual accererator. + +1. accelerator.vendor +2. accelerator.name + +Once you have found an existing system with a similar accelerator or if you do not have an accelerator, +match the following processor specs as closely as you can. + +1. processor.name +2. processor.ISA +3. processor.uArch + +If there is not an exact match that is okay, we provide steps for customizing the configuration to match your system in :doc:`add-a-system-config`. + + + + + diff --git a/docs/identifying-system-hardware.rst b/docs/identifying-system-hardware.rst deleted file mode 100644 index 75061b43c..000000000 --- a/docs/identifying-system-hardware.rst +++ /dev/null @@ -1,31 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -============================== -Identifying a System with Similar Hardware -============================== - -The easiest place to start when configuring a new system is to find the closest similar -one that has an existing configuration already. Existing system configurations are listed -in the table in :doc:`system-list`. - -From the perspective of setting up a benchpark system configuration, if the following -specs match the system would be considered similar: - -1. processor.name -2. processor.ISA -3. processor.uArch - -Optional additional matching can be determined by looking at: - -1. accelerator.name -2. interconnect.name - -If there is more than one that is similar from a hardware perspective, it can be further -narrowed down in the next step when looking at software similarities (:doc:`identifying-system-software`). - - - - diff --git a/docs/identifying-system-software.rst b/docs/identifying-system-software.rst deleted file mode 100644 index 327d56adf..000000000 --- a/docs/identifying-system-software.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -============================== -Identifying a System with Similar Software -============================== - -To run any of the benchpark \ No newline at end of file diff --git a/docs/index.rst b/docs/index.rst index 8d0e45d73..1c684164a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -12,23 +12,12 @@ 1-getting-started 2-benchpark-list - 3-opt-edit-experiment + 3-benchpark-workflow 4-benchpark-setup 5-build-experiment 6-run-experiment 7-analyze-experiment -.. toctree:: - :maxdepth: 1 - :caption: New System Workflow - - run-benchpark-on-new-system - identifying-system-hardware - identifying-system-software - customizing-system-hardware - customizing-system-software - validating-system - .. toctree:: :maxdepth: 1 :caption: FAQ @@ -49,6 +38,7 @@ :maxdepth: 1 :caption: Contributing + identifying-similar-system add-a-system-config add-a-site-specific-system-config add-a-benchmark diff --git a/docs/run-benchpark-on-new-system.rst b/docs/run-benchpark-on-new-system.rst deleted file mode 100644 index eb3d91fd8..000000000 --- a/docs/run-benchpark-on-new-system.rst +++ /dev/null @@ -1,23 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -============================== -Running benchpark experiments on your system -============================== - -To run any of the benchpark experiments on your system you will need to follow this workflow. - -.. image:: /_static/images/new-sys-workflow.png - - -1. Identify a similar system based on: - a. hardware specs (:doc:`identifying-system-hardware`) - b. software specs (:doc:`identifying-system-software`) -2. Customize your system for new: - a. hardware specs (:doc:`customizing-system-hardware`) - b. software specs (:doc:`customizing-system-software`) -3. Validate your system: - a. run existing benchpark experiments (:doc:`validating-system`) - From cd730aa32f9a95f531c8cc1821555a6d5f1dbbe6 Mon Sep 17 00:00:00 2001 From: dyokelson Date: Thu, 26 Sep 2024 18:20:08 -0700 Subject: [PATCH 04/10] fix spelling error --- docs/identifying-similar-system.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/identifying-similar-system.rst b/docs/identifying-similar-system.rst index ca019bb24..97c2d25bc 100644 --- a/docs/identifying-similar-system.rst +++ b/docs/identifying-similar-system.rst @@ -11,7 +11,7 @@ The easiest place to start when configuring a new system is to find the closest one that has an existing configuration already. Existing system configurations are listed in the table in :doc:`system-list`. -If you are running on a system with an accelerator, find an existing system wih the same accelerator vendor, +If you are running on a system with an accelerator, find an existing system with the same accelerator vendor, and then secondarily, if you can, match the actual accererator. 1. accelerator.vendor From 5141009288408a522c0e92204c12e39abf2ccd71 Mon Sep 17 00:00:00 2001 From: dyokelson Date: Thu, 26 Sep 2024 18:29:40 -0700 Subject: [PATCH 05/10] updated a few more files --- docs/add-a-system-config.rst | 17 +++++++++ docs/add-an-experiment.rst | 5 ++- docs/validating-system.rst | 10 ----- docs/workflow.rst | 73 ------------------------------------ 4 files changed, 21 insertions(+), 84 deletions(-) delete mode 100644 docs/validating-system.rst delete mode 100644 docs/workflow.rst diff --git a/docs/add-a-system-config.rst b/docs/add-a-system-config.rst index 06ba5a9ee..3879e8a5e 100644 --- a/docs/add-a-system-config.rst +++ b/docs/add-a-system-config.rst @@ -158,6 +158,23 @@ If defining a specific system, one can be more specific with available software and packages, as demonstrated in :doc:`add-a-site-specific-system-config`. + +TODO: New System Steps: +------------------------ + +We provide an example of editing the generic_x86 system configurations. + +The main driver for configuring a system is done by defining a subclass for that system in a ``var/sys_repo/{SYSTEM}/system.py`` file, which inherits from the System base class defined in ``/lib/benchpark/system.py``. + +As is, the x86_64 system subclass should work for most x86_64 systems, but potential common changes might be to edit the number of cores per cpu, compiler locations, or adding external packages. + +TODO: Examples of making these changes... + +Once the system subclass is written with proper configurations run: +``./benchpark system init --dest x86_64`` + +This will generate the required yaml configurations for your system and you can move on to experiments. + Validate the System ------------------------ diff --git a/docs/add-an-experiment.rst b/docs/add-an-experiment.rst index 9356a70f0..816af61ae 100644 --- a/docs/add-an-experiment.rst +++ b/docs/add-an-experiment.rst @@ -6,8 +6,11 @@ ==================== Adding an Experiment ==================== +. -Experiment Specifications are created with ``experiment.py`` files, each located in ``benchpark/var/exp_repo/experiments/${Benchmark1}``. + +Experiment Specifications are created with ``experiment.py`` files (that inherit from the Experiment base class in ``/lib/benchpark/experiment.py``), + each located in ``benchpark/var/exp_repo/experiments/${Benchmark1}``. Variants of the experiment can be added to utilize different *ProgrammingModels* used for on-node parallelization, e.g., ``benchpark/var/exp_repo/experiments/amg2023/experiment.py`` has variant ``programming_model``, which can be diff --git a/docs/validating-system.rst b/docs/validating-system.rst deleted file mode 100644 index 09ebded1d..000000000 --- a/docs/validating-system.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -============================== -Validate a New System -============================== - -To run any of the benchpark \ No newline at end of file diff --git a/docs/workflow.rst b/docs/workflow.rst deleted file mode 100644 index 3f1482751..000000000 --- a/docs/workflow.rst +++ /dev/null @@ -1,73 +0,0 @@ -.. Copyright 2023 Lawrence Livermore National Security, LLC and other - Benchpark Project Developers. See the top-level COPYRIGHT file for details. - - SPDX-License-Identifier: Apache-2.0 - -============================== -The Benchpark Workflow -============================== - -A benchpark workflow consists of the following steps: - -1. Initialize the system: build the correct system configuration files (i.e. for the hardware you want to run on) -2. Initialize the experiments: build the experiment configuration files -3. Setup the experiments: validate that the experiment and system configurations are compatible, create ramble input files and experiment root directory -4. Build the experiments: build the source code, and setup the experiment directory structure within the root directory (can this be combined with prev step?) -5. Run the experiments: run the code built in the previous step, output files are stored in proper directory structure -6. Analyze the experiments: ramble generates a summary of results - -Once you have cloned the repository and installed it (Getting Started page) you can either run an existing configuration as is, or create a new one for your system by following the steps below. - -1. ** Initialize the System ** - -To run benchpark on a system that does not have an existing configuration in var/sys_repo already, it is recommended to either start with one of the existing configurations that is similar, or start with a generic example (e.g., generic_x86). - -We provide an example of editing the generic_x86 system configurations. - -The main driver for configuring a system is done by defining a subclass for that system in a var/sys_repo//system.py file, which inherits from the System base class defined in /lib/benchpark/system.py. - -As is, the x86_64 system subclass should work for most x86_64 systems, but potential common changes might be to edit the number of cores per cpu, compiler locations, or adding external packages. - -TODO: Examples of making these changes... - -Once the system subclass is written with proper configurations run: -``./benchpark system init --dest x86_64`` - -This will generate the required yaml configurations for your system and you can move on to experiments. - -2. ** Initialize the Experiments ** - -Similar to systems, all experiments are configured by an Experiment subclass that inherits from the Experiment base class in /lib/benchpark/experiment.py. - -To initialize the saxpy experiments for a cpu only system you would run: - -To update it for a gpu system... - -3. ** Setup the experiments ** - -Once the experiments you want to run have been initialized you can run setup, this will validate that the system and experiment are compatible and create a root directory for your experiments. - -``benchpark setup `` - -4. ** Build the Experiments ** - -Now that the experiments are validated, ramble will build the source code for the experiments and further setup the directory structure for each experiment: - -``ramble -P -D . workspace setup`` - -5. ** Run the Experiments ** - -To run all of the experiments in a workspace: - -``ramble -P -D . on`` - -6. ** Analyze the Experiments ** - -Ramble will automatically analyze and create a summary of experiment results in a workspace. - -``ramble -P -D . workspace analyze`` - - - - - From e6eabc02006fa205ecf232e5762dd758f743b681 Mon Sep 17 00:00:00 2001 From: dyokelson Date: Thu, 26 Sep 2024 18:43:19 -0700 Subject: [PATCH 06/10] fix formatting --- docs/add-an-experiment.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/add-an-experiment.rst b/docs/add-an-experiment.rst index 816af61ae..0901ebe47 100644 --- a/docs/add-an-experiment.rst +++ b/docs/add-an-experiment.rst @@ -10,7 +10,7 @@ Adding an Experiment Experiment Specifications are created with ``experiment.py`` files (that inherit from the Experiment base class in ``/lib/benchpark/experiment.py``), - each located in ``benchpark/var/exp_repo/experiments/${Benchmark1}``. +each located in ``benchpark/var/exp_repo/experiments/${Benchmark1}``. Variants of the experiment can be added to utilize different *ProgrammingModels* used for on-node parallelization, e.g., ``benchpark/var/exp_repo/experiments/amg2023/experiment.py`` has variant ``programming_model``, which can be From 5c905ddffcfa1d09c60e7f34359412dbb6d8fcc7 Mon Sep 17 00:00:00 2001 From: dyokelson Date: Fri, 27 Sep 2024 17:07:40 -0700 Subject: [PATCH 07/10] few more docs updates --- docs/add-a-system-config.rst | 60 ++++-------------------------------- docs/add-an-experiment.rst | 10 ++++-- 2 files changed, 14 insertions(+), 56 deletions(-) diff --git a/docs/add-a-system-config.rst b/docs/add-a-system-config.rst index 3879e8a5e..b7ac2c877 100644 --- a/docs/add-a-system-config.rst +++ b/docs/add-a-system-config.rst @@ -40,53 +40,9 @@ example the Tioga system is defined in:: ├── tioga ├── system.py -Static System Configurations ----------------------------- +The following yaml files are examples of what is generated for a system after it is initialized: -``benchpark/configs`` contains a number of static, manually-generated system -definitions. As an alternative to implementing a new ``System`` class, you -can add a new directory with a name which identifies the system. - -The naming convention for the systems is as following:: - - SITE-[SYSTEMNAME-][INTEGRATOR]-MICROARCHITECTURE[-GPU][-NETWORK] - -where:: - - SITE = nosite | DATACENTERNAME - - SYSTEMNAME = the name of the specific system - - INTEGRATOR = COMPANY[_PRODUCTNAME][...] - - MICROARCHITECTURE = CPU Microarchitecture - - GPU = GPU Product Name - - NETWORK = Network Product Name - -Benchpark has definitions for the following (nosite) systems: - -- nosite-AWS_PCluster_Hpc7a-zen4-EFA - -- nosite-HPECray-zen3-MI250X-Slingshot (same hardware as Frontier, Lumi, Tioga) - -- nosite-x86_64 (x86 CPU only platform) - - - -Benchpark has definitions for the following site-specific systems: - -- LLNL-Magma-Penguin-icelake-OmniPath - -- LLNL-Sierra-IBM-power9-V100-Infiniband (Sierra, Lassen) - -- LLNL-Tioga-HPECray-zen3-MI250X-Slingshot - - -The following files are required for each nosite system ``benchpark/configs/${SYSTEM}``: - -1. ``system_definition.yaml`` describes the system hardware, including the integrator (and the name of the product node or cluster type), the processor, (optionally) the accelerator, and the network; the information included here is what you will typically see recorded about the system on Top500.org. We intend to make the system definitions in Benchpark searchable, and will add a schema to enforce consistency; until then, please copy the file and fill out all of the fields without changing the keys. Also listed is the specific system the config was developed and tested on, as well as the known systems with the same hardware so that the users of those systems can find this system specification. +1. ``system_id.yaml`` describes the system hardware, including the integrator (and the name of the product node or cluster type), the processor, (optionally) the accelerator, and the network; the information included here is what you will typically see recorded about the system on Top500.org. We intend to make the system definitions in Benchpark searchable, and will add a schema to enforce consistency; until then, please copy the file and fill out all of the fields without changing the keys. Also listed is the specific system the config was developed and tested on, as well as the known systems with the same hardware so that the users of those systems can find this system specification. .. code-block:: yaml @@ -154,15 +110,11 @@ file # batch_queue: "pbatch" # batch_bank: "guest" -If defining a specific system, one can be more specific with available software versions -and packages, as demonstrated in :doc:`add-a-site-specific-system-config`. - - -TODO: New System Steps: +Example: Creating a New System: ------------------------ -We provide an example of editing the generic_x86 system configurations. +We provide an example of creating a new system, based on the generic_x86 system configurations. The main driver for configuring a system is done by defining a subclass for that system in a ``var/sys_repo/{SYSTEM}/system.py`` file, which inherits from the System base class defined in ``/lib/benchpark/system.py``. @@ -173,9 +125,9 @@ TODO: Examples of making these changes... Once the system subclass is written with proper configurations run: ``./benchpark system init --dest x86_64`` -This will generate the required yaml configurations for your system and you can move on to experiments. +This will generate the required yaml configurations for your system and you now validate it works with a static experiment test. -Validate the System +Validating the System ------------------------ dryrun dynamic system static experiment diff --git a/docs/add-an-experiment.rst b/docs/add-an-experiment.rst index 0901ebe47..9b0796098 100644 --- a/docs/add-an-experiment.rst +++ b/docs/add-an-experiment.rst @@ -6,10 +6,12 @@ ==================== Adding an Experiment ==================== -. +Similar to systems, Benchpark also provides an API where you can represent experiments +as objects and customize their description with command line arguments. -Experiment Specifications are created with ``experiment.py`` files (that inherit from the Experiment base class in ``/lib/benchpark/experiment.py``), +Experiment Specifications are created with ``experiment.py`` files +(that inherit from the Experiment base class in ``/lib/benchpark/experiment.py``), each located in ``benchpark/var/exp_repo/experiments/${Benchmark1}``. Variants of the experiment can be added to utilize different *ProgrammingModels* used for on-node parallelization, @@ -19,6 +21,10 @@ or ``openmp`` for an AMG2023 experiment using OpenMP (on a CPU). These files, in conjunction with the system configuration files and package/application repositories, are used to generate a set of concrete Ramble experiments for the target system and programming model. +An experiment is initialized with TODO: ``benchpark experiment init...`` + +Initializing an experiment generates the following yaml files: + - ``ramble.yaml`` defines the `Ramble specs `_ for building, running, analyzing and archiving experiments. - ``execution_template.tpl`` serves as a template for the final experiment script that will be concretized and executed. From 50c2e3536f9e8a2072ba00f4bf26dcf9127a5d4c Mon Sep 17 00:00:00 2001 From: dyokelson Date: Fri, 27 Sep 2024 17:10:43 -0700 Subject: [PATCH 08/10] more docs updates --- docs/add-a-system-config.rst | 6 ++---- docs/add-an-experiment.rst | 2 +- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/add-a-system-config.rst b/docs/add-a-system-config.rst index b7ac2c877..80317cc3e 100644 --- a/docs/add-a-system-config.rst +++ b/docs/add-a-system-config.rst @@ -12,10 +12,8 @@ System specifications include details like - How many CPUs are there per node on the system - What pre-installed MPI/GPU libraries are available -A system description is a set of YAML files collected into a directory. -You can generate these files directly, but Benchpark also provides an API -where you can represent systems as objects and customize their description -with command line arguments. +A system description is a set of YAML files collected into a directory. Benchpark provides an API +where you can represent systems as objects and customize their description with command line arguments. Using System API to Generate a System Description ------------------------------------------------- diff --git a/docs/add-an-experiment.rst b/docs/add-an-experiment.rst index 9b0796098..1f8d435b2 100644 --- a/docs/add-an-experiment.rst +++ b/docs/add-an-experiment.rst @@ -10,7 +10,7 @@ Adding an Experiment Similar to systems, Benchpark also provides an API where you can represent experiments as objects and customize their description with command line arguments. -Experiment Specifications are created with ``experiment.py`` files +Experiment specifications are created with ``experiment.py`` files (that inherit from the Experiment base class in ``/lib/benchpark/experiment.py``), each located in ``benchpark/var/exp_repo/experiments/${Benchmark1}``. From a2d126f99562f34da91cbd6113fe13ca6b1cf040 Mon Sep 17 00:00:00 2001 From: dyokelson Date: Wed, 2 Oct 2024 17:36:06 -0700 Subject: [PATCH 09/10] Update docs for allocation modifier --- docs/add-an-experiment.rst | 39 +++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/docs/add-an-experiment.rst b/docs/add-an-experiment.rst index 1f8d435b2..37ee07a6b 100644 --- a/docs/add-an-experiment.rst +++ b/docs/add-an-experiment.rst @@ -35,7 +35,7 @@ Benchpark Modifiers In Benchpark, a ``modifier`` follows the `Ramble Modifier `_ and is an abstract object that can be applied to a large set of reproducible -specifications. Modifiers are intended to encasulate reusable patterns that +specifications. Modifiers are intended to encapsulate reusable patterns that perform a specific configuration of an experiment. This may include injecting performance analysis or setting up system resources. @@ -93,24 +93,25 @@ The modifier checks the resources requested by the experiment, computes the values for the unspecified variables, and checks that the request does not exceed the resources available on the system. -To use the resource allocation modifier with your experiment, -add the following in your ramble.yaml:: - - ramble: - include: - - ... - - ./configs/modifier.yaml - config: - ... - modifiers: - - name: allocation - applications: - ... - software: - ... - environments: - - ... - - '{modifier_package_name}' +The resource allocation modifier is used by default in your experiment. However, +it will only calculate values if you have not specified them yourself. + +If you do not specify values, it will assign the default values as listed below. + +.. list-table:: Default Values For the Allocation Modifier + :widths: 20 80 + :header-rows: 1 + + * - Variable + - Default Value + * - n_nodes + - (n_ranks / sys_cores_per_node) OR (n_gpus / sys_gpus_per_node) whichever is greater + * - n_ranks + - (n_nodes * n_ranks_per_node) OR (n_gpus) + * - n_gpus + - 0 + * - n_threads_per_proc + - 1 Profiling with Caliper Modifier From c88504607c894c6513b612bc952d92c9457215bd Mon Sep 17 00:00:00 2001 From: dyokelson Date: Wed, 2 Oct 2024 17:57:39 -0700 Subject: [PATCH 10/10] added examples for experiment init --- docs/add-an-experiment.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/add-an-experiment.rst b/docs/add-an-experiment.rst index 37ee07a6b..8b713f01a 100644 --- a/docs/add-an-experiment.rst +++ b/docs/add-an-experiment.rst @@ -21,7 +21,11 @@ or ``openmp`` for an AMG2023 experiment using OpenMP (on a CPU). These files, in conjunction with the system configuration files and package/application repositories, are used to generate a set of concrete Ramble experiments for the target system and programming model. -An experiment is initialized with TODO: ``benchpark experiment init...`` +An experiment is initialized with the following command, as well as any additional variants that have been defined in your experiment.py passed in as key-value pairs: +``./bin/benchpark experiment init --dest {path/to/dest} experiment={experiment_variant} programming_model={prog_model_variant}`` + +For example, to run the AMG2023 strong scaling experiment for problem 1, using CUDA the command would be: +``./bin/benchpark experiment init --dest amg2023 programming_model=cuda workload=problem1 experiment=strong`` Initializing an experiment generates the following yaml files: