From 6db2fee8fce59b12b06108cbcedf11cefcaf67a5 Mon Sep 17 00:00:00 2001 From: Amen Ayadi Date: Fri, 28 Feb 2020 14:45:22 +0100 Subject: [PATCH 1/3] seldon-core userguide. https://github.com/cetic/fadi/issues/91 @banzo --- doc/SELDON.md | 81 +++++++++++++++++++++++++++++++ doc/images/logos/seldon_logo.jpg | Bin 0 -> 36197 bytes 2 files changed, 81 insertions(+) create mode 100644 doc/SELDON.md create mode 100644 doc/images/logos/seldon_logo.jpg diff --git a/doc/SELDON.md b/doc/SELDON.md new file mode 100644 index 0000000..7ae5fc8 --- /dev/null +++ b/doc/SELDON.md @@ -0,0 +1,81 @@ +Seldon +========== + +

+ + ELK logo + +

+ +[Seldon Core](https://www.seldon.io/tech/products/core/https://www.seldon.io/tech/products/core/) is an open source platform for deploying machine learning models on a Kubernetes cluster. It extends Kubernetes with **its own custom resource SeldonDeployment** where you can define your runtime inference graph made up of models and other components that Seldon will manage. + + +##Install Seldon Core + +#### 1. Deploy Seldon core + +Set seldon-core-operator.enabled to `true` in the `values.yaml`. + + +## Deploy your model + +#### 1. Wrap your model ( Dockerize it ) + +To allow your component (model, router etc.) to be managed by seldon-core it needs to be built into a **Docker container** and to expose the appropriate [service microservice APIs over REST or gRPC](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/internal-api.html). + +To wrap your model follow these [instructions](https://docs.seldon.io/projects/seldon-core/en/latest/wrappers/README.html). + +#### 2. Create your inference graph + +Seldon Core extends Kubernetes with its own custom resource **SeldonDeployment** where you can define your runtime [inference graph](https://docs.seldon.io/projects/seldon-core/en/latest/graph/inference-graph.html) made up of models and other components that Seldon will manage. + +A **SeldonDeployment** is a JSON or YAML file that allows you to define your graph of component images and the resources each of those images will need to run (using a Kubernetes PodTemplateSpec). The following is minimal example for a single model, in YAML: + +``` +apiVersion: machinelearning.seldon.io/v1alpha2 +kind: SeldonDeployment +metadata: + name: seldon-model +spec: + name: test-deployment + predictors: + - componentSpecs: + - spec: + containers: + - name: classifier + image: seldonio/mock_classifier:1.0 + graph: + children: [] + endpoint: + type: REST + name: classifier + type: MODEL + name: example + replicas: 1 +``` +The key components are: + +* A list of Predictors, each with a specification for the number of replicas. + * Each defines a graph and its set of deployments. Multiple predictors is useful when you want to split traffic between a main graph and a canary or for other production rollout scenarios. +* For each predictor a list of componentSpecs. Each componentSpec is a Kubernetes PodTemplateSpec which Seldon will build into a Kubernetes Deployment. Place here the images from your graph and their requirements, e.g. Volumes, ImagePullSecrets, Resources Requests etc. +* A graph specification that describes how your components are joined together. + +To understand the inference graph definition in detail see [here](https://docs.seldon.io/projects/seldon-core/en/latest/reference/seldon-deployment.html) + +#### 3. Deploy your model + + Once you have created your inference graph as a JSON or YAML Seldon Deployment resource ( the previous step ) you can deploy to Kubernetes with kubectl: + +``` +kubectl apply -f my_deployment.yaml +``` +To delete ( or manage ) your **SeldonDeployment** you can use kubectl for the custom resource SeldonDeployment, for example to see if there are any models deployed: + +``` +kubectl get seldondeployment +``` +to delete the model `seldon-model`: + +``` +kubectl delete seldondeployment seldon-model +``` \ No newline at end of file diff --git a/doc/images/logos/seldon_logo.jpg b/doc/images/logos/seldon_logo.jpg new file mode 100644 index 0000000000000000000000000000000000000000..ed325e35ee8d38d6c98bf02282001df886622175 GIT binary patch literal 36197 zcmeFZc|4SD-#>n$8vA6=I%SDSij;k*5JD0{OeNWpWM9UJ>_rrnvJiM=W(2$LW)WM-Ed%AP^cFYFc_$26{R=dQN5*C@U`)jE|R#hv&H98R6sn zrv-R;L@u5>efHe>^XFm0vWha23TLFwOCFqrf`*2Mo{s()1H&=N6Fes*|Jz^WYJiEF zV*KzX6$L*)$wWcLL_w|tVBmEgrug{<{OyZ^lIqamBM@pDS~~C-ijD%56jW4{ho}x8 zK6D8DZ6x?};1JVc<`d^sjT6T_OoLoXD zg+)$@o|nFGQASqo@)cDz^{X11`Zo;>jcys6SlqXKU}bG%>*Va>>gMj@>HjPsFeo@A zH0sss=$JQe-^C`Uq^5mL&&d3gUr<<7TvGa_tfscEzM=8k_omLS?w;Ph{(-@X$*Jj? z*}3_JMcn$v=GOKOes}L+UK9Y;Z_@(*|7~Kw%!>(}7v-TtREHo3^P-@10|yn;p~EN6 z9bs0{gWPpw;g@_$&3Y*^Deo(dfYc2v+r1|pwCsZCCxmbZQ~Noye{EuZ|D&1xF|q%c z*C0SoMFHMCDkcC9?Cgmo`_le<{Cf@l9RvT4fq%!qzhmHEih&HP(!fC}!7g>?h)Gvk zxYQ-14hnI$Fo`f{>TqtuqPBg>N2HsoldY8IEQW;;VkRXSP%y+8W!bG`zJ$GVyD*<| zH(o5b?qyTdiTQ8U5iK`+UKtMGRb{PlDJaPL9;mTqhJ%(PB7m}Z4oy(9yy;kQUdY;0To-n?A2-BS*2 zYqDv=;x(ZnxmnEGHb@r>k(kceTZsLRFZz2%j!E9yWhItDw<1=UoHwQbIimXIJs;`n zj@xgav+-Oix=IETMsMDq%G)#uOtQ-y6lzWVepqUA1geq z{+7-ZVhE*=;(&c{4{Rp;55X?1e&SuP{N# z`h73!w|@1<7cWb(n6#phmXcuZf+vs#~O?y z*7-hGsMnEBe!=l2&0er5dLBtHyPwj%AG;~JTy^WqQmWzFrkYcN=m{J&_zdB4#{2jF z#nRsbkGj7=HFo!Evl`v{r<5Qum+ZeZZZE)Xe;wgzLK5LHPD~4-!!d3xo>#}T9 zWWeF>ZjRNXZ8G3zvwYvZh^=T6PH&EXGB1da8r9v>&zPu;NZQUe*%s*yus-#5MCw9p zM}SKe1&+>!t7h<8e_`33=QIw zFTdJo8GCZpLy1)L;okJ}^DM)FV{*~FQ4(h6Lj%f|rO_5Di=IS?=d`p0XWOZB{b~Y; z-TA50KK_YByZEZ^GtzYaid`{}YV#wMy6!_0R}2*#~RylUB8ob4;iUZ&8wG3Uo`mfRQNOGLttb?BO4?KYmYXosZKe8GUM5 zP3&7@X-uz6%&6LnzLZnBJ1RIH&)8sLe)&W>R|iKDTdgVMM7FC^n^@c3P}^vPE6QI^ zR3!OJMDhX0WKJwA_iO6adjHN)#=`8!N;|FdM!f@Ft5mjQa?Aq@UtrB#YD$7bW0DSC zFvPB{baIy?35-v7GN5AimYyDGaJn#bBvLjb858Dh01sYs{W!mT4_+3$Lk4Ur$-tY9 zIqT%A*#PS6e(8trhpP>_-Bo4FnHYnbxd&Zm$1>!dq3?{Wvm@#zrWT+7`S86?fuIz0divQDt7S)yw`xK=FG^-kj`9GWU)dz}l9sb)7Fa z>QH5?xuvJd3jBNgPuAds3}+%OS8fXXn|poCPt<9@%)I}MmaHg`-CI~a19J}d*yO7n zEK<<^#H)VuzW)AO8r2=ew%yHaosX5L70+B9S#4``YWY5CCwJXv8pScJDUw{Ko4~>9 zQsz}!AJ)7GF$ZC2?oyPJL|U%PdTZYD2@ij z&#_j-ZZRd=xHtA6Z`}9G9C6pz`f#EDnmBR3KcXw`Qhn_wMH*~_kXVBTCSjM)N%hfU z)v=1MS}mrHah_@DTc{&Z_3kUPqaPliS*q(M?Cz$0H@t99KT17RHR^)s;*)IX@h9W2?3oc=RN|y6tS&tReIw8yRyvYe zwEC!e5?YE$E6U_yTgQ`YjZ4n54~X+?bF*64qEtiZ$`2cG3WWoFAESfyq8=aikO)_J z;Bo;erGwg(mWeTdFGVDh0ju4facuNC16hF*qyBA6A+{%&7ZrB~hg&V~&p6|UqGTi^ zhxa|9jSG!>YIduS`>pS0esb;&;(7n7984u;SJ)QfkKxjT&c`XqpKx%rg-8e-N3M*7 zm}?NkS{7t-Pd{Lc7gZ>daNti7=5R(*SWRkkU!UyLFL705Ej1*BGuXMHV?V9L7i2Er z_j$Kkj-zO=ULu)^)n(O^fsIv^CkZ29fGAT6POdQTn^X@L@1_wdDQ)z)H%A6|>m|v+ zBrI4W}$N#-~)?kq}D0~X+z9mdvbFFrc6Zyon0qC%fNA^rPD)2m$TE`$51nJB(5 zI|Au*3+AdiZZZV_5eCX>OKYP7Kf5xOLd+x)BxY`_6+PrjQ?M&OF#??}% ze=s2{N&Y!5KWv$vc=7Ar#@pSu_dcEf`RkW=GXxnuGLT3r>JLR**s#5L&i?YbI{jUa z`6HtBJ|4k2E4EF@fe|*#yVl8}-~P?5{%72hN=+ua(sg4Wt{!hG^XgGISdFmoMr@9< zSQM^h#(w2EWUc%h;Rrm(MpL2d>Uhgig%Yz1`!-=yCRQZ5(rJC9e&Jir7{k1kL8+7u zbihIZgj=HtQ@5|&lBaiEd{0#Z};NoHgdhSji?76wRxRro(j#L z(=$IuFn%P{D@Yy6cJmH!7D(?<^$42RFuelS8|8+`$j}LbZerMDhZQ5HSNq+(tl%XY zImUZ-J6(EJhr5~-*&PTel}-~4|BxF_UrVLSsaj*YiF`LLoWrYq3sY#vy|D6LR_!>; zuq!;;5N#}=sHaGRkbd)6ZNC^vz` zt*O}cr7;S8V6$NVyLJ>SJq#X%=&byvXM9&mwEx!nS20gmaucmd!R&sRX_&D>`CFj? zGEmOX29M0hTA;Zb>fYjXCF;atZlXV)tELjG8?^QWp%Z>VaiYlr%i|Nhd}Hz$W>N*M zjr!Kwj^vzpZIo$F5H0Y2&wI&QcgIqBYO~&{FM+{j$@}dg4zZ!;S2l$AupgCN#>LF( zgC)6vk3PJ)9G;9Gwf6iN_sZq26R2=l{}^jayPk```K)JhHMqzxR3ztsYusVg(+R&A zqZ)qR%2mcdFG@F&O+1y#u7||uZk8=h28z%YLr#*DtaqDEtZl)Ymkbn6j8joHX4swc zn_^(gn}g(`umqx5r`=4!!%k_IB-Mo;Qt*75&R`z6V|S+>gR}I9*99uQ3UYm$NLelpPDd%-%}0k5SX zO9n0_a|6jTg^MdYW zmqQX3UJlMFnSKV3;R4F+^MB%kBH2roZdX6QYO3scXrD@&?xPM`Kbb`a{vTVWgZuS3 zVfP>|=zog~ztLH`(ks1P_2<=78YW(o_|p?9Lq+~JPCZ`?HtQFC8l_Q9=?hbR@HKaS z#|UOs((~}adztmAQniOfidLKdhDu;ii7DvHY1t{cZhA>ZLg6XLhV&aS5;4y(^VS`3 z>|G8rz^2UIZ3rctBr${0mvV&+e0_!>7{IXa@mi$77c6Ap2onl-lcG}f4tVz*CM5Q_ z&g&A6sgi+uunhj|XNvptnIgFH*FHd5ty9TB9|3$dWB2I80Bz$W> zlEYf>;zIwNQ}+_1N_zXhCQz!L&il%ubv}&8R`SOj0DWTjqK9-EFQ9t*)l%<96Wp(z z^Sjc!>&gUez;)B;Il`x-iq|*NB6abipxdka%A{0M^b3iVGXuoF0A)TjDq)5fKvJlB zZ(<6I8hlW6b?W;U>rl(lDK`6>Ho59U3bJn0YtzHo2`%D)R~p-_C!*1TYEh5|eJ1FK`Lt7BgkfF+yZnT55DR3lR?J!&FUDvEQw(|6? z|GXCdWr4EFG|6dwG0b7OvSaC|yoh3|H~CY%{uZ}~a-j{&rg|khH43ErWFWvTx&}W& zJ2nKj;Dkkzfu%k*ZhSBnA4mrDqRY`OGhmwCg8tBEU_N)I#5ySbE%*A|o>69BwDpDu zSAFWUDP|Gjc9w`G^G~vDbN5Js$Tm9ZEsvejMUc=!GY<&u!I07A8t^cBJSsA&msgdN zzCg%|Mqk4Ty%}=0X$S7B%nEm-w(`YV<&kz>HEpr%%RU};G6;(>CsQp%g5F^zSuW0~ zn=;OKnvm2V^-SYg?@z8qnBjmZ3uR08DLececz& z>{}+Eu}s3JJC1ZrK>_AVxJ9{8kqr5R9egOed<|;D6>@S;r#LdgXuaBDf8NRDwf^%N z4)^l2ML$G0CSpPyUp?Ak&*>LiMSRaknaoM}u-h`-9~e`S*^InqwHLY-DpnR_7n)wU zwv_okoueH`;{(IT2-F9^^i2Iu)UghmAOj&nG8ZV%Ep7;J%Ay^T%g8`U-h}~V8=-#r zo`-oHHh-c2LJE>qYvlOh5P}Xsv@7c%h^Nlmkb)j64Bu%X15yc%B@>~8$))Ew1NqAe zXVNW`MbAEWWd{}EOErIFqTe&p-yX9@Muz$aK^afsv1wh}bc`D9C&&|BP1E;p#lwDHR-)rwyZ zxm0h_?!v|&Zm8lcu3ZfoIPa8NwKOAzn%6Aas{bj#v2Bbsz(aT17hmBmYIhtSz2tpU zqIZEep~4pRjB(oOL%4rJlGu3G(mM%<)kh!ib+(FKo!+^p0srB9{DBQm!2#FM_fyn|bGIRm|XZ6-)i~tBRbSXZQO0lmOfZJ{fF<+=o*$x5Ye+uKOpP zQ9k0hta1p6eBo_szO^r{eQ`+PGWB=gMykmE3%8+N@Mr)@_ctm1h)=<%!Ff;B(M+W9 z8#_DB7`3&h{hqJdq56mu?o*Q|i>ADGxfb30br|-;MEZll5_F6m|v*dbfeCAIB&o+CA`>uzLi{+CjTkpPaR;jufc zroi@Ln0Rx^D?1tmg=aig{eZ@vMWsv6M%#+mL>jB z8(u{7ZPU#D4;*k&sH%CWV!-{5zmFo*!ftfo&X|RV<3pm`Ww8@P2A4hc`Y|3Gp(~!7 zJ0fyv<1ALVJErT>H66-)#6(gjBc}!KNMI_8wHNA72E0LRHjvfpEe*=Y4DO#PQWrm0 ze&xK->9Y!!Mhb<5jBn`BQ}Zj_St)Fmvh$Qp33*RN$ge|kdLcde8hj5Fy?@}7qWSS5xKFjU z;%D9@2Cr=yG*vvC<#w8P+qZ zL9Tnc8BTE4%*vpwLiU?6Y#Naq@!E$nFXK;_7eBZ-**RfV6o2*P<>)A3(dJEoSiewU z3r0_H#+g4`U?XH^n=sOvIg0RgGCGmA7kA(gZ$oU~^J3GlPwb_SfU=@|GTU1+@Y(a? zMB8U{_(FB`AMu2K>rH`c)^Cp$m#k5homfig-6omDp z=*wJsx~UtQd%pl1UgSh)AHz*Et;YLStqvuMzvoN4ssO=B7-Mx(INm6{;nH&$>o2gp zx~dO%scbag&E*RP_Pcbt#kb7l21!3ugeqIKPV2~L(TA3pJ7CYuYDEOZy1#ri-F)6I zL3^4Em~8&XWgT3i7+oDnq>~_!+?=`iYWHz^Q?RTDB0pnQmkiv-nYi^8584ORZw+TL z6jj&@=3i93E~)M9hJ!Xn(wSD)0eh@xnx1xltOLOntL@W9>N zwV?BRjj{4HC45NnHqm{P?d#t*hTXimKo3FM^US9GB`?_^gm{(Fwxz<^?c;qN7=|jT zLHCchHsfsW<@ZE1vt-s8LE5cWOG*(zOV zZ+LyMGKw<)-0b9;_YwP-eUVR~Ii5X#F*nZliA{Ow3q@B}bFNQ|BK@JH&$(!x9$u3c z#KIf)Var!DZ972%TlF?NxAU(}wmc>Sw`zXGj^rpBW~|4aGS)n2c<8Lm@$FRfZ9-m_ zbv@)mc+tWhUvgQ@JJtd2>Y?MCcl!wYeQ#a&3_pfsYov~>ezk^$97Ua56{A;`u;IO^ zUJ&(w4uTlNTc)fpx9OyP>MeOZT3t}RI6p$2rMT2$+c=*W6?G%5>|Tc|Yv=*;;8Igz zn=#2r=y%Zy^S*1aj#-UYpSSB#b@aKIKe*i&Do0vqTZHF0PgvRvec7l`58N(A*p`G5 z_?}OLH+QqGtu=w(oETvm=2)Y(Urg}yDMf1>lb%|vKPDM7+Ji?~^_H*xf znnsIZ9}hH~gW+aG@r1XLwCr`HG(xetTTj(7;nsiGByx9HflNX2yH>o#txgz~Bb1gi z=b9k2m@rD+{gmFSC%KqO*Uy7vb;u|4#d>E*>7lPeO5cG*9VG{zRCAjSJ z7Ci4syfyNUg3~ZFW{-utArOOB_r_6aC|I$BEcKv2zn9(8W;%By)#vjMr43~ ze>`C?1T_vi&y&_% z8bqFT?U1I|Mm;EOTNMo%$aq6K6hS%}@qg=Da*!P3JATgcuSf_6Lcnu@fr zi_!!pswnV`vyH7d^EA3kR$)!~FB*;|eYH8)&jTo`>0S~W`8}25X-NA>ZR9?J0zT%D znRo<@K|s&E5sR8AemeC6s#o4q-}m0(3BT@1Gyg~J9^-wfs6VeMuzMqP5B=9Q>G)+) zsb)0A7BRmpiuzxdH4z#ug8oDX{&I_8e_2*vrJKoBAi@9h9r|CpmK{H-9TzY#$iT8S z)y&)ED4QE1k#Qq_pG3aj^91B$_u9tip3Y(20(jS^1~L~JPsYhNBzgxU8#I)b)oll{ zW3?z+_ZxSW-lR61QP&CyiBZ`UkHZ9t<@~{Apu)r?%=CMZ*50MYY;vGu#1>J5+$4=c z#P-oJ&`1%&7Yr( z&A|taNSEaxM6YI$Z;K;25wfIC%sOW~d}$Oxq_L)|{rT%T5p=gyKHIGdF+cQTaw$%*@ zM-=P5tb(clchhSIuMDH63`w^X=+YJp4e!uDpk|}ldnsvm!&iwL1#yo*@lHWDD{;$?Q=QaoY{Xu| z9_8V7REv%bxLy%O2_U??IoR9HYVLjUlTA^)afKc_ZcR1rDYmTbGy(A|>bBpz;_7r! zBU?90op8uI9e1R{zaCYvUG!0G81FM~8?)5*a$H@8Z>(l4X*&_hm7M;is5Jje)Zik` z@mQ&%$!OaAgiBEV#6-$+7^Syzk2lSy)4h)@^CE)=?Rl#-)H6s+p0GB7(5Z|>XemKz zQu|_k>}k2;H}LoLA0*@XF05wDGl5DE57#jIc@-aid%rsVxK_rFmc=PLw}((>;S4fh z?E=Bh+byj~=oj`nH`}WK<9z`j{-(%M(KRHj^ScL;zP^F)oywXR z+z1TF1EbRg~K$~ya8!q7+U zN(L*J9!#$4i+P^$6xpT%yl7hwYGoo86@ad=oYX?{eY=|~pZ4;3a0<+$t!qn=ag(C0 z17TwgW2_LL7?D1`;v8lid|N}YMNBX%e&%gNyW@(vBhG(9?R6|$Tks~G%<`%@R+vPEs_lSK2{F+KuI)epvF);8|KOtJsKzR&i=%0;*3HTQq{#( z=|afolFyQSYp@phnUt+uF?5lTh1}Wr`OVKBABAbsWNii(Ba8}rPNUyk=%Xf8ufrLO z90bF~8lyu$TMC|?S6MJ+H$T~(?Te8Ap+b;+PO7wvaaZ3q-h68=@~jZe?U93`>f8%e zX*8KxCfL}VVP6z3fsevKKQUs#bbF_`&!%kDITF?XkmM8HD*7a%E+VmZ3X2ce&rIJX z*2lK=J**l_9lmtrHbe6kPiS}Q@uSGad&{d1Ca-O~qGccUMDR9prsgQRw)8LVFj-Ky zR{8`bkV+;*)AA3H+x|;;d0*=$)#?m-Ujcp zv1~F@9?#YmGX@rxzOMu^Iqa;zx~p#JUhx@bFGX02qFSi(6N)sn2=Om8=Fi3DpFPK4 zG<<-(x0|pgUH5FR&+(H%2Y%_Ay+C8^O4!K`-sLnYV#B>A)LN9-heS%Sp<~<5?t0aU zYZ1k;tv4}w^=m|bnWKzD^$&gCo{r?x|F9sXc z5Fo}#q}^`9!3Ro6s`geS42Tky*;ZcvmvREAno@vdtMDS>l{IGh1P94)YrP(+yMGz= z{X!V2h6yzjBr5iB%F;VzAtsjgbWG1H7HjW_eP==NzZqaP$Cyi_(`)e`erJky3Q zs`C?Rn~5|SKDNTB=!m#=gDKKq83uG9RJ@Jxa)I0tOz!uF1u`UD?lNO*%Nrn<7cp;= zcDH?Li|(RaJ|ey%t)yz5s6zN7p`x`(p9-%-24Rc@J4m>85EvIc8`4v{n`3&N%zuYNo%S=eA?7A zta<>SyD_|Qb%P<6uQeRO7s}|9%DA_Tn{IyR)ed%FoR7HwM9qM!D9UemL!5)G9vCvl#RXk~@3xw?{pMsuTD%GLTn}Yarac1i9tPFlKNS)Ysz_Wy?Fe1Nc7O}@ zA*0aJ!0qFQ^7cSYoOvHs3yIyeYKTAFLtw~4zS4{EW@ffuor0CWLsmjF6crn2_HX17 zR3=s~*6p3&GtYW-vg9?Ale^&5a&6zV$Ch4%din9bY!@bGVh{e*E%(7P@!>|bPwa5ho+?}OXcXEx>W>6XO zJxY6r9v&z((^f)7H5|HsLnewkN~Gh)`RkuPodtM-L>fEcy1eXB_MWN7oP14|cf9-p zi?S}(!PmuYFRrT{~D6#{Y-0SVGm zUcj^Gm*hZmQ(9bd>}K&fop-Cy$4FH~a;&|4irnj|8)sOIuC1>aT`BLpPn=wPmJnmW z14Acek7|2vMxM^FaJlj9j{i6QFL7F5i#JxI>RT95PrqCys=D2%7fx*pZ;^TA_{iPN z$O`jqf@GrERoU*s&9htF(gY7zQeWu8Wp{=5^l+ZCeYYX~c}t|qB+_O#M3vRxE$DvS z52&qM4wsYsz;rD|T1ZXj%=~CR?m?QZa&*iD#0CO?OZt)<*xYb|#lY+!@C)#gT$k(+@6DA76*JRX({j`+)`+ zf61NEW@0lJTlRQ5Lj1=ypHug;%(8nv^X}}}8Y6Vc z^&=47^Ey#8cKs5d_|o!wZ` zfCS94yd@Zt{9y(~6=6F8;^NE-3EWz($+xT0mao*y6dveTF(W@P!pD`No?G0e<#tOq zzZ8~+PSPB?aqENJ=WPLh#392UL6a{WYK@d!UwpbfkSA-Yz6&GHz{Py{eU)aDt30r6 zD;$pO-xPE|hP+8C%ilF!y#4`J9g*h65#;SKF~-)wt=@Ad{@VR&o$u;BjL@l0UpdWR ztL>k^|5qz7bfC#P8C(2WF#72Gj@q-ZYOz73J_j-YXF`qL^A%1`+&9Az2q2Y4hW;Np zo~$wXa#B{WE%~#w_#Y|dcP-2VzQO<^{XvO+YVZmYE4}j&`lR-;b?szOOMZsfqw8z_ z*`P?VV-$#)Ht2*eHNl90VU8tOgH_E0NEJySKR9grUmMPU{n%(l^E{vV9R~_g_Ki8p1<1xgA*WPxjTyshZ1z6CYYxxES({ z`=V$8{Jqd3q?J-!#_j&$kIi8=CZ~)#UXMq#7?=QJ%M#C&gp`EwNQ=|0Ws7Y@<|n#> ziQF1{nfhC;-d^St`?7Xw3@C#Rvns7D$H%_ljbvn00@npM2<#~xY^_pwSiGPT7v2kX zUblF0nNEgUA<%m|;&D<$H|^R1(79|Zxx}Ru95my{UiOl4dCqoj9xNeaHz_7wA8eMD z1ABzFEGBVOted~J^8K!>W^K^>aw3OvaViwM8!Rnb+J2780@C;{Bs=t>A4Ih#GE>)fy5fydY$3YFlTnE z!aZfB1*I45BuJ6B(`kuAXx5DH{O1e}kBO>ZS4D0iP7z$O)$J8eT)5r0hit!DjXxIR zxeDhA1O>+HkeCAuxr{033t>!{Y|c4iNc@q;j{mAC^qi%!$Ia(bRV+H7;$l{n|D>6e z;kDV$8KL8D*VWV~CE;BA?|AR4f_`)D@$hHYspdiMR)b3?1jnBMSdNcmy-~5^EIOu(AeK2YrizHp!Zn{kdj!t0C zncPZY?XsD-yzxRVZzhwqFFu{qYut~*PAr=Y3VjZjw-haGa_@TBEO3VrSobMQu;qhd08SP}L;qAES6t_wRPH z&rj`a1M6plDGPID;xcbN2XTm?LUnQWzdGiullzrLDGF z|Fq~!)H&iXAswq=B<{)AF`}rO)Fc({7nUwFDn6zYls$Jou+&R}tBsS+4TwZl1VPs4 zQJV7)D=vQ=9T%!rhkq4rLYYG+ZtJJWJc>Zq%=TZ?c^Q{rm2$y4^$=fROupjrzsW3r z2DSh2`KP{O!?G72ir*CDhD|lg?}r-DXDd=mKL`bO*Baq~;EOa2Hf%R!Cp=DyP)qvq zWogr)r7vh~VJFsXI_AxG{?P4)$GB?&#waR;r{PJ=gc(cyg@gya&i!Xz(uKMEp_g}UfP~z~m=zb`zWzSvr#=#|B=ZcMs)U1}P0)5_M{FjA3nPX@UP{<|& z$)78DU8F&aLaf~H+J?>c`c>T<4bp-n>GHqSBfeiKvJu#g`&SbEvj`)5{>qV!f8o%7 zKS3~={$I>Q3g1FZz`F+ zp3qS7qjGFJy7*>yg}gJQRZyxyH~%$!UlvUbs-6+gOoF!E>UkNjO<2B5yt`DX8POd& zvE=l%iyOr~&%&6QX<9PF-wqjeKB#UuuzS;PBg6Ylz} zqeHN{0Ql~jcAar(Vx%wWCdH3~NzIQPat`RJ67E+BE0aGqy7F{OuSj>iB^`^-wVQ(u z6b~TkxMn)`I|dt%DYfi^Z3l;uu)o};zo-!&5-W#Oa&~IFyD3QXeN`+l`MZ(vDb`-wqAhfE6u!ncMt6=jS51+JIFn`&yyq>;wq)HHt zCegM$z-uR4ecJuLdks$%p=v)!zW>|xOAJ=Gx6H?d{gIMAd?ziH;rI=o4QB(wt*7|j zdUy2|j!B1;&AQmuiUftZ6}RKCaZYMz_bMFAqzr?PnrWlF6la>>3CF9&s4E)3ola=! z@dW2oi5iCl!i&7`UMP}$Fs2nD0``jU((Rs^T6Q#7LEQ5e#I_OmNkE?8%^^2erM}11 z(^5};eL`80Dv6A%2aEDcu`2v$oX`zqzK(F>GfVK!3ZP@M77+tuB#k;yOk)CLSvlNS z(e|Hg--9ju`=D-u+~8D!YjyPIm43q4cgfY&n(?K$Us)rhdoXwn;aYgDU;<}-=Z z_V~as=Pz-$*`(}8RnUwBmS{&i**V#@Y`kN8XK`Qd{aPXj-mD)7u|$^07K%r#hbPsHX(Bo^wa}JsOG6*pe4_(8W~y&1fY|2s7+*wg2$s!i_50X*MIe zvmIxKr7}n~N_Y3x;k2{yEW5^=GUg(tMnSEpHw2b$inQEI%?%w@?CT@o=JA2Q^)XXl zGGmBOxAex$_leAP$p~N-`pz%taHq^a9*E&)A;^B*e@L1|B>N)gvww{LD|PKtUyNZ{ zK-Q4?&}FAbQ&84#yrzp#Hn60v>#}%LVtYTE+Y8Nftm?{J@ZA?A1Eo6Z&c)eJd1 z_N<1XS0T#VFP-q8ZPy1ysFu)l-Vd+|A)1$kI=UpdBltH14v~R;GT=-SgxPMMRRR28 zdNLs>jBF+(LaI(;9dBhjwDfACrkds?_}~Mp7%VfK3)nvsx`96AeFsYho?ml~!fd-{ z4AU78AH28qGm(4#F*2*;cf}8sYJ3ycO1k^Tim)ui?Y&sQ{y-#QTSC#0*cbQ|JJ4ts zvGRGo(eW$E11~8r`2Uzu*f4(zTGbO14tumq@SSxD5=eXRCs%=nPT*r~3GuK?!u~D? z8DIjv>YxD3`~j{T`-Z`T2qY?W&w<2QTQj!F){9!3Mi5kC*dKiCARWvF6=W1Y*h?Tn zHLEGM0NUPAI#j!nF&V%_qHwIJS{Zj!l3!UHkq|}(f)ko=Q(A;$StsaEp$I{Dr|EiicyI7F#?lw$4+e*w{0?c+DQOlz&X%RtiE@&li5e z@K2oPz@bq{*ajI%lsyOW891AE^Qv3u%maC2_hbjE|B?jp@Y2k0Ek*lTwTN}m1=!XM zl;kUBn13KF^XEo_*!wz2Q7Mc5;|Be+9fJME84XDRohO4(u03Y zmG~>GMI8OLQvZ?YeouJ6KSJZd7qN+6t&pbP|MupL&$9;jw!S%ydEUN?Z5R^Abh;z| z^y#UPpp?8bC^eIHPe9D7M(>@sRZ`s%twr`lI465v)@QugeHUoRDBm;sm2zXuTE7Gs z((RCe&4Zc@#%TD_#;G-?Or94yppdCng#3!&A{ZB3fUy?ggB$jrp4}Nu0@JlEFnPc- zb`S%(cgjkFPqaxKl88t-d(h}N|7q7~{K&^*7_(|t@nK@>OJ7vQx#Ps6B$q9NztbUD z!B;&PY)u4>=zDnk$qSDHYYYgQ<0!estX3s?+pu(kL`9H{*Ftg$Vt>hlcocNp!5m61_6)GOv+PFvxFpv8QD3~fjSnUO$v$3*2 z5C$klt3+^PA+Y**sqUgD8lg6t)lh65Milyu68RmSN|JVe>+UtN)mkBGU*6p|An3D-`;q<5)qp8-5x{()6Qi#b*;R>nK{#!B#JR35>Tx?$fZ?UYz zQ-Pz$P1>svGBkMkFMo#_wh`CG1~YzgkLF%`8K$f;wk$ zd7#2FS&35Ui+8VTTOMrukU}H33MO$z=qCJy%Y?9v%#IQw1Y5CbVTOETH4oaYABf;v zltZw|T@fu@TG*r$j*?5U!Rd-wH+syV$yOl*ry$H1}{UO;S6<^F= zL}QX|;&V?=l}-KZjV!j~nV{NmLr=NO&f;EkU0GZVkkAVHql7UuqPI*Q#yQw>X+*94 z;!aQtX6m+tV#Z!COOlQm)QB$5;_TijGgnXW-5`ptbiujzm@o5x3rf=e(Cz!`hgj1? zX|bFO4w8d65=Ql#O1h0alJS9K29y`;jE+BfhDG-EmM%`rFc@a+5lR$N!faV~tcFrH zAFQzI-JUD^ePjGrk2kx|4~d&}ZZ3;goFhbJ27+5KAm=GUU_wt&V(K&6q_Xc1aY1Xx-w_C>t0KV$N~@46 z4!l;q*K}1EGdC!qyvah0Xe8)|A0q=541l+ls+Vvm+-k&02deb+No*=rB;DjGUxJSz z;p;Z~BzR3DqAM#BE5b#kx|OWB*LU$;Kqjd2QTnMI;$bqMz?M(l+}2k~!hdp&X$`(A z4Ux6hy?bI6v*NsG17zLeC_MLc!%^PP*0v^3shRcD>!0+{T!o zBm|yDfUj||GXFQj^k0=_|KBx4f96^J>s2M4n@jKaGbv4nu5>od@! zUAjw*+jR1p@~YH+bFSZn2c{t?uTGNVtMIJK!Lv-%Cc9Heq+Ppho6qeZ`R%!P2K;mh zf!0^?w`^oVOy2ajnUZJFGh!Muu@~-{l7Zuwgpji4X1KB1bYr~tPRGZk7iaVypOrnA z5ZyR^8C4B-T5*BJU0OjgTWbPDHa2?|oDw#VU)htekVyRVh=!`k{BnQ2&n1<`FEtz- zxtdubCJi|u>(W?lQ(O{@tOUxB{B}t1P+!^F^&|*<+Sw9f-7*a^U&0;T-OM z5_InRH+n66Z0rA_{V$NAz2QHfjzW$>vWs zt3qE&ViNv-#vhmxzPh2tF8L}2cQ}`x*XC;0DmtWMLt7I+p(crJGie8x?SNJP zpBGCHUgg(u%}91~U%)AKw@I>{mkb15Ccy)ouS`j^;!a~VSe|E{x}oS~K=b7L)!F)X zgL$1-ixDSMa*(U@b#S~PpIH-kv)JaI#zz{uQ6z-#>NdvUAB-%wwKTi{Ayx6qRf|(+`t_h7%Ut~VHoDoN6xU?3CDzKKi}@bkLzgh67f7gg zDygD=WYRPTU)F?!jcBxsfuhi#D(qOv?VdMnYk<=ZL#4gF6Cy>_YY59j_qOw`BL`! zRLs@wT!iB4+>XZOt382kqV>AaDc#vAUm{W%)st4me$?qryO~O@t)K2%x=D;yUu(pf zH5clsI>9deR~7TmKWB@~b$3$7iN6_eiIb!IraQ91F{|rxw<+PVs$GXNOb$P^jVdiN zb|47L-)@j>sClAL4e$4Pu_>c8-4{G;+G;2N)sdzC{6mAONX`7n?xUVu`;%2jVseGN z6?EGw_^xXBss}^ijG96_hiK#Qo#%T@3(96}45Z0Q4 z)C}DNQk=tHmhOMw1@w1;{A*wze82u{CvR2yFU`I`n>eUe|Juvj^w&<`f685c=P;Ci zRV({CqBjy&#K^#jI4V-*3^;`tUj<1JsWSduhmZH)S%kkO=iig`!J}x=$VgE@_8X%& zOr~Ft4X-y%7^r6oKQ-r_T8nVJlD{x|lsKI(I`}+n9Zhu7@7|ABj4%O~l|%@7g;;I> zfm>y|xh0$M*`Mbb}f^)Kh2_|PASyg_Cnw_;d z8E+^d7)riC?EVP7!5Im5RL>FpG-_%PtJwp-x5aFv4d(Xf?x{pRTGR zGaVe*ataNxYI(KHhhA_bXB-2-PcDERApqVdCdNt-^(D8uSrrn)??!xTT=q)vkvIDI zn^bGCnv+yj5ZgI%=%p^VIzQj@?6d0Aw)G&OgM<;J9Vg&ywp^$8BL=3G^dons6jFSS zM>O((+1o=+r9U-Av>Vm2pL>Mqy1x4fBKftOB>KcBpfYQg77$Zc>sfPVFz++yN;_vZ zbd~P{?AxB7B=stTm7X%p)}5na5EZTf=Sz8T#pxsSV*iIiiG@|@buXd{TTfpti4|uE ze$L1PMP1F(0e+mIeUoNfPmtqYcyd(^_F|nqN8^PPIf6%RsWF)R$SUV8G2+Y)EL?=D z;8!-0@gP>-+T14f*5nHFy>11X8*>AwEn?QJZYZ4N zk%IZUjht-M&=teox7eL4ZVsxOh`K$1Hi5lLpyY*gq-A;@m-s7~FzG$!@(I2;-_5O8s4h9W8}7a@s2+K z!9f(!L6WD>T21oYG9Kca)|R)0P6Q#+tq`fHJ` zXdD;**?GJ7TjwpPD}!mnTfa;TN@w~VtfpxrV4pRU?nLR-h~0>bqUPAu&ebCB@06yd zYMtLkLyw^3@DX*fGi@Q>s>tjSGLTdL`g@7&U?rRSX8>B`D|wI{#FpCvNSvV0sb4MU z@;q|6{GCsiOJv(yyo}PgJp#|*J%CN9g=1e6ls^|oDQ?F!t{BPkJiJ5=SsYTIv|G0Q zBW?WuQ*95D(gqpeG~VS(zMZlAz#KJd4v!Igzk^o886*sD8gr_dv&>3N^li3!G!}Wj zwb{_$gv+&Is~R~^iKn4=gD&3~`q8^{PRD;%jUG;=x0T9PV5DHJU8AYx?731A;&@2^ zL{nKHq`$k`@%Ehmns&qMI~Fm%D`dboUVRG|;Otl7(HNs6y!cjbiRluX5lE95!A~{0 z;c{RiFkKBhZOME2)Uo5w{lr3X(7-FyeXgS3Rp_J3is~grZoC=CkE$$~=x2(^rv&l( zraVvou4mV=r5-+Y-mlnO<0Hexn@T2=oPje`iHeQ8TWl}d`*LVbscOJt?|ojc?TJ3M z8n9xf>$f&2|K;>sLjH>d5_V}fV&-;iXs9^4}wS& z5Ty%5REXFp0#RC2RGLVM^b!OCMT&rcbficpq98~|x^zPCO^_xDCJ;ghXX*aFeca{T zamM%K-aYQ{2LoO*GV*4w`OY=xGoSe^njHv2KxAQpkd?dNRn|6o+Kh*7Pa@j@Hhbxz zp4cb9MKPOZ*fk>nL;tY>qH(aE%$)P4F;zKxps5`wZf6j0S8g)jw zoJBqS(P*+cQo@Ftnm^6cU*1P#S`{(h$nf-E*X@SCsYziesV**BB{LeKCqT^RhW&g& zk_U~z)-B_!0Cw=t+@>P&X+K|J{47&gD*n-LkJQB@0}1iY0B$pW^9a(}ZvO$Mo)Eb# zF^7DX%z7K&kFndIkXL39x++>)cX^)6@ia38qCzCrd^7rOkGxh_hO*&5ik0m;wVqr0 zs_ESc1_GvRX^zE*|B#nPJQq{!9^ zQ5G@`wl%eo?L<4VNI(9=_^V)Np zN#zlMO$HTX_97+w!?Ld2pZpwdEYq^w+8x;Mqc)pbx*BMvSc0Djh~8wqUEyDmTS_wp z<%4e>b%e1FmvjG!WdGX)2Lyoq)^YA6b1lQ_V&>zo=(u8pX8DPk+xeO3uCrL`4n&$X zP+O$KFpeA^cG;$rgslQ1Qwa3u3r`-v8~yzAJGy_q_y@dw;>2jJy?;ualD?B@ZQ#N+ zL@gLN_!{LL<9rx3JbL6)E1f^5w_sH)35M519&S1omiHEIGxD17(ld2hrdII_r$d4h zgK7xGee6ea`oAV}`|=6lHv*438=UW5iG9Wkq04QNTMu0Cj{>cO)37aS^$d~$q~0!x zs0+M&hV@X7$KT^iNfDa7_c{Ox&!R&-j{w^ad#b1U`Q;9>w5AJTXgl2Q4pc6z3~?J&2@OhjfNgge{qp>xa+Iyr~EYo850SEe;wyA=)#6j~;$j zGt7|t7B*R2VorYFw{6o|ygIDBreRsMkL$>?NQD=02u~>p>?wvZ)knc;2Xc66T&(n) zQ{q*RJl-6cdMoN8vnVBZ&yU2ye)ki|L2bw$h^V;kcUkz)eBABg>NMP|x+K zz!WOftWqX&%B2<&J2{|o#XV=-Ot4=zv2gYG{igfH(bnK286e zQ6Si_mwVHAhNaFK7hOPpAIn6oukiFqduGKV%(5ktLpA+CF34JRw|$Mt)Xto!i5%|m zKN`}+3~QPwhIsO6xPz(n0GDyPUS>SQhwj(+)ls23I}itacO@Cx#-f(8D{pArWMRNz zY-m)=4#oB$^m;)T8x2oa+10M=d9{$|!O@5FZ&V`930P31~?6wclQZ za(lXDiD|E3-W{RxShBKg6GJBQAzrp_9yXaHa_uX<@?9e1(N?=e$&n3i7C}^1k_%i* zp!=NlMkVbrz1?cdZgwQbS|Z0c!&kYBMio@&Oth1m$i~-WR*aN(-IV)XP{V7c1*qx=vmn2a_bS1c(=H*N3GBC@FVI+chPNakvDvwe{Yyr*u1yU#ZfbEr%9r0jth9aW!<)! z5phyn;}_D(`Vmdv#od=AMdRw^P!pci(@s88#41f6(qB*yW#KV+!!Mn zE#Bg*eK|TDp!kgjax%=J1mN&dt4_G%dM$L>$FH^s9{BXhUO)6P_d*6Kaz<%+-;1Q9 zz0A9vlgFo?Y}i$xl_7+QP{18aD~IBqEL>v<;Z=Jj$m?4}DYU3!5gDZ!9PsOjjEwA$ z0L58Be2q6pa-*8m+a4k2Blo^Z-GQ!k2yZnl4SIwFr3f6`$B)X=fsD)+B2|ya#6AK zPRDgBZQR_~Dxtw661G;dgJ0`KVI2WI%2|_z98n9k#U!tJI}NufQtUC;o){=`0ubnO zPk3s*zwfrrCP0Mmqu?*a@*Ql!7b5H_h|R=-DXr^=l4QER;&w0AHx;G4hOsp*8S9Lm zvyV@WJ$j9Du_nl3-$f3q@~e{Wseq~Fcj905(k=yE!IR<7H5pZxKFRGhEUZo%s{S1R z{M4v3$d3DM0`U1UQ7-$YLxp*ui^|@~pUU36E86r(R;Yi8vqfs zS&2bWb({s(CsU}cWdt_r$T&h9Gi&d zW(cIFrx#A6onlB8!``yu?w3A>KXB(stuT_`?P3oAc!&+nIP}ogar(E^qgg;dXpYT+&_?^+hEM=f%kD&_s-B>3!OiyB=bGIwY z1)O}$x~bCMP67Jquj*SZVFZ4khZ`zgyE_?O&01^6x!-p$2N)hH(_*ZRKDUvujXnce zG2Q*I0b1m)iy6n9ZuxTh?(^C}3VE91R6G4ujGbI~k3Y;mYZy1Kqh@eV$+GhmgCda= zOgk$5xupJ^Yue%Jdt0%Q4P})%^UV?U7iQjn4nIo`?IYy79j?L9jgnAjOZ(+GgAsAj z6H*GCmQ!3pLWQ?cVP_-;ek)~LK&;k~#0@cW?M1VyO-KZ7-mV(OHwDb#?Hy=|~f?Kn#IWY3H>|VeBh5@2ixm0~QHya{VM2 zT6*MaTm!~UgCA5R@;79haX7lV)&SjVbEVL07klwdviQZp{3zvHJ!PS9N(C*DV;|8A zaiB6;;rEg;duC&0Yi`3=oKXSLB>8k*0R!DHkV%8TLf7Am>0B>!o9r7`EWGGk4q>HQ z1jAhagPzzks^w+Wv?})jtiBFI&C#2aJ&1uMd6N3n zDkqmz1Z;Wb$_H?gL9o&e#8z2hz;6{y5xtv&1yzXlT#|fL`PR(4&yqQaAs3W+PntfY z>_7NZ9<%Gm--BHgnS4_8((*%fz z7cvoC7QLT^0eZC{&d#T}lgD}&pVZ*9hXPkzMC*0Ev{B1SfLoyOs0wOH-Hm-LFE($c z2(_#T*W6@Dp)PK*G>OD5+`ljRuvKy8xNYkD_fKpuq@Kx18{AX;$JKQUY_y|+J@ZtN z+ovW;exI~gGe;>-EiY-llmd4rkVR=H4>BHQ^*ybJ=z+0QnL}<{6|XYaKlR>$sJ6%K zIi@;vnx4qVVpi$x1m?oueTWzbbKyzy!}yCCH?&_dgr7O^8p(vyH(D*k>S5yEc|ml44=5$8olXWkSTPl?oS%C@B7 zMV{W=wPN2NeH_-u-!Zc_?pJcHQycYrs`$gi@0^IzQ|-F!cjJ0=7$}hbC|RExUm6Bq z(7fHP;f!XbusKa)*tPH9x|P|nBK%@-)9j@8S9HaPS_+xfC@imKsPVL2sIfh( zr%Kg__Eg=&J{+k;>iB;A$WU4q>kH>*j)qOG4p(*J6aub&RAa#sZZ58!jeqU*rAKD8 z-s8@s2hABizCQyH?B4|jQW#%HD+*fOwGU5el~IHs5=enfj-#hWtj;&EoKu3H@=|43 zBIJ0Vz@J_$k+iU$y`ed%2wUQGA-j7!wK6+py(ov?h`Q6QYml2;^{gao!zFxZ_MVG* zM4O}5>mI#+i=Ly=wIBrOT$@h1&f}?9VYfYdn`Pm0dIQ}jndsorBD{CGm5+3s0klq( zCAt0^xql_IKK?O{-EniY;nPplKv+lV1255b#3@DV&#o7%*$=mBHxrGK&@b4T_EW}% zRPr5fo4B!}KEl?rSKK-mYRH6b(+RV0TeL%&+jIGwIBtatz5j~0NcEgwPle`yS>OG_ zS!74)m%U$+&9jE!!z7J(_db~{HJi=Wq%=V@5Xvf_Dqv5;YI>__d%aX8gzBAuR9H@)EN3qf~r6y5O3CzX|B1e?9C+_X(w+R+~GBOEOe937$5TU*= zKm$nE>`Y|MRCs_ohk2!&-8n*YiZy zK@r7|QhLr8=2dLm4~>tvXt%Y`ogmOYA}9Bh7QD{4Myy#YyfwX4YrclrK)lJRgmDus z>mrz-vG-n9=TG0&wS7xa@9JWIe&Xa6zIR5-yR$3%ZPONm#?8GBgHiGsgqY*&j?1cI zhPj+@=N?t@4puu0X5^>_viWvqlaa~IJrYe1=9grGg7r*yrv{0!|Ge=WfduobV@?lL zF_ENAKiEXxm3!RgxMD64ECWKdZyIh_3_FlcY6+moO#>N^hv+&b5{!x_ zRs*&XAlmU@%mDE$kjm9r3$(fcFbnv0q-xR_;MRGwfP-PS?Erw#0JzUf17P0m2%zbq zaA*oxVe|%7nyLV3ddVO^1KcZPz_!Dq@nytCQQBtoR00JG*i7V!?Hvdp-Gy`mcHR8s z4rFr&(!T>)1@fQ|D&7B|-CxV??IaZ*p1TE3ZtxAWDH*arL7E2X#$-Fay5wvB2WuEX z%TYw-4g321Ecr=r!y`KS7gvy|S&6Mva#w&-P2iTQPL9XK`uIPmC7;MY(Gp zTWTcnuahlIy?MRKM0*q8H0qWiEFbY_+nH5rK-^gVVnEoRZ8w0?J;Q(QuLcB~M}J&f zr=?yUhUY+l>5nT6$!*A_|^ztA)Qwu*?X!XZ*So z&h56w*P@hlwx%ZPx%7YfH~)t%&787!P1tU$7zvgBjkrf149NBaa1|5(jLDwrz36wU0LS=*2g`&{l(a!+lR z&7|os&%)ma6KwTVq9+dsFL9kzI{1&+B$~n^1QyW=XGfvhz(}kh`0!++>rRtSQN5ke z6h>{TV5K#PH%O8NnjYQRhym0-FkVux2c8xck~GyFh-Dq9rf9ggv(tR0UH8E0sI=rj z^FM*{NI7d3pK-d!zUv6oOi|3$Qvh9?t^_@Unfu-xB%Z4RwNvHCgs9#$pfh&vSXumO z9%Dw8_|w-VXQ;C>kCDv=ale20AH2uEDRF)nCIZh;XUF84P!Sgz!Z6G^J90MfBJOkc`&?L6Y~HjWVWIK@qFjCrAlfXIW}x6k8@s|7e^n{ zff+T#OqSF$hOz{zFP1MJe^Z;;rj&r(*vM5xi|FCAYeZO7D6KE+O2IH=NMgvD9+%13=^K z8Od{%CxalE@0q=IbbmD7xW3Tusd*48rR^qe;PJv4X658^E)c2$s8w}(T z6V10^7q(O+pBB7L34Mir0w@Kf0YkXlthYjig11og@z0XIg4+&}Nb zddAxU@@B$kcHQgA-ol*s6lst4t{X;LA1)lODNhf7pGD0auC!6dwMRcY`23l0hnm}U zRI;7Aej2K^?UcByEG93hdF@rY_2u}0WM@Rd)LvFfPHtnjk``-tpiUhfqH1P1cl3s_ zDHttw%|OD>z?Trsu3ghg6B7YRmvO3%#rxfksZ7g80!v3>~6=0HS#M){;=SS~4ZbUZyN-1a=$8`0{CU-G5lNE}WlW*hil z&#oQ&4im|-VR|05d?d?LY9zdJ%1r)1Fdr05nnjD6?V?FF?pI#4LH`l zwa=>NFIHfI<|IU1uuIybRTH8;KB-dMLb@Y&azJn+ZHOAYE>N?N!-YzR*KFn827y5b|Zet?$qw-Ub!vjZ_~#Scv! zBP4~(Ym!D}jk+rzE#8LVVa*7oR}0d)X8yGYL{2J{q-@A7I5ZXtrb5`6A<77T@q&aW z87J$u8s`Y*^&VT*<1Dmi84VTx|o&`#-WPZ^F=T-?V?;n_INWGLbZhB((D-DhHQE>7Pw7>G`^L=ia zb@9D{*FF^8rb(og@`&n$hKt}Zmju|1;pUUys_Y6b#ebS`*$f4hL(+xedOY-rKv&*I z;XeCK)~2M)-mN{6ru{2~(`>St4@2r5{92(nqTH-=;70XZuIlQJO}E_j%=tul9ZX~VD97w# z<#=Wr-YA92Lp{mK>*!=`zA!5J@Uu++*NayG0sT05%)M`2(bmz9Ts_Ci=fw%mCu*!@ zH&;x7(9?^#=Ia&Wu2+~|Hh}A)g$IoQoc!Wy)Hyxu%|1yZrkGLIUZ^hO%fc1;#O}n4 z4%vs9a$ma{W@eZyAk_4iUcek`Hm{?T`ZOyY7kl;&a`Qatzc7*|*RYs$AZfHHn)zNB zCc0@3s%^BI@l>zJ=+x9VHE6a**-NK({xQU6LNWWMPy`pVW13KCP95DfiLA{J8Vy5$ znQ-}eYLKVHrBjcea%-w`&}*gZgG1;73kH&PF9KES$G;#`|J`xh_3O1iLP~$K2OY;ORpqUG z)e$TFnu&nDR}-DL4^?h)TOtP?eLQar6tGC^9`a8P^zD22tRbcaaH&hUaZw3 zZ8J$*gWXVMBEopI-3rB5f86jq(0nMwTu zML#j@G&Dx9Q3GY!-?s7jfl|ZmC4NzCk?WwAXvD1^cBR{1+c54m8kJA+hY(ELT}>R& z{@jGMZ2MV#jGl{WM9HE_U$Jqb+2+>=#z%2k%7WxQnHnNSZDjF>Ggrk~4KFD7bPtzl z?{U3FjP-B$`1XdTn<{hQR@z7}_Trpt!W9)&1--Pl=f_!b*?q#sBYlG1ET1x|My%Vg zZG*)NK~H{YUC{>J*GZSja}YhR0|6~7LblBvuydZ~xpw$klJP@Rf3PeW#$oge?>xI=Z>=Mmw^62I^pX6<4%wp19T!Qws_1ETSah{pQ^_Os zt(aLB*KgaF+PkvmE=|L?WM;^5%a*z1hO6_?^+UTh*3keZI2B)lbBr^c`l zH2nT;ak1%p9Q?b?O6d-S*q59kN}p;VEIBlEZg=1-ilXq^z$$32WGVlAlfNE>|1c={ zDuRt2a6`CVO_%4rG&@v=KeRnb`>KHHqDD2@Yf4O*o_l2o$>tOY-Dj^RfMa#G$rphn z8I6T9Q0PGMqbo{;1677LeFN^V{X;vD*c!dw$dPti2@Mr(9*n_VfnKLr0zM+540OVL zA1~Cf+bd;ih8_O_F57cxY85o8eVS9Z@pZ z*2zq=wsl@sc@)vk(y;>>hmG;?Kt5GkkB>_jKlJ7NsOh&S#eTPNagluV!tM$i-U}yk z(ja@{6Nq3#!E*(uxt7JaYuv9cNXtqM+W&v5$9}!`gCFv@RnmWAn_W@ku@-1(jFT;I z?%+wkung1yjHT#liwxJK+LKKv2`AUz9mIWkJX*XMI@OXE0NZDKs(~;_7#44X?kAK! zId#cFq3|Zdms8|{TN)BVZueY2NGJz)U Date: Wed, 27 May 2020 11:30:18 +0200 Subject: [PATCH 2/3] Add Seldon link to doc --- doc/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/README.md b/doc/README.md index c5279c3..b39231c 100644 --- a/doc/README.md +++ b/doc/README.md @@ -6,5 +6,6 @@ FADI Documentation * [Reverse proxy](REVERSEPROXY.md) - Traefik reverse proxy configuration * [Security](SECURITY.md) - SSL setup * [TSimulus](TSIMULUS.md) - how to simulate sensors and generate realistic data with [TSimulus](https://github.com/cetic/TSimulus) +* [Machine learning models management](SELDON.md) - how to package and score machine learning models using [Seldon Core](https://www.seldon.io/tech/products/core/) -For tutorials and examples, see the [examples section](../examples/README.md) \ No newline at end of file +For tutorials and examples, see the [examples section](../examples/README.md) From 520512edeac9a54d5c492c81d1293808308b2de0 Mon Sep 17 00:00:00 2001 From: Sebastien Dupont Date: Wed, 27 May 2020 12:08:18 +0200 Subject: [PATCH 3/3] Doc review --- doc/SELDON.md | 77 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 47 insertions(+), 30 deletions(-) diff --git a/doc/SELDON.md b/doc/SELDON.md index 7ae5fc8..ac9d32d 100644 --- a/doc/SELDON.md +++ b/doc/SELDON.md @@ -1,37 +1,48 @@ -Seldon +Manage machine learning models with Seldon Core ========== -

- - ELK logo +* [Install Seldon Core service](#install-seldon-core-service) +* [Deploy your model](#deploy-your-model) + * [1. Package your model](#1-package-your-model) + * [2. Create your inference graph](#2-create-your-inference-graph) + * [3. Deploy the model to the Kubernetes cluster](#3-deploy-the-model-to-the-kubernetes-cluster) + +

+ + Seldon Core Logo

-[Seldon Core](https://www.seldon.io/tech/products/core/https://www.seldon.io/tech/products/core/) is an open source platform for deploying machine learning models on a Kubernetes cluster. It extends Kubernetes with **its own custom resource SeldonDeployment** where you can define your runtime inference graph made up of models and other components that Seldon will manage. - +[Seldon Core](https://www.seldon.io/tech/products/core/) is an open source platform for deploying machine learning models on a Kubernetes cluster. It extends Kubernetes with **its own custom resource `SeldonDeployment`** where you can define your runtime inference graph made up of models and other components that Seldon will manage. -##Install Seldon Core +## Install Seldon Core service -#### 1. Deploy Seldon core - -Set seldon-core-operator.enabled to `true` in the `values.yaml`. +To deploy the Seldon Core service inside your FADI installation, set `seldon-core-operator.enabled` option to `true` in your FADI `values.yaml` configuration file and reapply the chart: +```yaml +seldon-core-operator: + enabled: true + usageMetrics: + enabled: false +``` ## Deploy your model -#### 1. Wrap your model ( Dockerize it ) +### 1. Package your model -To allow your component (model, router etc.) to be managed by seldon-core it needs to be built into a **Docker container** and to expose the appropriate [service microservice APIs over REST or gRPC](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/internal-api.html). +To allow your component (model, router etc.) to be managed by Seldon Core it needs to be built into a **Docker container** and to expose the appropriate [service microservice APIs over REST or gRPC](https://docs.seldon.io/projects/seldon-core/en/latest/reference/apis/internal-api.html). -To wrap your model follow these [instructions](https://docs.seldon.io/projects/seldon-core/en/latest/wrappers/README.html). +To wrap your model follow the [official Seldon instructions](https://docs.seldon.io/projects/seldon-core/en/v1.1.0/python/index.html). -#### 2. Create your inference graph +NB: currently only Python is ready for production use, but other languages ([Java, R, Go, ...](https://docs.seldon.io/projects/seldon-core/en/latest/wrappers/language_wrappers.html)) are compatible. -Seldon Core extends Kubernetes with its own custom resource **SeldonDeployment** where you can define your runtime [inference graph](https://docs.seldon.io/projects/seldon-core/en/latest/graph/inference-graph.html) made up of models and other components that Seldon will manage. +### 2. Create your inference graph -A **SeldonDeployment** is a JSON or YAML file that allows you to define your graph of component images and the resources each of those images will need to run (using a Kubernetes PodTemplateSpec). The following is minimal example for a single model, in YAML: +Seldon Core extends Kubernetes with its own custom resource `SeldonDeployment` where you can define your runtime [inference graph](https://docs.seldon.io/projects/seldon-core/en/latest/graph/inference-graph.html) made up of models and other components that Seldon will manage. -``` +A `SeldonDeployment` is a JSON or YAML file that allows you to define your graph of component images and the resources each of those images will need to run (using a Kubernetes PodTemplateSpec). Below is a minimal example for a single model, in YAML: + +```yaml apiVersion: machinelearning.seldon.io/v1alpha2 kind: SeldonDeployment metadata: @@ -53,29 +64,35 @@ spec: name: example replicas: 1 ``` + +[ref](https://docs.seldon.io/projects/seldon-core/en/v1.1.0/graph/inference-graph.html) + The key components are: -* A list of Predictors, each with a specification for the number of replicas. - * Each defines a graph and its set of deployments. Multiple predictors is useful when you want to split traffic between a main graph and a canary or for other production rollout scenarios. -* For each predictor a list of componentSpecs. Each componentSpec is a Kubernetes PodTemplateSpec which Seldon will build into a Kubernetes Deployment. Place here the images from your graph and their requirements, e.g. Volumes, ImagePullSecrets, Resources Requests etc. -* A graph specification that describes how your components are joined together. +* A list of **`predictors`**, each with a specification for the number of replicas. + * Each predictor defines a graph and its set of deployments. Having multiple predictors is useful when you want to split traffic between a main graph and a [canary](https://martinfowler.com/bliki/CanaryRelease.html), or for other production rollout scenarios. +* For each predictor, a **list of `componentSpecs`**. Each `componentSpec` is a Kubernetes `PodTemplateSpec` that Seldon will build into a Kubernetes Deployment. Place here the images from your graph and their requirements, e.g. `Volumes`, `ImagePullSecrets`, Resources Requests, etc. +* A **`graph`** specification that describes how the components are joined together. -To understand the inference graph definition in detail see [here](https://docs.seldon.io/projects/seldon-core/en/latest/reference/seldon-deployment.html) +To understand the inference graph definition in detail see the [Seldon Deployment Reference Types + reference](https://docs.seldon.io/projects/seldon-core/en/latest/reference/seldon-deployment.html) -#### 3. Deploy your model +### 3. Deploy the model to the Kubernetes cluster - Once you have created your inference graph as a JSON or YAML Seldon Deployment resource ( the previous step ) you can deploy to Kubernetes with kubectl: +Once the inference graph is created as a JSON or YAML Seldon Deployment resource, you can deploy it to the Kubernetes cluster: -``` +```bash kubectl apply -f my_deployment.yaml ``` -To delete ( or manage ) your **SeldonDeployment** you can use kubectl for the custom resource SeldonDeployment, for example to see if there are any models deployed: -``` +To delete ( or manage ) your `SeldonDeployment` you can use kubectl for the custom resource `SeldonDeployment`, for example to see if there are any models deployed: + +```bash kubectl get seldondeployment ``` -to delete the model `seldon-model`: -``` +To delete the model `seldon-model`: + +```bash kubectl delete seldondeployment seldon-model -``` \ No newline at end of file +```