diff --git a/sig-support/meetings/027-2024-02-21.md b/sig-support/meetings/027-2024-02-21.md index 34fd592d..9c9b5f4c 100644 --- a/sig-support/meetings/027-2024-02-21.md +++ b/sig-support/meetings/027-2024-02-21.md @@ -16,7 +16,15 @@ - [Transcript](#transcript) ## Participants -- + +- Andrey Arapov +- B S +- Damir Simpovic +- Maxime Beauchamp +- Rodrigo Rochin +- Scott Carruthers +- Tyler Wright + # Meeting Notes @@ -24,9 +32,8 @@ - Scott assigned two issues to himself related to protobuf. https://github.com/akash-network/support/issues/184 - The group talked about https://github.com/orgs/akash-network/discussions/486 - Artur is working on https://github.com/akash-network/support/issues/183 and tracking this issue. It is marked "p1". -- Scott looked -- Core team looked at this issue. There is a lot of complexity with provider attributes updates. There is a lot of data. -- +- Core team looked at the issue with provider attributes fees. There is a lot of complexity with provider attributes updates. There is a lot of data, and thus the fee does make sense. + ### Support @@ -37,10 +44,228 @@ - Slow roll out will allow for sanity testing of code base and documentation. +### Action Items + +- The group will look at the new docs and make sure things look clean. +- Feature discovery service will be rolled out to more providers on the Akash Network. +# **Transcript** +_This editable transcript was computer generated and might contain errors. People can also change the text after it was created._ -### Action Items +Tyler Wright: The provider bid price I believe. +Tyler Wright: And… + +Tyler Wright: I believe you're going to do some initial provided attribute rather change pricing. I believe that there was going to be some internal quick testing to determine if there should be a discussion or an issue. I just want to see if that has happened. Otherwise we can just continue to track it inside of the ticket and go from there. + +Tyler Wright: All right, welcome to six biweekly meeting. This is a six support on February 21st 2024. How much appreciated everybody for joining during these six support biweekly meetings the core team goes through any issues that are waiting triage. If anybody here that's added any of those issues want to talk about the issues and more detail. They're more than welcome to participants can also talk about any issues past or present that are either open or closed just discuss status or Solutions Etc. + +Tyler Wright: From there. I usually hand it over to someone like Roger or somebody else from the Insiders or vanguards. It's very active and Discord. I can kind of give an update on support going on and any new issues that might be coming up from users and using So again, we'll talk what touch on that but the agenda usually starts and we'll start again today with handing it over to Scott that will go through any issues and support repo that nutria thing. I also see Max on the call here again, as I do every couple of weeks got out to Max and the cosmos team anybody has any issues related to Cloud most feel free to drop those in the cloud. + +Tyler Wright: Repo as well as in the ecosystem Cloud most Channel. I know that Max is a great job. And the rest of the team do a great job of responding to any issues that the community members or core team or anybody else sees with the cloud most product and they Continue to update it and improve it. + +Tyler Wright: Same thing on the index of side. But again anybody has any issues Cloud most related. We can save those to the end and discuss any of those issues here, but we start off by going through any issues in the support repo and they caution Network organization and that's related to the core code base. So again, if anybody has any issues that are core related as you all know, feel free to drop those in the support repo and then don't label them at all. What happens is again, you can just add a waiting triage label and during these biweekly meetings we go over look at the issues and put an importance on the So without further Ado, I'll hand it over to Scott to lead us through triaging any issues are waiting Triads over the last couple of weeks. + +Scott Carruthers: All right. Thanks. Tyler else share out my screen. + +Scott Carruthers: Is everyone able to see my screen? + +Scott Carruthers: So I was just reviewing things that are awaiting triage prior to the call and… + +Tyler Wright: Yes. + +Scott Carruthers: I think we maybe have just three or four. So I think we should be able to fly through these pretty quickly. So going from the top type of typescript protogeneration that Max opened based on. recent discussions and this being owned by the core team primarily and certainly will be Consulting. I know that Maxi did some work on this prior, so certainly be In communication with him and coordination with him. I'm just going to assign this. To myself since I'm going to be primarily taking this on. So this was not only for typescript but also JavaScript and possibly rust in the future as well. So I'll be in the protobuf generation. So again awesome myself to that this issues and this initiative is for generating. + +Scott Carruthers: Protocade definitions for other languages other than go has been discussed. pretty Ad nauseam over the last couple weeks. So I don't think they're prior any questions on this. Let me know if there's any questions but this is something that I'll be deep diving into and focusing on over the next couple weeks. + +Tyler Wright: Scott do you know what you want to label it? it's the P2 for you, or is it more of a key one? + +Scott Carruthers: Sure, I can apply a label as well. + +Scott Carruthers: Yeah, I would definitely classify it as a P2. + +Tyler Wright: excellent + +Scott Carruthers: Okay, next up update core packages to go version 1.12. So already has some applicable labeling but some of the arter just open to track and that some of the he'll be updating. They core code base with himself. So I will assign Archer. + +00:05:00 + +Scott Carruthers: This issue. + +Scott Carruthers: Again, this will be going through both the node and the provider code base and updating. Go version 2 1.22 so pretty simplistic. I just again something that we're just using for tracking internally in the core team. + +Scott Carruthers: so feedback for small providers that are again their wallets drained, so Andy I was actually looking at this prior to the call, I don't know if you've had a chance to review this but is actually wondering if at least just initially and obviously we can do additional assignment if necessary, but as I read through this this is very much. into something that possibly a new provider needs some guidance on typical tactics and techniques for so that if you read through the text of this or talk about their wallet getting drained because of withdrawal + +Scott Carruthers: transactions so obviously we can adjust that 12 or 24 hours is if necessary in also talking about their wallets being drained it bidding on very small workloads. So obviously we have some techniques so in the pricing script that other providers are using to ensure that they're not bidding on. Workloads would point one CPU if necessary. So yeah, are you okay with me assigning this to you? I'll send it as a P2 and I think this is just something that you could pretty easily respond to in this conversation with some initial suggestions. + +Andrey Arapov: Yeah, I think… + +Tyler Wright: go ahead and + +Andrey Arapov: what you said exactly what's happening. But I was checking, the transactions for visual. They are really cheap. They are a hundred times less than I send so and… + +Scott Carruthers: Yeah. + +Andrey Arapov: they don't use it happen that often the usually happened once twice a day something like this. So even if he did like it more often, I think it's more maybe even related to that updated attributes that this consuming three akt because if he puts there, something like Small amount also maybe he doesn't know about the deposits 5kt when there are bits happening. Maybe he just check the balance, but just It doesn't I think require that much attention to be honest, but we could just drop simple message that hey this is that and you need to make sure that you have enough make it is in Bottle depending on the amount of CPUs. + +Scott Carruthers: Yeah. + +Andrey Arapov: And that's it. + +Scott Carruthers: Yeah, I clearly agree probably in five minutes. You could probably at least I'm just responding and… + +Andrey Arapov: Yeah. Yeah. + +Scott Carruthers: discussion to see if he has any. And yeah, that's an interesting point as well. I wonder if these cognizant of Return of the five dollars is put in or a five akt that's put into escrow and Elise closes. So yeah, it may just be a new provider that just needs our guidance on a number of design issues. + +Scott Carruthers: hm support. this actually opened last month, so I'm not sure why this wasn't assigned prior. So this is a topic that's been getting a lot of closely recently as we're looking to bring more. we went through the exercise of doing training Through Thumper and now we're have some other candidates that are interested in doing AI model training on the Akash Network and shared memory. As something that's critical to those AI workloads in the training. So again something that we've been discussing both internally and other six calls significantly, so I know that + +Scott Carruthers: We have some people possibly that will be assigned and focused on this so I'm ready for right now. I'm just assigned archers to this, but there may be other people that are assigned to this in the very near term and it's already labeled a P1, which I think is definitely appropriate. But I hear a handed someone have a question. + +Tyler Wright: No, Roger just put something in the chat. But yeah,… + +Scott Carruthers: Okay. + +Tyler Wright: I think you're all set. + +Scott Carruthers: So I guess I'll just take a pause because of this being such a significant issue and point of attention. Does anybody have any questions shared memory support and the fact that we're very much focused on this and are hoping to allow this through a cost appointments somewhere in the next week or two. So any questions or concerns on this front? + +00:10:00 + +Scott Carruthers: Yeah. + +Scott Carruthers: so again, we have some other issues. I'm not sure why these are actually been open for a long period of time. Come and haven't had assignments one essentially. Labeled as not an issue is protobuf brush generation, which we already talked about for typescript. So again, I will assign myself to this as the Proto seat generation for other languages other than go is and then I'm going to be very much focused on. + +Scott Carruthers: I'm not gonna touch right now. This one that's labeled not an issue. We could review that. Later, but that goes through all the issues that have been open the last couple weeks and actually a couple that have been out there without assignment spending a little bit longer than that. So I guess I'll just open it up for discussion. I think that brings us through the triage and assignment any questions or concerns on any of these issues? + +Scott Carruthers: Okay, if there's no questions or concerns, I'll turn it back over to you timer. + +Tyler Wright: excellent much appreciated stuff again, if anybody has any + +Scott Carruthers: Yeah, we get a free continue to track it and our next session when Archer is on. He may have some thoughts of his own but yes, sir, Taking a step back. So make sure this clear what we're talking about. There's been a couple of issues raised recently or provider concerns recently anytime you update. Your provider attributes it's costs three dollars or I'm sorry keep on messing up aktn and hours a currently cost three akt on average to update provider attributes. So there's been some discussion recently in the community on why? + +Scott Carruthers: Those transactions are so much higher than any other transaction. So, the least creation or deployment creation or those kind of transactions are typically very cheap and they update a provider attributes is much more expensive myself and Andy and others in the core team looked at this yesterday and I mean as basically just a matter of the complexity. So obviously on any blockchain they calculation of transaction costs. They're built into that is a estimation of the computational power and a complexity of an operation. So if you look at provider attribute, updates + +Scott Carruthers: the amount of data that's included in the message body significantly higher than it would be for a delegation which the body would only have a short string obviously when we're doing provider attribute updates. It can be that the Json body is much larger. So we basically understand where we don't want to be dismissive of the concern, but updating a provider actually views should be Or other infrequent operation. We definitely are cognizant of the fact that if you're adding new GPU models or enabling persistent storage or something like that. There are legitimate activities that require provider attribute updates, but that's simply in our investigation. we don't see any evidence of this being a bug within the code basis just if you look at the + +00:15:00 + +Scott Carruthers: gas wanted and gas used estimation when the client is sending a provider attribute update. They're at least wave Castle being set to Auto based on the complexity of the operation is calculating that this is a much more computational heavy more complex operation. And yeah, I mean, that's really + +Scott Carruthers: All there is to it we can continue to look at this and Andy and I actually haven't had a chance to test this but we're discussing yesterday. It's possible that you could get those. provider attribute updates through a transaction by adjusting gas Auto to something manual and possibly it would be possible to get that transaction at a cheaper. + +Scott Carruthers: Champa damir. I know they had your hand up for a second. Did you have any thoughts on these regards? + +Damir Simpovic: Just my two cents Simply because I was doing those. + +Damir Simpovic: I probably did over a hundred of those. but I didn't honestly even know it's granted the akt PRI price was much lower but once + +Damir Simpovic: Once I figure everything out. I haven't touched anything. I don't think That this is really such a big concern honestly. even if it is But three akt or… + +Scott Carruthers: Okay. Yeah that. + +Damir Simpovic: three and a half akd. I don't see that. + +Scott Carruthers: I'm as one of our most prominent providers. I'm glad that you voice that opinion because that's certainly our opinion internally as well that this shouldn't be a huge concern because updating your provider attributes should be a very infrequent activity as I said earlier and you are echoing those Sentiments of it not being a concern and should be infrequent activity. yeah, I think that's obviously if we continue to get some pushback or send a satisfaction within the community we can look at this further. But yeah, I mean, just simply this is a more expensive transaction based on the complexity. + +Tyler Wright: Thank you Again, we'll leave that is a discussion right now, but that's a discussion in the chat. So if others want to get involved, that would be great so we can again. Get more people involved at least in the conversation. Yeah, so I'll continue to push that through we'll maybe talk during about six maybe talk about it during State providers. But again, I heard it a couple of great opinions here. So if we can get those into discussions, I would be great. + +Tyler Wright: All right moving on with the agenda at this point. We usually Reach out to someone like Rodrigo who I see here now and others that are insiders vanguards or active in Discord and other channels in the community. + +Tyler Wright: I just wanted to see on the support side. Was there anything new that anybody has been seeing in terms of new questions new comments new concerns with the network that maybe documentation can be updated or a processes maybe updated. Has anybody seen any new themes from the community? Go ahead damir. + +Damir Simpovic: So what I've noticed was a few questions regarding. when somebody Creates an order but that's not start loose. Basically, it doesn't accept any of the orders and the deployment sometimes do not get automatically closed. + +Damir Simpovic: by the providers and In which case the 5 akt requires to place a bid? remains in escrow indefinitely I think I've seen twice this already. So not sure what's going on. this is just looking further it has happened yet. Or perhaps it did but it's not and really big scale. But if somebody could even use this maliciously to drain. the smaller provider wallets because for one open order they will basically drain. 50 or + +00:20:00 + +Damir Simpovic: 100 provider it's so for five akt they gain 5880 or 500 a value quote unquote. So, I don't know. perhaps something to + +Tyler Wright: Perfect. Thank Damir. I'm taking some notes. I don't know if anybody else has seen some of those same questions or anything or has any comments or questions about what damir just mentioned. + +Tyler Wright: again, I'm taking some notes and I'll make sure that we add any additional action items related to what damir just mentioned there Roger. Don't feel the need to talk. I know Max you're probably doing a million things too, but I see you both active in Discord is or anything. That you all have seen that you wanted to call out. + +Rodrigo Rochin: nothing specifically just more users that are previous weeks and scammers still there, but we handle them as soon as you + +Tyler Wright: excellent + +Maxime Beauchamp: Yeah, nothing. Yeah when I said + +Tyler Wright: Thank you, damir. Cool, so we talked about support in Discord and other channels Scott has led us through issues that are awaiting triage and we've looked at any support issues in that repo. + +Tyler Wright: Max is here. And again, if anybody has any questions or concerns or any issues related to Cloud most I know Max is very responsive. Is there anything else that anybody wants to discuss related to support at this time? + +Tyler Wright: Go ahead Scott. + +Scott Carruthers: so I just want to make sure everyone is aware where we're at with the introduction of feature discovery on the network. So I've been in communication directly both with Broadway and damir about this and obviously Andy as well, but I've been testing feature discover heavily over the last couple of weeks and we have been making small adjustments based on. a minor issues that we've discovered during testing where to the point that we want to start rolling this out on to mainnet Providers and we want to do so in a limited fashion and with some selective providers to make sure that as this gets rolled out to Maine that we don't experience any other issues or there's not any other code adjustments that need to be made so currently we haven't merged they + +Scott Carruthers: the provider release candidate to Main because We don't want new providers coming out to the network to default to this new version that includes feature Discovery, but we just want to roll it out to a few providers. I would think that would be something on the magnitude of world out to five select providers. Make sure they're not having any issues over the first couple of days and then we'll merge the feature Discovery into Maine and then start rolling out in Mass to the rest of the network. So this doesn't require a RPC node update is only a provider. day and mostly some changes to the inventory operator within the provider code base. So as I said earlier rotary and damir are definitely both cognizant of this because I reached out to them yesterday as possibly being a couple of our first providers and he has actually already + +Scott Carruthers: experimented with this release candida on his main provider as well and also just some breaking changes and breaking news that applies to That Andy will find interesting. the current release candidate that's out right now, which is zero five zero RC 15. We notice both through. + +00:25:00 + +Scott Carruthers: Andy's experience on his mainnet provider and also some additional testing on Sandbox providers that there's a issues where labeling so along with feature Discovery being a mechanism where we can see the amount of gpus and other Hardware on the network feature Discovery also introduces automatic labeling. So no longer are providers going to have to manually label nodes for GPU resources or persistent or storage resources, but they feature Discovery through that those Hardware Discovery mechanisms will do automatic application of kubernetes labels. + +Scott Carruthers: Andy earlier this week notice and we've also experienced this on some sandbox providers that the labeling is falling off after a 30 minute window and we've already identified the issues as something with a kubernetes watcher that and this is why Archer isn't on the call today. He's making the code change the necessary to alleviate that issue. So soon and I think in the next couple hours, we're gonna see I think what would be RC 16 dropped and then what's that is available. I had conversation with rotary about this this morning. We were going to upgrade his provider, but now we're waiting this new release candidate with this latest issues remediation, so we don't want any upgrade in their providers and their operators until we make that announcement that that issue is alleviated, but I just want to bring it up on this call so that + +Scott Carruthers: obviously rodray and damir are very foundational in our support of providers in the community as well. So again, we're only going to roll this out to five select providers initially and then when we get to the point of doing the main net upgrade, obviously want to make sure that you're cognizant of these changes. So this will be announced but I kind of just wanted to give a quick status update on where we're at with feature discovering probably what they successful look like for the remainder of the week. + +Scott Carruthers: Rodrigo, go ahead. + +Rodrigo Rochin: So yeah, just wanted to mention that ready and I read the docs in. y'all know I'm not an expert like the mirror and Andrew your spot. But ducks and instructions look perfect. So shouldn't have a problem with you already. + +Scott Carruthers: Yeah, that's great. And that is actually another reason that we wanted to just do this for five initial providers not only doing certainly checks of the code base but also some sending checks to the documentation and this week upgraded his provider to rc15 and he said he was pretty much disabled to copy and paste all the docs. So we got some initial great feedback from the documentation. But if there's any need for clarity, we definitely want to get those in place the documentation before we send it out the community as well. + +Scott Carruthers: Yeah, that was it. Let me know if there's questions on that + +Andrey Arapov: Scott I just wanted that if that's at 5.14 not 15 yet and when I was checking 5.14 the storage wasn't returned. So the presentation storage was empty in provider status information. And I know if you had the chance to test it more with 0.15. I just had my time for that yet. + +Scott Carruthers: Yeah, so yeah, let me go back and maybe I wasn't entire eclair when that is precisely the issue that we're rectifying currently and the reason why arter isn't on this call is he's looking at that so with the introduction of so when we were looking at your provider or this week, we saw the feature Discovery wasn't reporting persistent storage. And the reason for that was the necessary labels were on kubernetes nodes. So that issue is now been identified and it's a issues where kubernetes washer falls off after 30 minutes. So that's precisely what arter is. currently introducing new code to rectify so when rc16 or whatever the next release candidate is issued will definitely want to do retesting of that. But that's precisely what the current focus is on. + +Andrey Arapov: Nothing for colorification. Everything is clear now. + +Scott Carruthers: Okay, great. Go ahead, damir. + +Damir Simpovic: Regarding the sandbox provider. I actually started doing this today that discussed with Yeah, I mean it's just actually started doing this and most of the stuff is done. all I need to do is to actually just install the + +00:30:00 + +Damir Simpovic: Your Castle but not sure if I could run. + +Damir Simpovic: I could run an RPC note, which is in the sandbox if that I need some kind of a Snapchat or is there. something somewhere that I can download from + +Scott Carruthers: Yeah, so we could have approach that into that two ways. Andy, and I actually is two weeks ago. I went through the spin-up of a new RPC node in a sandbox and it was a little bit painful based on. Snapchat I use I get into this and show you how so we definitely could go through the procedure of spinning up RPC node in your provider and your sandbox provider and we can definitely go to the process but there's really no reason in sandbox. You just can't use the public RPC endpoint and again the new Provider updates do not require any RPC updates. So using this the public RPC node in sandbox should be fine. Then that would definitely be the easier path. + +Damir Simpovic: All right. Thank. + +Tyler Wright: Thank Anyone have any other questions really want to discuss before I let all go. + +Tyler Wright: Again much. + +Scott Carruthers: and it And… + +Tyler Wright: I'll go ahead Scott. + +Scott Carruthers: if it wasn't query shampa, so appreciate building out a Sandbox provider and testing feature Discovery there before you update Euro plots. We'll want to I was just that you wait to the actual provider build until the latest release candidate is release which should be in the next couple hours. So I'll definitely reach out to you directly once we're ready to proceed with that. + +Damir Simpovic: Thank That will Certainly not happened earlier than tomorrow. So no worries. + +Scott Carruthers: Okay. Sounds good. Thanks. + +Tyler Wright: Of anything else that anybody wants to discuss before I let y'all go. + +Tyler Wright: If anybody has any ideas that they want members of the core team to demo or anything like that related to testing environment set up Etc. I know that's Scott and Archer I've done this previously so you can look at previous video recordings from I think of six support meeting, maybe two or three ago, maybe 24, but again if there's any other Concepts that anybody wants more clarification on when technical standpoint feel free to reach out to me or feel free to drop it in Discord channel for six support and we can talk about adding those to a future agenda. + +Tyler Wright: Much appreciate everyone's participation today again. If you're watching this later, feel free to join these meetings in the future and get involved in contributing to core code whether it be on the passport coach core code side where you can find issues in the support repo or supporting Cloud notes on their continuing growth and development in the cloud most repo, but there's a number of ways to get involved, but again much appreciated everyone's time today. Hope everyone has a great rest of your day and we shall talk soon. Thanks again now. + +Scott Carruthers: Thanks, everyone. + +Andrey Arapov: I think + +Rodrigo Rochin: You later. + +Meeting ended after 00:34:34 👋 -# **Transcript**