From ffe3617c8a5b19e17dd957e7cb341818bec787b3 Mon Sep 17 00:00:00 2001 From: George Appiah <48031636+iamGeorgePro@users.noreply.github.com> Date: Sun, 21 Apr 2024 07:43:31 +0000 Subject: [PATCH 1/2] Create meeting notes for sig-support #29 Signed-off-by: George Appiah <48031636+iamGeorgePro@users.noreply.github.com> --- sig-support/meetings/029-2024-03-06.md | 126 +++++++++++++++++++++++++ 1 file changed, 126 insertions(+) create mode 100644 sig-support/meetings/029-2024-03-06.md diff --git a/sig-support/meetings/029-2024-03-06.md b/sig-support/meetings/029-2024-03-06.md new file mode 100644 index 00000000..ac3bd665 --- /dev/null +++ b/sig-support/meetings/029-2024-03-06.md @@ -0,0 +1,126 @@ +# Akash Network - Support Special Interest Group (SIG) - Meeting #29 + +## Agenda + +- Participants will look at issues in [Akash Support Repo](https://github.com/akash-network/support/issues). +- Scott Carruthers will lead triaging issues, and discussion of implementation for issue fixes. +- Scott will discuss Support from the community and ask for any community feedback. +- Participants can have an open discussion of any other issues or support related items on Akash Network. + +## Meeting Details + +- Date: Wednesday, March 20th, 2024 +- Time: 07:00 AM PT (Pacific Time) +- Recording: Coming Soon +- [Transcript](#transcript) + +## Participants +- Andrew Hare +- Andrey Arapov +- Deathless +- Maxime Beauchamp +- Scott Carruthers +- Tyler Wright +- Zeke E + +## Meeting Notes + +### Introduction and Updates +#### Tyler wright +- Core team and community members discuss issues in [Akash Support Repo](https://github.com/akash-network/support/issues) +- Invitation to community to bring forward any issues +- Lighter attendance expected due to conferences +- Mainnet 10 upgrade discussed (March 13th) +- Parameter change for deposit amounts (from 5 AKT to 0.5 AKT) +- Change went into effect on March 11th or 12th and documentation updated accordingly +### Triage of Open Issues +- Scott facilitated the triage process +- Reviewed open issues in Akash Network support repo +- Identified unassigned issue regarding memory issues on archival nodes +- Andrey provided insight on memory usage and suggested further review +- Scott tagged Arthur for further investigation and decision on the issue +- Andrey Arapov suggested converting the issue into a discussion +- Scott agreed to convert the issue and assigned Andrey for follow-up +- Andrey requested to leave the issue open for reference while creating a discussion + +#### Issue opened by Community member(Issue 195) +- Issue opened by Community member, discussed by Scott and Maxime. +- Scott reviews the issue, notes absence of reliable Per test and local developments. +- Mention of unreliable sandbox and need for local development environments. +- Maxime explains difficulty for new developers to run Cloud locally. +- Discussion on the dated nature of the issue and potential new solutions. +- Maxime assigned to review issues in Cloudmos repository. +- Scott offers support to Maxime throughout the process. + +#### Review of open issues in provider resource release process ( Issue 193). +- Discussion on issue of provider should release resources immediately after never accepted deployment was closed. +- Scott and Andrey share experiences and observations regarding delays in resource release. +- Mention of variability in delay times, ranging from 3 to 45 minutes. +- Andrey provides examples of extended delays, up to 6 minutes, and references a possible 25-minute delay. +- Speculation on the cause, including potential issues with the inventory operator. +- Connection made to related ticket 192 regarding issues with the inventory operator's data polling. +- Andrey suggests further investigation into the connection between the two issues. + +#### Issue reported by Deathless regarding abnormally large GPU and RAM allocatable statistics. +- Discussion on an urgent issue reported by Deathless, involving abnormally large GPU and RAM allocatable statistics. +- Andrey suggests that the issue may be related to the upgrade of the inventory operator from 0.4 to 0.5, with potential impacts on resource visibility. +- Andrey describes the need for manual restarts of the inventory operator to address the issue. +- Scott reclassifies the issue as a P0, indicating urgent attention required, and notes that it's already assigned to Archer. +- Discussion on the impact of high memory allocatable stats, affecting bidding and deployment processes. +- When memory allocatable stats are high, bids are not received if the request uses a wildcard for the GPU model, but bids are received if a specific model is specified. +- Uncertainty whether the issue should be classified as a P0 or P1, but its impact necessitates urgent attention. +- Andrey notes similar issues with the Hurricane and European providers, persisting even after inventory operator restarts. +- Scott highlights the severity of the issue, considering it one of the most severe ongoing issues in the network. +- Agreement to prioritize urgent attention to the issue. + +#### Unresponsive status endpoint issue reported by a provider. +- Andrey discusses the issue of a provider's unresponsive status endpoint, noting other ongoing issues with the Hurricane provider. +- Scott acknowledges Andrey's comments and suggests waiting for further testing before reclassifying the issue. +- Zeke mentions a Discord conversation where a user reported unreachable port 8443, prompting clarification on the issue. +- Andrey suggests updating the engine X Ingress operator config to allow port 8443, which was enabled with the provider upgrade to version 0.5. +- Scott clarifies the issue and concludes the review of unassigned issues, opening the floor for further discussions. + +### Update on Shared Memory (SHM) enablement over SDL. +- Zeke inquires about the timeline for Shared Memory (SHM) enablement over SDL. +- Scott informs Zeke that the code for SHM enablement is complete, with Adam as the primary engineer, and they are awaiting the release. +- Documentation updates have been staged to include SDL examples and necessary attributes for providers to support SHM. +- Scott mentions that the release timing is uncertain, but once released, all documentation will be updated accordingly. +- Zeke expresses satisfaction with the progress, noting the ability to deploy SHM on Akash without manual intervention. +- Scott confirms Zeke's observation and concludes the meeting, passing control back to Tyler. + +### Efforts to combat spam in the Discord server. +- Tyler starts the agenda by mentioning the recent efforts to reduce spam in the Discord server. +- He notes that keywords have been added to a block list, resulting in a decrease in spam. +- Tyler encourages community members to reach out if they encounter any spam-related issues. +- Zeke, a newly appointed Vanguard, acknowledges the effectiveness of the auto-moderator in catching spam. +- Apart from spam, Zeke doesn't identify any significant thematic issues in Discord recently. +- Tyler appreciates Deathless and others for their contributions in addressing issues and providing feedback in the provider and deployment channels. +- He encourages community members to report any issues or suggest improvements for the user experience in the support channel or special interest groups. + +### Other Matters Discussed +- Introduction of Andrew, a new senior software engineer at Overclock Labs. +- Andrew will continue to join future calls and is currently working behind the scenes on various tasks. +- Tyler praises Andrew's work ethic and integration into the team. +- Reminder to mark calendars for upcoming SIG clients and SIG providers meetings. +- Tyler anticipates discussions on provider auditing and feature discovery in the upcoming meetings. +- Appreciation expressed to all participants for their time and energy. + +### Closing Remarks +- Tyler asks if there are any remaining support or core code-related topics to discuss. +- He suggests closing the Sig chain issue related to mainnet 10, which has passed the vote and the upgrade successfully occurred. +- Tyler plans to follow up on several high-priority issues, including 195, 194, 193, 192, and 190, at the next SIG support meeting or sooner. +- Scott informs the group about the migration to the new documentation site over the weekend and asks everyone to report any missing documentation or formatting issues. +- Tyler emphasizes the importance of reporting any anomalies or formatting issues in SIG docs for prompt resolution. +- Tyler acknowledges Scott and Andrey's efforts in ensuring the documentation is clean and ready for community use, thanking them for their oversight and collaboration with Piyush and his team. + +## Action Item +- Tyler to close the Sig chain issue related to mainnet 10 after obtaining consensus from the group. +- Max to follow up on issues 195 and 194 and determine the necessary actions. +- Group members to report any missing documentation or formatting issues in SIG docs for prompt resolution. +- Scott and Andrey to continue overseeing and ensuring the cleanliness and usability of the documentation site. +- Scott to reach out to Arthur for an ETA on the release of SHM enablement. +- Scott to coordinate with Archer to escalate the urgent issue by Deathless for immediate attention. +- Scott Carruthers to tag Arthur for further investigation and decision on the memory issue. +- Andrey to conduct further tests to replicate and investigate delays in resource release. +- Scott and Andrey to monitor related ticket 192 regarding issues with the inventory operator's data polling and assess its connection to the current issue. +## Transcript From 4d0a02ea00ec6e7eb8ea716d07ae03cddd3c59bb Mon Sep 17 00:00:00 2001 From: Drinkwater Date: Thu, 25 Apr 2024 09:35:46 -0400 Subject: [PATCH 2/2] Update 029-2024-03-06.md added transcript Signed-off-by: Drinkwater --- sig-support/meetings/029-2024-03-06.md | 310 ++++++++++++++++++++++++- 1 file changed, 309 insertions(+), 1 deletion(-) diff --git a/sig-support/meetings/029-2024-03-06.md b/sig-support/meetings/029-2024-03-06.md index ac3bd665..7c0254b4 100644 --- a/sig-support/meetings/029-2024-03-06.md +++ b/sig-support/meetings/029-2024-03-06.md @@ -123,4 +123,312 @@ - Scott Carruthers to tag Arthur for further investigation and decision on the memory issue. - Andrey to conduct further tests to replicate and investigate delays in resource release. - Scott and Andrey to monitor related ticket 192 regarding issues with the inventory operator's data polling and assess its connection to the current issue. -## Transcript + +# **Transcript** + +_This editable transcript was computer generated and might contain errors. People can also change the text after it was created._ + +Tyler Wright: All right, welcome everybody to seek support to biweekly. I believe this is number 30 or 31 this support biweekly has been going on since open framework has started. This is a biweekly meeting where the core team and members of the community go over any issues and the support repo inside the accost Network organization that are waiting triage if there's anybody specifically from the community that is brought any issues forward and wants to discuss an issue. This is the biweekly meeting to join where you can meet with members of the core team discuss the issues and again, they can be prioritized and look to be solved. + +Tyler Wright: I know Scott who's been leading us through really the last couple especially in my absence will be leading us today. after we go through the issues awaiting triage usually be looked to members of the community to share any feedback that they've gotten and Discord Telegram and other channels. I know usually we have a number of insiders that join us, but there are a few conferences that insiders members of the core team Etc at so I was expecting a lighter week than usual. With that said the six support meetings transcripts notes and recordings should be up to date out of the last couple of Sig meetings. I know that again Scott lettuce through triaging any issues. He talked about the mainnet pen upgrade which happened last. + +Tyler Wright: Wednesday, March 13th. So week from today last week around this time. There was a plans Network upgrade that took less than five minutes and required again. Now, they upgrade their nodes and then providers have to upgrade their RPC notes. So again, thank you to all about data set participated and all the providers and have since upgraded their nodes and obviously thank you to members of core team here that help with documentation testing Etc. + +Tyler Wright: In addition to the main Net10 upgrade there was a parameter change that what didn't require Network upgrade but did go on chain for boat that's lowered the big deposit amounts for both the tenant side and the provider side believe it changes from five akt to 0.5. Akt. So again that went into effect. I believe March 11th or 12th and has been tested and seems to be Working properly end of the documentation has since been updated. So let's just say recap of some of the things that have been happening over the past a couple of weeks since the last six support meeting and even a little bit prior to that. But without further Ado, I'll hand it over to Scott to triage any issues of waiting triage on just be taking some notes in the background. but Scott's feel free to + +Scott Carruthers: All right. Thanks Tyler. Let me share my screen. + +Scott Carruthers: Okay, is everyone able to see my screen? + +Scott Carruthers: thumbs up it sounds good. So I will. Go into the lakash network support repo and take a look at the open issues. I looked at this a little bit before our call and it does look like there's a couple that don't have assignment or prioritization so we can go through those. first one that we come across that has been created since our last six Port is a review of our provider auditing requirements and Andy has already assigned himself to that. So I'll just progress down to the first unassigned issue. So this was open by Andy so Andy may have a comment and yell rely on you for a little bit of review, but this is + +Scott Carruthers: An issue where we are seeing some amount of memory issues. in certain queries on archival nodes. I Andy would you like to give some more thoughts on this? + +00:05:00 + +Scott Carruthers: Okay. + +Scott Carruthers: Yeah. + +Andrey Arapov: you can see there in the screenshot 140 gig of memory or 157 gig so it means you just needs more memory for its operations,… + +Scott Carruthers: Yeah, I'm just gonna Put a note in that we're leaving this open despite the fact that it… + +Andrey Arapov: but for one hand,… + +Scott Carruthers: carried by increase Ram allocation + +Andrey Arapov: I will think to maybe close it but another hand I'm thinking to leave it for Arthur to take a look and decide if that's expected because if it's not expected then it probably needs to be addressed. I mean the fact that it uses that whole amount of memory for this queries because maybe it's not supposed to use that much of memory And just maybe tie garter just to take a look and decide if he's okay to close it as it is, I would like him just like to go through just understand what happened and decides if it's something that we should really take care. I mean they closer look or not. + +Andrey Arapov: + +Scott Carruthers: Yeah, I tagged him there with that comment share. + +Andrey Arapov: maybe it's growing this time. We don't know so it's definitely not like you wanted to it's the lowest score in terms of priority,… + +Scott Carruthers: And also assign them. + +Andrey Arapov: but that's my five sense. + +Andrey Arapov: Maybe it's worth to change the title of this issue or maybe just leave it because it's not running out of memory anymore since he increased the memory, but it's rather consumes lots of memory, basically. More of a question. + +Scott Carruthers: Yeah. + +Andrey Arapov: Maybe I think it's even belongs more to but you can converse it into a discussion rather than the issue I believe. + +Andrey Arapov: I don't think that there is a possibility to convert it like, the issues into discussion, but I can do it later. + +Scott Carruthers: Okay. + +Andrey Arapov: Yeah, I think it was just a great probably discussion and based on this issues and just leave it in under discussion and close this issue. + +Scott Carruthers: you want me to quote you want me to close the issue and you can just create just Okay. + +Andrey Arapov: No, no just leave it so that I don't forget it. and then I'll create a discussion and refer to this issue because it's not an issue. + +Scott Carruthers: okay, I'll sign you on this as well just for Is a visible there? Okay. Sure. + +Andrey Arapov: Yeah things. + +Scott Carruthers: Okay, so I was actually reviewing this issue earlier today. So this was opened by a Community member. I am going to go back to people that are currently on call. I don't believe that this person is currently on the call. but I okay. + +Maxime Beauchamp: It's narrow. + +Maxime Beauchamp: the new team member + +Scott Carruthers: Okay. + +Scott Carruthers: So yeah, so I just was briefly coming through this before. Her call as you can see There's no reliable Per test and local developments and so I thought I don't know if any so max. Maybe you have some context for this but he's staying in those currently no reliable blockchain for tests and local development for instance tests local environments prior to either download and important DB or uses unreliable sandbox, so I guess there's two then I could write this in the comments and possibly get some further feedback, but the two obvious thoughts that come to mind is so we have the ability to spin up local development environments, which is well documented within our engineering documentation, and I'm also + +Scott Carruthers: Not entirely clear on the comment of unreliable sandbox. So I don't know if anyone else might have some further thoughts on the kind of the Genesis for this. + +Maxime Beauchamp: Yeah, so what he's talking about is to run Cloud those locally. and this is sort of like the indexer and we + +00:10:00 + +Maxime Beauchamp: apparently run the indexer since Genesis and there's a few reasons why we need to have the whole. Database locally as well for the API to run properly or the indexer to run property. So it's a bit tricky for new developers to start working on that locally on Carlos. So yeah, that was the issues that we had with yarrow as a new team members start working on combos. And we discussed it a few times I can see that this issue is created last week. and I think it's a bit dated as we + +Maxime Beauchamp: We were taking it new Solutions. there's another issue I think for tracking this on cloud will austerity so I'm not sure… + +Scott Carruthers: Okay. + +Maxime Beauchamp: if that issues needs to be there. + +Maxime Beauchamp: entirely + +Scott Carruthers: How about would this be a good Next Step Max? I could assign you to this issue and then you could do a review of the issues that's opened in. The cloud must repository and then determine if it's appropriate to close this issues. + +Maxime Beauchamp: Yeah. + +Scott Carruthers: Okay. + +Scott Carruthers: All Yeah Max obviously as a progresses if you need our support. On this in any way, I'm sorry. I have a better understanding of the issue and based on the indexer. It's if not necessary to work with the full may not database. So yeah, obviously as you review that other issue or if you need a support on this, just let us know. + +Scott Carruthers: so I believe that leaves us with two remaining unassigned issues. So the next one is provider should release resources immediately after. Not accepted or the deployment was closed. So I'm rather familiar with this issues. Andy you open this so you may have some thoughts as well. I'll probably assign this to Archer for initial triage and he can triage it further from there, but I know that I was looking at this I think. you and I were looking at this on the same provider and seeing that in my experience the inventory operator when a bid was closed. There was a lag possibly. And this is a very rough approximation but in my testing maybe a lag of two to five minutes between + +Scott Carruthers: when the bid is closed and when those resources are made available again via the inventory operator, so I believe that's generally the issues that were talking about any for their thoughts or clarifications any + +Andrey Arapov: yeah, sometimes it was taking even up to 35 minutes depending on which provider and which condition so it's kind of + +Andrey Arapov: It's not too deterministic to really reproduce in terms of time. But yeah, so, whenever there are all the requests in the network and providers are bidding on them. And even if they did not need lost a bit those resources on some providers in some cases not all they are getting locked up for longer than they should because normally if rather didn't beat on it or if it did but it lost that beat supposed to immediately release this resources. I'll never it's okay to log them up well with this open because it can accept it. But if it wasn't accepted and supposed to release and that can create problem and provider will be just getting running out of resources in case if somebody will be generating lots of requests every 15 minutes and sort of attack sort of + +00:15:00 + +Andrey Arapov: And yeah, I think it's important to look at this closer and I think it started happening since the upgrade to the provider From 0.0.4. So yeah, but at least it doesn't happen across all the providers and only specific cases. + +Scott Carruthers: yeah, so in my experimentation with this, so I was closely looking at the status of the bid on chain and if they was created via another provider or just the bid was closed from the loy. So the deployment was closed. I witnessed that they've close immediately on chain in the inventory operator would take about three to five minutes to recognize and release those resources. I was never able to reproduce the Scenario of the 30 to 45 minutes and I was even testing it on the provider that reported it. So we were somewhat of the belief that there was something else in play. So Andy, have you actually observed or you able to replicate a 30 to 45 minute? Failure to release the resources because I've never seen anything. That extreme I know. + +Andrey Arapov: Yeah, the day that I've tried it I reproduced it and that's what the sticker describes and can you try to scroll down? Maybe there was some update. Maybe to the bottom of it. + +Andrey Arapov: And slightly up if it speaks something about this time delays or something. + +Scott Carruthers: So this says after six minutes, it's still locked. And I did. + +Andrey Arapov: Yeah, so yeah, it was just additional test that I checked if there is something about this five, I mean the mid timeout which is the old value so it was locked for longer than six minutes and then I just didn't want to wait longer. I just realize that probably it will be there for longer time because I run several tests on this problem one of this providers and this one particularly. And yeah, I mean, of course I can run more tests to see if I can still reproduce it. But I believe it's still there and don't think it will be just gone by itself. + +Scott Carruthers: No, I don't think it would be gone by itself either and I think it's very easy to reproduce. So there's a several minute delay. So I was just curious that if we were able to reproduce that extreme delay of 30 to 45 minutes, but here you have an example of six minutes, which is obviously abnormally a long time. So, that's fine. + +Andrey Arapov: Yeah, in the beginning in the top of the ticket. I think I wrote there is Christian to 25 minutes as well. Yeah, yeah,… + +Scott Carruthers: Yeah, okay. + +Andrey Arapov: because I didn't just put the numbers randomly. So it's essentially run some tests on my own. + +Scott Carruthers: Yeah, I'm just saying that the most I could reproduce was three to five minutes but it looks like you were able to reproduce something much greater. So okay, it's fine. + +Andrey Arapov: Yeah. + +Scott Carruthers: the thoughts are comments on this issue. + +Andrey Arapov: not really but I think maybe it's related to the inventory operator the way how it pulled the data because normally what I know, it pulls every 15 minutes and there are some issues if you look at the next ticket 192 why the reports successfully High multiplic in order to operate his hair, yeah this one. so I think it's kind of connected slightly upper one into + +Andrey Arapov: and I will + +Andrey Arapov: about online to have + +Andrey Arapov: So you can open it up. + +Scott Carruthers: yeah, yes so this is actually an issues that this provider ping me yesterday as well. + +Andrey Arapov: So I think + +Scott Carruthers: So yeah, he goes through a period of time where so the provider I'm talking about is + +Scott Carruthers: deathless This is just and… + +Andrey Arapov: yeah. + +Scott Carruthers: then yeah, he's showing so I've seen this issue where we have abnormally and I know that Max has seen this issue before as well. We get abnormally large GPU alcatable. statistics I have never seen this with anything outside of gpus but yeah deathless is definitely currently experiencing this with a Ram allocatable + +00:20:00 + +Andrey Arapov: Yeah what I think is that when we upgraded the internet operator on the provider to all point, five four, I mean, so when we jumped from 0.4, I think even before in this older version of Provider, sometimes when the media device plug-in is getting stopped or restart or just being in the beginning of the insertation. It has in impacts on the amount of the GPU provider sees and when you restarted or when you wait a little bit and then it kind of picks itself, at least it was back then with 0.4 but with 0.5 when you were one + +Andrey Arapov: Because there is entirely changed the way how we winterize things from understood. I think it doesn't self recover. So it needs to be manually restarted as soon as you see this high of excessive amount of Resources and the last command one before the last one. I've just updated five hours ago. Yeah, because I was reporting on the North quite often recently. I was testing these growth AI seeing you so I had to was looking up the GPU in some conditions and I need to restart an old quite often and after it was about it up and I've noticed that issues was always coming back again. So I needed to restart the interior operator. Sometimes it was even reporting zeros. + +Andrey Arapov: So kind of interesting stuff, but they definitely needs to be getting the soft recovered from this issues. + +Scott Carruthers: I was just reading through your comments from five hours ago as well. So yeah, I've Actually reclassified this as a p zero, I think this needs some urgent attention. and it's already assigned to Archer. So + +Andrey Arapov: I think so. But another hand I was thinking when it doesn't show that amount of available resources is unusable like zero or something. So it shows that you have way too many resources one hand and why I'm saying it maybe not be zero, but maybe if you want because providers are able to offer the resources, they just able to offer to much of the resource. So then they have so well another hand. Yeah. It's probably zero. + +Scott Carruthers: yeah, so deathless actually reported yesterday that for some reason when he's getting a high memory El capable stats. so this is a strange issue. understand exactly where the correlation in this is but deathless yesterday was reporting that so he only had a certain node that was reporting incredibly High memory allocate allocatable statistics and whenever was in play and whenever that issue is current he was unable to rent out GPU or he wasn't deployers were not getting bids from his provider if + +Scott Carruthers: The sdl and so again this is kind of strange because you want to think there's any correlation here, but he was reporting and was able to reproduce it. When those memory lkdable stats are very high. If a sdl is requesting a GPU and it has a wild card for the model his provider won't bid if you specify RTX 30 90 or whatever. He has a specific model it will bid. But if he's using a wild card for the GPU, it doesn't bid during so I guess that's still debatable if it's a P0 or P1, but it definitely is having other ramifications. I think it's something that we need pretty urgent attention on. + +Andrey Arapov: The other Handler I think it should really be zero and also hurricane provider. I noticed today. also the comment to the second last one, but European provider is experiencing this issues now persistently,… + +Scott Carruthers: Yeah. + +Andrey Arapov: even if I restart the inventory operator, it's still keeps reporting with the worker now,… + +Scott Carruthers: Yes. All right. + +Andrey Arapov: so it's persistent issues. definitely + +Scott Carruthers: I would actually say this may be one of the most severe issues going on in the network right now. + +Andrey Arapov: no. + +Scott Carruthers: So yeah, we'll definitely try to get some urgent attention on up. + +Andrey Arapov: Okay. Thanks. + +Scott Carruthers: and then + +00:25:00 + +Scott Carruthers: Yeah, no, probably. Yeah,'s Zeke can death was just joined. So yeah, we were just going through the discussion of Resource the inventory operator is given abnormally allocatable amounts at where we are previously just seen on gpus but now we're starting to see it on memory as well. So the case is the issues fully up to date and I requested it from a P1 to a P0 because we understand this is probably the most urgent issue that we have on the network right now. So requested to a P0 to try to get some attention. + +Scott Carruthers: Okay, and then that leaves are very last unassigned issue. Andy do you want to so this was open some two weeks ago do you want to is this + +Scott Carruthers: so the provider stops responding to the status endpoint are you saying that they provide ourselves responding both to the rest API 8443 status endpoint and the grpc part of a44 simultaneously here + +Andrey Arapov: Yeah, it's the most important honestly and it's when the paper writer was enabled. Actually, I have not tested it back again because there are some other issues on the hurricane provider such as the abnormal reports of the resources, And whatever there no restarts. No issues. Okay. I am that's my last command that I said that I want to enable the appealing back again to see if I can still reproduce it, but I haven't got back time to do that issue because those focusing on the other issues and that other issues also. Boiled up with the abnormal report of the resources of thinking maybe that's one of the reasons for that for this issue. I will need to get back to it. + +Scott Carruthers: Okay. + +Andrey Arapov: But I think it's not. + +Andrey Arapov: Be zero of no nor P1 because I didn't have anyone reporting this issue. + +Scott Carruthers: Yeah, I have it as P2 and I'll just put in the comment that waiting for their testing by yourself. + +Zeke E: And I saw someone on Discord this morning put in. + +Andrey Arapov: Yeah. + +Zeke E: A basically saying that it happened to them. It's in the providers Channel. + +Zeke E: + +Scott Carruthers: So just for clarity what exactly did they experience? + +Zeke E: There so they said that Port 8443 isn't reachable. But April 4 is 844 is so + +Scott Carruthers: Okay. + +Andrey Arapov: That must be another thing. It must be that they need to update the engine X Ingress operator config to allow the port 844 because this one is the new part that was enabled since we upgraded the providers to version 0.5. So Yeah,… + +Zeke E: Katrina that's probably up. + +Andrey Arapov: so Zeke if you can just bring me the link to this conversation. I will reply to the person. + +Scott Carruthers: Wait, so just for clarity. I thought Zeke were reporting the opposite. So I thought you were saying that a44 they grpc. Is working… + +Zeke E: The grpc is working. Correct. + +Scott Carruthers: but a four for three is not okay. + +Andrey Arapov: There's the opposite. Okay. just let me know. + +Zeke E: Yeah, I'm sending it to. + +Andrey Arapov: Yeah, yeah. I'll take a look. + +Scott Carruthers: so that concludes our review of unassigned issues are there any For the discussions on any of the issues that we talked about or issues that we haven't talked about this morning that anyone would like to review further. + +Scott Carruthers: Okay. + +Zeke E: I might have missed this but is there a timeline for shm enablement over SEL? + +Scott Carruthers: so as you are probably aware Zeke a number of the cash core team are a kubecon today. So yesterday we got the update that Arthur has so basically Adam about an edge was the primary engineer working on this so it's code complete and we're now just waiting they release I've actually already staged to documentation update so that as soon as this is release, we'll have sdl examples and also, Guys who are providers adding necessary attributes to their provider config for shm supports or ready to go and the code is completed. We're now just waiting for the release that will unlock as shm with team members at kubecon. + +00:30:00 + +Scott Carruthers: I'm not sure when that's going to happen and what exactly we're waiting for. But yeah, that's basically the update that I have. Is this Coco complete and we're just waiting for the release to be cut that will support sham. And as I said, once that happens all the documentation and other bits are updated, so I actually was wondering if we might actually see that release this morning. That didn't happen if you want to. Maybe reach out to Arthur when he's available. Maybe you could get a ETA on when we think that release was going to become available. But that's all the information I currently have. + +Zeke E: Okay, sweetie. It's perfect. Now we can actually Deploy srock on the cost. + +Scott Carruthers: yeah, without the provider having to go in and yeah. + +Zeke E: And manually a bit. Yes. + +Scott Carruthers: Okay, so yeah great question Zeke for because there may be other people that are wondering that because there's been some chatter about this system native support and like I said is code complete so we should be releasing that good update on that and if there's not anything else that concludes our issues review, so I'll hand it back to you Tyler. + +Tyler Wright: Thank you very much, Scott Andre and others. Again, if anybody has any issues that they see over the next couple of weeks, please feel free to add comments any open issues already or if you see any new issues go through the proper mechanisms to again Ask insiders vanguards another members of the community and then if it gets escalated from there, then it'll be added to the support repo. As Max kind of covered they're also is a cloud most repo where any issues related to Cloud most live. I know the cloud most team does a great job of if anybody has any issues that they see in Discord and ecosystem Cloud most Channel or again elsewhere Cloud must team great issues for those and then continues to update those issues so you can monitor + +Tyler Wright: The Cardinals repo as well as roadmap and the support repo to see all the things that are being worked on. + +Tyler Wright: cool, At next part of the agenda. We usually see if any community members are here to give any updates on. Behaviors or themes I've seen and Discord or other channels related to support real quick. I know that the last couple of insiders meetings. There's been a real effort to improve issues around spam and Discord server. There's been a number of keywords that have been added to a list for the block lists and it seems like there's been some significant less spam. We'll continue to monitor that and talk about that during the support repo as well. As this is something that Akash insiders are actively talking about during The Insider office hours. + +Tyler Wright: But again, if anybody has any issue related to spam feel free to reach out to an Insider or myself and will continue to monitor that very closely. Is there any other issues I see Zeke here who has recently been made a Vanguard. So again, shout out to all that Zeke does Zeke have you seen anything in Discord? I know that you just shared some stuff with Andrey related to Port issues. But is anything else thematically that you've seen over the last couple of weeks and Discord that might be worth sharing with the Core team and the other members of the community. + +Zeke E: I mean right off the bat out in the auto model has been very very helpful. And catching spans. So I really like that. + +Zeke E: In terms of schemes. I think that's kind of just been the main one. I mean, yeah, nothing. I can really think about the top of my head. Other than that. + +Tyler Wright: I know that again deathless and some other members of the community have brought for some issues in provider Channel deployment channel so much appreciate content participation obviously deathless giving a number of people feedback sharing opinions thoughts and helping people with the class. So again much appreciate those efforts if anybody sees any glaring themes or issues related to whether it be use of product any of the tool documentation anything that could use update to make the user experience better for a cost users again, feel free to drop something and big support or feel free to attend another special interest group whether it be docs otherwise, + +00:35:00 + +Tyler Wright: Is there anything that anybody wants to discuss that's support or core code related that hasn't been discussed already? + +Tyler Wright: Real quick just for the consensus of this group. The Sig chain issues. It relates to the main Net 10 is still open. I know that has again passed on vote and then the upgrade is successfully happened last Wednesday. Is it fine if I close that issue? For repo just want to get thumbs down from this group. On closing the sigchainish cool. + +Tyler Wright: I will close that issue. I know there's a number of issues that will be followed upon, specifically 195 related to the memory of our archival node. I know that Max will follow up on issues number 194 and figure out where that should go and then issues 193 and 192 which are quite high priority as well as 190 will be major focuses over the next couple of weeks. So we'll look to follow up with those certainly at the next six support meeting if not sooner. + +Tyler Wright: Is there anything else that anybody wants to discuss before I'll let you all go. ahead Scott + +Scott Carruthers: It's just one thing to mention to this group to help us support the community and just make sure that everyone is aware I believe is over this past weekend. We migrated to the new documentation site. So definitely. if you see any chatter in the community, like any missing documentation or any formatting issues or anything of that nature and just so that you're aware in general if you go to akash dot Network, which is right now redirects to network with the path of Docs. So you can visit either of those paths or it's all also the documentation site that is linked off of the network home site. + +Scott Carruthers: So yeah, just make sure the entire group is aware so it should be very clean. they developers of the new documentation site did a diff on the old documentation site. and so hopefully there should be complete parody and there's not any documentation missing so this group is aware. We went through that migration and please let us know if you see any anomalies or any missing information + +Tyler Wright: Yes, great call out If you all see any missing information or anomalies or formatting issues, please drop a message and Sig docs with the specific documentation linked that would be extremely helpful and again will continue to push to have that as clean as possible. But do want to shout out the efforts of Scott and Andre who were very Hands-On and making sure that the documentation matched up as much as possible. I know that push and his team, we're very busy and we're trying their best but much appreciate Scott and Andre's oversight on making sure that documentation is clean and ready to be used by the community because I know a lot of time effort and work put into. + +Tyler Wright: Both like updating the developer documentation. Also the engineering documentation that's been added more recently. So again, thank you very much for bringing that up Scott and all the efforts here. + +Tyler Wright: if there's nothing else then again much appreciated everyone's time and efforts today during this call again. Feel free to drop any comments inside any specific issues. Please stay involved in Discord. And if you see anything worth calling out, please call it on the six support Channel Sig docs or any other respective channel that makes sense and again much appreciate everyone's time and energy today. me, + +00:40:00 + +Tyler Wright: he does not have to speak but I will call out that again the overclock Labs core team. Obviously Max and Andre is a part of that have also added another senior software engineer and Andrew who is on this call and will continue to join these calls in the future. He's working on a number of things behind the scenes and getting his feet wet. But he's a very very cease and go developer that again has gone for the rigorous process of bedding amongst the team and he's been enjoyed to work with the thus far so I'm sure many folks here and listening in we'll get to know him more as time goes on but just want to shout out does not have to say anything at this time. + +Tyler Wright: I hope everyone has a great rest of their day. Thank you very much. For participation today. There's a number of sigs again a couple of sigs got moved from this week to next week because of all the travel, but please add to your calendar Sig clients and Sig providers. I'm sure we'll do some talking about provider auditing some other stuff related to Future Discovery, and I know a couple of client related things. So again Market calendars for next week, because it'll be a basic week next week, but much appreciate everyone's time and energy today. Hope everyone has a great rest of your day. Thank you very much. + +Scott Carruthers: it's + +Andrey Arapov: Thank you. + +Meeting ended after 00:41:46 👋 +