meeting notes Sig-support #50

sig-support/meetings/050-2025-11-26.md

@@ -0,0 +1,270 @@
+# Akash Network - Support Special Interest Group (SIG) - Meeting #50
+
+## Agenda
+
+* Review of issues in the support repository and triage status
+* Discussion on recent **console issues** and post-upgrade behavior
+* Community reports related to **shell access and console reliability**
+* Updates on **infrastructure roadmap**, including bare metal and VM support
+* Provider-related issues and upcoming fixes
+* Open discussion on additional support and ecosystem topics
+
+## Meeting Details
+
+- Date: Wednesday, November 26, 2025
+- Time: 07:00 GMT-7
+- [Recording](https://lwi7ysyeqad4sp5wfluprrnokifc2y57wpngrrzajvmvcohpylwa.arweave.net/XZH8SwSAB8k_tiro-MWuUgotY7-z2mjHIE1ZUTjvwuw)
+- [Transcript](#transcript)
+
+## Participants
+
+* Scott Carruthers
+* Rodri R
+* Benjamin B
+* Zeke Ezagui
+* Fenil Modi
+* Kenny R
+
+---
+
+## Meeting Notes
+
+* **Opening Remarks**
+
+  * Scott Carruthers opened the meeting and confirmed he would facilitate due to Tyler being unavailable.
+  * Noted that the meeting was auto-recorded.
+  * Explained that the session would likely be brief due to the absence of pending triage items.
+
+---
+
+* **Triage Overview (Scott Carruthers)**
+
+  * Scott reported that there were **no issues in the “Awaiting Triage” state** at the time of the meeting.
+  * Confirmed that recent issues had already been handled and assigned asynchronously as they came in.
+  * Stated that no triage actions or assignments were required during this call.
+
+---
+
+* **Console Issues and Network Upgrade Impact**
+
+  * **Rodri R** raised concerns about recent console issues, including the inability to close deployments prior to the network upgrade.
+  * Rodri noted community reports where users could not see deployments or account balances in the console.
+  * Rodri also mentioned that the console continued to show **maintenance** and **read-only mode banners** after the upgrade.
+  * **Scott Carruthers** explained that the recent network upgrade was intended to fix the deployment closure issue.
+  * Scott clarified that lingering banners were due to **backend work still being completed** by the console team.
+  * Scott stated that backend work was expected to conclude shortly and that banners should be removed once complete.
+
+---
+
+* **Shell Console Access Issues**
+
+  * **Benjamin B** reported an increase in community complaints related to **shell console access**, primarily via the web console.
+  * Benjamin noted uncertainty over whether the issue originated from the console interface or the underlying shell service.
+  * **Scott Carruthers** shared that he had not experienced shell access issues, noting his heavy reliance on the CLI.
+  * Scott confirmed that **CLI-based shell access appeared stable** and unaffected.
+  * Scott committed to tracking the issue and ensuring the console team is aware of the reports.
+  * **Rodri R** confirmed that CLI usage remained stable for him and reiterated that issues seemed isolated to the console.
+  * Scott stated that shell access issues were unlikely to be directly related to the recent network upgrade but requested continued feedback post-upgrade.
+
+---
+
+* **Trials Status and Console Updates**
+
+  * Scott provided an update on **trials being temporarily disabled** due to account conversion work by the console team.
+  * He noted that trials were expected to be re-enabled shortly.
+  * Scott stated that announcements would be made via Discord once trials were fully restored.
+  * **Rodri R** confirmed during the meeting that deployment closure was now functioning correctly.
+  * Rodri reported seeing multiple banners in the console, alternating between maintenance mode and blockchain unavailable.
+  * Scott acknowledged the observation and advised waiting for the official console team announcement.
+
+---
+
+* **Infrastructure Roadmap: Bare Metal and VM Support**
+
+  * Scott introduced ongoing work on **bare metal support** for the Akash network.
+  * Explained that customer demand, particularly for **AI model training**, is driving the need for full host-level control.
+  * Scott shared that a viable platform for delivering bare metal support has already been identified.
+  * Noted that the initiative is well into the **proof-of-concept stage**.
+  * Stated that bare metal support is expected to be delivered first, followed by **traditional VM support**.
+  * Indicated that delivery is anticipated within the next quarter, with potential earlier availability.
+  * Highlighted bare metal and VM support as significant enhancements to the Akash ecosystem.
+
+---
+
+* **Provider Issue: Non-Bid Transactions Impacting Bidding**
+
+  * Scott raised awareness of a known provider-side issue affecting bid submission.
+  * Explained that if a provider performed **non-bid transactions** such as withdrawing funds, bids would stop being written on-chain.
+  * Clarified that provider logs would still show bidding activity even though bids were not recorded on the blockchain.
+  * Noted that providers had to restart their provider pods to restore normal operation.
+  * Scott confirmed that the issue has now been fixed.
+  * Announced that a **new provider release** would be published shortly.
+  * Advised providers to upgrade once the release is announced.
+
+---
+
+* **Open Discussion and Closing Remarks**
+
+  * Scott opened the floor for additional questions or discussion topics.
+  * No additional issues were raised by participants.
+  * Scott noted that further provider-specific discussion would continue in the upcoming **SIG Providers call**.
+  * Confirmed that the meeting transcript would be reviewed and shared.
+  * Thanked all participants and formally closed the meeting.
+
+---
+
+## Action Items
+
+* **Scott Carruthers**
+
+  * Coordinate with the console team to ensure backend work completion and banner removal.
+  * Track community-reported shell console access issues and escalate as needed.
+  * Announce provider release fixing non-bid transaction bidding issues.
+
+* **Console Team**
+
+  * Complete backend account conversion work.
+  * Re-enable trials and remove maintenance and read-only banners.
+  * Communicate console status updates via Discord.
+
+* **Providers**
+
+  * Upgrade to the upcoming provider release addressing the non-bid transaction issue.
+
+* **Community Members**
+
+  * Report any ongoing console or shell access issues post-upgrade.
+  * Monitor Discord announcements for console and trial updates.
+
+---
+
+## **Transcript**
+
+Zeke Ezagui: Yep.
+
+Scott Carruthers: Yeah, good morning.
+
+Scott Carruthers: Maybe this will be even quicker if it's just you and me. I guess we go through the motions for the recording, but that would be very quick.
+
+Scott Carruthers: Hey, Rodri.
+
+Rodri R: Hey guys, how are you doing?
+
+Scott Carruthers: Good. How are you?
+
+Rodri R: All good as well.
+
+Scott Carruthers: I think I might just give it another minute and we'll get started. Tyler couldn't join this morning, so I'm just going to run the meeting. And I actually looked in GitHub earlier and there's no awaiting triage issues. So, I'll go through maybe one or two issues that I find interesting. one of them I'll probably just make a brief mention of and then we'll cover up more in providers.
+
+Scott Carruthers: but yeah, it's probably going to be pretty quick since there's no triaging to do and Tyler's not here. probably make it rather quick.  You guys have anything you want to  There we go.
+
+Rodri R: Got it. Yeah. there has been a few kind of issues on the console and people not being able to close deployments and me as well. I haven't been able to close the deployment for a while. not sure…
+
+Rodri R: if it's fixed now. I haven't tried it after the upgrade, but I will end up in today.
+
+Scott Carruthers: Yeah, that was the whole point of the upgrade.
+
+Rodri R: Yeah. Yeah. …
+
+Scott Carruthers: It was to rectify that issue. So, there obviously have been a decent amount of issues with console recently with trials and the inability to close deployments. but yeah, that should all be rectified now.
+
+Rodri R: and also some people were saying that their funds were not appearing and that some were appearing but they couldn't see their deployments and…
+
+Rodri R: also cons console was still showing a maintenance and read only mode even after the upgrade.
+
+Scott Carruthers: it was showing that after the upgrade…
+
+Scott Carruthers: because of some backend work that the console team needed to complete. I believe it was supposed to be completed at the end of yesterday. but I haven't got a recent update by the console team. But yeah, it should either come out of u maintenance mode and the associated banner. if it hasn't already, it should very soon. I want to start the recording before we get into any more discussions. because we should probably …
+
+Rodri R: I thought you knew had already done
+
+Scott Carruthers: Yeah, problem. so I could briefly mention some of that stuff that we just reviewed with the recording.
+
+Scott Carruthers: So, if there's not any objections, I'll start the recording. Ben, did you want to ask a question before we started the recording or…
+
+Zeke Ezagui: It's being recorded.
+
+Scott Carruthers: good to start?
+
+Benjamin B: No, we can take it after we start recording.
+
+Zeke Ezagui: Scott, I think Tai might have started auto recording or something.
+
+Scott Carruthers: Okay. Did you never mind then. we recorded all that. So, good console discussion.
+
+Rodri R: like things
+
+Scott Carruthers: Yeah. Yeah. I didn't realize that this meeting started auto recording, so that's fine. Anyways, go ahead, Ben. Did you have a summit
+
+Benjamin B: Yeah, the one thing I've noticed besides the console issues that I think have been fixed for the most part is that a lot of users have reported issues with the shell console. And I wrote something in the Vanguard channel and…
+
+
+### 00:05:00
+
+Benjamin B: it doesn't seem like we know about what the current issue is, but it's becoming more and more frequent.
+
+Scott Carruthers: And are you talking about shell access from the CLI or…
+
+Scott Carruthers: console or both? Yeah.
+
+Benjamin B: Most people don't use CLI, I think. So it's mostly reports from console.
+
+Scott Carruthers: Okay.
+
+Benjamin B: I'm not sure if it's related to console or if it's related to the shell console overall.
+
+Scott Carruthers: Yeah, I haven't had any of those. So, I'm kind of the opposite of a lot of the community. I heavily utilize the CLI and I usually use console when I need to test something specifically. so I haven't had any access with shell access via the CLI, but yeah, we'll definitely track that if it's been brought up in Discord and make sure the console team is aware. Go ahead, Ry. Okay.
+
+Rodri R: Yeah, on CLI I haven't found any issues only it's only on console like I said before and like Ben says several people as well are mentioning this issue issues not issues just  Take.
+
+Scott Carruthers: So, I wouldn't necessarily think this would be Yes.
+
+Scott Carruthers: I wouldn't necessarily think that this would be related to the upgrade, although we would have to know the nature of the issue to say that definitively, but I wouldn't have expected that to be directly impacted by the recent network upgrade. but let us know if you're still seeing the issues post upgrade. so as I mentioned, I think most were already on the call.  Tyra wasn't able to join us today. So, I'm just running the meeting. and the process that we normally go to kick off the call and take things out of awaiting triage is not necessary today. So, if I go into awaiting triage, there's no issues that are currently in that state. So, we don't need to go through that process today.
+
+Scott Carruthers: I think that's mostly just we've been handling and doing assignment of issues as they've come in. So again, no assignment necessary during this call. So we just had some good discussion about console and the need for the recent network upgrade. so hopefully the inability to close deployments and trials will both be resolved. The deployment closure issue be resolved and I know that they were doing some backend work on the console team to do some account conversions to be able to reenable trials. And like I said, I haven't got a most recent update on that.
+
+Scott Carruthers: I thought it trials were going to be reintroduced and active again yesterday. but maybe some delay there but it should definitely be very soon and that banner should disappear from console when that's so we should put out announcements and insiders of the discord channel so that you guys are aware of it once trials are enabled again but you should also be able to see it via that indicator with the banner.  Go ahead. Right. Right. Okay.
+
+Rodri R: So yeah I just open the console right now. I can confirm that the deployment closed as it should. It disappeared. but I do see the banner still and it changes from blockchain unavailable to the console being on maintenance mode banners.
+
+Rodri R: Two banners. but yeah, we'll wait on your announcement or console team's announcement for that.
+
+Scott Carruthers: Okay.
+
+Scott Carruthers: Yeah, I would have been very surprised if you were so unable to close deployments because as I said that that issue has definitely been rectified. So, sounds like it may be taking a little bit of time for trials to become enabled again. that's the current banner, but yeah, hopefully the console team will be we're complete in removing that very shortly. since there's no triage that needs to be done during this meeting, I was just going to call attention to a couple of issues that we can see on the board.
+
+Scott Carruthers: and Zeke, I don't know if you have anything else that you want to touch base on or obviously, all those from the community. If there's any other issues that I'm not going to cover right now and are of interest, feel free to call those out when I'm done. but again, basically just a couple of issues that have come up that I think are some interest.  So bare metal support on Akos this is something that I've had some involvement on but it's primarily being worked by other core team devs. we've see a lot of demand and have talked with some customers the Louisville control over our host for running AI model training.
+
+
+### 00:10:00
+
+Scott Carruthers: the bare metal support will unlock certain customers that need full control over a host. so we have the initiatives to introduce both bare metal support and VMs on Akos in the short term. so right now based on some customer conversations there's a focus on bare metal support. and we already have what we believe is the platform that we'll use to deliver that feature. so this is one of the predominant issues by our backend devs right now and we're pretty far down the proof of concept. so hopefully we'll be able to add bare metal support on the AOS network quickly followed by traditional VM support. don't quote me on the time frame, but hopefully we'd be able to over both of those in the next month or two. definitely in the next quarter.
+
+Scott Carruthers: So I think bare model support and VMs in combination should be exciting introductions to the chain. the other thing that I wanted to briefly touch on and I won't go into this too much because we have a six provider call coming up and I can get into it a little bit further then but this issue that I opened now some time ago I'm not pointing to the right message where it
+
+Scott Carruthers: I'm just going to Why am I not seeing the issue? I won't even point to the issue, but just another update. I feel like I'm staring at it and I'm just missing it. but there's a known issue in the provider codebase right now that if there's a transaction occurs from a provider account that's essentially non bid activity. so an example would be if a provider wants to withdraw funds from their account. over a couple month period of time, they've Zeke said it was just closed 10 minutes ago. So, that makes sense.
+
+Scott Carruthers: I saw it on the board earlier, but I just want to make sure that the community is aware of this issue and that we're about to publish a release that providers can upgrade to resolve an issue that if there was any non-bid activity from the provider account the provider would stop writing bids to the blockchain and you'd have to restart the provider pod to recover.  And I was going through an example of a non-bid activity transaction that may have occurred would be after a month or two per a month to two month period of time. A provider has acred profits and they're just trying to do a traditional transaction bank send to another account. They're essentially withdrawing their profits.
+
+Scott Carruthers: if you did a TX bank send from that account, again you would see the provider bid and provider bid log and provider logs, but the bid actually wouldn't be written to the blockchain and you would have to restart the provider to recover. So, we now have rectified that issue. so we'll announce a release that all providers can upgrade to rectify that issue shortly.  So off the top of my head, those are the main things I wanted to make people aware of that we have bare metal and support eminent on the Akash network. and also if there's any providers on the call that were impacted by that issue that I just reviewed want to know that we've made progress there. I know we'll be rectified shortly on the network as well.
+
+Scott Carruthers: any questions about the issues that I just reviewed or anything else on the board that hasn't come up yet that anyone would like to discuss. and I just give one last chance and we can keep it kind of short today outside of issues. So we had some interesting discussions on council and its status earlier in the call. we've now gone through some issues. I've opened it up if anyone had any discussion items that haven't come up. how about it just in general? even outside of issues, any other topics of interest that anyone would like to discuss? if there's nothing else, we'll keep it short today. This call was recorded.
+
+
+### 00:15:00
+
+Scott Carruthers: So Tyra will go through the transcriptions and post them in the related repo. but yeah with that if there's nothing else we'll end shortly. All right. Thanks for everyone for joining and obviously if you have any questions after the call ends how to reach us. So reach out if anything comes up. Thanks everyone.
+
+Benjamin B: Hey girl.
+
+Rodri R: back to you
+
+Zeke Ezagui: Thanks.
+
+Scott Carruthers: 
+
+Scott Carruthers: Talk soon. Hey.
+
+
+### Meeting ended after 00:16:13 👋
+