Siddhartha Ghosh: know this meeting is being recorded and, as always, the recording will be available from the meeting page in which I will share in the chat shortly.
Siddhartha Ghosh: So today's agenda.
Siddhartha Ghosh: Is.
Siddhartha Ghosh: boom first make which city words about the last outage scatters and if you have any feedback we welcome.
Siddhartha Ghosh: And then there will be a short presentation right john blast on chain and casper cross good evening with the PBS upgrade now it is possible to schedule jobs from cheyenne to casper and vice versa, but yet not quite there some configuration changes needs to be in place.
Siddhartha Ghosh: But john who is.
Siddhartha Ghosh: configuring an architect in the whole solution for you.
Siddhartha Ghosh: will take this chance to introduce you to this nice feature and then.
Siddhartha Ghosh: Our director who.
Siddhartha Ghosh: did give a presentation about today, Joe at that point, it was NWS see three deployment during early.
Siddhartha Ghosh: February, I think it was the first enhance so to six months now, so wherever we are our our about our readiness and things like that it can say keywords and, since in.
Siddhartha Ghosh: Dealing gum at least three to four in house for last three to four and I have been hearing this questions about.
Siddhartha Ghosh: Allocation the status time lines and things like that I thought it would be nice for the article is our deputy director and also it's the allocation process.
Siddhartha Ghosh: will see if he works and he will get a chance to ask questions directly to him, and then I will open the floor for.
Siddhartha Ghosh: Discussions at any point in time to fill a dirty dozen patients or otherwise the jack Bingo or unmute yourself to ask questions so with that let's let me hand over to make.
Mick Coady: Thanks it i'll share my screen here to second.
Mick Coady: city.
Mick Coady: Okay, can you see my screen okay.
Mick Coady: Yes, okay so us likes it mentioned, he asked me to just say a few words about.
Mick Coady: The outage that we went through last week.
Mick Coady: As you know, the from all the communications that have taken place in the preceding months the outage was scheduled to start a week ago, yet yesterday on Monday July 26 and run through yesterday.
Mick Coady: In actuality we were able to finish, that the outage ran from last Monday, through Friday evening.
Mick Coady: Friday late Friday night actually.
Mick Coady: In all this was in preparation for direct getting ready for a great show in the power requirements that that new system is going to require at nw sc.
Mick Coady: So let's see.
Mick Coady: So.
Mick Coady: As the communications mentioned, we plan to return the ups back systems.
Mick Coady: Hopefully, by last Wednesday on the 28th that would have been casper glade campaign storage and the rest that you see here.
Mick Coady: In actually we were able, we weren't able to restore it until Friday at early afternoon because of faulty ups system that was discovered when we were bringing the systems backup turns out that there was a faulty switch on the back of one of the systems that.
Mick Coady: I didn't see pictures I don't have pictures for you, but apparently it's big enough, it requires a forklift.
Mick Coady: To move in and out so it was a little trickier than what we had hoped, but the the contractors in the staff at nw sc were able to finally get that repaired on Friday morning last week.
Mick Coady: On the, on the other side of the coin cheyenne was not planned to be restored until yesterday, but once the all the power work was.
Mick Coady: was completed by the contractors in sizzle staff, we were able to begin the final phases of the checkout and of cat of showing in in we were able to return the entire system last Friday night about two and a half, three days ahead of schedule.
Mick Coady: So I want to give a little bit of recognition here so as you might expect, there were dozens literally dozens of people who are involved from sizzle in the effort, but I want to.
Mick Coady: give special recognition to Michael coucher head of the is gc group in showing in, who was the overall project lead and then from the boulder.
Mick Coady: Teams Jared baker and Emma shop were responsible for coordinating internally the all the efforts that were going on.
Mick Coady: Between the htc systems engineer the consulting groups, as well as the staff up at NWS see I think they just did a terrific job and helped make my job, a lot easier, with the communications to users throughout the week.
Mick Coady: So.
Mick Coady: Just wrap this up sizzles perspective is the outage went pretty much as well as possible, with the exception of the faulty ups switch that.
Mick Coady: We ran into.
Mick Coady: We we were thrilled with how well things went, we would like to hear what your perspective was on that.
Mick Coady: If you don't feel comfortable or want to mention it right now in this meeting, please feel free to reach out to myself or sit and let us know and we'll take that back we'd like to hear you know.
Mick Coady: Particularly, but I would about the communications and whether that was adequate too much or how that fit and what the impact was maybe from the delay or from getting the systems back earlier to you so with that i'm done.
Mick Coady: sit back to you okay.
Siddhartha Ghosh: All right, anybody would like to give your feedback.
Siddhartha Ghosh: You must become unmute yourself and speak or provide in the chat as well.
Mick Coady: Okay i'm gonna stop sharing said okay.
Siddhartha Ghosh: Okay john it's your turn now.
John Blaas: just give me a second to share my screen.
John Blaas: You all see my screen, yes, all right great alright so i'm going to take a few moments to talk a little bit about pure scheduling, which is a feature that we hope to release to users, within the next two weeks.
John Blaas: So cheyenne well beginning three new queues which will furnish jobs over to casper So these are cues that you won't be able to submit to directly and your jobs will then end up on the.
John Blaas: casper complex within one of three q's at the moment that I have planned the htc Q.
John Blaas: The visualization queue which targets the tp 100 notes and the gpu gpu queue which targets the V 100 nodes of the four and a blade configurations.
John Blaas: casper will be gaining the following cube which will furnish jobs over to cheyenne which will just be simply named cheyenne to keep it simple.
John Blaas: There is a possibility of adding more tools in the future as needed, but these will be the ones that will start off with.
John Blaas: So with these fresh in queues will start to you know, introduce a need for actually specifying resources for job placement.
John Blaas: This hasn't been a huge need for cheyenne because it's a largely homogenous cluster with only cpus but with casper with been experimenting with the putting in place more job resources that the users can request, such as n gpus cpu type and gpu type.
John Blaas: So, with the jobs in casper if you specify a cpu type of cascade like or sky like that will eventually end up on one of the be 100 nodes.
John Blaas: Or you know htc know, but if you would specify a cpu type of broad well and put that job into the casper main queue it would eventually filter down into the cheyenne queue, which would then push the job over to the cheyenne complex.
John Blaas: So you know, the low here, I have some examples of you know, some specifications for the selections of resources that you could give so this first example, give you a gpu gpu job targeted at a be 100 node the second job request would give you a glimpse.
John Blaas: A gpu gpu job on a V 100 know that also has a cascade light cpu type.
John Blaas: And then we also just provide an example of just submitting a job to the htc Q from cheyenne as well, and then also an example of you know, submitting a job from casper to the shy and complex.
John Blaas: So you know just for you know.
John Blaas: A visualization of how this actually looks so you know, on the shy and complex we don't have any kind of routing Q structure, and so you basically just submit directly to you know the queue you want.
John Blaas: This will be no different for the htc this and gpu gpu cues on the cheyenne complex but SEC, you know they will route those jobs, specifically to the corresponding cues on the casper complex.
John Blaas: And then on the casper complex, since we do use the routing Q structure we won't keep the cheyenne complex in the routing Q structure so that.
John Blaas: If you, you know request a job with that cpu type of rodwell it will eventually get into that cheyenne queue.
John Blaas: But we will also allow users to submit directly to that cheyenne Q, just to make it simpler for a start, and then that'll direct directly to the race or Q and we may add more cues as needed.
John Blaas: But that's a quick look at a pure scheduling those coming within two weeks back to you, sir.
Siddhartha Ghosh: Thank you john.
Siddhartha Ghosh: You have any questions for john.
Siddhartha Ghosh: nascar or flashing the chat as it will pretty.
Siddhartha Ghosh: Okay.
Siddhartha Ghosh: Thank you.
Mick Coady: it's Nice.
Dave Hart: Alright sorry, can I ask a question as a Dave.
Dave Hart: yeah i'm on that john So the question I would have is so, how would you how does this process prevent someone will say a large shine allocation from just taking over the casper cluster.
John Blaas: So when their job ends up on the casper cluster they'll get their fair share is determined, based on the casper fair share, not the cheyenne complex fair share.
Dave Hart: So it will be evaluating their casper allocation and they'll say okay perfect that's why I want to double check and make sure people knew that.
Siddhartha Ghosh: Thanks Thank you okay it upon knowledge to done, please.
Irfan Elahi: yeah so our main reason for giving an update is that there has been some change in schedule so that's the kind of main change from the last.
Irfan Elahi: update that I gave so a couple of weeks ago the met with HP and HP confirmed that the chip shortages and backing the supercomputer industry and their ability to manufacture and deliver.
Irfan Elahi: Many of the large systems in us so sb informed us that the must slip the delivery of the Rachel to January 25 2022 in it's a delay of hundred and 16 days or nearly four months.
Irfan Elahi: If you remember the original date for delivery was October 120 21 now.
Irfan Elahi: You know i'm sure most of you i've been keeping up with the trade press and the computer industry and and are aware of the chip shortage that demand for chips.
Irfan Elahi: sort significantly during the pandemic as a lot of people are purchasing laptop gaming consoles.
Irfan Elahi: And spirit and televisions and also many companies where we're buying equipment to mitigate workforce and personal impact of lockdown.
Irfan Elahi: And you know there's big shortage of gaming consoles and also the auto industry, the TELCO industry everybody seeing this impact and the fact that.
Irfan Elahi: We during our contract negotiation discuss the pandemic and HP did make some changes and they added pandemic language to the contract, so this was not.
Irfan Elahi: On I totally unexpected.
Irfan Elahi: The web preparing for it, so what we will do is, we will extend our we are working to extend the maintenance and support for cheyenne for an additional six months so cheyenne original decommission was July 1 we are looking at December 31 2022.
Irfan Elahi: Now this is the date they are willing HP is willing to commit to on the contract, because we have to do a contract modification.
Irfan Elahi: With the new dates, but they are confident at this time that it might be earlier, I know I will provide you a big as I get it so basically.
Irfan Elahi: You know, we are working on the smart sheet schedule and so asd will start anywhere in late April early May and the reason I can't give you the exact date.
Irfan Elahi: Is because when we plugged in all of this hundred and 16 day.
Irfan Elahi: delay its kind of pushing some of the smart sheet wave into me and i'm working with the project manager to figure out why it's padding are adding some extra data so right now it's.
Irfan Elahi: it's kind of smart sheet is saying is may 11 but my working says it should be April 27 so we'll try to reconcile that but hopefully by next and hug I give you an exact date when the is the can start.
Irfan Elahi: Now That said, I also wanted to give you some update on the accomplishments you know, the new three megawatt transformer.
Irfan Elahi: that's required for who has successfully been installed commissioning we left it running for a night and it's working fine.
Irfan Elahi: And Nick mentioned before about ups, we also did work to tie in the ups for rachel's infrastructures and storage system so that's in place.
Irfan Elahi: Also, the ethernet backbone network that is called by for us it was upgraded, so that it can host all of the ethernet connections required for individual compute and storage.
Irfan Elahi: That was very complicated work and i'm glad that gets behind us, we also received the case storage system for deliberate has been installed.
Irfan Elahi: And admin and operate sure their training and skills transfer is underway we've had a number of training sessions and the system application and facility team are on track with respect to all of their work, as well as readiness for receiving a WHO.
Irfan Elahi: they're also one thing i'd like to mention is that there are some fixed up components that are required in the facility.
Irfan Elahi: That are also impacted by this the chip shortage there's a light switch gear that is needed for good, he has intelligent breakers and chips, and it has been delayed.
Irfan Elahi: You know, originally we were expecting a delivery on July 30 now it's looking like July and September 17 and also, there are some mechanical power transfer switch these delays are not going to impact the.
Irfan Elahi: Facility schedule itself per se, but it has forced us to change the order of task and work around, so we will still get our facility ready in time, even if.
Irfan Elahi: The delivery was in first week of October, but it did make it a little bit more complicated in in you know in changing our cast around a little bit in that sense, so with that you know i'll take any questions that anyone may have on the doritos delivery and commissioning.
Siddhartha Ghosh: You notice go ahead.
Rajesh Kumar: And, just a quick question How does that impact cheyenne availability means they can be available for more time now.
Irfan Elahi: That is correct, as I mentioned, we are not going to decommission shine on July 1 2022 as was originally planned.
Irfan Elahi: We are working to extend the maintenance and support, so we can keep shine all the way to the end of December 2022 so that's a six month extension, even though this is less than four month delay, we want to keep cheyenne.
Irfan Elahi: available, we simply can't be without a supercomputer and i'm hoping that there wouldn't be any additional delay, but we are trying to put some buffer in in there, so did that answer your question.
Rajesh Kumar: Yes, and just just a quick follow up on that means How does that impact the nsc and the large allocation projects.
Irfan Elahi: That is a question for Dave Hart.
Siddhartha Ghosh: I guess it will address that in during his presentation or.
Dave Hart: I just thought let's finish with our questions for fun and i'll take questions and such.
Rajesh Kumar: sounds good, thank you.
Siddhartha Ghosh: And anyone else.
Siddhartha Ghosh: Have any questions, but if one.
Siddhartha Ghosh: Is the chat or just pick up.
Siddhartha Ghosh: The day now it's.
Mick Coady: Your time to sit sit just one.
Mick Coady: Quick thing up.
Mick Coady: will be putting.
Mick Coady: publicizing this are announcing this in tomorrow's daily be about the direct shoe delay, so if there's any further questions, hopefully that'll address them.
Siddhartha Ghosh: Thank you.
Dave Hart: and make make sure reach out to me, so we can include some of the some information about allocations as well, or maybe a complimentary item in the daily be so.
Dave Hart: Okay, so yeah so just to start, I was not sure what I was going to say today exactly because it wasn't clear to me, yet when I was gonna be able to say anything about the delay.
Dave Hart: Sorry, Joe so i've been that's part of the reason that that hasn't come up yet is we were making sure all the, the process has been gone through before I made that announcement more publicly so.
Dave Hart: yeah I wasn't even sure it was going to happen today at this meeting, so I don't have anything directly prepared, but obviously with the four month delay we're going to be shifting some of our plans for allocations in the fall.
Dave Hart: In short, the accelerated scientific discovery process will largely stay the same as terms of its deadlines that's primarily because, if I slip asd deadlines for months that lands us in the holiday season.
Dave Hart: Then leading into the Ms and agu season, so I my plan is to have those requests sorted and lined up in advance of the holidays, we may slip it slightly, but that it won't go past the the thanksgiving holiday to sort out the asd projects and reviewed them.
Dave Hart: As far as nsc and Chapman university allocations this fall we're going to make them solely on cheyenne as normal.
Dave Hart: So we'll have more than a year left of cheyenne at that point, so that should be sufficient for making this next round of fall allocations in both cases.
Dave Hart: Given the four month delay and the asd program the spring opportunity in April will be plenty of time to get the the next round of system of allocations on ready and approved for the ratio so.
Dave Hart: Four months works out pretty well in fact it with the allocation cycle so April April meetings and proposals will be reviewed for dirty joke.
Dave Hart: In the normal for the normal chap cycle and the normal nsc cycle, the lab allocations will be set up and so forth between now and then i'll be having conversations with the anchor director and others about exactly how that will how the.
Dave Hart: msc program a particular and the lab allocations will be dealt with and i'm sure the Executive Committee will be engaged in that, but some of that is still tbd.
Dave Hart: More or less that's the basic plan for dealing with allocations schedule i'm happy to field other questions that you might have at this point.
Dave Hart: To answer your question.
Rajesh Kumar: Yes, thank you.
Okay.
Siddhartha Ghosh: So anybody has any questions for.
Siddhartha Ghosh: me or all of us just speak up at this point, or just use the chat window to pleasure investing.
Dave Hart: In a big group for not any questions on.
Dave Hart: That the consulting group grilled me.
Dave Hart: endlessly on that topic so.
Dave Hart: Obviously, one question i've been asking along the way, is making sure that the gpu nodes Andre Joe or man well managed, so that we don't have people unintentionally or intentionally taking over those nodes given there so few of them relative to the larger partition of the.
Dave Hart: amd nodes so that that's something we're watching and monitoring.
Siddhartha Ghosh: The high me ask a question, and so this is regarding, that is, the video being pushed and is the review process is not so will the evaluation committee the little bit lenient on the fact that the readiness part, we will have four more months we did.
Dave Hart: yeah we can certainly take advantage of that, but you know.
Dave Hart: It sort of depends on where they still depends on where they are at the time right we don't want an idea that's just been formulated for new code to be developed, it would still have to be code that's.
Dave Hart: Still in ready to run make science runs or very close to that point with a clear plan for finishing that development effort.
Dave Hart: It even so brand new code may or may not be a good best fit for the asd program given so many uncertainties in that in that vein, with the system itself.
Dave Hart: If you know it should be fine if it runs well on cheyenne but there's still that element of risk we you know the asd program is really aimed at taking science ready codes and giving them that chance to really take over a large scale system it's not the opportunity to try.
Dave Hart: brand new code that that's just been finished and hasn't had a chance to be you know have some.
Dave Hart: Real world testing, you know at us substantial scale.
So.
Siddhartha Ghosh: Actually.
Siddhartha Ghosh: As far as the cpu part is concerned you're absolutely right, but for gpu, as you know, we do not have a platform to test and try at that skill and this is the skin.
Siddhartha Ghosh: Absolutely, and many of us that are working with the users on those projects so.
Dave Hart: I do expect will be a little more leniency on the gpu side.
Dave Hart: Unless some folks there are projects out there that do have gpu code ready and running elsewhere that you know would be.
Dave Hart: have an opportunity here, so will the panel let's see what submitted and we'll look at the what the what's available and if there's room certainly there'll be a little more latitude on the gpu code readiness.
Thank you.
Siddhartha Ghosh: Any more questions from anybody on a related topic before we.
Dave Hart: Go on, easy on me today.
Siddhartha Ghosh: And that's what I.
Dave Hart: Well i'm always available for questions, please don't hesitate to shoot me an email the heart@uconn.edu i'll field your questions, then as well.
Siddhartha Ghosh: Thank you.
Siddhartha Ghosh: Okay, thank you, everybody for your time.
Siddhartha Ghosh: As always, the meeting recording and the proceedings will be there in the meeting page, which by this time flashed in this.
Siddhartha Ghosh: chat window and I will also send you an email with the link with that, let me end this meeting, and thank you for your time.