cloud.
Mick Coady: said, you could make or find a Co host.
Siddhartha Ghosh: I have to make a Co host okay.
Mick Coady: You could.
Irfan Elahi: Or you could just share the screen whatever works.
Siddhartha Ghosh: Let me give you.
Siddhartha Ghosh: Make your The co host and so.
cool.
Siddhartha Ghosh: Okay now you are co host so you know yeah.
Irfan Elahi: So, can you guys see my screen.
Mick Coady: yeah are from wicked.
Yes.
Irfan Elahi: So let me know whenever you're ready said.
Siddhartha Ghosh: I think you should go ahead, since we had Preston time, I would like to have as much time as possible for you.
Irfan Elahi: yeah Thank you everyone and welcome to the kickoff for the encourage PC users group I wanted to share basically some details of the new system now, this is just.
Irfan Elahi: The first of many engagements, we will have so i'm going to try to stay a little bit more at the high level, but we definitely plan to deep dive.
Irfan Elahi: Especially for those user environment and the software stack and some of the internals of the hardware, including interconnect.
Irfan Elahi: So just to give you a little bit his historical perspective we kicked off this project in June of 2018.
Irfan Elahi: And you know I started putting the core team together almost a in June of 2018 and the core team included technical evaluation team business benchmark facility team.
Irfan Elahi: As well as a contract and legal team, but the most important was the science requirement advisory panel.
Irfan Elahi: Because he wanted to get input from our scientist and users are not just looking at how cheyenne was utilized and look at the metrics and data that we collected from cheyenne, but we also wanted to be forward looking so we had.
Irfan Elahi: A number of meetings with the science requirement advisory panel and over 30 plus a use science cases that I was used to form the technical requirement for the new system.
Irfan Elahi: We also engage the technology development divisions and other subject matter experts from within sizzle to get a very.
Irfan Elahi: Total solution, not just from user perspective, but also from facility, as well as how we can meet in the new system going forward, because one of the most important.
Irfan Elahi: Things that s rap and users have been pressing us on is they want more storage, but they also want system that's available and usable so we put a lot of emphasis and i'll talk a little bit more about that in detail on.
Irfan Elahi: So the main objective of NWS he was to provide hype, a highly productive beta and intensive supercomputer for in car.
Irfan Elahi: and users and application, but also tried to see if we can build on the success of cheyenne especially what we learn from cheyenne and also invest in s access game and gpu architecture and also cloud computing technologies, because there is a lot of.
Irfan Elahi: Proof that these could become prevalent.
Irfan Elahi: technologies in the next five years now also we wanted to make sure that we could enable our application to optimization and refactoring.
Irfan Elahi: I can make use of these new technologies and also one of the guidance from pseudo Council was to make sure that the users remain productive throughout the transition to nw se and beyond, and that's why we do not plan to decommission cheyenne until the end of 2022.
Irfan Elahi: So, after consultation and user requirements we I am looking at our budget, we decided that the new sbc systems system should be at least three times.
Irfan Elahi: The cases same capability of cheyenne and working with the benchmark team we created a suite of bird benchmarks and that's the benchmark, you know.
Irfan Elahi: running on all of cheyenne would get us one season or cheyenne sustained equals performance.
Irfan Elahi: So we basically provided that sweet as well as guidance to the vendors and said, well, you must at least get minimum of three cheyenne and also we wanted 60 petabyte usable file system.
Irfan Elahi: Another guidance and the rfp was that the split should be at 20 and the split is.
Irfan Elahi: Not by budget or the node count, but on the cce APP itself so MC 80% of the seaside a cpu.
Irfan Elahi: And 20% is gpu another thing that we've been kind of getting requests from not just other labs but also management and nsf is cloud bursting capability, so we did invest.
Irfan Elahi: In cloud bursting capability in terms of deliverables it did not cost us money, but the new scheduler will have a gui interface that will allow you to launch instances on azure and aws.
Irfan Elahi: Primary will be targeting EMS, and some of these special campaigns, because it does require additional funding, so we do not, you know so will not provide the funding.
Irfan Elahi: If people have cloud credits are you car and car has that funded setup then users will be able to burst out the scheduler would be configured to do that.
Irfan Elahi: And also, I won't go into too much details, but basically our plan is to start the accelerated science discovery in January 1 week of January and then maybe after three months we'll open the system to the rest of the users, we also had options to grow, the system if needed.
Irfan Elahi: So one of the cool aspect of this pyramid which, as far as I know, we have never done in the past.
Irfan Elahi: Is after the request for information or the Roi phase we had co design workshops, where we invited the vendor and technology providers to sit down with us for the.
Irfan Elahi: whole day and basically kind of working side by side, you know, in a collaborative and consultative approach create solution and innovation and improvement.
Irfan Elahi: On the rfp responses so basically the Co design outcome was we configured and architected a system that we are able to provide real time input and feedback Otherwise, what happens is that.
Irfan Elahi: vendors respond to our rfp we look at the rfp and we don't like certain things, and then we asked them to go for an F rp and that can become problematic, but what we did here was we basically in co design workshop were able to iron out a lot of those.
Irfan Elahi: issues and that made the rfp response more.
Irfan Elahi: targeted towards our requirement, it was made much closer to our requirement now, we did not have a smooth sailing with this rfp there were a ton of constraints and issues.
Irfan Elahi: One of the biggest challenges that I saw was we have limited domestic SPC vendor to a high performance computing xp acquired SGI and Korea so basically we were you know.
Irfan Elahi: down to just IBM HP and Dell on IBM is not cost competitive they were extremely expensive, you know and the problem with.
Irfan Elahi: Dell was that they do not provide quotes for.
Irfan Elahi: A product that are not ready to ship, so we were asking them to quote a product that would be another 18 months away from delivery date so it was challenging for down.
Irfan Elahi: Because our federal facility takes time, so we, you know that would be pretty much did not been.
Irfan Elahi: The other thing is that small vendors like penguin computing and others like Aspen systems, they do not have.
Irfan Elahi: The resources to run our benchmark because it's very resource intensive so they just sit out these kind of big pyramid and that's why i'm working with tdd and others to see if we can.
Irfan Elahi: simplify our benchmarks, if you look at what oakridge is doing they're trying to take a lot of the workflow and put it out there on the.
Irfan Elahi: You know, make it publicly available and make it more closer to synthetic benchmark, so we need to also look at that if you want to invite some of the smaller.
Irfan Elahi: vendors and my apologies i'm going fast, because I don't have too much time.
Irfan Elahi: Also code 19 added significant challenges, most of the vendors have problem getting price warranty and they also had issues with getting resources to do some of the rfp one and also there is constraint upon us that we have to buy America.
Irfan Elahi: Because of the by America academic boo or a task and a Lenovo we're very much interested, but we could not.
Irfan Elahi: You know, invite them because they are not American companies, so we are limited by that.
Irfan Elahi: So what are we getting we are basically getting a cray system used to be called shasta now it's crazy ex.
Irfan Elahi: We are the peak will be just about 20 better flop 60 petabyte usable it's about 82 petabyte of raw storage.
Irfan Elahi: And here's The good thing the SES up that we are getting is actually 3.51 so it's more than what we had asked for and, as you can see 2.743 is on traditional cpu.
Irfan Elahi: Which is basically an empty Milan and then the point, six, seven, you know it's almost about 70% of shine in terms of CSS is on gpu and all of these gpus are the nvidia redstone a 110 Sir or graphic processing unit.
Irfan Elahi: gpus now, we do have confidence and create dream so we're really excited about that they have a very large install base oakridge argon nurse El capitan at Lawrence livermore and even more recently.
Irfan Elahi: Sorry El capitan at Lawrence livermore and more recently Los Alamos also worked with the cray.
Irfan Elahi: So that they have a pretty large install base Another thing we are doing differently with this one is that there would be an onsite.
Irfan Elahi: person so that will reduce our time to resolution for problems.
Irfan Elahi: And the scheduler and the system is going to be PBS so you will not have to change your job submission script, we are also planning to move all of casper from learn to PBS so going forward, we will have a single our scheduler.
Irfan Elahi: To basically i've you know i'm not going to spend too much time on this slide this is kind of what I already discussed, I just wanted to share that graphics of what the racks will look like and I will share the slides with the set so he can put it out there.
Irfan Elahi: Some more details, so there are about 2570 nodes total are 2488 our compute while at to our gpu the cpu nodes will have our dual socket 64 core 2.6 gigahertz Milan.
Irfan Elahi: Our cpu these are seven X one too, but I appreciate, if you don't share that out because amd has asked us not to these are not yet publicly.
Irfan Elahi: available, so this is under nda please don't share some of these details on gpu, as I mentioned, is.
Irfan Elahi: You know we'll have a single socket SMB 64 for same host cpu as the as the other one, but it also has four nvidia redstone.
Irfan Elahi: gpus, these are all SMS for which means this all will be envy link interconnected for gpu direct and also all of these will have 40 giga Bytes of high bandwidth memory, these are all hdmi to.
Irfan Elahi: memory, the interconnect on this is the crease slingshot it's about 200 gigabit per second.
Irfan Elahi: per direction and the latency is a you know 1.7 2.6 microseconds so it's pretty comparable to my Linux.
Irfan Elahi: Now, it will also have a login nodes six login nodes are just pure cpus like two of them are more you know have cpus but also have gpu is more for compile ation and on interactive work.
Irfan Elahi: The operating system on the new system is a variant or a micro community of susie Linux it's crazy Linux environment, as I mentioned that the schedule will be PBS.
Irfan Elahi: But we do plan to support all kind of container especially any container that has supports the open container initiative standard will be supported on this machine, especially docker and singularity.
Irfan Elahi: For programming environment, you have the CV or the programming environment that will support open MP 4.5 as well as mpi version 3.1.
Irfan Elahi: The create performance analysis tool will be included, and then we also have the nvidia compiler environment, just to make you aware of the.
Irfan Elahi: nvidia compiler environment that formerly this was called PGI now it's more of freely downloadable.
Irfan Elahi: compiler, but we will have full support on this which the free version does not so we do have the PCs, I think, is what nvidia calls it.
Irfan Elahi: So we will be able to escalate our problems, and then we'll have 50 seats of Intel Palo studio as well, and then the file system, as you will notice is different, its luster and not GPS.
Irfan Elahi: Now I will encourage you to discuss with Sid and others and i'll also be attending these ad hoc meetings, if you have requirement for other software stack but also reach out to Brian Wendy Wendy who's the lead from the application side for NWS see.
Irfan Elahi: For storage I mentioned 60 petabyte usable if you can expand it or develop if we have the money in the future, it has about 300 giga Bytes per second aggregate bandwidth, but we will also connected to glade and our campaign storage and from there, it will have 200 gigabyte you know.
Irfan Elahi: We plan to use a flash for a Meta data and, as I mentioned before, it will have luster file system and not GPS.
Irfan Elahi: So I just wanted to give you a little bit quick background.
Irfan Elahi: On the on the different type of nodes so the correct cabinet which they call a lump us.
Irfan Elahi: basically consists of eight chassis and each of these chassis can support it compute blades which are in the front and eight network leads, which are installed in the rear of the cabinet.
Irfan Elahi: Now the good thing about these kindnesses there is no black plane in the chassis to provide network connection between the compute blades.
Irfan Elahi: it's actually auto no connections between the network blades and the compute blades that act provide the network connections, so this is different than cheyenne.
Irfan Elahi: And the benefit of this that without having a bad plan it means nothing is hardwired so we can simply swap out the nodes are you know to.
Irfan Elahi: Change the network blades and so you know you're not tied to to you know, a one type of cpu and lot of.
Irfan Elahi: vendors what they're doing is they say that these cabinets can have a life of 10 years, so instead of completely taking everything off the floor and replacing it, you can simply replace the computer late again that's something to see.
Irfan Elahi: Now, each of these are cabinets cab home 64 compute blades and 64 switch plates.
Irfan Elahi: And, and you can have 256 cpu only nodes are 128 gpu only known.
Irfan Elahi: The cpu nodes are you know will have eight memory channels that are fully populated with one dimple channel all of these are dual rank memory, especially because we want to provide the optimal memory performance.
Irfan Elahi: And we are giving you guys a sufficient that ability of 256 gigabyte of memory.
Irfan Elahi: per node for the cpu for the gpu nodes it will have 512 instead of 256 again all memory child are fully populated with DDR for 3200 Dems.
Irfan Elahi: You know, an e gpu is configured, as I mentioned before, single socket for nvidia gpus are not the difference between a cpu and gpu node in terms of net network is that gpu nodes would have for network injections get to that will give you almost 800 gigabits per second.
Irfan Elahi: So just to touch on luster the power five file system will not be dp fs it will be luster and it's going to use cluster store he 1000 our hardware, which is.
Irfan Elahi: crazy and it will be concerning do into a single luster file system called scratch and it will have a single namespace that is posits compliant and you can access it through a single mount point we.
Irfan Elahi: expect that the file system scrubbing policy will be about six months or even more so, you will see a different scratch scrubbing policy, not the 90 day you have right now.
Irfan Elahi: So very quickly touching on something that is you know rasen basically reliability availability serviceability usability and manually and.
Irfan Elahi: said, this is not the the south Indian Curry rossum that you and I are aware of this is a different.
Irfan Elahi: So basically what we are trying to do is make sure that electrical and mechanical components are ups, we want to make sure that the story and file system has a higher availability, which is 99%.
Irfan Elahi: But we are also trying to take advantage of features and functions that Christmas and provides for higher rassam.
Irfan Elahi: You know, and this is something we've been very, very focused on while working to architect, the system with the cray.
Irfan Elahi: This is an interesting slide that you might want to look at in your free time but it gives you the history of supercomputing at n car from 1960s to 2025 because that's when you know we plan to keep operating the new system.
Irfan Elahi: it's I know it's pretty small, but you can magnify it and I, this is mostly in your free time you can look at it, so this is a good slide that I wanted to share with you, it gives you a little bit.
Irfan Elahi: idea of how much flops the new system will be so you will see with cheyenne and the new system, we will be above 25 pepper flops.
Irfan Elahi: And then, this is similar but it gives you the storage capacity, if you look at campaigns to read with the 66 petabyte.
Irfan Elahi: glade which is 38 by by object store, which is five and then you look at the 60 petabyte that will come, so we have a lot of storage once the system will get installed, obviously we will start decommissioning some of the older.
Irfan Elahi: storage in blade this kind of just gives you the maximum power consumption so you'll see will be reaching close to 4.5 megawatt megawatts a year.
Irfan Elahi: You know, if I may be mistaken, but I think cheyenne city is four or five megawatt total but anyway that's just off the top of my head don't hold me to it.
Irfan Elahi: This is a slide that I really like because it gives you the sustained mega flop per second per Watt if you compare cheyenne which has about 73 sustain mega flop or walk while the new system will have hundred and 71 So you can see that the new system is much more energy efficient.
Irfan Elahi: This is my final slide I couldn't go, not the final there's one more, but this kind of gives you when we will be accepting this machine, I hope, to complete the acceptance test in December.
Irfan Elahi: Open it for an accelerated science discovery in January and then in late March early April, it will be open to all the users.
Irfan Elahi: I also wanted to touch on some things we are doing on casper you will be also hearing about this at the Web talk tomorrow, but we are adding we added recently a new gpu nodes.
Irfan Elahi: we've also moved RD off of to their own dedicated node lot of users were complaining about that that idea was kind of had longer Q where wall clocks so they were kind of.
Irfan Elahi: causing issues, but we also adding 64 high throughput computing knowledge, these will all have SDR you know trainer at four gigabyte memory.
Irfan Elahi: They will be new dedicated login nodes on casper this was another user requirement and then, as I mentioned, learn will be migrated to PBS so you will have a common scheduler across the board.
Irfan Elahi: So that kind of ends my slides i'm four minutes overtime any questions, I will be happy to take that.
Brian Vanderwende: So are found there were a couple questions that popped up into the chat one from Jim edwards is will gpu direct storage be available.
Irfan Elahi: So that's something we have not.
Irfan Elahi: looked at, but I would like to get some our requirement from Jim you know what does.
Irfan Elahi: He mean by gpu direct storage and and and how will he take advantage of it and absolutely we can look into it.
Irfan Elahi: So I need that you know so sit, maybe, can you work with the and Brian both of you are, because these are two different you know and get has a requirement and then definitely I think that's a very interesting question.
Brian Vanderwende: yeah we'll follow up with Jim that sounds good.
Mick Coady: Another question or fun from ethan gutman was how will cloud bursts interact with glade or other storage.
Irfan Elahi: So that's actually pretty challenging because one of the challenges of cloud is that you know.
Irfan Elahi: You can't run in the cloud, you can firing and then just run something unless you have the data, you need to run your jobs, so you know, there are very.
Irfan Elahi: Few different options, you know one is that you basically have some kind of a data common data Center environment that's very readily accessible it's on the edge of cloud.
Irfan Elahi: You know, and there are ways to do that, the other is that you basically get some persistent storage on the cloud where you can pre staged your data before you run it.
Irfan Elahi: that's really costly, because you have to continuously pay for that persistent cloud the other option that we can do is you basically upload your data.
Irfan Elahi: Fire you know after you have your instance ready run your job when you're done you download your data back, but then that has its own challenges, because you have to pay for moving data to and from the cloud.
Irfan Elahi: You know so right now, all of these things are being looked at, I think.
Irfan Elahi: You know, over the next few months and couple of years, we need to find a happy medium that works for us and is cost effective.
Irfan Elahi: But as I said, you know, there are costs involved in all of these my my apologies, I have to jump into another abr meeting I will be more than happy to come back at the next meeting and answer additional questions.
Irfan Elahi: said is that Okay, if I drop off.
Mick Coady: Okay, I think we can hit, I think we can handle it from your phone Thank you.
Irfan Elahi: very much.
Siddhartha Ghosh: Thank you, it fun.
Mick Coady: So one of the questions to the group was Dave Gill asked a good question about the nda parts that are fine alluded to, I believe he was referring to the clock speeds on the chips.
Mick Coady: At that point Dave.
Siddhartha Ghosh: Any more questions make or Brian that we should be discussing.
Brian Vanderwende: Yes, good.
Mick Coady: Go ahead, Brian go ahead, sorry.
Brian Vanderwende: yeah so john Klein asked about the luster file system and if it's going to be mounted on casper.
Brian Vanderwende: And I see bill Anderson is in this meeting, so he could speak to this as well, but I believe that the the luster is, it should be mounted on to casper as far as the GPS storage being mounted on wc three I think the current plan is is no.
Brian Vanderwende: But, and specifically i'm talking about the the the the scratch space.
Brian Vanderwende: In glade but again bill can speak more to that if he if he has anything to add.
Bill Anderson: hi Brian yeah some of those plans are still being finalized.
Bill Anderson: Like you said, and I know Jared on this call, he could weigh in as well, but our plan is to mount the scratch from NWS see three onto casper there is discussion about mounting.
Bill Anderson: Some of the campaign's tour and home file system current home file system on to end with three but but no final decision has been made there is my understanding.
Brian Vanderwende: Thanks so.
Mick Coady: I think said I think that's it for questions, so I apologize if I missed, one that someone asked earlier.
Siddhartha Ghosh: Okay yeah that's suddenly we can go back to the questions and all my, as I mentioned in the beginning, the.
Siddhartha Ghosh: Whole session is being recorded any of the issues that we haven't addressed, we can come back to the next meeting, and this will be a monthly meeting, as we have mentioned earlier in our weekly, so let me get on to the this user group structure, what we are expecting so.
Siddhartha Ghosh: Can you see my slide.
rgarcia: Yes.
Siddhartha Ghosh: Okay, so here here is what we had as agenda, we already covered it founds top and then few logistics, I will get to that in a minute I will to group, and this meeting series, the structure of the meetings and communications that we would like to have.
Siddhartha Ghosh: Our current priorities and then, finally, I will hand over to Brian for application team perspective, once we get to that we will know more about it.
Siddhartha Ghosh: So a few logistics all meetings in the series will be recorded well, the only reason we want to do that is to be able to capture all the conversations that will be happening, and as you realize that a lot of people in the meeting.
Siddhartha Ghosh: And we need to capture most of the opinions or most of the questions that you have or concerns you have so that we, we are able to address those.
Siddhartha Ghosh: We will share with Members the recordings minutes agenda slides etc all participants muted, by default, as you join as you probably have noticed already.
Siddhartha Ghosh: Comments question us raise hand feature, if you want the.
Mick Coady: We want to speak.
Siddhartha Ghosh: or just flashing the chat window, as you have been doing already or we are monitoring the chat you may email this a mailing list anytime for the low end of this response, which will be a little more persistent response over there it's a shared inbox type.
Siddhartha Ghosh: group, where we, we will be monitoring all the males.
Siddhartha Ghosh: Then about in hug and the meetings, this is a user group user group that is for you and by you, your needs, this is about your needs how you want us to serve you.
Siddhartha Ghosh: about our services, what do you like do not like we do not have guarantee, but you want us to offer you have experienced elsewhere, you want us to offer, for example.
Siddhartha Ghosh: Already in the chat window, you will you have seen the mention about the docker container came, so we do not have a very extensive very know what to call.
Siddhartha Ghosh: elaborate or more extended support for docker container currently, but we do support docker container but similarly.
Siddhartha Ghosh: that's just an example, many other things that we do not support currently, but if you need us or want us to offer this is the forum, we would like you to Boise are concerned.
Siddhartha Ghosh: about the meeting, ideally, we administer and set platform for your voice, so it is your platform, or we are just giving you the context, so that you were able to speak.
Siddhartha Ghosh: Up you want to organize it differently it's up to you, we are just giving you this organization to modify to your needs now the Community communication well the easiest way to communicate will be send a mail to get you caught it to sorry.
Siddhartha Ghosh: Give me a second.
Siddhartha Ghosh: sorry about that.
Siddhartha Ghosh: So.
What is happening.
Brian Vanderwende: So I think if you use, you know arrow keys to move on the slides.
Siddhartha Ghosh: Instead of clicking probably that's better.
sorry about that.
yeah.
Siddhartha Ghosh: So this is our web page, we in the invitation mail I have given the link of this page.
Siddhartha Ghosh: This the wiki the wiki is our primary communication medium where you, we can have persistent storage of all of our communication in all the sharing that we will need to have will be there all the meeting minutes recordings, etc.
Siddhartha Ghosh: Anything you want to.
Siddhartha Ghosh: create a new page or add a comment add an agenda for future, a future meeting, this is the place where you would.
Siddhartha Ghosh: Go and.
Siddhartha Ghosh: replace your concern, so they are women teenager minutes slides links to recordings, etc.
Siddhartha Ghosh: Current priorities in wc three obviously then next year by this time we'll have the machine fully in production so, particularly to give the gpu partitions enterprise, as you all know, our flagship codes.
Siddhartha Ghosh: not suited for gpu partitions so we want new applications either email deal or any model like empathic or any other models.
Siddhartha Ghosh: That are that have been already voted on to gpu.
Siddhartha Ghosh: To focus on gpu partitions is the port projects for gpu participants your training needs anything that we are missing, this is just to give you an example of things that we need to discuss over here.
Siddhartha Ghosh: And then I hand over to Brian Brian for more detail about the application day focus or in wc three brand new tech from here.
Brian Vanderwende: yeah thanks said so i'll just briefly describe what what our team is doing right now so there's there's a bunch of deployment teams that have been structured.
Brian Vanderwende: As we you know, work towards the deployment of NWS C three and one of the teams that Ray Kelly, and I are leading is this application team, I would say, this is the most user centric team were concerned about things like application readiness, we want to make sure that.
Brian Vanderwende: Our you know our flagship codes, but also these newer codes are ready to run on NBC NBC three and, similarly, the end of usc three is ready to run those codes.
Brian Vanderwende: We want to make sure that anybody who's interested in getting spun up using gpus or it has questions about say the PBS scheduler configuration or.
Brian Vanderwende: Concerns about the amd chips or anything like that that we can make sure that you feel like you're ready to move to the new machine when it's deployed.
Brian Vanderwende: we're also charged with constructing or at least prototyping the user environment so things like you know the software stack things that work well for you right now things that you anticipate being a challenge, maybe in a more heterogeneous computing environment.
Brian Vanderwende: Those are the kinds of things that we're interested to do some outreach and get your feedback on so you know that doesn't have to happen in this meeting, of course, because.
Brian Vanderwende: These are, these are parallel efforts, I would say, but I just want to bring attention to what we're doing and.
Brian Vanderwende: If you want to reach out to us about any of these concerns, you can send myself and or rory an email and we'll certainly be in touch and things like you know some of the questions that have come up here about you know how workflow managers will work with the.
Brian Vanderwende: With the PBS configuration for the scheduler and you know it certainly any questions about gpu computing and how we intend to structure that on the new machine, those are great things for us to discuss with you, moving forward.
Brian Vanderwende: So yeah that's a that's kind of just a summary of what will be will be doing.
Siddhartha Ghosh: Okay, so well, we are now opening the floor, you can ask questions over the chat or.
Siddhartha Ghosh: Even mean you know if you want us prefer to speak just let us know abroad or you can unmute yourself If not, our will and will.
Brian Vanderwende: fix.
Ethan Gutmann: I just I just asked this in the chat but i'll repeat it here i've.
Ethan Gutmann: i've had issues getting code to compile and create a system it's just because they they wrapped different compilers and all sorts of weird ways can we get access to some kind of small test system with the cray programming environment built on it.
Ethan Gutmann: So that we can test out our compilation at a time.
Brian Vanderwende: yeah so so i'll speak to that.
Brian Vanderwende: We have, I guess, a number of different stages of access that we're looking into in the short term we're hoping we're trying to get access and we're planning to get access to some HP internal machines to play with that.
Brian Vanderwende: You know that that'll probably be just you know our applications team and some other select folks, but we will have a test machine that gets deployed late in the summer.
Brian Vanderwende: That you know will be our machine and I think at that point we can we can work more closely with folks.
Brian Vanderwende: In terms of getting all the wrinkles out of that that great computing alarm, because that that is one of the risks that we're paying attention to is is making sure that you know our default compiler and mpi and software stack that we get from the vendor plays nicely with our applications.
Thanks.
Siddhartha Ghosh: More questions, and particularly the university folks I know inertia wyoming folks.
Siddhartha Ghosh: were pretty quiet so.
Kartik Iyer: Hello hi yeah it's got the here.
Kartik Iyer: For the new machine will be testing a few.
Kartik Iyer: Standard codes to check the performance of the machine and.
Kartik Iyer: What will be the process of selecting those codes from your end.
Siddhartha Ghosh: Okay, let me take that first and you take it from there guarding.
Siddhartha Ghosh: The.
Siddhartha Ghosh: As I mentioned, our process of selection of this machine is based on the performance or the capacity, so we already have a nw sc benchmark suite.
Siddhartha Ghosh: For that and different applications has different weight associated with the performance contribution and that's how we get to that factor of, say, 3.5.
Siddhartha Ghosh: cheyenne equivalent and things like that So yes, vendors are required by the contract to make the cheyenne equivalent performance that they have.
Siddhartha Ghosh: projected to get out of this machine if they fall short of that they will have to provide additional hardware or thing mo a bit bs back in terms of money or this needs to be discussed how they will compensate that but there by contractually they are required to.
Siddhartha Ghosh: Keep the performance level that they have committed to for the money that we are paying to them.
Kartik Iyer: gotta tanks ashutosh out one more follow up question so once the machine is up and running, then you have like an early user user period where we can test our code, so that when you're applying for computer time we can use that say scaling data.
Siddhartha Ghosh: Yes, you will have to do that in fact the earphone spoke about he is the period advanced scientific discovery roughly we are expecting.
Siddhartha Ghosh: If everything goes well from January to January 1 2022 till April 1 2022, we will have this three month period dedicated to advanced scientific discovery and this will be a mega project to something like 20 million or more hours.
Siddhartha Ghosh: No pride scientific project that needs to be completed within this to be three month period now in order to apply for this project, the first thing you need, we need to ensure that the codes are ready to run scale well and.
Siddhartha Ghosh: But farms well into the machine so obviously there will be our early user Phase I you'd assume it will the prior to January MINS maybe December sometime.
Siddhartha Ghosh: When you need support your code and it needs to come up with the performance number in order to show that your code is performing when now is the process will start in my.
Siddhartha Ghosh: Microsoft wall says right, it will be something like.
Siddhartha Ghosh: September timeframe this year.
Siddhartha Ghosh: So you may have to produce the benchmark numbers from some other platform like cheyenne if you are not planning to use the gpu partition.
Siddhartha Ghosh: To come up with a projection that how you your code will perform on new in wc three grey system but.
Siddhartha Ghosh: After the system is come back up online, certainly, you have to read tweak your code to get to what where you have projected to.
Kartik Iyer: Okay, thanks, a lot so.
Mick Coady: I said said, been asked a good question to related to asd about when.
Mick Coady: When do we expect that the calls for asd projects will be going out.
Siddhartha Ghosh: A little bit floyd but my crystal ball says it will be some time during August September timeframe.
Siddhartha Ghosh: And Ben is one of the priority is the participants I believe so, certainly will will contact him directly and otherwise there will be nsf solicitation for this as the project is the is much bigger than the usual enter allocation process.
Siddhartha Ghosh: On much wider segments that we usually say this.
Siddhartha Ghosh: In a wider.
Siddhartha Ghosh: You user base our circulation.
Mick Coady: Then let's note that answers your question or not so and then Dave Gill had another good question about internal training, you know for user training on gpus and would that.
Mick Coady: fit under the applications effort that brian's been talking about, I believe.
Mick Coady: It it won't directly fall under the gave and will fall under directly under the applications teams.
Mick Coady: charter or effort, but we have a parallel effort going on within sizzle that we've aranka has tagged the gpu tiger team.
Mick Coady: Most of consulting group is involved in that, as well as some of the other there's some Members.
Mick Coady: Of you folks who are in this meeting, who are also going to be part of that and the training for users, both internal to sizzle internal to end car and and universities will be the responsibility of that gpu tiger team and you'll be hearing more about that here very shortly.
Brian Vanderwende: So said there's a question from JEREMY sour and I think.
Brian Vanderwende: You know, you can take this one so it's is there any notion of how allocations and charging will work for gpus will gpu nodes be a separate Q from cpu nodes.
Siddhartha Ghosh: JEREMY This is something we are discussing at this point in time, of the framework, what we are.
Siddhartha Ghosh: leaning more I would say is GP will be completely separate out of the.
Siddhartha Ghosh: usual allocation poll that this system will have our any of our system has currently.
Siddhartha Ghosh: So.
Siddhartha Ghosh: It will be on the project basis.
Siddhartha Ghosh: When either form university or from.
Siddhartha Ghosh: An car or elsewhere.
Siddhartha Ghosh: Users may request for gpu resources.
Siddhartha Ghosh: For dedicated run and though the file system and other things will be shared.
Siddhartha Ghosh: The gpu partition and they will be completely separate and there will be separate you for gpus now we'll be here i'm assuming the application is such that.
Siddhartha Ghosh: If it is gpus and take application, it will be running on gpu and cpu centric will be on cpu but could there be applications were part of it is really running on gpu partition and communicating with its other part which is mainly cpu partition in theory, this is possible.
Siddhartha Ghosh: In.
Siddhartha Ghosh: Even in practice the system will support such applications, but in practice will such applications show up in our mix I doubt very much but, if indeed it is or will be flexible to that as well.
Supreeth Suresh: i'm sorry I said just to follow up on that.
Supreeth Suresh: The thing you said just now, if we have a mix so currently it's not doable on casper and Chan, but it will be doable on end wc three, where we have a mix of cpu doing one task and mix of gpu doing under the desk yeah yeah that's right.
Siddhartha Ghosh: that's exactly that is single mpi applications can have a gpu portions running on the gpu partition and the cpu portions are running on the sequel partitions and communicate with each other, under the same or y combinator yes, thank you.
Supreeth Suresh: yeah I was thinking of, for example, CSM if we replace some dynamical core with the gpu model we can.
Supreeth Suresh: run CSM like that get the dynamical code will be running on gpu and other dynamical call for example ocean model.
Siddhartha Ghosh: Yes, this is very much possible yes.
Mick Coady: Okay, said jms to another good question to says, can you talk more about how glade in the new file system will interact and what will happen to Grade one showing in his decommissioned.
Siddhartha Ghosh: Okay.
Siddhartha Ghosh: glick is the current glade I mean not that campaign storage campaign storage is also under the same glade partition some so.
Siddhartha Ghosh: Current glade will go every the project and the collection space will go every with the cheyenne.
Siddhartha Ghosh: step is that quite right bill.
Bill Anderson: yeah so yeah the blade scratch that's on cheyenne that would go away when cheyenne goes away that's my understanding of course there's the new there'll be new scratch on and wc three things like work and P that are currently in glade.
Bill Anderson: that'll be those will be transitioned over into campaign storage space is the current guidance we're getting from management, in other words kind of long, so it will need to last longer beyond and not be purged would be part of kind of campaign store in some fashion.
Bill Anderson: And file system on NWS see three the native file system there will just be scratch i'm not sure that addressed the question Jim or not.
Mick Coady: yeah bill I think it's probably safe to say that nothing is set in stone, yet right that's.
Mick Coady: Right little lot of these policies and and decisions are still being worked out and, in fact, what were some of the guidance we're looking for from this group.
Mick Coady: To help.
Mick Coady: form those policies, yes.
Siddhartha Ghosh: yeah So if you have any opinions along this line, we very much want to to wise that or any users here.
Siddhartha Ghosh: I invite you to revise your preference and things like that.
Mick Coady: said, perhaps, since this is so, you know, obviously a very important.
Mick Coady: message matter to us to all users, might we might consider if it's if the rest of the group is okay with it is making that an agenda item or a discussion item that next month's meeting.
Mick Coady: Or at the storage advisors meeting which a lot of these folks are part of.
Brian Vanderwende: yeah just that I mean you know to level set on that you know, obviously, about the hardware has been procured you know we have this.
Brian Vanderwende: 60 petabytes of last year, coming in and and you know there's other moving parts with storage hardware, but a lot of the way that it's structured and the policies and the quotas.
Brian Vanderwende: How it's partitioned you know that there's certainly still flexibility on that, so this is, you know the earlier, we can get that input.
Brian Vanderwende: These easier, we can move things around.
Siddhartha Ghosh: Okay, any other questions or topics that you would like to discuss will capture the discussion threads and put it under this in hug wiki.
Siddhartha Ghosh: When we will share the meeting minutes in a mail to you will have links to the wiki.
Siddhartha Ghosh: Particularly for university folks you need to use CIT password in order to get to our wiki if you are not familiar with it and current folks are already using the wiki so they know pretty much.
Mick Coady: said the Charles Jackson asked a question, let you take it will there will there be topics for these meetings.
Siddhartha Ghosh: And the topics are the agenda items is.
Siddhartha Ghosh: Basically, up to.
us to decide.
Siddhartha Ghosh: Already we found few agent item, as I can see related to storage stuff and.
Siddhartha Ghosh: you're also a follow up item I have not done from Jim the gpu data so probably We will report back after working with Jim.
Brian Vanderwende: Exactly.
Siddhartha Ghosh: What the requirements are and what the follow up decision or the discussion from our side, so no please pick up and that will be my suggestion to you not only.
Siddhartha Ghosh: You can do it over here, but not necessarily not only here but, once you get the wiki link there you add your comment, create your page to voice your concerns or agent item for the next meeting.
Siddhartha Ghosh: comment on topics we'll we'll try to correlate those things and form the agenda item for the next meeting when that's my preference at this point, thank you.
Brian Vanderwende: I think you know a lot in those deployment groups will happen in one month, so perhaps bill and I are are some some assortment of folks can put together a.
Brian Vanderwende: You know, a couple slides just as a starting point for the discussion on say the storage and how the gpus will be interact interacted with and things like that.
Siddhartha Ghosh: Okay, and JEREMY has asked a question currently we have 328 100 gpus To begin with, how much your own.
Siddhartha Ghosh: More more for assuming a need for expanding gpu partition with naacp three title lifetime and doubling well.
Siddhartha Ghosh: At this point I would not rule that out the doubling.
Siddhartha Ghosh: But it all depends on the issue, say, Jeremy.
Siddhartha Ghosh: Quite frankly, this is our biggest worry when that is.
Siddhartha Ghosh: The good use of this partition, that is the very reason we have this DDT or the gpu tiger team which are aggressively working with.
Siddhartha Ghosh: You and others to come up with enough science cases to make this debut partisan busy for the time being, we do have our funds or do we will have funds are moving forward to expand gpu partitions.
Mick Coady: and
Mick Coady: said Dave just asked another good question of the 70 or more folks on the on this call about how many are from the university community.
Mick Coady: I think he's concerned are they well represented.
Mick Coady: So here.
Mick Coady: You have those numbers.
Siddhartha Ghosh: I do not know how many have giant adult so let me answer this, this is a good question.
Siddhartha Ghosh: Obviously this, since this is by selection.
Siddhartha Ghosh: I tried to come up with a criteria for selecting members out of this 3000 plus users, that we have, we cannot consider all of them, or all of you.
Siddhartha Ghosh: Believe because that's on practical or impractical to manage, so I, we will have to select so the selection criteria we already mentioned in the web pages, but it will be nice to spell it out over here.
Siddhartha Ghosh: We picked up all the nsc or majors jacuzzis like from an seo tip is there, anybody who has allocations for lot past two years, more than 5 million quarter hours on our system.
Siddhartha Ghosh: The P eyes of all the nsc and jack we have considered or we have also considered any users who have spent more than 5 million quarter hours we're past two years.
Siddhartha Ghosh: Then we tried to make it equitable the make enough participation for a wyoming folks because wyoming has a special allocation within our systems, right from the beginning, after we started this in wc.
Siddhartha Ghosh: and
Siddhartha Ghosh: With all this well folks from different a community as well as the developer folks like Dima doors and few others who are helping other users, they are not and themselves burning lot of cpu hours but helping other users to.
Siddhartha Ghosh: or code or use our systems, so all this more folks together we have about.
Siddhartha Ghosh: Close to 250 users total so that number is well represented within university and college and university of wyoming.
Siddhartha Ghosh: Then we sent this invitation to all of these users, out of which we got back positive response from about 120 users incur a slightly more dominated over university, because in Korea this I responded more currently.
Siddhartha Ghosh: Out of this hundred and 20 users you leave out about 90 is this from the support site and then or we have roughly 60 users from university in inca and about 40 users from the University.
Siddhartha Ghosh: And this call about 110 or so are invited, I do not know how many have joined.
Siddhartha Ghosh: And as we mentioned the criteria is pretty loose, we are not denying anybody from participating, rather we are inviting if you feel compelled to fill the represent a particular group particular usage particular model.
Siddhartha Ghosh: We encourage you to spread this word out to your colleague, we can invite them.
Siddhartha Ghosh: that's how the processes currently.
Mick Coady: So they just call out right at 10 o'clock with I think a couple more questions we do you want to handle those and just.
Siddhartha Ghosh: Okay, so let me quickly handle.
Mick Coady: The one.
Brian Vanderwende: Go ahead, I was gonna say there's one question from Eric about current gpu utilization and I see JEREMY pendant on this as well, I mean utilization is a is a.
Brian Vanderwende: You know there's a couple different ways, you can define that but one important thing to keep in mind is the casper is more of a.
Brian Vanderwende: test and development resource than a production resources for as far as the be 100 We certainly have codes that run there in a production sense.
Brian Vanderwende: But we're really looking for people to use those be 100 gpus there to prepare themselves for the bigger resource that will come online next year.
Brian Vanderwende: But in that sense, you know purely from scheduler availability, we see more time periods, where there's contention for those resources and a bit of a wait, then.
Brian Vanderwende: Then they're running idle, but there is a there is a an up and down curve, since it is more, you know, since it does have a development aspect to it and and that's a bit of a fuzzy answer it's not a number but but I I think we're seeing.
Brian Vanderwende: A good amount of interest on those resources.
Mick Coady: yeah i'm sure that the.
Mick Coady: All the conference in the world that the the gpu users will say that the contention is increased significantly lately.
Siddhartha Ghosh: Okay, since we are a little over 10 so unless there is any burning question, I will prefer to aim to respect everybody's time here.
Brian Vanderwende: And if you do have a question that you know, has you know, is more than a simple clever quick clarification question sending that to the email that could help us incorporated into the next.
Brian Vanderwende: You know the following meetings as well.
Siddhartha Ghosh: Okay, thank you, everybody.
Siddhartha Ghosh: it's nice thing.
Siddhartha Ghosh: and see you in a month that is next month six March, if I remember right the next meeting date.