Siddhartha Ghosh: Notification that it's been recorded now.
Siddhartha Ghosh: The meeting is being recorded now CB do audio chat transcripts as is or will be available in this in car and hug wiki page.
Siddhartha Ghosh: We have a shorter agenda today I will briefly update you on is the projects, the announcement that we have already made some time a couple of weeks back.
Siddhartha Ghosh: And then make will talk about our upcoming long outages and after that ugly request Dan to President is GP GP work under Prime of project.
Siddhartha Ghosh: We are trying to highlight the the ritual preparedness activity in grip so in future meetings, if any of you would be interested or like to present your work or your spiritual preparedness, I appreciate getting back to us, or just pawns a male or surgeon suggestion agenda.
Siddhartha Ghosh: So with that, let me start on the is depart.
Siddhartha Ghosh: so easy on the radio, as you know, when we deploy a new supercomputer the past three months we dedicate to a very large scientific projects for very large.
Siddhartha Ghosh: Very large computing scientific projects and kind of computing campaign, the idea is.
Siddhartha Ghosh: Few of these projects will have access to pretty much the whole machine.
Siddhartha Ghosh: During which you can get a lot of your science done in case, your computing.
Siddhartha Ghosh: workflow or the computing scenario fits that kind of model, so we have already solicited.
Siddhartha Ghosh: This is the projects for in car, as well as in universities So these are two independent tracks.
Siddhartha Ghosh: So it will be five to six cpu on the products and three to four about gpu only projects, just to remind you all 80% of the ritual will be sleeping only and 20% of the capacity is coming up coming out of gpu that's about 320 GPS and 2488.
Siddhartha Ghosh: cpu based modes.
Siddhartha Ghosh: So per cpu based projects.
Siddhartha Ghosh: Okay there's a question for Jim edwards though no Max cpu go projects we in fact we do encourage such projects, we are not anticipating that there will be a lot of such projects, but we do encourage.
Siddhartha Ghosh: Next cpu gpu projects in if you have a next step up projects me either of these projects if it satisfies this in the leader of this partition that is cpu only core hours exceeds 30 million or gpu only four hours exceeds with kiki you are.
Siddhartha Ghosh: pretty much.
Siddhartha Ghosh: In the competition.
Siddhartha Ghosh: So major competition and needs to be done by in January, February timeframe, we expect the competition will ramp down on during March, but it has to be completed all by March 31 because after that the system will be opened up for general general the general population.
Siddhartha Ghosh: By ID February, we will have a checking just to make sure all the projects that we have selected.
Siddhartha Ghosh: And is already on track to complete by 31st of March and we use up the right to terminate any projects that are not up to speed or released or or we can see that they won't be able to make it.
Siddhartha Ghosh: And we will have a backup or standby list other projects engaging or replace those projects that are not making sufficient projects by article.
Siddhartha Ghosh: Or we also expect all the data analysis to be completed and data to be involved by 31st of March 2023, that is, we will have little over and do all of this on this.
Siddhartha Ghosh: Specific in car.
Siddhartha Ghosh: This is the height.
Siddhartha Ghosh: And page.
Siddhartha Ghosh: letter of intent or you like, what do you would like to do, how are we aligned with your project we're in a strategic plan and all that needs to be submitted through labs by July 26 that is by 26th of this month.
Siddhartha Ghosh: And we expect three to four such a request will be coming from each lab at most in and out of which there will be a short list of project or I can be hope to be able to notify you by the August, or so, so that you prepare your full is the proposal.
Siddhartha Ghosh: And that is due by September 10 then RD of the world, the final selection and by that time, we hope that our test system will be ready for you to test and try or quote your project, so will will give you access to the selected project team to get ready for the day Joe by.
Siddhartha Ghosh: me.
Siddhartha Ghosh: So and.
Siddhartha Ghosh: This, this is the link, let me go to.
Siddhartha Ghosh: The chat.
Siddhartha Ghosh: The link has.
Siddhartha Ghosh: Let me do that later and go to the.
Siddhartha Ghosh: presentation and.
Siddhartha Ghosh: So.
Siddhartha Ghosh: And here's the university park and diversity it's all the point, everything is one chock full proposal do by September 3.
Siddhartha Ghosh: It will be just like your calendar and chapo guzman you have to have a nsf funded project.
Siddhartha Ghosh: Even if it is not directly related to the nsf funded project, it should be aligned with the nsf funded project and that's our requirement for universities.
Siddhartha Ghosh: And, just like incur project, you will have access to the test system by mid October or once your project is selected.
Siddhartha Ghosh: So with that, let me hand over to me for making.
Some.
Siddhartha Ghosh: Digital it again.
Mick Coady: So yes, thanks um.
Mick Coady: Can you hear me okay.
Siddhartha Ghosh: yeah I can hear me.
Siddhartha Ghosh: Okay Scott.
Mick Coady: I don't I don't have any slides to share at this point I just was going to give everybody a quick update.
Mick Coady: brief update and reminder that we are planning for a full.
Mick Coady: outage htc systems outage later this month, it will begin early Monday morning on July 26 and the full outage will.
Mick Coady: Or the yeah most of the systems will be back up by early.
Mick Coady: Sometime on Thursday that Thursday July 29, and that includes all of casper glade file systems campaign storage Jupiter hub H pss stratos quasar in shines login notes.
Mick Coady: Now shines compute nodes will remain down throughout the maintenance period and but are expected to be back up on the following Monday August 2.
Mick Coady: We, there are some there there's a lot of work being done in.
Mick Coady: In here there's a lot of work that's already started as a matter of fact, on the power infrastructure in at NWS see.
Mick Coady: The but in preparation, so that we can minimize the amount of time that the systems are down.
Mick Coady: And that's really all I really want I think I have to say about this at this point, but i'll be glad to field any questions at this point.
Mick Coady: Okay doesn't sound like it, so I will add up a couple let's see oh sure cecile so they will start on Monday July 26.
Mick Coady: Early that morning and the.
Mick Coady: The.
Mick Coady: will be the full the outage the maintenance work will extend through August 2 the following Monday, but one more time the.
Mick Coady: casper glade campaign storage Jupiter hub HP SS stratos quasar in shines login nodes are expected to be returned to service by that Thursday so approximately three days now.
Mick Coady: There is a full announcement on this, a reminder announcement coming out in this morning's daily bulletin, and that will run through right up till beginning of the outage the Friday before so if and I will.
Mick Coady: Put a slide add a slide to what said posts on the wiki for this as well.
Mick Coady: let's see.
Mick Coady: JEREMY you had a question I didn't quite catch it.
Mick Coady: Okay.
yeah.
Siddhartha Ghosh: Good equals bills important stock index here.
Mick Coady: yeah i'll post JEREMY that's a good point i'll post these in the chat here as soon as i'm done talking and so that they're captured this in here to.
Mick Coady: Any other questions or comments.
Ana Barros: hi this is Anna is there anybody that we can talk to about the asd university project proposals to get advice and enter exchange ideas.
Mick Coady: said I think that's more for you.
Siddhartha Ghosh: yeah yeah yeah.
Siddhartha Ghosh: Are you asking about seeking collaboration here, or you need more clarification or more.
Siddhartha Ghosh: yeah.
Ana Barros: I would, I would, I think I would actually both.
Siddhartha Ghosh: Okay.
Ana Barros: I think it would be more you know, the fact that I have somebody to to work on a really large problem, but he may need some support because he.
Ana Barros: doesn't have the experience.
Siddhartha Ghosh: Some it's my just my God, first of all, we encourage and six such wellman's solace excite assistance when we are proposing that you get back to us so Anna.
Siddhartha Ghosh: something you all to be mentors what I will do, I will, after this meeting, we will start a conversation chain and direct you to the right folks and for this kind of proposal, but this is also my open and.
Siddhartha Ghosh: announcement to all of you, if any of you interested or kind of thinking that you do have a large project, but how to get there, and how to prepare your proposal or how to.
Siddhartha Ghosh: arrange your competition, so please get back to us, so we will assist you in a test try to assist humans and it's not that we have a lot of resources to do with that, but at least we will give it a shot.
Ana Barros: Thank you.
Kartik Iyer: Hello hi i'm my name is karthik.
Kartik Iyer: I know one thing of this going to be a web link or a page where we can.
Kartik Iyer: go through the proposal details.
Siddhartha Ghosh: Online Oh yes, men so.
Siddhartha Ghosh: It was there in the slide as well as I have posted in the chat can find it out, or shall I do it once again.
Kartik Iyer: I should be able to fish it out.
Kartik Iyer: Okay, thanks.
yeah.
Kartik Iyer: Thank you.
Kartik Iyer: So just to add to the conversation that's.
Kartik Iyer: Just started, so we can ask, we can consult with you to find out whether this particular computation of problem that I have that good fit for Derek you will there be some consultant, who can help us.
Siddhartha Ghosh: Sure, please, we do encourage that consultation and.
Siddhartha Ghosh: I talked about that to announce.
Siddhartha Ghosh: But I just forgot it it's my yes.
Kartik Iyer: he's no sweat a lot, I really appreciate your advice, so I can just open a ticket or, should I just send you an email or how does this work.
Siddhartha Ghosh: But I would appreciate, if you can.
Siddhartha Ghosh: open a ticket mints that way, if things are tracked.
Siddhartha Ghosh: Most likely, most of the conversation will happen outside the ticketing system, because this.
Siddhartha Ghosh: Is not.
Siddhartha Ghosh: Really, a.
Siddhartha Ghosh: problem but.
Siddhartha Ghosh: At least.
Siddhartha Ghosh: In order to track or to assign a particular individual.
Siddhartha Ghosh: It would be helpful to go through a ticketing system.
Kartik Iyer: Instead, appreciate it.
Siddhartha Ghosh: any more.
Siddhartha Ghosh: questions on on any more.
Siddhartha Ghosh: Interest in any of you were interested in is Dr you have any questions that we have not addressed with today's it now.
Siddhartha Ghosh: To Jacqueline.
Mick Coady: said Craig had a good question about whether the nsc proposals will be coming out this fall and, if so, if they'll be for computing on cheyenne or Great show.
Mick Coady: i'm not sure the answer of that, and what I posted in the chat was that will ask sizzle and in car management and get back to everybody about that.
Siddhartha Ghosh: activity in the last in some way to pass the similarity same question and.
Mick Coady: Did.
Siddhartha Ghosh: A heart give.
Siddhartha Ghosh: me a coveted off and some of the heat is on on my answer coating dead, so the government fall nsc the requirement will be that you will have to come.
Siddhartha Ghosh: It will be only for a sham festival it only for the region, so the cheyenne in a seat and that needs to be completed by the usually we give you one year time but by the time your fault Tennessee starts the cheyenne lifetime access the way we see it is less likely to be.
Siddhartha Ghosh: less likely to be the case that we will get one week fully one yet so in the solicitation who will make that clear that you have to arrange your competition for less than a year, a year will be eight or nine months or so.
Siddhartha Ghosh: For very true, I think the fast nsc solicitation will be on fb dot info but i'm not too sure about this.
Siddhartha Ghosh: So I will get back to you.
Siddhartha Ghosh: Okay, so go ahead.
Supreeth Madapur Suresh: yep.
Supreeth Madapur Suresh: So I have a quick question about the deadlines for the test test system, so I thought, like the dojo will be open by January 1 but in the in the slide mentioned as mid October.
Supreeth Madapur Suresh: If it's mid October for the test system.
Supreeth Madapur Suresh: How many notes will be there, like what's the configuration of those things.
Siddhartha Ghosh: it's pretty small system.
Siddhartha Ghosh: Less than a blackman certainly not to her basics.
Siddhartha Ghosh: it's mostly for the portability voting and they you have to make sure that your problem fix their it may not have an entrepreneurial spirit, but this is a good question.
Mick Coady: said well i'll look that up and try and have an answer for everybody, before the end of this meeting.
Siddhartha Ghosh: Okay yeah.
Mick Coady: It in the chat.
Siddhartha Ghosh: yeah it wouldn't be nice Thank you.
Yes.
Supreeth Madapur Suresh: awesome Thank you yeah.
Siddhartha Ghosh: Okay sounds like we do not have any more questions, so let me hand over to Dan for presenting his preparedness with DEMO project with gpu gpu Dan that's all yours, you are co host you should be able to share screen.
Daniel Howard (CSG, NCAR): yeah messages after great music reopens You may think, because of security settings but just a moment now try to get that working so.
Daniel Howard (CSG, NCAR): um.
Daniel Howard (CSG, NCAR): Let me see how.
Daniel Howard (CSG, NCAR): It goes okay.
Daniel Howard (CSG, NCAR): Okay i'm back now, and I think I have the sharing enabled and I should be able to start in just a moment.
Daniel Howard (CSG, NCAR): to share.
Daniel Howard (CSG, NCAR): And okay so um it's good seeing this first slide here enable presenter mode and then.
Daniel Howard (CSG, NCAR): so good.
Daniel Howard (CSG, NCAR): yeah it looks good okay so uh yeah so my name is Daniel Howard, I am devotee who had been working together with Brett Sanders and Adam loop, for the past couple months on the Primo models, the pillow vessel investor in addition model.
Daniel Howard (CSG, NCAR): partnered with the university uc Irvine and Zeppelin floods as small offshoot out of their research group.
Daniel Howard (CSG, NCAR): So you know this project is primarily using open ACC in order to implement a gpu version of their model.
Daniel Howard (CSG, NCAR): We chose opening CCS it's very accessible towards other developers, including Brett and Adam, who don't experience with kuda but it turns out, it was.
Daniel Howard (CSG, NCAR): simple enough to work with that to get them up to speed additionally we're also able to leverage a gpu hackathon during our work together, so we have Dave knowlton matt stack and mojito touchy nanny and video and SAS dsc who also contributed some work towards this particular project.
Daniel Howard (CSG, NCAR): So the Primo APP itself.
Daniel Howard (CSG, NCAR): is basically a.
Daniel Howard (CSG, NCAR): Explicit value find a dime scheme for for advancing solutions in a.
Daniel Howard (CSG, NCAR): Flood and coastal in addition model on a.
Daniel Howard (CSG, NCAR): Particular we've been using testing models in in the Houston and Miami areas Houston Houston hurricane Harvey as a particular test case, this is the solution on a Cartesian grid.
Daniel Howard (CSG, NCAR): Initial restoring the project and looking through the Code, the respected the V mug solvers to be the most expensive part in terms of what's contributing to time being spent in the model.
Daniel Howard (CSG, NCAR): And that's part of what we were looking at look at, but a nonetheless that there's some new new approaches that Brett Sanders had developed towards like.
Daniel Howard (CSG, NCAR): Solving this particular problem using this duel grid approach and then having these pieces live near took relations across each data cell so there's this upscaling process that happens within the the the vaster grits.
Daniel Howard (CSG, NCAR): and his initial project developing the model, it was a cpu based model working.
Daniel Howard (CSG, NCAR): With single buzzes multiple data design and there's a paper that was published in 2019 on that particular version just sipping on the API version of that model.
Daniel Howard (CSG, NCAR): it's all in fortran but our focus is basically to then convert this to at least starting off as a single gpu version you're using open ACC.
Daniel Howard (CSG, NCAR): And then eventually as nice progress towards multi gpu and petition with mpi API where Huda and opening CC and open AC is since essentially is cuda so it's just hidden from us as a programmer typically so the mpi recruiter can work effectively with open ECC nonetheless moving forward.
Daniel Howard (CSG, NCAR): I felt like it'd be useful within this group to talk about like how we approach this particular project.
Daniel Howard (CSG, NCAR): So, in terms of like working with open ACC the code itself involved in initial voting of the vessel data that section could not necessarily be accelerated in the gpu do to data movement limitations.
Daniel Howard (CSG, NCAR): So the parallel I O part part of the project that needs to be done is a separate goal from the open ACC and gpu petitions but, nonetheless, our primary target for open ECC was you know, Miss basically encapsulating and running kernels within the time stepping up to code.
Daniel Howard (CSG, NCAR): And, including, as I mentioned earlier, the vm in flux over teams, so our suggested approach when you know converting any model of code to a gpu version using ACC.
Daniel Howard (CSG, NCAR): Typically, you want to start if you're familiar with the opening say see basically you you so directors based approach.
Daniel Howard (CSG, NCAR): So you'll surround regions of code that can be paralyzed and put on the gpu with these ACC parallel regions Patmos Patmos in T C c++ and i'm for trying to use this bang dollar sign ACC.
Daniel Howard (CSG, NCAR): comment in order to tell the compiler that this is a pillow region and kind of use kernels are parallel regions of kernels are ones where.
Daniel Howard (CSG, NCAR): The the compiler is left to determine whether or not it can be effectively paralyzed and parallel region is basically where you can have more control at the House paralyzed.
Daniel Howard (CSG, NCAR): Typically, parallel can be better, but you said to be more careful and how you're setting that up and speaking of that and.
Daniel Howard (CSG, NCAR): being careful and how you're doing so it's important, as you are making these changes to verify the correctness of the Code as you move forward.
Daniel Howard (CSG, NCAR): a useful tool is the flag T equals auto compare which you can add a compile time and that will basically.
Daniel Howard (CSG, NCAR): Make the model run on both the cpu and the gpu simultaneously and compare the output between the two whenever data is copied between the two devices.
Daniel Howard (CSG, NCAR): there's some usually actually some discrepancy between the values between the House and device but usually that order of there is usually on the.
Daniel Howard (CSG, NCAR): Order of 10 and that might have 16 something like that, but you get like larger differences and obviously there's an issue that you have to deal with, but that particular.
Daniel Howard (CSG, NCAR): tool, you can use the otter compare tool, it was very helpful towards ensuring accuracy of the model it's important to start with just the ACC kernel and parallel regions and and use that then the theatrical Perry like first before adding data regions.
Daniel Howard (CSG, NCAR): Since.
Daniel Howard (CSG, NCAR): That moving the data out and doing that manually is camp is probably the more difficult aspect of using open SEC.
Daniel Howard (CSG, NCAR): Whereas if you just use the the ACC kernels unparalleled the data movement has handled for you automatically but basically every time you encounter data region it's always copied and copied out and so that can cause.
Daniel Howard (CSG, NCAR): The the model to perform really poorly initially, but then, once you then add the data regions and the third step, you can to minimize data movement.
Daniel Howard (CSG, NCAR): That will again I want you to to speed up the model to the point where you're hopefully getting to a performance based you're trying to achieve.
Daniel Howard (CSG, NCAR): But uh as I was saying, so the district, to reiterate that the parallel regions themselves without any debt debt of agents.
Daniel Howard (CSG, NCAR): Basically, the compiler automatically creates data regions immediately surrounding those criminals or parallel regions, but then each time you counter them to copies in and out, and that makes it be very slow so initially it will be slower when you start from step one.
Daniel Howard (CSG, NCAR): But if you don't do this process, it can be much harder to isolate bugs if you say, do the colonel's and data regions, at the same time and trying to figure out if maybe.
Daniel Howard (CSG, NCAR): It Colonel is causing the issue or when the data regions as President issue so that's just an important point to make in terms of.
Daniel Howard (CSG, NCAR): The workflow that should be trying to work towards in terms of how to.
Daniel Howard (CSG, NCAR): refactor code for gpus using open opening sexy.
Daniel Howard (CSG, NCAR): So in the freemium model.
Daniel Howard (CSG, NCAR): You know, as discussing we had these data, regions and these.
Daniel Howard (CSG, NCAR): And compute regions, and so the gpu Colonel regions.
Daniel Howard (CSG, NCAR): In our in this particular model we have this meantime you send me a duping through at over each time step.
Daniel Howard (CSG, NCAR): And what actually ended up working out really well for our case was to actually encapsulate that whole meantime new within a within a large data region.
Daniel Howard (CSG, NCAR): and basically moving all the data, before we start the time the onto the gpus and do that appropriately as necessary so like you know copy in or create if it's 10 being.
Daniel Howard (CSG, NCAR): That there's different directives, you can use that might that will be prudent to consider depending on the use cases necessary for each that are you putting on the gpu.
Daniel Howard (CSG, NCAR): And then within that time that you can then have the gpu kernels either side, like within a functional call and so like so, for example, we have this like called flux is routine the pair of the ACC parallel bluejeans are actually inside that flexibility, not shown, but you can like.
Daniel Howard (CSG, NCAR): You know, separate the concerns of the program in terms of like you know you have these routines elsewhere, you don't necessarily have to.
Daniel Howard (CSG, NCAR): have them all within the same space, but uh it's important, at least with whenever you are working on the gpu just to be mindful of the the data environment you're always you're in where the data is located.
Daniel Howard (CSG, NCAR): So if you ever have to find like say cpu cpu code within that time.
Daniel Howard (CSG, NCAR): If you're looking like the Center of this.
Daniel Howard (CSG, NCAR): scale studio code if there's any secret code there that saving lives on data that was our variable that was on the gpu you have to copy it back to the host the cpu.
Daniel Howard (CSG, NCAR): And then, when you're done with the cubby back to the device to keep working with it, so you have to be mindful of those sort of considerations, but you want to if, in any case, you want to minimize any of the data movements and try to keep as much the competition on a gpu as possible.
Daniel Howard (CSG, NCAR): and basically Encapsulating the whole confusion with the larger data region is very helpful in terms of minimizing any amount of data data movements as as the program encounters any parallel region.
Daniel Howard (CSG, NCAR): Within say, for example, these this flux routine or advanced of routine called um but uh you know beyond just the.
Daniel Howard (CSG, NCAR): So yeah the main points, the data region can be as big as you want like you don't necessarily have to like keep the data regions around each of the kernels so that can be useful, just in terms of keeping your.
Daniel Howard (CSG, NCAR): Keeping the code readable but it's in it's also it's very useful, the same keep to just like make the databases as possible, so long as the other variables and data can actually fit within the memory on the gpu if that becomes an issue, then you will need to pick take a different approach.
Daniel Howard (CSG, NCAR): expect cetera so uh but beyond the design of the code itself the I think the I thought the main emphasis, I would like to take in this particular talk with us about.
Daniel Howard (CSG, NCAR): How do like use various tools to supplement your your design workflow towards improving performance of your gpu kernels and computing so.
Daniel Howard (CSG, NCAR): Two very useful tools that I recommend and like workflow patterns that nvidia recommends is the inside systems and that's like a few tools there's also inside graphics but that's the only useful if you're more often do you like a visualization workflow.
Daniel Howard (CSG, NCAR): But a each these profilers are useful towards towards assessing the performance of the model and help you then isolate where additional work would be needed in terms of.
Daniel Howard (CSG, NCAR): Improving the model these particular tools, or at least included on casper and under the mth PC compiler tool fit.
Daniel Howard (CSG, NCAR): When you load that module you'll then be able to run beyond the compiler themselves the video for competitors or pj compilers you can also then been the insight systems and site compute a.
Daniel Howard (CSG, NCAR): command command line interface modules in order to fund these profiles on your programs and the pretty simple all you have to do is pre pangea program with these.
Daniel Howard (CSG, NCAR): commands and says, for the cases of insect systems and nc you for the case of insight compute the additional arguments that you add in order to specify how you want to put out on but a basic one for like an SEC system just census profile and your Program.
Daniel Howard (CSG, NCAR): You can always learn more about like specific ways, you can.
Daniel Howard (CSG, NCAR): different options, you can run alongside these commands using the manual pages for those commands running man assessor man and to you or see the videos docs online.
Daniel Howard (CSG, NCAR): But some good options for emphasis include you know, specifying the output name you usually like the PDS job name easily enough and i'll just add a Q tip.
Daniel Howard (CSG, NCAR): Hugh D R EP for for their report on file names, as well as specifying the kinds of choices, you want to make sure we're on a trace like open ACC code, which is just coutinho versus maybe open cl code, you can specify that.
Daniel Howard (CSG, NCAR): If you want to track memory usage of your program you can add could have memory usage SAS equals true will give up but similar to if you've used.
Daniel Howard (CSG, NCAR): and video visual developed before they'll also include a sequel file by default, or you can specify it to be a SCI fi csv file of all statistics of the profile run and then you can run automate automate any any data analysis on the profile results directly through those databases.
Daniel Howard (CSG, NCAR): So, for example here, so this is basically our initial profiling results when we did our first pass of converting it to an open open SEC gpu version this, so this is showing a screenshot from the insight systems gui.
Daniel Howard (CSG, NCAR): So that at least the nice thing with open SEC automatically name kernels based at the sub routine and number line on So you can see, like exactly where the code and and which sub routine steps of issues that can be taking place.
Daniel Howard (CSG, NCAR): You can even further segment that profile by adding NV tx rages it's a particular module you can add or additional library link with like c++ code.
Daniel Howard (CSG, NCAR): That then allows you to call certain functions to then surround regions, the code to them have the compiler mark demarcate where it's it's time has been spent, as is taking samples throughout the code.
Daniel Howard (CSG, NCAR): So, initially, initially with with our case you know, using our.
Daniel Howard (CSG, NCAR): Our kernels surrounding like various routines one of the teams, we were using was this advanced routine that I mentioned earlier, and that the no pseudo code.
Daniel Howard (CSG, NCAR): Basically, this advanced your team was like taking up so much of the time, and each of the the time steps that you basically can't see any of the other.
Daniel Howard (CSG, NCAR): kernels being done, and so, like you know you can see from the output of the screenshot that you know it's taking up a large bar graph of like the the time period and like if anything you can't even see the.
Daniel Howard (CSG, NCAR): The smaller channels outside so clearly there's an issue there, and if you want to expand down to the advanced view the lower part and can see that, like the.
Daniel Howard (CSG, NCAR): Advanced routine itself at the within each time to be sticking like you know, half a second, while the other kernels are you know, taking on the order of microseconds.
Daniel Howard (CSG, NCAR): So obviously there's an issue there that we needed to do to address and so as we push forward we actually were trying to figure out how to fix that.
Daniel Howard (CSG, NCAR): towards towards testing to to fix that particular 17 we determined that it was a capella treating one of the another separate team within the advanced your team that was causing excessive data movement.
Daniel Howard (CSG, NCAR): To be very specific and how that works within open ACC as if you want to like learn more about.
Daniel Howard (CSG, NCAR): Opening CC program you're not familiar with it there's one other director, you can use for the ACC routine, which is allows you to specify a specific 17 to be compiled to run on a.
Daniel Howard (CSG, NCAR): On the gpu and so, then when say like when this say called within a credible it knows to them on the gpu version of that of that a separate teams.
Daniel Howard (CSG, NCAR): code.
Daniel Howard (CSG, NCAR): So in this particular case.
Daniel Howard (CSG, NCAR): it's in most cases it's actually possible and usually efficient sufficient to just use the ACC routine specify is if this is critical function, or if it's like a if it's parallelizable it can be like a simple sequential function.
Daniel Howard (CSG, NCAR): But uh depending on how your pitch a passing arguments to that routine you can maybe cause issues, so not particular case.
Daniel Howard (CSG, NCAR): So we had like our advanced routine.
Daniel Howard (CSG, NCAR): And we had the vfr routine That was a sequential in scope, but what we're doing you know we're passing.
Daniel Howard (CSG, NCAR): Your variables to this routine you know X, Y and at a slice of a an array, you know as he went to like a small range of array.
Daniel Howard (CSG, NCAR): It was, it was a small range should like it wasn't useful to try to paralyze it even further so so you still wanted to be sequential.
Daniel Howard (CSG, NCAR): But by doing this, the slicing, at least within fortran what ended up doing is the end up creating that copy of that very each time that.
Daniel Howard (CSG, NCAR): method was called rather than passing the address of that of that particular subset of the right so then when, as it accounted that immediate very, very slow so actually what we ended up doing.
Daniel Howard (CSG, NCAR): In order to fix, that is, we had to then basically in line that that function manually.
Daniel Howard (CSG, NCAR): Within the code and that actually ended up fixing the problem.
Daniel Howard (CSG, NCAR): And we're able to avoid that issue with the how the compiler was treating encountering that particular spot vfr sub routine.
Daniel Howard (CSG, NCAR): So these type of issues, you can encounter and we end up you know to fix it that way, and once you fix it you we actually then Monday the profiler again and we sell much more improved results you know, we now have this much shorter duration of the advanced your team.
Daniel Howard (CSG, NCAR): Section now taking.
Daniel Howard (CSG, NCAR): Like you know, half a half a millisecond you know five five and 14 microseconds and that's on the comparable order the other other routines as well and.
Daniel Howard (CSG, NCAR): As an interesting note, you can we also initially I didn't show this in for this talk, but like we actually initially ran like an armed forces model.
Daniel Howard (CSG, NCAR): On force profile of on that model for just the the cpu version.
Daniel Howard (CSG, NCAR): And after we made this fixed for the advanced subroutine we now got similar like proportion of time spent with each through teams.
Daniel Howard (CSG, NCAR): That we saw on the cpu now also now on the gpu so it did show that we're achieving semak do throughput of these on on both devices, but now you know with the much, much greater.
Daniel Howard (CSG, NCAR): compute intensity possible on the gpus.
Daniel Howard (CSG, NCAR): So from the case of our hurricane Harvey test case for 100 ticket items like a test run we actually are able to achieve a 5056 X beat up about one V 100 gpu which is one cpu.
Daniel Howard (CSG, NCAR): And so, this beat up is isolated primarily for the compute intensive fusion version of the code that time setting loop does not account for the.
Daniel Howard (CSG, NCAR): Data visualization and io we had an read and all the data which is still doing a bit of a slow down enough and model but that's that's actually typical and most work closer you know almost each speed up.
Daniel Howard (CSG, NCAR): The compute issues you'll you'll be left with having to deal with io and so that that's where we're focusing on now currently for our future efforts.
Daniel Howard (CSG, NCAR): But, nonetheless, you know results, highlighting that you know we see improvements as we, as we so this is for like our upscale factory, basically, increasing the size of the problem.
Daniel Howard (CSG, NCAR): But like if we don't get that much improvement over time, it is because of that io issue we are encountering so the beyond the.
Daniel Howard (CSG, NCAR): Moving forward will plan to continue to address issues as well as then work towards the the Multi gpu implementation for this particular model.
Daniel Howard (CSG, NCAR): Before I move forward having gone through these the inside systems part of our profiling go there any questions at this point.
Daniel Howard (CSG, NCAR): Okay, I think i'll set for feel free to interrupt me anytime.
Daniel Howard (CSG, NCAR): So as before we go from your call easy we'll start with insight systems and then, if you want to go even.
Daniel Howard (CSG, NCAR): deeper into performance optimization go, then the next step is to use the insight compute profiler which will allow you to analyze specific kernels within the code.
Daniel Howard (CSG, NCAR): So the SI systems will focus on the overall code, but onsite sit compute will focus on just individual kernels within the code.
Daniel Howard (CSG, NCAR): So nc you will function, the same way you just pretend it to your to your model you're running and you can set various.
Daniel Howard (CSG, NCAR): options to that the same just like you would answer this.
Daniel Howard (CSG, NCAR): So in this case, you know some useful options include like a you know you can set the the full scope of all of all metrics that SA P, you can measure the dash F just means you override the file you out but can specify the name of the file report.
Daniel Howard (CSG, NCAR): At one of the most more useful ones is you can then you can also use the Colonel next option, you can then specify what name of the.
Daniel Howard (CSG, NCAR): Colonel you want the inside compute to to specifically be profiling, so, in our case, we wanted to analyze advanced from earlier.
Daniel Howard (CSG, NCAR): So we just named the and advanced 1509 gpu that's like the name, we saw from the insect systems, and you can focus on that particular Colonel otherwise, if you leave that out.
Daniel Howard (CSG, NCAR): The nicu and said compute will just analyze every single kernel encounters and, but you can limit say you know, maybe you skip a few to like do you have the gpu warm up or you can only like.
Daniel Howard (CSG, NCAR): A profile.
Daniel Howard (CSG, NCAR): A certain number of like Colonel launches so in this case, we only wanted to profile just two launches and this case there's actually like to kernels happening there's a there's a reduction not happening within the.
Daniel Howard (CSG, NCAR): method in the kernel itself.
Daniel Howard (CSG, NCAR): You want to make sure get book each of those any case for this initial.
Daniel Howard (CSG, NCAR): analysis that we saw with advanced.
Daniel Howard (CSG, NCAR): You can then we basically have here in the screenshot a comparison between the the inline vfr version of the model and the function based, to be brief version of the model, so the the the blue bars is the improved upon.
Daniel Howard (CSG, NCAR): And the the yellow bars is the one with it.
Daniel Howard (CSG, NCAR): I.
Daniel Howard (CSG, NCAR): Actually I think they might be backwards, the blue one is the.
Daniel Howard (CSG, NCAR): One with the enlightened and the yellow one is the one with that said as a function.
Daniel Howard (CSG, NCAR): And so you can they see that the memory bandwidth percentage has been has much increased with the.
Daniel Howard (CSG, NCAR): Improved model, which is one of the metrics we want to, we want to be.
Daniel Howard (CSG, NCAR): Improving upon.
Daniel Howard (CSG, NCAR): But you still notice that the the so the sm percent right there that's the student multiple posture occupancy rate.
Daniel Howard (CSG, NCAR): That is actually built to be low for both models and that's just indicative that is still a relatively small model for our test case that we've been running.
Daniel Howard (CSG, NCAR): And so we've actually run a few other additional test cases on even larger models and you know we're then seeing that you know the gpus have much more ability to handle much larger problems and even.
Daniel Howard (CSG, NCAR): We initially through reddit so uh that that was a good indication so far have.
Daniel Howard (CSG, NCAR): have additional of additional work could be doing, but, as you can see that there's lots of data that comes out of an SEC computer, it can be a little bit overwhelming in terms of what's important.
Daniel Howard (CSG, NCAR): And this is just like once one the initial page, you often see oftentimes I mean one of the some of the more important aspects to.
Daniel Howard (CSG, NCAR): Look at in terms of your your profile analysis, but you know you can do a roof line analysis with with within the the model on you know checking to see like the the.
Daniel Howard (CSG, NCAR): The intensity of arthritic intensity able to achieve, given the theoretical theoretical intensity, you should be able to achieve and the Dom that's.
Siddhartha Ghosh: A useful metric to.
Daniel Howard (CSG, NCAR): us as well i'm.
Daniel Howard (CSG, NCAR): Here you have a few workload analysis can see the type of operations that are emphasized that the data model, depending on the the invitation you we have used and so with the.
Daniel Howard (CSG, NCAR): As you can see, for the floating point 64 kind of line there's the I ended up miss mislabeling them with a vfr enlightened current but the the enlightened being the blue bars and the.
Daniel Howard (CSG, NCAR): yellow line being the functional version where it's just a function call, we see that, like a certain amount of time is spent just moving data around with the Al use the algorithm luxury.
Daniel Howard (CSG, NCAR): units, as well as some additional time scene and the the load stole units, the LSD years.
Daniel Howard (CSG, NCAR): And so you end up like losing a lot of time spent with actually doing floating point calculations, which is what you want to see that graph higher than most of the others.
Daniel Howard (CSG, NCAR): Ideally, even you wanted to like take so at this point is really only using 15% of the of the time spend the model so there's still some improvement public likely to be made, but uh in a case it's moving towards that direction is is some of the type of.
Daniel Howard (CSG, NCAR): Improvements that you can use the SI compute to to investigate and then see where those improvements can be made.
Daniel Howard (CSG, NCAR): Additional parts of antique compute you know that those.
Daniel Howard (CSG, NCAR): memory workload analysis can.
Daniel Howard (CSG, NCAR): See how that has been moved around with and the.
Daniel Howard (CSG, NCAR): gpu in its various caches.
Daniel Howard (CSG, NCAR): Now to save you know if you're seeing issues here.
Daniel Howard (CSG, NCAR): You can then work on still trying to cook called us memory together make make it so the the the memory is in line with each other as as moved around.
Daniel Howard (CSG, NCAR): That with an open ACC there's a cash directive to help to specify man we're going to use often and frequently within a kernel to keep it within a.
Daniel Howard (CSG, NCAR): A local cache it's bit more readily access regularly a while the colonel's running Those sort of optimizations you can then see maybe an impact here.
Daniel Howard (CSG, NCAR): Being to to the additional more detailed analysis that insect compute offers and.
Daniel Howard (CSG, NCAR): Emily like there's that, as I said, there's there's lots of data, you can go through through anti compute.
Daniel Howard (CSG, NCAR): i'd recommend I can't go to detail and everything one important thing, maybe to look out for any of.
These.
Daniel Howard (CSG, NCAR): warning signs symbols that that can appear.
Daniel Howard (CSG, NCAR): they'll often give you like a little recommendations of what to look at what my important so there's even a section that can like.
Daniel Howard (CSG, NCAR): highlight specific you know line numbers within your code, where the profiler has seen an issue and that can help point you in the correct direction of where you might want to look at towards further optimization.
Daniel Howard (CSG, NCAR): But there's only so much information there within the.
Daniel Howard (CSG, NCAR): SEC, if you can offer that.
Daniel Howard (CSG, NCAR): is important, maybe, just like you focus on a few and just be on the lookout for something important and once initially so you don't get overwhelmed at first otherwise, if you want more information, you want to like keep going.
Daniel Howard (CSG, NCAR): diving down to performance optimization there's plenty of documentation that video offers, as well as other resources towards how to interpret some of these results.
Daniel Howard (CSG, NCAR): One last thing I focused, at least with.
Daniel Howard (CSG, NCAR): insight compute if you go to the page here the there's like at the top left is a source and there's a drop down box, you can change that.
Daniel Howard (CSG, NCAR): Usually it's like details page is what your default we pulled pulled up, but you can change it to source and you can actually jump in on the code and see where time is back within the within each.
Daniel Howard (CSG, NCAR): Each line and actually then Compare that to the to the Assembly code, so if you want to go really deep, you can go that that developed but um.
Daniel Howard (CSG, NCAR): That that might not be of interest anyone, but nonetheless these these metrics here can these highlight where time is being spent, you can like see where.
Daniel Howard (CSG, NCAR): Perhaps if you if there's an opportunity there within specific function calls how to then improve upon specific areas of the of the model.
Daniel Howard (CSG, NCAR): So, as I said, lots of profile data what what is important so like to remind us that occupancy occupancy is an important metric to look at that's like the the.
Daniel Howard (CSG, NCAR): sm percent because we saw earlier, the streaming multiple sister.
Daniel Howard (CSG, NCAR): Just the number of active that's versus the Max number of fence computing unit so it's pointing to check whether the gpu gpu is.
Daniel Howard (CSG, NCAR): Doing multiple sensors are actually being kept busy or not, if though occupies fancy you can you can try adjusting work group size versus a gang worker vector sizing across the colonel's and how you specify that with an opening CC or if you're in gouda you can do the same thing.
Daniel Howard (CSG, NCAR): You have more direct ritual with cuda but opening see gives you some control just the same.
Daniel Howard (CSG, NCAR): Sometimes the occupancy can also be impacted by other things such as you know, with our initial case, we had a memory, to put issue as memory had to be moved around with the.
Daniel Howard (CSG, NCAR): inline vs functional function, based on vfr routine.
Daniel Howard (CSG, NCAR): But so that that that came up with the issue of business efficiency, where we saw very little floating point operations being done, and so you can look at.
Daniel Howard (CSG, NCAR): various different all those operations to see what's useful to you to work towards perhaps trying to improve.
Daniel Howard (CSG, NCAR): That those can do issues there that can lead to like Colonel stalls is like waits for move data to move around that can be an issue was like a memory dependency we're waiting for.
Daniel Howard (CSG, NCAR): That that sliced array to be created and moved on to the gpu and that that that was taking time because we like actual execution dependencies.
Daniel Howard (CSG, NCAR): And you're waiting for different tasks are finished synchronization issues memory thermals perhaps you know bandwidth is not sufficient amount of data, it needs to be moved now the issues like that.
Daniel Howard (CSG, NCAR): and, beyond that, like you know the last one is just the peak mammoth percentage that was the.
Daniel Howard (CSG, NCAR): Second, bar below the sm percentage that remember percent so that that's just checking to see how much of the achievements you actually achieving for the gpu so as I mentioned earlier, you look for like coalescing memory loads story about values and local cache memory.
Daniel Howard (CSG, NCAR): or V using data and that's like the main thing when you're often starting with open UCC code is like if you're still having issues in terms of moving data back and forth thing hosting device, if that is a sub optimally implemented.
Daniel Howard (CSG, NCAR): you'll have you'll see this feedback with bandwidth percentage suffers just the same.
Daniel Howard (CSG, NCAR): And so, this is like some final takeaway some fun, you know learning about.
Daniel Howard (CSG, NCAR): Profiling with nvidia insight and say compute.
Daniel Howard (CSG, NCAR): it's important to refactor gpu codes and incremental steps in order to avoid bugs the the auto compare tool is a great resource in order to ensure that you're maintaining correctness is your code.
Daniel Howard (CSG, NCAR): is important also then minimize data movement, so when we were initially talking about the the you know surrounding the whole compute the.
Daniel Howard (CSG, NCAR): Time setting with the database and you know, basically, by doing so we only moving data once and then.
Daniel Howard (CSG, NCAR): The second time when we exit gap loop and then any small times for like maybe constants within the types of things so as much as possible, try to minimize data movement through things do judicious use of data regions as much as possible um and sharing from Court by running.
Daniel Howard (CSG, NCAR): Through the when you're using profile is this point just you know.
Daniel Howard (CSG, NCAR): Go go through the steps so important to first one answers to just get an overview overview of the model, you can maybe choose other providers if you're comfortable with like our map or.
Daniel Howard (CSG, NCAR): or tower other providers, but it's good to just get to have an overall overall idea of how you model was performing.
Daniel Howard (CSG, NCAR): And like see where you're needing to improve upon that using that initial overview of the output from those providers and then, when you want to make even further optimization you can then use insight compute.
Daniel Howard (CSG, NCAR): and using that can help you say check if, like particular sections of code is being implemented effectively by the compiler like in our case I showed earlier with the.
Daniel Howard (CSG, NCAR): Call it to that vfr function.
Daniel Howard (CSG, NCAR): And so that's you points you know we end up making major progress being able to partner with the gpu hackathons a poke events that are offered regularly throughout the year, and so I suggest you know.
Daniel Howard (CSG, NCAR): Organizing instead of saying specific time for your.
Daniel Howard (CSG, NCAR): Small and medium sized.
Daniel Howard (CSG, NCAR): team to to focus on improving and models and you can use those gpu hackathons to not only have that space to dedicate towards towards working on on.
Daniel Howard (CSG, NCAR): Your model development but also then being paired with other mentors and often people from experience that other hbc centers or at film and video themselves.
Daniel Howard (CSG, NCAR): To to support your development just the same.
Daniel Howard (CSG, NCAR): And lastly, as an important thing to maybe mentioned, you know i'm focusing on optimizing the code making perform better, but a.
Daniel Howard (CSG, NCAR): code opposition is less important to falling good stuff and engineering practices it's always good to you know, make sure developing unit tests and documenting your procedures well and maintaining a recording your improvement over time.
Daniel Howard (CSG, NCAR): Simply and in a robust database, so that you're able to refer back to any changes you've made easily and.
Daniel Howard (CSG, NCAR): Unable to then share your event, you share your code and publish it in a manner that's accessible to others, and not just developers within your team.
Daniel Howard (CSG, NCAR): So.
Daniel Howard (CSG, NCAR): Just some useful resources, I imagine the slides can be shared later, but there's some textbooks just specifically on gpu profiling there's like a section in the parallel I prefer computing text by adobe into MMA.
Daniel Howard (CSG, NCAR): The cooking whoo text is a very useful resources, the same if you want a specific webinar that those.
Daniel Howard (CSG, NCAR): Specifically deeply into systems or compute separately and CSA has a few that they published.
Daniel Howard (CSG, NCAR): oak Ridge national that they have a few slides as well webinars they've done.
Daniel Howard (CSG, NCAR): You also want them to lose like in videos blogs, they have some useful posts about how to go about inspecting kernels and using the tools like insight compute.
Daniel Howard (CSG, NCAR): And of course there's always a developer documentation which is just as equal and point of reference, as well, can be pretty dense, but it has all the information I think you would need towards a further.
Daniel Howard (CSG, NCAR): Using these fellows to that to the best of their abilities.
Daniel Howard (CSG, NCAR): In any case, that about sums up from what all I have other any questions from about the use of insight systems computer or other gpu porting with open ACC and.
Daniel Howard (CSG, NCAR): i'm happy to take your take your questions.
Siddhartha Ghosh: please feel free to unmute yourself and ask question or just.
Siddhartha Ghosh: Like your question with jack.
Daniel Howard (CSG, NCAR): i'm seeing some questions or they see some comments request that chat i'm see farm service net is there any reasons to not to use or not use for Trans do concurrent for automatic gpu acceleration, but by MP4 trance dash STD park compilation flag.
Daniel Howard (CSG, NCAR): That I wouldn't I don't think there's any reason not to use it use it or not, if anything, the the standard implementations like that give offer like portability across.
Daniel Howard (CSG, NCAR): different systems, I think more easily than, say even open ACC open ACC actually will will support and has supported as of the past week piece the past two major versions so.
Daniel Howard (CSG, NCAR): I can't say like you know, using reason to use one or the other, I think they're both equally fine i'm.
Daniel Howard (CSG, NCAR): Adoption currently have, like the the standard language like do concurrent or sad part.
Daniel Howard (CSG, NCAR): is not yet prolific so I can't say towards a future viability of those methods, but I mean like they're all part of that fortune Center now so that that's that's a.
Daniel Howard (CSG, NCAR): Good approach to make, and I think it will be supported so I don't think you have a issue to use it or not, you can either stay was just opening CC or use the the standard petitions and reporter and has there been updated with the most recent portion versions.
Daniel Howard (CSG, NCAR): see any other questions.
Daniel Howard (CSG, NCAR): I have not seen any others, but feel free to unmute yourself or ask away but um otherwise.
Daniel Howard (CSG, NCAR): said, we will be posting the slides somewhere for others to review they'd like.
Siddhartha Ghosh: Yes, so all of these slides will be posted to enhance, we can say we can enhance.
Siddhartha Ghosh: The talks will be recorded and recordings will be there and yeah certainly I ICO ECO signals common but it's a very nice and informative presentation, thank you then it's a good job.
Daniel Howard (CSG, NCAR): Well Thank you everyone I think i'm going to pause share and.
Daniel Howard (CSG, NCAR): Otherwise.
Daniel Howard (CSG, NCAR): I think we're at the top of the hour, so i'm not sure what your plans are the rest of Agenda Bella i'll let you take it back said.
Siddhartha Ghosh: yeah Thank you Daniel.
Siddhartha Ghosh: yeah so since we are at the top of the hour and we.
Siddhartha Ghosh: We exhausted our agenda items to so let me end this call, and, as I mentioned in the beginning, and then go for all the integral postulating hyperventilation screwed that things will be recorded and have a little too hard week, you will have options to.
Siddhartha Ghosh: Make comment on.
Siddhartha Ghosh: people's featured it and as.
Siddhartha Ghosh: soon have a nice day bye.