Sidd: start the recording so recording.
Sidd: started.
yeah.
Okay.
Sidd: With that hit is our agenda and the Rachel latest update, in fact, we do not have an update at this point in time, we are still hoping.
Sidd: HP K, will be able to deliver in their do committed time but, as you know, from the industry news things basically as and when we hear more from them or a different commitment date from them, who will keep you posted.
Sidd: on designing digital end user environment that's that will be our main topic today the Brian will cover that part and following that.
Sidd: We are conducting a series of gpu training contemplating very different many different aspect of it, as you know, 25% of our compute resources will be gpu.
Sidd: Resources mainly so Dan will give you our current thinking and then ask you to provide your feedback that kind of training, you need the format of training.
Sidd: The schedule best suited to and things like that so either here or offline probably Dan has some of the Google form the mechanism to manage that he will get into the more detail.
Sidd: And then there is an upcoming gene for PBS deducted rights, which is the standard out standard error that you download from your jobs.
Sidd: oftentimes it can give problems john last we'll talk about that, and we are making some changes on our end to protect the nodes from crashing for some of the problematic jobs, some of these, we have already described in our documentation, but.
Sidd: As happens with their new users or who may not go through all the documentation or may not go through all that Vice pieces and have any evidently killed notes, so we are making some changes, we will we believe affect you positively, but let john speak more about it and.
Sidd: We welcome your feedback and then we'll open the floor for upload for open discussions so with that this is our territory studio, this is the exact same slide I present did last time so with that, let me hand over to.
Sidd: Brian this key so brad are you ready.
Brian Vanderwende: yeah i'm ready said, I think we may have gotten our wires crossed a little bit I was going to talk about kinda this morning I think you're that's what you'd asked me to talk about.
Sidd: my apology.
Sidd: So, Brian will be talking about condo okay i'm sorry about that.
Brian Vanderwende: No that's Fine, let me.
Sidd: just stop sharing and then it's your.
Brian Vanderwende: I will say, though, I can make a plug tomorrow, and this is probably where the confusion was tomorrow i'm going to give it a work in progress, talk on in our sizzle the whip session.
Brian Vanderwende: Talking a little bit about designing the ratio user environment, and this is content that's really.
Brian Vanderwende: You know I prettied up the slides a little bit, but it's stuff that i've talked about in various short presentations in this in the ad hoc meetings in the past.
Brian Vanderwende: So I don't think anything should be that new to this crowd having said that i'm happy to share that slide deck after I cleaned up a bit tomorrow.
Brian Vanderwende: With this audience so i'll i'll give that to sit and he can put that on the on the wiki page for those who are interested in can't make the the Web talk.
Brian Vanderwende: Because I know that tends to be a bit of smaller audience.
Sidd: yeah you can see, at this slide but probably in the next in her use the replay to so that they can also provide their feedback or.
Brian Vanderwende: Oh sure yeah that sounds reasonable, and I think at that point, maybe there's you know the spec work would be a little bit further along and I think that could.
Brian Vanderwende: You know, be a bit bigger component of the talk yeah sure.
Sidd: Thank you.
Brian Vanderwende: Can you see my screen said.
Yes, okay.
Brian Vanderwende: Okay, so this isn't going to be too long, you know, last time I talked a little bit about my intentions early I guess our intentions was a effort that was kind of collaborative between us and some some folks on the earth earth science.
Brian Vanderwende: or system data science group but.
Brian Vanderwende: Basically there's been a long, you know, a long running desire to have official kind of support on our systems, and this is something that we've sort of prototypes and put in place over the last month.
Brian Vanderwende: This would be you know certain.
Brian Vanderwende: Right now, the intention is to just support Python environments with this.
Brian Vanderwende: You know, will support other types of environments more natively you know, on the terminal you'd still use a module to load like matlab and are those in the Jupiter have.
Brian Vanderwende: There would be kernels that go through the you know the Jupiter interface that are maybe created via concubine scenes, but it should look apparently native.
Brian Vanderwende: But this is mostly to.
Brian Vanderwende: A make content available officially via a module and be move our package library which we've traditionally built from source to kind of infrastructure.
Brian Vanderwende: So there's now a module in place it's it's called conduct latest I didn't use our typical version and scheme for this just because kinda you know you can basically auto update it at any time.
Brian Vanderwende: And I think we, at least, certainly for the kinda executable itself, we want to keep it as up to date as possible, so this is this allows us not to have you know.
Brian Vanderwende: weekly module that just quickly becomes out of hand and annoying for you to track.
Brian Vanderwende: So you can just do a module load condo on cheyenne or on casper and get the module now one thing that's been tricky in the past is.
Brian Vanderwende: kinda is very insistent that you initialize kinda in your startup Shell scripts before you can activate any environment spin that designed for the past couple of years didn't wasn't before that, but.
Brian Vanderwende: You know that's kind of the route they've gone down and that's been tricky to.
Brian Vanderwende: To therefore support via module because if you do a module load and you only have access to like certain kind of commands and not the most important one of activating environment, then that's not very satisfying.
Brian Vanderwende: But through a lot of testing, we were able to figure out a way to make this work.
Brian Vanderwende: Basically, now, you will load the module and then you can run all the kind of Sub commands, including the activate and deactivate commands.
Brian Vanderwende: This will override when you load the module it will override the kind of binary in your user environment, so if you do initialize one in your say your batch RC or your tcs HR see a file.
Brian Vanderwende: When you have the module loaded you'll be using our binary so that version of conduct which, in practice, doesn't have huge.
Brian Vanderwende: implications unless your version of conduct is like wildly out of date, then you may see some behavioral differences, but the important thing is that it's active in your environment only when the module is loaded so we're not making any sort of lasting changes to your user environment.
Brian Vanderwende: This is done pretty easily and seashell we had to write a bit of an activation script and bash to make all this work, but it's a relatively lightweight modification.
Brian Vanderwende: I think it should be pretty robust to maintain moving forward, so the nice thing here, the bottom line is that with this module you can now activate.
Brian Vanderwende: manage content environments and you'll have no need for a Shell initialization no no need to put stuff in your bash RC or tcs hrs the file.
Brian Vanderwende: and worry about you know where that's getting sourced and all that, which I think this topic of conversation for us recently.
Brian Vanderwende: So it's really more native to kind of the htc style obviously this doesn't really have implications for Jupiter have users, but this will play in on Jupiter have a little bit.
Brian Vanderwende: Overall philosophy here has been to have the least impact as possible, but also to have some customizations in there that we think the majority of users might want to appreciate.
Brian Vanderwende: But we only impose those changes on the user, environment and an additive way we don't override anything you said so again the module adds this initialization bypass only if the user is not already initialized.
Brian Vanderwende: The we'd set auto activate based to false and that way your Python and whatever other executable that kinda is installing into its base environment to support things.
Brian Vanderwende: don't get added to your to your path, when you live the kind of module because that that just leads to unexpected behaviors.
Brian Vanderwende: And I think it makes sense to follow a usage model, if you look on the kind of module depend, you have to activate any environment to make any non conduct changes to your environment.
Brian Vanderwende: All of the settings that we've placed the configuration settings for our condo are in the root install of kinda in its kind of RC.
Brian Vanderwende: So if you're familiar with the way it kinda is configured there's a hierarchy of settings you have this kind of RC and the route then bet is below the precedence of a user level kind of RC file, which are going your home directory, so you can override that way.
Brian Vanderwende: and environment variables take precedence over everything so initially we had thought about maybe setting some of the settings in the module using environment variables, but then you'd have no way of overriding.
Brian Vanderwende: If you so desire, so we thought that this kind of lowest level precedents made sense for the settings that we were offering to you.
Brian Vanderwende: Some of the things that we've done we've set the default environment location, so if you create an environment, it will go into your glade work user condo he envies.
Brian Vanderwende: This overheads a default kind of behavior trying to put everything in your home directory which, if you have one environment might be okay, if you have a lot of entitlements.
Brian Vanderwende: That quickly doesn't scale, given the size of the home directory, and the fact that kind of environments can be gigabytes big.
Brian Vanderwende: kind of does some stuff to save space, but it's still you know can eat up a fair chunk of your home.
Brian Vanderwende: So in our minds work is the logical place to put it, however, this is setting a default location so kinda will still check your home directory.
Brian Vanderwende: You know if you have existing environments it's not like it, should it shouldn't it should still be able to find those it's just new environments will be placed in your county and these in work.
Brian Vanderwende: And as well coming has a package cash that it'll download the packages that have been unpacked to create environments this.
Brian Vanderwende: Again defaults to your home this mix even lessons, because these are really temporary files, so if you load the module your default package cash will be sent to your scratch.
Brian Vanderwende: And you know it'll stick around for the length of the purge which is fairly long these days and honestly, I think, by the time it would hit the purge if you haven't used it there's probably a newer version of that package.
Brian Vanderwende: So we think that this is a right cadence and the right locations, then, and then we'll set your default channels to become to forge and also the end part channel to support the geocache.
Brian Vanderwende: packages, so if you want to add any you, you could add them via your condo RC or you could specify them on the command line when you create environments or install packages for those would be the defaults.
Brian Vanderwende: Now, as far as environments The thinking is to continue offering an in car package library, but now, instead of it having its own homegrown infrastructure, it would just use the content infrastructure.
Brian Vanderwende: So the idea here is that it would be at a faster rolling cadence because it's quite a bit easier to install packages when you're not building from source.
Brian Vanderwende: So the current plan would be to refresh this these npl libraries every month, but basically three versions available, so there would be a.
Brian Vanderwende: Future library, which would be basically this month, then there would be the default, which would be the previous months and then.
Brian Vanderwende: A legacy when I supposed to be two months back.
Brian Vanderwende: But the default would roll forward every month with being the previous So if you go on there right now there's a November and then there's October and the October, one is the default.
Brian Vanderwende: So just to give people like a month time to maybe see that the changes that are in the new version don't break anything.
Brian Vanderwende: You know, and it would give two months time to make any sort of accommodations there.
Brian Vanderwende: could certainly revisit that cadence if it's problematic for people, but this kind of keeps things up to date and it's now kinda in its recent versions has a has a nice clone feature, just like we have in the mpl.
Brian Vanderwende: So if you wants to stick with a content environment longer than you can create a personal clone of it in your work directory and lock those versions down pretty nicely, so I think there, I think that cadence makes sense for what we're offering as default.
Brian Vanderwende: The default library is the only one that would be available in the Jupiter house and that's just to keep the.
Brian Vanderwende: You know the Jupiter hub splash screen relatively clean it's already a bit busy so we don't want to have lots of versions of this npl in there that are.
Brian Vanderwende: Making it very long and scrolling.
Brian Vanderwende: Certainly, seeking Community input on really everything, but especially some of these questions, we could possibly offer a machine learning version, or we could put the machine learning libraries into the default npl.
Brian Vanderwende: I haven't done this, so far, and the reasons for that are that it I it seems like.
Brian Vanderwende: Every machine learning project sort of is tied to a different version of the machine learning library, because the pace of development on those libraries so fast.
Brian Vanderwende: more so with you know some of the like tensorflow then maybe psychic learn a little bit more stable ones like those.
Brian Vanderwende: But the other problem is that a lot of the cutting edge versions are generally installed via pip not kinda.
Brian Vanderwende: And while the while the condo developers, are they seem to be working pretty hard on better inner inner ability between tip and kinda.
Brian Vanderwende: it's very prototype according to them, so i'm leery to install machine learning back just be a pip and make the environments potentially more fragile, but we could do that if there's strong demand and there's kind of a consensus around the versions, people would want.
Brian Vanderwende: The other question that I would have is the preference for the mpi libraries and in the in the mpl.
Brian Vanderwende: With users prefer them to be machine specific that we build from source and maybe have like an empire hbc channel or something.
Brian Vanderwende: You know you'd have like a mpt based version on cheyenne you have a open API version on casper or maybe if we wanted to have it similar we can do it with mpi on both it's tough to be separate builds because they're different operating systems.
Brian Vanderwende: Or do we use the end pitch offerings that don't have things like scheduler integration and all that but still function, the other kind of forge currently i'm doing the ladder and that's a to allow to have a single environment serving both systems and also.
Brian Vanderwende: That, in my mind, you know it increases the portability of your workflow from system to system.
Brian Vanderwende: Because you might be using that impish mpi on say, like the university system or something and it's going to work exactly the same as it does here, but it might not be optimal for using like our infinite band fabric and things like that so there's a trade off there.
Brian Vanderwende: And then the last question is how best to support him car projects that use our Python libraries, I know a few out there that load that activate the mpl the existing mtl.
Brian Vanderwende: or clone it and then build on top of it and use that to serve as the basis for you know what a build on, like some of the CSM post processing tools.
Brian Vanderwende: I believe met plus uses that on our systems, so you know I want to make sure that we're supporting those projects in you know, the best way that we can.
Brian Vanderwende: I think, maybe, creating a clone of whatever the default environment is the time that they want to version lock again would make the most sense, and we could even put that in our common library location and label it appropriately if if people reach out to us and discuss that but.
Brian Vanderwende: You know that's still an open question and certainly i'd be happy to work on a case by case basis if people reach out to me and how we can support you.
Brian Vanderwende: So those are the areas that I think we would like feedback, I just want to show real quickly.
Brian Vanderwende: That this is live so right here i'm on.
Brian Vanderwende: casper and if I do a MOD so First, I guess, I can do, which.
Brian Vanderwende: Under it's I don't I don't not currently activating it using my dash RC or anything if I do a module load kinda now should have it there there's are glad your Apps kind of our install.
Brian Vanderwende: can do a condo end list.
Brian Vanderwende: And right here, you can see i've got the base environment which is doesn't have much and so activating the base environment isn't that useful.
Brian Vanderwende: there's the three mpl versions that we're serving via this install and then I can still see the versions, that I personally installed with my own mini condo in the past, so.
Brian Vanderwende: You should be able to see the full list of all the environments, that you have a history of using in addition to ours, and you can see right here we've got an October version of the mpl and November version I installed yesterday and then.
Brian Vanderwende: Sim link basically that points to the prior month, so this point October, so if I were to do kinda activate ppl there we go, so I didn't have to do any sort of shell initialization I just have environment active and I have that version of Python there and let's see I can do.
Brian Vanderwende: well.
Brian Vanderwende: Anyway, the other thing I wanted to show was.
Brian Vanderwende: Sorry for bouncing around screens here.
Brian Vanderwende: If I go into the Jupiter hub.
Brian Vanderwende: You can see that there are now two and DL libraries that are available here there's the mpl 379, which was the one that's built from source the legacy one I guess at this point.
Brian Vanderwende: Then that's obviously the one that's been around for a while and then there's this npr kinda one which I don't plan to have.
Brian Vanderwende: You know the version explicitly called out, because that will flow moving forward, I guess that does bring up a point as well.
Brian Vanderwende: They, and this can be used on both these extra confused on both systems so they're they're portable in that sense.
Brian Vanderwende: But the currently the condo environments that we offer these empty all environments they're all Python three seven based, and the reason for that is that some of you have may have encountered Python versions newer than three seven have a have a copy routine.
Brian Vanderwende: That stalls that locks on G pfs for certain sizes of data.
Brian Vanderwende: If it's divisible by a certain number the command just will will stall, this is an ideal so that's why we recommend it to people in the past and we maintain Python versions that are stuck at three seven.
Brian Vanderwende: Eventually, we certainly would like to support the more recent versions of Python, but we want to offer something that's stable as possible on our platforms.
Brian Vanderwende: Unfortunately, seems to be a situation where the Python developers say hey, this is the GPS issue IBM go fix it and IBM like well you know it will fix it at some point, you could fix it so we're stuck in the middle waiting for somebody to actually do the do the fixing.
Brian Vanderwende: In the meantime, all the pythons that we offer will be three seven based until that until that gets resolved, but.
Brian Vanderwende: that's pretty much my overview certainly we're going to work on documentation next and then do sort of a formal announcement in the daily be, but this is live it's available.
Brian Vanderwende: And we would certainly appreciate your testing feedback and let us know that this is useful for you and I think it hopefully will alleviate some of the need for users to maintain their own personal mini Kaunda and maybe even maintaining their own personal environments.
Brian Vanderwende: So yeah that's about it, so if there any questions i'm happy to take them or otherwise, we can move on and since.
Mick Coady: Brian you have a few questions.
Brian Vanderwende: Excellent let me.
Brian Vanderwende: Take a look.
Brian Vanderwende: Okay, Brian says, are the npl environments i'll stop sharing.
Brian Vanderwende: Are the npl environments available.
Brian Vanderwende: In terms of kinda environment our files or any changes or additions.
Brian Vanderwende: yeah so.
Brian Vanderwende: By nature, there is the, the environment, I am will file sitting in the environment, and it could be tracked down by the intrepid user I, I think we could facilitate making that easy to find and it sounds like there's an interest in that so.
Brian Vanderwende: Let me take it, let me think about that a little bit how best to make that available and and I can let you know, Brian maybe we can iterate a little bit on that.
Brian Vanderwende: john says, are the.
Brian Vanderwende: Are all of the various idiosyncrasies of the casper cheyenne environment documented somewhere.
Brian Vanderwende: I don't know if it's a general question or Python specific john.
clyne: I don't know if you're fine and I think you answered the question I was just asking if the.
clyne: Basically, the usage of content on the sizzle systems is documented and I think you just said that that's next.
Brian Vanderwende: yeah yeah I really wanted to get this in people's hands, because if we if some you know some folks if they friendly user said.
Brian Vanderwende: This is cool, but I really like it to do this, you know we'd like to get that all straightened out before we we formalize the documentation for a broader group but yeah it's I would say, hopefully, in the next two to three weeks well I haven't officially documented and announced.
clyne: Soon, thank you.
Brian Vanderwende: David belly says i'm wondering about the timing of this one car pilot be going away yeah so the Sir certainly understand that question.
Brian Vanderwende: understand that concern, perhaps.
Brian Vanderwende: No, I don't I don't intend for it to go away, however, it will eventually he off, so I would say it's currently in a state where you know about a month ago we released a new version of it there's a.
Brian Vanderwende: installation with kind of the latest source builds I anticipate that's going to be the last source built npl unless there's some pressing need that we have moving forward.
Brian Vanderwende: But it will stick around as long as the current software stack sticks around so you know eventually cheyenne will be decommissioned, and the cheyenne virginal go away at that time.
Brian Vanderwende: i'm sure eventually will refresh casper at some point, because it's just something we need to periodically do.
Brian Vanderwende: And at that point that traditional and car pilot would probably go away as well just because I don't think we want to rebuild it at that point, unless, of course, this kind of approach doesn't pan out.
Brian Vanderwende: You know if it turns out there's still more of a benefit for users, doing the car pilot we could look at that, but yeah I would say, probably, certainly not in the next like.
Brian Vanderwende: You know, a few months I would expect it to go away anywhere.
Brian Vanderwende: So we're not trying to push people off of this right away or onto this right away.
Brian Vanderwende: See DJ says, thank you for putting this together, is there a plan to include desk mpi with Python mpi sport.
Brian Vanderwende: So yeah it's.
Brian Vanderwende: it's it is supported in there it's again it uses the M pitch.
Brian Vanderwende: mpi that just gets installed by Canada so it's not tied in say to PBS there's a bit of probably host file work that you have to do if you've used that before on another system it's you know it's not necessarily is.
Brian Vanderwende: as easy as a source built mpi but that's again, where I think we could offer we can certainly offer and car hbc channel that has those packages that are custom built for the systems.
Brian Vanderwende: And we can either put those in the default mpl or we could provide some instructions for how to clone the mpl and then install the the version of mpi for pi that's that's native to.
Brian Vanderwende: cheyenne or casper depending on which system you're you're planning to run on but really at this point I would like to introduce that system dependence, if we can at least in the data and the default libraries.
Brian Vanderwende: And that should really all be all it's necessary for desktop obviously there's like settings files and stuff like that and that's something I think that we.
Brian Vanderwende: We want to offer via some means right now we have a desk settings file in the current module but it's probably a bit out of date and needs a bit of revision so maybe we move that to the condom module maybe we keep it in a car i'm still need to play with that a little bit.
Brian Vanderwende: Alright well you know if anybody has any other questions feel free to send me an email.
Brian Vanderwende: Or you know reach out to.
Brian Vanderwende: CSP in any in any means that you like happy to take you back you know moving forward on this so thanks.
Sidd: Thank you, Brian if anybody at this point to like to unmute themselves or ask questions.
Garth D'Attilo: As it said I got a quick question this is guard yeah so Brian just just to be clear we're looking at trying to take this stuff out of what we would say the Python module now and make it part of a condo the condo.
Garth D'Attilo: setup but the functionality will remain the same, my concern being you know, there are a bunch of packages like even Carter pi which require external libraries to function like project.
Garth D'Attilo: But this would be focused on providing that that base level suite of the most commonly things it wouldn't necessarily change.
Garth D'Attilo: what's already available for scientists, but rather augment and then the second part of that is, if we do need additional things we would come to you guys and say Oh, this would be really helpful.
Garth D'Attilo: Going forward to to help our science, the you know help our scientists not have to build things.
Garth D'Attilo: Because kind of builds can be a little difficult, depending on what.
Garth D'Attilo: The package is.
Brian Vanderwende: yeah yeah exactly um you know we it's the same intent, as with the Python module that we're trying to offer some of those packages.
Brian Vanderwende: are really just a suite of packages that we know about the users, like us, at both very common levels and also things that aren't as commonly used but but are important.
Brian Vanderwende: into one window one libraries people on to maintain that themselves.
Brian Vanderwende: And certainly think you know, like the geography, libraries and things like that are are all in there.
Brian Vanderwende: They are using the condo binaries instead of you know buyers that we build from source.
Brian Vanderwende: In practice, I think that should be fine, but there may be edge use cases where that is an ideal so, then we could we could work with people if that's if they need to like somehow link into an executable their building or something like that.
Brian Vanderwende: But.
Brian Vanderwende: But yeah we're trying to offer the same functionality that we did with the source builds in the Python module history.
Garth D'Attilo: And then from the users perspective, they were really just looking at an additional sort of the couple commands here right and low condo and then.
Garth D'Attilo: load whatever environment, you want to within condo so that's going to be sort of the bigger change that people would see as they try to use this particular.
Brian Vanderwende: Basically, instead of doing module and Python and car pileup they would be module load kinda kinda activate and deal.
Garth D'Attilo: Okay, and so, as long as I think, as long as that's broadcast out this is great, you know we use Khan over the place, but we've had issues of replicating huge packages.
Garth D'Attilo: Right so eating, some of them are ridiculous specially the geography stuff or you might get like a landmass thing that's like gigabytes.
Garth D'Attilo: yeah so as long as we can work through some of those things and make sure we're not replicating all those and that's sounds like it's the tent.
Garth D'Attilo: This this sounds sounds good, but I just want to clarify, you know the user impact, to make sure that you know people, people will be aware of what they have to do to make the change i'm sure that'll be well publicized and then you know that the options for them, should they run in.
Garth D'Attilo: trouble or want to add something so if that's part of the documentation of hey contact this person about this, or just contact sizzle support if you if you need that.
Brian Vanderwende: yeah and I think that's one nice thing about moving the contest that it's a lot easier, you know.
Brian Vanderwende: Like you said there's still packages that are problematic but it's a lot easier to install something or update a package using content than it is using paper or a source build.
Brian Vanderwende: So you know if it's something where it's really a one off and maybe the person who just clone our package library and do a kinda you know kinda upgrade package or do.
Brian Vanderwende: A content install if it's not there already.
Brian Vanderwende: But if they think that they're going to need it moving forward for a while, or that they have colleagues that will need it, and certainly will be able to reach out to us and we'll just install it into our our base npl and the nice thing about kinda is that.
Brian Vanderwende: If you have multiple versions of it, it will use hard links as much as possible if you're doing it on the same on the same file system, so we should that's part of the reason that i'm not concerned about us offering a monthly cadence on the NPs because the file system, the file size.
Brian Vanderwende: implications of that aren't too bad as long as the number of packages don't change too much so that's also why I think we could offer like specialized versions of it as needed for people with their requests.
Garth D'Attilo: And then the final thing you know we've noticed that people, we can provide a cloud environment but they'll add their own right, as the user environment those things start to get slowly out of date and sometimes cause conflicts, as you update.
Garth D'Attilo: condos their plan to sort of help people mitigate that as they go forward.
Garth D'Attilo: With a set of instructions.
Brian Vanderwende: yeah I think I think that's probably the biggest documentation question is how much will dive into like in personal environment creation and maintenance, you know I think the condo documentation itself is is pretty good.
Brian Vanderwende: Obviously it people need to read it.
Brian Vanderwende: But I don't want to replicate stuff that's already out there and pretty good, so I think we need to figure out where there are in-car specific pieces of advice we can offer and then otherwise direct people towards the the official documentation talking about the same subjects.
Garth D'Attilo: That sounds good, thank you very much.
Brian Vanderwende: Thanks Carl.
Mick Coady: icahn right Brian you got a couple hands up yeah.
Sidd: john is.
clyne: That the thanks it so Brian i'm curious and I apologize if you if you've covered this already but.
clyne: You know, most of the user community that i'm familiar with it uses kinda they you know this install economy themselves, they set up their own environments.
clyne: What, what is the you know what is the incentive for these people to you know now move to a sizzle provided content environment, and you know, maybe do away with their existing customer environments mean is that a is that a goal is to try and get people to adopt.
clyne: A sizzle provided environment why Why would anyone want to do that now and i'm not being flippant but i'm just.
clyne: No, no.
Brian Vanderwende: that's it's a good question, so I think the goal is to take the load off of users who are who are who don't want to have to do that, but have had to do that.
Brian Vanderwende: So if somebody.
Brian Vanderwende: You know, really prefers to maintain their environment they like that level of control, we don't want to do anything that prohibits that I think it still would make sense for users to use our condom module.
Brian Vanderwende: Just because then they don't have to you know, keep updating the kind of binary itself but they could use that to manage all their environments and that should work pretty natively for them.
Brian Vanderwende: The the benefit for you know again for users who don't want to maintain their environments is we're going to keep things relatively update on this will have a couple of versions on offer and it'll be pretty easy to clone so.
Brian Vanderwende: Ideally people won't have to do that moving forward, given the breadth of the packages that we want to offer there's a file space consideration as well if you're using our npr library.
Brian Vanderwende: I think right now that clocks in about 2.53 gigabytes so you know if you're putting that it's not huge you know, but if you're putting a bunch of those, especially in your home directory like kinda likes to do by default that can you know that can not be ideal so.
Brian Vanderwende: My personal feeling is that we're not trying to force anybody to do the civil way of things you know, if you prefer to manage things yourself, then.
Brian Vanderwende: By all means that's that's that's great you'll you'll have a deep understanding of the infrastructure, but we really want to support people who don't want to do that, but it had to.
clyne: That all makes sense to me, and you know, I guess, I would say, as you're getting ready to prepare user documentation, maybe kind of calling out this issue you know you do have these as a user, you do have these options, and these are maybe the trade offs between the two.
Brian Vanderwende: That makes sense, thanks, Jonathan.
Brian Vanderwende: and David.
David Bailey: yeah I was gonna say that this space is a key thing because there's like every time I try to do a mini condo my home directory fills out, but no, my question is more about.
David Bailey: Who should we send them feedback to about this module directly to us is or how.
David Bailey: I just noticed I just tried it out and one of the problems is Python is loaded as a module by default and so when you do a module that makes you unload the other Python module to do it so just anyway, I just wondered, who we should be sending that to so.
Brian Vanderwende: Oh sure yeah and that's actually something we went back and forth on based on initial feedback, is whether to make those modules conflict or not.
Brian Vanderwende: I would say, until we officially announced this in the daily Bulletin feel free to just email me directly.
Brian Vanderwende: Since it kind of me and a couple of others in the team are leading the development of this but, once we announce it to the broad user Community then at that point, I think, submitting a ticket probably does make the most sense, though i'm happy to take emails at any point.
David Bailey: sounds good.
Brian Vanderwende: Alright, thanks everybody.
Mick Coady: bring you here to a couple of more questions in the chat sorry.
Brian Vanderwende: that's Okay, speaking of card up, I would encourage you want to have some of the common oh yeah so DJ we actually do that with the npl and.
Brian Vanderwende: I I can't remember if we've done that, in a way that's very generic or if it's very tied to that npr library that's a good point and we'll make sure that that also functions for the content and yell it probably doesn't at this point, but we will make sure that does that.
David John Gagne: So yes, sounds good, thank you.
Brian Vanderwende: And then I think.
Brian Vanderwende: Daniel just made a comment for people to digest so.
Brian Vanderwende: Probably leave time for other speakers.
Sidd: yeah Thank you Ben Thank you Dan.
Sidd: For them exciting common, but now it's your chance to present the one gpu training then go ahead.
Daniel Howard (CSG, NCAR): yep, let me just go ahead share my screen activate.
Daniel Howard (CSG, NCAR): Thanks window open.
Daniel Howard (CSG, NCAR): Okay, so here, here it is.
Daniel Howard (CSG, NCAR): See percent okay so uh yeah just briefly go over I already put in the chat every quest for a survey i'd like people to fill out, they can basically trying to develop a gpu training series for for researchers and stuff engineers here at anchor as well as a larger community.
Daniel Howard (CSG, NCAR): And so, essentially in order to best serve you we want to understand the audience needs in terms of some of these trainings So if you go ahead and click that link or.
Daniel Howard (CSG, NCAR): happen to have your phone, I just want to scan the qr code, you can go there and fill it out it's kind of long but, like most of them are short.
Daniel Howard (CSG, NCAR): questions, but if you have time to put some thought to open the ones that appreciate that nonetheless.
Daniel Howard (CSG, NCAR): Just try to focus on your personal experience you don't necessarily have to have gpu experience already.
Daniel Howard (CSG, NCAR): But if you have somebody that helps if they're just maybe something more advanced topics mountain be planning in the future, but uh.
Daniel Howard (CSG, NCAR): Otherwise, all the sponsors anonymous and you're welcome to contact me for additional personalized feedback if you'd like towards what these resources are planning to offer soon so.
Daniel Howard (CSG, NCAR): So we're currently testing and in the process of formulating and developing that training series on gpu computing.
Daniel Howard (CSG, NCAR): So topics that we were considering, including you know we're going to start with basic you know kind of basic introduction for like insured it up computing like like most maybe new users might expect.
Daniel Howard (CSG, NCAR): That might start with at least, just like especially just like you know why choose diffuser next scientific project and we try to orient that towards.
Daniel Howard (CSG, NCAR): You know, specific needs within like earth science community, but then you go on to like you know director based programming and programming using profiles, etc, as well as additional advanced topics, later on, which could could include like multi gpu programming specific.
Daniel Howard (CSG, NCAR): gpu Python package examples using perhaps Julian go to jail depending on interests.
Daniel Howard (CSG, NCAR): and other sort of tools that might be useful within that space, the the main impetus, at least for this is, you know, obviously we're having great show coming online soon and we'll have an increased amount of resources available for.
Daniel Howard (CSG, NCAR): gpu computing and car and so we'd like to.
Daniel Howard (CSG, NCAR): Basically, help enable that as much as possible.
Daniel Howard (CSG, NCAR): And, depending on how like how much experience and and and time you might have to to learn from those materials that we can help them form at least the best course of action for make taking advantage of the new hardware coming online.
Daniel Howard (CSG, NCAR): In addition to the V 100 we already have on casper.
Daniel Howard (CSG, NCAR): One of the main takeaways Of these, we want to make sure to to to include a part of the training.
Daniel Howard (CSG, NCAR): Is that, especially with like directed based programming is that it's relatively simple to to go through these through a a simple approach to to.
Daniel Howard (CSG, NCAR): refactor same many models are going to use gpus perhaps maybe might be the difficult question is whether or not it's effective for your particular scientific application.
Daniel Howard (CSG, NCAR): say if it's like the publicize large enough or safe, you might you might think you might want to go into kuda programming, which you might meet me more.
Daniel Howard (CSG, NCAR): And you're debating if you want that more refined control of the gpu and try to help users ascertain when and where it would be most useful them and help to help develop more skills towards effective gpu development with with the debate show and casper currently.
Daniel Howard (CSG, NCAR): So basically with the survey already shared in the chat you know we just want to know what needs, do you have with respect to gpu doing.
Daniel Howard (CSG, NCAR): So we already.
Daniel Howard (CSG, NCAR): had quite a bit with in terms of talking with Julie cat and kind of Paul about their project Python work and X Dev work leading trainings.
Daniel Howard (CSG, NCAR): And the in Python ecosystem and learning from like their discussions with them and their trainings they've already helped you know their main key takeaway was knowing your audience, hence the survey.
Daniel Howard (CSG, NCAR): I know it's not bother the representative of everyone within the empire and hbc user community but, nonetheless, you know any information, we can get would be really appreciated would help inform the type of resources, that you will get in the future.
Daniel Howard (CSG, NCAR): So, and so doing, and we want to make sure that we're preparing these these topics materials to be clear.
Daniel Howard (CSG, NCAR): And and and make sure we're detailing exactly we're playing be covering ahead of time, so that you know you know exactly what you might be getting into doing each session and can pick and choose what would be most beneficial towards you.
Daniel Howard (CSG, NCAR): So say you might be already a profession gpu program you might not be interested in the introductory sessions.
Daniel Howard (CSG, NCAR): But you can plan to attend the later ones, and we were already planning to do the development of these series to be called mentoring sessions and thus then allow them to be archived and and watched passively.
Daniel Howard (CSG, NCAR): asynchronously at your own learning pace, apparently, according to the project by the a team apparently 10 times more people watch the recordings rather than participating live.
Daniel Howard (CSG, NCAR): So these are some of the experiences that we're playing to incorporate towards how we implement our virtual training series.
Daniel Howard (CSG, NCAR): As well we'll plan have a code of conduct, just to make sure that the run well and may projects.
Daniel Howard (CSG, NCAR): could be a approach to include across the trainings for.
Daniel Howard (CSG, NCAR): Eating learning.
Daniel Howard (CSG, NCAR): But the down the line you were planning to develop this this like a series of seminars and it will be like archived and you could regularly say like we offer them, but over time, will need to.
Daniel Howard (CSG, NCAR): check the validity of relevance of each of the training materials offer offered as a best practices and new tool say come out within the gpu computing space.
Daniel Howard (CSG, NCAR): So currently available training materials we've already seen some brief actually had order the lead alongside us a gpu workshop, as well as some secret trainees, to help leading the session developing them tales there that's already available on github and see what was offered.
Daniel Howard (CSG, NCAR): So, especially from that session basically indicated that we need to split separate material, the more bite sized chunks.
Daniel Howard (CSG, NCAR): The interactive live coding examples, though useful there's difficult to coordinate and authentic significant longer than expected and given say the time constraints and many people to commit to a seminar series might might constrain decisions to to include such interactive sections.
Daniel Howard (CSG, NCAR): Not not including them but that's part of the one of the questions were having this over here, to see if you prefer, say live coding type seminars i'm learning gpu coding or for.
Daniel Howard (CSG, NCAR): More asynchronous or or just live engagement, without necessarily a interactive coding section part and maybe you know, on your own time, you can test out things done.
Daniel Howard (CSG, NCAR): But a additionally within say if we did have those live coaching sessions setting up those environments and even access to gpus can be expensive, expensive and feasible for large groups.
Daniel Howard (CSG, NCAR): You might not want to like shut down portions of like the reserve portions of the gpus on our current systems to make those.
Daniel Howard (CSG, NCAR): webinars available to a larger audience, but it would depend on how many people end up wanting to attend these these sessions nonetheless many rituals already exists out online, as you know, imagine, many people have probably explored somewhat their.
Daniel Howard (CSG, NCAR): GP computing and in considered that their own utilization of which.
Daniel Howard (CSG, NCAR): But our goal is ideally to contribute specifically within the earth science in maine and build upon those materials that can relate to that space, and that is to.
Daniel Howard (CSG, NCAR): replicate what's already out there, so so trying to to build resources that that would specifically serve earth scientists and researchers.
Daniel Howard (CSG, NCAR): beyond just the general practitioner everything gpu computing unless there's already resources here which you're welcome to if you're not already seen them that they're very, very helpful on terms of what might be useful today in terms of seeing what's what's out there.
Daniel Howard (CSG, NCAR): But already you know well known, especially with nsf.
Daniel Howard (CSG, NCAR): emphasis towards up computing and the funding that increased funding towards towards which many earth earth science projects are already incorporating gpus into usage and deployment Wharf CSM darts best eddie that they have may they already are using gpus and various manners.
Daniel Howard (CSG, NCAR): So during this training we've already discussed somewhat and we're evaluating the the idea of extracting kernels some some of these models or other places that might be relevant to highlight development approaches and their experiences with n.
Daniel Howard (CSG, NCAR): gpu computing and see how see how well we can leverage.
Daniel Howard (CSG, NCAR): Those.
Daniel Howard (CSG, NCAR): Already already implemented best practices towards sharing with others and new year's, this is about how they might approach GB computing within their own projects.
Daniel Howard (CSG, NCAR): So other questions that remain, though in that space is are the current gpu needs within the Community, more of a.
Daniel Howard (CSG, NCAR): tactical variety and like learning the software implementation like how to program things, is it the access to hardware which.
Daniel Howard (CSG, NCAR): Obviously, dealing with the relatively soon, or is it, like other problems like I about bottlenecks are softer problems like you know software lifecycle management that those are those are pertinent.
Daniel Howard (CSG, NCAR): Questions that would need to be addressed in terms of what might be most important to cover in the training they were aiming to develop.
Daniel Howard (CSG, NCAR): And I think one of the more fundamental questions, I think we even talked I briefly mentioned earlier, already was just.
Daniel Howard (CSG, NCAR): dmo identify which scientific problems and other senses sciences can benefit from gpu computing.
Daniel Howard (CSG, NCAR): Helping users determine whether or not you know the poem size is large enough that they're able to like the code is like.
Daniel Howard (CSG, NCAR): parallel lines parallelizable in such a way that I can be implemented on a gpu effectively and also the data movement, then back and forth under the gpu is actually manageable says that you know when you do make that we factoring.
Daniel Howard (CSG, NCAR): factoring process for your model, perhaps you actually are achieving a speed up and not ending up with a slowdown, because that can sometimes happen, depending on the the applicability of other particular models to work towards GP computing.
Daniel Howard (CSG, NCAR): and the last thing that's often at least i've been considering, at least in terms of like that there's often a.
Daniel Howard (CSG, NCAR): disconnect between, say, sometimes it needs between like a physics based up competing.
Daniel Howard (CSG, NCAR): Projects versus like Ai ml based up computer projects and how much are those in fact distinct in terms of what maybe should be taught within this.
Daniel Howard (CSG, NCAR): You know seminar series or project we're blaming delayed in terms of the types of skills that are necessary for say like you know, a compiled language type fortran.
Daniel Howard (CSG, NCAR): c c++ approach versus you know many of the Python Julia type packages that already have implementations in a sense within them, but they're just different use cases and how you would interface with the gpu gpu and each each paradigm.
Daniel Howard (CSG, NCAR): And so, trying to adjust the needs of both of those sides of Community might be important towards deciphering the approaches will take in terms of what content will be producing and adding over time to the to the gpu training series.
Daniel Howard (CSG, NCAR): Now that's the main main point and listen just engaging with you all here today is actually just trying to learn from you what you all need so again, if you can fill out that survey.
Daniel Howard (CSG, NCAR): Today, while i'm talking about finishing up this integrating that that I very much appreciate that and you're more than welcome to still again reach out to me for additional feedback, as you might find relevant but uh Thank you and happy to take any questions.
Thank you.
Sidd: everybody else, please.
Sidd: Do yourself or ask question in the chat make it glide.
Mick Coady: yeah, I just wanted to let folks know those of you who in this meeting will be at the gpu tiger team meeting later this week we'll review this again and go over some of these points in more detail so.
Mick Coady: that's it thanks it.
Sidd: Right john.
clyne: yeah thanks for that.
clyne: i'm just curious, I may have missed this, but what is the timeframe for the one do you hope to offer the training series.
Daniel Howard (CSG, NCAR): yeah so like we're currently debating between December and January, instead of starting the series on this actually one of the questions.
Daniel Howard (CSG, NCAR): The other complication currently is just that there's a lot of upcoming conferences and holiday time obviously that might impact people's like preference towards say like starting such a such a series doing which so.
Daniel Howard (CSG, NCAR): It was thought perhaps you'd have like the introductory sessions like starting out like saying December and then we'd like.
Daniel Howard (CSG, NCAR): Probably continue those out a little bit in January and they get to more advanced topics, later on, but it might be worthwhile to start like in.
Daniel Howard (CSG, NCAR): January, just to make sure we have all of our ducks in a row, for example, terms of like what we plan to be offering and making sure the material is a high enough quality for for.
Daniel Howard (CSG, NCAR): For community but um yeah if you think this is one of the survey questions you're welcome to provide feedback to enter your own preference but we're able to ask start offering say i'd say December in terms of the introductory material.
clyne: Thank you.
Sidd: Any more questions for them.
Sidd: If not, let me open thinking that and let me open this floor for any general questions feedback or the topic that you like, to be covered in the future and.
Daniel Howard (CSG, NCAR): there's also one question that chatter just saw from David.
Daniel Howard (CSG, NCAR): Yes, I will we have a baby duration to test on.
Daniel Howard (CSG, NCAR): Not that i'm aware of.
Daniel Howard (CSG, NCAR): there's some discussions, I think, say, with nvidia for like a that, where some people have had access to a test system, therefore, the 100 gpus.
Daniel Howard (CSG, NCAR): But.
Daniel Howard (CSG, NCAR): there's not a current pressing need for.
Daniel Howard (CSG, NCAR): For example of safe access to a 100 100 hundreds.
Daniel Howard (CSG, NCAR): Unless.
Daniel Howard (CSG, NCAR): But you, you can maybe make that preference now in the survey, at least in terms of if that would be highly desirable towards having an access to a 100 type nodes versus the current view and hundreds.
Brian Vanderwende: And I guess More generally speaking there, there will be a test machine for great job but.
Brian Vanderwende: that the time the timing of that and coordinating that for the for these sessions, I think, is what Daniel was getting out but I don't think these sessions are going to be built around that, at least not in the current timeframe.
Sidd: Thank you, Dan.
Sidd: Any other questions concerns.
Sidd: Our topic that you would like to discuss our discuss in the future in hoc meeting you can either unmute yourself and speak or.
Sidd: Just type in the chat.
Mick Coady: said, where you going to have john last week.
Sidd: Is presentation i'm sorry, my apologies, we have another topic to cover shot.
Sidd: Thank you, Nick john Please go ahead with your previous direct drives, that is, the upcoming gene PBS changes.
John Blaas (NCAR): You know just one moment.
John Blaas (NCAR): coming in the December maintenance will be putting in place a direct right hook on cheyenne so this direct right hook has been in place on casper since early January of this year.
John Blaas (NCAR): But we plan to put this on cheyenne as well, so essentially what this will do is they will look at your PBS stash K invocation and if you have specified.
John Blaas (NCAR): Oh, we are yo it'll you know tap attack on the D directive for you and then Similarly, if you just select, for you know only output, a logging or only error logging.
John Blaas (NCAR): And so that will allow you to direct right to your final location, instead of locally on the node this helps you when you know, for instance, a bunch of your note bunch of the nodes in your job go down, you lose your output, because it was stored locally.
John Blaas (NCAR): So additionally this right hook will also ensure that you're not trying to write your output and error logging to any of those local temporary directory directory or file locations, so you can see a listing of some of the locations that we check for.
John Blaas (NCAR): But that's about it.
John Blaas (NCAR): Any questions.
Sidd: Any questions from dawn.
Sidd: Anyway, this is well we're beta because most often you wonder where is your standard out standard error is going, unless you directly want you to do the final equation like would it be option.
Sidd: it's better that as and when it produces.
Sidd: Besides it's being offered for little it will be directed dumped into your final location.
Sidd: It looks like they're doing any question for john Thank you john so let me open the floor for any other feedback any other topic that you would like to cover in the future, or.
Sidd: Integrating or any topic of discussion that you would like to bring us bring up for now.
Sidd: P sounds like.
Sidd: it's already done.
Sidd: So we are.
Sidd: Nearly the top of the hour, so thank you for joining in the same hub and see you next month.
bye.
Brian Vanderwende: So, really.