1
00:00:01.380 --> 00:00:03.120
Mick Coady: Okay start over again here.
2
00:00:04.290 --> 00:00:11.280
Mick Coady: want to remind everybody that this is my last meeting the fact, three weeks from.
3
00:00:12.990 --> 00:00:15.030
Mick Coady: Tomorrow is my last day.
4
00:00:16.170 --> 00:00:27.600
Mick Coady: And i'll miss this group and lead, you know meeting with you already regularly but worry will be taking over as the lead or coordinator for the for the group.
5
00:00:28.710 --> 00:00:37.230
Mick Coady: In fact he'll probably start before this meetings over with because I have i'll have to leave early to make a doctor's appointment so.
6
00:00:38.460 --> 00:00:54.570
Mick Coady: So that that's not awkward in the middle i'll just say farewell so on right now, and look forward to meeting and seeing you all down the road sometime I hope sincerely hope so, so thanks for all your participation and help.
7
00:00:55.710 --> 00:01:04.380
Mick Coady: Over these last several months and i'm sure you're going to be continuing to do some really cool things.
8
00:01:05.520 --> 00:01:08.160
Mick Coady: In the months and year ahead so.
9
00:01:09.930 --> 00:01:24.060
Mick Coady: So then on to the next agenda item which is Brian is going to say a few words about gpu support in the CRATE containers container as it is today so Brian take it away.
10
00:01:25.380 --> 00:01:26.250
Brian Vanderwende: yeah thanks MAC.
11
00:01:27.660 --> 00:01:30.120
Brian Vanderwende: And i'll try to follow the tone of that.
12
00:01:33.120 --> 00:01:48.960
Brian Vanderwende: yeah so as some of you may be aware, Sid primarily worked with some collaborators at cray HP to get a container stood up and installed and available to everybody on.
13
00:01:49.590 --> 00:01:58.950
Brian Vanderwende: On our cheyenne and casper systems and that's been really useful for supporting efforts and some compiler testing efforts that are going on on the cpu side.
14
00:01:59.940 --> 00:02:05.520
Brian Vanderwende: But maybe about I would say over the holidays right before the holidays, we worked with.
15
00:02:06.120 --> 00:02:19.650
Brian Vanderwende: Some HP folks and got the container in a state where it can also basically it imports, the nvidia gpu driver and provide some limited support for compiling and running applications using.
16
00:02:20.640 --> 00:02:29.220
Brian Vanderwende: gpu offload and the cray cray compilers so certainly you're good you know you could use the nvidia compilers in the container that's all by mounted in.
17
00:02:30.060 --> 00:02:36.930
Brian Vanderwende: I don't know what value that provides, but this This gives you kind of a sneak sneak peek into the.
18
00:02:37.380 --> 00:02:45.930
Brian Vanderwende: Support from the CRATE compilers which, as it is in that release is fairly limited for nvidia targets they package.
19
00:02:46.590 --> 00:03:03.270
Brian Vanderwende: A package scientific libraries, with support for the amd gpus but we're still waiting on the equivalent support for the nvidia gpus, so I think that the level of support will be a lot higher in a future release of the container hopefully maybe in about a month's time we'll get that.
20
00:03:04.320 --> 00:03:11.670
Brian Vanderwende: But it's available now there's instructions that we're going to release I posted them on the slack and some people are.
21
00:03:12.000 --> 00:03:26.370
Brian Vanderwende: The gpu slack But some people are members of we're going to put some documentation out there either this afternoon or tomorrow, probably paired up with a daily Bulletin item so keep an eye out for that and feel free to give it a test try.
22
00:03:27.690 --> 00:03:35.610
Brian Vanderwende: The as far as crazy gpu support they have open ACC support in the fortran compiler, but that is relatively old.
23
00:03:36.780 --> 00:03:44.610
Brian Vanderwende: Open ACC standard I think it's basically about two point O sport they're working on getting into three point over the coming year, but.
24
00:03:45.060 --> 00:03:53.730
Brian Vanderwende: You know that's an iterative process that they're working on and the container is again a little bit behind where the the current release of the cray compiler suite is.
25
00:03:54.870 --> 00:04:04.380
Brian Vanderwende: So yeah there'll be more details in that documentation and hopefully the next release will have a bit more open ACC support, as well as that open MP offload support as well.
26
00:04:06.000 --> 00:04:09.870
Brian Vanderwende: happy to take any questions but otherwise just keep an eye out for that and the daily Bulletin.
27
00:04:14.520 --> 00:04:17.790
Brian Vanderwende: Alright, well, I guess, very next feel free to push forward.
28
00:04:18.870 --> 00:04:31.530
Mick Coady: Okay i'll do it doesn't sound like there's any questions so i'll pick I put roy's name there in case I had to split before here so um let's see we'll.
29
00:04:32.580 --> 00:04:33.210
Mick Coady: go there.
30
00:04:34.230 --> 00:04:39.030
Mick Coady: So we haven't talked about the gpu development Q on casper.
31
00:04:40.380 --> 00:04:48.030
Mick Coady: For a few months, I just wanted to I was curious myself how much usage did.
32
00:04:49.080 --> 00:04:49.800
Mick Coady: It had been getting.
33
00:04:50.880 --> 00:04:59.910
Mick Coady: So you can see that, over the past one and two months it's been getting fairly fairly regular use there's some.
34
00:05:02.610 --> 00:05:18.510
Mick Coady: familiar faces and names in there, particularly somebody named Siena I don't know what that's about that probably the heaviest user there, but many of the folks that are here in this meeting, and part of the GT GT or rig regular.
35
00:05:19.650 --> 00:05:22.830
Mick Coady: users and take advantage of that Q so.
36
00:05:23.940 --> 00:05:36.060
Mick Coady: I would suspect, and I think, as long as there's support for it and the usage is there will continue to provide that that cue it seems certainly beginning.
37
00:05:37.200 --> 00:05:47.550
Mick Coady: I think the kind of usage, we would hope for and expect if you have any feedback for us, please let us know and we'll take some feedback right now as matter of fact, if.
38
00:05:49.530 --> 00:05:52.590
Mick Coady: anybody has any anything to add to that.
39
00:05:58.620 --> 00:05:59.370
Mick Coady: Good one brain.
40
00:06:01.620 --> 00:06:12.240
Mick Coady: Okay, and sound like it so Okay, then we'll continue move on and really the big item, for I think the.
41
00:06:13.020 --> 00:06:29.760
Mick Coady: agenda item that should I was expecting to take the most time today is that what asked Daniel to give us an update on where we we've where our efforts have been on planning for the gpu training tutorials that will be starting.
42
00:06:30.960 --> 00:06:45.780
Mick Coady: Here, in early next month it's I think it's very fair to say that this has been the focus for at least PSG in supreme and seeing that have been very generous with their time and support as well.
43
00:06:46.800 --> 00:06:47.490
Mick Coady: Here so.
44
00:06:48.630 --> 00:07:00.390
Mick Coady: And I thank them for that it's very likely I won't be here by the end of Daniels talk so i'll say so on now and take care of everybody, Daniel take it away.
45
00:07:09.600 --> 00:07:11.910
Mick Coady: Let me know if I need to make you co host Daniel.
46
00:07:12.990 --> 00:07:15.030
Daniel Howard (CSG, NCAR): My MIC is like not working well i'm going.
47
00:07:17.100 --> 00:07:17.310
Mick Coady: i'm.
48
00:07:17.400 --> 00:07:18.600
Daniel Howard (CSG, NCAR): Looking for the green box.
49
00:07:19.650 --> 00:07:23.580
Daniel Howard (CSG, NCAR): But if you can like me and i'm going to share my screen and be helpful, saying I can't.
50
00:07:25.200 --> 00:07:29.910
Mick Coady: Just made you co host Okay, so you should be able to.
51
00:07:31.080 --> 00:07:31.260
Daniel Howard (CSG, NCAR): You know.
52
00:07:31.980 --> 00:07:34.590
Mick Coady: rory rory may have to stop sharing.
53
00:07:35.280 --> 00:07:37.260
Daniel Howard (CSG, NCAR): I think I just I just forced him to stop so.
54
00:07:37.800 --> 00:07:38.070
Okay.
55
00:07:40.140 --> 00:07:41.400
Daniel Howard (CSG, NCAR): So yeah any case.
56
00:07:42.480 --> 00:07:53.610
Daniel Howard (CSG, NCAR): So we've been working on basically developing a training series, and in the meantime I guess Intel that starts in February, you know there's also some Community resources for computing that i'll share this.
57
00:07:54.540 --> 00:08:00.540
Daniel Howard (CSG, NCAR): refill update, as well as the results from the survey that we did a while back on do for computing but.
58
00:08:01.380 --> 00:08:04.590
Daniel Howard (CSG, NCAR): First, just you know, probably don't need to remind people too much here, but you know just.
59
00:08:05.340 --> 00:08:14.400
Daniel Howard (CSG, NCAR): You know kind of like white up competing still relevant you know the these accelerators are one of the most performant in terms of theoretical performance that is achievable and so.
60
00:08:14.970 --> 00:08:29.460
Daniel Howard (CSG, NCAR): Encouraging users to to have a broader adoption of these resources will be appropriate, so, as you can see, you know, we have the amd and nvidia gpus there as well as our next gen cpu the MB epic there and the gpus there, at least in terms of.
61
00:08:30.570 --> 00:08:38.370
Daniel Howard (CSG, NCAR): pick up on performance obviously i'm far exceeding that where it really shines those is the memory bandwidth so making sure that we can.
62
00:08:39.690 --> 00:08:52.350
Daniel Howard (CSG, NCAR): support some of the applications that are able to achieve and utilize that band was effectively is as important in terms of the training materials will be developing towards encouraging users to effectively use these new hardware systems.
63
00:08:53.640 --> 00:09:00.060
Daniel Howard (CSG, NCAR): So, nonetheless, you know there's already been past collaborations in the space in terms of.
64
00:09:02.460 --> 00:09:10.890
Daniel Howard (CSG, NCAR): Using gpus in whether modeling, and these are just some examples of that of these properties putting two recent multi port 10 workshop series.
65
00:09:11.970 --> 00:09:12.330
Daniel Howard (CSG, NCAR): So.
66
00:09:13.170 --> 00:09:14.850
Daniel Howard (CSG, NCAR): he'd be able to be a part of that community I think it's.
67
00:09:14.850 --> 00:09:22.170
Daniel Howard (CSG, NCAR): Important to encouraging us beyond just the scoop and to also join the Community and hopefully contribute to that to the space effectively.
68
00:09:22.920 --> 00:09:32.250
Daniel Howard (CSG, NCAR): it's definitely been our best interest so nonetheless you know we have previously done a gpu training workshop series of more.
69
00:09:33.060 --> 00:09:43.380
Daniel Howard (CSG, NCAR): smaller scale, with thanks to stand to present other student trainees, as it gets out there that we are, we will be updating soon with the new material for the upcoming one setting in February.
70
00:09:45.960 --> 00:09:48.810
Daniel Howard (CSG, NCAR): down the line will be used that that's trading.
71
00:09:49.440 --> 00:10:00.660
Daniel Howard (CSG, NCAR): github for further updates as technology changes, as well as just encouraging a community of collaboration there as well as new as this this this group already exists here to the GT and and other.
72
00:10:01.110 --> 00:10:08.220
Daniel Howard (CSG, NCAR): avenues, such as like the open SEC slack channel, for example, that people could not be aware of and participate in in terms of.
73
00:10:09.660 --> 00:10:19.680
Daniel Howard (CSG, NCAR): You know, learning the space, but nonetheless to do the development of this course, we already have a curated Google drive i'll put a link here in the chat.
74
00:10:21.660 --> 00:10:25.770
Daniel Howard (CSG, NCAR): Basically, to this presentation document so you can go in and click the links here if you'd like.
75
00:10:27.000 --> 00:10:35.520
Daniel Howard (CSG, NCAR): You can you can go there and that has basically a large selection of documents, but at least relevant and hopefully said I recent some more recent than others about.
76
00:10:36.390 --> 00:10:44.010
Daniel Howard (CSG, NCAR): recruiting that, hopefully, will you will find useful beyond say, and we will be using some as references in terms of the development of materials here.
77
00:10:44.580 --> 00:10:51.990
Daniel Howard (CSG, NCAR): For these training workshop series there's a simple documentation page, for example, linking to various like no.
78
00:10:52.410 --> 00:10:57.060
Daniel Howard (CSG, NCAR): documentation pages that have video provides for like their their APIs and new faces and such.
79
00:10:57.600 --> 00:11:04.710
Daniel Howard (CSG, NCAR): But additionally there's just no other training courses like that have been offered that overage and nurse them up you relevant for people to explore.
80
00:11:05.550 --> 00:11:13.500
Daniel Howard (CSG, NCAR): For example, there's like one in depth one going kuda but since kuda is very demanding in terms of a learning experience.
81
00:11:14.100 --> 00:11:29.880
Daniel Howard (CSG, NCAR): we're we're we're studying more towards the opening CC and directed based approaches, because most of our training that we plan to be offering unless you feel free to explore some of these materials at your leisure and we hope that someday will be useful, especially to this group here.
82
00:11:31.410 --> 00:11:44.670
Daniel Howard (CSG, NCAR): Beyond that So what should we be adding to this space, because you know, obviously from highlighting the external material there's already a lot there so from that survey, we did recently here, so the results from that.
83
00:11:45.870 --> 00:11:58.260
Daniel Howard (CSG, NCAR): We mostly had domain, scientists and, and these are stuff engineers contributing to that survey strong interest in fortran and Python and less so, and C c++ and much, much less than other languages.
84
00:12:00.000 --> 00:12:06.600
Daniel Howard (CSG, NCAR): i'm excited to hopefully see some response and Julia but there's not many, unfortunately, but maybe that will change over the coming year.
85
00:12:07.800 --> 00:12:08.550
Daniel Howard (CSG, NCAR): In any case.
86
00:12:09.720 --> 00:12:16.830
Daniel Howard (CSG, NCAR): We only have about a therapy least according to this survey that we did responding a current usage of gpu or workflows and model development.
87
00:12:18.480 --> 00:12:28.050
Daniel Howard (CSG, NCAR): Not unless the majority of them do still anticipate a great need for companies up computing so that is something we want to be make sure, making sure we are satisfying in terms of the material we preparing.
88
00:12:29.280 --> 00:12:34.500
Daniel Howard (CSG, NCAR): Again, people, preferring fortran and Python still in terms of learning the space.
89
00:12:35.190 --> 00:12:46.950
Daniel Howard (CSG, NCAR): So I think that just comes with the territory ecosystem sciences so but that's kind of the point in terms of our trainings will probably focus more and that side of things, given that most of the models which will be working with are still in fortran.
90
00:12:48.540 --> 00:12:52.650
Daniel Howard (CSG, NCAR): and beyond that you know just helping people to like.
91
00:12:52.680 --> 00:12:53.970
Rory Kelly: them during the space you.
92
00:12:53.970 --> 00:13:08.730
Daniel Howard (CSG, NCAR): know that you can see some things up the results here for the live training versus actually a majority are indicating a preference on offense for like a synchronous learning do like X addressing our coverage cereals, so we want to make sure that we're developing a good.
93
00:13:09.780 --> 00:13:19.950
Daniel Howard (CSG, NCAR): platform that people cannot access outside of the live trainings as well, in addition to what we're doing in person via zoom for for these trainings.
94
00:13:21.510 --> 00:13:22.080
Daniel Howard (CSG, NCAR): So.
95
00:13:23.520 --> 00:13:27.960
Daniel Howard (CSG, NCAR): Just summarizing all of that, you know fortunate and Python based techniques ideal.
96
00:13:30.120 --> 00:13:42.390
Daniel Howard (CSG, NCAR): Nonetheless, the average ability, I couldn't overall from the group from that first set of results was about 1.7 out of five it's about 1.5 from researchers and doing a fit for for software engineers.
97
00:13:43.440 --> 00:13:48.240
Daniel Howard (CSG, NCAR): So there is some of the discrepancy in terms of trying to meet those needs across those types of groups.
98
00:13:49.650 --> 00:13:57.600
Daniel Howard (CSG, NCAR): So, nonetheless, certainly be crying concerns and Sunday open ended questions was achieving portable code and funding sources as well as.
99
00:13:58.110 --> 00:14:11.010
Daniel Howard (CSG, NCAR): Having just not sufficient time for for doing a lot of the development work and refactoring required to port the gpus and our emphasis on the the database a piece, which, of which often are far less time to do that.
100
00:14:12.570 --> 00:14:23.310
Daniel Howard (CSG, NCAR): Conversion deadline, perhaps you might offer further training and say kudos for like you know highest amount of optimization possible, but we want to at least get people started as fast as possible, so doing that.
101
00:14:23.880 --> 00:14:30.030
Daniel Howard (CSG, NCAR): database of coach I think would be our best course of episodes of technical development of the materials.
102
00:14:31.230 --> 00:14:40.830
Daniel Howard (CSG, NCAR): So, in terms of the the that one aside for funding sources that there's some you know, obviously outside of my most most everyone's awareness to you know to like nsf grant proposals and.
103
00:14:41.250 --> 00:14:44.940
Daniel Howard (CSG, NCAR): Opportunities you know the Microsoft interest, perhaps, and looking into songs.
104
00:14:45.300 --> 00:14:50.280
Daniel Howard (CSG, NCAR): And videos opportunities, you know they have this applied research accelerated program the US of a hardware program you say you want like a.
105
00:14:50.670 --> 00:14:58.050
Daniel Howard (CSG, NCAR): workstation with me on that that's separate from the cluster that that might be an opportunity, perhaps to explore they have two different I didn't like the second one there but.
106
00:14:58.770 --> 00:15:11.940
Daniel Howard (CSG, NCAR): You might look into some of the opportunities and we plan to advertise that accordingly to encourage you know additional support craftsman video themselves where this specific program or even potentially provide video.
107
00:15:13.110 --> 00:15:17.280
Daniel Howard (CSG, NCAR): Software architects to support some projects if if warranted.
108
00:15:18.780 --> 00:15:26.010
Daniel Howard (CSG, NCAR): So did you be training series itself, we already have the the topics we've covered are on the left side of the screen you're welcome to.
109
00:15:26.460 --> 00:15:33.330
Daniel Howard (CSG, NCAR): Follow the qr code or click the link if you pull up this slide document to see a more complete comprehensive this thing of all these.
110
00:15:33.810 --> 00:15:42.090
Daniel Howard (CSG, NCAR): trainings are plenty of you're covering the last set of like Python more oriented section that's still somewhat in flux, so that might change down the line.
111
00:15:43.290 --> 00:15:48.960
Daniel Howard (CSG, NCAR): There wasn't a lot of interest in number from the survey more than like two pi and not by sight buying so.
112
00:15:49.680 --> 00:16:05.610
Daniel Howard (CSG, NCAR): But we'll see we'll see how that pans out down the line that there's still so obviously what might be a need for across all those Games and especially with respect to some machine learning and data science workflows as that's becoming more popular and on the earth science.
113
00:16:07.830 --> 00:16:09.030
Daniel Howard (CSG, NCAR): frontier of research space.
114
00:16:10.170 --> 00:16:20.010
Daniel Howard (CSG, NCAR): Nonetheless, I just want to make a small plug you have that session hands on with that that's a cute so you know these are profiling and analysis tools for for models and relevant.
115
00:16:20.550 --> 00:16:29.850
Daniel Howard (CSG, NCAR): I sent myself to present that session because mcs je w June 30 if you have a work, I would like the opportunity to be able to to work with it an npr model.
116
00:16:30.510 --> 00:16:44.820
Daniel Howard (CSG, NCAR): Maybe that hasn't been looked at recently in terms of stp performance and ideally use that as an example, within this session for for this training program so if you'd like to collaborate with me and others perhaps on.
117
00:16:45.420 --> 00:16:54.150
Daniel Howard (CSG, NCAR): going through the process of analyzing that performance and then allowing me to then share some of those results and the process of doing that.
118
00:16:54.780 --> 00:17:03.030
Daniel Howard (CSG, NCAR): Within this session that we I think it'd be very interesting to our community to have as a real life example, as opposed to maybe some of your examples that.
119
00:17:03.330 --> 00:17:13.380
Daniel Howard (CSG, NCAR): I already have available, but maybe not might not be as interesting Nonetheless, if you feel feel free to reach out reach out to me if you have a interest in doing that and happy to take your questions, thank you.
120
00:17:31.740 --> 00:17:33.690
Rory Kelly: Okay doesn't sound like their questions for.
121
00:17:33.780 --> 00:17:40.410
Rory Kelly: Daniel at the moment, so i'll go back to the rest of the agenda, and I think this kind of more or less is.
122
00:17:42.540 --> 00:17:53.280
Rory Kelly: The rest of the agenda so we'll just have some closing remarks and comments I guess here at the end, first of all, if you know does anybody have anything they want to bring up in this forum for this particular meeting.
123
00:17:54.300 --> 00:17:56.100
Rory Kelly: i'll give you a few seconds to respond here.
124
00:18:01.440 --> 00:18:04.170
Rory Kelly: Could someone maybe say something so I know that my microphone is working.
125
00:18:05.340 --> 00:18:06.060
Brian Dobbins: it's working.
126
00:18:06.510 --> 00:18:08.040
Rory Kelly: All right, you guys are the best thanks.
127
00:18:11.790 --> 00:18:16.620
Rory Kelly: Well, I guess, as there are no comments or questions or anything that moment, I want to just take a minute.
128
00:18:17.640 --> 00:18:25.320
Rory Kelly: To say what you know as Nick said at the beginning he's going to transition out of this meeting because he's retiring and it's going to fall to me to lead it in the future.
129
00:18:25.950 --> 00:18:36.690
Rory Kelly: And in this meeting, and a number of meetings we've been talking recently about you know how we can sort of refocus things originally we were supposed to have test hardware on the ground for the incoming machine.
130
00:18:37.140 --> 00:18:45.090
Rory Kelly: And we were supposed to be preparing for Rachel to actually be in the lab and, obviously, because of delays the schedule is shifted but i'd like for these meetings to not lose momentum.
131
00:18:45.720 --> 00:18:56.700
Rory Kelly: So I think you know, maybe next month i'll prepare some thoughts about how we can refocus this and keep it useful to the Community, you know just looking through the participants, the people that are on they were.
132
00:18:57.270 --> 00:19:04.920
Rory Kelly: Really biased towards sizzle we're really biased towards the people that are sort of you know deployment people and less so.
133
00:19:05.160 --> 00:19:11.580
Rory Kelly: You know the people attending this meeting or less so, of people that are developing in using and running gpu code doing science on gpus.
134
00:19:12.240 --> 00:19:23.640
Rory Kelly: So a few of the things i'd like to see going forward with this meeting, are you know, involving more people from other labs more people that are actually doing science and using gpus and production.
135
00:19:25.080 --> 00:19:33.930
Rory Kelly: You know, trying to make this meeting useful towards targeting people that are currently developing or want to develop and don't know how to develop on gpus.
136
00:19:34.320 --> 00:19:41.730
Rory Kelly: I think a lot of the training that that Daniel is in charge of the Daniels taking up here will be helpful towards that but i'd like to listen other people's ideas.
137
00:19:42.570 --> 00:19:53.070
Rory Kelly: towards you know sort of making this meeting, more useful to as we're kind of waiting for hardware to come in, I know it's kind of a strange gap to fill because we had all this momentum and then suddenly we don't have a machine.
138
00:19:53.910 --> 00:20:01.140
Rory Kelly: But I think it gives us an opportunity to focus and be ready for when the machine eventually shows up, so I guess.
139
00:20:02.250 --> 00:20:10.860
Rory Kelly: That said, we know we don't need to waste too much of your time today i'll try to put together an agenda for us to talk about that next month, but in the meantime, if you have ideas of.
140
00:20:11.190 --> 00:20:19.590
Rory Kelly: stuff you'd like to see feel free to contact me directly or save up thoughts, you know, in your own notes for maybe a meeting next month, and we can talk about where to go from here.
141
00:20:21.600 --> 00:20:27.900
Sidd: What he said, you also have a wiki page, perhaps you can consider.
142
00:20:30.030 --> 00:20:35.610
Rory Kelly: Well, I don't have a wiki page but MC MC and if he does, you can transition it to be as well and yeah.
143
00:20:38.100 --> 00:20:41.610
Sidd: make me planning to continue to maintain that.
144
00:20:44.910 --> 00:20:50.820
Rory Kelly: MC MC may have signed off yeah he was on his way to a doctor's appointment, so he might not have okay.
145
00:20:51.180 --> 00:20:53.040
Sidd: Okay anyway let's take it offline.
146
00:20:54.210 --> 00:20:58.440
Rory Kelly: yeah but that's something you know if people want to develop some sort of you know.
147
00:20:59.250 --> 00:21:08.730
Rory Kelly: outside of the meeting documentation that's relevant to the meeting that wiki page or you know some other format that would be good feedback to know too so maybe we can get your thoughts on that next month.
148
00:21:10.710 --> 00:21:20.280
Rory Kelly: And I guess that's all I have for folks today so i'll look forward to talking to everyone next month, and you know, in the meantime, have a good and productive month and be ready to share your thoughts next month.
149
00:21:21.360 --> 00:21:21.930
Rory Kelly: Thanks everybody.
150
00:21:25.860 --> 00:21:26.220
Sidd: Thank you.
151
00:21:26.850 --> 00:21:27.480
Matthias Rempel: Thank you.
152
00:21:31.530 --> 00:21:31.920
John Dennis (he/him): Thanks.