2020-11-30

Agenda Item 1:

Quick Update from Ben on REL-3.0

Discussion:

Comments from Ben:

Ben: Not a lot to cover, I am just working on the 3.0-beta update. It’s a challenge because we want to integrate 3.0 with new features from version 2.4.  The plan is to have a release branch that is not part of the main trunk so that people can test it. This is the plan for this week. Otherwise I am working on the cloud coefficient generation part with lots of delays because of technical and bureaucratical issues. In the next technical meeting I will give an overview of the CRTM cloud coefficients and Legendre coefficients.

Ming: Ben I think this is a good plan.

Result:

Merging REL-3.0 is ongoing

Tasks:

Next TM Ben will present on Cloud scattering properties.

Responsible People:

Benjamin Johnson

Deadline:

Agenda Item 2:

Emissivity Update from Nick Nalli

Discussion:

Ben: Nick, we haven’t seen you in a while, do you want to give an update?

Nick: There are a couple of things. Next week is the ISSI meeting. Steve English has asked me to give a talk on the IR emissivity. Obviously, we will talk about the state-of-the-art. I have been working with Jim Yung on the simulation side. He has been working on double difference calculations. We are doing that to eliminate all contamination. There is a jump 900 and 925 and there is a gap in the temperature dependence. We found that in the transition zone between temperature-dependence and no temperature dependence the data is not good enough. Additionally, we are working with Sergio at kCarta and we found some odd artifacts between 800-1000. I think what is happening is that the LUT I have is over-tuned to the atmospheric emissivity term. I am doing some more tests with Sergio. The question I have for Ming Chen is, would it be a problem if I would regenerate LUTs and give them to you?

Ming: Probably not. But the dataset is becoming very large, so we will probably need to find a regression model.

Mark: Nick, how big is your LUT?

Nick: Basically, what I provide Ming is a very high resolution LUT which can be pretty large.

Ming: About 2GB.

Mark: What’s the resolution?

Ming: 0.2 wavenumber.

Yingtao: Is it really necessary that it is this large?

Nick: That’s right. Let me know if there are any better point spacings for you, spectral, wind spacing etc. .

Ming: I saw that Yong showed some of his latest advances in data assimilation with GSI, so we need to probably work with him.

Nick: I am finding this out that it is very complicated. Ming, that’s great, I am glad to hear that and at some point, I might have a new table for you.

Ben: Thanks for the update. The ISSI meeting is an ocean emissivity meeting that is focused on MW and the goal to have Nick involved is to have some IR awareness. The JCSDA is gonna host the datasets for the meeting once we decide on a name for the reference meeting. There will be some issues on what can be made public. It’s similar to what Ming has been doing in the past years. Unfortunately, the meeting is closed to the public. I will give you guys an update on the ocean emissivity part.

Result:

Nick Nalli will present IR Emissivity results at ISSI meeting.

Link to the ISSI meeting:

https://www.issibern.ch/teams/oceansurfemiss/index.php/first-meeting-agenda/

Tasks:

Deliver IR Emissivity LUT to Ming Chen

Responsible People:

-       Nick Nalli

-       Ming Chen

Deadline:

Agenda Item 3:

OpenMP Discussion

Discussion:

Mark: Ben, have you received any feedback from CRTM users on the REL-3.0 code, particularly on the OpenMP? I looked at the RTTOV code yesterday, they have two options for OpenMP. Because for MIRS they don’t have that, they’re just doing that at a profile by profile level. Maybe Haixia can say whether they need OpenMP at the profile level.

Ben: It really depends on the application. GSI benefits from the profile level. I haven’t received any complaints. The biggest user of the CRTM is JEDI. I haven’t heard any complaints from the NCEPlibs team.

Mark: For the JEDI are they assimilating at the profile level?

Ben: Our goal is just to get people to use the CRTM with lots of profiles at the same time. I like the idea of parallelizing the CRTM at a channel level for hyperspectral instruments.

Ben: When we did some early tests with OpenMP on a channel basis that also improved MPI load balancing.

Mark: Maybe for the AOP2021.

Haixia: I am sorry, Mark I just want to make sure that according to what I know the version we will use is CRTM 2.3. I do not hear that NCEPlibs is implementing a version 2.4.

Ben: Yes, they are testing that for the next-next release.

Haixia: Good. According to what I know OpenMP slows down Huija’s postprocessing.

Ben: I will be working with Huija specifically on that issue.

Ming: Ben, whether we do the OpenMP for profiles inside or outside the CRTM it’s pretty much the same.

Ben: There’s been a number of optimizations made in the CRTM and I think there is a general willingness on their side to adapt.

Haixia: I think you will need to let Huija know. I don’t know if OpenMP will have a significant speedup in GSI. Everything will be moved to JEDI.

Ben: Yes, by the time JEDI is going operational we want to have all these optimizations in place. GSI is important, I agree, you can still run REL-2.4. I was surprised by Mark’s last test with the speedup because it didn’t show any improvement. I’ll arrange a meeting with Huijia and there are also folks in Wisconsin who are working on improving performance.

Mark: Right now, we have a profile level OpenMP. Later if we add a channel level OpenMP that will add a lot of benefits for CRTM users. I think we should put that in the AOP.

Ming: But Mark I think that the Forward model uses OpenMP, but what about the K-Matrix?

Mark: For the K-Matrix that’s about the same level of effort.

Ben: We’re borrowing a software engineer from the JEDI team who will be working on optimization specifically, Mark Olah. He’s very good at that.

Ming: Ben, Jim’s first try was over channels, right?

Ben: Yes, but we ran into the same problems Mark mentioned and so we went with the profile approach. For a long time, we were counting on OSS to be the optimizier for hyperspectral instruments but that seems to have stalled.

Mark: For the K-Matrix the channel level should work, it’s not finalized yet but the benefit is very linear. So, we should thank Paul.

Ben: Yeah, my gratitude towards Paul is enormous.

Mark: A long time ago John Derber asked the CRTM team to use threads, not OpenMP, for the CRTM. It looked like it’s not working. Maybe today the software is not working. I don’t see any issue to do OpenMP at the channel level.

Ben: Let’s talk coefficient generation. My policy is just to be as transparent as possible.

Mark: For the version 3.0 does it include a LUT for the scattering properties for clouds and aerosols. Right now, for aerosols it’s still GOCART and for clouds it’s Gosheng Liu.

Result:

N/A

Tasks:

-       N/A

Responsible People:

N/A

Deadline:

N/A

Agenda Item 4:

Transmittance Coefficient Comparison

Discussion:

Ben: We are running out of time. Let’s quickly address the issue with the Transmittance Coefficient Generation. First, I would like to give Yingtao a chance to talk.

Yingtao: We should make sure that there are no overlapping tasks in the AOP2021.

Ben: The AOP2020 clearly states that the JCSDA is doing the coefficient generation

Ming: I have the problem that the AOP is more like a general guide line. I think that Yingtao raised some very good issue. Especially everybody in this group is involved in many different aspects but we need to have some focus.

Ben: The reason we have some documents is to have accountability for some deliverable. And that is what we are working on, how to hold individuals accountable.

Yingtao: It’s a Community Model that should encourage everybody to contribute.

Mark: I think for AOP2021 the first priority is to focus on the release of REL-3.0.

Ben: The one thing I want to get at with Yingtao is how can we have better access and collaborate with you?

Ben: I am more than happy if Patrick and you are collaborating, in fact I encourage them collaborating.

Yingtao: I don’t know which party will be responsible for generating this coefficient generation package.

Ming: Yingtao you just said that the CRTM is a community model. Why do you now want to separate it?

Ben: It’s not only about delivery, it’s also about the collaborative effort of development and testing.

I agree with Ming, this is going to be a Kevin-Ben discussion. For now, the best we can do is share, share, share. I don’t know where the gaps are on your side and knowing that will help me plan for AOP2021.

Ming: So right now, your role is more the coordination and Mark maybe does more science contributions. In the past there was one person

Yingtao: Ben, you just let me keep in charge of the coefficient generation? I will use Github as the repository.

Ben: I think I can’t do that since we really need the capability now to create new coefficients now.

Yingtao: So, you want to grab the coefficient generation capability.

Ben: I am not sure I fully agree with your definition of grabbing. We need the functionality now and we can’t rely on you to deliver it at some unspecified point in the future.

Mark: I would suggest that everybody wants to make the CRTM better why don’t we use a compromise. If Yingtao says he can deliver the package and Patrick is also working on that then Patrick maybe can make the package better. I know that in the past the package was very complicated with many lines just for the LBL computations. Patrick can work on the parallel implementation. I checked a Linux box with lots of cores. In the past we didn’t

Result:

Tasks:

Responsible People:

Deadline: