2021-02-25

Agenda Item 1:

CRTM 2.4 Post-Release Update

Discussion:

Introduction by Ben:

Ben: Let’s get started. It’s a busy time of year. We just had our first round of AOP discussions and they went well. I wanted to share with you the updates on CRTM v3.0. Before we get started, any general discussions? – No. I will share my screen now [presentation]. Starting with version 2.4, the CRTM is public domain now. The reason was that there were some discussions with using CRTM as a library and we had the agreement from NOAA. These slides just highlight some things we have done recently. We released v2.4 recently on Github as a fast release. We added a new regression testing system. Version 3 is an ongoing effort and we are wrapping up some things from v2.4. REL-3.0 will include some of the code from Mark Liu. All this is supposed to be done until the end of March, which is a tall order. Patrick has been working very hard on the coefficient generation and supported by Isaac.

Sarah: What do you mean by beta release?

Ben: I will release a beta version internally. A release in March will be hard. We are working with Scott Sieron at Penn State. Greg Thompson and I have been working closely with this group to see what the Microphysics efficiencies are. We support a new build system, which is ecbuild from the European Center, and we also have a standalone cmake system. The idea is to replace the autoconf build system. And we had some new sensors and bug fixes in v2.4.

CMAQ aerosol specification has been added by Yigntao Ma. There are still some ongoing issues with that. I created some experimental cloud coefficients.

Hui-Ya: Have you done some tests how much time you are saving with OpenMP?

Ben: That’s on the current slide. Daniel has been working on OpenMP over channels. The benefit there is not quite as substantial as with over profiles, but it is still very good. The scaling is somewhat nonlinear, so it’s dropping off, but at the very least we will have a 3-times speedup for most IR sensors. It’s actually optimized over both. I don’t know if Daniel is here.

Haixia: Ben, so for the GSI we are doing a profile-by-profile for channel sections, so GSI is going to be 3.4 times faster?

Ben: We expect it to be even better. We haven’t tested it for hyperspectral instruments yet, but that’s where it is supposed to shine.

Haixia: But it’s not in the EMC version on Github?

Ben: No, I will make a new release.

Haixia: Can you also provide a tutorial?

Ben: Yes, it’s really very simple. You just have to specify an environment variable. So Haixia and Emily reported some issue with v2.4 early on. Initially there were some issues with v2.4 BTs. They were significantly colder. I was curious about this because this only occurred for the case of specifying the cloud fraction to a specific value. Haixia may say more about this. After we found this out, the differences were much smaller.

Haixia: This is maybe related to the 2.4 cloud fraction threshold of 10e-6.

Ben: Yes, I talked Alan Geer. He said they use 10e-5 but he agreed that it could lead to an error if the actual cloud fraction is smaller than 10e-5. They change it on their end to let the cloud fraction go smoothly to zero. This is a longer discussion probably more suitable for the technical meeting.

Haixia: Sure.

Ben: Thanks for looking at this. I will not go through the subsequent detailed slides. One item is the pycrtm package from Bryan. Just wanted to talk a little bit about the testing system. The way it is designed, if you are familiar with cmake, you can just provide different sensor names and it will run all tests for this instrument. The first time the test runs it will create a baseline and if you make a change the tests might fail and that gives you an indication on what changed. That’s part of the agile process. Patrick and Cheng are heavily involved in that. Ming has been working on CSEM. He has delivered the code to us. Nick has provided his updated tables.

Ming: Ben, I think in the 3.0 beta version Mark implemented some surface model from UV to visible. I already implemented it in CSEM. If we can set up a Github from NOAA. I think that was actually good work for systematic testing.

Ben: I can’t take full credit, Paul did a lot of work. Let’s talk more about CSEM later. Cheng do you want to talk about CMAQ?

Cheng: Sure, what Ben is showing here are the CMAQ single scattering properties You notice that for the CMAQ table the extinction coefficients are much lower than the default. So, if we use this to compute the AOD on the next slide, this is MERRA 2 climatology data. The bottom line is that with CMAQ we have a one order of magnitude lower AOD. That’s what Ben referred to that there are some issues with the table.

Ben: Yes, that’s an ongoing discussion. Cheng, can you also talk about the GEOS-5 table?

Cheng: The GOCART tables had a lot of development in the last decade, so they are different from the default. I was hoping to combine the GOCART GEOS-5 table with the current table and merge it into one for CRTM aerosol simulation. The code changes are finished and I check if they make sense. I do see some problems.

Ben: We’re having a discussion with Orlando because thehy have a very flexible object oriented GEOS interface. This is the long-term idea similar to JEDI to have a generic structure that doesn’t contain anything aerosol related.

Mariusz: Yes, I can add we have two UFOs now, and GOCART produces AODs that are 60% higher.

Cheng: Is this what you presented at AMS?

Mariusz: Yes.

Hongli: That is good news. We have the same result. So, when you test the GEOS-5 LUT I can also have a GSI test.

Cheng: Sounds good.

Ben: I think the whole point is that there is a broad effort between NOAA and NASA across different groups to make it easier for people to pick up different aerosol configurations. Of course, you have to understand what you are using.

So, for the summary: For AOP2021 I wanted to share this with the group. The transmittance coefficient package is an ongoing effort. I’ll be working with the STAR group and there’s also aerosols, getting cloud LUTs updated. Ping Yang is adding a UV range to his scattering properties. Surface properties will be updated and I will be working with Ming and Nick on that. There’s a new thing called SIMOBS app, which is just a front-end application for the UFO. This will be a very Python-heavy task to run the CRTM as a standalone. We are also looking at AI applications to speed up things like transmittance coefficient generation. I don’t want to go into this in more detail. We will just use AI as another tool. Ok, that’s all and a lot of stuff in here.

Questions?

Hui-Ya: Did you notice that IO-speed got slower for netCDF?

Ben: I think there’s a trade-off but it’s something we will need to test because it depends on the disk system you are using.

Hui-Ya: In our upcoming publication we found that the IO speed became slower. In the postprocessing side it became a lot slower. It’s about from 45 seconds to 1 minute.

Ben: Ok, that’s substantial. We will provide binary support but we want to get away from it. NetCDF is a lot easier to work with and we are hoping to offset this with OpenMP.

Hui-Yang: Right now we are very anxious to see the OpenMP version because the postprocessing times are about 8 minutes.

Ben: Any updates?

Patrick, do you want to talk about the transmittance coefficient generation?

Patrick: Yes. As you already mentioned the TROPICS coefficients have been delivered and that also made it necessary to change the surface polarization. Right now, I am working on IASI-NG. The existing code has hard-coded band limits and I am working on extending them from 3 to 4.

Ben: How many channels does IASI-NG have, around 15000?

Patrick: Yes, about that much.

Ben: That shows how helpful OpenMP and netCDF will be for an instrument with 15000 channels, especially in the polarized case.

Kevin: Have we heard anything about the channel selection?

Ben: No, not yet. Haixia, do you know something?

Haixia: No, not sure yet.

Ming: You already mentioned that you moved the v2.4 into 3.0beta?

Ben: A quick look at the MW clearsky will show you if H-V will be negative.

Haixia: I think the IASI-NG channel selection is not clear yet. Last year we had a meeting, but I need to follow up.

Ben: Yeah, the PC score approach is something that we are not following.

Haixia: We are still assimilating reconstructing radiances, not the principal components.

I think EUMETSAT is going to do a study about that.

Mark: Haixia, for the PC score for IASI, those approaches are about 5 years ago Xu Liu from NASA developed something for the CRTM to include hyperspectral instruments and ECMWF is also using the Xu Liu method.

Haixia: We are not requiring the CRTM to use the PC score method.

Mark: About 5 years ago Xu Liu was funded briefly by the JCSDA to do this.

Haixia: We already have a person in our group to reconstruct radiances.

Ben: Andi or Scott?

Haixia: Kristen.

Mark: The original PC model is from NASA?

Ben: No, for the Europeans it’s developed by Marco.

Mark: That’s probably the same thing what NASA developed.

Ben: We will be assimilating the reconstructing the radiances, not the PC scores.

Mark: In the US we have PC scores from NASA, and AER developed generic OSS method.

Ben: Kevin may have some updates on OSS, the initial implementation stalled.

Kevin: Yes, it’s just something we haven’t picked up to support. Right now, there’s some re-evaluation. AER did some testing for CrIS. The other caveat is that we don’t have access to the OSS coefficient generation. I also wanted to talk about the IASI-NG method. There is an approach that makes IASI-NG look like IASI.

Mark: There was a discussion to save work by downgrading the second generation Metop to IASI-NG. For the application part there was no change.

Ben: Anything else?

Mark: Hongli organized the meeting on Thursday on aerosols. It’s time for us to find out the one acceptable solution.

Ben: If you are on this call and you are not on the list, just tell Hongli.

Sarah: I’m actually looking for a discussion on aerosol physics. Cheng wants her work to be consistent with model physics and postprocessing so we need coordinated effort. Now I have a new project to put aerosol backscatter into a UFO and I am also working with Barbara on the lidar operator.

Mariusz: Let me comment on making everything consistent. For NASA we should use GOCART tables. Someone raised an issue that UPP is using ASCII files instead of netCDF files.

Sarah: Yeah, UPP is actually the same table.

Mark: For the aerosol it’s more complicated than what we have seen before. People have a high expections for consistency between GOCART and CMAQ and also observed AOD from Aeronet and GOES-R. It might be needed to be an EPIC project. For the model part it should be checked if the mass for CMAQ and GOCART is similar. In the group it should be easy to compare the mass from each type. Second, on the CRTM side we need to check the consistency between GOCART and CMAQ. Third, if we want to utilize the benefit of aerosols, we need to minimize the difference towards observations.

Mariusz: I completely agree. It’s a problem that requires a lot of work. Especially in CMAQ when you have the model variance and you have to account for that, it’s substantial work.

Sarah: That is right. If the definition is not consistent we are comparing apples to oranges.

Ben: I have been bending over backwards to accommodate some of the more challenging partners. Hui-Ya, what are the postprocessing plans?

Hui-Ya: UPP was just used as pass-through.

Sarah: They read a mixing ratio in UPP.

Hui-Ya: I don’t know the details but a lot of them are pass-through. I know that Sarah put in the optical depth. A lot of data handling.

Ben: Anybody else have an updated.

Yingtao: I have a question about coefficients. I got an email from Yenlong, he is using v2.2. Do we have a repository for old coefficients?

Ben: I do, I can contact him but I would encourage him to update.

Yingtao: We should probably have an update policy.

Ben: We do. We support everything above v2.2.3.

Ming: I found an EMC release that is v2.1.3.

Ben: Yes, that is the oldest supported version. It is the easiest way to get it from the website.

Haixia: We are currently using v2.2.6, I’m confused myself.

Ben: v2.3 is the current operational version. I go to the NCEPLIBS meetings fairly regularly and update those folks. It’s past 4 o’clock. So unless there’s something else, send me an email.

Result:

N/A

Tasks:

N/A

Responsible People:

N/A

Deadline:

N/A