Agenda Item 1:

AOP 2021

Discussion:

Introduction by Benjamin Johnson:

Ben: A couple of things, we’re getting started with the generic AOP2021, the idea from management is to change things a little bit. I am mentioning this because you are involved in the in-kind contribution. The core focus of AOP2021 is “applications”. The goal is to do something with these codes that have been developed, like JEDI. This is just a general way we want to go with the Joint Center, to give partners software that is more practical. What you will be hearing about in the next month or two is applications, how to engage the user. The other part is, the CRTM AOP structure will change with the way how we work. To be more application-focused it’s going to shift slightly to a more user-oriented approach. This is going to be a management task, a dedicated outreach task. We are in the process of building multiple applications that people can use even outside the CRTM. As you know the Air Force is using RTTOV. We are looking to get them to switch to so-called “American Models”. The goal is to give them applications that give them a lower barrier of entry, to market what we do better.

The reason I am telling you to do this is because as in-kind contributors I will be soliciting advice from you in the next few months on how to do that. This is also an opportunity to expand or shrink your role in the future. I have already been getting some feedback from Mark Liu. Over the next few months we will flesh out the AOP2021 and I will share it with you.

Any questions? Things you are concerned about?

Mark Liu: Ben, we discussed the AOP, maybe I can show you a few slides, so you can get an idea.

[shares slides]

Mark: Just a few slides for the CRTM-3.0. One slide is for the 2.4 new feature and the computation time comparison I did. To myself the new features are very good. I looked at the OpenMP comparison. But I haven’t looked at the Python interface. I did a timing study for the parallelization. I did 100 profiles with 155 layers. The code is from Patrick, he provided it to me, it’s called the Nakajima-King example. I wanted to test whether the results are right or not. I added a cloud to check the results. Currently the REL-2.4 has some small issue, the results are not fully correct. I am using the optimized ifort, since this is what the operational user is using. The Forward model took 15 seconds, the K-Matrix took 30 seconds. That’s normal. The code in 2.4, I like that you added OpenMP and you are still able to compile the code even without using OpenMP. The single thread results are very much similar, just some small differences. Then for the threads 1,2,3,4 they use OpenMP and they are very much similar. I used 2 threads and it cut time by about half. I used 4 and it cut time by another half. That’s very good. Sorry, here is a typo. The bottom one is for 3.0. It looks like it’s faster than 2.4. I tried to make the code faster but there are too many places to change. Once I fixed the code, all of them gave the same result and the finite difference consistency also agree.

Yingtao: What’s the major change from 2 to 3?

Mark: The major change is the vectorized model. Here I included what Patrick and Jim did for the OpenMP and what you did for CMAQ.

Ben: Did you run 3.0 with 1 or 4 Stokes elements?

Mark: Just one, to make the same comparison. I thought I could speed up the code, but the package is too big. I can show the results. You can see here [shares screen] those values for VIIRS the first column is for K-Matrix and the second column is for AD. The third column is for cloud water amount. The fourth column is for the cloud particle size. The fifth is for the TL and the sixth is for the Finite Difference. The v2.3 below gives the same result. For the TL and AD model I haven’t added OpenMP yet. For timing of v3.0 timing looks good. The proposed Makefile is to keep the user interface the same as for v2.3, that is either have binary or netCDF. I have discussed that in emails with Ben and Cheng Dang. As long as the user provides the file, no change is needed. One thing I realized is that even though we use it often, it’s still not easy to include the netCDF library. So, for option 1 (single core, binary only), the netCDF source will not compiled. For option 2 (single core, with netCDF), then the user can modify the makefile. The third option (OpenMP and binary), the user doesn’t need to modify the makefile and option 4 is the full OpenMP and netCDF. Probably I can tell you some historical thing. To do the CRTM extension we did some work in 2016 but actually started about 5 years ago, but there was no project to direct this activity. 4 years ago, we got the GPS-PR proposal funded, but the funding was very small, not enough to get the v3.0 funded. Too many places to change, the Forward model, look-up tables, it’s not a small task. Here we had a plan in the past. In the past we also had the RTTOV tasks, so we can compare with RTTOV v12. We know the RTTOV long term development. Here is just a simple comparison with a polarized Monte Carlo model. This is the story what we did in the past. Back to Ben. Just wanted to discuss whether you like the option 1,2,3,4.

Yingtao: Mark, you said that the aerosol will be specified by the user. What if both coefficient files are present?

Mark: When the user wants to use the coefficients, he only has one choice.

Yingtao: So, you have to remove the file from the directory?

Mark: No, in the CRTM code is a preprocessor that handles this automatically.

Yingtao: So, you still have a compiler flag to indicate whether you want to compile netCDF?

Mark: No, there’s a two-line code in the CRTM that determines that. We will provide all the coefficient files, binary, netCDF.

Yingtao: How do you determine which option you are going to use?

Mark: In the makefile.

Yingtao: So, in the makefile you have flag to determine option 1,2,3,4?

Mark: Yes, just type ‘make -option’.

Ben: I wouldn’t spend too much time on this. We are moving away from autoconf to cmake based building. The other question I have, you would have to include a preprocessor directive not to include netCDF.

Mark: Yes, but it’s only 4 lines in the code in total.

Ben: Ok, but that’s a substantial change. We want to get users to use netCDF. We are trying to change the culture. Yes, we will still provide binary support. It’s more work for us and less transparent for the users.

Mark: You just provide an option, just like with the OpenMP.

Yingtao: That’s a preprocessor flag a user has to add?

Mark: No, we provide that. For the user is simple. Option 1 is default, then for netCDF type make -nc would be option 2. Then for OpenMP maybe it would be ‘make -omp’ and so on. That would be easier for version 2.4.

Ben: That’s the direction with cmake. NCEPLIBS is moving to cmake and this allows to be more modern and consistent with other software.

Yingtao: For cmake you will still have to provide the options to the users. You have to have a switch there.

Mark: No, right now in the CRTM initialization you need to tell whether you will have to specify whether you want binary or netCDF.

Yingtao: Yes, but when you switch to cmake you will still have to provide a flag to the user. I mean the netCDF format or OpenMP. When you compile the CRTM you will have to ask the user whether he wants to use the options.

Ben: No matter what we do this will all be covered in the README files. I agree that we want to provide the flexibility whether or not to use OpenMP.

Mark: But you have a default option to use the same configuration we had in the past.

Ben: Thanks, I really appreciate Mark, especially the timing tests. What I would like to see is the same test but with 4 Stokes elements. Then I will ask, Patrick is also here, actually for UV and Visible the slowdown is from Rayleigh scattering. For a cloud you only have one layer but for Rayleigh you have to compute scattering for all layers.

Yingtao: For the UV region?

Mark: No, for the code that I am asking is for all spectra. This function should speed up the code a lot. Right now, for every layer we have to do the scattering calculations.

Ben: I think there is an analytical form that allows you to get the scattering cross-section.

Mark: Yes, but that is very easy. We need a code for the full scattering.

Ben: The other possibility is to do some regression training.

Mark: Yes, that’s a good idea to use a pre-calculated table. You only need two parameters as input.

Ben: The Rayleigh scattering phase matrix only depends on the angle.

Mark: Yes, for the CRTM we have an additional angle from the satellite. Otherwise when you have a discrete grid that’s easy to have a table.

Ben: I guess we can come up with a fast way to do this. There might be an OpenMP solution up here internally.

Mark: At least what we can see at least for v3.0 is to fix the OpenMP bug. We can do UV and Vis first and MW later. For the Vis and UV we do radiance perturbation but for the polarized BT, the radiance can be negative so we can never convert that back. I have solved this problem during my PhD time in Germany. It’s solved but it needs more coding.

Yingtao: Why would the user need to input the RTSolution?

Mark: For the AD.

Yingtao: That is computed by the CRTM itself, right?

Mark: No, for the AD you need to input the perturbation.

Your perturbation can be negative. The problem is when your perturbation is negative there is no way to compute the BT perturbation to radiance.

Yingtao: So, you say you have a solution for this?

Mark: Yes, we need to be able to solve this for the user.

The radiance value can change a lot as a function of the wavelength.

Yingtao: IR and VIS people use radiance all the time.

Mark: No, you are wrong.

Yingtao: To be specific, short wave IR.

Mark: Right, but even at shortwave IR like 3.7 micrometer the users always use BT.

Yingtao: So, there’s nothing wrong with radiance, it’s just convention.

Mark: Yes, but we need to make the user comfortable.

Ben: The way how other models do this is to just use vertical IV and IH. I do remember reading a publication of yours that addressed this issue, might have been your dissertation.

Result:

Ben will solicit feedback from in-kind contributors in the coming months for CRTM applications for AOP2021.

Tasks:

-       Solicit feedback from in-kind contributors

Responsible People:

Benjmain Johnson

Deadline:

Before AOP2021 deadline.

Agenda Item 2:

CSEM Discussion (continued from last meeting)

Discussion:

Ben: I wanted to give Ming or Yingtao a chance to say whether the Gitlab repository is available right now?

Ming: The STAR repository should be available in a few weeks. There are two streams that we are going, the JCSDA and STAR repositories. Normally we rely on the CRTM repository but meanwhile we have the CSEM repository. With CSEM we need to have a preliminary version of REL-3.0. I would like to have a preliminary v3.0 version as soon as possible. For the Gitlab I have come up with a solution I can share with you.

Ben: What would be a solution?

Ming: Let me see.

Yingtao: Do you have any suggestions for the repository, Ben?

Ben: I said that in the email before. The key part is just that feedback comes up to the development repo.

Ming: You will have that. I hope that we can keep the CRTM as one part. I wanted to replace the dedicated surface repository with the CSEM. Meanwhile we will maintain a CSEM repository as in the EMC vlab. We will use that one for the further development, especially implementing the disk model. We will keep an independent version in the future. You can mirror the CSEM to Gitlab or the JCSDA repo. But I think that this one is just for development. For the production release I don’t know. Right now, we have two ways to release the CRTM.

Mark: My suggestion is, right now people don’t know what CSEM you have and why you don’t plug it into the CRTM v2.

Ming: Mark, first of all CSEM was designed for CRTM 3.0, there are a lot of structure changes. Right now, we have CRTM 2.4.

Mark: That’s why I am suggesting to do everything step by step. Otherwise it will be difficult to find errors. Then for v3.0 we use the software to merge it. There will still be some manual work to merge the conflicts. If you think about all those interfaces, that’s a lot of work.

Ming: Mark, I think you are thinking about work we have already done. 2.3 or 2.4 is the same. We have already tested CSEM with 2.3.

Mark: That is good, you are already done for the 2.3?

What I suggestion is why don’t you give the 2.3 version of the CSEM to the CRTM, then Ben can merge it with 2.4 and Ben can build the v2.4.1.

Ming: No, that’s my question. You say that we should merge CSEM with 2.4. Do you think we can justify a minor release with these major changes?

Mark: You think too much about minor and major changes.

Ming: …

Mark: It’s up to you, just a suggestion.

Ming: Mark, we should have had this one 2 years ago. So, we should have set up a repository on the JCSDA side.

Yingtao: I think the major concern for Ming is the credit for the work he has done. Everytime we develop a new component we better give the author credit. In the future the honor of this component will always be forwarded to Ming.

Ming: I don’t consider this an honor. It’s just about general development. Ben has a general roadmap, but Mark, you say we can implement that In 2.4 or 3.0. I don’t like random things. We planned to implement CSEM in 2016 v3.0. We are still working with 2.4 but we need to realize why we are still here.

Mark: Ming, let me explain. I don’t have a problem but we want things to be done effectively. There is no issue. The 2.4.1 can have a CSEM release and for v3.0 we can still have a CSEM release.

Ming: Mark, tell me why you are using 3.0 for the CRTM.

Mark: This is just because we said it but I don’t have a problem with 2.5. But there are a lot of things to be done for one version. So many bugs to figure out.

Mark: …

Ming: That’s what we discuss in STAR with Kevin. Sometimes we need to do things step by step. If the CSEM is going to get in, we talked about 3.0 for a long,  long time. Everybody, even at NASA, is asking when will 3.0 come out? Now we are having a 2.4 minor release. That’s ok, the CRTM has a large user community. We need to come up with an AOP. But in this case, I can implement in v2.4, v2.3 it doesn’t matter.

Yingtao: So, Ming you are going to implement CSEM in version 3.0.

Ming: Yes.

Ben: We are developing v3.0 right now.

Ming: You don’t need to worry, I can implement the CSEM in the v3.0 directly. But we need to have a solid plan.

Mark: You can discuss that. My experience is that doing things step by step this is more reliable. If you want to jump to v3.0 that’s fine with me. For v3.0 are you also going to match up the polarized BRDF?

Ming: If you say you have a vector surface part I can add that.

Yingtao: …

Ming: I need to see first of all, once we decide that for v3.0 we need to do vector radiative transfer.

Mark: Months ago, I gave the version to Kevin and Ben. I have no authority to tell Ming what to do.

Ming: Ben is most important since we are relying on the JCSDA for the development repository. Right now we have several versions running.

Ben: That’s your choice.

Mark: Ok, maybe later I can get Ben and Kevin together.

Ben: I am working on that, Mark you don’t have to worry about that.

Mark: If you have a better idea how to implement the makefile idea, we can work on that.

Ben: Thank’s for your help. Do you plan to send the v3.0 beta at some point?

Mark: Yes, I plan to give you a version in early December because I am trying to look how to add the OpenMP to TL and AD.

Ben: That’s not a big requirement for us.

Mark: The ECMWF is using TL and AD.

Ben: We would be happy to help with that as well.

Mark: I can add OpenMP, I have some experience now.

Ming: One thing, for the beta version you don’t have a lot of vector parts but for the surface part I will give Ben the one code with the surface part removed and then we can set up a repository.

Mark: For the management when I release the v3.0 beta I can only send the code to Kevin and Ben because otherwise there will be a problem.

Ming: Ben can set up a repository and I can work from there.

Mark: Yes, I can send it to Ben.

Ben: The same day I get the code it will go on Github and you have access to it Ming.

Ming:…

Ben: I can’t plan without knowledge about the code. I have no idea what needs to be removed, what needs to be added. Seeing the code will help me plan.

Ming: …

Ben: Things need to be removed. We will also have to plan what is not covered by CSEM.

Ming: So, the first thing to do is to set up a repository.

Ben: I have an alpha version there already.

Ming: 3.0?

Ben: Yes, it’s been there for months.

Ming: I checked the alpha part and couldn’t see any vector parts.

Mark: Maybe this week I can give the code to Patrick and we can do some testing.

Yingtao: Under JCSDA-internal?

Ben: Yes.

Yingtao: I am looking at it and I can’t see it.

Ming: I couldn’t see any vector code.

Mark: For 2.4 the Atmosphere structure messed up OpenMP because it was changed INOUT.

Ming: We will fork the JCSDA repo into the NOAA Github and do pull requests back. Normally we finish everything in the NOAA Github. We also have some internal CRTM tasks that are not covered in the AOP, such as STAR CalVal. In this case we want to have a unified team but sometimes we have different focus.

Ben: We are running about 30 minutes late.

Ming: For CSEM I will put the code into the CRTM. We will have some dedicated surface development. We will implement several models, not only ATLAS. Github is just used to facilitate the collaboration but we need to have focused work. That’s the way we are doing things.

Ben: I can’t control what you do, but the recommended approach is to use the JCSDA repo for development. Whenever you make CSEM public, there will be a public clone on Github.

Result:

No conclusive result for the CSEM code.

Tasks:

-       Send v3.0 beta to Ben and Kevin

-       Create a v3.0 beta repository on Github

-       Provide CSEM code

Responsible People:

-       Liu Quanhua

-       Benjamin Johnson

-       Ming Chen

Deadline:

ASAP

Agenda Item 3:

pyCRTM and GPU Updates

Discussion:

Ben: Bryan do you have anything to discuss?

Bryan: No, just put up some Zenhub issues but I still want to contribute as soon as I am finished writing my recent paper.

Ben: Daniel?

Daniel: Bryan helped me with build on Hera, now I am able to build fv3-jedi and I am able to run all tests.

Ben: What are the next steps?

Daniel: I am trying to modify the CRTM code inside the bundle.

Ben: We are appreciating all documentation you have.

Daniel: I also wanted to ask for my superiors if you have any plans where the GPU acceleration is supposed to go in terms of collaboration?

Ben: No collaboration with Nvidia.

Ming: Ben, you said you will send us some general guidelines for the AOP?

Ben: Yes. Daniel…?

Daniel: No, I just meant collaboration plans in general.

Ben: For GPU acceleration in general there are no specific plans outside AOP2020. It’s really up in the air and there are a lot of issues with GPU acceleration. We would hope that you could demonstrate some potential speed up. If your results show that it’s beneficial we will invest more time in it.

Mark: Has the GPU acceleration already been done?

Daniel: Yes, I am trying to modify the CRTM inside the fv3 bundle now and run some timing studies.

Ming: Ben, do we have some plan to parallelize the hyperspectral sensors?

Ben: Yes, you are right. We will have IASI-NG coefficients ready fairly soon. But I think the profiles are also significant. I don’t have a dedicated OpenMP engineer anymore. I am trying to get Tom to hire someone again.

Result:

-       Continued collaboration from Bryan Karpowicz

-       Daniel has compiled fv3-bundle no Hera

Tasks:

-       Run timing studies with GPU fv3-bundle

Responsible People:

-       Bryan Karpowicz

-       Daniel Abdi

Deadline:

N/A