2020-09-21

Agenda Item 1:

CRTM-2.4 Release discussion

Discussion:

Update by Ben:

I have merged all the work from Tong from REL-2.3.1 into the repository together with the changes from Jim Rosinski. I have also included the bugfixes such as cloud fraction. The limit checks for LOG and EXP are still remaining, but it shouldn’t impact continuity. Cheng and Patrick, mostly Cheng, have finished the work on netCDF I/O for Aerosol coefficients.

What is remaining:

-       I am in the JEDI code sprint to write ctests for CRTM. The recommendation has been to implement a ctest for standalone CRTM and a separate build system for people who don’t want to use CMake.

-       The other part is documentation. Patrick, Cheng, and I will work heavily on documentation, probably in the first week of October. In the second week is the Quarterly Meeting which takes a lot of effort to fit all the pieces together. I’ll send out an email requesting your contribution slides. Please also create Zenhub issues for your work before that. It doesn’t have to be perfect, but we need to keep track of all the in-kind contributors’ work. So just take a moment to go into Zenhub. If you’re confused about this, please send me an email.

Any comments?

Haixia: Ben, I saw you closed the cloud fraction bug issue. Are you opening another one?

Ben: I don’t know why I closed it. I think I moved it from one repo to another.

Haixia: So, you are still working on it?

Ben: Yes, I am working on it right now, and I will mention you in it.

Yingtao: I raised an issue that is probably connected to Hongli. Right now, we are having a release and trunk version of the CRTM. The trunk version is difficult to use for non-CRTM people. Should we have some kind of protocol when we want to update something?

Ben: My comment is that all of the development should take place in the “trunk” version. I have a script that simplifies the build process and that I can show you. It will create a “release build”. The way I have been testing it is that I create a feature branch where I move everything from the build directory to the top folder and delete everything. The other way is just testing within the trunk structure.

Yingtao: We had some kind of fixed protocol whenever someone finishes a feature branch we create a release version. We shouldn’t let people work on the trunk version. The other thing is, I have a branch, can we just let Hongli review it?

Ben: When you just have a feature branch, that doesn’t trigger a review. When you want to merge your changes, you can make a pull request. It is also possible for you to keep your feature branch and never merge. I’m okay with that. But in general, we encourage everybody to merge their branch.

Yingtao: So, if we want to merge a feature branch into develop we have to submit a pull request?

Ben: Yes, we have a minimum of two reviewers, mostly me, Patrick and Cheng. I can go through this with you offline.

Yingtao: Can you give the script to Hongli as well?

Ben: Yes.

Yingtao: Another question, have you modified my code on the branch for netCDF?

Cheng: No, not yet. But if you think it doesn’t make too much change I can go ahead and look at it.

Yingtao: Yeah, because now we have these look up table in the binary format.

Cheng: I suppose I can start looking into that.

Haixia: I also want to mention the status of IASI-NG.

Ben: Before you say something Patrick, I just wanted to say that as far as I know they decided against using the PC-score approach.

Patrick: Yes, so I managed to compile the old Fortran applications from the trunk for interferometers like IASI and I am currently busy to modify the cluster job shell scripts to run a IASI validation case.

Haixia: I think the channel selection for IASI-NG has not been fixed yet. So you first want to generate IASI coefficients and then look at IASI-NG as soon as the channel selection is final?

Patrick: Yes.

Ming Chen: In the new release are you planning to release the coefficient generation package?

Ben: Yeah, so here’s the plan. We will release the coefficient generation package in a separate repository. The release will be restricted because there are security issues. It will be separate from the CRTM repository but actively linked.

Ming: You also mentioned the license thing. I don’t know what kind of impact that will have.

Ben: This is what I’ve heard: the CRTM and all JCSDA software will be released under two open source licenses: 1. Creative Commons, which is the least restrictive open-source license.  I will have to work with Kevin on that. From the NOAA legal side, all codes from civil servants are automatically public domain. The remaining questions is whether some GPL parts can be converted to CC. I will be working with Kevin to get the least restrictive license possible, which is what NOAA wants as well.

Ming: This is something difficult in the current organization. Yingtao and Cheng are working on aerosols. Can we put some simple license there for the developers? The original motivation for the CRTM was to make the code public. Sometimes university contributors want to make contributions but this code belongs to UCAR and this to STAR.

Ben: I understand and let me just mitigate this. CC0 is the least restrictive license. The challenge here is that new developers can maintain their copyright. Let’s say Nick develops a new emissivity model, he can keep his copyright.

Ming: For me, I just want everything simple. We have been in this team for a long time and we have experience which one is better. The code that will come to the trunk will have a lot of testing. Normally the code that is shared with a tag, they don’t need the full trunk. Only if they have some positive results they will come to the trunk.

Ben: You are correct. We do need those standards and we do have them. There are two things to talk about here. We have two branches, master and develop. Develop are things that we deem interesting to include. Master will be updated very slowly for operational releases. Once we move to a public releases master code will be very stable and very well tested. I agree with your concern on that we need better communication on what goes into master and develop.

Ming: Normally for the develop we have a milestone. If we should accept something then it needs to achieve the milestone. We need to consider some management issues for the repository, how to upgrade. If it’s better to have some document for contributors. Somehow, we don’t know exactly technical details.

Ben: You’re free to talk about technical details but I also don’t want to force you into this discussion. You can talk about what you are developing for the repo. But your point is taking that we probably need to have a document how we will have governance of the repository.

Ming: That’s okay for me but anyway I think that for my surface part I wanted to set up a meeting with you last week and probably there is some misunderstanding there with the CSEM. So, there is no problem. But I am just afraid that if we don’t make everything is clear right now there will be problems in the future. The CSEM has been there for a long time. Whenever I want to do something I always need to request something with the CRTM repository. I need to be able modify my CSEM part. Normally in this case we ask the developer to do a lot of testing and then do a minor release. I don’t know if this step is the development stage. <unintelligible> The user doesn’t see any difference. We just need to make everything more flexible. The developer has a lot of things in the lower level. If you want to keep the current code in the CRTM that’s going to be very difficult. For example with the current interface, adding a new look up table is very difficult and requires changes.

Ben: I have a seen the current CRTM code and I don’t have any concern about including a new look up table in there.

Ming: You know that the surface part is different from the atmosphere part. For surface we have different types. We need to have a systematic way of including that. Also, CSEM the surface part was designed by top-down. It’s a very low infrastructure model. Comparison and optimization is a big issue.

Ben: Right.

Ming: We can have it for the 3.0 Release.

Ben: This is what needs to happen: you have to be on the CRTM_dev repository soon. We haven’t seen any code. We need to see it to decide how to move forward. You have full access to the CRTM_dev repository. If you’re blocked anywhere, let me know. We need to work with Kevin to get you started on the CRTM_dev repository.

Ming: For me there is no problem. <unintelligible>

Ben: Okay, just…

Ming: I can show you we shared some things on S4, such as the configuration with GSI. The other part is, the surface part, CSEM is kind of a general model. It’s not like the CRTM interface.

Ben: No. I need to see the code to understand how CSEM can be part of release 3.0. In order to do this, I need to have full access to your code. If you’re doing development for GSI, that’s outside the scope of our work.

Ming: There’s no development for GSI.

Ben: Alright, let’s have more communication offline. Sounds like you’re pretty close to having something you can share.

Ming: For me I think there is no problem, but I need to talk to Kevin.

Ben: Alright, my impression was that Kevin is fully on board.

Ming: For you I think you can accept the code and there is no problem.

Ben: ok.

Ming: The only thing is I think the 3.0 beta version being in the UCAR repository is not acceptable. I want to see especially the vector part.

Ben: A lot this stuff is not even done yet. The vector stuff specifically has not been merged into develop yet.

Ming: Do you have a plan yet?

Ben: We are aiming for an early release in February but with COVID it may slip by a month or so. But we need to really look into the CSEM NOW.

We are coming towards the end of the hour, maybe we can continue offline.

Cheng: We had the pull request in the last week on aerosol netCDF. If you are interested you can have a look at it and make suggestions.

Result:

-       This week there will be a REL-2.4 alpha version available.

-       Ming Chen needs to share the CSEM code on Github.

Tasks:

-       Ben will provide a REL-2.4 alpha

-       Ming Chen needs to share the CSEM code on Github

-       Interested contributors should have a look at Cheng’s pull request for aerosol netCDF I/O.

Responsible People:

-       Benjamin Johnson

-       Ming Chen

-       Cheng Dang

Deadline:

CW39, 2020