This is a place where we can share our comments, questions, tips, and experiences in using Amazon Web services. The AWS documentation is good, but it does have some gaps and, in the vast tome of documentation it can be hard to find the specific details we need to know.

See the child pages for details and please do add your own pages and comments.  If you figure out how to do something useful, please enlighten us all.  If you find a particularly useful tutorial, please share it.  We're in this together.

AWS has extensive documentation and many useful tutorials.  If you find other useful links please share them here

Alces Flight is one of several (free) third-party services that help you to create an HPC cluster on AWS.  This is recommended by Kevin as relatively user-friendly. See here for specific information on using Alces flight with AWS

Action Items from Thursday 7/12/2018

1) Intel folks suggest be cautious of fabric issues at AWS. AWS may still be using ethernet which is not as capable as IB, etc. on production machines such as theia, discover, wcoss, etc. If FV3 is going to be run at production resolution (C768?) this would be a good opportunity to compare computational performance of AWS vs. e.g. theia.

2) Also, earlier discussions with AWS representatives indicated that parallel file system access is NFS, not a more scalable parallel file system such as lustre or GPFS. This may or may not be important for JEDI.

3) Intel folks had this information w.r.t. using the Intel compilers on AWS (note that we can build locally but need the Intel runtime to run elsewhere):

"Our people who use AWS said they build on a local machine and transfer the binary to AWS. However, to run the binary you need the runtime and you can download it free from:

https://software.intel.com/en-us/articles/installing-intel-parallel-studio-xe-runtime-2018-using-yum-repository

I think this is the easiest – and by using this technique, you don’t have to pay money for building time as well."


4) Apparently the GOES-16 data is available here at AWS: https://registry.opendata.aws/noaa-goes/


  • No labels

1 Comment

  1. Mark Miesch

    Thanks Jim for this useful information.  The down side of using this approach for the intel compilers for JEDI is in continuous delivery.  Ideally, we would want Amazon to build the code, compile it, and test it using both the GNU compilers and the intel compilers.  If we build the code elsewhere and just transfer the binaries then this automated testing becomes less practical.