WRF & WPS CMAKE Tutorial

[William.Hatheway]

So with the latest addition of CMAKE to the WRF and WPS infrastructure, will there be a new users guide instruction or update to the tutorial website that talks about how to build WRF and WPS with CMAKE with step by step guides?

 

[William.Hatheway]

So with the latest addition of CMAKE to the WRF and WPS infrastructure, will there be a new users guide instruction or update to the tutorial website that talks about how to build WRF and WPS with CMAKE with step by step guides?

bringing this back to attention. I can install WRF with the compile_new and configure_new, but the output is built in the _build folder. so everything in the top level directory is still there but also duplicated in the _build. How can we make CMAKE install in the WRF folder and not a subfolder /WRF/_build?

Also how to set paths to hdf5, netcdf, jasper, libpng, etc when they are not in the standard location?

Honestly the cmake building is more confusing then the original one because it assumes everyone knows cmake and all the tutorials on the website, youtube, readme, etc still reference the old method.

 

[William.Hatheway]

couple of more questions that I came up with.

why does cmake use less memory then the old method?
how does cmake differ?

 

[islas]

I'll answer the questions/comments here in the order they appeared in this thread:
1) We are lacking on a instruction guide on working with the new system, however I encourage you to read the doc/README.cmake_build file in WRF (admittedly this is missing in WPS). In an effort to reduce the documentation burden, only things that differ from official CMake documentation have been noted. Otherwise, for CMake-specific functionality we've referenced the CMake documentation.
2) The final output is installed to an install/ folder. From there, once inside that folder, the workflow is identical to the old system. For alternate control, please refer to the doc/README.cmake_build
3) Please refer to the CMake find_package() documentation on providing most external packages, this could be added to the doc/README.cmake_build file. Finding of netCDF is simpler though and is noted within the readme.
4) As the new system is not yet the default (hence a distinction with *_new) most existing documentation should still reference the old system. As we transition to the new build system, more info should be generated.
5) It most likely uses less memory since the build rules have been streamlined to only do necessary preprocessing. For instance, now not all files are run through standard.exe, C preprocessing, and multiple in situ rewrites with sed.
6) Running ./configure_new VERBOSE=true will output the make commands similar to what is you see in the old make system. While there is a lot of extra directory and file bookkeeping that CMake does, this will also give you an idea of the difference in the compile command when comparing the analogous command from the old make system for the same file.

 

[William.Hatheway]

I'll answer the questions/comments here in the order they appeared in this thread:
1) We are lacking on a instruction guide on working with the new system, however I encourage you to read the doc/README.cmake_build file in WRF (admittedly this is missing in WPS). In an effort to reduce the documentation burden, only things that differ from official CMake documentation have been noted. Otherwise, for CMake-specific functionality we've referenced the CMake documentation.

I'll take a look at that

2) The final output is installed to an install/ folder. From there, once inside that folder, the workflow is identical to the old system. For alternate control, please refer to the doc/README.cmake_build

how would you build it to be pointed back to the main & run folder so that it follows the official online tutorials?

3) Please refer to the CMake find_package() documentation on providing most external packages, this could be added to the doc/README.cmake_build file. Finding of netCDF is simpler though and is noted within the readme.

okay thank you

4) As the new system is not yet the default (hence a distinction with *_new) most existing documentation should still reference the old system. As we transition to the new build system, more info should be generated.

makes sense

5) It most likely uses less memory since the build rules have been streamlined to only do necessary preprocessing. For instance, now not all files are run through standard.exe, C preprocessing, and multiple in situ rewrites with sed.

that's very useful

6) Running ./configure_new VERBOSE=true will output the make commands similar to what is you see in the old make system. While there is a lot of extra directory and file bookkeeping that CMake does, this will also give you an idea of the difference in the compile command when comparing the analogous command from the old make system for the same file.

I will try this

 

[William.Hatheway]

I'll answer the questions/comments here in the order they appeared in this thread:
1) We are lacking on a instruction guide on working with the new system, however I encourage you to read the doc/README.cmake_build file in WRF (admittedly this is missing in WPS). In an effort to reduce the documentation burden, only things that differ from official CMake documentation have been noted. Otherwise, for CMake-specific functionality we've referenced the CMake documentation.
2) The final output is installed to an install/ folder. From there, once inside that folder, the workflow is identical to the old system. For alternate control, please refer to the doc/README.cmake_build
3) Please refer to the CMake find_package() documentation on providing most external packages, this could be added to the doc/README.cmake_build file. Finding of netCDF is simpler though and is noted within the readme.
4) As the new system is not yet the default (hence a distinction with *_new) most existing documentation should still reference the old system. As we transition to the new build system, more info should be generated.
5) It most likely uses less memory since the build rules have been streamlined to only do necessary preprocessing. For instance, now not all files are run through standard.exe, C preprocessing, and multiple in situ rewrites with sed.
6) Running ./configure_new VERBOSE=true will output the make commands similar to what is you see in the old make system. While there is a lot of extra directory and file bookkeeping that CMake does, this will also give you an idea of the difference in the compile command when comparing the analogous command from the old make system for the same file.

Does the same answers apply for WPS too? I'm running into issues with it as well @islas

 

[William.Hatheway]

Does the same answers apply for WPS too? I'm running into issues with it as well @islas

I know you are busy getting ready for the mpas/wrf training but I didn't want this to get lost in the threads.

Also I would like to know what exports are needed for the different cmake USE functions or where to find them. For example if I want to use netcdf4 w/ hdf5 do i need a netcdf4_root and hdf5_root export or a *_inc *_lib export?

 

[islas]

The design for WPS CMake is also generally the same.

For the other `USE_*` options, yes they should generally be able to be used in conjunction with a `_ROOT` variable. For OPENMP, MPI, and netCDF, as those are supported by the executables in your environment you should normally not need to provide a `*_ROOT` to make use of those. That said, if you are running with multiple installs and really want something that isn't the first found thing in your path or not in your path at all, using the `_ROOT` variables will allow overriding.

What issues are you having with WPS?

 

[William.Hatheway]

The design for WPS CMake is also generally the same.

For the other `USE_*` options, yes they should generally be able to be used in conjunction with a `_ROOT` variable. For OPENMP, MPI, and netCDF, as those are supported by the executables in your environment you should normally not need to provide a `*_ROOT` to make use of those. That said, if you are running with multiple installs and really want something that isn't the first found thing in your path or not in your path at all, using the `_ROOT` variables will allow overriding.

What issues are you having with WPS?

mainly i'm having issues determining which *_ROOT variables need to be set since I do run multiple versions of WRF and have them installed in different places.

 

[islas]

If you mean various versions of dependencies, then yes for the most part it would be good to explicitly state the *_ROOT variable for any dependency. The USE_* variables are (somewhat) independently controlled options enabled from external software (here differentiated from "features" that are part of WRF, ex ENABLE_CHEM). The USE_* do not always correspond to an equivalent *_ROOT (ex USE_IPO), it is more that these USE_* can be turned on/off and the build will try its best to build as specified and so they are exposed at the top-level. As a counter example, netCDF is not in there as there is no option to build without it, and when chem is enabled Flex and Bison also become requirements that are not able to be toggled.

Unfortunately, as somewhat noted above, the inter-dependencies of when things are used and option wrangling is not straight forward. This, however could in part be addressed in the readme documentation, at the very least listing which packages must be found and which might be used.

 

[William.Hatheway]

If you mean various versions of dependencies, then yes for the most part it would be good to explicitly state the *_ROOT variable for any dependency. The USE_* variables are (somewhat) independently controlled options enabled from external software (here differentiated from "features" that are part of WRF, ex ENABLE_CHEM). The USE_* do not always correspond to an equivalent *_ROOT (ex USE_IPO), it is more that these USE_* can be turned on/off and the build will try its best to build as specified and so they are exposed at the top-level. As a counter example, netCDF is not in there as there is no option to build without it, and when chem is enabled Flex and Bison also become requirements that are not able to be toggled.

Unfortunately, as somewhat noted above, the inter-dependencies of when things are used and option wrangling is not straight forward. This, however could in part be addressed in the readme documentation, at the very least listing which packages must be found and which might be used.

That would be helpful in the readme. Just to ease use if anything.

 

[haiqingsong]

If you mean various versions of dependencies, then yes for the most part it would be good to explicitly state the *_ROOT variable for any dependency. The USE_* variables are (somewhat) independently controlled options enabled from external software (here differentiated from "features" that are part of WRF, ex ENABLE_CHEM). The USE_* do not always correspond to an equivalent *_ROOT (ex USE_IPO), it is more that these USE_* can be turned on/off and the build will try its best to build as specified and so they are exposed at the top-level. As a counter example, netCDF is not in there as there is no option to build without it, and when chem is enabled Flex and Bison also become requirements that are not able to be toggled.

Unfortunately, as somewhat noted above, the inter-dependencies of when things are used and option wrangling is not straight forward. This, however could in part be addressed in the readme documentation, at the very least listing which packages must be found and which might be used.

When will WRF 4.7 be released?