Developer Documentation

Developer Tutorial

In this tutorial we will look at how to setup a developer workflow using some helper scripts provided by spack-manager. These helper scripts are not really necessary since in the end we are just using the spack develop feature. This uses a spack environment which is a way to orgnaize the software you will be building/working with, similart to other environment concepts (conda env, pyenv, etc). The spack develop feature is simply a way of tagging specific software pieces as ones which you want to be able to modify the source code and have the changes propagate to its dependencies downstream through the spack install process.

For this tutorial we will take a moderately complicated scenario to illustrate the value of spack develop. The scenario is you want to make changes in Trilinos, run the unit tests in nalu-wind and ensure the exawind-driver still builds.

The following outline can server as a quick-start guide since it provides the key commands for each step.

Outline

1. Setup spack-manager
   • git clone --recurisve https://github.com/psakievich/spack-manager
   • export SPACK_MANAGER=$(pwd)/spack-manager
   • source $SPACK_MANAGER/start.sh
2. Create an environment
   • spack manager create-env -d $SPACK_MANAGER/environments/demo
3. Activating an environment
   • spack env activate -d $SPACK_MANAGER/environments/demo
4. Add root specs
   • spack add exawind+hypre ^nalu-wind ^trilinos
5. Add develop specs
   • spack manager develop --repo-branch https://github.com/trilinos/trilinos develop trilinos@develop
   • spack manager develop nalu-wind@master
6. Concretize and install
   • spack concretize
   • spack install
7. Code changes and rebuilding
   • spack cd -s trilinos # +code changes
   • spack install
8. Running tests
   • spack cd -b nalu-wind
   • spack build-env nalu-wind ./unittestX --gtest_filter=SomeFilter*
9. Advanced topics a) Parallel builds b) Combinatoric builds

Setup spack-manager

First setup spack-manager following the setup instructions

Create an environment

Next we will create a spack environment. We will do this using the spack manager create-env command.
All commands in spack have a ready help section that can be accessed with the -h or --help arguments. For example spack manager -h provides the following output:

# spack manager -h
usage: spack manager [-h] spack-manager commands ...

commands that are specific to spack-manager

positional arguments:
  spack-manager commands
    create-env          convenience script for setting up a spack environment
    develop             a more intuitieve interface for spack develop
    find-machine        get the current machine detected by spack-manager

optional arguments:
  -h, --help            show this help message and exit

Any of the subcommands in spack manager are custom extension commands that have been written to work with the other data embedded in spack-manager.

The spack manager create-env command has several optional arguments, but for the sake of the tutorial we will only use one argument -d to specify a directory where we want to create our environment.

So to create our environment we can run:

spack manager create-env -d $SPACK_MANAGER/environments/demo

and a directory will be created at this location that gets populated with all the machine specific data stored in spack-manager. The $SPACK_MANAGER/environments directory is a holding location for environments, but it is not required for you to create environments there. Feel free to create them any where on the filesystem.

If we navigate to this directory you will see two files:

# ls -lh $SPACK_MANAGER/environments/demo
include.yaml  spack.yaml

include.yaml is a concatenation of all the machine specific information (packages.yaml, 'compilers.yaml, configs.yaml etc.). This includes the available compilers, package preferences, and configuration settings. Further details of the config files are outside the scope of this tutorial but can be found in the spack configuration files documentation.

spack.yaml is what will define our environment and its contents will currently look like this:

spack:
  include:
  - include.yaml

  concretization: together
  view: false
  specs:

We will track changes to this file through out the rest of the tutorial. This file can be modified manually, or through spack commands. In the tutorial we will use the latter of these two options.

Activating an environment

Next we will activate the environment. Activating an environment restricts spack's functionality to specifically what is defined in the environment.

To activate this environment run:

spack env activate -d $SPACK_MANAGER/environments/demo

or use the shorthand function provided by spack:

spacktivate $SPACK_MANAGER/environments/demo

Any directory with a valid spack.yaml can serve as the location for a spack environment, and you do not need to be in this directory when so long as the environment is active. If you want to switch to another environment you can deactivate the current one spack env deactivate (or despacktivate for the shorthand version), and then activate another one.

If you open a new shell and want to work in an environment you've already configured all you have to do is activate it. You can also query the current environment status with spack env st.

Adding root specs

You may notice that at the bottom of the spack.yaml file there is an entry for specs:. These are called root specs. Root specs are what we tell spack we want in our environment. Spack solves for all the dependencies to create a full environment at the concretization stage.

Root specs are either added as a list

spack add exawind
spack add nalu-wind
spack add trilinos

which gives:

#spack.yaml
spack:
  include:
  - include.yaml

  concretization: together
  view: false
  specs:
  - exawind
  - nalu-wind
  - trilinos

Or the dependencies can be chained in a single spec like this:

spack add exawind ^nalu-wind ^trilinos

which gives:

#spack.yaml
spack:
  include:
  - include.yaml

  concretization: together
  view: false
  specs:
  - exawind ^nalu-wind ^trilinos

These two spack.yaml files are functionally equivalent.

Add develop specs

Next we will create a develop spec. Typically all of the packages that spack will install will be cached away inside spec and the contents of the builds are not readily available or modifiable. However, a develop spec is one where you can control the source code location and when you make changes to it spack will do an incremental build.

We can choose to let spack clone the git repo for us by running

spack develop nalu-wind@master

This will create the directory demo/nalu-wind that will be a clone of the nalu-wind source code.

Or we can clone the repo ourselves and tell spack where to point to for the source code

git clone https://github.com/Exawind/nalu-wind.git
spack develop --path nalu-wind nalu-wind@master

You will notice that to add a spack develop spec you need the package name (nalu-wind) and a version (master). This tells spack what repo to clone if you are going to have spack clone it. You may also wonder why we are using nalu-wind instead of nalu-wind. nalu-wind is a package we've added in spack-manager to make it easier for developer workflow. We've also created amr-wind and exawind packages, but the none developer versions are also acceptable. If you have any questions about the packages or can't remember the options for the spec's you can run spack info [package] to get information about any package.

Now that we've added a develop spec we can concretize. This is how spack finalizes the environment withe the requirements and constraints communicated through the spack.yaml file. To concretize run:

spack concretize

Now you can install/build:

spack install

Once the build completes we can look inside the source directory where we will see a series of files

• spack-build-out.txt: the build output for the development package
• spack-build-env.txt: the build environment used (sourcing this file will allow you to enter the exact environment used to build the software)
• spack-build-[hash].txt: the build directory where the object files, and executables can be found.

Making Changes

If you'd like to change something about the packages in your environment, say you want to switch from a Release build to a Debug build or add a new variant to a package, you can modify the spec's in the spack.yaml file. However, if you do this you must also run

spack concretize -f

This is how you signal to spack that you made a change to an existing spec and need spack to recreate the dependecy tree. This will also result in new hashes for affected packages since the specs have been changed.

Running Tests

To run regression or unit tests you need to do two things:

1. Get to the build directory
2. Call your testing commands in the appropriate environment

For item 1) spack creates a build directory with the format spack-build-[hash] inside your source code directory. You can get to this directory from anywhere by running the command

spack cd -b [package]

This command is telling spack to go to the build directory of the package you'd like. It is extra helpful if you've re-concretized your environment and have build directories from multiple hashes in your environment.

For item 2) it is importnant to remember that spack is building with a different environment from the one you used to call the spack install command. The build shell has a unique environment that should be accessed to run tests. If you wish to run tests you will need to make sure you have that environment available to your current shell.

There are two approaches for doing this. The first is to use a the spack command spack build-env. This command will let you execute any command in the same environment that a package was build with.

For example, if you wish to run the regression tests for nalu-wind you can run the following.

spack cd -b nalu-wind 
spack build-env nalu-wind ctest [any ctest args]

This has the advantage of keeping your current shell unmodified, but there is some overhead for the command you'd like to execute.

The other option is to source the environment into your current shell. When spack does a build it creates a spack-build-env.txt file that captures the build environment. This is located at the same directory level as the spack-build-[hash] directory. If you source this file you can run any of the build or test commands and your environment will match the build environment. You can also get the output of this file by running the spack build-env command without any arguments. More information on that command can be found via spack build-env -h. The main disadvandtage of sourcing the build envronment directly into your working shell is that unexpected changes might occur (python or git version may change).

To do incremental builds you can re-run spack install, or if you've already sourced spack-build-env.txt then you can navigate to the build directory and re-run ninja or make like it was a manual build outside of spack.