mediocreatbest.xyz

Scientific code should be compiled and tuned to best match their processor architecture. Some processors have special instructions for maximum performance; for instance, Intel has SSE instructions, and AMD has their specific instructions too. Code compiled for one architecture doesn't necessarily run on another which is the root of the problem.

The primary symptom you would see is a segfault error of an illegal instruction.

29 Illegal instruction     (core dumped)

This is especially prevalent in building and using Linux Containers for HPC systems. On these systems, although you can use container images that have already been built, creating new container images is more challenging because you have to build them separately and then load them onto the HPC system.

For example, I build some of our container images on my server and then use them on one of several supercomputers. For most systems, this is fine because they both happened to be Intel Broadwell chips, but now one of the machines is AMD k10 based. In cross-compiling for the other machine, I ran into a few hiccups, which is what this post is about.

Case Study: Spack

Spack is a tool to help build and manage dependencies for HPC systems. It makes things relatively easy because it allows you to specify the target architecture directly.

First, you can use spack arch to determine what it believes your architecture is on the target machine. In my case, this yielded a value linux-scientific7-k10, according to the triplet platform-os-target (Architecture Specifiers). The most important value is the target.

Next, you can change your spack install command to specify that target. For instance:

$ spack install mpich target=k10

In the case that you're using a spack.yaml configuration file, you can add the packages subtree and specify your target under packages > all > target (Concretization Preferences).

spack:
  specs:
  - mpich
  packages:
    all:
      target: [k10]

One last thing you can do is find the -march and -mtune parameters to use for other cross-compiling endeavors. The easiest way to do this is to check the source code (microarchitectures.json) for your target. For k10, this entry looks like:

{
    // ...
    "k10": {
      "from": "x86_64",
      "vendor": "AuthenticAMD",
      "features": [
        // ...
      ],
      "compilers": {
        "gcc": {
          "name": "amdfam10",
          "versions": "4.3:",
          "flags": "-march={name} -mtune={name}"
        },
        // ...
      }
    },
    // ...
}

Based on the compilers > gcc subtree, we can see that -march=amdfam10 and -mtune=amdfam10. As well, if these weren't specified, we could also check x86_64 for its fields.

Case Study: Python

Python makes it relatively easy to change the compiler target, at least using pip. For this, you need to set the CFLAGS environment variable before installing dependencies (StackOverflow answer that references this).

In my case, I already had the -march and -mtune flags I needed, so I just had to run:

$ CFLAGS='-march=amdfam10 -mtune=amdfam10' \
>   python3 -m pip install networkx

Case Study: Rivet

Rivet is a suite of libraries for particle physics simulation and analysis. For end-users, it is compiled using a bootstrap script, which internally calls into the Rivet libraries' autotools configure script.

To tell autotools what architecture to build for, there are some options like --build, --host, and --target which are all a little confusing. They each require a triple of cpu-company-system (System Type – Autoconf). In general, you can ignore --build and --target for most regular libraries, only using them when compiling compilers (Specifying Target Triplets). To make things easier, you can use shorter names for your architecture and a script will normalize them (gcc/config.sub).

In my case, I wasn't able to find out exactly what name should be used for my target architecture, while I did already know what -march and -mtune names I wanted. So, instead, I found that you can just pass CFLAGS directly to the configure script and it will do the right thing (GitHub issue that references this feature).

$ ./configure CFLAGS='-march=amdfam10 -mtune=amdfam10'
$ make
$ make install

Also, the Rivet bootstrap script doesn't expose a way to give the configure scripts any extra arguments. Instead, it uses a function that just passes an install prefix along. The replacement is as follows:

# before
function conf { ./configure --prefix=$INSTALL_PREFIX "$@"; }

# after
function conf {
  ./configure --march=amdfam10 --mtune=amdfam10 --prefix=$INSTALL_PREFIX "$@"
}