Scientific Computing

This CMake project builds GCC and high level prerequisites such as GMP. This avoids needing to manually download and configure each project. This assumes fundamental prerequisites such as libc, autotools, make, CMake are present. The 3-stage compiler bootstrap is enabled to help ensure a working, performant GCC.

“Security through obscurity” alone does not actually confer security. Obscuring code simply increases effort for someone who wishes to use code for unauthorized purposes. For Matlab “.m” code, two methods to partially obscure the underlying “.m” code for locally-run Matlab-based algorithms are:

• compile Matlab code to executable (OS-specific and Matlab version-specific)
• convert Matlab code to pcode (OS-agnostic and Matlab version-agnostic)

In general, when critical IP needs to be made available for user data while keeping IP non-public, this is done by providing web services. For example, the IP of Google, Bing, Office 365 is usable via web services, but generally the core code remains non-public.

Matlab pcode() can obscure directories or lists of files to “.p” code that runs on Matlab on any supported OS. Newer versions of Matlab use a more powerful obscuration algorithm that is not backward compatible.

For sites with poor Internet connection, conda install mypkg will fail and not resume a partially downloaded package. Workaround: use URL from the error message with curl to download the package and install directly.

curl --location --remote-name --continue-at https://conda.anaconda.org/conda-forge/win-64/<package file>

Resume download interrupted by reissuing the command to resume download. Install from the tar.bz2 file by

conda install <package stem>.tar.bz2

CMake FindPython prioritizes location over version number. Prior to CMake 3.15, even specifying Python_ROOT could be overridden if the other Python was a higher version.

Using the Python interpreter in CMake should generally be via ${Python_EXECUTABLE} instead of Python::Interpreter. CMake provides the imported target Python::Interpreter only when the CMAKE_ROLE is PROJECT. This means that Python::Interpreter is not available when using CTest, which is often when using the Python interpreter is desired. Normally, to use Python interpreter from a CMake script, including in execute_process or add_test, use Python_EXECUTABLE.

Example:

find_package(Python COMPONENTS Interpreter REQUIRED)

add_test(NAME Foo COMMAND ${Python_EXECUTABLE} myscript.py -arg1 value)

For system Python or other cases where “site-packages” is a non-writable directory, the pip --user option is necessary to install a Python package under the user home directory. However, if using Python virtualenv (with or without conda) the pip --user option is invalid. Environment variables set by Python indicate when a virtualenv is being used by Python currently.

“pip” is important for locally installed packages, since pip via pyproject.toml will automatically use the latest setuptools. This is quite important as too many user systems have too-old setuptools. The project’s pyproject.toml file should contain at least:

pyproject.toml:

[build-system]
requires = ["setuptools>=61.0.0", "wheel"]
build-backend = "setuptools.build_meta"

Detect Anaconda environment by existence of environment variable CONDA_PREFIX

CMakeLists.txt

find_package(Python COMPONENTS Interpreter REQUIRED)

# detect virtualenv and set Pip args accordingly
if(DEFINED ENV{VIRTUAL_ENV} OR DEFINED ENV{CONDA_PREFIX})
  set(_pip_args)
else()
  set(_pip_args "--user")
endif()

To install a package (named in CMake variable _pypkg) from PyPI:

# install PyPI Python package using pip
execute_process(COMMAND ${Python_EXECUTABLE} -m pip install ${_pip_args} ${_pypkg})

To install a local package in development mode (live changes):

execute_process(COMMAND ${Python_EXECUTABLE} -m pip install ${_pip_args} -e ${CMAKE_CURRENT_LIST_DIR})

CMake internally in Modules/FindPython/Support.cmake detects Python virtualenv.

One of the powerful uses of YaML with CI systems such as GitHub Actions (GHA) is the ability to setup job matrices. Job matrices allow deduplication of jobs with terse specification, encouraging better test platform and parameter coverage. GHA strategy matrix allows excluding jobs, including via exclusion arrays. An example strategy matrix below excludes shared build on macOS, due to bugs in third party library on macOS.

jobs:

  unix:

    strategy:
      matrix:
        shared: [true, false]
        img: [
          {os: ubuntu-latest, fc: gfortran},
          {os: macos-latest, fc: gfortran-13}
        ]
        exclude:
          - shared: true
            img: {os: macos-latest}

    runs-on: ${{ matrix.img.os }}
    env:
      FC: ${{ matrix.img.fc }}

This strategy results in 3 CI jobs:

• os=ubuntu-latest, shared=true, fc=gfortran
• os=macos-latest, shared=false, fc=gfortran-13
• os=ubuntu-latest, shared=false, fc=gfortran

Getting Python can be tricky for license-restricted users e.g. government and corporations. Building Python can be an arcane process without the automation of a high-level build system CMake. Python uses Autotools on most platforms except Windows where Visual Studio is used. The libraries need to be built with specific options and the forums are full of suggestions for tweaking Python build scripts etc.

The CMake project to build Python elides those issues for Linux/macOS platforms at least. It builds basic requirements of Python including expat, ffi, bzip2, xz, readline, zlib and more.

In CMake, undefined variables can evaluate as false in simple if(x) statements. For comparison operations like LESS GREATER, undefined variables do not operate like “false”.

if(x LESS 1)
  message(FATAL_ERROR "undefined not less 1")
endif()

No matter what value is compared to undefined variable “x”, the if() statement will not be true. As in most programming languages, a key best practice for CMake is to ensure variables are defined with a proper default value.

Building an Autotools project as a CMake ExternalProject saves the time of converting that other project to CMake. This technique makes it easy to automatically build that other project when it’s not easily installable otherwise, or you wish to build it optimized. This technique does not needlessly rebuild the ExternalProject each time the main CMake project is rebuilt.

It is useful to first check that Autotools is available to avoid emitting errors at build time. The goal is to emit errors about missing packages during build system configuration instead of during the build. We do this by including the file FindAutotools.cmake.

INSTALL_COMMAND

Note that “-j” option is NOT used to avoid race conditions in install scripts that might intermittently fail

CONFIGURE_HANDLED_BY_BUILD true

avoid constant reconfigure / rebuild.

“my_LIBRARY” is the known library file(s) built by the Autotools project.

include(GNUInstallDirs)
include(ExternalProject)

set_property(DIRECTORY PROPERTY EP_UPDATE_DISCONNECTED true)
# don't recheck for updates to the Git repo at subsequent CMake reconfigure

set(config_flags)  # parameters desired for ./configure of Autotools

set(my_LIBRARY ${CMAKE_INSTALL_FULL_LIBDIR}/${CMAKE_STATIC_LIBRARY_PREFIX}mylib${CMAKE_STATIC_LIBRARY_SUFFIX})

find_program(MAKE_EXECUTABLE NAMES gmake make mingw32-make REQUIRED)

ExternalProject_Add(mylib
URL https://github.invalid/username/archive.tar.bz2
CONFIGURE_HANDLED_BY_BUILD true
CONFIGURE_COMMAND <SOURCE_DIR>/configure ${config_flags}
BUILD_COMMAND ${MAKE_EXECUTABLE} -j
INSTALL_COMMAND ${MAKE_EXECUTABLE} install
TEST_COMMAND ""
BUILD_BYPRODUCTS ${my_LIBRARY}
)

add_library(mylib::mylib INTERFACE IMPORTED GLOBAL)
target_include_directories(mylib::mylib INTERFACE ${CMAKE_INSTALL_FULL_INCLUDEDIR})
target_link_libraries(mylib::mylib INTERFACE "${my_LIBRARY}")
# need the quotes to expand list
add_dependencies(mylib::mylib mylib)

For Ninja BUILD_BYPRODUCTS is necessary to avoid “ninja: error: “lib” needed by “target”, missing and no known rule to make it”

Some Autotools projects may need a “bootstrap” before “configure”. Add this script if needed:

ExternalProject_Add_Step(mylib
bootstrap
COMMAND <SOURCE_DIR>/bootstrap
DEPENDEES download
DEPENDERS configure
)

CMake ExternalProject allows building a wide variety of subprojects isolated from the main CMake project. For GNU Make Makefile projects, it is necessary to invoke the make command. However, there are several programs named “make” across operating systems. To help ensure the correct GNU Make is selected, we do:

find_program(MAKE_EXECUTABLE
NAMES gmake mingw32-make make
NAMES_PER_DIR
DOC "GNU Make")

A real-life example of CMake with Makefile ExternalProject has multiple Make invocations to build separate Make target groups, where later Make targets depend on the other Make targets being built first. We just show a snippet here for clarity, omitting definition of some of the obvious variables used.

CONFIGURE_COMMAND ""

since Make doesn’t have a configure step, so we must define this blank, as otherwise CMake will try to find a CMakeLists.txt in the external project code.

BUILD_COMMAND

builds the first target(s) that are required by targets in subsequent steps. If there’s no subsequent targets, this is the only build step.

INSTALL_COMMAND

Note that “-j” option is NOT used to avoid race conditions in install scripts that might intermittently fail.

BUILD_BYPRODUCTS

In general we point this at the “installed” files, as otherwise “ninja: error: “lib” needed by “target”, missing and no known rule to make it”. Ninja is stricter than Make about the target to source graph.

include(GNUInstallDirs)

set_property(DIRECTORY PROPERTY EP_UPDATE_DISCONNECTED true)
# don't recheck for updates to the Git repo at subsequent CMake reconfigure

find_program(MAKE_EXECUTABLE NAMES gmake make mingw32-make REQUIRED)

set(my_LIBRARY ${CMAKE_INSTALL_FULL_LIBDIR}/${CMAKE_STATIC_LIBRARY_PREFIX}mylib${CMAKE_STATIC_LIBRARY_SUFFIX})

ExternalProject_Add(mylib
URL https://github.invalid/username/archive.tar.bz2
CONFIGURE_COMMAND ""
BUILD_COMMAND ${MAKE_EXECUTABLE} -j -C <SOURCE_DIR>
INSTALL_COMMAND ${MAKE_EXECUTABLE} -C <SOURCE_DIR> install prefix=${CMAKE_INSTALL_PREFIX}
BUILD_BYPRODUCTS ${my_LIBRARY}
)