Scientific Computing

Variables that are larger than a few kilobytes often should be put into heap memory instead of stack memory. In Fortran, compilers typically put variables with parameter property into stack memory. A good practice in Fortran is to put non-trivial arrays intended to be static/unchanged memory into an allocatable, protected array. Example:

module foo

implicit none (type, external)

integer, allocatable, protected :: x(:,:)

contains

subroutine init()
  allocate(x(1024,256))
  !! in real life, this would be some constant data array or
  !! expression filling the "constant" array x.
  x = 1
end subroutine init

end module


program bar

use foo, only : init, x

call init()

if (any(x /= 1)) error stop "did not init"

end program

In this example, x is approximately a one megabyte variable, assuming kind=int32. Even though the compiler may not warn if we instead declare this variable as parameter, it can cause segfaults and other seemingly random runtime errors.

Normally we would use a derived type instead of a bare module, but we did it here for simplicity.

Fortran allocate large variables

If the variable to be allocated is about one gigabyte or larger, sometimes special techniques are needed, even on systems with very large amounts of RAM including HPC. This is especially the case on Windows systems.

The error messages one may get upon allocating large variables in Fortran include:

Error allocating <N> bytes: Not enough space

Segmentation fault (core dumped)

For Windows, a peculiar limitation is that each variable (including allocatable) cannot exceed the virtual paging file size, even if the Windows computer has large amount of RAM that isn’t being exceeded. The paging file size may be inspected and set under: Control Panel | System and Security | System | Advanced system settings | Advanced | Performance | Settings | Advanced | Virtual memory

In general, the compiler may need to have the memory model flag set for the situation. This flag has a set of implications.

• Intel Fortran: -mcmodel
• GCC Gfortran: -mcmodel

Note: This problem is resolved by macOS 11.4.

As noted by Hector Martin and others, early macOS 11 appeared to have a possible kernel bug causing excessive SSD write wear whenever the SSD was in the “on” state. One can use “smartmontools” to check SSD write history:

brew install smartmontools

smartctl --all /dev/disk0

Note that SSD on state time can be much less than powered-on time. This is especially the case for the Mac mini, which may sit powered on but unused for the majority of the time by some users.

Thankfully as noted by Jonas Ribe, Hector Martin and others, macOS 11.4 appears to have fixed this SSD write bug. Thankfully we haven’t see the 100+ TB of excess SSD wear pre-11.4 as Jonas did. We saw less than 5TB of excess wear on mostly idle, continuously powered on Minis.

The CMake test frontend CTest can easily select subsets of tests. While there are more advanced CTest test selection options, two of the most common and easy to use test subset selection methods are by regex selection of names, labels and/or fixtures exclusion. Assuming the project has a meaningful test naming scheme, one may trivially select tests by either or both of the ctest -R and ctest -E flags. For this article, assume the tests are named:

alpha:egg
alpha:bacon
beta:egg
beta:apple
gamma:egg
gamma:orange

where each test was setup in CMakeLists.txt like:

add_executable(alpha_egg alpha_egg.c)

add_test(NAME alpha:egg COMMAND alpha_egg)

The colon has no special meaning, and CTest names may use special characters if desired. When figuring out how to use CTest test selection, it’s very helpful to also add the ctest -N option, so that test names are printed without running the tests. For all examples, we assume the user working directory is PROJECT_BINARY_DIR or is using ctest --test-dir.

One may select all the “egg” tests by:

ctest -R egg

Suppose one wishes to exclude the test named “beta:egg”:

ctest -R egg -E beta

To run all tests except those with “beta” in the name:

ctest -E beta

A more sophisticated test selection scheme requires setting test labels in the respective CMakeLists.txt like:

set_property(TEST alpha:egg beta:apple gamma:egg PROPERTY LABELS "unit;gravy")

Combinations of test labels and test names regex can be used to select subsets of tests. For example:

ctest -R egg -L unit

Note also the option ctest -LE, which works like ctest -E for labels.

CTest fixtures can also be excluded with the ctest -FA option. This allow not rerunning expensive FIXTURES_SETUP tests when not needed.

Set labels for all targets and tests in a directory like:

set_property(DIRECTORY PROPERTY LABELS linalg)

where “linalg” is the desired label(s).

Git stash is often used to hold temporary work that wasn’t yet ready for a commit, perhaps during a rebase. The stash history is per repo. Over time, one may accumulate numerous stash entries that are no longer relevant and may desire to cleanup this clutter.

View Git stash entries by:

git stash list

View the contents of a particular stash entry by:

git stash show -p "stash@{0}"

where the number in braces corresponds to the git stash list index.

To remove a stash entry, sliding up all the entries older than it:

git stash drop "stash@{N}"

where “N” is the entry index to remove.

If one wishes to recover a dropped Git stash entry, it may be possible:

TeXLive is a well maintained and often updated LaTeX distribution that works for any OS. Download TeXLive net Installer and run. Click Advanced and use the “basic” scheme, which is well under one GB to start.

When building a document and it seems to be missing a package, note the error messages. Find and install needed packages via tlmgr using commands like:

• find package by filename: tlmgr search --global --file fullpage.sty
• find fonts/packages by name: tlmgr info ieee
• install package: tlmgr install lmodern

Use a LaTeX IDE (GUI editor) like TeXstudio.

When a new major version of TeXLive is released, simply repeat the procedure above and point any programs like TeXstudio to the new TeXLive install directory.

Windows Terminal is a useful and welcome open source addition to any developer working on Windows. It has no prerequisites and works well with multiple WSL distros.

Get Windows Terminal via winget:

winget install --id=Microsoft.WindowsTerminal

Python can extract .zst Zstd files using the zstandard package. Zstd is widely supported and available across operating systems. Make a Python function extract_zstd to extract all files and directories inside the .zst file.

Related: Matlab extract .zst file

Zstd is a modern performant file archiving standard widely used to replace .zip, gzip, etc. With modern “tar” extract .zst like:

tar -xf arc.zst

If needed for older tar, specify the program “tar” should use to extract .zst:

tar --use-compress-program=zstd -xf arc.zst

Matlab can extract .zst files.

For very old “tar” one may get:

zstd: error 25 : Write error : Broken pipe (cannot write compressed block) tar: Error opening archive: Child process exited with status 25

In this case, use a two-step process to extract the .zst file fully:

zstd -d myfile.zst   # creates tar file "myfile"
tar -xf myfile       # extract the original file/directory hierarchy

Matlab websave or ftp might not work in cases where a plain “curl” or “wget” command works. A symptom of this issue is HTML is downloaded instead of the intended binary file. Websites such as Dropbox recognize the HTTP User Agent of curl and Wget and mutate the web server response to be automation-friendly. Since Matlab is much less commonly used than Python, curl, Wget, etc. this user agent-dependent behavior results. We recommend understanding why Matlab websave doesn’t work, or use the low-level Matlab HTTP Interface.

To use curl from Matlab, recognize this may require unique setup for each system, despite that curl is included (preinstalled) in modern operating systems including Windows.

The extra quotes around “url” allow for arbitrary characters to be used in the URL that can confuse shells like zsh. The “-L” option to curl allows redirects.

function curlsave(filename, url)

cmd = "curl -L -o " + filename + " '" + url + "'";

assert(system(cmd) == 0, "download failed: " + url)

end

Linux systems with multiple curl versions installed may need to set an environment variable to prioritize. Set the filename as appropriate for computer (ensure the file exists).

LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libcurl.so.4 matlab

This can help errors like

curl: (48) An unknown option was passed in to libcurl

Even this doesn’t always work, so we recommend understanding why Matlab websave doesn’t work, or use the low-level Matlab HTTP Interface.

CMake configure_file can serve to generate source code files including Fortran. For templates beyond a few lines of code, it may be useful to read the template from a file. This requires setting CMake directory property CMAKE_CONFIGURE_DEPENDS to have CMake regenerate the file if the template file changes as in the example below.

file(READ my_template.in.f90 template_code)
configure_file(example.in.f90 example.f90 @ONLY)

set_property(DIRECTORY PROPERTY CMAKE_CONFIGURE_DEPENDS my_template.in.f90)

#example use
add_executable(main ${CMAKE_CURRENT_BINARY_DIR}/example.f90)