Scientific Computing

mpi_f08 Fortran interface

Fortran MPI programs should use the Fortran mpi_f08 interface:

use mpi_f08

OpenMPI, MPICH, and Intel MPI have long supported Fortran “mpi_f08”. For Windows, WSL well supports “mpi_f08” via OpenMPI or MPICH.

MPI constants like mpi_comm_world and mpi_real are Fortran derived types.

For legacy user programs if needed, access the MPI legacy integer value via the %mpi_val property.

use mpi_f08

integer :: comm = mpi_comm_world%mpi_val
!! %mpi_var emits the legacy integer

Fortran MPI examples

Too much data that is still not enough

This example uses the aurora, which is produced around most planetary bodies due to energetic particle kinetics as the particles penetrate the ionosphere. Optical instruments such as cameras give a line integrated measurement for each pixel (angle) of the imagers. This data can be useful for tomographic techniques, when the location and orientation of the camera is well known, and multiple cameras with overlapping field of view exist.

However, this rich data can be greatly supplemented and even superseded by other instruments, especially incoherent scatter radar, where 3-D + time data are available due to volume integrated target returns. Many analyses rely on those thin (~ 0.5 degree FWHM) radar beams to complete an analysis. We rarely know the needed orientation of the radar beams beforehand, and many ISR cannot change the location of their pre-programmed beams. Although as AESA they can steer almost instantaneously within the radar backend processor limits.

This is just a geospace example of too much data, but not enough to gauge individual analyses without additional processing techniques.

MINGWROOT environment variable

By convention, the environment variable MINGWROOT tells the path to MinGW64 (just above bin/, lib/, include/)

  • MSYS2: MINGWROOT=%SYSTEMDRIVE%\msys64\mingw64

This variable may be needed to modify the GNU Octave PATH on Windows when using “system()” calls with executables compiled by MinGW. A similar issues exists on Windows with Matlab and Parallel Computing Toolbox, that provides its own mpiexec.

We made a function to workaround these issues.

Eliminating non-https external links

With a website / blog having thousands of pages and many thousands of external links, it is impractical to check external outbound link quality with any regularity. Informal link checks revealed that non-https:// websites had a substantially higher chance of becoming a defunct site that gets snapped up by spammers and scammers. To help mitigate some of the risk of websites going to unintended destinations, we decided to eliminate almost all non-https external links.

An increasing number of undesired websites are enabling https both to improve SEO and trick visitors. However, this additional friction anecdotally for the external links we’ve seen go bad has so far been rarer for https:// URLs. We have seen https:// sites be replaced by undesired content, but what often happens is the spammer doesn’t bother to setup the certificates correctly, so either the website won’t load if HSTS was used, or there are prominent warnings that the user has to click through.

There’s nothing to stop spammers from correctly setting certificates, but we feel https-only external links currently afford a meaningful benefit.

WSL2 date time skew error

WSL2 (including with Windows 20H1 2004) is known to have issues with having the WSL clock get hours or days behind actual time after the computer is suspended. This issue was not seen in WSL1, but upon upgrading to WSL2 has been almost immediately apparent to multiple people that reported this issue. This causes errors with build systems (including GNU Make and Ninja) and SSL verification among others.

A workaround for this, when it occurs (have to keep doing workaround) is to synchronize the software clock to the onboard hardware clock from WSL Terminal:

hwclock -s

or if suitable from Windows Terminal:

wsl --shutdown

If that doesn’t work, try using NTP from WSL Terminal:

ntpdate time.windows.com

This issue has been noted at WSL GitHub Issues:

Other issues are linked from those

Fix BibTeX error with .bbl file

Sometimes cryptic errors occur if there was a syntax error in a .bib BibTeX bibliography file that doesn’t disappear even when the .bib syntax is corrected. The fix for this is often to delete the auto-generated files:

Example: top-level LaTeX file “main.tex”. The compilation generates main.bbl and main.aux among several others. Try:

rm main.aux main.bbl

pdflatex main
bibtex main
pdflatex main
pdflatex main

Delete Git tag / release version

Deleting or updating a GitHub or GitLab release leaves the Git tag, which must also be deleted to reissue a corrected release.

  1. delete the Git tag locally. Here we assume the release tag was v1.2.3

    git tag -d v1.2.3
  2. delete the Git tag on remote site

    git push -d origin v1.2.3

    This puts the release notes from v1.2.3 into “draft” mode–hidden from the public.

  3. Make the necessary changes, then do the release again, reusing the draft release notes.

Each Git hosting service has distinct approaches to releases and tags:

Python zipfile recursive write

This function recursively zips files in directories, storing only the relative path due to the use of the “arcname” parameter. Otherwise, you get the path relative to the root of your filesystem, which is rarely useful.

import zipfile
from pathlib import Path


def zip_dirs(path: Path, pattern: str) -> T.List[Path]:
    """
    recursively .zip a directory
    """

    path = Path(path).expanduser().resolve()

    dlist = [d for d in path.glob(pattern) if d.is_dir()]
    if len(dlist) == 0:
        raise FileNotFoundError(f"no directories to zip under {path} with {pattern}")

    for d in dlist:
        zip_name = d.with_suffix(".zip")
        with zipfile.ZipFile(zip_name, mode="w", compression=zipfile.ZIP_LZMA) as z:
            for root, _, files in os.walk(d):
                for file in files:
                    fn = Path(root, file)
                    afn = fn.relative_to(path)
                    z.write(fn, arcname=afn)
        print("write", zip_name)

Matlab parfor parallel plotting

Matlab plotting can be quite slow, as can Python Matplotlib plotting. Sometimes, Matlab parfor can be used to plot in parallel, when all parfor restrictions are met. However, parallel plotting in Matlab doesn’t always work, or may work on some operating systems but not others. So use great caution if making a “parfor” plotting loop–it may not work for others.

Problems when trying to do relatively simple plots in parallel:

Warning: A worker aborted during execution of the parfor loop. The parfor loop will now run again
on the remaining workers.
Error using distcomp.remoteparfor/rebuildParforController (line 194)
All workers aborted during execution of the parfor loop.
Warning: worker(s) crashed while executing code in the current parallel pool. MATLAB may attempt
to run the code again on the remaining workers of the pool, unless an spmd block has run. View the
crash dump files to determine what caused the workers to crash.

Change file ownership

When desired and permitted by the computer filesystem, it’s possible to change file ownership. This may be necessary when a file was inadvertently created with root / admin ownership by mistake, and it’s necessary for a general user to edit or access the file. Changing file ownership can have unexpected consequences, like removing the ability of others to access the file or run a program depending on the file. Therefore, file ownership changes should be done only when necessary and with consideration for others who may depend on the file.

These examples assume the file “example.txt” is in the current directory and the user logged in should own the file. These tasks are similarly done in other languages such as Go or Julia.

As long as the user has filesystem permission, Python can easily change file ownership across operating systems.

import shutil
import getpass

shutil.chown("example.txt", user=getpass.getuser())

On Windows check ownership of the file by Command Prompt:

dir /q example.txt

or PowerShell:

(get-acl example.txt).owner

Windows uses the takeown command to change file ownership. For simplicity we assume the desired user is logged in and executing the command.

takeown /f example.txt

On macOS / Linux check ownership of the file by

ls -l example.txt

For simplicity we assume the desired user is logged in and executing the command chown:

chown $whoami:$whoami example.txt