Scientific Computing

Ninja build on Red Hat

The binary executables for Ninja 1.9.0 may not work on RHEL 7 due to incompatible libc. The symptom of this is like:

ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21’ not found (required by ninja)

This was a known issue with the Ninja release artifact build process that was fixed.

Workaround: if a current version of Ninja is not available, use Ninja 1.8.2. This may work for other “older” Linux distros.

Why use CNAME flattening for apex domains

CNAME flattening was popularized in part by Cloudflare. We have used CNAME flattening successfully for years with multiple web-hosting providers. DNS A records are a pre-WWW artifact necessary to resolve the apex domain example.invalid from www.example.invalid or radar.example.invalid.

Implement CNAME flattening

To enable CNAME flattening on DNS records, first screenshot or otherwise backup DNS settings. Don’t do this experiment during busy times, better to use on a little-used or test website first to ensure it works correctly with the proposed setup.

  1. Determine the web address the web hosting provider puts the website at. E.g. for GitHub Pages it would be username.github.io or at Netlify username.netlify.app
  2. Remove DNS A record for example.invalid that points to a specific IP address
  3. Add a CNAME record pointing example.invalid to the server address from step #1

For GitHub username joe with GitHub Pages site at joe.github.io, with desired web address example.invalid: make a CNAME DNS record with example.invalid as an alias to joe.github.io.

A or AAAA DNS records are unneeded with CNAME.

Notes

  • Cloudflare CNAME flattening article
  • You may see some old 2014 blog posts initially complaining about CNAME flattening, but these initial hiccups have been resolved long ago.

Fortran submodule and CMake

Fortran submodule is supported by all popular Fortran compilers. While designed as a way to better manage public/private exposure of variables in large Fortran modules, submodules can also be used to seamlessly switch in/out advanced functionality.

For example, the GEMINI 3-D ionospheric model was created with raw binary file I/O. Since we had already written an object-oriented HDF5 interface, we integrated HDF5 file I/O into GEMINI. To help ensure a smooth transition with seamless fallback to raw binary if HDF5 wasn’t available, we used Fortran submodule with CMake. The user would call file_read and file_write subroutines with the same name, regardless of whether HDF5 was enabled. CMake would switch in submodule files depending on whether HDF5 was working or not.

Requirements

Fortran submodule requires adequate support from the Fortran compiler, and from the build system. CMake and Meson fully support Fortran submodule.

Compilers supporting Fortran submodule include:

  • Gfortran ≥ 6
  • Intel oneAPI
  • Cray
  • IBM XL / OpenXL
  • Flang
  • NAG
  • Nvidia HPC SDK

CMake

Rather than maintain a compiler feature table, in general we create simple test programs and verify that they compile–all automatically handled within CMake.

Insert into CMakeLists.txt

include(CheckSourceCompiles)

set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)
# save link time, only compile is needed

check_source_compiles(Fortran
"module b
interface
module subroutine d
end subroutine d
end interface
end

submodule (b) c
contains
module procedure d
end
end

program a
end"
f08submod)

if(NOT f08submod)
 return()  # or make FATAL_ERROR here
endif()

Selectively enable program functionality using submodule in CMakeLists.txt. This example is for HDF5:

add_library(io io.f90)

if(USE_HDF5)
  target_sources(io PRIVATE hdf5.f90)
else()
  target_sources(io PRIVATE raw.f90)
endif()

Complete examples of submodule are provided.

GitHub / GitLab Pages to Netlify

While both GitHub Pages and GitLab Pages are adequate for most personal, group and project pages, when website size and / or traffic have grown beyond what is feasible for these solutions, a more comprehensive hosting provider like Netlify may be considered. Netlify provides its own CDN, so those that had been using Cloudflare for DNS and CDN can configure Cloudflare to provide only DNS, if they so choose. Netlify is free for single users, allowing a private GitLab, GitHub or Bitbucket repo (or other suitable source) to deploy to a public custom domain HTTPS website. SSL certificates can be user-provided or can be created through Netlify for your custom domain.

Why transfer site to Netlify

Netlify provides a comparison of GitHub Pages and Netlify. GitLab Pages allows user choice of static site generator (Hugo, Jekyll, etc.). GitHub Pages can using GitHub Actions for Hugo. GitLab Pages private repos have a monthly runtime quota. Netlify has a monthly traffic quota on the free tier, and monthly build quota. For sites that are becoming very popular, GitHub Pages will simply want you to move elsewhere, while Netlify will have a paid plan to offer. This process may be too burdensome for those with limited IT or bandwidth resources, or simply the lack to time to learn how to do this.

Netlify uses webhooks to detect a git push to the website GitLab repo, and then builds the site. Netlify has a CDN and DDoS protection built-in. Even if the other features aren’t needed, a key feature is the ability to have the website code in a private repo with unlimited public website deployments and traffic.

Build minute limits (such as on GitLab and Netlify) can legimately be worked around by building the site locally on your laptop and pushing the publish-ready HTML.

Transfer site to Netlify

Note: This process may take down your site for a day or two if things go wrong. Even under normal conditions, all site visitors may need to allow an HTTPS exception due to SSL certificate error since Netlify requires all DNS servers to update before generating the domain certificate.

  1. if not already on GitLab, copy your website repo to GitLab (any name repo is fine).
  2. disable Auto DevOps and ensure no file named .gitlab-ci.yml exists.
  3. Login to Netlify using Gitlab, which will ask for your website repo.
  4. pick a custom Netlify subdomain like mycompany.netlify.app. Ensure this site is totally working before proceeding.
  5. Set Cloudflare or whatever your DNS provider is to point CNAME or A to mycompany.netlify.app (THIS IS THE PART THAT CAN TAKE YOUR MAIN WEBSITE DOWN!)
  6. Under Netlify Domain Management → HTTPS → Verify DNS config, ensure the verification completes. Until the DNS change propagates worldwide, your main HTTPS domain visitors are getting SSL verification errors. They can use https://mycompany.invalid instead of https://mycompany.invalid temporarily. Do this at a low traffic time range! If using Cloudflare CDN, the old records may point to DigitalOcean while the new records point to *.netlify.app

Git commit date / time / author edit

If the Git commits have already been pushed to remote, this process will require other users of the repo to reset, rebase or reclone, as for any Git operation that edits history. If the Git commits have not already been pushed, then this process will not require extra steps from other repo users.

Show Git commit AuthorDate and CommitDate by

git show <commit_hash> --pretty=fuller

for the most recent commit, simply:

git show --pretty=fuller

edit last commit only

To reset the author of the last commit to the current Git username and email, as well as setting AuthorDate and CommitDate to the current time:

git commit --amend --reset-author --no-edit
--reset-author
reset date/time/author to current
--no-edit
skip opening text editor

edit previous commits, including already pushed

Use git rebase -i as usual and for the commits to reset author / date, change the operation to e to edit each by:

git commit --amend --reset-author --no-edit

Specific date setting

Specific commit times can be set with a combination of the “–date” option and environment variable GIT_COMMITTER_DATE.

Reference

GitHub commit troubleshooting

Use PyPI to distribute Python packages

This procedure is for PyPI Warehouse.

Assuming a Python package named “myprogram”, have a directory structure like

pyproject.toml
myprogram/
   __init__.py

A minimal pyproject.toml is all that’s required for a PyPI Python package. The package “version” number must be distinct for each release, or PyPI will prevent upload.

Ensure prereqs are installed:

pip install --upgrade twine build

One-time setup: sign up for a PyPI account. Create ~/.pypirc with the PyPI token.

In the project directory, create the package:

python -m build

Upload the package to PyPI:

python -m twine upload dist/*

Now the package is live to the world on PyPI, installable via pip install.

HDF5 on Intel oneAPI for Windows

Intel oneAPI on Windows provides an easy way to use Fortran MPI on Windows. The Intel oneAPI compile and link commands on Windows are distinct from those on Linux, perhaps reflecting the internal use of Visual Studio on Windows. The HDF5 1.10.6 release changed the naming convention for the HDF5 Fortran library files on all operating systems.

  • old: hdf5hl_fortran.
  • new: hdf5_hl_fortran.

If experiencing trouble finding HDF5 with CMake, try FindHDF5.cmake. FindHDF5 specifically works with Intel compilers and HDF5 across operating systems including Windows.

To build HDF5 library, including on Windows, use CMake to build HDF5.


If you use include/static you will get errors like

error LNK2019: unresolved external symbol H5GLOBAL_mp_H5F_ACC_TRUNC_F referenced in function MAIN__

ConnectBot cannot import OpenSSH keys

The free open source SSH app ConnectBot allows connecting to SSH servers with port forwarding using public key authentication, including ED25519.

Sometimes it’s necessary to share SSH keypairs on multiple clients. Perhaps the server owner isn’t willing to bother with more than one SSH client key. ConnectBot has an open issue since it cannot import OpenSSH keys created on a PC.

Generally users should create unique SSH public/private keypairs for each device. Sharing keys between devices means if a device is compromised, deleting its key from ~/.ssh/authorized_keys on the SSH server disables all other devices sharing that key.

Workaround by creating an SSH keypair in ConnectBot. Copy the ConnectBot-created public/private keypair to the PC ~/.ssh directory.

The stem (filename without extension) of the public and private keys must match. The public key should have a .pub suffix, while the private key has no suffix.

Thereby the same SSH keypair is used on the phone with ConnectBot and the PC with OpenSSH client.

git diff feature branch with unstaged changes

To integrate changes from branch “feature1” without cherry picking can be done file by file or line by line.

Compare current unstaged changes file-by-file, with the ability to line-by-line implement changes:

git difftool feature1

Whole branch compare does NOT allow line-by-line edits, so perhaps open a second terminal and “git checkout” files from the feature branch as desired.

git difftool -d feature1

Meld and Visual Studio Code are each good GUIs for Git difftool and mergetool.