Scientific Computing

Login show load avg, free memory, free disk

Display a login message using a message of the day (MOTD) with this script under /etc/update-motd.d/

The login message you’ll get will include:

Last Boot..........: 2020-04-12 14:29:29
Memory.............: 822 MB (Available) / 1021 MB (Total)
Load Averages......: 0.03, 0.08, 0.10 (1, 5, 15 min)
Running Processes..: 12
Free Disk Space....: 81 GB of 253 GB on /

Python script for MOTD:

#!/usr/bin/env python3
import sys
import psutil
from datetime import datetime
import shutil
from pathlib import Path

lastboot = datetime.fromtimestamp(psutil.boot_time())
vmem = psutil.virtual_memory()
drv = Path('~').expanduser().anchor
du = shutil.disk_usage(drv)

print("Last Boot..........:", lastboot)
print(f"Memory.............: {vmem.available//1000000} MB (Available) /  {vmem.total//1000000} MB (Total)")
if sys.platform == "linux":
    print(f"Load Averages......: {psutil.getloadavg()} (1, 5, 15 min)")

print("Total Processes....:", len(psutil.pids()))
print(f"Free Disk Space....: {du.free//1000000000} GB of {du.total//1000000000} GB on {drv}")

Ubuntu uses Python script “/usr/bin/landscape-sysinfo” to print a similar MOTD on login.

Ninja job pools for low memory CMake builds

An increasing number of systems have multiple CPUs, say four, six or eight but may have modest RAM of 1 or 2 GB. An example of this is the Raspberry Pi. Ninja job pools allow specifying a specific limit on number of CPU processes used for a CMake target. That is, unlike GNU Make where we have to choose one CPU limit for the entire project, with Ninja we can select CPU limits on a per-target basis. That’s one important benefit of Ninja for speeding up builds of medium to large projects, and why we see increasing adoption of Ninja in prominent projects including Google Chrome. This is another reason why we generally strongly encourage using Ninja with CMake.

Specifically, CMake + Ninja builds can limit CPU process count via target properties:

The global JOB_POOLS property defines the pools for the targets.

Upon experiencing build issues such as SIGKILL due to excessive memory usage, inspect the failed build step to see if it was a compile or link operation, to determine which to limit on a per-target basis.

Example

Suppose that 500 MB of RAM are needed to compile a target and we decide to ensure at least 1 GB of RAM is available to give some margin. Thus we constrain the number of CPU processes for that target based on CMake-detected available physical memory. The appropriate parameters for your project are determined by trial and error. If this method still is not reliable even with a single CPU process, then a possible solution is to cross-compile, that is to build the executable on a more capable system for this modest system.

CMakeLists.txt includes:

set_property(GLOBAL PROPERTY JOB_POOLS one_jobs=1 two_jobs=2)

cmake_host_system_information(RESULT _memfree QUERY AVAILABLE_PHYSICAL_MEMORY)

add_library(big big1.c big2.f90)
if(_memfree LESS 1000)
  set_property(TARGET big PROPERTY JOB_POOL_COMPILE one_jobs)
endif()

Related: tell CMake to use Ninja

Visual Studio update Ninja build

The Ninja build executable for Visual Studio location can be determined from the Visual Studio terminal:

where ninja

The factory Visual Studio Ninja version may be too old for use with CMake Fortran projects. If needed, replace the Visual Studio Ninja executable with the latest Ninja version, perhaps with a soft link to the ninja.exe desired. Add user permission to create symbolic links.

Save figure SVG from Matlab or Matplotlib

Matlab or Matplotlib will save infinite resolution vector graphics SVG format, viewable in web browsers. SVG is usable by LaTeX.

  • vector graphics (SVG or EPS) allow nearly infinite zooming without loss of quality–excellent for line plots and contour plots
  • SVG is viewable by any web browser, and is usable from LaTeX
  • EPS is more commonly used in LaTeX
  • PNG is raster graphics, so has finite (blocky) resolution

Here are examples of saving figures to SVG from Matlab and Matplotlib.

Python

To save figure handle fg, simply do fg.savefig('myfig.svg').

from pathlib import Path
from matplotlib.figure imoprt Figure

fn = Path('~/Documents/mycoolfig.svg').expanduser()

data = [1,2,3,4]

fg = Figure(constrained_layout=True)
ax = fg.gca()
ax.plot(data)

fg.savefig(fn, bbox_inches='tight')

Matlab

Matlab figures in general are saved by exportgraphics.

data = [1,2,3,4]

fg = figure();
plot(data)

exportgraphics(fg, 'matfig.svg')

NetCDF4 segfault on file open

NetCDF4 Fortran library may compile successfully and run for simple programs but segfault on programs where HDF5 is linked directly as well as NetCDF4.

A reason one might directly link both HDF5 and NetCDF is a program that need to read / write files in HDF5 as well as NetCDF format. The symptom observe thus far is the program segfault on nf90_open().

The fix is to compile HDF5 and NetCDF for yourself.

GFortran duplicate use of submodule / module

Gfortran 9.3.0 is sensitive to overlapping / duplicated use elements in a module - submodule hierarchy. That is, if a procedure is used in multiple places in the module - submodule hierarchy, only use the procedure once at the highest necessary level of the hierarchy.

This is perhaps best shown by example:

module foo
implicit none (type, external)
contains
subroutine bar()
end subroutine bar
end module foo

module parent
use foo, only : bar
implicit none (type, external)
interface
module subroutine baz
end subroutine baz
end interface
end module parent

submodule (parent) child
use foo, only : bar  !< this is unnecessary and triggers the Gfortran 9.3.0 error
implicit none (type, external)
contains
module procedure baz
end procedure baz
end submodule child

The error message from Gfortran 9.3.0 is like:

$ gfortran -c .\dupe.f90

dupe.f90:17:17:

   17 | submodule (parent) child
      |                 1
   18 | use foo, only : bar
      |                   2

Pytest ignoring Meson subprojects

Meson projects may contain Python code, including Meson subprojects. However, the Meson subproject code may not be relevant to the top-level Meson project Python code. Then, Pytest Python test suites may fail when the subprojects/ directory tree is searched and unwanted tests are run.

Ignore directories with Pytest: while this example is for Meson subprojects, it is obviously applicable to many other Python projects.

Add to file “pyproject.toml” in project top-level directory:

[tool.pytest.ini_options]
norecursedirs = subprojects .* build dist CVS _darcs {arch} *.egg venv

Python paramiko SFTP example

Paramiko Python SSH library provides a convenient SFTPClient that allows easy transfer of files over SSH. A distinction from the command line utility sftp is that “put"ing a file must include the full destination path including filename, to avoid OSError.

This Paramiko example shows copying a file from local computer to remote computer over SSH in Python.

from paramiko import SSHClient

source = "foo.txt"
dest = "~/Documents/foo.txt"

with SSHClient() as ssh:

    ssh.load_system_host_keys()
    ssh.connect("server.invalid")
    with ssh.open_sftp() as sftp:
        sftp.put(source, dest)