Scientific Computing

Matlab plotting can be quite slow, as can Python Matplotlib plotting. Sometimes, Matlab parfor can be used to plot in parallel, when all parfor restrictions are met. However, parallel plotting in Matlab doesn’t always work, or may work on some operating systems but not others. So use great caution if making a “parfor” plotting loop–it may not work for others.

Problems when trying to do relatively simple plots in parallel:

Warning: A worker aborted during execution of the parfor loop. The parfor loop will now run again
on the remaining workers.

Error using distcomp.remoteparfor/rebuildParforController (line 194)
All workers aborted during execution of the parfor loop.

Warning: worker(s) crashed while executing code in the current parallel pool. MATLAB may attempt
to run the code again on the remaining workers of the pool, unless an spmd block has run. View the
crash dump files to determine what caused the workers to crash.

When desired and permitted by the computer filesystem, it’s possible to change file ownership. This may be necessary when a file was inadvertently created with root / admin ownership by mistake, and it’s necessary for a general user to edit or access the file. Changing file ownership can have unexpected consequences, like removing the ability of others to access the file or run a program depending on the file. Therefore, file ownership changes should be done only when necessary and with consideration for others who may depend on the file.

These examples assume the file “example.txt” is in the current directory and the user logged in should own the file. These tasks are similarly done in other languages such as Go or Julia.

As long as the user has filesystem permission, Python can easily change file ownership across operating systems.

import shutil
import getpass

shutil.chown("example.txt", user=getpass.getuser())

• username
• change file ownership

On Windows check ownership of the file by Command Prompt:

dir /q example.txt

or PowerShell:

(get-acl example.txt).owner

Windows uses the takeown command to change file ownership. For simplicity we assume the desired user is logged in and executing the command.

takeown /f example.txt

On macOS / Linux check ownership of the file by

ls -l example.txt

For simplicity we assume the desired user is logged in and executing the command chown:

chown $whoami:$whoami example.txt

The “python_history” file contains recent commands entered into the Python interpreter. “python_history” files with incorrect permissions can cause a message on exiting Python like:

Error in atexit._run_exitfuncs:
PermissionError: [Errno 13] Permission denied

Find the owner of ~/.python_history

• macOS / Linux: ls -l ~/.python_history
• Windows / PowerShell: (get-acl ~/.python_history).owner

If it does not show the current username, that’s likely the problem.

Fix

This changes the file ownership to the current user.

• macOS / Linux: chown $whoami:$whoami ~/.python_history
• Windows PowerShell: takeown /f $home\.python_history

Notes

• disable ~/.python_history

git-filter-repo is a Python program that is generally recommended over Git filter-branch and BFG Repo Cleaner. Other users of your repo will get an “unrelated history” error so they will have to reset or reclone the repo.

Example

Suppose an incorrect email address was used to make previous Git commits. To update Git commit email address history from the Git repo directory:

git filter-repo --mailmap mailmap --force

where file “mailmap” has mailmap file format email addresses to change like:

Jane Doe <jane@new.com> <jane@old.com>
John Smith <john@new.com> <john@old.com>

PEP517 / PEP518 introduced pyproject.toml as a supplement and even replacement for setup.py. Pyproject.toml has been adopted by major Python projects including PyTest. Some computers, especially those using system Python may be configured without a user site-packages directory. When pyproject.toml is present, systems without a Python user site-packages directory may fail on pip install -e . with errors including:

running develop WARNING: The user site-packages directory is disabled. error: can’t create or remove files in install directory

The fix below will work with or without pyproject.toml, installing just like usual:

• pip install . → under user site-packages, a static copy of the package (any package code changes require package reinstall)
• pip install -e . → Python-style link in user site-packages to this code directory (live development code)

Note: some systems require python3 -m pip instead of plain pip.

Fix

A general fix is to make setup.py contain site.ENABLE_USER_SITE = True. If the computer admin has locked down the user directory (very rare) this may still not work.

This installs this package, as well as any prerequisites specified and not already installed under the Python user directory site.USER_SITE.

#!/usr/bin/env python3
import site
import setuptools

# PEP517 workaround
site.ENABLE_USER_SITE = True

setuptools.setup()

Notes

• Pip issue discussing this matter

curl and Wget are popular web interface command line utilities that among their many features can resume interrupted or canceled downloads. curl is a library with a command-line frontend, while Wget is a command line tool.

Daniel Stenberg’s curl has a permissive license that has perhaps led it to become very widespread, installed “from the factory” in operating systems from Windows, macOS, Linux to automobiles and many other embedded systems. curl’s man page is perhaps best used by text search.

curl -O -C - https://mysite.example

-C -

--continue-at - continue a previous transfer, automatically determining byte offset.

A distinct Wget capability is recursive downloads.

wget -C htttps://mysite.example

NetCDF4 is a powerful scientific data format based on HDF5 standard. NetCDF4 is directly usable by ParaView including the CF (Climate and Format) metadata convention. We have created a FindNetCDF.cmake that efficiently finds NetCDF libraries for use in CMake.

Example:

Place FindNetCDF.cmake file under project cmake/ then from CMakeLists.txt:

list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake)

find_package(NetCDF COMPONENTS Fortran REQUIRED)

# example target
add_executable(main main.f90)
target_link_libraries(main PRIVATE NetCDF::NetCDF_Fortran)

Using HDF5 from any language can be intimidating if directly using the low-level HDF5 API. HDF5 interfaces have sprung up for popular languages that Make HDF5 trivially easy to use, such as Python h5py. Now Fortran also has an easy, powerful object-oriented and functional HDF5 interface named h5fortran. h5fortran builds and connects to the project using CMake or Meson. h5fortran uses Fortran 2018 standard code and is tested across multiple compilers including GFortran and Intel oneAPI. The efficient h5fortran interface is polymorphic (rank/dimension and type-agnostic) from scalars through 7-D arrays. A companion library for NetCDF is nc4fortran.

Examples

H5fortran is simpler to use than even Matlab HDF5 interface.

Functional interface example:

use h5fortran

call h5write('foo.h5', '/x', x)

call h5read('bar.h5', '/y', y)

Object-oriented interface examples:

use h5fortran

type(hdf5_file) :: h5f

call h5f%open('test.h5', status='new')

call h5f%write('/value1', 123.)

call h5f%close()

Using NetCDF4 from any language can be intimidating if directly using the low-level NetCDF4 API. NetCDF4 interfaces have sprung up for popular languages that Make NetCDF4 trivially easy to use, such as Python netcdf4. Now Fortran also has an easy, powerful object-oriented NetCDF4 interface named nc4fortran. nc4fortran builds and connects to the project using CMake or Meson. nc4fortran uses object-oriented Fortran standard code and is tested across multiple compilers including GFortran and Intel oneAPI. The efficient nc4fortran interface is polymorphic (rank/dimension and type-agnostic) from scalars through 7-D arrays. A companion library for HDF5 is h5fortran.

use nc4fortran

type(netcdf_file) :: hf

call hf%open('test.nc', status='new')

call hf%write('/value1', 123.)

call hf%close()

Nc4Fortran is simpler to use than even Matlab NetCDF4 interface.

IPyParaView from NVIDIA brings the interactive local / remote 3-D visualization power of ParaView to Jupyter Notebook. IPyParaView works on Linux, macOS and Windows.

First install ParaView, which is free and open-source. ParaView does not require a discrete GPU. ParaView will generally detect an use a discrete GPU if present, or use the CPU otherwise. CPU-integrated GPUs like Intel Iris are also used by ParaView.

We suggest using Miniconda Python and making a separate Conda environment for IPyParaView. Make the Python release match ParaView. Then install packages:

conda install ipython jupyterlab traitlets

Setup IPyParaView:

git clone https://github.com/nvidia/ipyparaview
cd ipyparaview

• Windows: from PowerShell run ./build.ps1
• macOS / Linux: run ./build.sh

Further details and examples are noted in the IPyParaView example repository.