Scientific Computing

Unix-like shells and Windows both limit how much data a new process can receive. Exceeding the limit produces Argument list too long (E2BIG) on POSIX, or a truncated / rejected command line on Windows.

ARG_MAX is the combined size of the argument vector and the environment block that exec / CreateProcess will accept.

getconf ARG_MAX

or from Python:

import os
print(os.sysconf("SC_ARG_MAX"))

POSIX defines ARG_MAX as the maximum length of the arguments to the exec family, including environment data. That means:

• every argv[i] string (plus its terminating NUL)
• every NAME=value environment string (plus NUL)
• implementation-specific accounting for pointer arrays, null terminators, and alignment

On Linux, the executable path is also included in the E2BIG check.

On Linux, GNU xargs reports how much of the ARG_MAX budget the current environment already uses:

true | xargs --show-limits

POSIX’s guaranteed minimum is only _POSIX_ARG_MAX = 4096 bytes, which could be a consideration on micro embedded systems. Real systems ARG_MAX values are usually much larger on modern systems. POSIX does not define a portable way to calculate the exact remaining room because implementations may account for pointers, null terminators, and alignment differently.

Linux

Two independent limits apply.

ARG_MAX: overall combined size of the argument vector and environment block.

• at most 1/4 of the current soft RLIMIT_STACK
• at most 3/4 of the kernel’s _STK_LIM (8 MiB) → 6 MiB ceiling
• at least 32 pages (128 KiB floor, since Linux 2.6.25)

Default stack is 8 MiB, so getconf ARG_MAX is typically 2097152 (2 MiB). Raising ulimit -s raises the reported value up to the 6 MiB cap; dropping the stack limit drops ARG_MAX toward 128 KiB.

<linux/limits.h> ARG_MAX header value is a compile-time minimum, not the runtime limit. Instead, use getconf or sysconf(_SC_ARG_MAX).

MAX_ARG_STRLEN: This is the limit constraining export of a huge variable.

A single argument or environment variable (NAME=value\0) cannot exceed MAX_ARG_STRLEN. Exceeding MAX_ARG_STRLEN results in execve returning E2BIG, even when the total is well under ARG_MAX. Raising MAX_ARG_STRLEN requires rebuilding the kernel.

macOS

Darwin does not have Linux’s per-string MAX_ARG_STRLEN cap. One argument or one environment variable can consume almost the entire ARG_MAX.

Typical values:

kern.argmax is not tunable at runtime.

The budget still includes the environment. A 128 KiB PATH can contribute to exec failures when combined with the rest of the environment and the command arguments, even though 1 MiB looks ample on paper.

Shell builtins (echo, printf) are not subject to ARG_MAX. External binaries are.

Windows

Windows splits the problem across several APIs. There is no getconf ARG_MAX.

So:

• A program calling CreateProcessW can pass a ~32k character command line and a large Unicode environment.
• Anything that goes through cmd.exe or a .bat file dies at 8191.
• PowerShell and the Win32 APIs can set a single variable closer to the 32,767 character documented max; cmd cannot.

The environment block format is:

NAME=value\0NAME=value\0\0

Names cannot contain =.

How the two limits interact

Setting a huge variable in the current shell often succeeds. The failure happens at the next exec:

# Linux: this assignment may work in bash
export HUGE=$(python3 -c "print('x'*200000)")
# this exec fails with E2BIG because 200000 > MAX_ARG_STRLEN
/bin/true

On macOS the same pattern fails when len("HUGE=")+200000 plus the rest of the environment exceeds ARG_MAX.

On Windows, cmd.exe will not even accept a set line longer than 8191 characters, while CreateProcessW with an explicit environment block can go much further.

Prove the Linux per-string cap

import os, subprocess

for n in (131069, 131070, 131071):
    env = os.environ.copy()
    env["H"] = "x" * n
    try:
        subprocess.check_call(["/bin/true"], env=env)
        print(n, "ok")
    except OSError as e:
        print(n, e)

On a 4 KiB page Linux kernel, 131069 typically succeeds and 131070 fails because H= plus the value and the terminating NUL must fit within MAX_ARG_STRLEN.

Workarounds

• Do not put large payloads in the environment or argv. Use a file, pipe, socket, or response file (@args.txt).
• xargs / find -exec … {} + chunk argv to stay under the total.
• On Linux, ulimit -s only helps the total, not MAX_ARG_STRLEN.
• On Windows, call CreateProcessW directly (or PowerShell / Python subprocess with env=) instead of going through cmd.exe.
• Keep PATH short; it is the environment variable that most often eats the budget.

Summary

The 128 KiB figure often quoted for Unix is the Linux per-string cap (MAX_ARG_STRLEN), not the total, and not a universal Unix number.

On macOS, Terminal commands can be specified to run as a Rosetta CPU arch translated binary using the arch command. If supported by the macOS version, for Apple Silicon CPUs Rosetta can be installed with the following command:

softwareupdate --install-rosetta

Then run an x86_64 command like:

arch -x86_64 <command> <command options>

To run cmake as an x86_64 binary on an Apple Silicon Mac:

arch -x86_64 cmake -B build

Check the architecture of the current shell process:

arch -x86_64 uname -m

Universal or x86_64 binaries required

When using

arch -x86_64 <command>

to run a program, the program must be a universal binary or an x86_64 binary. If not, the error message will be similar to:

arch: posix_spawnp: <command>: Bad CPU type in executable

For programs like CMake that are designed for cross-compiling, command CMake to build for x86_64 by setting the CMAKE_OSX_ARCHITECTURES variable to x86_64:

cmake -B build -DCMAKE_OSX_ARCHITECTURES=x86_64

cmake --build build

It’s desirable to run the native version of the app rather than the Rosetta CPU arch translated binary. Determine if an app is native, Universal, or other architecture with lipo:

lipo -archs /path/to/program

lipo doesn’t search $PATH, so you need to provide the full path to the program, which can be done like:

lipo -archs "$(command -v program_name)"

For example

lipo -archs "$(command -v cmake)"

A symptom of running an x86_64 app on an Apple Silicon CPU without Rosetta is like

arch -x86_64 /usr/bin/true

arch: posix_spawnp: /usr/bin/true: Bad CPU type in executable

Or

❯ cmake-3.13.5/CMake.app/Contents/bin/cmake

zsh: bad CPU type in executable: cmake-3.13.5/CMake.app/Contents/bin/cmake

iso_fortran_env is respected by modern Fortran compilers, as implied by its name. iso_fortran_env is part of the Fortran 2003 specification and popular Fortran compilers added it by calendar year 2010 generally.

Why use iso_fortran_env terminal I/O: legacy programs written before Fortran 2003 often write to terminal with:

write(*,*) 'The value of X is ',x

or

write(6,*) 'The value of X is ',x

This is a problem when trying to debug with text output to terminal, especially where someone has used “6” for file I/O unit by mistake. Or, if trying to write to a file with an uninitialized unit number, stderr gets redirected to file fort.0.

For operating systems including macOS, Windows, and Linux the standard terminal units correspond like:

Legacy code making use of open(unit=) number in this range can cause unexpected behavior. Please use open(newunit=...) to avoid conflicts with the standard terminal units. Of note, GFortran can use environment variables to change the default unit numbers for standard input, output, and error as in the table above.

Example

Print repeatedly to the same line, and combine prompt text on the same line with input.

iso_fortran_env terminal I/O: example prints to stdout, then stderr and finally asks for user input with a prompt on the same line.

program myterm
use iso_fortran_env
implicit none (type, external)
character(1000) :: usertxt  ! 1000 is an arbitrarily large number
integer :: ios

! could also just use print *,'printed to stdout'
write(output_unit,*) 'Printed to stdout'

write(error_unit,*) 'printed to stderr'

! prompt with caret on same line as input, here using a greater than sign >
write(output_unit,'(A)',advance='no') ' >'
flush(output_unit)

read(input_unit,"(A)", iostat=ios) usertxt
! trap Ctrl-D EOF on Unix-like systems to avoid crashing program
if (ios/=0) backspace(input_unit)  ! ctrl D gobble

write(output_unit,*) usertxt

end program

If stderr goes to file fort.0

If stderr from error_unit gets written to a file fort.0 instead of being printed to screen, this is an indication of open(u)ing a file without first setting a value for u, which might default to 0.

Unless needing to persist a file opening between calls of a function/subroutine, normally open a file with newunit.

program myfile
use iso_fortran_env
implicit none (type, external)

integer :: u, ios
character(1000) :: fn ! 1000 is an arbitrarily large number
character(1000) :: dat

print *, "please input file to open"

read(input_unit, '(a)', iostat=ios) fn
if(is_iostat_end(ios)) stop

! open file, using better to ask forgiveness than permission principle
! status='old' means generate error if file doesn't exist
open(newunit=u,file=fn, status='old',action='read',iostat=ios)
if (ios /= 0) then
    write(error_unit,*) 'could not open file ',trim(fn)
    error stop 'file IO error'
endif

read(u,'(A)') dat
print *,'first two lines of ',trim(fn),' are:'
print *,trim(dat)
read(u,'(A)') dat
print *,trim(dat)

close(u)  ! implicitly closed at end of program, but as good practice...
end program

Related:

• Fortran iostat error code
• Fortran stream buffering

UTM virtual machine host has a built-in display, but its terminal is rather limited and currently doesn’t allow scroll back. It’s convenient to use the serial device with a terminal program like screen from the host OS. To configure the serial device, power off the VM. Add a Serial Device under the UTM VM settings. Before starting the VM, note on the VM settings main page (starts with “Status Stopped”) the name of the “Serial (TTY)” device like /dev/ttys005 or similar.

The GNU screen program can be used from the host OS to connect efficiently to the guest OS.

brew install screen

From the host OS Terminal, connect like

screen /dev/ttys005

The serial device with screen enables scroll back through the terminal. On the host OS, edit the file “~/.screenrc” to include the following lines:

defscrollback 10000
termcapinfo xterm* ti@:te@

defscrollback 10000

Sets the default scrollback buffer to 10000 lines in screen.

termcapinfo xterm* ti@:te@

allow scrollback to work as expected.

Homebrew can select GCC versions (and versions of other Homebrew formulae) like:

brew install gcc@15

Then create a file like “~/gcc-15.sh” with the following content:

v=15
prefix=$(brew --prefix gcc@$v)/bin

export CC=$prefix/gcc-$v CXX=$prefix/g++-$v FC=$prefix/gfortran-$v

Activate the GCC version in this shell session by running:

source ~/gcc-15.sh

If there is still trouble, try modifying SDKROOT to find a compatible SDK for the selected Homebrew GCC version.

Using an older version of Homebrew-distributed library or program could be useful to workaround bugs, or if one wants to still use their Intel CPU Mac hardware with Homebrew as GCC 16 isn’t supported by Homebrew on Intel CPUs on macOS for example. The Homebrew bottle binaries or source compile scripts must be available for the CPU architecture and OS version of the computer.

Do not use shell aliases in non-interactive shell scripts to stay shell-agnostic.

In Bash, aliases are not expanded in non-interactive shells unless the script first invokes

shopt -s expand_aliases

In Zsh, aliases are expanded in non-interactive scripts as well. However, while on macOS the default interactive shell is Zsh, non-interactive scripts with the usual #!/bin/sh shebang run Apple’s Bash in POSIX mode instead of Zsh.

List the aliases defined in the current shell (Linux, macOS, BSD) with:

alias

Our practice is to avoid aliases in non-interactive scripts and use explicit paths or functions instead. For example, to pin a user-specified GCC for CMake, pass the compiler paths directly:

#!/bin/sh

prefix="$HOME/.local/bin"
gcc_version=17
CC="$prefix/gcc-$gcc_version"
CXX="$prefix/g++-$gcc_version"
FC="$prefix/gfortran-$gcc_version"

cmake -B build -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_Fortran_COMPILER=$FC

cmake --build build

Windows PowerShell

Although the preceding discussion is for Unix-like systems, Windows PowerShell can also use aliases and has similar considerations for non-interactive scripts.

To list the aliases defined in the current PowerShell session:

Get-Alias

An analogous script for non-interactive PowerShell scripts would be like:

$prefix = "$HOME/.local/bin"
$gcc_version = 17
$CC = "$prefix/gcc-$gcc_version"
$CXX = "$prefix/g++-$gcc_version"
$FC = "$prefix/gfortran-$gcc_version"

cmake -B build -DCMAKE_C_COMPILER=$CC -DCMAKE_CXX_COMPILER=$CXX -DCMAKE_Fortran_COMPILER=$FC

cmake --build build

Adobe Reader can export and view a list of comments through the Comment tab on the right side. To do this from a command line program (for example to feed AI analysis / training) consider a Python script using the high performance PyMuPDF (fitz) library:

import fitz
import sys

file = sys.argv[1]

doc = fitz.open(file)

for i, page in enumerate(doc, start=1):
    for a in page.annots() or []:
        info: dict = a.info

        print(
            f"page {i}  {a.type[1]}  "
            f"author={info.get('title')!r}  "
            f"text={info.get('content')!r}"
        )

        # for highlights, get the actual selected words:
        if a.type[1] in ("Highlight", "Underline", "StrikeOut", "Squiggly"):
            print("  quoted:", page.get_textbox(a.rect))

On Linux, the CUPS PDF program saves printed PDFs to the ~/PDF directory from any program.

apt install printer-driver-cups-pdf
# or
dnf install cups-pdf

# then
systemctl restart cups

Verify a PDF printer is visible:

lpstat -p -d

The PDF output directory can be configured by editing the “Out” directory in /etc/cups/cups-pdf.conf

Higher quality PDF might be obtained by using the PDF print options in the specific program, if available.

LibreOffice PDF Conversion

Alternatively, LibreOffice can print compatible documents to PDF directly from the command line in headless mode like:

soffice --headless --convert-to pdf file.docx

Libreoffice can be installed like

apt install libreoffice-common

Related: convert image stack to PDF

The Fortran standard does not define a specific Fortran module file format. Each compiler vendor has a unique incompatible Fortran module file format. Fortran module files are not portable between different compilers or even different versions of the same compiler.

LLVM Flang

The LLVM Flang .mod files generated are legal Fortran syntax – they are text files. The .mod format gives the version number, which may be seen like:

head -n1 <moduleName>.mod

The output starts like:

!mod$ v1

GNU Fortran (GFortran)

The GFortran header version is defined in module.cc as variable “MOD_VERSION”. GNU Fortran “gfortran” .mod files are GZIP and is not documented

Examine Gfortran .mod file header like:

gunzip -c <moduleName>.mod | head -n1

The output starts like:

GFORTRAN module version ‘15’ created from …

Intel oneAPI (ifx)

Intel oneAPI .mod files are a proprietary binary format. It is possible to determine the version of the .mod file by using od to look at the first 2 bytes of the .mod file.

od -An -N4 -d <moduleName>.mod

The first number is like “13” and is the module format version. This version may change over time as oneAPI internals change. The second number is the update version, which is fixed at “1”.

There is an undocumented “ifx” compiler option -switch:fe_module_dump that outputs the module and submodule data in text format.

NVHPC and AOCC

NVIDIA HPC SDK (NVHPC) and AOCC compilers generate .mod files that are text files. The format for legacy Flang module files is distinct from LLVMFlang Fortran module files.

The .mod file is a text file, beginning with the version number.

head -n1 <moduleName>.mod

The output is like:

V34 :0x24 dummy

Cray Fortran

By default, Cray Fortran stores uppercase DUMMY.mod filenames. This can be made lowercase with the ftn -ef flag. The Cray Fortran .mod format is proprietary, but the version number might be seen like:

head -n2 <moduleName>.mod

Related: Fortran submodule file naming