Scientific Computing

Python 3.7 was released in June 2018, adding performance to common operations, and adds user-visible changes in the following categories.

The boilerplate copy-paste required for Python classes can seem inelegant. Python 3.7 data class eliminates the boilerplate code in initializing classes. The @dataclass decorator enables this template.

@dataclass
class Rover:
    '''Class for robotic rover.'''
    name: str
    uid: int
    battery_charge: float=0.
    temperature: float

    def check_battery_voltage(self) -> float:
        return self.aioread(port35) / 256 * 4.1

Python 3.7 introduced breakpoint, which breaks into the debugger.

x=1
y=0

breakpoint()

z = x/y

It’s very common to have more than one version of Python installed. Likewise, multiple versions of the same library may be installed, overriding other versions. For example, system Numpy may be overridden with a pip installed Numpy.

Python ≥ 3.7 gives the absolute path and filename from which the ImportError was generated.

from numpy import blah

Python < 3.7:

ImportError: cannot import name ‘blah’

Python ≥ 3.7:

ImportError: cannot import name ‘blah’ from ’numpy’ (c:/Python37/Lib/site-packages/numpy/init.py)

The popular and efficient argparse module can now handle intermixed positional and optional arguments, just like the shell.

from argparse import ArgumentParser
p = ArgumentParser()
p.add_argument('xmlfn')
p.add_argument('--plottype')
p.add_argument('indices',nargs='*',type=int)
p = p.parse_intermixed_args()   # instead of p.parse_args()

print(p)

python myprogram.py my.xml 26 --plottype inv 2 3

Namespace(indices=[26,2,3], plottype=‘inv’, xmlfm=‘my.xml’)

whereas if you have used p.parse_args() you would have gotten

error: unrecognized arguments: 2 3

Note: optparse was deprecated in 2011 and is no longer maintained.

Python ≥ 3.7 can do

import a.b as c

instead of Python ≤ 3.6 needing

from a import b as c

The discussion makes the details clear for those who are really interested in Python import behavior.

Python ≥ 3.7 disassembler dis.dis() can reach more deeply inside Python code, adding a depth parameter useful for recursive functions, and elements including:

• list comprehension: x2 = [x**2 for x in X] (greedy eval)
• generator expressions: x2 = (x**2 for x in X) (lazy eval)

Case-insensitive regex sped up by as much as 20x.

Python 3.7 added constants that allow controlling subprocess priority in Windows. This allows keeping the main Python program at one execution priority, while launching subprocesses at another priority. The ability to start subprocesses without opening a new console window is enabled by subprocess.CREATE_NO_WINDOW. The confusingly named but important universal_newlines boolean parameter is now named text. When text=True, stdin/stderr/stdout will emit/receive text stream instead of bytes stream.

pyproject.toml allows fine-grained control of supported Python versions. The minimum Python project version is set like:

[project]
requires-python = ">=3.10"

Python versions required specification can be a range of versions or specific list of version(s)

Pytest is the de facto standard for Python unit testing and continuous integration. To be complete in testing, one should test the interactive console scripts that for many Python programs is the main method of use.

Console script testing can be added through Pytest Console Scripts addon, but I usually simply use subprocess.check_call directly like Pytest Console Scripts addon does.

Note that “sys.executable” is the recommended way to securely get the Python executable path, to ensure testing with the same Python interpreter.

import pytest
import subprocess
import sys

def test_find():
    subprocess.check_call([sys.executable, '-m', 'mypkg'])

This is for a package configured with __main__.py or __init__.py such that in normal use, the user types:

python -m mypkg

to run the Python program.

Matlab should generally be installed NOT using sudo. Upon upgrading operating system, or if you installed Matlab on a laptop using a docking station, and then run off the docking station, Matlab may complain about a changed host ID.

If Matlab is already installed, but won’t open the desktop due to a licensing error,reactivate Matlab:

$(dirname $(realpath $(which matlab)))/activate_matlab.sh

• Matlab activation reference
• Matlab Host ID

Get the host ID (MAC address) by:

ip a

look for the WiFi link/ether hexadecimal value. If connected to the internet via WiFi, you can confirm the correct device by comparing the value for inet or inet6 vs. https://ident.me

Install to the home directory and do NOT use sudo. Make a directory for Matlab installs:

mkdir ~/.local/matlab

Start the Matlab install NOT as root or sudo

./install

Install to directory like “~/matlab/” Activate via Internet and sign in to select the license key.

GQRX is popular for RTL-SDR receivers on Linux:

adduser $(whoami) plugdev

apt install gqrx-sdr rtl-sdr librtlsdr-dev

You can also download the latest release of GQRX.

In GNU Radio Companion, look for the RTL-SDR Source block.

Test RTL2832 PLL Frequency range:

rtl_test -t

Output should be like:

E4000 tuner

Found 1 device(s): 0: ezcap USB 2.0 DVB-T/DAB/FM dongle
Using device 0: ezcap USB 2.0 DVB-T/DAB/FM dongle
Found Elonics E4000 tuner
Supported gain values (18): -1.0 1.5 4.0 6.5 9.0 11.5 14.0 16.5 19.0 21.5 24.0 29.0 34.0 42.0 43.0 45.0 47.0 49.0
Benchmarking E4000 PLL...
E4K PLL not locked for 53000000 Hz!
E4K PLL not locked for 2217000000 Hz!
E4K PLL not locked for 1109000000 Hz!
E4K PLL not locked for 1248000000 Hz!
E4K range: 54 to 2216 MHz
E4K L-band gap: 1109 to 1248 MHz

R820

Found 1 device(s):  0:  Realtek, RTL2838UHIDIR, SN: 00000001

Using device 0: Generic RTL2832U OEM
Detached kernel driver
Found Rafael Micro R820T tuner
Supported gain values (29): 0.0 0.9 1.4 2.7 3.7 7.7 8.7 12.5 14.4 15.7 16.6 19.7 20.7 22.9 25.4 28.0 29.7 32.8 33.8 36.4 37.2 38.6 40.2 42.1 43.4 43.9 44.5 48.0 49.6
[R82XX] PLL not locked!
Sampling at 2048000 S/s.
No E4000 tuner found, aborting.
Reattached kernel driver

Record the entire passband ~ 2 MHz bandwidth, not just the demodulated audio. Example command:

rtl_sdr ${TMPDIR}/cap.bin -s 1.8e6 -f 90.1e6

Press Ctrlc to stop recording after several seconds so that your hard drive doesn’t fill up. You can read the cap.bin file in MATLAB, Python or GNU Radio.

Troubleshooting:

• is RTL-SDR recognized? Before and after inserting the RTL-SDR receiver into the USB port of your Linux PC, type:

  lsusb

  should show Realtek device.

• try a different, non-USB 3 port (USB 2).

• librtlsdr0 provides file /lib/udev/rules.d/60-librtlsdr0.rules that allows the RTL-SDR stick to be recognized upon USB plugin.

• dmesg should show dozens of messages with RTL2832 when the USB receiver is plugged in

Other popular programs for the RTL-SDR:

• MATLAB RTL-SDR support has several examples and a free eBook. Matlab also supports USRP and PLUTO SDR hardware among others.

• GNU Radio (start with GNU Radio Companion graphical SDR IDE)

  apt install gnuradio

• pyrtlsdr: pure Python wrapper for librtlsdr and less bulky than GNU Radio.

• SDR#

• CubicSDR

HDF5 files do not have an error recovery mechanism and do not journal. There is an optional per-variable error checksum Fletcher32 to detect data corruption. Checking/comparing file size alone is not an adequate check for HDF5 corruption.

Here a few easy techniques to check for corrupted HDF5 files.

Python HDF5 checking script checks HDF5 files for corruption and optionally finds the corrupted block(s) and variable(s)

HDF5 shell tools are installed by:

• Linux: apt install hdf5-tools
• macOS
• Windows: use MSYS2 or WSL

h5stat file.h5

Print the data values in the file

h5dump file.h5

HDFview GUI appears to use the Fletcher32 checksum to show a red question mark if corruption is detected. Another curiosity is that the Object reference is 2³² - 1 on the corrupted variable.

Related: HDF5 GUIs to view and edit variables in .h5 files

Using CMake and a few dashes of modern Fortran techniques, the ASCII text art Peanuts / Snoopy calendar lives again on GitHub. PDF output and inputting of one’s own ASCII image files is now possible.

In Spyder IDE when using Command Line Options under:

Run → Configure → General Settings → Command Line Options

be sure no line ending is in the parameter string. Otherwise, opening the Run Command may have error:

EOL while scanning string literal

Try deleting trailing spaces–also check any of the interword spaces if it’s still happening.

Virtual machine image sharing between computers works best with operating systems that are not license-bound to hardware. Linux and ReactOS are good candidates. Windows is not a good candidate for virtual machine sharing since Windows checks the hardware. The image can reside on a fast USB drive or network share.

Global ITU license-free frequencies

Individual countries may choose the details for their lands, or may open up additional frequencies. Generally this is a place to start.

• 26.957-27.283 MHz ISM band (ITU RR 5.150)
• 40.66-40.7 MHz ISM band (ITU RR 5.150)

Country-specific license-free

I don’t list every allocation because they are so numerous–just a few countries I ran across. This is a conceptual table–confirm yourself.

Global ISM bands (from ITU), detailed

Related:

• 900 MHz global license-free spectrum
• US license-free 25-60 MHz radar