Python-based data analysis is dominating many fields.
In astronomy and geoscience applications, Python dominance is increasing.
Consider a widely valued language like Python inside of a niche, virtually unknown language outside a few specialties.
Installing Python for the first time:
Miniconda Python
is a 100 MB download–it won’t initially take several gigabytes of hard drive space like Matlab and IDL.
Package distribution can occur via a website or preferably, a centralized conda or pip repo.
For example, to install popular data analysis modules:
conda install pandas scipy matplotlib spyder
IDL vs. Python syntax gives NumPy/Python equivalents to common IDL commands
IDL is a bit distinctive in its syntax.
IDL and Matlab syntax remind me a bit of Fortran, while Python is C-based, including that Python has 0-based indexing and C-order array axes.
Mayavi also gives advanced 3-D visualization.
Jupyter Notebook makes an IDE in your web browser, and you can make remarkable animated, interactive 3-D plots as well with
Ipyvolume.
conda install jupyter
pip install ipyvolume
Ipyvolume interactive flow field.
HDF5 can handle large and small datasets quickly and easily.
Assume HDF5 file terabyte.h5 with double-precision float variable X of size 100000x2048x2048 (3.4 TB).
Let’s load the first frame of the 2048x2048 image data, and write it variable first to image.h5.
The with syntax uses Python’s context manager, which closes the file upon exiting the indented code section under with.
importh5pywith h5py.File('terabyte.h5') as f:
img = f['X'][0,...]
with h5py.File('image.h5') as f:
f['first'] = img
Removing the left N columns from a text file can be done with the one-liner below.
This example trims off the first 2 columns of each row of a text file “old.txt”, resaving as “new.txt”:
cut -c2- > new.txt < old.txt
For example, clean diff output of all the plus/minus signs or eliminate unwanted characters in the first column(s) of a text file.
Old Fortran or COBOL programs may have columns 73-80 filled with vestigial card sequence numbers.
Trim rightmost columns as in this example.
This keeps the first 72 columns of text from “ancient.for” on each row, resaving as “ancient.f”.
cut -c-72 > ancient.f < ancient.for
Any trailing whitespace (up to 72 characters for this example) remains.
Wake-on-LAN can also work worldwide (over WAN Internet), in case a PC 10,000 km away gets turned off accidentally.
To turn on a target PC from anywhere in the world, first setup the target PC and router port forwarding.
For simplicity/clarity we assume the following arbitrary parameters, which will be unique for the target Linux PC:
target interface: eth0
target PC static LAN IP: 192.168.1.100
target public WAN IP: 1.2.3.4
public WoL port forward: 10009
target MAC: 00:11:22:33:44:55
Remote Linux PC: modern Linux distros such as Ubuntu have WakeOnLan configured in NetworkManager.
Settings → Network and select the wired LAN interface to use for WakeOnLan.
under the “Ethernet” tab, seee that “Wake on Lan: Default” is checked.
No Wake on LAN password is needed for most use caes.
copy down this “Device” hexadecimal string, this is the MAC address to turn on the PC remotely.
Confirm the correct MAC address by typing in Terminal
ip a
Have someone nearby it in case it doesn’t turn it back on–shutdown remote PC for testing.
If PC didn’t turn on, ensure BIOS/UEFI settings have Wake On Lan enabled.
Try a discrete Intel Ethernet NIC, some motherboard NICs don’t work for Wake-on-LAN
Stan Solomon and the NCAR team have greatly cleaned up GLOW auroral energy deposition and kinetics model.
The new version number is 0.981, and since it’s mostly to Fortran 90/95 standards,
it’s easier to use and modify, along with being more stable.
Updated GLOW Python/Matlab code is available for new
GLOW 0.981.
The single line IF in both classic and modern Fortran only “captures” the first statement.
If you use a semicolon, the code after the semicolon is always executed!
Fortran 77 had
mod(),
which could output negative numbers.
This is often NOT the behavior you want, as other languages’ mod() functions only output positive numbers.
FFmpeg audio option -f alsa -ac 2 -i hw:1,0 uses -ac 2 for stereo to hardware address 1,0.
For default audio device, use -f alsa -i pulse.
Obtain hardware addresses via arecord -l.
Note that monoaural -ac 1 doesn’t work on some hardware.
-c:a aac -ar 44100 is AAC encode audio at 44.1 kHz sampling frequency (passing audio up to about 22 kHz, CD-quality).
FFmpeg video option -f v4l2 -r 10 -i /dev/video0 say to acquire video from webcam /dev/video0 at 10 fps, using the native resolution of the webcam.
Force webcam resolution like -s 640x480.
-c:v libx264 -pix_fmt yuv420p -preset ultrafast -g 20 -b:v 2500k encodes video using the H.264 video compression algorithm, prioritizing reduced CPU utilization, in blocks of 20 frames with a target bitrate of 2.5 Mbps.
FFmpeg config -threads 0 -bufsize 512k lets FFmpeg choose the number of CPU threads, with a 512kB buffer.
Adjust buffer size as needed–tradeoff between latency and robustness.
-f flv rtmp://a.rtmp.youtube.com/live2/YOURSTREAM &> stream.log uses the FLV container format, live stream to YouTube, with stream ID YOURSTREAM.
Don’t let others know the Stream ID!.
Screenshare: use the same command as above, swapping -f v4l2 -r 10 -i /dev/video0 with
-f x11grab -r 10 -s 1024x720 -i :0.0+0,24
-f x11grab -r 10 -s 1024x720 -i :0.0+0,24 sets screengrab at 10 fps, starting in the upper left hand corner, send a 1024x720 pixel video.
If your screen resolution is more than 1024x720, there would be unsent regions at the bottom and/or right side.
Restream.io has a free basic service that streams to over 30 sites, from a single incoming stream from FFmpeg.
While FFmpeg can “tee” live-stream to multiple sites, this uses N times the data bandwidth for N sites, which doesn’t scale well.
Restream.io takes one incoming stream and sends it to multiple sites, saving the streaming creator’s Internet data bandwidth.
This is useful for slow Internet connections or where Internet connection cost is expensive per gigabyte.
The free service from restream.io works for YouTube Live among others.
restream.io can send stream notifications to the feed.
restream.io sends preview/link to stream
use Titles on restream.io to set stream titles all at once (before stream start)
12000x5400 PNGs are available since 2010 (6000x2400 back to 1995).
This
Python program
downloads composite reflectivity
data
rapidly using multiple download threads.
Unfortunately a colormap-data table or function isn’t readily available.
Of course it’s inside the NEX2IMG program, but I would have to hack this a bit to extract the function.
Another approach would be to use the HSV map to bin the PNG data.
