This file is part of eRCaGuy_dotfiles: https://github.com/ElectricRCAircraftGuy/eRCaGuy_dotfiles

(click to expand)

See also
Serial terminal programs
picocom
Scripting the sending and receiving of commands over serial
Transferring files over serial

Serial terminal programs

picocom [my favorite]
1. https://github.com/npat-efault/picocom
2. See my installation instructions & notes below.
minicom
Arduino Serial Monitor and Serial Plotter
eRCaGuy_PyTerm - a datalogging serial terminal/console program which I wrote, written in Python.
1. https://github.com/ElectricRCAircraftGuy/eRCaGuy_PyTerm
2. Can be used in place of minicom, picocom, or the Arduino Serial Monitor.
3. This program works fine, but for general logging, you might consider picocom, however, instead. See its logging command just above.

`picocom`

Background

picocom [my preferred command-line serial terminal program]: a serial terminal command-line program to communicate with serial devices such as Arduinos or embedded Linux boards over serial. Use it as an alternative to minicom or the Arduino Serial Monitor.
https://github.com/npat-efault/picocom
1. Manual pages:
  1. man format: picocom.1
  2. HTML: picocom.1.html
  3. Markdown [my preference]: picocom.1.md
  4. PDF: picocom.1.pdf

Linux Installation [tested and works]

First, add yourself to the dialout group so you can access serial devices without using sudo or being root. See my instructions here: ../arduino/README.md.

# Add your user to the "dialout" group
sudo usermod -a -G dialout $USERNAME
# Now, log out entirely from Ubuntu, then log back in. You will now have the privileges of
# this group! Let's verify that. You should see `dialout` as one of the words printed 
# when you run this:
groups $USERNAME

Install dependencies

sudo apt update 
# Install `lrzsz`. This gives you access to the XMODEM, YMODEM,
# and ZMODEM protocol executables such as `rz` and `sz`, used to send and
# receive files. More on this below. 
sudo apt install lrzsz
# OR (this also installs `lrzsz` automatically as a dependency of minicom)
sudo apt install minicom

Download, build, & install

# --------------------------------------------------------------------------
# 1. Download and build the program
# --------------------------------------------------------------------------

# cd to wherever you'd like to download the program
cd ~/GS/dev
git clone https://github.com/npat-efault/picocom.git
cd picocom
make  

# The `picocom` executable has now been created and can be run directly
# (withOUT "installing" it first) simply by using the appropriate path to
# it.
#
# Examples:
#
# View the help menu and the version; the version is listed on the first
# line
./picocom --help
# See just the version line from the help menu
picocom -h | head -n 1

# For more-detailed usage, see the "General Usage" section I wrote below.

# View the "man" (manual) pages manually like this:
man ./picocom.1
# See just the version line from the man pages
man picocom | tail -n 1

# --------------------------------------------------------------------------
# 2. Install the program by adding symlinks to it to your PATH variable
# --------------------------------------------------------------------------

# create the ~/bin dir if it doesn't already exist
mkdir -p ~/bin
# add a symlink to the executable
ln -si "$PWD/picocom" ~/bin

# re-source your ~/.profile file; don't know what "source" means?
# Read my answer here: https://stackoverflow.com/a/62626515/4561887
. ~/.profile

# `picocom` will now be in your PATH since Linux Ubuntu's default
# ~/.profile file adds it to the path like this:
#
#       # set PATH so it includes user's private bin if it exists
#       if [ -d "$HOME/bin" ] ; then
#           PATH="$HOME/bin:$PATH"
#       fi

# --------------------------------------------------------------------------
# 3. Install the man (manual) pages for the program
# [keywords: how to install man pages; installing man pages installation]
# --------------------------------------------------------------------------

# add a symlink to the man page so you can do `man picocom`
# See: https://askubuntu.com/a/244810/327339
sudo mkdir -p /usr/local/share/man/man1
sudo ln -si "$PWD/picocom.1" /usr/local/share/man/man1/

# Update `man`'s internal database
sudo mandb

# Verify the new manpage, with the new `picocom` version at the very bottom,
# is now properly installed and working!
man picocom

# The version numbers here should now match:
# 1. See just the version line from the help menu
picocom -h | head -n 1
# 2. See just the version line from the man pages
man picocom | tail -n 1

Windows Installation [WIP; not yet functional]

Install and configure MSYS2 per my instructions here: Installing & setting up MSYS2 from scratch, including adding all 7 profiles to Windows Terminal.
Install and configure Git Bash.
Open Windows Terminal. Open two terminals:
1. Git Bash, for git commands.
2. MSYS2: ucrt64, for building.

In the Git Bash terminal, clone the repo:

# cd to wherever you'd like to download the program
cd ~/GS/dev
git clone https://github.com/npat-efault/picocom.git

In the MSYS2 terminal:

# Update MSYS2
pacman -Suy

# Install dependencies
pacman -S msys2-runtime-devel

# build
cd ~/GS/dev/picocom
time make
###### fails :(

# $ time make
# cc -Wall -g -DVERSION_STR=\"3.2a\" -DTTY_Q_SZ=0 -DHIGH_BAUD -DUSE_FLOCK -DHISTFILE=\".picocom_history\" -DLINENOISE -o linenoise-1.0/linenoise.o -c linenoise-1.0/linenoise.c
# linenoise-1.0/linenoise.c:108:10: fatal error: termios.h: No such file or directory
#   108 | #include <termios.h>
#       |          ^~~~~~~~~~~
# compilation terminated.
# make: *** [Makefile:65: linenoise-1.0/linenoise.o] Error 1

General Usage

# Run picocom, connecting to serial device "/dev/ttyUSB0" at a baud rate of
# 115200
# - To exit, type Ctrl + A, Ctrl + X. See the readme here:
#   https://github.com/npat-efault/picocom
#   Search that document for "To exit picocom you have to type" and "C-a, C-x".
# - Ctrl + C does NOT exit the program because you may want Ctrl + C to actually
#   be forwarded to your remote serial device instead! So, Ctrl + C gets
#   forwarded to your device. Ctrl + A, Ctrl + X, therefore, is required to
#   exit picocom. 
picocom --baud 115200 /dev/ttyUSB0

# Change the serial baud rate **of the active terminal** to 115200.
# - see more `stty` cmds in my "git & Linux cmds" doc
stty 115200

# View the help menu and the version; the version is listed on the first line
picocom --help

# Check the man (manual) pages
man picocom

# Logging:
# You can data-log all serial input/output by appending to a log file like this; 
# see `man picocom` and search for `--logfile` for details
picocom --baud 115200 --logfile serial_log.txt /dev/ttyUSB0

Start running the program with, for instance:
```
picocom --baud 115200 /dev/ttyUSB0
```
Exit the program with Ctrl + A then Ctrl + X.

Press Ctrl + A, Ctrl + H for the special command help menu while running. Here is the full menu displayed when you run this:

*** Picocom commands (all prefixed by [C-a])

*** [C-x] : Exit picocom
*** [C-q] : Exit without reseting serial port
*** [C-b] : Set baudrate
*** [C-u] : Increase baudrate (baud-up)
*** [C-d] : Decrease baudrate (baud-down)
*** [C-i] : Change number of databits
*** [C-j] : Change number of stopbits
*** [C-f] : Change flow-control mode
*** [C-y] : Change parity mode
*** [C-p] : Pulse DTR
*** [C-t] : Toggle DTR
*** [C-g] : Toggle RTS
*** [C-|] : Send break
*** [C-c] : Toggle local echo
*** [C-w] : Write hex
*** [C-s] : Send file
*** [C-r] : Receive file
*** [C-v] : Show port settings
*** [C-h] : Show this message

Some of my most-common commands are listed below.

Ctrl + A, Ctrl + B = set baud rate (type it in after)
Ctrl + A, Ctrl + U = baud rate UP (increase baud rate)
Ctrl + A, Ctrl + D = baud rate DOWN (decrease baud rate)
Ctrl + A, Ctrl + C = toggle local echo on and off
Ctrl + A, Ctrl + W = write hex; ex: type 41 42 43 44 to send the ASCII chars ABCD. Delimiting chars you may choose to use, such as spaces or colons, for example, are ignored. So, 41424344 and 41:42:43:44 both send the same thing. Do not, however, include the 0x part before hex. Ex: to send 0x41 just type 41, not 0x41.
1. See the ASCII table here: https://en.wikipedia.org/wiki/ASCII#Printable_characters.
2. See also man picocom and search for "Write hex".
Ctrl + A, Ctrl + S = send a file over serial to the remote device.
1. By default, this requires the ZMODEM sz (send) and rz (receive) executables to be available on both the host computer and the remote serial device, in order to work. Those executables are usually part of the lrzsz package, and can also be built as part of Buildroot. See more on this below.
2. See also man picocom and search for the section titled "SENDING AND RECEIVING FILES". You can read the latest version of the man pages online here: https://github.com/npat-efault/picocom/blob/master/picocom.1.md.
Ctrl + A, Ctrl + R = receive a file over serial from the remote device.
1. Same requirement as the send command, described just above.
Ctrl + A, Ctrl + V = show port settings, such as baud rate, flow control, parity, databits, stop bits, etc.

Scripting the sending and receiving of commands over serial

See here for starters: How to write automated scripts for picocom. See this link and more info. in my "References" section below too.

WIP:

ATTEMPT 2: WORKS! (but ONLY if picocom is open in another terminal 1st) (update: it just works now since I added the sleep 0.1!) This is a pretty complicated synchronization problem. You need to: start listening, wait a moment, send the cmd, wait a moment, stop listening, block on sending to the pipe, have the first thread read from the pipe after the 2nd thread blocks on writing to it. Done.

#!/usr/bin/env bash

# some test commands to automatically send over serial (and receive the replies)
cmds_array=()
cmds_array+=("ls /sys")
cmds_array+=("uname")
cmds_array+=("pwd")

response_array=()

# First, configure, open, and leave open the serial port
picocom --noreset --exit --quiet --baud 115200 /dev/ttyUSB0

if [ ! -p "/tmp/serial_pipe" ]; then
    mkfifo /tmp/serial_pipe
fi

read_serial_response_and_pipe_it_back() {
    # see my new answer here: https://unix.stackexchange.com/a/720853/114401
    response_str="$(timeout 0.5 cat /dev/ttyUSB0)"
    printf "%s" "$response_str"
    printf "%s" "$response_str" > /tmp/serial_pipe
}

# send all commands and receive all their responses
for i in "${!cmds_array[@]}"; do
    read_serial_response_and_pipe_it_back &
    sleep 0.1  # ensure the cmd above has time to start reading before we continue

    cmd="${cmds_array["$i"]}"
    printf "%s\n" "$cmd" > /dev/ttyUSB0
    # printf "%s\n" "ls /" > /dev/ttyUSB0
    sleep 0.5

    response_array+=("$(cat /tmp/serial_pipe)")
done

# print all responses
for response_str in "${response_array[@]}"; do
    printf "==start==\n%s\n==end==\n\n" "$response_str"
done

ATTEMPT 1:

#!/usr/bin/env bash

# some test commands to automatically send over serial (and receive the replies)
cmds_array=()
cmds_array+=("ls /sys")
cmds_array+=("uname")
cmds_array+=("pwd")

response_array=()

# First, configure, open, and leave open the serial port
picocom --noreset --exit --quiet --baud 115200 /dev/ttyUSB0
# make a pipe we will need below for some basic inter-process communication
# (IPC)
if [ ! -p "/tmp/serial_pipe" ]; then
    mkfifo /tmp/serial_pipe
fi

# send all commands and receive all their responses
for i in "${!cmds_array[@]}"; do
    # 1. spawn a background process to start listening for the serial response, 
    # passing the response to a pipe to be read by this process
    cat /dev/ttyUSB0 > /tmp/serial_pipe &
    spawned_pid="$!"

    # 2. send the cmd; the response will get received by the spawned process
    # above and then piped back to this process
    cmd="${cmds_array["$i"]}"
    printf "%s\n" "$cmd" > /dev/ttyUSB0

    # 3. delay a bit to let the spawned process receive the command, then read
    # from the pipe and kill the spawned process above
    sleep 0.5
    response_array+="$(cat "/tmp/serial_pipe")"
    kill "$spawned_pid"
done

# print all responses
for response_str in "${response_array[@]}"; do
    printf "=====\n%s\n=====\n" "$response_str"
done

Transferring files over serial

Background

Transfering files over serial can be done via ZMODEM. See: Unix & Linux: How to get file to a host when all you have is a serial console?. The preferred technique would be to use the ZMODEM error-checking protocol (see here: Wikipedia: ZMODEM) via lrzsz, which provides the rx, rb, and rz modem file receive functions (and is available on Buildroot too), and the sx, sb, and sz modem file receive functions.

Or, you could use the kermit protocol in case XMODEM (rx and sx), YMODEM (rb and sb), and ZMODEM (rz and sz) are not available.
However, if neither lrzsz nor kermit is available, then you can encode binary files into text using uuencode (I can't get this to work at all for me) or base64 (preferred), and then just manually send them over using picocom and cat or picocom and ascii-xfr, as mostly explained by this answer here.

To install lrzsz and ascii-xfr on Linux Ubuntu, do this:

sudo apt update
# NB: installing `minicom` **alone** includes both `minicom` **and** `lrzsz`!
# (and `minicom` includes the `ascii-xfr` executable too).
sudo apt install minicom 

# Or, to install just `lrzsz`. This gives you access to the XMODEM, YMODEM,
# and ZMODEM protocol executables such as `rz` and `sz`, used to send and
# receive files. 
sudo apt install lrzsz

To learn about the ingenuity of the ZMODEM auto-resend and error-checking protocol, see here: https://en.wikipedia.org/wiki/ZMODEM:

ZMODEM replaced the packet number with the actual location in the file, indicated by a 32-bit number. This allowed it to send NAK messages that re-wound the transfer to the point of failure, regardless of how long the file might be. This same feature was also used to re-start transfers if they failed or were deliberately interrupted. In this case, the receiver would look to see how much data had been previously received and then send a NAK with that location, automatically triggering the sender to start from that point.

Scenario 1: computer to bare-metal or RTOS-based microcontroller

Transferring a file over serial from a Linux or other computer to a custom bare-metal or RTOS-based microcontroller such as Arduino or STM32:

In other words: the target serial device does NOT have access to the Linux tools such as cat or rz or sz which we will use in the other scenarios below.

You'll have to do this all custom. Modelling your approach after how ZMODEM (see quote from this link just above) works would be a great idea. You'll have to write a custom application on both the remote serial device as well as on your local host machine.

Write a Python, C, or C++ program to send serial data from your host computer to the serial device.
Start out by sending a CLI command such as send_file to the remote serial device. Have this make it switch from an "ASCII CLI-interface" mode to a "binary file receive" mode.

The remote device will then send a NAK (negative acknowledgement) to the host machine specifying that it needs the host machine to begin sending the file at byte 0.

The NAK will contain a uint32_t i_file index which specifies the index location in the file which it would like to receive next, and at which point the sender (host machine) should start sending.

Make this a binary packet with header, payload, trailer. In C++, that might look like this:

// NB: the NAK packet is a **fixed size** packet.

// Magic numbers to mark the start and end of a NAK packet.
constexpr uint32_t PACKET_NAK_START = 1234567890;
constexpr uint32_t PACKET_NAK_END = 987654321;

#define FILENAME_MAX_LEN 20

struct __attribute__ ((__packed__)) HeaderNak
{
    /// A unique, random 4-byte number to indicate the start of this
    /// type of packet
    const uint32_t packet_start = PACKET_NAK_START;
    /// A NAK packet counter to help detect missed packets
    uint32_t packet_counter = 0;
    /// A timestamp of the time at which this was sent from the sender
    uint64_t timestamp_ns;
};

struct __attribute__ ((__packed__)) PayloadNak
{
    /// Name of the file requested
    char filename[FILENAME_MAX_LEN];
    /// The file index at which point the device would like the sender
    /// to begin sending the file data again
    uint32_t i_file;
};

struct __attribute__ ((__packed__)) TrailerNak
{
    /// Some sort of packet integrity checksum, such as an XOR, CRC,
    /// MD5, SHA256, etc, over the header and payload portions of the
    /// packet.
    /// - Make this a CMAC or HMAC, which is basically just
    /// a SHA256 hash over the whole packet plus a pre-shared key, 
    /// if you'd like to also ensure packet authenticity to ensure the
    /// packet came from a trusted source.
    uint32_t checksum;
    /// A unique, random 4-byte number to indicate the end of this 
    /// type of packet.
    const uint32_t packet_end = PACKET_NAK_END;
};

struct __attribute__ ((__packed__)) PacketNak
{
    HeaderNak header;
    PayloadNak payload;
    TrailerNak trailer;
};

The NAK packet will be identified and parsed by the magic starting number, known size, magic ending number, and checksum.

The host machine will then begin sending over the file to the remote serial device using a packet structure which might look like this in C++:

// NB: the FILE data packet is a **variable size** packet, with a payload of
// perhaps 1 to 512 bytes or so. This means that the number of bytes
// actually serialized and sent over the wire can vary, but the 
// `sizeof(PacketFile)` is fixed and known at compile-time. 

constexpr uint16_t MAX_NUM_BYTES = 512;
    
// Magic numbers to mark the start and end of a FILE contents packet.
constexpr uint32_t PACKET_FILE_START = 5555567890;
constexpr uint32_t PACKET_FILE_END = 987655555;

struct __attribute__ ((__packed__)) HeaderFile
{
    /// A unique, random 4-byte number to indicate the start of this 
    /// type of packet
    const uint32_t packet_start = PACKET_FILE_START;
    /// A File packet counter to help detect missed packets
    uint32_t packet_counter = 0;
    /// A timestamp of the time at which this was sent from the sender
    uint64_t timestamp_ns;
};

struct __attribute__ ((__packed__)) PayloadFile
{
    /// Name of the file being sent
    char filename[FILENAME_MAX_LEN];
    /// The total number of bytes in this file being sent.
    uint32_t file_size;
    /// The file index at which point the bytes being sent in this packet
    /// begin.
    uint32_t i_file;
    /// The number of bytes (actually used and sent over serial) in the data
    /// array below, for this packet.
    uint16_t num_bytes;
    /// An array of the file bytes being sent over by this packet. 
    /// NB: this array must be sized to hold up to `MAX_NUM_BYTES` for 
    /// processing on each end, even though only **num_bytes** of it 
    /// will actually be serialized and sent over the serial wire!
    uint8_t bytes[MAX_NUM_BYTES];
};

struct __attribute__ ((__packed__)) TrailerFile
{
    /// Some sort of packet integrity checksum, such as an XOR, CRC,
    /// MD5, SHA256, etc, over the header and payload portions of the
    /// packet.
    /// - Make this a CMAC or HMAC, which is basically just
    /// a SHA256 hash over the whole packet plus a pre-shared key, 
    /// if you'd like to also ensure packet authenticity to ensure the
    /// packet came from a trusted source.
    uint32_t checksum;
    /// A unique, random 4-byte number to indicate the end of this 
    /// type of packet.
    const uint32_t packet_end = PACKET_FILE_END;
};

struct __attribute__ ((__packed__)) PacketFile
{
    HeaderFile header;
    PayloadFile payload;
    TrailerFile trailer;
};

You may need to make the sender wait a bit after sending each file packet.
1. Depending on the baud rate and processing speed of the recipient (ie: if the baud rate is too high and/or the processing speed of the recipient too slow), you may need to make the sender delay a small amount after sending each file chunk packet, to give the receiver time to process the new packet, including computing the hash, ensuring the packet is valid, etc.
Since each packet contains the file_size, the i_file index at which point in the file this data begins, and the num_bytes of the file sent in this packet, the remote serial device receiving the file can easily know when the while file has been received.
1. If it ever detects a corrupted or missing packet, it will send a NAK packet to the sender to indicate where it needs the sender to begin sending again, and then it will pick up from there, reconstructing the file with the newly-received data.

That's the gist of it! Go make it happen. :)

Packetizing and error-checking serial data is actually pretty fun, I think. It's one of the more-enjoyable aspects of embedded software, to me. I love using sophisticated techniques like that described above to send complex data over primitive interfaces. I enjoy that challenge.

Scenario 2: Linux to Linux as plain text withOUT ZMODEM

Transferring a file over serial from a Linux computer to a Linux computer where the destination computer DOES have cat but does NOT have access to the sz and rz ZMODEM protocol executables:

If you don't have access to a good binary file transfer program like ZMODEM's sz and rz programs, then you can either:

Write your own, following my layout in Scenario 1 above (hard), OR
Just send the file over as encoded text using common Linux tools (easy).

I'll cover how to do the latter:

How to send a file over serial as encoded text using common Linux tools:

On your local machine, install picocom. Follow my instructions above.

WIP

# prepare file on sending side
split
# encode
base64

# connect to remote board on picocom
picocom --baud 115200 /dev/ttyUSB0
cat > file  # (run this on destination computer)

# (run next 2 cmds on source computer)
pv # the file over to see a progress bar; OR:
cat file > /dev/ttyUSB0  # from sender

# Ctrl + C the receiver
sha256sum # verify it


# decode
# recombine
# done!

# Improvements:
# 1. increase baud rate until you start to see errors
#  For me, on a device capable of 3 Mbaud, the highest I could go without errors was the one just above 115200
# If using an error-checking scheme such as Scenario 1 or Scenario 3, you could probably go much higher baud rate.
# 2. cross-compile lrzsz using Buildroot, send it over, then move to Scenario 3 for bigger file (ex: 80 MB)

Scenario 3: Linux to Linux WITH ZMODEM

Transferring a file over serial from a Linux computer to a Linux computer where the destination computer DOES have cat, AND the sz and rz ZMODEM protocol executables:

WIP

References

I absolutely could not have solved Scenario 2 nor Scenario 3 above without help from these answers here. I had never even heard of ZMODEM, nor did I know cat could be used to receive data over stdin like this, prior to reading them. These answers were invaluable to me.

***** Unix & Linux: How to get file to a host when all you have is a serial console?--by @J. M. Becker
***** Unix & Linux: How to get file to a host when all you have is a serial console?--by @Warren Young
The content in this question itself was also very useful: Unix & Linux: How do I use a serial port on Linux like a pipe or netcat?
Unix & Linux: What is the complementary command to 'rx' for xmodem transfer? - taught me about the existence of XMODEM (the first), YMODEM (an improvement), and ZMODEM (the best!)
Binary transfer script for minicom, as a GitHub gist: https://gist.github.com/cstrahan/5796653 - where I first learned of the pv command to track file transfer speed and progress (percent complete). Ex: pv -B 10K file_to_send.txt > /dev/ttyUSB0
Unix & Linux: How do I use a serial port on Linux like a pipe or netcat? - excellent content in the question itself, including how to create and use a local pipe (an inter-process communication (IPC) mechanism) via mkfifo, which is really cool.
1. I've added an answer here too: https://unix.stackexchange.com/a/720397/114401
SuperUser: What is the fastest and most reliable way to split a 50GB binary file into chunks of 5GB or less, and then reassemble it later? - how to split large files, then recombine them using cat.
How to write automated scripts for picocom - very useful! Use picocom -rX -b 115200 /dev/ttyUSB0 or picocom --noreset --exit --baud 115200 /dev/ttyUSB0 (same thing) to simply configure the serial port, in place of stty, then exit, leaving the serial port open, configured, and connected so you can cat or otherwise manually write to or read from it.
Difference between "cat" and "cat <" - difference between cat somefile and cat < somefile
https://linux.die.net/man/1/rz - man rz
lrzsz: free x/y/zmodem implementation official page: https://www.ohse.de/uwe/software/lrzsz.html
lrzsz package in Buildroot: https://github.com/buildroot/buildroot/tree/master/package/lrzsz

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Table of Contents

See also

Serial terminal programs

`picocom`

Background

Linux Installation [tested and works]

Windows Installation [WIP; not yet functional]

General Usage

Scripting the sending and receiving of commands over serial

Transferring files over serial

Background

Scenario 1: computer to bare-metal or RTOS-based microcontroller

Scenario 2: Linux to Linux as plain text withOUT ZMODEM

Scenario 3: Linux to Linux WITH ZMODEM

References

Uh oh!

FilesExpand file tree

serial_terminals_README.md

Latest commit

History

serial_terminals_README.md

File metadata and controls

Table of Contents

See also

Serial terminal programs

picocom

Background

Linux Installation [tested and works]

Windows Installation [WIP; not yet functional]

General Usage

Scripting the sending and receiving of commands over serial

Transferring files over serial

Background

Scenario 1: computer to bare-metal or RTOS-based microcontroller

Scenario 2: Linux to Linux as plain text withOUT ZMODEM

Scenario 3: Linux to Linux WITH ZMODEM

References

`picocom`