Create Indestructible SBC's with overlayfs--Part II--uv, flask, and systemd

Last time I covered using overlayfs on a Linux Single Board Computer to survive unwelcomed power hits. 

To understand this post you may want to read or skim the last one...This time I wanted to finish the indestructible SBC setup by adding a Python virtual environment using uv, creating a basic webserver using Python flask, and getting the flask service to auto-start at boot using systemd.

If you are a Linux/Python admin you may already know how to do all of this; consider skipping this post and check out these guys instead. 

As usual, I am writing the steps because I forget everything. 

Pi Zero--512MB RAM but RPi's text-only OS still gave me a surprising amount of extra space to mess around in an overlayfs configuration. 

BUT FIRST--A WORD FROM THIS BLOG'S SPONSOR:


                           
 

After getting your SBC set, you will probably want to design a PCB for your SBC then get it fabricated.

For this, please check out PCBWAY. 

For December 2025 PCBWAY is featuring some terrific holiday specials, details here.

For instance: a special where cool PCB solder mask colors like purple and matte black are a super low price: ten 99 x 99 mm double-sided PCB's for $5USB + tax, tariff, shipping. 

Super affordable!

PCBWAY can fabricate PCBs using full color! Details here

In addition to top shelf PCB fabrication they also do fantastic work with assembly3D printinginjection molding, and much more. 

Their staff is extremely helpful and PCBWAY always turns work around quickly. 

As always--you can help this blog by checking out the PCBWAY site. Thanks.

MAKING CHANGES TO THE READ-ONLY SBC

From last post: we set the Raspberry Lite OS to read-only to survive power hits using overlayfs, but today we are writing OS changes. Here's how:

  • I removed the PI's SD card
  • I mounted it on my PC using an SD to USB adapter  
  • I edited /boot/firmware/cmdline.txt, getting rid of overlayroot=tempfs at the end of the statement, 
  • then saved cmdline.txt. 

Then I put the SD back into the RPI and started it up--it was now a normal read-write device.

When I was ready to go back to overlayfs mode, I added overlayroot=tempfs back to the cmdline.txt using vi, saved, and rebooted. 

WHAT IS UV?

I'd been administering Python environments for years; the number of tools needed to set up/maintain python3 projects-- pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and so on, was troublesome. 

Imagine my excitement when I first read about uv, a single tool that kicks the other Python admin tools' butt. 

Read about uv here; a good getting started video is here.  

For Raspberry Pi OS, I opened the Linux terminal  and used the statements below to create a new project called pyflask using uv (Update: if using other linux distros you may need to put sudo in front of these commands or elevate to root):




  
#hint: to make my SSH terminal not look crap, I used UTF-8 

#encoding (I use secureCRT9.6-- 

#properties > appearance > encoding 

#default is "automatic" which wasn't working)

#install uv 

apt update

apt install curl #needed for next command--may already be installed on your SBC

#install uv using less HD space vs using pip install.

curl -LsSf https://astral.sh/uv/install.sh | sh

#SET PATH needed for terminal app to find uv when you try to run it

#put uv into PATH--for RPi I could skip this step and just restart bash; not sure w other distros

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc

#note, for some distros .local above is replaced by .cargo

#restart bash  

bash  

#is uv working?  This should show a list of basic uv commands.

 uv
 
 
 


CREATE A NEW PYTHON PROJECT AND VIRTUAL ENV USING UV

 




#hint: to make my SSH terminal not look crap, I used UTF-8 
#encoding (I use secureCRT9.6-- 
#properties > appearance > encoding 

#default is "automatic" which wasn't working)

#install uv 

apt update

apt install curl #needed for next command--may already be installed #on your SBC

#install uv using less HD space vs using pip install.

#cd to ~ you should do the next command from your home dir.
uv init pyflask #pyflask is the name of the project

#cd to the project folder you just created.
cd ~/pyflask

uv run main.py  
#shows you successful hello world.

#ACTIVATE THE NEW PROJECT
#you still must be in the project dir.
#must use "source" in the command below or term will open
#a new term, make the change, and pop you back to old one....
source .venv/bin/activate 

#INSTALL PYTHON PACKAGES
#in project root folder....
#uv add [whatever]--use this instead of pip, faster!

#let's add flask
uv add flask

#ASIDE--with UV, WHERE DO YOUR FILES GO?
#see what is where in uv--useful cmd!
uv tree

#in project root.  See below. note-- never edit anything inside the #.venv dir and its subdirs.
#/home/pi/my_weather_bot/  <-- YOUR PROJECT ROOT (You work here)
#├── main.py   <-- Your code goes here
#├── utils.py                   <-- Your code goes here
#├── data/                      <-- Your data folders go here
#│   └── weather_log.csv
#└── .venv/                     <-- THE VIRTUAL ENV (Do not open/edit)
 #   ├── bin/
  #  ├── lib/
   # └── pyvenv.cfg


 

 

After all of this I had a working venv for project pyflask:



GETTING FLASK GOING....

Flask is a simple webserver for Python--more here.  

Using flask I created a webserver that generated "hello world" when viewed from my browser--using only a few lines of Python.  First using the terminal I created a new file to hold the webpage:

touch /home/charlie/pyflask/app.py

vi /home/charlie/pyflask/app.py

I then pasted this python code to the new app.py file:



#we used uv to install the flash module in our venv already.

from flask import Flask 

app = Flask(__name__)

@app.route('/')

def hello():

    return 'Hello, World!'

if __name__ == '__main__':

    app.run(host='0.0.0.0', port=5000, debug=True)

 

...Going back to the terminal I started the new app.




#START PYTHON APP using UV
#use this to test your flask setup.

uv run app.py  

#if you just use python3 app.py the path gets wacked and it won't work

#term will show something like what is below--if flask is ready 
#to process browser GETS and PUTS

########################################

 #* Serving Flask app 'app'
 #* Debug mode: on
#WARNING: This is a development server. Do not use it in a production #deployment. Use a production WSGI server instead.

 #* Running on all addresses (0.0.0.0)
 #* Running on http://127.0.0.1:5000
 #* Running on http://192.168.5.200:5000
#Press CTRL+C to quit

 #* Restarting with stat
 #* Debugger is active!
 #* Debugger PIN: 430-777-572
 
 



ACCESSING THE FLASK WEBPAGE FROM A BROWSER

Once I had everything above working it was time to test from a browser:

#if I open browser or curl using PC on

#the same wifi subnet as RPi Zero using port 5000 

#I see hello world.  

#Your IP will be different, but I hope you get the idea.

http://192.168.5.200:5000/

GETTING FLASK TO AUTO-START

Our indestructible SBC's test app (in this case app.py) had to start on power-up every time.  

There were lots of ways I could have done this, but to me the "Debian way" was to use systemd--video series about how systemd works starts here

Let's get right to it--using the Linux terminal:

#we need to create a .service file for our python app.

#in Raspberry pi and maybe other debian likes create this in this dir.

#/etc/systemd/system/

#remember that systemd only works with

#the full path to a file or executable

#the command to create the .service file using vi on a raspberry pi is this.

sudo vi /etc/systemd/system/flaskapp.service

#what I put in this new file....






[Unit]
Description=My Flask App

# Wait for the network to be ready before starting (crucial for Flask)
After=network.target

[Service]

# The user the app should run as 
User=charlie

# The group  
Group=charlie



# The folder where your app.py is located
WorkingDirectory=/home/charlie/pyflask
 

# The command to start the app. 
# format: /path/to/python /path/to/app.py
ExecStart=/home/charlie/pyflask/.venv/bin/python3 /home/charlie/pyflask/app.py

# Automatically restart the app if it crashes
Restart=always

# Send Python output to the system log immediately
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target
 

 
 


FINISHING UP OUR BUILD

#let's load it; first, make the system reread all systemd config files...

sudo systemctl daemon-reload

#enable at boot

sudo systemctl enable flaskapp.service

#start it

sudo systemctl start flaskapp.service

#see its status

sudo systemctl status flaskapp.service

#stop the service (no changes needed to the service file for this)

sudo systemctl stop flaskapp.service

#restart the service

sudo systemctl restart flaskapp.service

OUTTRO


I ran the steps above on a Raspberry Pi Zero W, and it worked great--I was ready to start building Linux- and Python-based projects that power up and down like any other DiWHY build.

Now that I have a framework for a SBC based DiWHY project, what should I build?  

No idea. 

I have read about connecting CODECS and DACS to Raspberry Pi's, I might start there and design my own experimenters' boards; wait, there's more! Tons of support for what we usually do with Microcontrollers and C/C++ but instead, RPi and Python:

OLEDs? Here.
Rotary encoders? here.
ADC?  Here.
GPIO and bit-banging?  Here.
SPI and I2C? Here.

But for now it's time to inhale zero fumes. 

See ya next time!

 

Create Indestructible SBC's with overlaysf -- Cut the Power, Scotty!

A fun discussion at a geeky tech meetup lunch:

You have a Linux single board computer or "SBC" (or any Linux host) as the basis of a cool DiWHY audio project; what is the best way to shut it down?  

Easy!

sudo shutdown -h now #syntax depends on distro

The command makes the OS spin everything down without stressing the file system.

But what if someone disconnects the host's power without issuing said command? 

Any habitual Linux user knows: bad idea. At the next power up the file system will attempt to recover from the abrupt shutdown, and usually can, but every now and then: not as much.


However--there must be a way to do this. There are Linux based routers, set top boxes, Internet TV's and everything else, routinely surviving the aforementioned power loss. 


 

How do they work? 

Huge shoutout to group member MVCL who pointed me to OpenWRT, a Linux based router appliance, and the technology it uses to survive power hits--overlayfs--info here.

In the coming months I'd like to create an SBC based DiWhy appliance that can survive and abrupt power loss. Let's get the basics working.

CHEAT CODE--OVERLAYFS ON A NOVEMBER 2025 RASPBERRY PI:


The folks at RPi made this easy.

Get a Raspberry Pi (I used one of these) and Raspberry Pi OS Lite (I downloaded it here)


Then:

sudo apt-get install overlayroot

 Edit this file:

sudo vi /boot/firmware/cmdline.txt  
# add to the end of the single line in cmdline.txt: overlayroot=tmpfs   

Reboot--

You are good--the OS is read only; contents on the SD card are copied to RAM during boot; end user changes are stored entirely in RAM.

If you pull power your OS configuration is safe; RAM contents are lost; at power up the process starts over.

That's it.

Are we done? 

Nooooo.....

  • What about other distros (like Debian 13)?  Same steps?
  • How does overlayfs work at a deeper level?  
  • How do we change the read-only OS to do things like add programs, change configuration files and perform updates?

TESTING OVERLAYFS WITH VM's

For proofs-of-concept I created 2 VMware Workstation Pro Debian VM's: One to experiment with overlayfs' basics, another to create a "kiosk system" that could survive power hits.

Creating the first VM was pretty easy, here are the commands to enter into the Linux terminal: 




#in Debian, add user your username to sudoers
#this makes sudo commands work like ubuntu.

su - #use the minus to preserve paths for normal user.
usermod -aG sudo charlie 

#(use your ~ username instead of charlie, elmo).

#OverlayFS requires 4 directories to work.
#lower directory: the OS
#Upper directory: where changes are stored
#work directory: the kernel prepares files before moving them to #upper layer
#merged directory: the mountpoint the end user sees

# Create a main project folder
mkdir ~/overlay_lab
cd ~/overlay_lab

# Create the four required directories
mkdir lower upper work merged

#What these directories will be used for.
#overlay: The device name (can be anything, but overlay is standard).
#-o: Specifies we are adding options.

#lower dir: The bottom layer. This is a directory used by overlayfs.
#upper dir: The writable top layer

#This is a directory used by overlayfs
#work dir: The directory for internal operations 

#(must be on the same partition as upperdir).

#merged: The mount point.  This is what the end user sees
#-----------------------------------------------

#put data into lower dir. from overlay_lab dir:
echo "I am a base system file." > lower/base_config.txt
echo "Do not delete me." > lower/important_data.txt

#-----------------------------------------------

#run the overlayfs mount cmd. watch the line wrap!

sudo mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged

#now you can see in merged these 2 files that came from lower.

#what does it all mean?

#changes to files in merged directory are mimicked to upper 

#deletions in merged directory cause a "whiteout" file in upper, 
#"masking" the deleted file

#files added to upper (copying a read only OS for instace are copied to merge dir

#if you edit a file in upper the edits do not appear in merged.

#if you edit fstab incorrectly you'll brick your SBC. This is why we 
#snapshot experimental VM's!

#edit fstab so your new configuration isn't lost at reboot 

#you are going to want to create a VM snapshot before you mess with fstab. 

overlay /home/charlie/overlay_lab/merged overlay noauto,x-systemd.automount,lowerdir=/home/charlie/overlay_lab/lower,upperdir=/home/charlie/overlay_lab/upper,workdir=/home/youruser/overlay_lab/work 0 0





All of this means that you have created a new mount at startup; "merge" is the directory where your active files will reside. Other directories are hands-off--pretty easy.

No Sh*t Sherlock: Docker makes heavy use of overlayfs. Didn't know that! Video here.

AND NOW--A WORD FROM THIS BLOG'S SPONSOR:


               
 

After getting your kiosk going, you might want to design a PCB for your SBC then get a bunch of them fabricated.

For this, you should use PCBWAY. 

This month (December 2025) PCBWAY is featuring some awesome holiday specials, details here.

PCBWAY has a service to fabricate PCB's using any and all visible colors. That's right, full color PCB's! Details here

In addition to top notch PCB fabrication they also do fantastic work with assembly3D printinginjection molding, and so so SOOO much more. 

Their staff is super friendly and PCBWAY always turns work around quickly. 

I'm always impressed! 

As always--you can help this blog by checking out the PCBWAY site. Thanks.

CREATING THE "KIOSK" VM


On a second Debian VM here are the steps I used to configure the kiosk VM:




 
#root and login
su -

#make sure overlayroot is installed. Should be....if not....
apt-get update
apt-get install overlayroot

#add overlay module on next boot. 
#we are adding overlay as a single line to the bottom of the modules #file.

echo "overlay" | tee -a /etc/initramfs-tools/modules

#edit grub; do this carefully 

vi /etc/default/grub

#change the GRUB_CMDLINE_LINUX_DEFAULT to this (quotes are needed)
GRUB_CMDLINE_LINUX_DEFAULT="quiet overlayroot=tmpfs"

#save the file (":wq!)

#rebuild grub
update-grub

update-initramfs -u

#reboot.
sudo shutdown -r now
 



file system (dh -f) should look like this.


The main thing: overlayroot is now mounted as /

For me, I could force-power-off the VM and restart--yep, all seemed OK.  

After several reboots I didn't see tons of inode rebuild errors and whatnot. After many power cuts the virtual machine survived all power hits A-OK.

As I added more apps and code to the VM  (covered in the next section), I could continue to run df -h to see if /overlayfs was filling up.  

So far, so good.

HOW DO I CHANGE THE READ-ONLY FS?

To make minor changes to the read only partition:

overlayroot-chroot

(make changes--apt installs, for instance)

when done...

exit

I could use this for apt installs and updates...."exit" sometimes threw errors, but my changes stuck.

-----------------

Another method: this flipped me back to normal Debian once; when I rebooted the OS it was back to overlayfs mode. 

I read this is best for major OS changes and configuration file redo's.

  1. Reboot the VM.

  2. At the GRUB menu, press e.

  3. Find the line with overlayroot=tmpfs.

  4. Change it to overlayroot=disabled.

  5. Press F10 to boot. (You are now in a standard, persistent Debian environment).

  6. to check: mount | grep overlay # produces no output. overlayfs is not #running.

  7. Run your updates, then reboot to re-enable protection.

Finally, I deployed a trick for advanced Linux nerds.....allowing a menu choice in GRUB to disable overlayfs, visible from the initial grub menu seen at startup--info here....

Editing grub configurations is not for the faint of heart and can potentially brick your SBC's OS configuration; if you aren't into moderate to advanced Linux tweaking you should probably skip this option.

#--------------------------------------------------

Follow the "go to normal once" mode using e key, above.

Once in normal mode create some text we will need later.

grep -A 20 "menuentry '" /boot/grub/grub.cfg >>  grub.txt

Now use vi to open 2 files at once:

vi /boot/grub/grub.txt /etc/grub.d/40_custom

once in vi you see grub.txt....issue this command (no : needed)

20yy

this yanks the top 20 lines of the grub.txt file. 

in vi, issue escape then this command

:bn

this flips you to the 40_custom file, then type

p

this pastes the 20 lines from the bottom of the grub.txt file

...now edit the 40_custom file to match what I have boxed in red:


reboot.

If all went well you get a new option each time you reboot: choosing what is boxed below puts you into "normal" mode. Make OS changes; reboot, and pick "Debian GNU/Linux" to go back to overlayfs.



MORE ABOUT OVERLAYFS ON MY RPI SBC


SBC setup for overlayfs. Most important component: coffee.



As I said at the beginning of this post--getting overlayfs working on a current Raspberry Pi was a piece of cake.  or--should have been.

I hit a few bumps getting it working....because I will forget all of this in about 3 hours, here's more detail about what I did:

I used a RPi 4B purchased in 2018.  

Next I downloaded the latest "Trixie" text-only Raspberry PI OS and RPi imager. 

Getting wireless to work in text-only RPi Trixie was a giant time waster; raspi-config was throwing up errors that made no sense such as:

Could not communicate with wpa_supplicant

 entering the error messages into AI took me down rabbit holes that were equal nonsense.  
 
In the end, I didn't let the RPi imager create a custom configuration. Each time I did the Wi-Fi configuration on boot was broken and couldn't be easily fixed.

I ended up doing a standard install of PI OS and configured Wi-Fi settings on first boot via its wizard.

I had to type in the SSID exactly (doh); but, why did that not work at first?  

After some head scratching: what I called my home Wi-Fi SSID and what the RPi thought it was called were different.

The command in the "Trixie" version of RPI OS to list of available Wi-Fi SSID's was something I'd never seen before:

sudo nmcli dev wifi list  #yeh, I knew that.

There was a difference in case, RADISH vs. Radish. I entered Radish, then raspi-config worked OK; no bonkers error messages. 

With wireless working, I sudo raspi-config again, option 3, interfaces, and enabled SSH.

Cool, what IP to SSH into?

ip -a

This revealed the IP of the RPi. I could SSH into the host.

Next I followed the same steps for overlayfs on Raspberry Pi, stated at the top of this post, only 3 steps were needed.

sudo apt-get install overlayroot

then

sudo vi /boot/firmware/cmdline.txt  
# add to the end of the single line: overlayroot=tmpfs

sudo shutdown -r now  #reboot to overlayfs

To go back to normal RPI (not overlay): I pulled the SD card and edited /boot/firmware/cmdline.txt on my Windows 11 PC (Raspberry OS is formatted fat32, so yes, this worked), getting rid of overlayroot=tempfs then saving cmdline.txt.  

Finally, I made a copy of the single line in cmdline.txt, add the string above to one of them, and comment out the line I didn't want using a #. 

So, editing and saving cmdline.txt on my PC to what is immediately below meant a normal "Read/Write" boot:

sudo vi /boot/firmware/cmdline.txt
#sudo vi /boot/firmware/cmdline.txt overlayroot=tmpfs

While editing cmdline.txt on my PC to this meant an overlayfs "Read Only" boot.

#sudo vi /boot/firmware/cmdline.txt
sudo vi /boot/firmware/cmdline.txt overlayroot=tmpfs

One more tidbit: if I was in overlayfs mode and I tried to do a sudo apt upgrade the terminal would return "Killed".  It didn't tell me I was trying to write to a RO device, instead it just said "killed," but, whatever, and what do I want for free?

WHAT'S NEXT?


I need to get rid of the SBC's mandatory login, then conjure a DIY app for the junker RPi.  (update--this is covered in the next post--go here.)

RPi supports the peripherals we know and love--I2C, SPI, serial UART, all the usual stuff; perhaps what can be done with an Arduino Nano can be done here....I have this old sequencer idea (here) that perhaps I could continue to work on....  

Regardless: overlayfs is an important technology....instead of just entering the cheat code I dug in a bit further. Now I know more about a widely implemented and important technology. 

This knowledge will ultimately enrich my professional skills, my sense of confidence, and my piwece of mind.

There is a lesson here? Nope. Next time: just use the damn cheat code.