Create Indestructable SBC's with overlaysf -- Cut the Power, Scotty!

A fun discussion at a geeky tech meetup lunches:

You have a Linux single board computer or "SBC" (or any Linux host) as the basis of a cool DiWHY audio project what is the best way to shut it down?  

Easy!

sudo shutdown now #or something like this--depends on distro

....of course. The command makes the OS spin everything down without stressing the file system.

But what if the linux system is an appliance and someone yanks the power cable without issuing said command? 

Any habitual Linux user knows: bad idea. The file system will attempt to recover from an abrupt shutdown, and usually will, but every now and then: not as much.

But--there must be a way to do this!

There are Linux based routers, set top boxes, Internet TV's and everything else, routinely surviving abrupt power loss. 

 

How do they work? 

Huge shout out to group member MVCL who pointed me to OpenWRT, a Linux based router appliance, and the technology it uses to survive power hits--overlayfs--more info here.

Could I get it to work?  

Yep..

CHEAT CODE--OVERLAYFS ON A NOVEMBER 2025 RASPBERRY PI:

The folks at RPi abstracted all the tough configuration away.

Get a Raspberry Pi (I used one of these) and Raspberry Pi OS Lite (download it here)

Install the OS 

Then:

sudo apt-get install overlayroot

 Edit this file:

sudo vi /boot/firmware/cmdline.txt  

# add to the end of the single line in cmdline.txt: overlayroot=tmpfs   

Reboot--

BAM! you are good--the OS is read only; contents on the SD card are copied to RAM during boot; end user changes are stored entirely in RAM.

If you pull power your OS configuration is safe; RAM contents are lost; at power up the process starts over.

That's it! 

Are we done? 

Nooooo.....

• What about other distros (like Debian 13)?  Same steps?
• How does overlayfs work at a deeper level?  
• How do we change the read-only OS to do things like change configuration files and perform updates?

TESTING OVERLAYFS WITH VM's

For proofs-of-concept I created 2 VMware Workstation Pro Debian VM's: One to experiment with overlayfs basics, another to create a "kiosk system" that could survive power hits.

Creating the first VM was pretty easy, here are the steps:

#in Debian, add user your username to sudoers

#this makes sudo commands work like ubuntu.

su - #use the minus to preserve paths for normal user.

usermod -aG sudo charlie #I am user charlie, you are not.

#use your username instead of charlie, elmo.

#-----------------------------

#OverlayFS requires 4 directories to work.

#lower directory: the OS

#Upper directory: where changes are stored

#work directory: the kernel prepares files before moving them to #upper layer

#merged directory: the mountpoint the end user sees

# Create a main project folder

mkdir ~/overlay_lab

cd ~/overlay_lab

# Create the four required directories

mkdir lower upper work merged

#What these directories will be used for.

#overlay: The device name (can be anything, but overlay is standard).

#-o: Specifies we are adding options.

#lower dir: The bottom layer. This is a directory used by overlayfs.

#upper dir: The writable top layer.

#This is a directory used by overlayfs

#work dir: The directory for internal operations 

#(must be on the same partition as upperdir).

#merged: The mount point.  This is what the end user sees.

#-----------------------------------------------

#put data into lower dir. from overlay_lab dir:

echo "I am a base system file." > lower/base_config.txt

echo "Do not delete me." > lower/important_data.txt

#-----------------------------------------------

#run the overlayfs mount cmd. watch the line wrap!

sudo mount -t overlay overlay -o lowerdir=lower,upperdir=upper,workdir=work merged

#now you can see in merged these 2 files that came from lower.

#what does it all mean?

#changes to files in merged directory are mimicked to upper 

#deletions in merged directory cause a "whiteout" file in upper, #"masking" the deleted file

#files added to upper (copying a read only OS for instace are copied #to merge dir

#if you edit a file in upper the edits do not appear in merged.

#if you edit fstab incorrectly you'll brick your SBC. This is why we #snapshot experimental VM's!

#edit fstab so your new configuration isn't lost at reboot 

#you are going to want to create a VM snapshot

#before you mess with fstab. (watch the wrap for the following cmd)

overlay /home/charlie/overlay_lab/merged overlay noauto,x-systemd.automount,lowerdir=/home/charlie/overlay_lab/lower,upperdir=/home/charlie/overlay_lab/upper,workdir=/home/youruser/overlay_lab/work 0 0

All of this means that you have created a new mount at startup; "merge" is the directory where your active files will reside. Other directories are hands-off--pretty easy.

No Sh*t Sherlock: Docker makes heavy use of overlayfs. Didn't know that! Video here.

AND NOW--A WORD FROM THE BLOG'S SPONSOR

After getting your kiosk setup going, you might want to design a hat for your SBC then get a bunch of them fabricated.

For this, you should use PCBWAY. 

This month PCBWAY is featuring some awesome holiday specials, details here.

PCBWAY has a service to fabricate PCBs and other materials using any and all visible colors. Imagine that--full color PCB's! Details here. 

In addition to top notch PCB fabrication they also do fantastic work with assembly, 3D printing, injection molding, and so so SO much more. 

Their staff is super friendly and PCBWAY always turns work around quickly. 

I am always impressed! 

As always--you can help this blog by checking out the PCBWAY site. Thanks.

CREATING THE "KIOSK" VM

On a second Debian VM here are the steps I used to configure the kiosk VM:

#root and login

su -

#make sure overlayroot is installed. Should be....if not....

apt-get update

apt-get install overlayroot

#add overlay module on next boot. 
#we are adding overlay as a single line to the bottom of the modules #file.

echo "overlay" | tee -a /etc/initramfs-tools/modules

#edit grub; do this carefully 

vi /etc/default/grub

#change the GRUB_CMDLINE_LINUX_DEFAULT to this (quotes are needed)

GRUB_CMDLINE_LINUX_DEFAULT="quiet overlayroot=tmpfs"

#save the file (":wq!)

#rebuild grub

update-grub

update-initramfs -u

#reboot.

#------------------

file system (dh -f) should look like this.

The main thing: overlayroot is now mounted as /

For me, I could force-power-off the VM and restart--yep, all seemed OK.  

After several reboots I didn't see tons of inode rebuild errors and whatnot. The virtual machine survived the power hits A-OK.

So far, so good.

HOW DO I CHANGE THE READ-ONLY FS?

To make minor changes to the read only partition:

overlayroot-chroot

(make changes--apt installs, for instance)

when done...

exit

I could use this for apt installs and updates...."exit" sometimes threw errors, but my changes stuck.

-----------------

Another method: this flipped me back to normal Debian once; when I rebooted the OS it was back to overlayfs mode. 

I read this is best for major OS changes and configuration file redo's.

1. Reboot the VM.

2. At the GRUB menu, press e.

3. Find the line with overlayroot=tmpfs.

4. Change it to overlayroot=disabled.

5. Press F10 to boot. (You are now in a standard, persistent Debian environment).

6. to check: mount | grep overlay # produces no output. overlayfs is not #running.

7. Run your updates, then reboot to re-enable protection.

Finally, I deployed a trick for advanced Linux nerds.....allowing a menu choice in GRUB to disable overlayfs, visible from the initial grub menu seen at startup--info here....

Editing grub configurations is not for the faint of heart and can potentially brick your SBC's OS configuration; if you aren't into moderate to advanced Linux tweaking you should probably skip this option.

#--------------------------------------------------

Follow the "go to normal once" mode using e key, above.

Once in normal mode create some text we will need later.

grep -A 20 "menuentry '" /boot/grub/grub.cfg >>  grub.txt

Now use vi to open 2 files at once:

vi /boot/grub/grub.txt /etc/grub.d/40_custom

once in vi you see grub.txt....issue this command (no : needed)

20yy

this yanks the top 20 lines of the grub.txt file. 

in vi, issue escape then this command

:bn

this flips you to the 40_custom file, then type

p

this pastes the 20 lines from the bottom of the grub.txt file

...now edit the 40_custom file to match what I have boxed in red:

reboot.

If all went well you get a new option each time you reboot: choosing what is boxed below puts you into "normal" mode. Make OS changes; reboot, and pick "Debian GNU/Linux" to go back to overlayfs.

MORE ABOUT OVERLAYFS ON MY RPI SBC

SBC setup for overlayfs. Most important component: the coffee.

As I said at the beginning of this post--getting overlayfs working on a current Raspberry Pi was a piece of cake.

Because I will forget all of this in about 3 hours, here's more detail about what I did:

I used a RPi 4B purchased in 2018.  

Next I downloaded the latest "Trixie" text-only Raspberry PI OS and RPi imager. 

Getting wireless to work in text-only RPi Trixie was a giant time waster; raspi-config was throwing up errors that made no sense such as:

Could not communicate with wpa_supplicant

 entering the error messages into AI took me down rabbit holes that were nonsense.  

 

In the end, I didn't let the RPi imager create a custom configuration. Each time I did the Wi-Fi configuration on boot was broken and couldn't be easily fixed.

I ended up doing a standard install of PI OS and configured Wi-Fi settings on first boot via its wizard.

I had to type in the SSID exactly (doh); but, why did that not work at first?  

After some head scratching: what I called my home Wi-Fi SSID and what the RPi thought it was called were different.

The command in the "Trixie" version of RPI OS to list of available Wi-Fi SSID's was something I'd never seen before:

sudo nmcli dev wifi list  #yeh, I knew that.

There was a difference in case, RADISH vs. Radish. I entered Radish, then raspi-config worked OK; no bonkers error messages. 

With wireless working, I sudo raspi-config again, option 3, interfaces, and enabled SSH.

Cool, what IP to SSH into?

ip -a

This revealed the IP of the RPi. I could SSH into the host.

Next I followed the same steps for overlayfs on Raspberry Pi, stated at the top of this post, only 3 steps were needed.

sudo apt-get install overlayroot

then

sudo vi /boot/firmware/cmdline.txt  

# add to the end of the single line: overlayroot=tmpfs

sudo shutdown -r now  #reboot to overlayfs

To go back to normal RPI (not overlay): I pulled the SD card and edited /boot/firmware/config.txt on my Windows 11 PC (raspberry OS is formatted fat32, so yes, this worked), getting rid of overlayroot=tempfs then saving cmdline.txt.  

I could make a copy of the single line in cmdline.txt, add the string above to one of them, and comment out the line I didn't want using a #.

WHAT'S NEXT?

I need to get rid of the SBC's mandatory login, then conjure a DIY app for the junker RPi.  It has I2C, SPI, serial UART, all the usual stuff; perhaps what can be done with an Arduino Nano can be done here....I have this old sequencer idea (here) that i could continue to work on....  

Regardless: overlayfs is an important technology....instead of just entering the cheat code I dug in a bit further. Now I know more a widely implemented and important technology. 

This knowledge will ultimately enrich my work skills, my sense of professionalism, and my piwece of mind.

There is a lesson here? Nope. Next time: just use the damn cheat code.