I'm working on a project to consolidate audio streaming devices into a single Raspberry Pi-based unit that serves multiple services on the network. There are a few automations I rely on using Home Assistant where I want to ensure the services are up and running at all times, and if for whatever reason they're not, I want things to self-heal or alert me if they don't.

Let's make a Watchdog script!

In an SSH or Command Line session on your Raspberry Pi (or any other Linux-based server, really.)

$ nano watchdog.sh

(I prefer nano, you can use vim or whatever you like)

Paste in the following script:

#!/bin/bash

# These are the services I care about, you can add any you need!
services=("plexamp.service" "raspotify.service" "shairport-sync.service" "mopidy.service")

for service in "${services[@]}"; do
    if systemctl is-active --quiet $service; then
        echo "$service is running."
    else
        echo "$service is not running. Attempting to restart..."
        systemctl restart $service

        # Check if restart was successful
        if systemctl is-active --quiet $service; then
            echo "$service restarted successfully."
        else
            echo "$service could not be restarted. Sending alert..."
            
            # Do things here (I am sending myself a notification using Pushover's REST API)
            
            # Replace <YOUR_PUSHOVER_USER_KEY> and <YOUR_PUSHOVER_API_TOKEN> with your actual values
            curl -s \
                -F "<YOUR_PUSHOVER_API_TOKEN>" \
                -F "user=<YOUR_PUSHOVER_USER_KEY>" \
                -F "message=<Player Location> - $service is not running and couldn't be restarted." \
                https://api.pushover.net/1/messages.json
        fi
    fi
done

Save your new Watchdog script and make it executable.

$ chmod +x watchdog.sh

Test it!

$ sudo ./watchdog.sh

If your desired services are running, you should see the following output:

plexamp.service is running.
raspotify.service is running.
shairport-sync.service is running.
mopidy.service is running.

Let's set it to run every 2 hours using cron

Edit the root user's crontab by:

$ sudo crontab -e

Insert the following into the bottom and save your changes. Make sure to use the correct path of your watchdog script.

0 */2 * * * /home/user/watchdog.sh

That's it! If your services crash, the script will attempt to restart them on the next run. If it can't restart them for any reason, or the services won't come back, the script will send you a notification (or perform any actions you want), allowing you to look into things before it gives you a headache.