Watchdog Write Up - (Pt 1)
Part 1: Local Crime Alerts / Python / Docker / Tech Stack
Recently, due to serious criminal activity in my neighborhood I took it upon myself to stay alert to what's going on. So I built an app that monitors the Richmond Police Department's active calls web page and processes the data to alert me to any activity of interest.
The App Itself
It consists of two parts running inside Multiple Docker Containers:
Watchdog (Python):
- Pulls data from RPD's Web Page
- Parses the web page into a list of Python dictionaries
- Iterate over the dictionaries adding various meta data and whether the call contains certain keywords or is within a defined vicinity.
- If a call is within vicinity and contains a keyword or is on my street (regardless of keyword) then it will notify me via SMS and print to a log which I can watch in real time.
- It will cache these results and then insert them into an SQLite DB to prevent duplicate processing of calls.
- Repeat all these steps every 45 seconds.
Some of the Watchdog Logic showing off how caching prevents reprocessing and error handling to catch funky errors from weird location formatting
This is what the watchdog service looks like during an alert event.
Web Server (Python / Svelte)
- Flask web server that selects data from an SQLite DB when hit at a certain endpoint
- A Svelte Web App that has a Read Only view of the DB with Search / Pagination.
A snippet of Svelte Web Component for the Table Header
A simple web app that displays the data in a table and refreshes the data every 45 seconds
Tech Stack
The whole app runs on a Raspberry Pi 3+ and the logic weighs about 587 lines (including tests). It also has an automatic backup and cache file removal that runs daily.
| Watchdog | Web Server |
Docker python:alpine | Docker python:alpine |
| Python3 | Python3 |
| Beautiful Soup (Web Scraping) | Flask (Web Server) |
| Twilio (SMS Messaging) | SQLite (DB Storage) |
| SQLite (DB Storage) | Svelte (Front End Compiler) |
| TailwindCSS (Styling) | |
| Cypress (Testing) |
A multistage Dockerfile that is used for the Web Server
Lessons Learned
- CPU Architectures are different and sometimes you have to change your tech stack to accommodate impediments in your production environment.
- SSH into devices on the local network to deploy changes remotely via Mobile / Laptop. Copying files between them.
- Docker-Compose for Multi-Container services
- Python Error Handling, Logging, and Pythonic ideas.
- Dipping my toes into Python Unit Testing.
- Become more comfortable with Rollup Config and Git
- More Regex!
What Is Next
Me simultaneously rejoicing and being concerned when I'm alerted by Watchdog
Updating the web view:
- Adding maps and learning D3
- Sorting and Search Functions
- Automating the CD/CI Pipeline so I no longer need to SSH into the server to deploy updates.
Sometimes weird formatting errors occur when parsing the location. This will cause the app to exit and I want to work towards adding these into a queue for reprocessing after manual review.