Rewriting the Civic Scraper: From Monolith to Microservices

When a municipal server changes its cert bundle, a monolithic scraper dies quietly in the dark. For months, our civic data pipeline lived inside a single, sprawling Python script fetching zoning laws, public notices, and city council minutes all at once. If the council page threw a timeout, the whole run halted. We lost hours of historical data to unhandled socket errors and memory leaks. Municipal websites run on brittle, outdated infrastructure with terrible uptime. Relying on a single script meant a broken CSS selector on the parks department page could blind us to emergency zoning changes. We needed isolation, predictability, and a way to test components without hammering live government servers. We rewrote the architecture in Go, breaking the monolith into single-purpose microservices running in Docker. Go gives us strong static typing, fast compilation times, and tiny memory footprints when spinning up dozens of concurrent workers. Each service handles one job. One container fe