The Publick network: the publick.org homepage and every town’s site, run
from this one repository with the Publick engine
at the version in engine-version.
LICENSE What may be reused and how: CC BY 4.0 for what Publick makes, the credit line,
and what keeps its own terms (311, public records); the code is MIT
LICENSE-CC-BY-4.0.txt The CC BY 4.0 legal text
engine-version The engine release every town runs, an exact tag; moved each morning by engine.yml
ADDING-A-TOWN.md The checklist for adding a town, from an empty folder to its first published site
home/ The publick.org homepage (index.html) and the page for a state with many towns
(state.html); their town lists are filled in by scripts/build_home.py
towns/<town>-<state>/ One folder per town, laid out as a town repository is:
config/<town>.toml everything town-specific, over the settings in defaults/
data/ collected data, committed by the daily run, with run.json (its last
daily run); a 311 town's requests and scorecard are kept in the documents
bucket instead, since 2026-10-13 (below), and what its AI summaries cost
in the private store (below), never in git
site/static/ files that replace or add to the engine's (the share image)
defaults/network.toml The settings every town shares ([summaries] model and prices, the documents bucket,
GoatCounter, 311 and digest settings), under each town's own config (below)
defaults/states/<st>.toml The settings every town in a state shares (its name, Census and BLS codes, the
glossary of its civic terms and laws, in each language)
states/ma/ Statewide sources, fetched once for every town: Massachusetts's DLS reports (other
states' figures are saved in the engine, or fetched by each town)
wrangler.toml The Worker that serves every site from the sites bucket
wrangler.scheduler.toml The Worker that starts the daily runs on time, and checks every site loads, hourly
scripts/import-town.sh Copies a town's own repository into towns/ (how the first three towns moved in)
scripts/build_home.py Writes the homepage's town lists, by state, with the network's counts and each town's
meetings this week (from each town's data/run.json) and whether it sends the weekly
email, and a page for each state with 10 or more towns; and the
network's sitemap.xml (an index of the homepage's and
every town's sitemaps) and robots.txt
scripts/build_status.py Writes the network status page, publick.org/status/
scripts/make_share_image.py Draws home/share/publick.png, the homepage's card on Substack and social media
.github/workflows/network.yml The daily runs, and builds on push and pull request
.github/workflows/engine.yml Moves engine-version to each new engine release once the towns pass on it
.github/workflows/evaluate.yml Runs the engine's minutes or translation prompt against the real model before a prompt change
ships, or the minutes as a batch at each effort, to measure the Batches API (by hand)
.github/workflows/worker.yml Deploys the Workers: the scheduler by itself when engine-version moves to an engine whose
scheduler changed, or wrangler.scheduler.toml changes; the sites Worker only by hand (an
engine whose sites Worker changed opens an issue asking for it)
Every site is built into the sites bucket (Cloudflare R2) and served by one
Worker, which picks the site by hostname: gloucester-ma.publick.org is
towns/gloucester-ma, and publick.org is home/. See the engine’s
pipeline/deploy.py and worker/index.js.
publick-scheduler Worker (wrangler.scheduler.toml), with one GitHub
schedule (12:17 UTC) as a backup, since GitHub starts those late or not at
all. Each start takes the towns that are due (their last fetching run,
daily or by hand, finished more than DUE_HOURS, 18, ago), oldest first; a start with nothing
due does nothing, and the next start makes up a missed one.
states/: Massachusetts’s DLS reports, only when a report
isn’t saved or is over a week old.MAX_JOBS, 100 jobs), up to ten jobs at a time, the slow ones spread
over the jobs by how long each took last time. Daily runs, runs by hand
that fetch, and pushes to main give the accessibility checks a sample of
each town’s pages (every hand-written page, and the first and largest of
each kind of record page); pull requests and rebuilds by hand without
fetching check every page.SUMMARY_BUDGET (a month can have its own in the secret SUMMARY_BUDGET_MONTH):
each run gives each of its towns a share of what’s left, new documents first.
What each town spends is kept in the private store, a bucket nothing serves
(the engine’s README, Cost records), not in git; the run’s budget job reads
it, and prints how much of the budget is used and each town’s share as
fractions, never as amounts, since the run logs are public. A manual run’s
catch_up is a percent of the month’s budget.data/311/requests.json and scorecard.json, most of what
the repository grew by, are kept in the documents bucket under
<town>-<state>/data/311/, read from its public address by any build and
saved back by the daily run; git keeps 311/status.json. The daily run of
2026-10-13 saved every 311 town’s there, and .gitignore has kept them
out of git since (their history stays). A new 311 town’s first run saves
its files to the bucket from whatever its pull request committed, such as
a backfill (fetch_311 --backfill), and the bucket’s copy wins after that.main: rebuilds and publishes the towns the push touched (every
town when engine-version or a workflow changed), records that in their run
records, and rebuilds the homepage and status page. Data isn’t fetched.engine.yml, 08:40 UTC): the
scheduler Worker starts the engine’s release at 08:20 and this at 08:40
(GitHub’s own schedules for both are a late backup). After the release, it opens a pull request moving engine-version to
it, waits for that pull request’s run to build and check every town (every
page of the canaries, the fewest towns with every kind of town between them,
and a sample of the rest), and merges it if every town passes; the merge
publishes them. A run that fails has its failed jobs run once more. If a few
towns still fail (at most one in 20, and at least one), it merges anyway and
holds them back: each keeps the site it has, and an issue (label
engine held back) names them. If more fail, or anything but a town, the
pull request stays open, the towns stay where they are, and the failed
workflow emails the owner. So the towns take at most one new engine a day,
and only one that passed on nearly all of them. For an
urgent fix, release the engine now (its Actions → Release → Run
workflow) and run Actions → Engine.DIGEST_TOWNS
(wrangler.scheduler.toml) its newest issue through Buttondown, to that
town’s readers alone. The signup form on a town’s /digest/ page (its
config’s [digest] signup) posts to the sites Worker, which passes the
address on to Buttondown, within the SIGNUP_LIMITER rate limit
(wrangler.toml). A new town joins a full week after its launch, not at
launch (ADDING-A-TOWN.md, step 8): its [digest] table, its place in
DIGEST_TOWNS, then Actions → Worker, which makes its Buttondown tag. A run makes two requests for each town and two more. See the engine README’s Weekly
digest.A daily run doesn’t fail when a town does; its data is still committed. Instead,
one GitHub issue, Towns need attention (label towns behind, assigned to
the account in the ALERT_ASSIGNEE secret), is opened when a town has had no good
update (published, with fresh data) for 30 hours, or a figure source’s checks
keep failing, or a state’s statewide checks have failed three times in a row.
Each daily run updates it and closes it when every town is caught up. An edit
sends no email, so the run also comments with the towns newly behind (from
python -m pipeline.network behind --new-since); a town already listed, or
one that’s caught up, sends nothing. If the daily runs stop altogether, the scheduler opens
The network’s daily runs have stopped (label network stopped) after 30
hours, and closes it when a run finishes. A pull request’s run still fails when
a town does, so a broken site can’t be merged.
Between the daily runs, the scheduler loads publick.org and every town’s
homepage (from the network’s sitemap) through the sites Worker every hour, at
:50. A site that doesn’t load, tried twice a few seconds apart, opens an issue such as
gloucester-ma.publick.org isn’t loading (label site down, assigned to the account in the
ALERT_ASSIGNEE secret, which Actions → Worker gives the scheduler); the issue lists every site down while any is, and
is closed when all load again.
publick.org/status/ is public, for the
towns’ readers: whether each site’s data is up to date, when each data source
(its config’s [freshness] table) last updated, and which data a failed or
missed update affects, in plain words. It leaves out the run’s internals
(commands, errors, timings), which stay in the run’s summary on GitHub Actions
and its email. Each run that fetches a town’s data writes the result to the
town’s data/run.json, committed with the data; after the run’s towns finish,
the homepage is rebuilt from main with the status page
(scripts/build_status.py) and published. A site shows “some data delayed”
when a step of its last run failed or a source is behind, and “not updated”
after 30 hours with no daily run. It never shows what running the network
costs or how much data it keeps.
The step-by-step checklist, from finding what the city uses to the first
published site, is ADDING-A-TOWN.md. It starts with the
engine’s helper, python -m pipeline.new_town <Town> <state> <city website
URL>, which writes the town’s folder with everything left for a person
marked # To fill in:. In short:
Add towns/<town>-<state>/ with its config/<town>.toml (start from the
engine’s tests/fixtures/town/config/gloucester.toml and its README), an
empty data/, and optionally site/static/share/<town>.png. Set
[site] domain to <town>-<state>.publick.org and [town] state_abbr to
the state’s code. Add an [analytics] table with prefix set to the town’s
folder, so its page views count apart on the network’s GoatCounter site, and a
[storage] table with prefix set to the town’s folder too (it defaults to
the config file’s name). Its
contact_email, <town>-<state>@publick.org, needs no rule of its own: the
catch-all rule in Cloudflare Email Routing (Setup, step 7) forwards it.
Merge, then run the workflow for the town by hand to fetch its data (set
catch_up to a few percent of the month’s budget to summarize its first
months’ documents in that run, within the town’s share). No DNS
change is needed: the homepage lists the town under its state on its own, and gives the state its own page once it has 10
towns.
What every town shares is in defaults/network.toml, and what every town in
a state shares in defaults/states/<st>.toml: the footer’s network name and
link, the GoatCounter site, the documents bucket, the summaries’ model and
prices, the 311 and digest settings, the state’s name and codes, and the
glossary. A town’s config leaves them out and takes them from there; a value
it sets itself wins. A table there fills in the town’s table of the same name
only when the town has it (an empty [digest] turns the signup on).
The glossary is [terms]: the civic terms that read the same everywhere in
defaults/network.toml (consent agenda, variance, town meeting, warrant
article), and each state’s laws and words in its file (Massachusetts’s free
cash and Order of Conditions, New Hampshire’s non-public session), the
state’s winning. Each term has a column for each language, its wording there
and its definition; another language is another column. Every town gets them
all: its glossary page, the definitions on its meeting pages, and the words
its translator is given and checked against (the engine’s
pipeline/glossary.py). A town’s own [[glossary]] entries are only what
means something of its own there (Nashua’s ordinance numbers, Gloucester’s
Retirement Board warrant), and win over a term of the same name. The engine’s pipeline/config.py
checks both files (its README’s “Settings shared by a network’s towns”), and
PYTHONPATH=../publick-engine PUBLICK_TOWN_DIR=towns/<town>-<state> python -m pipeline.config show
prints a town’s config with the file each value came from. A change to
defaults/network.toml rebuilds every town, and one to a state’s file that
state’s towns.
The town’s tax, budget, and school figures come from its state, through the
engine’s package for that state (the engine README’s States section): its
[finance] and [schools] tables take that state’s keys. Every New England
state has a package. Start from a town in the same state: Gloucester or Malden
for Massachusetts, Manchester for New Hampshire, Wallingford for Connecticut,
Burlington for Vermont, Lewiston or Bangor for Maine, South Kingstown for
Rhode Island; Lowell for a Massachusetts city whose Agenda Center has every
board and no calendar; Nashua for a New Hampshire city on CivicClerk. Portland
for a city whose areas are council districts ([town] areas), and Methuen,
Waltham, or Nashua for a school board whose district posts its agendas and
minutes on one page of links ([page_meetings]). A town in a state the engine has
no package for leaves those tables and the budget and schools sections out; it
still gets meetings, 311, unemployment, and housing. List each source in its [freshness] table, so the
status page shows it.
Add the officials section and an [officials] table (the engine README’s
config table): the mayor, the council, and the school committee, from official
city and school websites, with checked set to the day you checked them. Use
only city- or school-issued email addresses and the mayor’s office phone, not
personal numbers the city may also publish. Give a member a ward only if
the seat is elected by that ward, and wards if it’s elected by several (a
district of wards, as on Beverly’s School Committee).
Each town’s [officials] table is kept by hand. Check every town’s members
against the city and school websites after each municipal election, and
whenever a seat changes between elections (a resignation, an appointment to
fill a vacancy, new council or committee officers). Update the members and
checked in one pull request.
Each table’s next_election is the town’s next municipal election: November
3, 2026 for Bangor, Lewiston, Portland, and South Kingstown; Town Meeting
Day, March 2, 2027, for Burlington; November 2, 2027 for Beverly, Gloucester,
Lawrence, Lowell, Malden, Manchester, Methuen, Nashua, Salem, Waltham, and
Wallingford. Portland’s winners take office at the council’s inaugural
meeting, the first Monday of December. From the engine release that
reads it (publick-engine#114), a list is due a check on election day, or once
new terms begin if that’s later (the first month a member’s term_ends on or
after the election’s month, such as January 2027 for Lewiston and April 2027
for Burlington; a town with no term_ends, such as Lowell, is due on election
day). A week after, a list whose checked is
earlier shows as behind on the town’s About page and the status page, and
the town goes into Towns need attention. After checking, set
next_election to the election after; until then, the daily run warns.
Actions → Digest test → Run workflow sends each town’s newest issue now,
as it would go on Sunday, to the addresses in DIGEST_TEST_RECIPIENTS and
no subscriber, with [Test] before its subject. Left empty, engine sends
the issue on the town’s site now; given an engine branch, tag, or commit,
the town is built with that engine first, to see an engine change’s email
before it’s released. The newest issue is the last Sunday’s: a new one
appears on its Sunday.
About once a week while towns are added, compare each town’s upcoming
meetings with its city’s own calendar, Agenda Center or portal, and school
district site. With the engine on
PYTHONPATH and PUBLICK_TOWN_DIR set to the town’s folder:
python -m pipeline.listings lists the meetings shown as one and why. A
calendar meeting beside an Agenda Center one of the same board and day
that isn’t put together means the two name the board differently: add the
calendar’s name to [meetings.aliases].[[meetings.corrections]] entry
with the reason, a link to the evidence and the date checked. It can go
up in English: the next run drafts its Spanish, and a translation in
[strings.es] is used instead when there is one. Manchester’s Arts Commission entry for
November 9, 2026 is the first. The engine’s README says what each field
does; a correction comes down by itself when the city changes the listing.New Hampshire’s tax rates and school figures come from statewide files the
engine saves once a year, because the state’s websites refuse automated
requests. When the status page shows “New Hampshire state figures (yearly)” as
behind for a New Hampshire town, download the new files in a browser and run
the engine’s python -m pipeline.states.nh.extract (the engine README, New
Hampshire’s yearly figures) and merge that to the engine. The next morning’s
release and engine move (engine.yml) take it to every New Hampshire town.
With the sites bucket’s keys set (below), from a checkout of the engine:
python -m pipeline.deploy rollback --domain gloucester-ma.publick.org # the build before
python -m pipeline.deploy rollback --domain gloucester-ma.publick.org --build <build>
To go back to an older engine for every town, set engine-version to the
earlier release and merge.
In the Cloudflare account that holds publick.org:
publick-sites, with no public address.publick-sites only.publick.org zone, for deploying the Worker.AAAA records for * and for @, both to
100::. The address is never reached; the Worker answers first. Remove any
GitHub Pages A records for @.publick.org that must not
reach the Worker needs a route with no Worker, such as
files.publick.org/* (the documents bucket).@publick.org address, each town’s contact_email among them, is
forwarded.r2.dev address, no Worker binding), and an R2 API token
for it (the engine’s README, Cost records).In this repository’s Settings → Secrets and variables → Actions:
| Secret | What |
|---|---|
SITES_ENDPOINT |
https://<account id>.r2.cloudflarestorage.com |
SITES_BUCKET |
publick-sites |
SITES_ACCESS_KEY_ID, SITES_SECRET_ACCESS_KEY |
The R2 token from step 2 |
CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID |
The token from step 3, and the account ID |
ANTHROPIC_API_KEY, BLS_API_KEY, STORAGE_ACCESS_KEY_ID, STORAGE_SECRET_ACCESS_KEY |
As for a town repository (engine README, Secrets) |
ENGINE_PR_TOKEN |
For engine.yml, which moves the engine each day: a fine-grained personal access token with resource owner publick-org, this repository only, and Contents and Pull requests read and write. Required: what the workflow’s own token pushes, opens, or merges starts no other workflow, so the pull request wouldn’t be checked and the merge wouldn’t publish. It expires; replace it before it does (until then the engine doesn’t move, and the workflow’s failure emails the owner) |
BUTTONDOWN_SUBSCRIBE_KEY |
For the weekly digest’s signup form, which the sites Worker answers: a Buttondown API key with subscribers read and write, and sending disabled, so it can’t send email. Without it, the form says it didn’t go through |
BUTTONDOWN_SEND_KEY |
For the weekly digest’s Sunday send, from the scheduler Worker: a Buttondown API key with emails read and write, and sending enabled. Without it, nothing is sent |
DIGEST_TEST_RECIPIENTS |
The addresses Actions → Digest test sends a test of the weekly digest to, separated by commas. A secret so they stay out of the public run logs. Without it, the test sends nothing |
PRIVATE_ENDPOINT, PRIVATE_BUCKET |
The cost records’ bucket (step 8): https://<account id>.r2.cloudflarestorage.com, and its name |
PRIVATE_ACCESS_KEY_ID, PRIVATE_SECRET_ACCESS_KEY |
The R2 token from step 8 |
SUMMARY_BUDGET |
The network’s monthly budget for AI summaries and translations, in dollars. A secret, so the run logs never show it. Without it, no town summarizes |
SUMMARY_BUDGET_MONTH |
A month with a budget of its own, as YYYY-MM=dollars (several separated by commas); empty for none |
ALERT_ASSIGNEE |
The GitHub account the alert issues are assigned to (network, engine, and Worker workflows, and the scheduler Worker). A secret, so the run logs don’t name it. Without it, they’re assigned to no one |
SCHEDULER_GITHUB_TOKEN |
For the scheduler Worker: a fine-grained personal access token with resource owner publick-org, for this repository with Actions and Issues read and write, and for publick-engine with Actions read and write (so the scheduler can start the engine’s release). It expires; replace it (then run Actions → Worker) before it does. Until then, runs fall back to GitHub’s own schedule |
The scheduler’s Cron Triggers also need the Cloudflare account to have a
workers.dev subdomain: opening Workers & Pages in the dashboard once
creates it. The Workers don’t use it; both have workers_dev = false.
Then turn off GitHub Pages for this repository (Settings → Pages), and run Actions → Worker.