V21 · Operations & Administration

1. The Eskom app

Your settlement can only be as current as the tariff and network reference data behind it. The Eskom seat is where you publish those inputs; it is not where you negotiate or settle a contract.

Follow along. Seat: eskom@webill.net (Eskom; password supersecret). Path: sign in → Tariffs. You should see: a tariff table with charge counts, Add New Tariff, and a per-row Permutate action.

Where this is taught. The meanings of tariff, rate, DLF, TLF, and UoS are owned by V2 §8. Their settlement application is owned by V8 §4.

1.1 Know the boundary before you edit

The landing route points at Virtual Contracts. /eskom/virtual-contracts is the VWA register — registered Virtual Wheeling accounts, enrolled meters, the month’s refund obligation, and 45-day aging. It is taught in V17 §9. /eskom/wheeling-contracts is still a placeholder. Create and sign a per-contract agreement in V7, then read its monetary result in V8.

1.2 Build a tariff in the order the screen expects

Add New Tariff creates a named shell. Open that tariff’s Charges screen to add a Consumption Charge or Service Charge. Each form asks for Name, Index, Frequency, Season, TOU, Denomination, and Status; complete the commercial sheet before activating it. Consumption indices are energy based, such as KWH or MWH. Service lines use their own index basis, such as POD or FLAT, and must not be substituted for an energy rate.

Permutate creates missing HIGH/LOW × PEAK/STANDARD/OFF rate shells for eligible charges. It is deliberately disabled once every charge has already been permutated or activated: disabled means there is nothing left to generate, not that the page failed. A charge marked INACTIVE remains reference data but does not price settlement.

Careful. Permutate produces empty rate positions, not approved prices. Enter the published value in each inline row and press Save before marking a charge ACTIVE.

1.3 Worked example — roll the tariff forward safely

Data. full_demo · seed 42 · stage S02 (tariffs and losses) · illustrative 2026/27 rate sheet.

1. On Tariffs, choose Add New Tariff and save Megaflex 2026/27.
2. Open Charges and add the energy line. Set its denomination to cents, its index to KWH, its frequency to monthly, then leave it INACTIVE while you complete the rate grid.
3. Add the service line using POD and FLAT. It remains a separate charge: R1: 16,142 kWh × 120.0c ÷ 100 = R 19,370.40 energy; R2: 1 POD × R 450.00 = R 450.00 service.
4. Return to the tariff table and choose Permutate. Open Rates, enter each approved Season/TOU value, and use the row’s inline Save.
5. Activate only the charge lines whose full grid is approved. Check the site tariff selection afterward; settlement reads published data, never invents a missing rate.

What you see on screen is a saved rate row and an ACTIVE charge; this is the pattern to expect on your own page. The resulting settlement line belongs in V8 §7, not in this editor.

1.4 Maintain losses and UoS separately

Open Wheeling Charges at /eskom/loss-factors. The screen has distinct tables and distinct add actions: Add DLF, Add TLF, Add UoS, and Add Levy. Enter the field set shown by each form—DLF by voltage tier, TLF by zone and distance band, UoS by category and charge definition, non-bypassable levies by electrification-subsidy or regulatory-levy category and a c/kWh rate—rather than copying one factor into another table. The levy table’s Non-bypassable column is fixed Yes; it is not a checkbox.

Follow along. Seat: eskom@webill.net (Eskom). Path: Wheeling Charges → choose Add DLF, Add TLF, Add UoS, or Add Levy. You should see: a form for that specific reference type and its row in the matching table after save.

Do not diagnose a final statement by changing historical reference rows. Trace the period and its status first: settlement moves DRAFT → PRELIMINARY → FINAL, and a FINAL result needs its explicit correction path.

Where this is taught. Statuses and controlled re-settlement are taught in V8 §11. The municipal mirror is §2.

1.5 Your publication check before settlement

Before a new billing period, inspect one active charge from the tariff list through its rate grid and one network row through the loss-factor screen. Confirm the denomination is compatible with the number you entered: cents per kWh and rand per POD are not interchangeable. Record the published source and effective operational decision outside this screen according to your governance process.

If the expected price on a settlement differs, trace the contract’s selected tariff, interval classification, and status before changing a rate. That sequence tells you whether the issue is configuration, a period correction, or a misunderstanding of the ledger.

1.6 The operator billing portal

/operator is a new persona — not /eskom and not /so. Path: landing page → Eskom door → sidebar Billing (a POST, not a bookmark). The issuer chip is Eskom’s legal name. Search the issued register by month, entity, and organization. Month run approves and re-issues. Received remittances are the supply-only BRP hat.

B13. Doors are the interim identity story. Production Eskom users will need accounts; this path is what ships until that decision.

B14. Leaving transition (issuing_from) is an MO/ops act, not a switch on Settings. Until that month, off-takers see the amount in V15 §13 and still receive the bill from the utility. The first issued month is the month on or after issuing_from.

Fuel prices

/eskom/fuel-prices shows effective-dated fuel prices in R/GJ, their basis and source link. Missing fuels remain missing; references older than 120 days are marked stale. To record a new price, close the old series window and create a new sourced row. Overlapping windows in the same fuel/source series are refused. The seeded April 2026 gas price remains secondary-sourced pending the filed NERSA note. The diesel derivation uses a rounded R/GJ price, so its reconstructed R/MWh differs slightly from the original expenditure quotient.

For each quarterly gas update, obtain the filed NERSA/Sasol price notice, verify its effective date and units, close the prior same-source window, and enter the new R/GJ row with the document reference. Do not silently replace the April 2026 secondary source with an assumed official value. For the annual diesel reference, retain the Eskom expenditure and generation figures and the heat-rate basis used to derive R/GJ; document the rounding. Reopen a thermal plant’s dispatch card to check the newly pinned row. The register page is read-only; changes use the reference-data actions.

2. The Munix app

Your sites can appear eligible on a map yet still lack a municipal tariff or charge configuration. Munix is where the municipality publishes its participation state and its local reference data; your session is scoped to that municipality from the moment you sign in.

Follow along. Seat: <normalised municipality name>@webill.net (Munix; password supersecret). Path: sign in → Configuration. You should see: the municipal status chip, boundary/map information, participation facts, and wheeling charge controls when eligible.

Where this is taught. The map, site eligibility, and municipality object are taught in V2 §9. This chapter teaches the municipality’s publishing screens.

2.1 Set the operational status first

Configuration is /munix/configuration; /munix redirects there. Choose the chip that accurately describes current participation: Wheeling, Pilot, Partial, or None. This is a published operational fact, not a draft workflow. Org-facing maps and eligibility banners reflect it when those pages next load; there is no separate publish button.

The Embedded wheeling tariff and Monthly Admin Fee controls appear only while the status is Wheeling. Enter the values that the municipality has approved, then choose Save. Changing a map boundary or seeing a map does not create a fee; the charge configuration is its own state.

2.2 Worked example — turn a participating municipality into usable data

Data. full_demo · seed 42 · stage S02 (tariffs and municipal reference data) · illustrative municipal configuration.

1. Open Configuration and verify the current chip before changing it. Select Wheeling only when the municipality is genuinely available for wheeling.
2. The wheeling-charge area now appears. Enter 15.0c/kWh for Embedded wheeling tariff and R 850.00 for Monthly Admin Fee, then press Save.
3. The illustrative monthly network amount is R1: 16,142 kWh × 15.0c ÷ 100 = R 2,421.30; R2: R 2,421.30 + R 850.00 = R 3,271.30.
4. Re-open Configuration and confirm the saved values. Then open the V1 map in a separate session: the status is what map and eligibility surfaces consume, while the arithmetic itself is applied by settlement.

What you see on screen is the Wheeling chip plus populated, saved charge fields. This is the pattern to expect on your own page; do not expect an issued statement here.

Where this is taught. The monetary line and its ledger destination are taught in V8 §7 and V8 §8.

2.3 Maintain municipal tariffs without confusing ownership

At /munix/tariffs, you follow the same data sequence as Eskom: add a tariff, add charge definitions, Permutate missing seasonal/TOU positions, fill inline rates, save, then activate ready charges. A municipal tariff is available only to sites in that municipality. It does not replace national loss-factor maintenance, which remains the Eskom seat’s job.

Follow along. Seat: municipality email (Munix). Path: Tariffs → choose a tariff → Charges → Rates. You should see: the same charge/rate editing pattern as Eskom, inside the Munix shell and scoped to your municipality.

Use Permutate before you fill the grid. It is disabled when all eligible charges have already been permutated or activated, which is an expected completed state. A saved INACTIVE charge remains unavailable to pricing until you activate it.

2.4 Know what is deliberately absent

/munix/wheeling-contracts and /munix/wheeling-charges are placeholders. They are not alternate contract or charge registers. The former does not list agreements; the latter is not the Eskom DLF/TLF/UoS editor.

Careful. “Wheeling” status and the map colour answer whether the municipality is published as participating. They do not prove that a particular site has a valid tariff, a contract, readings, or a FINAL settlement.

Where this is taught. National tariff and loss-factor reference data is in §1; agreements are in V7; settlement proceeds DRAFT → PRELIMINARY → FINAL in V8 §11.

2.5 Your hand-off check

Before telling an organisation that municipal wheeling is ready, make the separate facts visible. Configuration must show the intended status; a Wheeling municipality must have saved charge values where the tariff is required; and the organisation must still complete its site and agreement work. These facts are related, but none is a substitute for another.

Careful. Do not use a placeholder route as evidence that an agreement or a charge has been registered. The current placeholders are intentionally honest about the feature boundary.

2.4 Municipal billing

Sidebar Billing POSTs to /operator/session with this municipality already in the session. The issuer chip is that municipality’s legal name, never Eskom’s. The rest of the walk — register, month run, received remittances, settings — is V15 §13. B13/B14 apply here as on the Eskom door: doors are interim; transition flip stays with MO/ops.

3. Background-process catalog

Every scheduled job, enqueue-driven worker, AshOban trigger, long-running sim process and operator mix task — what it does, when it runs, what you see, and what to do if it fails. Feature depth lives in the linked volume; this chapter is the operations index.

How to learn the clock. Narrative, step-by-step process and worked examples for the four rhythms: V1 · The Four Rhythms. Orientation preview: V0 §6. This catalog keeps failure modes, queues and retry behaviour.

Counts at the pinned config/config.exs. Oban cron entries and distinct workers: see COVERAGE.md Matrix 3. The complete run list is V0 §6; this chapter is the operations index (failure modes, queues, retries). Reconcile against COVERAGE when cron changes.

Schedules below are SAST as configured. Gate times for market workers still come from MarketRules (V9 §1).

3.1 The 24-hour dial

3.2 The month strip

3.3 Cron cards

FetchRates — reference FX publications

The market_data queue runs at 10:45 and 14:00 SAST on weekdays, with a fallback pass at 17:30 SAST. An existing primary publication makes the live fetch a no-op. A fallback pass uses the source policy only when the primary set for that date is absent. Calendar absence and provider refusals end the job; transient failures have at most three attempts with 30-second and two-minute waits before the remaining attempts.

Mocks remain the configured default. Live fetching requires explicit provider configuration and an effective WB.Fx.SourcePolicy row. Every publication retains publisher, channel, publication time and retrieval time. A Friday response to a Sunday request is refused. mix wb.fx.backfill --from YYYY-MM-DD --to YYYY-MM-DD requests SARB history through Frankfurter in chunks of at most 90 days; it refuses while the mock is configured.

Live valuation is not armed by this release. The desk must resolve PD-F8-1 before go-live: a trading-day SARB rate is published after the SAPP day-ahead gate. The existing conversion path keeps its exact-date rule; later P13M0 policy integration must be verified before using the new policy for live valuations.

Each card: what / when / what you see / if it fails / taught in. The complete 65-line crontab (every configured run) is V0 §6; counts in COVERAGE.md Matrix 3.

SettlementRunner — DAILY · WEEKLY · MONTHLY · QUARTERLY

ChainReconciliationRunner

LevyConservationRunner

RunBackcast

RunNightly

ProposeNominations

SubmitDamBids

FetchSchedule

ComputeImbalance

IdmMonitor

PollSo

ArchiveSoInstructions

RefreshPriceForecast

ReconcileMonth

MonthlyForecastRunner

FetchWeatherCycle · FetchWeatherDrift

CertificateRunner · Cadt.SyncRunner · Cadt.PublishRunner

Billing and insight (1st of month)

DamCadence · IdmCadence · DeliveryCadence · SettlementCadence · TrustInterest

DisputeClock · ApplicationClock · MecBreachScan · RunStochastic · RunFleet

3.4 Enqueue-driven workers and setup helper

These five are Oban workers but do not have an independent cron line. Their “when” is the event that enqueues them. BackfillMeterHistory is included because operators encounter it during S05, but it is a setup helper—not an Oban.Worker.

PipelineRunner

StatementRunner

ForecastRunner

RunAllocation

IngestReadingWorker

BackfillMeterHistory

3.5 AshOban triggers (4)

3.6 Long-running sim surfaces (3)

Setup and test pin two further transports so those runs never reach a network: :sapp_client → WB.Sim.SappSim and :fx_provider → WB.Fx.Provider.Mock. A missing published rate blocks rather than degrades — an operator seeing blocked Regional-desk actions should look at the FX register first. Live fetching requires explicit configuration and source policy; live valuation remains gated on PD-F8-1 and the later conversion integration. Taught in V18.

3.7 Mix tasks (5)

Queues to watch: default, proposals, setup, contracts, settlement, strategy, brp, forecast. FAQ for stuck jobs: §7.

3.9 The Market Operator’s Clear Day panel

Clearing a Trading Day is not a background job. It is four steps on the MO console, in order, with a human between the third and the fourth.

Publishing is separate from solving, and that is the point of the panel. The obligation is to publish by 14:00 on the day before delivery. An operator who has just re-run a day needs to look at it before it becomes everybody’s settlement basis — so Publish is its own button, behind its own confirmation, and a solved run sits in SOLVED until somebody presses it.

Solving again does not edit a published run. It creates a new one, and publishing that one supersedes the old one, which stays readable. That is what makes “why did my statement change” answerable months later.

Three things on the panel are worth reading rather than skimming:

• The marginal unit per period — who set the price. If it is not who you expected, the reason is usually one of the four inflexibility grounds (V9 §4.13).
• The [D-08] badge — the period’s price under the other reading of whether a curtailed load may set the price. It appears only where the two readings differ, and the difference can be four figures.
• The network model notice — whether the constrained run bound on a published System Operator model, on labelled assumptions, or on nothing at all. Under the provisional zone register it is the third, and the constrained schedule is then identical to the unconstrained one. The notice is what stops that being read as “the network was checked and found clear”.

Requires the HiGHS solver on the host (brew install highs, or your platform’s package). Unit commitment with minimum run and down times cannot be solved correctly by any greedy method, so an absent binary is a deployment fault rather than a reason to fall back — the panel says so instead of returning a schedule that quietly ignores commitment.

The public price board at /prices needs no login at all: the market price set is published information, and a register only participants can read is not published.

Hydrology fetch runs Tuesdays at 01:30 on the forecast queue. The generation bridge runs monthly at 04:50 on settlement, after certificate accrual, proposing fuel activity from FINAL generation. Neither job submits a tax declaration or arms a live provider.

Trading-control workers

Forecast evidence workers

The scheduler uses UTC. ScoreSkill runs Monday 04:30 UTC (06:30 SAST), creating one job per weather site and prior Monday week. Re-running identical inputs reuses the evidence version. BuildSoPack runs daily 07:30 UTC (09:30 SAST) for the next seven delivery days; BuildSoRolling runs hourly at :40 for six hours from the next full hour. Both retain category and data refusals as records. These workers build evidence and never push it to a live operator. The pack transport resolves to Mock unless explicitly configured otherwise.

Data fabric workers

FreshnessSweep runs every fifteen minutes; AttemptRetention at 01:10 UTC preserves ninety days and the latest attempt. DeadLetterSweep runs at 00:20 UTC. DeliverWebhook is event-driven on the events queue (concurrency five) with eight attempts per schedule. These jobs do not alter the monthly settlement-chain order.

DispatchPending runs each minute to recover committed outbox rows when an enclosing transaction did not forward its Ash notifications. A rolled-back source transaction leaves no delivery.

Plant telemetry workers

These four workers share telemetry concurrency five. Vendor pull workers are not installed: P11M2 remains trigger-gated. Compression applies to raw telemetry older than 30 days on a capable Timescale build; raw retention is 400 days. The hourly aggregate is retained. Monitor the feed registry and Oban for refusals and retries.

4. Demo & acceptance data

How the seeded world is built. This chapter states the procedures, profiles and safety rails in plain language. Copy-paste commands live on the control-plane tasks (mix wb.setup, mix setup.demo, mix setup.market).

4.1 Safety rails

Before any mutating setup command:

1. Stop Phoenix, Oban peers, IEx, Scenario Lab drivers and the live-meter runtime. Call WB.Setup.stop_live() if a live tick loop is still up.
2. Use a local, disposable database only. The control plane refuses MIX_ENV=prod.
3. Every data profile requires --reset — an explicit acknowledgement that teardown will destroy application data in scope.
4. Prefer an explicit --as-of YYYY-MM-DD and --seed (default 42) so demos reproduce. Budget ~2 GB free and ten+ uninterrupted minutes for a full build.

4.2 Profiles

Pass --include-market on foundation, full_demo or acceptance to append S16 after that profile’s last stage. The dedicated market profile already includes S16. See §4.7.

S18 · Regional desk is part of full_demo: a sponsored SAPP participation profile, a small FPM + DAM book, a cross-border path, and a shadowed month with a classified variance, all against SappSim and the fixture rate sets. The sponsor is labelled illustrative. Setup pins :sapp_client and :fx_provider to in-process mocks — those runs never reach a network. The trader-seat volume is V18.

market_month is the longest deterministic profile there is: it runs the fifty participants through every Trading Day of a month — day-ahead clearing, prices, publication, metering, allocation, balancing, statements — and then bills the month. See §4.8.

4.3 Stage graph S00–S17

Written agreement PDFs in setup

Dual-signed physical PPAs (SIGNED, ACTIVE, TERMINATED in the gallery) get a long-form PDF during setup — the same pipeline as live signing (V5 §6.3).

Download path after setup: Contracts → open a SIGNED/ACTIVE physical → executed-agreement download (/org/agreements/:id). Thread example: PPPA-202607-138209764 (IPP2 Atlantis Wind → OffTaker C) once that gallery ref is present.

4.4 The procedures

A — recreate the database, then set up once. Use this only when you intend to discard migrations’ application data and setup audit history. Do not use mix ecto.reset followed by full_demo: seeds.exs competes with the setup control plane.

mix ecto.drop
mix ecto.create
mix ecto.migrate
mix wb.setup --profile full_demo --reset --seed 42

B — teardown and rebuild while retaining the database shell. This keeps extensions, migrations, and setup_audit, then reconstructs the deterministic world.

mix wb.setup --profile full_demo --reset --seed 42

C — tear down once, then advance deliberately. Copy the manifest ID that S00 prints and resume that exact manifest at each next stage; this lets you inspect the boundary where a pack changes the data.

mix wb.setup --profile full_demo --reset --through S00 --seed 42
mix wb.setup --resume MANIFEST_ID --through S01
mix wb.setup --resume MANIFEST_ID --through S14

D — roll an already-sealed demo forward to today. When a prior full_demo was sealed on an older business clock (for example as_of 2026-07-18) and you want Strategy Overview, Scenario Lab and market artifacts to sit on today’s UTC date without inventing a new seed, use:

mix wb.setup.roll_forward
# equivalent:
mix setup.roll_forward

Options: --as-of YYYY-MM-DD (default: today UTC), --seed N (default: inherit the prior completed run’s seed, else 42), --profile full_demo|acceptance|lab, --dry-run, --force (rebuild even when as_of is already current). This is still destructive — it stops the live-meter runtime and runs the same reset-first rebuild as Procedure B with the resolved clock/seed. Stop Phoenix, Oban peers and Scenario Lab drivers first. The Demo Dashboard’s Update demo to today runs this same procedure in the background (§8.2).

Expected success: manifest status: "COMPLETED", current_stage: "S14", teardown_complete: true, no failure.

4.5 Credentials pointer

Every demo user password is supersecret. Seats include eskom@webill.net, mo@webill.net, so@webill.net, org seats *_@webill.net, municipality = normalised name + @webill.net.

4.6 Recovery playbook

If a setup stops, leave the failed world intact long enough to identify the manifest and failed stage. Read recent setup_audit records, then correlate the manifest with jobs on the setup queue. Resume the same manifest ID after correcting the cause; starting another destructive profile over a half-built world destroys the useful evidence and can leave mixed fixtures.

Careful. --reset is an acknowledgement that the profile tears down data in scope. The guard against MIX_ENV=prod, the live-overlap refusal, explicit seed/as-of values, and the manifest audit are safety rails—not inconveniences to bypass.

Lab profile for Scenario Lab drills: §5. Live profile for S15: §6. Simulated market overlay (S16): §4.7. A traded month (S17): §4.8.

4.7 Simulated SAWEM market

The ordinary demo cast already has a handful of BRPs (S03 parties, S08 BRP-T1-DEMO and the status gallery). The simulated market is an optional overlay of fifty extra participants that sit beside that cast, register with both the Market Operator and the System Operator, and submit a day-ahead series every day.

It is not part of default full_demo. S14 still seals the ordinary world first; S16 appends afterwards so existing counts stay valid.

Who is in the overlay

The off-takers carry industrial-scale load on purpose. A cast averaging a couple of megawatts each against 1 300 MW of generation would clear at the price floor in every hour of every month and settle to almost nothing; at 497 MW it peaks around 620 MW against roughly 840 MW of available supply on a winter evening, so the market has a marginal unit in it and the System Marginal Price moves.

Every organisation name starts with SIM, every BRP code with BRP-SIM-, and every meter is tagged SIM-MARKET. Sign-in emails are sim_ipp_solar_01@webill.net, sim_ot_mine_01@webill.net, sim_trader_01@webill.net, and so on. The password is still supersecret. Overlay generator meters (including coal MTR-NERSA-SIM-GEN-12 / 13 and gas 14 / 15) have no wheeling interval history until you Generate Reads on the site — that button uses the same technology shapes as the rest of the platform (coal baseload, gas morning and evening peak). SAWEM market intervals from S17 are a different table and do not fill that explorer.

Each participant is:

1. an organisation + user (S01-style credentials);
2. a site or trader pool (generator / off-taker / pool);
3. an ACTIVE BRP registration with the Market Operator (GENERATOR / CONSUMER / TRADER, Annexure 1 criterion, MPID assigned);
4. a Trading Unit (and Point of Delivery where the site’s GPS falls inside the provisional network model) with the System Operator via the same onboarding path the wizards use. A trader registers a Trading Unit with no Point of Delivery of its own: §12.3(1) makes it the counterparty at the IPP’s and the off-taker’s meters, and inventing a meter for it would double-count energy that already has two owners;
5. a 168-period §9.3 DAM series for tomorrow, shaped by technology or load (solar is daylight-weighted; coal is baseload; malls peak in business hours).

How to seed it

Dedicated rebuild (foundation + overlay, reset-first) — same shape as mix setup.demo:

mix setup.market
# equivalent:
mix wb.setup --profile market --reset --seed 42

Add the overlay to an existing profile without changing that profile’s own stages:

mix setup --include-market
mix setup.demo --include-market
mix wb.setup --profile full_demo --reset --include-market --seed 42

mix setup --include-market sets WB_SETUP_INCLUDE_MARKET=1 so priv/repo/seeds.exs appends S16 after foundation.

From the Demo Dashboard (/demo, §8): the Simulated SAWEM market card shows whether the overlay is present. Include simulated market seeds S16 in the background without teardown — the ordinary demo world stays. Use this when foundation or full_demo is already sealed and you only want the fifty participants.

WB.Setup.seed_simulated_market/1 is the same overlay write the button calls.

Daily DAM submissions

WB.Sim.Market.DamSubmitter is a GenServer started with the application when :simulated_market_submitter is true (dev only; off in test and prod). On boot it submits tomorrow’s series if the overlay is present and any unit is still missing one, then sleeps until the next 03:00 SAST — the same window as WB.Brp.Workers.SubmitDamBids. A missing overlay is a no-op: foundation-only boots stay quiet until mix setup.market or Include simulated market lands the cast.

Capacity follows a technology (or load) shape so the fifty participants do not all declare a flat MCR. A unit that already has an operative series for the Trading Day is skipped.

The Demo Dashboard’s market card reports participant counts, the next Trading Day, whether that day’s DAM is in, and whether the submitter is alive — and its daily checks can backfill Trading Days the 03:00 window slept through (§8.4).

S16 itself posts no meter readings — it declares tomorrow, it does not measure yesterday. Metering arrives with S17, which submits 20-minute coverage as a simulated Network Operator for every metered BRP-SIM-… Point of Delivery; the design it follows is docs/references/market/spec/no-meter-readings.md. Do not point the wheeling live-meter runtime at these participants — that runtime writes ts_readings for active PPA devices and is a parallel world.

S16 also seeds two months of synthetic market invoices so /org/market-accounts is not empty before anything has traded. Where S17 then trades one of those months for real, it cancels the placeholders and raises its own documents — see §4.8.

Data. market · seed 42 · stage S16 · fifty BRP-SIM-… parties beside the ordinary S03/S08 cast. Daily writer: §3.6.

4.8 A month of market trading

S16 gives you a market that can be looked at: fifty registered participants and tomorrow’s day-ahead series. S17 gives you a market that has run — every Trading Day of a chosen Billing Period taken through the Market Operator’s own chain, and the month closed with statements, invoices, payments and a billing board.

Everything on this page is written by WB.Sawem.Cycles — the same functions the scheduled jobs call, given an explicit clock. There is no second implementation of the market in the seed. A step that would miss in production misses here, on the cadence ledger, with its clause.

What a Trading Day of the seeded month does

Provisional then verified is not busywork. §12.2(2) gives seven days to verify metering and §13.6(1) keeps balancing preliminary until D+7, so a seed that only ran the official pass would leave the desk nothing to compare against.

What month end does

Collection comes before distribution because the Code puts it there (§15.3.4(2)(d),(e)), which is what lets a Shortfall reach the creditors in the same cycle rather than the next one.

Where the Trading Clearing Account has collected less than the month owes — which is what a defaulter does to it — every creditor is paid its share, not the first few in full and the rest nothing. §15.3.4(11) allocates a bad debt across the creditors in proportion to what each was owed, and the cash side follows.

Money in the seeded month

WB.Brp.Registration opens every party’s own bank account with the standard R3 million float when the registration is created. A month’s energy bill for the largest off-taker in the cast is around R43 million, so before the first Trading Day the seed tops each account up to a R250 million working balance and posts R25 million of it as credit cover. Both are labelled as notional funds from the participant’s own business, never as income from the market — the same honesty rule the opening balance itself follows.

Without the top-up every participant is an insufficient-funds refusal, and a month in which nobody could pay teaches nothing about §15.3.4: the ladder is only interesting when the participants who can pay do.

The monthly §15.1(1)(d) Market Operator Charge is R18 500 per party and lands in the ring-fenced MO Charge Account, which §15.3.4(6) forbids from ever funding a shortfall or a cascade.

The days that are not ordinary

A month of thirty-one identical days proves the pipe and teaches nothing, so the seed places each interesting failure deliberately:

Month end adds two more: the largest debtor pays 60 % of its invoice and cures its Credit Call inside the window; the smallest pays nothing and does not, so its Shortfall becomes Unsecured Bad Debt and the §15.3.4(11) cascade allocates it across the creditors. One month, both branches of the ladder.

How to seed it

From the Demo Dashboard (/demo, §8), with the overlay present: the Market trading card offers the last six complete Billing Periods. Pick one and press Populate trading month — or press Trade the last 3 months and let it take the incomplete ones oldest-first. It runs in the background, and the progress bars are read from the rows the seed has written, not from a counter in the page, so they survive a reload and a second browser.

From the command line:

# Add a month to an overlay that is already seeded — no teardown.
mix setup.market_month --month 2026-07

# Rebuild foundation + the fifty participants + the month, reset-first.
mix setup.market_month --month 2026-07 --rebuild

# A three-day smoke run.
mix setup.market_month --month 2026-07 --days 3

--month defaults to the last complete calendar month. WB.Setup.seed_trading_month/1 is the same write the button calls.

Roughly what it writes

For a 31-day month and the fifty-party cast:

Ten to twenty-five minutes on a laptop is normal.

Six meters that are not read, and why

Thirty of the fifty participants own a physical Point of Delivery: fifteen IPP meters and fifteen off-taker meters, each paired with a trader as counterparty so §12.3(1)’s two-BRP rule is satisfied. Six of those thirty sit at locations the provisional network model PROV-2026-07 does not cover, so they have no Network Zone — and §3.3(4)(a) makes jurisdiction a property of the zone.

The seed does not invent a zone for them. It does not submit readings it knows would be refused, and it does not give the affected units an Actual Energy of zero either, because a unit settled at exactly its schedule reads as a clean day rather than as one that was never measured. Those units appear on the Market Operator’s own unplaced-sites board (/mo/zones/unplaced) and in the run’s unplaced_points. The other twenty-four are metered, allocated and balanced.

The five traders with no wheeling book this month (BRP-SIM-TRD-16 … 20) are in the same position for the same reason: no meter, no balancing position — and still a full month of declarations, cleared schedules, settlement lines, statements, documents and accounts.

Running it twice

Idempotent, and resumable by construction. A Trading Day that has already published a clearing run, an official balancing run and an indicative statement for every party is recovered from the record rather than re-traded: the month gets its lines back from the published run and the balancing legs, and nothing is written twice. An interrupted month is finished by running the same command again.

§15.3.4 issues a statement once, so a repair run does not re-issue one. A revision is §15.3.5’s instrument, not a second issue of the same figures.

Where to look afterwards

Data. market_month · seed 42 · stages S00–S03 + S16 + S17 · the month named by --month, default the last complete one. Verified by test/wb/setup_acceptance/trading_month_test.exs (three days, gated on WB_SETUP_ACCEPTANCE=1).

5. Scenario Lab — pointer to V12

The Scenario Lab (/org/sim, development and demo builds only) is taught in V14 · Scenario Lab.

That volume covers purpose, the three seeded recipes (AUG-BASELINE, CODEX-S12-WARM, CODEX-S12-ISOLATION), every editor panel, Runs, Compare, the day log, Teardown, and the worked clone-and-run.

This chapter stays so older #5-scenario-lab links still land, and so the operations catalog can point at a neighbour instead of re-teaching the kitchen.

Gating (unchanged). The route compiles only when Application.compile_env(:wb_wheeling, :sim_enabled) is true (dev / test, off in production). At runtime WB.Sim.sim_enabled?/0 must also pass or you are redirected home.

Revised read

S12 also runs a Revised read demonstration against its participant meters: one FINAL interval and one DRAFT interval receive a substituted revision. The FINAL period keeps its settlement and gains a locked-period finding; the DRAFT period queues a re-cut. Original readings remain available. Repeating the script skips an already revised demonstration interval. If either state has no eligible interval, the S12 result explicitly reports no_eligible_interval for that state.

6. Live simulation mode

S15 turns on a continuous live-meter writer after a sealed S14 world. It is not another deterministic rebuild of history.

6.1 S05 history vs S15 live

Settlement and strategy treat live ticks like any other meter data once ingested — late cumulative reads still preserve energy (V2 §7).

S05 is therefore not a slow version of S15. S05 creates a fixed, inspectable past as part of a profile; S15 writes a moving present after that past has been sealed. Do not attempt to repair live drift by rerunning S05 over it.

6.2 Start live only from a matching seal

1. Complete a sealed S14 with known --as-of and --seed.
2. Stop conflicting processes (WB.Setup.stop_live() if needed).
3. Start live with the matching --as-of and --seed. The live profile is continuous and accepts no --reset.

mix wb.setup --profile live --seed 42 --as-of YYYY-MM-DD

4. The Simulator starts under WB.Sim.LiveMeter.Supervisor (not in the default Application children). Each tick enqueues IngestReadingWorker (§3).

The Demo Dashboard’s Catch up metering now (§8.4) runs this same profile against the sealed run’s own clock and seed — it is a button on the procedure above, not a different path. Update demo to today is the clock rebuild (§4.4), not this live-meter start.

Default wall cadence: 20 minutes per tick (virtual clock advances +20 minutes). Catch-up can run faster when behind.

Follow along. Seat: an organisation user after a sealed S14 world. Path: open a device or interval explorer, then observe after one live tick. You should see: a newly ingested reading rather than a rebuilt history.

6.3 Worked observation — one tick becomes a reading

Data. full_demo · seed 42 · S14 sealed, then S15 live · illustrative 20-minute tick.

1. Start S15 against the matching sealed profile. The GenServer creates the next simulated meter observation at the tick.
2. At one cadence, the virtual clock moves R1: 10:00 + 20 minutes = 10:20 SAST; it enqueues IngestReadingWorker.
3. The worker persists the reading. Open the device/interval explorer and verify the new timestamp rather than expecting a modified S05 record.
4. A daily settlement run can subsequently include eligible fresh intervals. It will not overwrite a FINAL period merely because another reading arrived.

What you see on screen is a later interval row after ingestion. This is the pattern to expect on your own page; the simulator itself is not a settlement approval action.

6.4 What you watch

• New readings appear on device / interval explorers.
• Settlement DAILY runner (§3) can pick up fresh intervals on its next pass.
• Lab Driver runs are separate — do not confuse an accelerated scenario month with S15 wall-clock ticks (§5).

6.5 Stop before any destructive work

Call WB.Setup.stop_live() before teardown, second setup, or leaving the machine. Setup refuses to overlap live ticks with a mutating profile run — that refusal is intentional (§7).

Credentials / safety still apply (§4). Never point live at a shared or production database.

Where this is taught. Meter and interval interpretation is taught in V2 §7. The ingestion worker appears in §3; controlled settlement status is taught in V8 §11.

6.6 A safe live-session checklist

Treat a live session as an observation window. Start from one sealed S14 manifest, keep its as-of date and seed with your notes, and tell other operators that S15 is writing intervals. When you are finished, stop the writer before opening a Scenario Lab drill or rebuilding fixtures.

If an interval does not appear, inspect IngestReadingWorker and the live supervisor before manually adding readings. Manual repair hides the very ingestion behavior that S15 was meant to demonstrate.

Plant telemetry companion

S15 starts LiveTelemetry.Simulator beside the meter simulator. It advances a virtual five-minute clock and submits deterministic Mock device payloads through the production normaliser and plausibility engine. Ten-minute feeds emit only on their own boundaries. It starts only with the guarded live profile; an ordinary application boot does not start it. The mock badge remains visible, and no vendor HTTP call is made.

7. Troubleshooting & FAQ

When a screen looks wrong, identify whether the source is configuration, readings, lifecycle, or a background worker before changing data. This chapter gives the first safe check and points to the section that owns the mechanism. If this summary conflicts with the owning volume, follow the owner.

Settlements and statements

1. Why is my settlement still DRAFT?
   DRAFT is a working result while the pipeline resolves data and can be re-run safely. As data becomes usable it can be PRELIMINARY; FINAL is the controlled close, so the normal runner does not silently clobber it. Check the period’s readings and pipeline status before you retry.
   → V8 §11

2. Why can’t I re-settle a FINAL period from the nightly job?
   FINAL is an operator-protected statement of the period, not an automatic cache. The runner may update DRAFT or PRELIMINARY work but requires the explicit correction/re-settlement path for FINAL. Identify the source correction first; changing a tariff or meter row without that trace is not a safe remedy.
   → V8 §11

3. Why is a statement not issued?
   StatementRunner is enqueued by settlement finalisation, so a non-FINAL period has no normal statement to issue. Confirm the period status and inspect the settlement queue before assuming a money calculation failed. A failed statement job needs its job evidence and retry path; it does not authorize editing ledger rows.
   → V8 §9 · §3.4

4. Why is the result empty after a re-run?
   The empty-run guard prevents a partial-meter tail from wiping a previously useful result. Treat the empty output as a data-completeness signal, then inspect the interval window and allocation inputs. Once the readings land, use the owned re-settlement route rather than fabricating zero values.
   → V8 §11

5. Why do IPP hours say provisional?
   Provisional means the calculation has incomplete or late meter evidence, not that the platform has lost energy. Late cumulative reads preserve the quantity and allow a controlled re-settlement. Compare the interval coverage first, then read the final report after data is complete.
   → V4 §8 · V8 §3

Deals and lifecycle

6. Why did my proposal expire?
   The hourly AshOban :expire_stale trigger moves a PENDING proposal past expires_at to EXPIRED. That is a lifecycle transition, not a deleted negotiation. Renew only when its product and max_renewals allow it; otherwise create the appropriate new deal flow.
   → V7 · §3.5

7. Why is my contract still SIGNED after its start date?
   The daily PhysicalPpa :suspend_unsigned trigger (and the virtual twin) moves AGREED deals past the 10-day sign window to SUSPENDED; revival is a mutual Request / Accept on Negotiations (V7 §5.1). The daily PhysicalPpa activation trigger moves SIGNED to ACTIVE when start_date ≤ today. Confirm the date and the trigger/queue evidence before changing contract data. If the lifecycle is valid but execution failed, follow the deal owner’s recovery route rather than backdating settlement.
   → V7 · §3.5

Meters, jobs, and live data

8. Why is an interval missing or red on the gallery device?
   CODEX-DEMO-GALLERY-MISSING-READ deliberately contains a gap so you can recognize incomplete coverage. It is expected fixture data, not a reason to repair the demo database. The resolution rules preserve energy when a later cumulative read arrives.
   → V2 §7

9. Why is an Oban queue stuck?
   First identify the worker and its owning queue—such as settlement, brp, strategy, forecast, or setup—then inspect its error and retry state. A stuck job usually reflects an unmet input or external boundary, not an invitation to create a duplicate job. Use the worker card to verify what it should have produced before retrying.
   → §3

10. Why did the IPP monthly forecast not fan out?
    MonthlyForecastRunner runs at 04:00 on the fifth and fans out a ForecastRunner per eligible active IPP. Check the active physical PPA and the forecast queue before expecting a report. A missing fan-out is not corrected by manually copying another IPP’s forecast.
    → §3.3 · V4 §8

11. Why are live ticks colliding with setup?
    A mutating profile and S15 would write incompatible assumptions into the same world, so setup refuses the overlap. Stop the simulator with WB.Setup.stop_live(), verify it stopped, then resume or rebuild from §4. Do not force a reset through the guard.
    → §4 · §6

Market and Scenario Lab

12. Why is a bid EXPIRED or showing a gate-miss?
    The bid was built after the D−1 gate determined by MarketRules; 10:00 is a typical gate, not a substitute for configured rules. Check the gate, worker timing, and bid source before attempting another submission. A retry after delivery has moved on is not a valid bid.
    → V9 §4

13. Why is the SIMULATED ribbon visible on MO or SO screens?
    A Scenario Lab run still owns those artifacts, and its status is not TORN_DOWN. Open /org/sim, identify the run by its name and seed, then inspect or tear it down when the exercise is complete. Do not read a ribbon-marked screen as production-like operational evidence.
    → V14 · §5 · V3 §17

14. Why does a Lab comparison show a drifted badge?
    The two runs no longer share comparable provenance or resulting trajectory: confirm scenario clone, seed, period, and edited controls in the day log. A baseline comparison should isolate one intended change, such as a day factor, rather than blend market, SO, and MO injections. Clone a clean baseline and rerun if you cannot explain the divergence.
    → V14 §10

Setup and seats

15. Why did setup fail mid-stage?
    Inspect the failed stage and its setup queue evidence, then resume the same manifest ID after correcting the cause. A second --reset on a half-built world removes the audit trail that explains the failure. Resume that manifest; do not start a second reset.
    → §4

16. Why is my password wrong after a rebuild?
    Demo seats use supersecret; Eskom is eskom@webill.net, and a municipal account is its normalised municipality name plus @webill.net. If the expected seat cannot sign in, correct the S01 pack and rebuild. Do not hand-edit password hashes, because that breaks reproducibility.
    → §4

17. Why can’t I find Scenario Lab on the landing page?
    The Lab requires compile-time and runtime sim_enabled gates. Production builds omit it, and a disabled runtime redirects the route home. Use a development/demo build and the lab/full-demo setup profile rather than attempting to expose the route manually.
    → V14 · §5

Eskom and Munix

18. Why is the municipality map the wrong colour?
    The Munix Configuration wheeling-status chip drives the V1 map legend. Confirm whether the published state should be Wheeling, Pilot, Partial, or None, save it in the municipality seat, then reload the organisation map. A colour is not proof that a tariff or PPA exists.
    → §2 · V2 §9

19. Why does Munix say Wheeling but the map still seems unusable?
    The status publishes municipal participation, while the map presents that fact to organisation users. It does not create an Embedded wheeling tariff, Monthly Admin Fee, valid site configuration, or agreement. Check the saved Munix charge controls and then trace the V1 site/deal path.
    → §2 · V2 §9

20. Why is Permutate disabled or doing nothing?
    Every eligible charge has already been permutated or activated, so there are no missing season/TOU shells to create. Open Charges and Rates to confirm the grid and saved values; the action never invents a price. Activate only after the approved grid is complete.
    → §1

The health probe fails, but the home page opens

Open /health/ready and read checks. A 503 identifies a critical dependency failure; /health/live only proves that the endpoint answers. Restore the named dependency or apply pending migrations. A missing HiGHS binary or stale weather produces degraded instead. An unconfigured /metrics returns 404 by design; configure the bearer token on the web listener and send it in the Authorization header.

8. The demo dashboard — /demo

One room for the whole demo estate. Everything §4 builds — the full demo world, the simulated SAWEM market, traded Billing Periods — is seeded, watched and kept up to date from this page, and the inventory of what was built is listed on it. The three demo tools the landing page used to host live here now.

8.1 Opening it

From the landing page (/), the Demo tools strip shows the full demo’s seal state in one line; Open Demo Dashboard opens the room in its own window, so it can sit beside the console you are demonstrating. The page exists only where demo tools are enabled (:sim_enabled — dev and demo builds); on any other build the route is compiled out and a stray visit lands back on /. No seat is required — like the Scenario Lab (§5), it is a harness, not an organisation login.

Follow along. Any dev/demo build. Path: / → Open Demo Dashboard. You should see: the world’s identity in the header (as-of date, seed, today), eight stat cards, four operation cards, the last three Billing Periods, the daily checks, and the tables of what was built.

8.2 The operations

Four cards. Every mutating button states in its confirmation exactly what it will do; every run happens in the background and the page stays usable.

8.3 Live progress

Progress is read from the rows the seeds write — the setup manifest, the published clearing runs, the statements — so it survives a reload, a second browser and a restarted server. Setup transitions additionally ring a PubSub doorbell (WB.Setup.PubSub) so every open dashboard refetches the moment a stage completes or a Trading Day lands, with a slow poll as the fallback while anything is running. If the page and the rows ever seem to disagree, believe the rows and reload.

8.4 Daily start-up checks

The estate has two daily obligations: the 20-minute meters must reach now, and every simulated participant must hold tomorrow’s day-ahead series. The Daily start-up checks panel owns both.

• Run at boot — default on. When on, a boot-time runner (dev/demo builds, :demo_startup_checks) performs both checks shortly after the application starts. The switch is stored in setup_audit.demo_settings, the schema that survives teardown — a rebuild does not reset it. When off, the two buttons below are the only way the checks run.
• 20-minute metering — how many metered PPA sides exist, the latest stored reading, and how many intervals behind now that is. Catch up metering now starts the live simulator through the S15 profile against the sealed world’s own clock and seed (§6); the simulator then back-fills every missing interval and settles into the live cadence. A running simulator needs no button — catching up is what it does.
• DAM submissions — the next Trading Day, whether every unit has submitted for it, and any recent Trading Days (three-month horizon — the same window the trading-month tools operate over) where a series is missing — the hole the daily 03:00 submitter (§4.7) never refills on its own. Backfill DAM now submits the missing days, oldest first; units already holding an operative series are skipped.

8.5 What was built

The stat cards and tables are the §4.2 world, counted and named: organizations with their BRP registrations and holdings (simulated participants badged sim), generator sites by technology and connection, off-taker sites by category, trader pools, and every physical and virtual PPA as a contract leg (who sells to whom — site, pool, organisation) beside its network side (Eskom, or the municipality each end sits in). Document counts split wheeling billing (tax invoices, credit notes) from market documents (invoices, self-billing, debit notes), and carbon certificates are summed from issued batches.

8.6 Safety rails

The §4.1 rails all hold here: dev/demo builds only, production refused below the page by the setup control plane, destructive actions confirmed in plain words, one setup at a time (a second Set up everything is refused while one runs), and the page itself writes nothing — every mutation goes through the same WB.Setup functions the mix tasks call.

8.7 Market phase before a demo

The phase shown on this dashboard is the register value, not a build flag. Changing it is a publish on Market Operator → Rules (/mo/rules) — open, review, confirm — the same ceremony as moving the price cap. It is not a deployment.

Before a sales demo, open /demo and / and /mo/rules and check that the three labels agree with each other and with what you are about to say in the room. A rebuilt demo estate seeds Phase 1 explicitly (WB.Setup.Packs.Market.Registrations, WB.Setup.Packs.Market.Simulated). Walkthrough: V9 §18.

Where next. Procedures and recovery: §4. The overlay’s cast: §4.7. What a traded month writes: §4.8. Watching live ticks: §6. What-if experiments on the sealed world: the Scenario Lab, V14.

Trading-control status

The daily status report includes armed registrations, PROPOSE drafts awaiting submission, individual check refusals in the last 24 hours, and approvals approaching expiry. Counts come from persisted rows. A refusal count counts check results, so one bid refused by two checks contributes two. Review the Controls tab for the affected bid and the Approvals inbox for decisions.

9. Registry connector operations

The carbon registry seam (WB.Carbon.Registry.Connector) is Mock in every committed environment. Live Verra and Gold Standard adapters exist; going live is a configuration change, not a code change.

Credentials

Environment variable names only — never values in config/*.exs:

Keys belong in KMS (spec §17), not in the repo and not in a shell history that gets pasted into a ticket.

Fixture capture

1. Point the adapter at a sandbox (or a live account you control).
2. Run each of the five callbacks once.
3. Sanitise: redact account identifiers and personal data; keep field names and shape.
4. Commit under test/support/fixtures/registry_connectors/<adapter>/ with the capture date and API version in that folder’s README.
5. CI replays the files through the real Req decode path with :carbon_registry_forbid_network set.

When a registry changes its API, re-capture. Production already stores every live response as an artifact, so drift shows up in data, not in guesswork.

Go-live checklist

1. Conformance suite green (verra_conformance_test.exs, gold_standard_conformance_test.exs).
2. Config map selects the live adapter for that registry only.
3. Credentials present at runtime; Mock remains the default for every other registry.

COAS manual procedure

COAS has no public API in v1. Operators complete each stage by uploading the document (hashed through Artifact) and capturing the reference:

A stage cannot complete without its artifact. A stage cannot skip its predecessor. After Cancelled, international-side actions refuse.

CAD Trust observer (/operator/cadt)

The CADT node is infrastructure beside the app, never an Elixir dependency. Hosting is invisible to code. CI and the demo estate run the Mock exclusively (:cadt_forbid_network in test).

/operator/cadt is the observer console: per-registry sync health (roots, last synced, :STALE with the recorded error) and the suspect ledger (:OPEN first, resolution trail on the resolved). Observation reads; this screen never holds keys and never writes to the node.

A genuine double-count hit blocks listing and retirement until an operator resolves it with an evidence artifact. The Certificate Desk shows the same incident as a blocking banner.

Demo step S22 seeds one colliding serial block against the Mock’s planted fixture, runs the sweep, and resolves the suspect so a fresh mix setup.demo always shows a found-and-resolved trail.

CAD Trust node — deployment, mirror, keys

The Connectivity Deck’s sizing for a CADT node plus Chia: quad-core, 4–8 GB RAM, ~500 GB disk, Ubuntu preferred. Optionally run CADT’s mirror database (MySQL/Postgres) as the analytics-friendly read path. The app talks to either through WB.Cadt.Client; swapping Mock for NodeApi (or a later mirror adapter) is configuration.

Testnet is the default. No committed config names mainnet. Publishing is the same software with a Chia wallet key (24-word mnemonic in real key management — it signs data-layer writes inside the node) plus a CADT API key that merely guards the node’s API. The API key reaches the adapter from CADT_API_KEY / CADT_BASE_URL at runtime. The adapter refuses publishing with :missing_api_key rather than defaulting.

The key ceremony itself is an ops runbook outside this repository. Record who performed it, when, and which KMS slot holds the material. Never paste a mnemonic, API key, or example secret into tickets, fixtures, or this manual.

10. Platform health and metrics

The Demo dashboard’s daily checks include Platform health, with a link to the complete readiness report. /health and /health/ready work on both listeners without a market token. ok and degraded return 200; a critical failure returns 503. /health/live always returns 200 when the endpoint can answer, so it is the process liveness probe.

The eight checks cover the database, pending migrations, TimescaleDB, Oban, the Oban leader, HiGHS, weather freshness and cluster peers. Each has a two-second deadline. An optional fault (such as a missing solver or stale weather) is degraded; stopped Oban, a failed database, pending migrations or missing TimescaleDB fail readiness. Mock feeds identify themselves as mock. These checks make no external provider request.

PromEx records web/API request latency, LiveView activity, database queries, jobs, VM usage, publication and gate-miss counts, API authentication/idempotency/rate-limit outcomes, budgets, feed/FX ages, leadership and readiness. Clearing metric names are registered for the later background-clearing milestone; they do not yet have emitters.

/metrics is available only on the web listener. It returns 404 until operations supplies METRICS_BEARER_TOKEN, then requires a matching bearer token (401 otherwise). Restrict the load balancer path to the Prometheus host and scrape each instance every 15 seconds. Grafana dashboard upload needs its configured host and credential; the application does not upload dashboards at boot. Never put tokens into links or the manual.

Honeybadger uses its environment key and the wb_wheeling app name. Moving a key out of a file does not rotate it: the operator’s dated rotation and revoked-key check live in docs/references/ops/DEPLOY-EC2.md. Stage 1’s rotation is still pending.

11. Feeds and delivered events

The operator’s problem

Webill does not invent every number it shows. Meters arrive from files and devices. Weather arrives from a forecast vendor. Carbon registries answer when you ask about a certificate. The market gateways send bids out and bring results back. A buyer may ask you to knock on their own server when an invoice is issued.

If any of those pathways goes quiet, the screens still look finished — they just show yesterday. The operator’s problem is therefore simple: name every pathway, know how often it should speak, and see when it last told the truth.

A named pipe, then the platform word

Think of the platform as a house with labelled pipes. One pipe carries 20-minute meter buckets. Another carries hourly weather. A third carries an outbound “invoice issued” knock to a URL you typed. You do not pour hourly rain into the meter pipe, and you do not judge a weekly registry check by a 20-minute clock.

That labelled pipe is a feed. The label on the pipe is the family. How often a reading is supposed to arrive is the grain (for example 20 minutes). How the pipe is allowed to run — live, pretend, or only when a person uploads — is the mode. The house rule for “this is still current” is the policy. Each try to use the pipe is an attempt. What you see on the register — fresh, stale, never, refusing — is the status, derived from those facts. It is not a mood.

A 20-minute meter feed does not redefine an hourly weather forecast. Grain belongs to the feed.

11.0 Words you need before the buttons

Abbreviations this chapter uses

Where this is taught. Night-job times live in §3. Plant-owner screens live in V4 §11 and V2 §11. Signed Market API subscribe/retry endpoints live in V13 §13.

How a reading becomes a status

flowchart TB
  A[Something arrives or is sent on a named feed] --> B[Record one attempt: OK, REFUSED, TRANSIENT, or OUTAGE]
  B --> C[Update last attempt and last success]
  C --> D[Apply today's policy]
  D --> E[Derive NEVER, FRESH, STALE, or REFUSING]
  E --> F[Fifteen-minute sweep shows the new badge and files a finding if it just went stale or refusing]

1. A worker, an upload, or a plant push uses a feed key (for example weather:mock or plant:{site}:{provider}:{start-date}).
2. The platform writes an attempt with an outcome.
3. If the outcome is OK and the clock is newer than the last success, last success moves forward.
4. Status is then derived — it is not stored as a separate opinion.
5. Every fifteen minutes FreshnessSweep re-seeds the catalog, recomputes every badge, broadcasts the change, and files a quality finding when a live feed has just become STALE or REFUSING.

Attempt history is kept for ninety days, and each feed always keeps its latest attempt even after that window. That job is AttemptRetention (01:10 UTC) — see §3.

Mode is not status

MOCK and MANUAL are modes. They never pretend to be live freshness.

A live feed without a policy is already STALE, reason no_policy. A missing rule is itself a stale condition. — on grain or last success means unknown / never succeeded, not a third clock.

Default policies (source note GD6 interim / F12 platform defaults, effective from 1 January 2020):

• Meter families — 2,400 seconds (40 minutes: two 20-minute grains).
• Everyone else — 21,600 seconds (six hours).
• Weather quotas — 10,000 calls per 86,400 seconds, except weather:solcast at 500 per day.

Policies are effective-dated. A new row with a later effective_from replaces the old rule without rewriting history.

Each feed family

The catalog has nine families. Eight have named pathways on the operations register today. PRICE is reserved: the enum exists; no seeded row uses it yet.

METER. Interval energy used to settle contracts. Imports are MANUAL (you drop a file). The live simulator and device ingest are scheduled. Setup backfill is MOCK. These rows are the settlement truth. A plant telemetry sample never replaces them.

TELEMETRY. Device evidence: power, charge, wind speed, hydrology, carbon MRV. Three shapes exist:

1. Platform rows — telemetry:hydrology (SAPP) and telemetry:carbon_mrv.
2. Per-plant rows — created when an owner registers a feed (plant:{site-id}:{provider}:{start-date}). See §11.2.
3. Not this family — carbon inverter project telemetry on a methodology pack, and 20-minute settlement meters.

WEATHER. Hourly forecasts. Demo rows resolve to the Mock provider until weather is armed. Open-Meteo, the Open-Meteo ensemble, and Solcast are named so a later go-live is a configuration change, not a new pipe.

PRICE. No key, no row, no policy. Do not invent a ticker here.

FX. Daily foreign-exchange rates. Demo uses fx:mock.

REGISTRY. Verra, Gold Standard, the mock connector, CAD Trust, and COAS. COAS is MANUAL — a person tracks it. Live Verra / Gold Standard adapters exist; going live is taught in §9.

BILL. bill:manual_upload — you bring a bill. Nothing is stale because a person did not upload today.

MARKET. How the organisation talks to the Market Operator, the System Operator, and SAPP, plus two outbound duties: the hourly Grid Code forecast pack and a curtailment claim. Gateways are 60-minute grain with a 15-minute cadence (900 seconds).

EVENT. One outbound row, events:webhook_out. Every signed delivery records an attempt on this feed. The human workflow is §11.1.

The named pathways on the register

These keys are seeded by WB.Feeds.Registrations. The Pathway column shows the provider (today that string is the key) and the resolved module.

Timezone: most rows require an explicit UTC offset on timestamps (utc_offset_required). Import adapters may declare their own rule.

Data. Catalog from WB.Feeds.Registrations.rows/0 as of manual 3.20.2. Demo modules resolve to Mock until a provider is armed. Last checked 2026-09-07.

How to use the Feed registry (/ops/feeds)

This is the operations estate, not an organisation login. It exists only where demo tools are compiled (:sim_enabled — dev and demo builds). On any other build the route is not there; a stray visit is sent back to /. No seat is required. There is no menu chip on the Demo dashboard today — you type the address.

Follow along. Any dev/demo build. Path: open /ops/feeds. You should see: heading Feed registry, a Family or status filter, a table (#feed-rows), and a Delivered events → link to /ops/events.

Filter (#feed-filter, label Family or status): All, each family, or MOCK / MANUAL / STALE / FRESH / NEVER / REFUSING. The filter is exact. WEATHER does not show plant telemetry.

The table live-updates when the sweep broadcasts a status change. You do not create feeds on this page. You read them. Creating a plant feed happens on the owner’s plant screen. Creating a webhook happens on Configuration or the Market API page.

Done looks like: every LIVE row you care about is FRESH, MOCK rows still say MOCK, MANUAL rows are idle unless someone should have uploaded, and no LIVE row says No policy.

Empty / refusal. An empty table after a filter means nothing matches — the pipes are still there. Production has no /ops/feeds at all.

How to use Data feeds on Configuration

The Feeds & webhooks tab of the Organization room (/org/configuration/feeds, V2 §2.6) stacks two cards in #organization-data-fabric:

• Data feeds — every feed you own, plus the shared METER and WEATHER rows (those have no organisation owner; every org may see them).
• Webhooks — §11.1.

Each feed row gives a plain name with its key under it (for example Meter devices, meter:device_ingest), the kind of data, a status chip and the interval (20 min, or — when the feed has none). The chip translates the status:

The same strip is attached to the Market API page (/org/brp/api) when a participant registration is selected. There the webhook scope is the party, not the organisation. Pick a registration first or Create webhook refuses with No participant registration selected.

Follow along. Seat: offtaker_c@webill.net (Organization; password supersecret). Path: sign in → Configuration → scroll to Your feeds. You should see: shared weather/meter lines and any org-owned plant feeds, each with mode and status.

11.1 Webhooks and dead letters

A webhook is a promise: “when this kind of thing happens, knock on this HTTPS address and sign the knock so I know it is you.” The signed knock is a delivery. If the far door never opens, the letter is not thrown away — it becomes a dead letter and waits for a person.

That is the outbound EVENT feed events:webhook_out.

What they are for

Think of a doorbell on your office, not a letterbox on ours. You do not come here to fetch every invoice. You hang a bell on your finance system (or your ERP, or a chat room that can take HTTPS). When Webill issues an invoice, we press that bell. Your system opens a payable without anyone staring at this screen.

A webhook is therefore for telling another system you already run. It is not a feed into Webill. We do not invent the address — you paste an HTTPS URL you control. The Webhooks card on Configuration says so under its title.

The kind of news you asked for is the topic (for example invoice.issued). One webhook is one URL plus the topics you tick. A URL holds one webhook; to change what it hears, edit its topics rather than adding a second row.

A worked example — invoice.issued

OffTaker C buys wheeled energy. When the seller’s bill is issued, C’s finance clerk should not have to open Configuration and hunt. Their accounts-payable system should already have a line: document number, amount, due date.

1. On Organization → Feeds & webhooks, under Webhooks, C pastes https://finance.offtaker-c.example/webill/hooks and leaves the default topic invoice.issued ticked.
2. They press Create webhook, press Copy beside the whsec_ secret, paste it into the finance app (so it can check the signature), then press I’ve stored it.
3. Later, a billing document for C is issued. Webill POSTs a small signed JSON body { type, timestamp, data } with document_id and number.
4. Finance answers 200. Recent deliveries shows invoice.issued with a Delivered chip and 1 attempt. The EVENT feed records an OK attempt.
5. If finance is down, we retry (see below). After a dead letter, C presses Retry once the server is back.

flowchart LR
  A[Seller issues C's invoice] --> B[Webhook invoice.issued]
  B --> C[POST to C's HTTPS finance URL]
  C --> D[Finance opens a payable]
  D --> E[Finance answers 200]
  E --> F[Recent deliveries shows Delivered]

Follow along. Seat: offtaker_c@webill.net (Organization; password supersecret). Path: Organization → Feeds & webhooks → Webhooks. You should see: the help line under the title, invoice.issued ticked, and Create webhook. Demo/dev can use the in-process receiver instead of a real finance host (§11.1 local receiver).

Data. The knock for this topic is wired: when a billing document is issued, both the issuer organisation and the recipient organisation are notified (WB.Events.Notifier). The table below says which topics knock, and where.

Each topic and why you would use it

Every topic name is listed here. Why subscribe is the job you would give your other system. Knocks from Your webhooks is what is built and wired today: only four topics are published into this doorbell.

• Configuration (organisation scope) offers only the three wired organisation topics: invoice.issued, credit_note.issued and settlement_period.final. Each carries a one-line description.
• The Market API page (/org/brp/api, party scope) offers the whole list, so the names match the Market Code / Market API vocabulary. A subscriber there can wait forever until a publisher is added. Some of those names can already be read by polling the Market API (V13 §13) — that is a different pipe from this POST.

Money and paper

The market day

Credit cover and notices

Standing data

dispatch.instruction is scoped to a party (your market registration). Create it on /org/brp/api after you select a registration. The other three wired topics are scoped to the organisation and belong on Configuration.

Create a webhook

On Configuration (organisation scope) or /org/brp/api (party scope):

1. Enter an HTTPS callback URL. An http:// address, or one with a user name or password in it, is refused at the field.
2. Tick one or more Topics. None ticked is refused at the field.
3. Press Create webhook (#create-webhook).
4. Copy the whsec_… secret from #webhook-secret-once immediately (Copy puts it on your clipboard). Press I’ve stored it and it is gone. Storage keeps only an encrypted copy. The secret is never included in the JSON body we POST.

The topic list is the table in §11.1 topics. The form default on Configuration is invoice.issued, because that is the everyday finance doorbell and it is wired today.

A URL that already has a webhook is refused (This URL already has a webhook. Change its topics in the list below.). If that webhook was deleted, the message says to restore it instead. If secrets were never provisioned, Create webhook flashes Webhook secrets have not been provisioned for this environment.

Each webhook is a row: endpoint, topic chips, a status chip (Active, Paused, Paused after failures or Deleted) and the last delivery. The row’s round buttons:

After a rotation, deliveries are signed with both the new and the old secret for 24 hours, so the far side can switch without missing a notice (Market API rule — V13 §13). The row says Old secret valid until … while that window is open. Rotating again inside the 24 hours ends the older secret at once. A topic a webhook holds but can never receive at its scope shows as an amber chip marked (not sent here).

What “done” looks like on the far side

We POST JSON to your URL with Standard Webhooks headers. A reply in 200–299 completes the delivery (DELIVERED). Timeouts, 408 / 425 / 429, server errors (500+), and network failures are TRANSIENT — we retry. Any other reply, including 410 Gone, is REFUSED — no retry; the delivery is dead at once.

flowchart TB
  A[You create a webhook and copy the whsec_ secret] --> B[An event of that topic happens]
  B --> C[We POST a signed JSON body to your HTTPS URL]
  C --> D{What did the door say?}
  D -->|200-299| E[DELIVERED — consecutive failures reset]
  D -->|timeout, 408, 425, 429, 500+| F[RETRYING — wait, then try again]
  D -->|other, including 410| G[DEAD at once]
  F --> H{Eight attempts used up?}
  H -->|no| C
  H -->|yes| I[DEAD — a dead letter]
  I --> J[Five dead letters in a row pause the subscription]

Retry gaps (then a small deterministic jitter, about ±10 %):

5 seconds → 5 minutes → 30 minutes → 2 hours → 5 hours → 10 hours → 24 hours.

Eight unsuccessful attempts exhaust a schedule. A retry left waiting more than 72 hours is swept to dead (DeadLetterSweep, 00:20 UTC) with the error Retry schedule abandoned for 72 hours.

Pause stops new events. In-flight deliveries keep their schedule (a paused subscription snoozes 300 seconds and tries again). Five consecutive dead letters pause automatically. Resume is always a human click — the toggle also clears the consecutive-failure counter. DISABLED kills in-flight work as dead (Subscription disabled).

The operations page (/ops/events)

Same gate as the Feed registry: :sim_enabled, no seat, type the URL or follow Delivered events →.

Follow along. Any dev/demo build. Path: /ops/feeds → Delivered events →. You should see: Subscriptions, Delivery log and dead letters, and an empty Attempt timeline until you press Timeline.

Retry re-enqueues the dead letter and keeps the old attempt rows. You get a new schedule; you do not rewrite history.

On Configuration, Recent deliveries lists the latest 20 deliveries across your webhooks: topic, endpoint, a status chip (Queued, Retrying, Delivered or Failed), attempts and time. A Failed row has Send again, which is the same Retry.

flowchart LR
  A[Delivery is DEAD] --> B[You press Retry]
  B --> C[New attempt schedule starts]
  C --> D[Previous Timeline rows stay]

Local receiver and the production key

Dev, demo and test deliver to an in-process receiver (WB.Events.Sink). It checks the signature on the exact bytes. Host failing-sink.example answers 503 on purpose so you can watch retries and dead letters without a real outage.

Production reads EVENTS_SECRET_KEY from deployment secrets (a base64-encoded 32-byte key). Provision it before anyone creates a subscription, and keep it across restarts. Losing it makes stored whsec_ blobs unreadable — you cannot verify or rotate until a new key is in place, and old secrets stay dark.

Careful. Copy whsec_ when you see it. Do not put EVENTS_SECRET_KEY or any whsec_ in this manual, in a ticket, or in a screenshot.

Where this is taught. How a machine verifies webhook-signature, the 20 KB payload cap, and /subscriptions/:id/deliveries/:delivery_id/retry are V13 §13. This chapter owns the human screens and the dead-letter story.

11.2 Plant telemetry feeds

A plant feed is a TELEMETRY row for one generator site, one provider, and one effective start date. Registering it on the owner’s plant screen creates the operations identity plant:{site-id}:{provider}:{from} and stores a hashed draft data schedule. Signing that schedule is a separate commercial act.

A commercial vendor row still needs a credential reference even while unarmed. Kind must match provider for mock / push / mirror.

Successful batches record accepted and rejected counts. The raw payload artifact is the bytes that arrived. Sample revisions keep a correction’s provenance — they never silently rewrite the first truth. Stale and recovered transitions update the feed’s operational status (the plant freshness sweep every five minutes, distinct from the registry’s fifteen-minute sweep).

These feeds are not settlement meters and not carbon inverter MRV. The plant screen links the feed, the draft schedule, and the immutable curtailment traces.

flowchart TB
  A[Owner registers mock, push, or mirror] --> B[TELEMETRY row appears on /ops/feeds and Your feeds]
  B --> C[Collector pushes or the inbox sweep reads a file]
  C --> D[Accepted and rejected counts; raw bytes stored]
  D --> E[Plant now panel can show a fresh sample]
  E --> F[Settlement still reads the 20-minute METER rows]

Follow along. Seat: ipp_2@webill.net (Organization; password supersecret). Path: Atlantis Wind → Strategy → Plant (or Dashboard Plant now → Open plant). You should see: path, latest observation time, and a mock badge on demonstration readings.

Where this is taught. How the owner registers, consents, and reads the panel: V4 §11 and V2 §11. The push API and mirror CSV schema: V13 §17. This chapter owns the registry row and the “not a settlement meter” boundary.

11.3 What runs while you sleep

You do not press these. They keep the pipes honest.

A rolled-back source transaction leaves no delivery. These jobs do not change the monthly settlement-chain order.

Where this is taught. The full clock, including the plant telemetry workers (mirror inbox, plant freshness, MEC scan, traces), is §3.