Orientation and current branch state. The measurement-ID reference lives on its own page.
| On-rXg audience | Off-rXg audience | |
|---|---|---|
| Consumer | The rXg itself. Marks APs online and offline, instruments client counts, radio channel/width/power, versions, and raises health notices. | AT&T DPA / VERTEX. Later also the contracted reporting engine. |
| Needs | Freshness and correctness of device state. A stale value here is a wrong AP list, a missed offline alarm, a bad client count in the UI. | Schema fidelity and completeness over 15-minute windows. A wrong value is a wrong report; a missing one is invisible. |
| Operator switch | instrument_from — an enum, :integration (default) or :telemetry. | enable_telemetry_payload_collection?, plus the webhook being configured at all. |
| Cisco | InstrumentFromCiscoTelemetryJob every 2 min → StatsTask::Base.run_instrumentation!.Plus SampleStatsJob every 3 min → run_stats! for our own metrics and graphs. | Grpc.new.to_kpi_report → webhook. |
| Ruckus | ProcessSciMessage#process_ap_status! / #process_ap_report! / #process_configuration_message!, per pushed message. | Mqtt.new.to_kpi_report → webhook. |
| OpenWiFi | Telemetry::UpdateHandlers::{AccessPoint,AccessPointRadio,WirelessClient}, per WSS message. | nothing — our own APs report nothing to AT&T today. |
| Failure visible? | Cisco: yes — HealthNotice raised on exception, cured on success. Ruckus: no — a log line only, since !5424. | No. Unmapped measurements are silently filtered; a failed push is unrecoverable. |
instrument_from is either/or. With :telemetry selected, a telemetry outage is a device-state
outage: APs stop being marked online, client counts freeze. That is a much sharper consequence than "the graphs have a gap",
and it is why the on-rXg audience cares about freshness far more than about schema completeness.stats schema, wrapped in a begin/rescue that raises a HealthNotice and cures it on the
next success — a real observability contract, including a precheck that distinguishes critical from non-critical missing
tables. Ruckus instrumentation happens inline, once per pushed SCI message, with no equivalent contract.process_ap_status! sets
online: true with the comment "assume we are online if we see this message". The only path that can mark an AP
offline is process_configuration_message!, which reads
connectionStatus == "Connect". Offline detection therefore rides a different, less frequent message type than
online detection. Transitions emit the alarm through AccessPoint#update_online_health_notice.self.try("process_#{type_key}!") — which is the on-rXg instrumentation call, not the payload storage
above it. It keeps one bad payload from killing the worker, which is right, but it means Ruckus AP state can silently stop
updating with only a log line, where the Cisco equivalent would surface a HealthNotice. Worth a follow-up: the
source comment already says an AdminAlert belongs there.For the off-rXg audience, a measurement makes four hops between a radio and AT&T. Each hop is owned by a different part of the tree, and each hop can drop data silently, which is why the catalogue page tracks what survives end to end.
| Hop | Lives in | What happens |
|---|---|---|
| 1. Ingest | rxg: console/app/lib/telemetry/{cisco,ruckus,openwifi} | Vendor data lands in three unrelated stores — the stats Postgres schema for Cisco, the telemetry_payloads table for Ruckus, and the metrics hypertables for OpenWiFi. See §3. Both audiences read from here. |
| 2. KPI derivation | …/cisco/grpc/to_kpi_report.rb · …/ruckus/mqtt/kpis.rb | Each vendor class exposes .new.to_kpi_report, returning a flat array of performance data elements. These are the only two such definitions in the repo, and they exist purely for the off-rXg audience. |
| 3. Inspection | Admin::Scaffolds::TelemetryPayloadsController | The admin surface that renders those elements as tables, per-KPI lookups, raw payloads, and the metric documentation tab. How you see what AT&T would get, without waiting for a push. |
| 4. Egress | att-templates: notifications.yaml:1524 | Webhook Measurement VERTEX KPI → POST /contentListener. Merges both vendors, maps IDs to protobuf ints, wraps in Measurement_event. Pushed every 15 minutes. |
| Key | Meaning |
|---|---|
| measurement_id | Snake-case string name. This is the join key to the protobuf enum — see the catalogue. |
| value / svalue | Numeric, or string for text metrics (version, STP mode, VLAN lists). |
| source_id | AP ethernet MAC, or the switch's primary MAC. |
| extra | Radio slot ID for APs; the port's burnt-in-address MAC for switch ports. |
| source_subtype | 0 = AP, 1 = switch. (The proto's SourceSubtype enum runs to 14; we emit two.) |
| source_subcategory | Band: 0 = 2.4G, 1 = 5G, 2 = 6G. Radio metrics only. |
The three vendor paths have different collectors, different storage, different query languages and different retention. They converge only at hop 2, and only two of the three converge at all.
| Path | Collector | Lands in | KPI derivation | To AT&T? |
|---|---|---|---|---|
| Cisco C9800 gRPC dial-out |
telegraf, configured by rxgd (Rxg::Task::Telegraf) from GrpcSubscription rows |
The stats Postgres schema — one hypertable per YANG path (e.g. stats.access_point_oper_data_radio_oper_data), each with a _tag sidecar joined on tag_id. No ActiveRecord models. |
72 calculate_*.rb over 76 SQL/ERB templates. Derivation runs in Postgres, against raw SQL built by schema introspection. | yes |
| Ruckus SmartZone VSZ SCI / MQTT |
StatsTask::ProcessSciMessage | telemetry_payloads via TelemetryPayloadRuck (the table's only STI subclass). Composite PK: message_type + device + sampled_at + type. |
Pure-Ruby procs in a single 454-line kpis.rb. Derivation runs in the Rails process, over cached payload bodies. | yes |
| OpenWiFi our own APs, rg_wss_client over WSS |
telemetry/openwifi/* + update_handlers/* | The metrics hypertables: ClientMetric, RadioMetric, ThroughputMetric. |
None. ActiveRecord scopes and *_graph models serve our own dashboards. | no |
The asymmetry matters constantly when reading either codebase: a Cisco metric is a SQL template you can run
against a schema telegraf owns, a Ruckus metric is a lambda over a JSON hash held in Rails.cache, and an
OpenWiFi metric is a row in one of our own hypertables. The same measurement_id string can mean all three.
Both vendor raw stores are deliberately short-lived, because the volume is untenable and the on-rXg audience only ever needs the latest sample. AT&T's own store is the system of record for anything on the off-rXg path — we cannot re-derive last week's measurement from either vendor store.
| Store | Horizon | Enforced by |
|---|---|---|
| stats.* (Cisco) | 2 hours | PurgeGrpcTelemetryJob every 30 min → Purger.purge_stats! → Timescale drop_chunks on every table in the schema. |
| Ruckus payload bodies | 20 minutes | Rails.cache TTL. The row persists with body: {}; TelemetryPayload#body is a cache read, so older payloads render {}. |
| Push cadence | 15 minutes | The webhook schedule. Mqtt.new defaults to a matching 15.minutes.ago window — which fits inside the 20-minute TTL by five minutes. That is the whole safety margin. |
| On-rXg sampling | 2–3 minutes | Cisco only: instrumentation every 2 min, stats every 3 min. Ruckus and OpenWiFi are push-driven, so their cadence is the controller's. |
The webhook nonetheless stamps collection_interval: 300 (5 minutes) on every measurement it
sends, with the comment "time in seconds?" in the source. That matches none of the intervals above.
The ERB body at notifications.yaml:1524 is where the two vendors converge. In order:
Telemetry::Cisco::Grpc::Grpc.new.to_kpi_report and Telemetry::Ruckus::Mqtt::Mqtt.new.to_kpi_report.device_manufacturer ('cisco' / 'ruckus').measurement_id is a key in the inline ATT_PROTOBUF_MEASUREMENT_TYPES hash — 71 entries. Anything else is dropped with no error.measurement_id underscores to spaces, and divides aggregate_throughput_avg / throughput_per_client_avg by 10⁶ to reach Mbps.reporter_id = the rXg's domain name, customer_id / venue_id from the att_venue_info CustomDataSet, collection_interval hard-coded to 300 s.Two consequences worth internalising. First, the ID map is a second source of truth that lives in a YAML string, separate from the Ruby that emits those IDs — so drift is invisible to every test either repo has. Second, the 300-second interval is asserted, not measured (see above).
Two branches carry the live work. Everything else in either repo is either folded into these, superseded, or a different effort that happens to touch KPI code.
| Repo | Branch | MR | What it is |
|---|---|---|---|
| rgnets/att-templates | hbh/combined-new-schema | !16 DPA Phase 2 Telemetry [New Schema] |
The schema and egress side. +399 notifications.yaml, +164 measurement proto, +162 schema_notes.md. Owns the
protobuf ID map, the VERTEX KPI webhook body, and the phase-2 switch and port measurements. Its history shows a deliberate
restore-from-master pass to shrink the diff down to just the telemetry change.
Description is currently empty — see §10. |
| rgnets/rxg | hbh/kpi-update | !5424 draft, 16 commits, 8 files, +708/−105 |
The producer side. Ruckus hardening and inspectability: payload collisions, PHY coverage, and the admin surface for seeing what AT&T would receive. Detail below. |
| Area | Change | Serves |
|---|---|---|
| process_sci_message.rb | sampled_at from integer to float milliseconds; RecordNotUnique retry that bumps the timestamp 1 ms so identical-instant events stop colliding on the composite PK; a blanket rescue around process_*! so one bad payload can no longer take down the worker. | both |
| ruckus/mqtt/kpis.rb | EHT ('be') PHY added to the MCS computation, plus nil guards for the crash on unsupported/legacy PHYs. | off-rXg |
| Admin UI | A metric documentation tab (323-line ERB), a per-KPI lookup action-link group generated from Mqtt::KPIS, a raw-payload popup, Raw Data as the default tab, and visual cards. | on-rXg (inspecting off-rXg data) |
| find_if_allowed | Override working around ActiveScaffold truncating fractional seconds out of composite-PK URLs — it reparses the ID and does a one-second bounding query. Directly downstream of the float sampled_at change. | on-rXg |
| api/telemetry_payloads_controller.rb | New get_ruckus_method_source collection action returning a KPI method's Ruby source. | on-rXg |
| access_point.rb | Methods made public so KPI and webhook code can call them. | off-rXg |
| Item | Audience | State as of 2026-09-17 |
|---|---|---|
| Ruckus payload collisions and PHY coverage !5424 on hbh/kpi-update | both | In review. Draft MR, 16 commits. Fractional sampled_at, unique-violation retry, EHT PHY, admin inspection surface. |
| Phase 2 schema and egress !16 on hbh/combined-new-schema | off-rXg | In review. Needs a description written before anyone else can review it usefully (§9). |
| Make Ruckus stream failures visible | on-rXg | Written, not yet run. On hbh/kpi-update: two new files, two hooked. Covers every failure on the raw MQTT stream, not only the instrumentation call — see §7. Needs rails test in a real environment. |
| Consolidate the duplicated SCI decodes channel width, PHY, TX power | both | Done, not yet run. Three decodes that existed twice or three times, and had drifted every time, now live once on
TelemetryPayloadRuck — see §8. |
Two masked defects in kpis.rbEXTRACT_RADIO_POWER_LEVEL Hash/object mix-up; CHAN_WIDTHS missing 320 MHz | both | Both fixed, along with a third found on the way — the empty-PHY case that a customer was actually hitting. See §8. |
| Measurement-map drift 12 IDs produced and dropped | off-rXg | Identified; needs a decision, not a fix. Which of the 12 should get protobuf IDs is an AT&T-facing question. Four of them are the volumetric ones, which ties to §9. |
Admin/API exposure hardeningget_ruckus_method_source allow-list; raw-payload HTML escaping | on-rXg | Identified, not fixed. Both are in code !5424 already touches. |
| Ruckus payload retention the 20-minute body TTL | both | Open question. Whether to persist bodies at all, and for how long, is unresolved. Everything downstream of it inherits the limit (§3). |
| Gateway metric producer the 3000-block IDs | off-rXg | Not ours. att-templates!20 (ldd/gateway_metrics) is the intended producer, stacked on hbh/combined-new-schema. |
| OpenWiFi handler failure visibility | on-rXg | Not researched. handler.rb has no rescue and no HealthNotice, so it may share the Ruckus gap. Nobody has traced how the WSS message loop treats a raising handler. |
Designed and written 2026-09-17; tests not yet run. The problem is §10 item 1: when Ruckus telemetry processing fails, the rXg's own view of its fleet silently stops updating, and the only evidence is a log line. Cisco raises a HealthNotice for the same class of failure. The goal is to make the two vendors symmetric without introducing an alarm that scales with fleet size.
Scope is the whole raw MQTT stream, not just the instrumentation call. The first draft of this section only
covered the rescue inside ProcessSciMessage. That is one of four places a message can die: the subscriber
callback in rxgd/bin/mqtt_subscriber_ruckus_sci has three more rescues, and two of them sit
around work that happens before ProcessSciMessage exists — SciMessage.decode, and the
constructor, where InfrastructureDevice.find can raise. Recording therefore happens at each rescue rather
than inside the processor.
The rXg already has a per-AP telemetry notice — AccessPoint#update_online_health_notice — and it is correct
for what it does: an AP going offline is a fact about one device that an operator must act on, and it fires on a state
transition rather than on a sample. An instrumentation failure is the opposite: it is a property of the pipeline,
not of any AP. A single bug in a shared code path would raise one identical alarm per AP, bury every unrelated notice in
the ticker, and send an email per AP on creation. So: one notice, fleet-wide, regardless of how many APs are affected.
These four constraints come out of reading create_or_append_to_logger_notice and cure_by_name,
and between them they dictate most of the design.
| Constraint | Consequence |
|---|---|
One uncured record per (cluster_node_id, name). | A single fixed name gives exactly one record, however wide the failure. |
Dedupe is hn.long_message.match(Regexp.escape(msg)) — an identical msg is a total no-op; anything new is appended. | The msg body must be a stable signature. Interpolating an AP MAC or a running count makes every occurrence distinct, so the text column grows without bound. That is the per-AP spam again, hiding inside one record. |
CustomEmail.deliver_for_event fires on record creation and on cure, not on append. | Appends are free. Raise-once/cure-once costs two emails per real incident — but a flapping raise/cure cycle costs two per flap. |
The dedupe guard wraps the whole hn.update!. | A stable msg also freezes short_message and detail_json at first sighting. Refreshing counts needs a direct write to the returned record. |
Aggregation moves out of the per-message path entirely, mirroring the Cisco structure of
InstrumentFromCiscoTelemetryJob + StatsTask::Base.
| Where | What it does |
|---|---|
| The four stream rescues per message |
Each logs exactly as it did before — the per-AP detail stays in the log, where it belongs — and additionally records a
failure signature counter. No HealthNotice call in the hot path. Recording never raises: a health-reporting
problem must not become the reason the stream stops. Stages: instrument for the processor's own rescue,
database for the reconnect-and-retry path, stream for the other two. |
| The signature | type_key + exception class + first application backtrace frame. Deliberately excludes the AP MAC,
any payload value, and any count, so that 500 APs hitting one bug collapse to one line. |
| A new periodic job ~5 min |
Drains the counters, raises or appends one notice under a fixed name, then refreshes counts and example
detail by direct write. perform_if gated on a ruckos device with telemetry enabled, matching
the Cisco job's guard. |
| The cure policy | Cure only after N consecutive clean windows (3 → roughly 15 minutes). Note that Cisco cures on every clean run, which means an intermittent Cisco failure does email on each flap — that part is not worth copying. |
Name it alongside the existing constants, so the pair reads symmetrically in the notice ticker:
| Vendor | HealthNotice name |
|---|---|
| Cisco | gRPC / Telegraf Telemetry Instrumentation Failed. (CISCO Telemetry) |
| Ruckus | SCI / MQTT Telemetry Instrumentation Failed. (RUCKUS Telemetry) proposed |
The two currently-masked defects in kpis.rb would surface as exactly two signature lines in one notice — which is the concrete test that the design works.
| File | Role |
|---|---|
| console/app/lib/stats_task/ruckus_telemetry_health.rb new |
Signature construction, recording, draining, and the clean-drain counter. Counters live in a redis hash via
Rails.cache.redis, which is what makes a count written by the subscriber process visible to the job process. |
| console/app/jobs/monitor_ruckus_telemetry_job.rb new |
The only place that touches the HealthNotice. Drains, raises or appends one notice, refreshes counts by direct write, cures after three consecutive quiet runs. Stays eligible to run while a notice is open, so switching telemetry off cannot leave an alarm stuck uncured. |
| console/app/lib/stats_task/process_sci_message.rb | The instrument rescue records instead of only commenting that something ought to. |
| rxgd/bin/mqtt_subscriber_ruckus_sci | The three subscriber rescues record. |
| console/test/lib/ruckus_telemetry_health_test.rb console/test/jobs/monitor_ruckus_telemetry_job_test.rb new |
The load-bearing assertions: two APs failing the same way collapse to one signature; the notice does not scale with failure count; an unchanged body is not appended twice while its count still refreshes; curing waits for the full quiet streak. |
touch_infdev_telemetry_last_seen_at is called on every message.Worked through 2026-09-17, prompted by a subscriber log from a customer rXg captured on 2026-09-10. The two audiences of §1 are served by two independent decoders of the same SCI payloads: process_sci_message.rb holds protobuf message objects, while ruckus/mqtt/kpis.rb reads JSON-parsed hashes of those same messages. Every piece of shared decode logic therefore existed twice, with nothing forcing the copies to agree — and each one had in fact drifted.
| Decode | Copies | How they differed | Consequence |
|---|---|---|---|
| Channel width channelWidth | three twice in kpis.rb alone |
Identical, but none had 320 MHz. The SCI protobuf declares no enum for the field and documents it only as "Channel width used by the WLAN", so the codes were derived by lab testing and the one meaning 320 is still unknown. | 320 MHz radios would silently record no mcs_avg, because width feeds the MCS computation. |
| Client PHY radioMode | two | The KPI path learned 'be'; the instrumentation path never did. |
Live at a customer. Every 11be client decoded to a nil PHY, was written out as an empty-string PHY metric, and then raised "Only ht vht he and eht are supported as PHYs, not " into a rescue that logged at debug level. Five times per direction in thirty seconds, no visible symptom. |
| TX power txPower | two | Both mapped max to 23 dBm and min to 10, but one read a dB reduction as 23 + offset and the other as 22 + offset. The KPI copy also called radio.txPower on the hash it had just read radio["txPower"] from, so any reduction raised NoMethodError. |
A radio at anything other than an endpoint reported one dBm lower to the rXg than to AT&T — and the KPI side could not report a reduction at all. Radios in the field all run Full, which is why it had not surfaced. |
All three now decode once on TelemetryPayloadRuck, beside MESSAGE_TYPE_LABELS — the Ruckus SCI
vocabulary both callers already depended on. McsComputation::VALID_WIDTHS is the single source of truth for
the widths the MCS maths accepts.
rxBytesR versus rxBytes_r is the same field seen through protobuf JSON naming versus the
protobuf accessor. Not a bug.receiveSignalStrength→rssi, rssi→snr), so at least the mislabelling is consistent.client_count has three derivations for what is nominally one KPI. Each is defensible for its own
consumer, but they will not agree, and AT&T's measurement 111 not matching the number in the rXg UI is a customer
question waiting to be asked.
| Consumer | Source | Quantity |
|---|---|---|
AccessPointRadio#curr_client_countthe admin UI | apStatus radio.wlans.sum(&:totalNumClients) | instantaneous total |
RadioMetric :client_countour graphs | apReport radio.binWlan.sum(&:avgNumClients) | sum of bin averages |
| AT&T measurement 111 | apReport distinct binClient entries per radio | clients seen during the bin |
Also unresolved: our EIRP model disagrees with the controller's. The instrumentation path logs
curr_txpower + gain = radio.eirp? as an explicit question, and on the captured customer AP the answer is no —
23+18 against the controller's 31, and 23+23 against its 45.
This concerns our own metrics tables — the on-rXg audience's dashboards and graphs — not the AT&T egress path, which keeps no history at all (§3). Two schema patterns coexist:
| Narrow (key-value) | Wide (bundled) | |
|---|---|---|
| Models | ClientMetric, RadioMetric | ThroughputMetric |
| Shape | metric_type + value_int/value_bigint | byte_delta_in/out, bytes_ps_in/out on one row |
| Identity | FK columns | String polymorphism: graphable_type, graphable_id, series_key |
| Buys you | New metrics with no migration — append to the enum | One read returns both directions and both rates |
| Costs you | Fragmented reads; TX, RX and RSSI are separate rows to join in Ruby | Rigid; non-bandwidth metrics do not fit; TEXT columns need dictionary compression |
The continuous-aggregate defect. 10 of the 11 *_metrics tables build their aggregates from an
agg_cols loop that emits AVG, MAX and MIN but not SUM.
Any volumetric query past the raw-retention window therefore either breaks the UI or silently returns an average of averages.
throughput_metrics is the only table that gets it right. Track this against rxg!5503 (dpt/metric-aggregate-tiers) before building anything new on top.
Ordered roughly by how much they would surprise a teammate. Items 3–5 are visible in the catalogue's drift tables.
RxgResourceFetchService globs app/controllers/api/*_controller.rb and
constantizes every one, and safe_constantize swallows NameError but not
NoMethodError — so a single API controller that raises while loading poisons the map for
every model. RxgBroadcastable then rescues StandardError into one log line per
broadcast. On that box, live updates were dead for AccessPoint, AccessPointRadio,
WirelessClient, TelemetryClient and TelemetryPayloadRuck alike.
add_action, which the rest_framework v2 upgrade of that date introduced, on a base still running v1. Nothing
to change in the code — but a patch must be cut against the base it is going to.gateway_* IDs (3000–3020, 3002 unused) and the 17 wlan_sensor_* IDs (2000–2016) are reserved in the map and emitted by nothing. schema_notes.md says as much; att-templates!20 (ldd/gateway_metrics) is the intended gateway producer.radio_gain and radio_power_level both map to 126 (deliberate, commented). temperature and switch_poe_budget both map to 4001 (not deliberate).EXTRACT_SNR reads client["rssi"] and EXTRACT_RSSI reads client["receiveSignalStrength"], with the comment "ruckus messed this up on their end". AT&T receives both 115 and 116 from that swap, so it needs to stay documented loudly.wlan_sensor_* block.| Question | Open this |
|---|---|
| What does AT&T actually receive? | catalogue.html |
| How does the rXg mark an AP online? | stats_task/base.rb (Cisco) · stats_task/process_sci_message.rb (Ruckus) · telemetry/update_handlers/ (OpenWiFi) |
| How is a Cisco metric computed? | console/app/lib/telemetry/cisco/grpc/kpis/calculate_*.rb + sql/ |
| Which YANG paths do we subscribe to? | console/app/lib/telemetry/cisco/grpc/{defaults,used_paths}.rb, and the GrpcSubscription scaffold |
| How is a Ruckus metric computed? | console/app/lib/telemetry/ruckus/mqtt/kpis.rb |
| What gets sent, and how? | att-templates/notifications.yaml:1524 |
| What did we promise AT&T? | att-templates/schema_notes.md |
| What is the reporting engine? | planning/att-reporting-engine/ on !5362 |
Generated 2026-09-17. Status sections reflect that date only; re-read the branch table before trusting it.