AT&T Telemetry & KPI — project context

Orientation and current branch state. The measurement-ID reference lives on its own page.

Provenance. Everything below was read out of code on 2026-09-17, not transcribed from earlier planning material: rgnets/rxg at hbh/kpi-update (32c0dfea95, master 360903154b) and rgnets/att-templates at hbh/combined-new-schema (0e44313f27, master 8dccd0ee84). The status narrative here goes stale within weeks; the catalogue page goes stale only when the schema changes.

1  Telemetry has two audiences

Start here. The same ingested data serves two consumers with different requirements, different code paths, and different failure modes. Most confusion about this effort comes from reasoning about one audience while looking at code that serves the other. The AT&T webhook and the measurement catalogue are the off-rXg half only.
On-rXg audienceOff-rXg audience
ConsumerThe rXg itself. Marks APs online and offline, instruments client counts, radio channel/width/power, versions, and raises health notices.AT&T DPA / VERTEX. Later also the contracted reporting engine.
NeedsFreshness and correctness of device state. A stale value here is a wrong AP list, a missed offline alarm, a bad client count in the UI.Schema fidelity and completeness over 15-minute windows. A wrong value is a wrong report; a missing one is invisible.
Operator switchinstrument_from — an enum, :integration (default) or :telemetry.enable_telemetry_payload_collection?, plus the webhook being configured at all.
CiscoInstrumentFromCiscoTelemetryJob every 2 min → StatsTask::Base.run_instrumentation!.
Plus SampleStatsJob every 3 min → run_stats! for our own metrics and graphs.
Grpc.new.to_kpi_report → webhook.
RuckusProcessSciMessage#process_ap_status! / #process_ap_report! / #process_configuration_message!, per pushed message.Mqtt.new.to_kpi_report → webhook.
OpenWiFiTelemetry::UpdateHandlers::{AccessPoint,AccessPointRadio,WirelessClient}, per WSS message.nothing — our own APs report nothing to AT&T today.
Failure visible?Cisco: yes — HealthNotice raised on exception, cured on success. Ruckus: no — a log line only, since !5424.No. Unmapped measurements are silently filtered; a failed push is unrecoverable.

Three asymmetries this exposes

  1. Telemetry replaces integration polling for device state — it does not supplement it. instrument_from is either/or. With :telemetry selected, a telemetry outage is a device-state outage: APs stop being marked online, client counts freeze. That is a much sharper consequence than "the graphs have a gap", and it is why the on-rXg audience cares about freshness far more than about schema completeness.
  2. Cisco pulls on a timer; Ruckus is driven by pushes. Cisco instrumentation is a job that runs every two minutes against the stats schema, wrapped in a begin/rescue that raises a HealthNotice and cures it on the next success — a real observability contract, including a precheck that distinguishes critical from non-critical missing tables. Ruckus instrumentation happens inline, once per pushed SCI message, with no equivalent contract.
  3. Ruckus can only mark an AP online from its two main message types. process_ap_status! sets online: true with the comment "assume we are online if we see this message". The only path that can mark an AP offline is process_configuration_message!, which reads connectionStatus == "Connect". Offline detection therefore rides a different, less frequent message type than online detection. Transitions emit the alarm through AccessPoint#update_online_health_notice.
Consequence for !5424. The blanket rescue added in that MR wraps self.try("process_#{type_key}!") — which is the on-rXg instrumentation call, not the payload storage above it. It keeps one bad payload from killing the worker, which is right, but it means Ruckus AP state can silently stop updating with only a log line, where the Cisco equivalent would surface a HealthNotice. Worth a follow-up: the source comment already says an AdminAlert belongs there.

2  The pipeline in one pass

For the off-rXg audience, a measurement makes four hops between a radio and AT&T. Each hop is owned by a different part of the tree, and each hop can drop data silently, which is why the catalogue page tracks what survives end to end.

HopLives inWhat happens
1. Ingestrxg: console/app/lib/telemetry/{cisco,ruckus,openwifi} Vendor data lands in three unrelated stores — the stats Postgres schema for Cisco, the telemetry_payloads table for Ruckus, and the metrics hypertables for OpenWiFi. See §3. Both audiences read from here.
2. KPI derivation…/cisco/grpc/to_kpi_report.rb · …/ruckus/mqtt/kpis.rb Each vendor class exposes .new.to_kpi_report, returning a flat array of performance data elements. These are the only two such definitions in the repo, and they exist purely for the off-rXg audience.
3. InspectionAdmin::Scaffolds::TelemetryPayloadsController The admin surface that renders those elements as tables, per-KPI lookups, raw payloads, and the metric documentation tab. How you see what AT&T would get, without waiting for a push.
4. Egressatt-templates: notifications.yaml:1524 Webhook Measurement VERTEX KPI → POST /contentListener. Merges both vendors, maps IDs to protobuf ints, wraps in Measurement_event. Pushed every 15 minutes.

The shape of a performance data element

KeyMeaning
measurement_idSnake-case string name. This is the join key to the protobuf enum — see the catalogue.
value / svalueNumeric, or string for text metrics (version, STP mode, VLAN lists).
source_idAP ethernet MAC, or the switch's primary MAC.
extraRadio slot ID for APs; the port's burnt-in-address MAC for switch ports.
source_subtype0 = AP, 1 = switch. (The proto's SourceSubtype enum runs to 14; we emit two.)
source_subcategoryBand: 0 = 2.4G, 1 = 5G, 2 = 6G. Radio metrics only.

3  The three ingest paths share nothing

The three vendor paths have different collectors, different storage, different query languages and different retention. They converge only at hop 2, and only two of the three converge at all.

PathCollectorLands inKPI derivationTo AT&T?
Cisco C9800
gRPC dial-out
telegraf, configured by rxgd (Rxg::Task::Telegraf) from GrpcSubscription rows The stats Postgres schema — one hypertable per YANG path (e.g. stats.access_point_oper_data_radio_oper_data), each with a _tag sidecar joined on tag_id. No ActiveRecord models. 72 calculate_*.rb over 76 SQL/ERB templates. Derivation runs in Postgres, against raw SQL built by schema introspection.yes
Ruckus SmartZone VSZ
SCI / MQTT
StatsTask::ProcessSciMessage telemetry_payloads via TelemetryPayloadRuck (the table's only STI subclass). Composite PK: message_type + device + sampled_at + type. Pure-Ruby procs in a single 454-line kpis.rb. Derivation runs in the Rails process, over cached payload bodies.yes
OpenWiFi
our own APs, rg_wss_client over WSS
telemetry/openwifi/* + update_handlers/* The metrics hypertables: ClientMetric, RadioMetric, ThroughputMetric. None. ActiveRecord scopes and *_graph models serve our own dashboards.no

The asymmetry matters constantly when reading either codebase: a Cisco metric is a SQL template you can run against a schema telegraf owns, a Ruckus metric is a lambda over a JSON hash held in Rails.cache, and an OpenWiFi metric is a row in one of our own hypertables. The same measurement_id string can mean all three.

Nothing on the AT&T path is durable

Both vendor raw stores are deliberately short-lived, because the volume is untenable and the on-rXg audience only ever needs the latest sample. AT&T's own store is the system of record for anything on the off-rXg path — we cannot re-derive last week's measurement from either vendor store.

StoreHorizonEnforced by
stats.* (Cisco)2 hoursPurgeGrpcTelemetryJob every 30 min → Purger.purge_stats! → Timescale drop_chunks on every table in the schema.
Ruckus payload bodies20 minutesRails.cache TTL. The row persists with body: {}; TelemetryPayload#body is a cache read, so older payloads render {}.
Push cadence15 minutesThe webhook schedule. Mqtt.new defaults to a matching 15.minutes.ago window — which fits inside the 20-minute TTL by five minutes. That is the whole safety margin.
On-rXg sampling2–3 minutesCisco only: instrumentation every 2 min, stats every 3 min. Ruckus and OpenWiFi are push-driven, so their cadence is the controller's.

The webhook nonetheless stamps collection_interval: 300 (5 minutes) on every measurement it sends, with the comment "time in seconds?" in the source. That matches none of the intervals above.

4  What the webhook actually does

The ERB body at notifications.yaml:1524 is where the two vendors converge. In order:

Two consequences worth internalising. First, the ID map is a second source of truth that lives in a YAML string, separate from the Ruby that emits those IDs — so drift is invisible to every test either repo has. Second, the 300-second interval is asserted, not measured (see above).

5  Branch and MR state

Two branches carry the live work. Everything else in either repo is either folded into these, superseded, or a different effort that happens to touch KPI code.

RepoBranchMRWhat it is
rgnets/att-templateshbh/combined-new-schema !16
DPA Phase 2 Telemetry [New Schema]
The schema and egress side. +399 notifications.yaml, +164 measurement proto, +162 schema_notes.md. Owns the protobuf ID map, the VERTEX KPI webhook body, and the phase-2 switch and port measurements. Its history shows a deliberate restore-from-master pass to shrink the diff down to just the telemetry change.
Description is currently empty — see §10.
rgnets/rxghbh/kpi-update !5424
draft, 16 commits, 8 files, +708/−105
The producer side. Ruckus hardening and inspectability: payload collisions, PHY coverage, and the admin surface for seeing what AT&T would receive. Detail below.

What is in !5424 specifically

AreaChangeServes
process_sci_message.rbsampled_at from integer to float milliseconds; RecordNotUnique retry that bumps the timestamp 1 ms so identical-instant events stop colliding on the composite PK; a blanket rescue around process_*! so one bad payload can no longer take down the worker.both
ruckus/mqtt/kpis.rbEHT ('be') PHY added to the MCS computation, plus nil guards for the crash on unsupported/legacy PHYs.off-rXg
Admin UIA metric documentation tab (323-line ERB), a per-KPI lookup action-link group generated from Mqtt::KPIS, a raw-payload popup, Raw Data as the default tab, and visual cards.on-rXg
(inspecting off-rXg data)
find_if_allowedOverride working around ActiveScaffold truncating fractional seconds out of composite-PK URLs — it reparses the ID and does a one-second bounding query. Directly downstream of the float sampled_at change.on-rXg
api/telemetry_payloads_controller.rbNew get_ruckus_method_source collection action returning a KPI method's Ruby source.on-rXg
access_point.rbMethods made public so KPI and webhook code can call them.off-rXg

6  In progress

This is the volatile section. It records what is actually being worked and what is only decided, so that nobody re-derives a conclusion or re-opens a settled question. Everything above it describes the system as it exists; this describes intent. Update it or delete it — do not let it quietly rot.
ItemAudienceState as of 2026-09-17
Ruckus payload collisions and PHY coverage
!5424 on hbh/kpi-update
both In review. Draft MR, 16 commits. Fractional sampled_at, unique-violation retry, EHT PHY, admin inspection surface.
Phase 2 schema and egress
!16 on hbh/combined-new-schema
off-rXg In review. Needs a description written before anyone else can review it usefully (§9).
Make Ruckus stream failures visibleon-rXg Written, not yet run. On hbh/kpi-update: two new files, two hooked. Covers every failure on the raw MQTT stream, not only the instrumentation call — see §7. Needs rails test in a real environment.
Consolidate the duplicated SCI decodes
channel width, PHY, TX power
both Done, not yet run. Three decodes that existed twice or three times, and had drifted every time, now live once on TelemetryPayloadRuck — see §8.
Two masked defects in kpis.rb
EXTRACT_RADIO_POWER_LEVEL Hash/object mix-up; CHAN_WIDTHS missing 320 MHz
both Both fixed, along with a third found on the way — the empty-PHY case that a customer was actually hitting. See §8.
Measurement-map drift
12 IDs produced and dropped
off-rXg Identified; needs a decision, not a fix. Which of the 12 should get protobuf IDs is an AT&T-facing question. Four of them are the volumetric ones, which ties to §9.
Admin/API exposure hardening
get_ruckus_method_source allow-list; raw-payload HTML escaping
on-rXg Identified, not fixed. Both are in code !5424 already touches.
Ruckus payload retention
the 20-minute body TTL
both Open question. Whether to persist bodies at all, and for how long, is unresolved. Everything downstream of it inherits the limit (§3).
Gateway metric producer
the 3000-block IDs
off-rXg Not ours. att-templates!20 (ldd/gateway_metrics) is the intended producer, stacked on hbh/combined-new-schema.
OpenWiFi handler failure visibilityon-rXg Not researched. handler.rb has no rescue and no HealthNotice, so it may share the Ruckus gap. Nobody has traced how the WSS message loop treats a raising handler.

7  Proposal: making Ruckus instrumentation failures visible

Designed and written 2026-09-17; tests not yet run. The problem is §10 item 1: when Ruckus telemetry processing fails, the rXg's own view of its fleet silently stops updating, and the only evidence is a log line. Cisco raises a HealthNotice for the same class of failure. The goal is to make the two vendors symmetric without introducing an alarm that scales with fleet size.

Scope is the whole raw MQTT stream, not just the instrumentation call. The first draft of this section only covered the rescue inside ProcessSciMessage. That is one of four places a message can die: the subscriber callback in rxgd/bin/mqtt_subscriber_ruckus_sci has three more rescues, and two of them sit around work that happens before ProcessSciMessage exists — SciMessage.decode, and the constructor, where InfrastructureDevice.find can raise. Recording therefore happens at each rescue rather than inside the processor.

Why this must not be a per-AP notice

The rXg already has a per-AP telemetry notice — AccessPoint#update_online_health_notice — and it is correct for what it does: an AP going offline is a fact about one device that an operator must act on, and it fires on a state transition rather than on a sample. An instrumentation failure is the opposite: it is a property of the pipeline, not of any AP. A single bug in a shared code path would raise one identical alarm per AP, bury every unrelated notice in the ticker, and send an email per AP on creation. So: one notice, fleet-wide, regardless of how many APs are affected.

What the HealthNotice API forces

These four constraints come out of reading create_or_append_to_logger_notice and cure_by_name, and between them they dictate most of the design.

ConstraintConsequence
One uncured record per (cluster_node_id, name).A single fixed name gives exactly one record, however wide the failure.
Dedupe is hn.long_message.match(Regexp.escape(msg)) — an identical msg is a total no-op; anything new is appended.The msg body must be a stable signature. Interpolating an AP MAC or a running count makes every occurrence distinct, so the text column grows without bound. That is the per-AP spam again, hiding inside one record.
CustomEmail.deliver_for_event fires on record creation and on cure, not on append.Appends are free. Raise-once/cure-once costs two emails per real incident — but a flapping raise/cure cycle costs two per flap.
The dedupe guard wraps the whole hn.update!.A stable msg also freezes short_message and detail_json at first sighting. Refreshing counts needs a direct write to the returned record.

The shape

Aggregation moves out of the per-message path entirely, mirroring the Cisco structure of InstrumentFromCiscoTelemetryJob + StatsTask::Base.

WhereWhat it does
The four stream rescues
per message
Each logs exactly as it did before — the per-AP detail stays in the log, where it belongs — and additionally records a failure signature counter. No HealthNotice call in the hot path. Recording never raises: a health-reporting problem must not become the reason the stream stops. Stages: instrument for the processor's own rescue, database for the reconnect-and-retry path, stream for the other two.
The signature type_key + exception class + first application backtrace frame. Deliberately excludes the AP MAC, any payload value, and any count, so that 500 APs hitting one bug collapse to one line.
A new periodic job
~5 min
Drains the counters, raises or appends one notice under a fixed name, then refreshes counts and example detail by direct write. perform_if gated on a ruckos device with telemetry enabled, matching the Cisco job's guard.
The cure policy Cure only after N consecutive clean windows (3 → roughly 15 minutes). Note that Cisco cures on every clean run, which means an intermittent Cisco failure does email on each flap — that part is not worth copying.

Name it alongside the existing constants, so the pair reads symmetrically in the notice ticker:

VendorHealthNotice name
CiscogRPC / Telegraf Telemetry Instrumentation Failed. (CISCO Telemetry)
RuckusSCI / MQTT Telemetry Instrumentation Failed. (RUCKUS Telemetry)  proposed

The two currently-masked defects in kpis.rb would surface as exactly two signature lines in one notice — which is the concrete test that the design works.

As built

FileRole
console/app/lib/stats_task/ruckus_telemetry_health.rb
new
Signature construction, recording, draining, and the clean-drain counter. Counters live in a redis hash via Rails.cache.redis, which is what makes a count written by the subscriber process visible to the job process.
console/app/jobs/monitor_ruckus_telemetry_job.rb
new
The only place that touches the HealthNotice. Drains, raises or appends one notice, refreshes counts by direct write, cures after three consecutive quiet runs. Stays eligible to run while a notice is open, so switching telemetry off cannot leave an alarm stuck uncured.
console/app/lib/stats_task/process_sci_message.rbThe instrument rescue records instead of only commenting that something ought to.
rxgd/bin/mqtt_subscriber_ruckus_sciThe three subscriber rescues record.
console/test/lib/ruckus_telemetry_health_test.rb
console/test/jobs/monitor_ruckus_telemetry_job_test.rb
new
The load-bearing assertions: two APs failing the same way collapse to one signature; the notice does not scale with failure count; an unchanged body is not appended twice while its count still refreshes; curing waits for the full quiet streak.

What this deliberately does not cover

8  The two paths had drifted apart

Worked through 2026-09-17, prompted by a subscriber log from a customer rXg captured on 2026-09-10. The two audiences of §1 are served by two independent decoders of the same SCI payloads: process_sci_message.rb holds protobuf message objects, while ruckus/mqtt/kpis.rb reads JSON-parsed hashes of those same messages. Every piece of shared decode logic therefore existed twice, with nothing forcing the copies to agree — and each one had in fact drifted.

DecodeCopiesHow they differedConsequence
Channel width
channelWidth
three
twice in kpis.rb alone
Identical, but none had 320 MHz. The SCI protobuf declares no enum for the field and documents it only as "Channel width used by the WLAN", so the codes were derived by lab testing and the one meaning 320 is still unknown. 320 MHz radios would silently record no mcs_avg, because width feeds the MCS computation.
Client PHY
radioMode
two The KPI path learned 'be'; the instrumentation path never did. Live at a customer. Every 11be client decoded to a nil PHY, was written out as an empty-string PHY metric, and then raised "Only ht vht he and eht are supported as PHYs, not " into a rescue that logged at debug level. Five times per direction in thirty seconds, no visible symptom.
TX power
txPower
two Both mapped max to 23 dBm and min to 10, but one read a dB reduction as 23 + offset and the other as 22 + offset. The KPI copy also called radio.txPower on the hash it had just read radio["txPower"] from, so any reduction raised NoMethodError. A radio at anything other than an endpoint reported one dBm lower to the rXg than to AT&T — and the KPI side could not report a reduction at all. Radios in the field all run Full, which is why it had not surfaced.

All three now decode once on TelemetryPayloadRuck, beside MESSAGE_TYPE_LABELS — the Ruckus SCI vocabulary both callers already depended on. McsComputation::VALID_WIDTHS is the single source of truth for the widths the MCS maths accepts.

Two that are not drift, checked

Still divergent by design, and worth a decision

client_count has three derivations for what is nominally one KPI. Each is defensible for its own consumer, but they will not agree, and AT&T's measurement 111 not matching the number in the rXg UI is a customer question waiting to be asked.

ConsumerSourceQuantity
AccessPointRadio#curr_client_count
the admin UI
apStatus radio.wlans.sum(&:totalNumClients)instantaneous total
RadioMetric :client_count
our graphs
apReport radio.binWlan.sum(&:avgNumClients)sum of bin averages
AT&T measurement 111apReport distinct binClient entries per radioclients seen during the bin

Also unresolved: our EIRP model disagrees with the controller's. The instrumentation path logs curr_txpower + gain = radio.eirp? as an explicit question, and on the captured customer AP the answer is no — 23+18 against the controller's 31, and 23+23 against its 45.

9  Storage and rollup

This concerns our own metrics tables — the on-rXg audience's dashboards and graphs — not the AT&T egress path, which keeps no history at all (§3). Two schema patterns coexist:

Narrow (key-value)Wide (bundled)
ModelsClientMetric, RadioMetricThroughputMetric
Shapemetric_type + value_int/value_bigintbyte_delta_in/out, bytes_ps_in/out on one row
IdentityFK columnsString polymorphism: graphable_type, graphable_id, series_key
Buys youNew metrics with no migration — append to the enumOne read returns both directions and both rates
Costs youFragmented reads; TX, RX and RSSI are separate rows to join in RubyRigid; non-bandwidth metrics do not fit; TEXT columns need dictionary compression

The continuous-aggregate defect. 10 of the 11 *_metrics tables build their aggregates from an agg_cols loop that emits AVG, MAX and MIN but not SUM. Any volumetric query past the raw-retention window therefore either breaks the UI or silently returns an average of averages. throughput_metrics is the only table that gets it right. Track this against rxg!5503 (dpt/metric-aggregate-tiers) before building anything new on top.

10  Known gaps

Ordered roughly by how much they would surprise a teammate. Items 3–5 are visible in the catalogue's drift tables.

  1. Ruckus instrumentation failures are invisible — the blanket rescue in !5424 swallows on-rXg instrumentation errors with a log line, where Cisco raises a HealthNotice. Addressed in §7, written but not yet run. The three defects it was masking are fixed in §8; the point of §7 is that the next three will not need a customer log to find.
  2. A patch applied to an older base can kill record broadcasting fleet-wide, silently. Seen in the field on 2026-09-10. RxgResourceFetchService globs app/controllers/api/*_controller.rb and constantizes every one, and safe_constantize swallows NameError but not NoMethodError — so a single API controller that raises while loading poisons the map for every model. RxgBroadcastable then rescues StandardError into one log line per broadcast. On that box, live updates were dead for AccessPoint, AccessPointRadio, WirelessClient, TelemetryClient and TelemetryPayloadRuck alike.
    Cause was version skew, not the branch: a patch cut from a post-2026-08-14 branch used add_action, which the rest_framework v2 upgrade of that date introduced, on a base still running v1. Nothing to change in the code — but a patch must be cut against the base it is going to.
  3. Two whole ID blocks have no producer. The 20 gateway_* IDs (3000–3020, 3002 unused) and the 17 wlan_sensor_* IDs (2000–2016) are reserved in the map and emitted by nothing. schema_notes.md says as much; att-templates!20 (ldd/gateway_metrics) is the intended gateway producer.
  4. Twelve IDs are produced and then dropped in transit — including every byte counter and port speed/status. See the catalogue.
  5. Two protobuf ints are double-assigned. radio_gain and radio_power_level both map to 126 (deliberate, commented). temperature and switch_poe_budget both map to 4001 (not deliberate).
  6. Ruckus SNR and RSSI are crossed at the vendor. EXTRACT_SNR reads client["rssi"] and EXTRACT_RSSI reads client["receiveSignalStrength"], with the comment "ruckus messed this up on their end". AT&T receives both 115 and 116 from that swap, so it needs to stay documented loudly.
  7. Ruckus offline detection rides a different message type than online detection (§1), so the two are not equally timely.
  8. Neither vendor store outlives the push interval (§3), so a missed or failed push is unrecoverable data loss rather than a delay.
  9. !16 has an empty description — the trunk MR of the effort is unreadable to anyone who did not write it.
  10. Phase-1 IDs 118–121 (packet retransmit, latency, jitter, packet loss) are in schema_notes.md but absent from the map; they appear to have been re-scoped into the wlan_sensor_* block.

Where to look next

QuestionOpen this
What does AT&T actually receive?catalogue.html
How does the rXg mark an AP online?stats_task/base.rb (Cisco) · stats_task/process_sci_message.rb (Ruckus) · telemetry/update_handlers/ (OpenWiFi)
How is a Cisco metric computed?console/app/lib/telemetry/cisco/grpc/kpis/calculate_*.rb + sql/
Which YANG paths do we subscribe to?console/app/lib/telemetry/cisco/grpc/{defaults,used_paths}.rb, and the GrpcSubscription scaffold
How is a Ruckus metric computed?console/app/lib/telemetry/ruckus/mqtt/kpis.rb
What gets sent, and how?att-templates/notifications.yaml:1524
What did we promise AT&T?att-templates/schema_notes.md
What is the reporting engine?planning/att-reporting-engine/ on !5362

Generated 2026-09-17. Status sections reflect that date only; re-read the branch table before trusting it.