Date of test : 2026-09-15
Location : RG Nets lab (rxg.factorialwebdevelopment.com)
Author : Henry Haller
1.SUMMARY
80 trials were run against a single Cigtech WF-660a (IPQ6018 / ath11k): on each
of two firmware builds, 10 cross-radio roams in each direction (2.4 -> 5 and
5 -> 2.4) and 10 intra-radio reassociations on each band as a control. The
test client, channels, client MAC, RF conditions and test script were identical
throughout; the only intentional difference was the firmware build.
CROSS-RADIO (moving a client between the AP's two radios)
rgnets-4.1.1-0.006 : 10 of 20 completed (50%). None within 2 s.
Median when successful ~5.0 s.
rgnets-4.2.4-0.011 : 19 of 20 completed (95%), all in under 1.02 s.
INTRA-RADIO (reassociating to the same radio) -- CONTROL
rgnets-4.1.1-0.006 : 20 of 20 completed. ZERO peer-delete timeouts. rgnets-4.2.4-0.011 : 20 of 20 completed. ZERO peer-delete timeouts.
The defect is confined to cross-radio transitions and fires on 100% of them in
BOTH builds (30 driver timeouts per 20 roams on 4.1.1, 21 per 20 on 4.2.4).
4.2.4-0.011 contains a workaround that shortens the driver's blind period from
3 s to 100 ms and cleans up afterwards. It does not fix the defect; it reduces
a user-visible connection failure to a rare one, at the cost of corrupting a
driver peer counter.
Critically, 4.1.1 is NOT a generally slower build: its intra-radio
reassociation median (0.577-0.589 s) is marginally FASTER than 4.2.4's
(0.669-0.708 s). The cross-radio failure cannot be attributed to general
regression.
2.HARDWARE AND SOFTWARE UNDER TEST
Access point under test
Model : Cigtech WF-660a
SoC : IPQ6018
Driver : ath11k, kernel 5.4.164
Management : rXg openwifi_wss (AP id 16652)
Address : 172.16.11.102, WAN MAC d4:ba:ba:a1:8c:00
Radio topology (the decisive property):
/sys/kernel/debug/ath11k/"ipq6018 hw1.0"/{mac0,mac1}
phy0 -> /sys/devices/platform/soc/c000000.wifi
phy1 -> /sys/devices/platform/soc/c000000.wifi
Both radios belong to ONE ath11k device and therefore share ONE peer table
(ab->peers), keyed on MAC address alone.
Radio configuration (fixed channels, not ACS, for all runs):
wlan0 / phy0 : 5 GHz, channel 149 (5745 MHz), 40 MHz, 27 dBm,
BSSID d4:ba:ba:a1:8c:01
wlan1 / phy1 : 2.4 GHz, channel 6 (2437 MHz), 20 MHz, 12 dBm,
BSSID d4:ba:ba:a1:8c:02
SSID "OW mPSK WPA2" on both radios, WPA2-PSK / CCMP,
per-device PSK via RADIUS, ap_max_inactivity=300
Test client
Model : Cigtech WF-189 (ipq53xx / ath12k, OpenWrt 25.12.3)
Configuration : rgminiloader stopped, "wifi down", single managed vif
"sta0" created on the MLO-merged phy00
Client MAC : 02:03:7f:12:97:21 (identical in all runs)
Control : wpa_supplicant driven over its ctrl_interface socket
Firmware images (rXg Device Firmware records)
id 215 : rgnets-4.1.1-0.006-openwrt-ipq60xx-cig_wf660a-squashfs-sysupgrade.tar
version_string TIP-rgnets-4.1.1-0.006-8d1ba61c
md5 738f278205257ae4d6818cf7a1c1c441 / ucentral schema dc9cad9
id 212 : rgnets-4.2.4-0.011-openwrt-ipq60xx-cig_wf660a-squashfs-sysupgrade.tar
version_string TIP-rgnets-4.2.4-0.011-954918b4e
md5 46ce1782e69efc22558e16c595412773 / ucentral schema 8491119
NOTE: commit 8d1ba61c in image 215 is the same commit currently deployed in
the field at the affected site. This is not an approximation of the
customer's build; it is that build.
3.METHOD
Cross-radio trials. Each trial is an independent roam attempt from a known
starting band:
1. Park the client on the source band by pinning its BSSID and performing
DISCONNECT / SCAN / RECONNECT. NOT MEASURED. This guarantees a known
starting band and a populated BSS cache for both bands, so a failed roam
cannot bias which direction gets tested. The park step also clears
wpa_supplicant's BSSID ignore list and re-enables the network (see
section 5) so each trial gets a fair attempt.
2. Clear the BSSID pin so a roam is permitted.
3. Issue "ROAM <target BSSID>" and measure elapsed time to the
CTRL-EVENT-CONNECTED event for the target BSSID. Only this is reported.
4. Abandon after a 25 s observation window -- deliberately wider than the
pass threshold so a slow-but-successful roam stays distinguishable from
one that never completes.
Intra-radio control. There is one BSS per radio, so there is no intra-radio
roam target. The control pins the BSSID and reassociates to it, which still
performs a real peer delete and peer create, but entirely on the owning radio
where no cross-radio mismatch is possible. Measures RECONNECT ->
CTRL-EVENT-CONNECTED. This is a reassociation, not a roam, and its latencies
are NOT comparable to roam latencies.
Timing is event-driven: a second socket is ATTACHed to the wpa_supplicant
control interface and the connection event itself is timestamped. Every
protocol event is recorded with an offset from the start of the trial, so the
phase in which a transition stalls can be identified. Per-trial wall-clock
windows are recorded and correlated afterwards against the AP's own
wall-clock kernel log, giving per-trial attribution of driver events.
Outcomes: completed <=2 s, completed <=5 s, completed at all (within 25 s), or
did not complete. A roam taking 5 s or 15 s is recorded as "completed" for
completeness but is not a success in any practical sense -- a real client
abandons the attempt long before that.
4.RESULTS
4.1 CROSS-RADIO, rgnets-4.1.1-0.006 (workaround ABSENT)
2.4 GHz -> 5 GHz, 10 trials:
FAIL FAIL FAIL FAIL FAIL FAIL FAIL 4.977s FAIL FAIL
5 GHz -> 2.4 GHz, 10 trials:
8.017s 5.017s 4.907s 4.877s 4.898s 4.978s 4.967s FAIL 4.968s 5.086s
Direction Attempts Completed <=5s <=2s min/median/max
2.4 -> 5 10 1 (10%) 1 (10%) 0 (0%) 4.977 / 4.977 / 4.977 s
5 -> 2.4 10 9 (90%) 6 (60%) 0 (0%) 4.877 / 4.968 / 8.017 s
COMBINED 20 10 (50%) 7 (35%) 0 (0%)
AP-side kernel signatures:
Timeout in receiving peer delete response ... 30
manually cleaning up (workaround) ........... 0
insufficient peer entry resource ............ 0
failed to find the peer / failed to setup rx tid 0 / 0
Every failure classified as "ended_disconnected(SCANNING)": the client did
not gracefully remain on its original band, it lost the link entirely and
fell back to scanning.
4.2 CROSS-RADIO, rgnets-4.2.4-0.011 (workaround PRESENT)
2.4 GHz -> 5 GHz, 10 trials:
0.958 FAIL 0.868 0.887 0.918 0.927 0.948 1.868 0.948 0.878 (seconds)
5 GHz -> 2.4 GHz, 10 trials:
0.602 0.746 0.947 0.917 0.897 0.957 0.957 0.907 0.887 0.977
Direction Attempts Completed <=5s <=2s min/median/max
2.4 -> 5 10 9 (90%) 9 (90%) 9 (90%) 0.868 / 0.927 / 1.868 s
5 -> 2.4 10 10 (100%) 10 (100%) 10 (100%) 0.602 / 0.917 / 0.977 s
COMBINED 20 19 (95%) 19 (95%) 19 (95%)
AP-side kernel signatures:
Timeout in receiving peer delete response ... 21
manually cleaning up (workaround) ........... 21
insufficient peer entry resource ............ 6 (ar->num_peers = -1)
failed to find the peer / failed to setup rx tid 0 / 0
4.3 INTRA-RADIO CONTROL, both builds
rgnets-4.1.1-0.006:
5 GHz : 0.606 0.308 0.588 1.669 0.589 0.299 0.659 0.299 0.619 0.289 s
10/10, min 0.289 / median 0.589 / max 1.669
2.4 GHz : 0.577 0.299 0.598 0.278 0.578 0.279 0.619 0.288 0.588 0.299 s
10/10, min 0.278 / median 0.577 / max 0.619
Peer-delete timeouts across all 20: ZERO.
rgnets-4.2.4-0.011:
5 GHz : 0.376 0.799 0.339 0.749 0.369 0.739 0.359 0.758 0.379 0.708 s
10/10, min 0.339 / median 0.708 / max 0.799
2.4 GHz : 0.407 0.709 0.359 0.699 0.398 0.709 0.388 0.739 0.409 0.669 s
10/10, min 0.359 / median 0.669 / max 0.739
Peer-delete timeouts across all 20: ZERO.
Reassociation latencies alternate strictly between a fast and a slow value on
every other trial, on both radios and both builds. A deterministic two-state
cycle of this kind is client-side behaviour (most plausibly PMKSA caching
alternating between a full 4-way handshake and a fast path) and is unrelated
to the access point.
4.4 SIDE BY SIDE
Metric 4.1.1-0.006 4.2.4-0.011
Cross-radio completed 10/20 (50%) 19/20 (95%)
Cross-radio completed <=2 s 0/20 ( 0%) 19/20 (95%)
Cross-radio median (completions) ~5.0 s ~0.92 s
Cross-radio peer-delete timeouts 30 21
Cross-radio workaround cleanups 0 21
Cross-radio create failures 0 6
Intra-radio completed 20/20 (100%) 20/20 (100%)
Intra-radio median 0.577-0.589 s 0.669-0.708 s
Intra-radio peer-delete timeouts 0 0
5.CONTROLS AND ELIMINATED ALTERNATIVE EXPLANATIONS
Workaround presence verified in the shipped binary, not inferred from the
version string. Searching the installed ath11k.ko for the workaround's log
string "manually cleaning up":
4.1.1-0.006 : 0 occurrences
4.2.4-0.011 : 1 occurrence
This is the only intentional difference between the builds.
The intra-radio control is a falsification test, and it passed. 20 peer
delete/create cycles confined to one radio produced ZERO timeouts on each
build, where 20 cross-radio roams produced 30 and 21 respectively. Cross-radio
operation is therefore a NECESSARY condition for the defect, not merely
correlated with it.
4.1.1 is not a generally slower or more broken build. Its intra-radio
reassociation median (0.577-0.589 s) is marginally faster than 4.2.4's
(0.669-0.708 s), and it completed 20/20 intra-radio trials. The cross-radio
result cannot be explained by overall build regression.
Signal strength is not a factor. Measured from the client with a clean scan,
three passes per build:
4.1.1 run : 2.4 GHz -41 / -42 / -41 dBm 5 GHz -49 / -48 / -48 dBm
4.2.4 run : 2.4 GHz -41 / -40 / -39 dBm 5 GHz -46 / -46 / -46 dBm
2.4 GHz was consistently the STRONGER band at the client despite its lower
transmit power (12 dBm vs 27 dBm), and conditions were equivalent across runs.
Roam failures cannot be attributed to a weak target band.
Channels and BSSIDs were identical in all runs (channel 6 / channel 149,
BSSIDs d4:ba:ba:a1:8c:02 / d4:ba:ba:a1:8c:01), verified after each firmware
change. An earlier reflash had allowed ACS to relocate the 5 GHz radio to
channel 36; channels were pinned for all runs reported here.
Access point health was confirmed independently: hostapd reported
status=ENABLED on both radios throughout, and initial associations (as opposed
to roams) completed normally in approximately 2.4 s on 4.1.1. The AP is not
broken for new clients; it fails specifically at moving an existing client
between its own radios.
A CLIENT-SIDE TEST ARTEFACT WAS FOUND AND CORRECTED. After two consecutive
failures wpa_supplicant places both BSSIDs on its ignore list (up to 60 s) and
temp-disables the SSID (CTRL-EVENT-SSID-TEMP-DISABLED, auth_failures=2).
Subsequent trials then failed for client-side reasons rather than AP-side ones,
understating the AP. An earlier 4.1.1 run affected by this reported 0/10 in
the 2.4 -> 5 direction; after clearing the ignore list during each (unmeasured)
park step, the same test reported 1/10. All 4.1.1 figures in section 4 are
from the corrected run. The 4.2.4 runs were unaffected, having never
accumulated enough consecutive failures to trip the mechanism.
6.ROOT CAUSE
A WMI-level capture was taken on 4.1.1 (ath11k debug_mask=0x12) during a single
cross-radio roam:
[2071.414207] WMI peer delete vdev_id 1 peer_addr 02:03:7f:12:97:21
[2071.414932] peer delete resp for vdev id 1 addr 02:03:7f:12:97:21
[2074.430404] Timeout in receiving peer delete response
[2074.430447] WMI peer create vdev_id 0 peer_addr 02:03:7f:12:97:21
The firmware answered the peer-delete command in 0.73 milliseconds. The driver
nevertheless waited the full 3.016 seconds before timing out and proceeding.
The response was not merely received, it was processed: that log line is
emitted from inside ath11k_peer_delete_resp_event(). The handler resolves the
owning radio with ath11k_mac_get_ar_by_vdev_id(ab, vdev_id) and signals THAT
radio's ar->peer_delete_done. On a cross-radio roam the waiter is a different
struct ath11k -- the radio the client is moving TO -- because
ath11k_peer_create() on the new radio initiated the delete of a peer living on
the old radio's vdev. The completion signalled is never the completion being
waited on, so the wait always expires.
The same capture contains its own internal control: peer deletes at 2065.650,
2080.190 and 2080.310 were all issued for vdev_id 1 by the radio that owns
vdev 1. Each received a response in under 1 ms and NONE was followed by a
timeout. Identical firmware, identical command, identical response latency;
the only difference is whether the requesting radio owns the target vdev.
vdev mapping confirmed from association events: vdev 1 = 5 GHz (rate_caps 6f),
vdev 0 = 2.4 GHz (rate_caps 6d).
CLIENT-SIDE CONFIRMATION. The same defect is directly observable from the
client. Protocol traces of failed 4.1.1 transitions show two signatures:
2.4 -> 5 failure:
+17.583s SME: Trying to authenticate with d4:ba:ba:a1:8c:01
+17.936s Associated with d4:ba:ba:a1:8c:01
+17.936s CTRL-EVENT-DISCONNECTED bssid=d4:ba:ba:a1:8c:01 reason=7
Reason 7 is "class 3 frame received from nonassociated STA". The client
associated successfully and was rejected in the same millisecond because the
AP had no peer for it. This is the peer-table defect seen from the far end.
5 -> 2.4 failure:
+17.561s Associated with d4:ba:ba:a1:8c:02
+20.928s CTRL-EVENT-DISCONNECTED bssid=d4:ba:ba:a1:8c:02 reason=15
Reason 15 is "4-way handshake timeout", and the 3.37 s gap corresponds to the
3 s peer-delete stall: EAPOL cannot flow because the peer is not functional.
A resident's device reports these as "connected, then dropped, no internet",
which matches the reported field symptom.
PER-TRIAL ATTRIBUTION (4.2.4). Correlating AP kernel events against individual
trial windows shows every one of the 20 cross-radio trials had exactly one
timeout and one workaround cleanup -- except the single trial that failed,
which had two of each. On 4.2.4 the failure mode is therefore a DOUBLED
cross-radio timeout within one roam, exceeding what a 100 ms workaround can
absorb. The six "insufficient peer entry" create failures were distributed
across six trials, five of which completed normally, so that condition is
retried transparently and is not on its own sufficient to fail a roam.
IMPORTANT CONSEQUENCE. The upstream issue WIFI-14998 states that "the root
cause for the response timing out is in the FW". These measurements contradict
that. The firmware responds in sub-millisecond time on every observed
occasion, including on the roams that fail. The defect is driver-side: the
driver waits on the wrong object.
7.THE WORKAROUND PRESENT IN 4.2.4, AND ITS SIDE EFFECT
4.2.4-0.011 carries the mitigation from WIFI-14998 (patch a-101 / commit
fbca032f4, authored upstream 2025-09-04). It makes two changes in
ath11k_peer_delete() / ath11k_wait_for_peer_delete_done():
1. The completion wait is shortened from 3 * HZ to 100 * HZ / 1000, reducing
the window during which the AP ignores frames from the station from 3 s to
100 ms.
2. On timeout the driver no longer aborts:
if (ret && ret != -ETIMEDOUT)
return ret;
It falls through and cleans its own peer database, preventing peers from
leaking until the driver reaches its maximum.
Change 2 has a side effect. The cleanup path decrements ar->num_peers, and on
a cross-radio delete that "ar" is the wrong radio -- the same mismatch
described in section 6. The counter is decremented on a radio that never
incremented it and goes negative:
ath11k c000000.wifi: failed to create peer due to insufficient peer entry
resource in firmware ar->num_peers = -1 ar->max_num_peers = 136
ar->num_stations = 1
This occurred 6 times in 20 roams on 4.2.4 and 0 times in 20 roams on 4.1.1,
confirming it is caused BY the workaround rather than by the firmware.
The corrupted counter PERSISTS beyond the roam that caused it. An intra-radio
control run logged a further such failure despite recording ZERO peer-delete
timeouts during that run. The workaround's decrement path only executes on a
timeout, so that failure cannot have originated in the control run; it was
residual state from a preceding cross-radio run, still failing peer creates
minutes later during ordinary same-radio reassociations.
ar->num_peers is therefore not transiently wrong for the duration of a roam --
it is left in a persistently negative state on the affected radio, causing
intermittent peer-creation failures for unrelated subsequent clients until the
driver is reset. On an access point in service this presents as a slow
accumulation of unexplained association failures rather than a self-clearing
condition.
The actual fix is patch a-106 (commit 77eeda58f), which corrects the radio
resolution so the driver waits on, and accounts against, the owning radio. At
the time of writing that patch exists only on upstream main and is NOT present
in 4.2.4.
8.SCOPE: WHICH HARDWARE IS AFFECTED
The defect requires two radios sharing a single ath11k peer table.
AFFECTED : WF-188n, WF-660a and other IPQ6018 platforms -- one ath11k
device ("ipq6018 hw1.0") exposing mac0 and mac1, both phys
resolving to the same platform device, therefore one shared
peer table.
NOT AFFECTED : WF-186H (IPQ5018), which presents two separate ath11k devices
with independent peer tables, so a band change is never a
cross-radio peer operation.
This is consistent with field observations at the affected site, where symptoms
are concentrated on the WF-188n population and absent from the WF-186H
population. NOTE: the WF-186H exemption is derived from driver topology and
field correlation; it has not been measured in the lab, as no WF-186H was
available.
9.CONCLUSIONS
1. Cross-radio (band-to-band) roaming on IPQ6018 running 4.1.1-0.006 is
broken. Half of all attempts never complete, none complete within 2 s, and
the 2.4 -> 5 direction completed only once in ten attempts. Failures leave
the client with NO connection -- it loses the link and falls back to
scanning rather than remaining on its original band.
2. Initial association and intra-radio reassociation are unaffected on 4.1.1
(20/20, zero driver timeouts, latencies marginally better than 4.2.4).
This explains why the field symptom presents intermittently as "cannot
connect or stay connected" rather than as an outage: only clients that
change bands are affected, and AP-level availability monitoring sees
nothing wrong.
3. 4.2.4-0.011 largely resolves the user-visible symptom on this hardware:
19 of 20 band transitions completed, all in under 1.02 s, against 50%
outright failure on 4.1.1. Upgrading affected IPQ6018 access points from
4.1.1 to 4.2.4-0.011 is supported by these measurements. It is a
mitigation, not a cure: one transition in twenty still failed.
4. The defect is not fixed in 4.2.4. It fires on 100% of cross-radio roams in
both builds. The workaround shortens the blind period and tidies up
afterwards, and in doing so leaves a driver peer counter persistently
negative, causing intermittent peer-creation failures that outlive the roam
which caused them. a-106 remains necessary for a correct resolution.
5. The upstream characterisation of this issue as a firmware problem is not
supported by measurement. The firmware responds in 0.73 ms; the driver
waits 3 s on the wrong completion object. This should be raised upstream,
as it changes what the correct fix is.
10.LIMITATIONS
- Sample size is 10 trials per direction per build for cross-radio and 10 per
radio per build for intra-radio: 80 trials total. At n=10 per cell the
per-direction split is not statistically stable. Two independent 4.1.1
cross-radio runs both gave 50% combined completion but differed on the
split between directions (40/60 and 10/90). The combined figure is the
robust one; the direction asymmetry is indicative only, though both runs
agreed that 2.4 -> 5 is the worse direction.
- One access point and one client, in lab RF conditions at close range
(-39 to -49 dBm). Behaviour at realistic cell-edge signal levels was not
characterised.
- Roams were initiated by an explicit wpa_supplicant ROAM command rather than
by a client's own band-steering logic. The driver-level peer operation
exercised is the same, but real client roam timing and retry behaviour will
differ.
- The test client is a WF-189, whose MLO-merged radio cannot scan the opposite
band while associated. Each trial's (unmeasured) park step therefore
involves a brief link drop. This is a property of the test client, is
identical across all runs, and does not affect the measured interval.
- The peer-leak exhaustion path (driver reaching max_num_peers = 136 on 4.1.1,
where the peer database is not cleaned) was NOT exercised. Because the peer
table is keyed on MAC alone, a single client MAC can leave at most one stale
entry, so demonstrating exhaustion would require accumulating roughly 136
distinct client MACs -- judged unnecessary once the roam failure itself was
established.
- Intra-radio results are reassociations, not roams (there is one BSS per
radio, so no intra-radio roam target exists). Their latencies are not
comparable to roam latencies.
- An earlier set of measurements taken with a polling-based harness (~50 ms
resolution, no ignore-list reset, no per-trial AP correlation) has been
superseded by the event-driven results reported here and is retained
separately as ipq6018_roaming.txt.v1. The two sets should not be mixed or
averaged. Of note, the superseded run reported 20/20 for 4.2.4 cross-radio
where the better-instrumented run found 19/20.
- The WF-186H exemption in section 8 rests on driver topology and field
correlation, not on lab measurement.