================================================================================ IPQ6018 CROSS-RADIO ROAMING TEST REPORT Band-transition failure on ath11k (IPQ6018) and its behaviour across rgnets-4.1.1-0.006 and rgnets-4.2.4-0.011 ================================================================================ Date of test : 2026-09-15 Location : RG Nets lab (rxg.factorialwebdevelopment.com) Author : Henry Haller -------------------------------------------------------------------------------- 1. SUMMARY -------------------------------------------------------------------------------- 80 trials were run against a single Cigtech WF-660a (IPQ6018 / ath11k): on each of two firmware builds, 10 cross-radio roams in each direction (2.4 -> 5 and 5 -> 2.4) and 10 intra-radio reassociations on each band as a control. The test client, channels, client MAC, RF conditions and test script were identical throughout; the only intentional difference was the firmware build. CROSS-RADIO (moving a client between the AP's two radios) rgnets-4.1.1-0.006 : 10 of 20 completed (50%). None within 2 s. Median when successful ~5.0 s. rgnets-4.2.4-0.011 : 19 of 20 completed (95%), all in under 1.02 s. INTRA-RADIO (reassociating to the same radio) -- CONTROL rgnets-4.1.1-0.006 : 20 of 20 completed. ZERO peer-delete timeouts. rgnets-4.2.4-0.011 : 20 of 20 completed. ZERO peer-delete timeouts. The defect is confined to cross-radio transitions and fires on 100% of them in BOTH builds (30 driver timeouts per 20 roams on 4.1.1, 21 per 20 on 4.2.4). 4.2.4-0.011 contains a workaround that shortens the driver's blind period from 3 s to 100 ms and cleans up afterwards. It does not fix the defect; it reduces a user-visible connection failure to a rare one, at the cost of corrupting a driver peer counter. Critically, 4.1.1 is NOT a generally slower build: its intra-radio reassociation median (0.577-0.589 s) is marginally FASTER than 4.2.4's (0.669-0.708 s). The cross-radio failure cannot be attributed to general regression. -------------------------------------------------------------------------------- 2. HARDWARE AND SOFTWARE UNDER TEST -------------------------------------------------------------------------------- Access point under test Model : Cigtech WF-660a SoC : IPQ6018 Driver : ath11k, kernel 5.4.164 Management : rXg openwifi_wss (AP id 16652) Address : 172.16.11.102, WAN MAC d4:ba:ba:a1:8c:00 Radio topology (the decisive property): /sys/kernel/debug/ath11k/"ipq6018 hw1.0"/{mac0,mac1} phy0 -> /sys/devices/platform/soc/c000000.wifi phy1 -> /sys/devices/platform/soc/c000000.wifi Both radios belong to ONE ath11k device and therefore share ONE peer table (ab->peers), keyed on MAC address alone. Radio configuration (fixed channels, not ACS, for all runs): wlan0 / phy0 : 5 GHz, channel 149 (5745 MHz), 40 MHz, 27 dBm, BSSID d4:ba:ba:a1:8c:01 wlan1 / phy1 : 2.4 GHz, channel 6 (2437 MHz), 20 MHz, 12 dBm, BSSID d4:ba:ba:a1:8c:02 SSID "OW mPSK WPA2" on both radios, WPA2-PSK / CCMP, per-device PSK via RADIUS, ap_max_inactivity=300 Test client Model : Cigtech WF-189 (ipq53xx / ath12k, OpenWrt 25.12.3) Configuration : rgminiloader stopped, "wifi down", single managed vif "sta0" created on the MLO-merged phy00 Client MAC : 02:03:7f:12:97:21 (identical in all runs) Control : wpa_supplicant driven over its ctrl_interface socket Firmware images (rXg Device Firmware records) id 215 : rgnets-4.1.1-0.006-openwrt-ipq60xx-cig_wf660a-squashfs-sysupgrade.tar version_string TIP-rgnets-4.1.1-0.006-8d1ba61c md5 738f278205257ae4d6818cf7a1c1c441 / ucentral schema dc9cad9 id 212 : rgnets-4.2.4-0.011-openwrt-ipq60xx-cig_wf660a-squashfs-sysupgrade.tar version_string TIP-rgnets-4.2.4-0.011-954918b4e md5 46ce1782e69efc22558e16c595412773 / ucentral schema 8491119 NOTE: commit 8d1ba61c in image 215 is the same commit currently deployed in the field at the affected site. This is not an approximation of the customer's build; it is that build. -------------------------------------------------------------------------------- 3. METHOD -------------------------------------------------------------------------------- Cross-radio trials. Each trial is an independent roam attempt from a known starting band: 1. Park the client on the source band by pinning its BSSID and performing DISCONNECT / SCAN / RECONNECT. NOT MEASURED. This guarantees a known starting band and a populated BSS cache for both bands, so a failed roam cannot bias which direction gets tested. The park step also clears wpa_supplicant's BSSID ignore list and re-enables the network (see section 5) so each trial gets a fair attempt. 2. Clear the BSSID pin so a roam is permitted. 3. Issue "ROAM " and measure elapsed time to the CTRL-EVENT-CONNECTED event for the target BSSID. Only this is reported. 4. Abandon after a 25 s observation window -- deliberately wider than the pass threshold so a slow-but-successful roam stays distinguishable from one that never completes. Intra-radio control. There is one BSS per radio, so there is no intra-radio roam target. The control pins the BSSID and reassociates to it, which still performs a real peer delete and peer create, but entirely on the owning radio where no cross-radio mismatch is possible. Measures RECONNECT -> CTRL-EVENT-CONNECTED. This is a reassociation, not a roam, and its latencies are NOT comparable to roam latencies. Timing is event-driven: a second socket is ATTACHed to the wpa_supplicant control interface and the connection event itself is timestamped. Every protocol event is recorded with an offset from the start of the trial, so the phase in which a transition stalls can be identified. Per-trial wall-clock windows are recorded and correlated afterwards against the AP's own wall-clock kernel log, giving per-trial attribution of driver events. Outcomes: completed <=2 s, completed <=5 s, completed at all (within 25 s), or did not complete. A roam taking 5 s or 15 s is recorded as "completed" for completeness but is not a success in any practical sense -- a real client abandons the attempt long before that. -------------------------------------------------------------------------------- 4. RESULTS -------------------------------------------------------------------------------- 4.1 CROSS-RADIO, rgnets-4.1.1-0.006 (workaround ABSENT) 2.4 GHz -> 5 GHz, 10 trials: FAIL FAIL FAIL FAIL FAIL FAIL FAIL 4.977s FAIL FAIL 5 GHz -> 2.4 GHz, 10 trials: 8.017s 5.017s 4.907s 4.877s 4.898s 4.978s 4.967s FAIL 4.968s 5.086s Direction Attempts Completed <=5s <=2s min/median/max 2.4 -> 5 10 1 (10%) 1 (10%) 0 (0%) 4.977 / 4.977 / 4.977 s 5 -> 2.4 10 9 (90%) 6 (60%) 0 (0%) 4.877 / 4.968 / 8.017 s COMBINED 20 10 (50%) 7 (35%) 0 (0%) AP-side kernel signatures: Timeout in receiving peer delete response ... 30 manually cleaning up (workaround) ........... 0 insufficient peer entry resource ............ 0 failed to find the peer / failed to setup rx tid 0 / 0 Every failure classified as "ended_disconnected(SCANNING)": the client did not gracefully remain on its original band, it lost the link entirely and fell back to scanning. 4.2 CROSS-RADIO, rgnets-4.2.4-0.011 (workaround PRESENT) 2.4 GHz -> 5 GHz, 10 trials: 0.958 FAIL 0.868 0.887 0.918 0.927 0.948 1.868 0.948 0.878 (seconds) 5 GHz -> 2.4 GHz, 10 trials: 0.602 0.746 0.947 0.917 0.897 0.957 0.957 0.907 0.887 0.977 Direction Attempts Completed <=5s <=2s min/median/max 2.4 -> 5 10 9 (90%) 9 (90%) 9 (90%) 0.868 / 0.927 / 1.868 s 5 -> 2.4 10 10 (100%) 10 (100%) 10 (100%) 0.602 / 0.917 / 0.977 s COMBINED 20 19 (95%) 19 (95%) 19 (95%) AP-side kernel signatures: Timeout in receiving peer delete response ... 21 manually cleaning up (workaround) ........... 21 insufficient peer entry resource ............ 6 (ar->num_peers = -1) failed to find the peer / failed to setup rx tid 0 / 0 4.3 INTRA-RADIO CONTROL, both builds rgnets-4.1.1-0.006: 5 GHz : 0.606 0.308 0.588 1.669 0.589 0.299 0.659 0.299 0.619 0.289 s 10/10, min 0.289 / median 0.589 / max 1.669 2.4 GHz : 0.577 0.299 0.598 0.278 0.578 0.279 0.619 0.288 0.588 0.299 s 10/10, min 0.278 / median 0.577 / max 0.619 Peer-delete timeouts across all 20: ZERO. rgnets-4.2.4-0.011: 5 GHz : 0.376 0.799 0.339 0.749 0.369 0.739 0.359 0.758 0.379 0.708 s 10/10, min 0.339 / median 0.708 / max 0.799 2.4 GHz : 0.407 0.709 0.359 0.699 0.398 0.709 0.388 0.739 0.409 0.669 s 10/10, min 0.359 / median 0.669 / max 0.739 Peer-delete timeouts across all 20: ZERO. Reassociation latencies alternate strictly between a fast and a slow value on every other trial, on both radios and both builds. A deterministic two-state cycle of this kind is client-side behaviour (most plausibly PMKSA caching alternating between a full 4-way handshake and a fast path) and is unrelated to the access point. 4.4 SIDE BY SIDE Metric 4.1.1-0.006 4.2.4-0.011 Cross-radio completed 10/20 (50%) 19/20 (95%) Cross-radio completed <=2 s 0/20 ( 0%) 19/20 (95%) Cross-radio median (completions) ~5.0 s ~0.92 s Cross-radio peer-delete timeouts 30 21 Cross-radio workaround cleanups 0 21 Cross-radio create failures 0 6 Intra-radio completed 20/20 (100%) 20/20 (100%) Intra-radio median 0.577-0.589 s 0.669-0.708 s Intra-radio peer-delete timeouts 0 0 -------------------------------------------------------------------------------- 5. CONTROLS AND ELIMINATED ALTERNATIVE EXPLANATIONS -------------------------------------------------------------------------------- Workaround presence verified in the shipped binary, not inferred from the version string. Searching the installed ath11k.ko for the workaround's log string "manually cleaning up": 4.1.1-0.006 : 0 occurrences 4.2.4-0.011 : 1 occurrence This is the only intentional difference between the builds. The intra-radio control is a falsification test, and it passed. 20 peer delete/create cycles confined to one radio produced ZERO timeouts on each build, where 20 cross-radio roams produced 30 and 21 respectively. Cross-radio operation is therefore a NECESSARY condition for the defect, not merely correlated with it. 4.1.1 is not a generally slower or more broken build. Its intra-radio reassociation median (0.577-0.589 s) is marginally faster than 4.2.4's (0.669-0.708 s), and it completed 20/20 intra-radio trials. The cross-radio result cannot be explained by overall build regression. Signal strength is not a factor. Measured from the client with a clean scan, three passes per build: 4.1.1 run : 2.4 GHz -41 / -42 / -41 dBm 5 GHz -49 / -48 / -48 dBm 4.2.4 run : 2.4 GHz -41 / -40 / -39 dBm 5 GHz -46 / -46 / -46 dBm 2.4 GHz was consistently the STRONGER band at the client despite its lower transmit power (12 dBm vs 27 dBm), and conditions were equivalent across runs. Roam failures cannot be attributed to a weak target band. Channels and BSSIDs were identical in all runs (channel 6 / channel 149, BSSIDs d4:ba:ba:a1:8c:02 / d4:ba:ba:a1:8c:01), verified after each firmware change. An earlier reflash had allowed ACS to relocate the 5 GHz radio to channel 36; channels were pinned for all runs reported here. Access point health was confirmed independently: hostapd reported status=ENABLED on both radios throughout, and initial associations (as opposed to roams) completed normally in approximately 2.4 s on 4.1.1. The AP is not broken for new clients; it fails specifically at moving an existing client between its own radios. A CLIENT-SIDE TEST ARTEFACT WAS FOUND AND CORRECTED. After two consecutive failures wpa_supplicant places both BSSIDs on its ignore list (up to 60 s) and temp-disables the SSID (CTRL-EVENT-SSID-TEMP-DISABLED, auth_failures=2). Subsequent trials then failed for client-side reasons rather than AP-side ones, understating the AP. An earlier 4.1.1 run affected by this reported 0/10 in the 2.4 -> 5 direction; after clearing the ignore list during each (unmeasured) park step, the same test reported 1/10. All 4.1.1 figures in section 4 are from the corrected run. The 4.2.4 runs were unaffected, having never accumulated enough consecutive failures to trip the mechanism. -------------------------------------------------------------------------------- 6. ROOT CAUSE -------------------------------------------------------------------------------- A WMI-level capture was taken on 4.1.1 (ath11k debug_mask=0x12) during a single cross-radio roam: [2071.414207] WMI peer delete vdev_id 1 peer_addr 02:03:7f:12:97:21 [2071.414932] peer delete resp for vdev id 1 addr 02:03:7f:12:97:21 [2074.430404] Timeout in receiving peer delete response [2074.430447] WMI peer create vdev_id 0 peer_addr 02:03:7f:12:97:21 The firmware answered the peer-delete command in 0.73 milliseconds. The driver nevertheless waited the full 3.016 seconds before timing out and proceeding. The response was not merely received, it was processed: that log line is emitted from inside ath11k_peer_delete_resp_event(). The handler resolves the owning radio with ath11k_mac_get_ar_by_vdev_id(ab, vdev_id) and signals THAT radio's ar->peer_delete_done. On a cross-radio roam the waiter is a different struct ath11k -- the radio the client is moving TO -- because ath11k_peer_create() on the new radio initiated the delete of a peer living on the old radio's vdev. The completion signalled is never the completion being waited on, so the wait always expires. The same capture contains its own internal control: peer deletes at 2065.650, 2080.190 and 2080.310 were all issued for vdev_id 1 by the radio that owns vdev 1. Each received a response in under 1 ms and NONE was followed by a timeout. Identical firmware, identical command, identical response latency; the only difference is whether the requesting radio owns the target vdev. vdev mapping confirmed from association events: vdev 1 = 5 GHz (rate_caps 6f), vdev 0 = 2.4 GHz (rate_caps 6d). CLIENT-SIDE CONFIRMATION. The same defect is directly observable from the client. Protocol traces of failed 4.1.1 transitions show two signatures: 2.4 -> 5 failure: +17.583s SME: Trying to authenticate with d4:ba:ba:a1:8c:01 +17.936s Associated with d4:ba:ba:a1:8c:01 +17.936s CTRL-EVENT-DISCONNECTED bssid=d4:ba:ba:a1:8c:01 reason=7 Reason 7 is "class 3 frame received from nonassociated STA". The client associated successfully and was rejected in the same millisecond because the AP had no peer for it. This is the peer-table defect seen from the far end. 5 -> 2.4 failure: +17.561s Associated with d4:ba:ba:a1:8c:02 +20.928s CTRL-EVENT-DISCONNECTED bssid=d4:ba:ba:a1:8c:02 reason=15 Reason 15 is "4-way handshake timeout", and the 3.37 s gap corresponds to the 3 s peer-delete stall: EAPOL cannot flow because the peer is not functional. A resident's device reports these as "connected, then dropped, no internet", which matches the reported field symptom. PER-TRIAL ATTRIBUTION (4.2.4). Correlating AP kernel events against individual trial windows shows every one of the 20 cross-radio trials had exactly one timeout and one workaround cleanup -- except the single trial that failed, which had two of each. On 4.2.4 the failure mode is therefore a DOUBLED cross-radio timeout within one roam, exceeding what a 100 ms workaround can absorb. The six "insufficient peer entry" create failures were distributed across six trials, five of which completed normally, so that condition is retried transparently and is not on its own sufficient to fail a roam. IMPORTANT CONSEQUENCE. The upstream issue WIFI-14998 states that "the root cause for the response timing out is in the FW". These measurements contradict that. The firmware responds in sub-millisecond time on every observed occasion, including on the roams that fail. The defect is driver-side: the driver waits on the wrong object. -------------------------------------------------------------------------------- 7. THE WORKAROUND PRESENT IN 4.2.4, AND ITS SIDE EFFECT -------------------------------------------------------------------------------- 4.2.4-0.011 carries the mitigation from WIFI-14998 (patch a-101 / commit fbca032f4, authored upstream 2025-09-04). It makes two changes in ath11k_peer_delete() / ath11k_wait_for_peer_delete_done(): 1. The completion wait is shortened from 3 * HZ to 100 * HZ / 1000, reducing the window during which the AP ignores frames from the station from 3 s to 100 ms. 2. On timeout the driver no longer aborts: if (ret && ret != -ETIMEDOUT) return ret; It falls through and cleans its own peer database, preventing peers from leaking until the driver reaches its maximum. Change 2 has a side effect. The cleanup path decrements ar->num_peers, and on a cross-radio delete that "ar" is the wrong radio -- the same mismatch described in section 6. The counter is decremented on a radio that never incremented it and goes negative: ath11k c000000.wifi: failed to create peer due to insufficient peer entry resource in firmware ar->num_peers = -1 ar->max_num_peers = 136 ar->num_stations = 1 This occurred 6 times in 20 roams on 4.2.4 and 0 times in 20 roams on 4.1.1, confirming it is caused BY the workaround rather than by the firmware. The corrupted counter PERSISTS beyond the roam that caused it. An intra-radio control run logged a further such failure despite recording ZERO peer-delete timeouts during that run. The workaround's decrement path only executes on a timeout, so that failure cannot have originated in the control run; it was residual state from a preceding cross-radio run, still failing peer creates minutes later during ordinary same-radio reassociations. ar->num_peers is therefore not transiently wrong for the duration of a roam -- it is left in a persistently negative state on the affected radio, causing intermittent peer-creation failures for unrelated subsequent clients until the driver is reset. On an access point in service this presents as a slow accumulation of unexplained association failures rather than a self-clearing condition. The actual fix is patch a-106 (commit 77eeda58f), which corrects the radio resolution so the driver waits on, and accounts against, the owning radio. At the time of writing that patch exists only on upstream main and is NOT present in 4.2.4. -------------------------------------------------------------------------------- 8. SCOPE: WHICH HARDWARE IS AFFECTED -------------------------------------------------------------------------------- The defect requires two radios sharing a single ath11k peer table. AFFECTED : WF-188n, WF-660a and other IPQ6018 platforms -- one ath11k device ("ipq6018 hw1.0") exposing mac0 and mac1, both phys resolving to the same platform device, therefore one shared peer table. NOT AFFECTED : WF-186H (IPQ5018), which presents two separate ath11k devices with independent peer tables, so a band change is never a cross-radio peer operation. This is consistent with field observations at the affected site, where symptoms are concentrated on the WF-188n population and absent from the WF-186H population. NOTE: the WF-186H exemption is derived from driver topology and field correlation; it has not been measured in the lab, as no WF-186H was available. -------------------------------------------------------------------------------- 9. CONCLUSIONS -------------------------------------------------------------------------------- 1. Cross-radio (band-to-band) roaming on IPQ6018 running 4.1.1-0.006 is broken. Half of all attempts never complete, none complete within 2 s, and the 2.4 -> 5 direction completed only once in ten attempts. Failures leave the client with NO connection -- it loses the link and falls back to scanning rather than remaining on its original band. 2. Initial association and intra-radio reassociation are unaffected on 4.1.1 (20/20, zero driver timeouts, latencies marginally better than 4.2.4). This explains why the field symptom presents intermittently as "cannot connect or stay connected" rather than as an outage: only clients that change bands are affected, and AP-level availability monitoring sees nothing wrong. 3. 4.2.4-0.011 largely resolves the user-visible symptom on this hardware: 19 of 20 band transitions completed, all in under 1.02 s, against 50% outright failure on 4.1.1. Upgrading affected IPQ6018 access points from 4.1.1 to 4.2.4-0.011 is supported by these measurements. It is a mitigation, not a cure: one transition in twenty still failed. 4. The defect is not fixed in 4.2.4. It fires on 100% of cross-radio roams in both builds. The workaround shortens the blind period and tidies up afterwards, and in doing so leaves a driver peer counter persistently negative, causing intermittent peer-creation failures that outlive the roam which caused them. a-106 remains necessary for a correct resolution. 5. The upstream characterisation of this issue as a firmware problem is not supported by measurement. The firmware responds in 0.73 ms; the driver waits 3 s on the wrong completion object. This should be raised upstream, as it changes what the correct fix is. -------------------------------------------------------------------------------- 10. LIMITATIONS -------------------------------------------------------------------------------- - Sample size is 10 trials per direction per build for cross-radio and 10 per radio per build for intra-radio: 80 trials total. At n=10 per cell the per-direction split is not statistically stable. Two independent 4.1.1 cross-radio runs both gave 50% combined completion but differed on the split between directions (40/60 and 10/90). The combined figure is the robust one; the direction asymmetry is indicative only, though both runs agreed that 2.4 -> 5 is the worse direction. - One access point and one client, in lab RF conditions at close range (-39 to -49 dBm). Behaviour at realistic cell-edge signal levels was not characterised. - Roams were initiated by an explicit wpa_supplicant ROAM command rather than by a client's own band-steering logic. The driver-level peer operation exercised is the same, but real client roam timing and retry behaviour will differ. - The test client is a WF-189, whose MLO-merged radio cannot scan the opposite band while associated. Each trial's (unmeasured) park step therefore involves a brief link drop. This is a property of the test client, is identical across all runs, and does not affect the measured interval. - The peer-leak exhaustion path (driver reaching max_num_peers = 136 on 4.1.1, where the peer database is not cleaned) was NOT exercised. Because the peer table is keyed on MAC alone, a single client MAC can leave at most one stale entry, so demonstrating exhaustion would require accumulating roughly 136 distinct client MACs -- judged unnecessary once the roam failure itself was established. - Intra-radio results are reassociations, not roams (there is one BSS per radio, so no intra-radio roam target exists). Their latencies are not comparable to roam latencies. - An earlier set of measurements taken with a polling-based harness (~50 ms resolution, no ignore-list reset, no per-trial AP correlation) has been superseded by the event-driven results reported here and is retained separately as ipq6018_roaming.txt.v1. The two sets should not be mixed or averaged. Of note, the superseded run reported 20/20 for 4.2.4 cross-radio where the better-instrumented run found 19/20. - The WF-186H exemption in section 8 rests on driver topology and field correlation, not on lab measurement. ================================================================================