Dual-Attention and Adversarial Transfer Networks for Sim-to-Real Cross-Orientation Wireless Sensing

Overview

Millimeter-wave (mmWave) human activity recognition degrades significantly when the user's orientation changes relative to the sensing links, because orientation shifts distort the Doppler signature and increase intra-class variance. Existing cross-domain wireless sensing methods either synthesize target-domain samples or learn domain-invariant features, but both remain dependent on measured multi-orientation data, and labeling such data for every orientation is labor-intensive.

This project develops S2M-Sense, a simulation-to-reality platform for cross-orientation activity recognition. Its objectives are:

• Expanding single-orientation depth-camera motion capture into orientation-diverse mmWave training data with a physics-guided channel simulator.
• Extracting activity-discriminative and orientation-robust representations from dual-link Doppler spectrograms with a parallel dual-attention network.
• Closing the simulation-to-reality gap through adversarial unsupervised transfer learning that requires only a small number of unlabeled measured samples.

Target Scenarios

S2M-Sense targets indoor activity recognition for smart homes, healthcare monitoring, and human-computer interaction, where cameras raise privacy concerns and wearable devices impose a burden on users. The person may perform an activity facing any direction within the joint coverage of two near-orthogonal mmWave communication links, which provide complementary views of the same motion.

Figure 1: Indoor passive sensing with a mmWave transmitter and receiver, where users perform activities at arbitrary orientations.

Sensing is passive and reuses OFDM communication signals. At the receiver, a reference channel and a surveillance channel are processed with the cross-ambiguity function (CAF) to obtain one Doppler spectrogram per link.

Figure 2: Experimental layout of the dual-link 60 GHz passive sensing system.

System Architecture

S2M-Sense consists of four components: channel-simulator-driven data generation, measured Doppler capture, dual-attention feature extraction, and simulation-to-reality transfer learning.

Figure 3: Architecture of S2M-Sense, combining the channel simulator, measured Doppler capture, the dual-attention recognition network, and the transfer learning network.

1. Channel Simulator: A depth camera estimates 34 body keypoints per frame, and adjacent keypoints are connected by ellipsoidal primitives. Under the quasi-static assumption, the channel impulse response (CIR) is $h(t,\tau)=u(t,\tau)+v(\tau)$, where $u(t,\tau)$ is the target-related component and $v(\tau)$ captures line-of-sight and static-scatterer paths. The contribution of the $k$-th primitive is

   $$u_k(t,\tau)=\lambda\sqrt{\frac{\sigma_k(t)\,G_k^{t}(t)\,G_k^{r}(t)}{(4\pi)^3\left(R_k^{t}(t)\,R_k^{r}(t)\right)^2}}\;e^{-j\phi_k(t)}\,\delta\big(\tau-\tau_k(t)\big),$$

   where the angle-dependent antenna gains follow the array factor of the Sivers uniform linear array. Motion sequences are refined by cubic spline interpolation and a One-Euro filter.

2. Orientation Expansion: Multi-orientation data are generated by rotating all keypoints by an angle $\alpha$ about a vertical axis through the waist keypoint, using a homogeneous transformation $\mathbf{T}(\alpha)$. The CIR is then recomputed for each target orientation, so a single recorded motion yields training samples over the full azimuth range.

3. Dual-Attention Network: Spectrograms from the two links are encoded by ResNet-18 into $\mathbf{A}_1,\mathbf{A}_2\in\mathbb{R}^{C\times H\times W}$, concatenated along the channel dimension, and batch-normalized. A position attention module captures long-range spatial dependencies, while a parallel channel attention module emphasizes activity-related channels and suppresses orientation-sensitive ones. The outputs are fused through a residual connection, $\mathbf{O}=a\cdot\mathbf{O}_p+b\cdot\mathbf{O}_c+\mathbf{A}$, where $a$ and $b$ are learnable weights initialized to zero.

4. Adversarial Unsupervised Transfer Learning: The source feature extractor $f_s$ and classifier are pre-trained on labeled simulated data. A target extractor $f_t$, initialized from $f_s$, is fine-tuned against a domain discriminator $D$ using unlabeled measured samples, alternating between

   $$\min_{D}\;-\mathbb{E}_{x_s}\left[\log D(f_s(x_s))\right]-\mathbb{E}_{x_t}\left[\log\left(1-D(f_t(x_t))\right)\right] \quad\text{and}\quad \min_{f_t}\;-\mathbb{E}_{x_t}\left[\log D(f_t(x_t))\right].$$

Figure 4: Dual-link feature extraction and concatenation, the parallel channel and position attention modules, and adversarial unsupervised transfer learning for activity classification.

Experimental Evaluation

The platform was evaluated on four activities (step, fall, sit, and kick) using a 60 GHz passive sensing testbed. Simulated training data and measured test data come from disjoint groups of volunteers to avoid identity leakage.

Parameter / Component	Specification
Carrier Frequency	60.48 GHz
Baseband Platform	NI USRP-2954R
RF Front End	Sivers phased-array antennas; two synchronized arrays capture reference and surveillance channels
Waveform	Training sequence with OFDM payload
Motion Capture	ZED 2i depth camera at 30 fps, interpolated to 2000 fps
Simulated Training Set	3000 spectrograms from 10 single-orientation videos of three volunteers, orientations sampled over $0^\circ$–$360^\circ$
Measured Test Set	480 spectrograms from three other volunteers at $45^\circ$, $135^\circ$, $225^\circ$, and $315^\circ$
Training	Adam, learning rate $10^{-4}$, 200 epochs, batch size 16

The simulator preserves the dominant Doppler structure of each activity, although measured spectrograms still contain clutter and amplitude distortions. Across all four activities and four test orientations, the structural similarity index measure (SSIM) between simulated and measured spectrograms ranges from 0.79 to 0.89.

Figure 5: Doppler spectrograms from the experiment (left) and the channel simulator (right) for falling down, forward kicking, sitting down, and single step.

All baselines were retrained on four simulated datasets: single-orientation, Link 1 with multi-orientation, Link 2 with multi-orientation, and dual-link with multi-orientation, and evaluated on the same measured test set. The proposed method outperforms WiDual, WiGRUNT, and ImgFi on all four datasets, with the best result obtained on the dual-link multi-orientation dataset.

Figure 6: Recognition accuracy of the proposed method and cross-domain baselines trained on four types of simulated datasets.

• Simulation Fidelity: Simulated and measured Doppler spectrograms achieve an average SSIM of 0.84 across 4 activities and 4 orientations.
• Simulation-Only Recognition: Training only on the dual-link multi-orientation simulated dataset yields 88.33% accuracy on measured data, which rises to 95% after adversarial transfer with as few as 16 unlabeled measured samples.
• Baseline Comparison: The 95% accuracy exceeds the best results of WiDual (78.96%), WiGRUNT (81.04%), and ImgFi (84.38%).