Fédération Française de Tennis
Efrei Research LabParis, FranceWhatSkeleton-based action recognition
StrokeNet
- 92.35%Top-1 on Fine-Tennis10 (StrokeNet-v1), ahead of all 12 compared methods
- 83.09Macro-F1 (StrokeNet-v2), most stable across seeds (± 0.03)
- −3 ptsTop-1 lost with 70% of ball frames masked; the ball-aware baseline loses ~30
Overview
The problem
Strokes such as a backhand and a backhand volley, or a lob and a forehand, share most of their body motion and differ in small details and in what the ball does. The ball is the decisive object, but in broadcast video it is small, blurred and often undetected. Existing datasets also lacked an expert taxonomy with synchronised pose and ball observations.
What was missing
Most racket-sport recognisers fuse ball information only after the pose sequence has been encoded, so contact evidence reaches the final decision but cannot guide how motion is represented. Pooling joints early loses the per-joint detail that separates neighbouring classes. Fine-grained benchmarks also suffer from label noise and subjective clip boundaries, and clip-level splits let a model recognise the player rather than the stroke.
Architecture
Swipe to see the whole diagram →
Technical details
- Inputs: AlphaPose 2-D pose from player crops, lifted to 3-D with MotionBERT (H36M-17 joints); TrackNetV3 2-D ball trajectory with per-frame visibility. Missing ball positions are interpolated for tensors but masked as unobserved.
- v1: four Bi-Mamba-2 stages with body-part cross-attention (BP-CA) between them and late cross-attention fusion; 31.8% of its parameters (2.89M) sit in the fusion modules.
- v2: ball-derived offsets ΔB, ΔC and Δt are added to the pose-derived SSD parameters (positive steps through softplus); dynamic graph convolution and per-joint downsampling keep joints separate; a next-frame ball forecast is an auxiliary loss.
- Joint and bone streams are trained separately and ensembled at score level. AdamW, 300 epochs, batch 128, EMA 0.999, three seeds.
- BadmintonDB: a shared two-player encoder with a player-interaction head over 18 classes.
What I built
- 01
Fine-Tennis10
A multimodal benchmark of 16,153 Roland-Garros stroke clips with expert classes, Hawk-Eye boundaries and synchronised RGB, pose, mesh and ball (details below).
- 02
StrokeNet-v1
A hierarchical Bi-Mamba-2 backbone in which anatomical body-part features attend to visibility-masked ball tokens between temporal stages, then late cross-attention fusion.
- 03
StrokeNet-v2
Visibility-gated ball offsets condition the state-space operator itself (input and output projections and the discretisation step), while each joint keeps its own temporal sequence.
- 04
Robust when evidence goes missing
Both variants keep most of their accuracy when ball frames are masked, and each tolerates a different corruption: dropped joints (v2) or temporal cropping (v1).
The dataset
Fine-Tennis10: the benchmark behind StrokeNet
Fine-grained stroke recognition needed a benchmark that did not exist: expert classes that a single broadcast view can resolve, clip boundaries no annotator has to guess, and every modality synchronised on the same stroke. Fine-Tennis10 was built from professional Roland-Garros singles broadcasts, with three FFT specialists.
Expert taxonomy
Defined with a state-certified coach of the National Technical Directorate, a data scientist and an innovation lead; only distinctions visible in one broadcast view are kept.
Boundaries from Hawk-Eye
Each clip is cut from official ball events (bounces and net crossings) synchronised with the video, not from subjective annotation.
Every modality, aligned
RGB, 2-D pose (AlphaPose), lifted 3-D pose (MotionBERT), a model-derived SMPL body mesh, and the 2-D ball trajectory with per-frame visibility (TrackNetV3).
Two evaluation protocols
A class-stratified split (12,919 train / 3,234 test) and a player-disjoint split that holds out 30% of the athletes, so a model cannot win by recognising the player.
| Class | Family | Definition |
|---|---|---|
| Service | Serve | Stroke initiating the point from the service motion. |
| Forehand | Groundstroke | Post-bounce forehand groundstroke or return, excluding drop shots, lobs, volleys and overheads. |
| Backhand | Groundstroke | Post-bounce backhand groundstroke or return, excluding drop shots, lobs, volleys and overheads. |
| Drop shot | Touch shot | Forehand drop shot intended to land short with reduced ball speed. |
| Half volley | Transition play | Struck immediately after the bounce, usually from a low pickup near the net. |
| Lob | Lob | High-arc shot sent over the opponent rather than driven flat through the court. |
| Backhand drop shot | Touch shot | Backhand drop shot intended to land short with reduced ball speed. |
| Backhand volley | Net play | Backhand volley struck before the bounce, typically near the net. |
| Smash | Overhead | Overhead attacking stroke struck above shoulder or head height, usually with a downward finish. |
| Volley | Net play | Forehand volley struck before the bounce, typically near the net. |
Main results
| Types | Methods | Publication | Modality | Fine-Tennis10 (Ours) | BadmintonDB (Public) | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R | P | B | Top-1 | M-Acc | M-F1 | Top-2 | SP | PP | Top-1 | |||
| CNN | ResNet3D | CVPR 2018 | ✓ | ✗ | ✗ | 63.31±0.20 | 53.81±0.18 | 53.85±0.06 | 85.96±0.15 | – | – | – |
| PoseConv3D | CVPR 2022 | ✓ | ✓ | ✗ | 85.65±0.25 | 73.60±0.19 | 73.71±0.41 | 94.44±0.29 | – | – | – | |
| GCN | ST-GCN | AAAI 2018 | – | – | – | – | – | – | – | ✗ | ✗ | 56.02±2.52 |
| MS-G3D | CVPR 2020 | ✗ | ✓ | ✗ | 89.73±0.10 | 79.99±0.04 | 80.71±0.38 | 95.73±0.17 | – | – | – | |
| BlockGCN | CVPR 2024 | ✗ | ✓ | ✗ | 90.18±0.26 | 78.65±0.58 | 78.80±0.35 | 95.79±0.08 | ✗ | ✗ | 57.82±2.21 | |
| ProtoGCN | CVPR 2025 | ✗ | ✓ | ✗ | 90.94±0.09 | 80.26±0.17 | 80.89±0.09 | 96.34±0.11 | ✗ | ✗ | 56.56±3.74 | |
| Transformer | TimeSformer | ICML 2021 | ✓ | ✗ | ✗ | 84.66±0.10 | – | – | – | – | – | – |
| Hyperformer | arXiv 2023 | ✗ | ✓ | ✗ | 89.46±0.19 | 78.21±0.18 | 78.23±0.23 | 95.85±0.07 | – | – | – | |
| TemPose-V | CVPRW 2023 | – | – | – | – | – | – | – | ✗ | ✗ | 57.64±1.87 | |
| TemPose-PF | CVPRW 2023 | – | – | – | – | – | – | – | ✗ | ✓ | 59.86±1.58 | |
| TemPose-TF (court-pos-z) | CVPRW 2023 | ✗ | ✓ | ✓ | 91.33±0.42 | 81.23±0.87 | 81.01±0.93 | 96.15±0.02 | ✓ | ✓ | 61.99±1.69 | |
| TemPose-SF | CVPRW 2023 | ✗ | ✓ | ✓ | 91.98±0.26 | 81.59±0.75 | 81.79±0.85 | 96.68±0.05 | ✓ | ✗ | 62.04±1.07 | |
| SkateFormer | ECCV 2024 | ✗ | ✓ | ✗ | 91.33±0.20 | 81.17±0.15 | 81.81±0.31 | 96.26±0.22 | – | – | – | |
| Simba | arXiv 2024 | ✗ | ✓ | ✗ | 88.68±0.22 | 76.87±0.37 | 77.01±0.23 | 95.32±0.18 | – | – | – | |
| Mamba | TSkel-Mamba-w/o CPKD | arXiv 2025 | ✗ | ✓ | ✗ | 88.86±0.18 | 79.81±0.44 | 78.55±0.54 | 95.97±0.02 | ✗ | ✗ | 57.53±3.92* |
| StrokeNet-v1 (Ours) | – | ✗ | ✓ | ✓ | 92.35±0.25 | 81.81±0.46 | 82.48±0.40 | 96.45±0.28 | ✓ | ✗ | 63.85±0.58 | |
| StrokeNet-v2 (Ours) | – | ✗ | ✓ | ✓ | 92.08±0.11 | 82.58±0.23 | 83.09±0.03 | 96.45±0.18 | ✓ | ✗ | 62.91±1.63 | |
Fine-Tennis10 pose results use score-level ensembles of joint and bone streams. BadmintonDB baselines are published results. * TSkel-Mamba evaluated by us. R/P/B = RGB / Pose / Ball · M-Acc = Macro-Accuracy · M-F1 = Macro-F1 · SP/PP = Shuttle / Player Position · court-pos-z = player positions zeroed.
Qualitative results
Demo
Limits and next steps
Interaction is one-way: pose does not update the ball stream, and trajectory uncertainty is not modelled beyond visibility. Half volley stays the hardest class (15 test clips). Next steps are reciprocal pose–ball conditioning, the opponent through the two-player adaptation, and court geometry from CourtAlign.
Beyond this project
Human–object interaction wherever an object decides the action: ergonomics and rehabilitation, gesture recognition, any recogniser that must keep working when one sensor drops out.
Research output
Paper
StrokeNet: Structured State Space Duality for Pose-based Stroke Recognition in Racket Sports
Ahmed Jouini, Faten Chaieb-Chakchouk, Alex Loth
WACV 2027
Accepted together with the Fine-Tennis10 benchmark; proceedings forthcoming.
Dataset
Fine-Tennis10 on Hugging Face
Derived modalities (pose, mesh, ball) under CC BY-NC 4.0; RGB on gated request. DOI 10.57967/hf/11172.
DatasetContact
Interested in the work?
I'm looking for AI research engineer roles in computer vision and video understanding, ideally in sports analytics, though not only. Ready to start a new journey.
Paris, France