GRASP: Generalizable Robotic Adaptation for Sim-to-Real Plant Interaction
Krishna Muvva5, Kunjan Theo Joseph2, Soumik Sarkar2, Santosh Pitla1, Yeyin Shi1, Nikhil Satyala1
University of Nebraska–Lincoln
Iowa State University
Osnabrück University of Applied Sciences
Arizona State University
University of Nevada, Reno
arXiv Page (coming soon)
GitHub Repo
Dataset & Weights
Assembly CAD files
ResultsTL;DR: GRASP replicates the agronomist's maize push test through a hierarchical perception–interaction–inference pipeline. A YOLO segmentation model, trained on SAM-3-generated labels from real field imagery, identifies the stalk; a classical IK-based controller then guides the manipulator through approach and alignment. A recurrent neural network (RNN), trained on interaction data collected in a GPU-accelerated deformable-body simulation with randomized FEM plant mechanics, infers soft/hard stalk stiffness from proprioception alone without requiring vision at inference. The system transfers to a real xArm6 + Bunker Pro robot setup, producing stiffness estimates that align with observed stalk lignification and maturity trends.
Introduction
Maize stalks fail from the inside. Stalk rot hollows out the stem long before visible symptoms appear, and by the time damage can be seen externally, the plant is often already lost. Current detection methods do not scale well: destructive sampling damages the plant, while hand-held push meters require a person to walk every row.
GRASP automates the agronomist's push test. The robot locates a maize stalk, performs a controlled diagnostic push, and estimates stalk stiffness from the resulting mechanical response.
The challenge is not simply making contact—it is producing a consistent and informative interaction. A useful measurement requires pushing at the correct height and direction while generating enough deflection to reveal stiffness differences without damaging the plant. To study this process, we built a GPU-accelerated simulation environment in NVIDIA Isaac Lab using deformable maize models with randomized FEM stiffness properties spanning more than an order of magnitude. This environment was used to generate large-scale interaction data and evaluate robot–plant contact strategies under diverse plant conditions.
The complete pipeline transfers to a real xArm6 mounted on an AgileX Bunker Pro platform, producing stiffness estimates that align with observed differences in stalk maturity and lignification.
Key Contributions
- GRASP Dataset — 21,541 field images auto-annotated through a SAM-3 → YOLOv8/v11/v26-seg pipeline, enabling zero-shot stalk segmentation with no manual labeling and real-time inference at 30 Hz on edge hardware.
- GRASP Platform — A UFactory xArm6 manipulator mounted on an AgileX Bunker Pro mobile base, paired with a ZED X stereo camera for real-time 3D stalk localization and autonomous push execution under field conditions.
- GRASP Framework — A sim-to-real robotic pipeline that locates maize stalks, performs controlled diagnostic pushes, and estimates stalk stiffness directly from proprioceptive interaction data.
Methodology
Hardware
The GRASP robot is a UFactory xArm6 (6-DOF, 5 kg payload, ±0.1 mm repeatability) mounted on an AgileX Bunker Pro tracked mobile base. A ZED X stereo camera provides RGB-D perception; a gripper-mounted depth camera provides close-range feedback for contact control.
Camera Mounting — drag to rotate and explore the full CAD file
Perception Pipeline
SAM 3 zero-shot annotations are used to train a real-time YOLOv8-seg model running at 30 Hz. Drag each slider to compare input and output.
Diverse Dataset used for training
SAM 3 Masks Case 1 and 2
3D Point Cloud
YOLOv11n vs YOLOv11l — stalk detection overlay
Isaac Lab Setup
Deformable maize plant models with FEM physics are placed in GPU-parallel Isaac Lab environments. Stalk stiffness is randomised log-uniformly over [5×10⁷, 10⁸] Pa each episode.
Maize Plant / Plant Mesh
Plant Mesh / Textured Render
Raw RGB / Segmentation
Scene Layout / Scene Variant
Isaac Lab Training
Results
The GRASP policy was evaluated both in simulation and on the physical robot across a range of stalk stiffness values. Key metrics include reward convergence, stiffness estimation accuracy, and sim-to-real transfer success rate.
30 Hz
Real-time perception
YOLOv8-seg on edge hardware
10×
Stiffness range covered
5×10⁶ – 10⁸ Pa randomised
21,541
Field images 0 shot annotated
Zero manual labelling
Sim→Real
Policy transfer
xArm6 + Bunker Pro
Real plants for testing
2/12 (Yellow)
2/19 (Yellow)
3/12 (Yellow)
2/5 (Green)
3/19 (Green)
YOLO Training curves
Stiffness Estimation (proprioception vs distillation methods)
Sim-to-real performance in real time (results are shown as P(soft))
3/19 (Green)
2/19 (Yellow)
🌽 Questions and Answers
Q: 🌾 Why simulate plants instead of just collecting real push data?
Real-field data collection doesn't scale. Stiffness varies by more than an order of magnitude from plant to plant and week to week — capturing that range means walking every row with a setup, running trials on plants that can't be reused, and still ending up with data too narrow to cover edge cases. Simulation lets us randomize stiffness over the full biological range in every episode, run thousands of parallel pushes overnight, and never harm a plant.
Q: 📏 Isn't measuring stiffness just measuring force?
Force alone tells you how hard you pushed, not how stiff the plant is. Stiffness is the slope of the force–deflection curve — that is, resistance per unit of bending. Push at the wrong angle, wrong height, or too fast, and that slope becomes meaningless noise. That's why GRASP enforces a quality contact: the arm must approach frontally, engage at mid-stalk height, and sustain the push long enough to drive a measurable 20° tilt. Only then does the GRU estimator see the clean interaction sequence it needs to recover Young's modulus reliably.
Q: 🌿 Can GRASP generalize to other crops?
The method generalizes; the specific policy needs retraining. GRASP's reward design is built on Euler–Bernoulli beam theory, which holds for any approximately cantilevered stem — sorghum, sugarcane, sunflower, young trees. Adapting to a new crop means swapping the deformable plant model and re-randomizing stiffness over that species' biological range. The reward structure, control stack, and GRU estimator stay the same. The real bottleneck is vision. End-to-end policies don't generalize well to new visual scenes, so detecting the stalk and localizing the push point on an unseen crop would require retraining the segmentation model on that species. This is the most practical barrier to sim-to-real transfer.
Q: 🤖 Why Does an End-to-End Reinforcement Learning Policy Fail for Sim-to-Real Plant Interaction?
Because the task combines perception, contact, and deformable plant mechanics, the RL policy must learn an extremely long causal chain from pixels to stiffness. Small sim-to-real errors in plant geometry, material properties, contact dynamics, and camera observations compound, causing the learned policy to overfit simulation behavior. RL also requires millions of interactions, which are impractical on real plants, and the latent variable of interest (stiffness) is not directly observable from vision alone. By decomposing the problem into perception, IK-based contact, and a proprioceptive stiffness estimator, GRASP avoids the sim-to-real fragility and poor sample efficiency of end-to-end reinforcement learning.
Q: 🔧 What can't GRASP do yet?
Quite a bit. GRASP currently handles one stalk at a time; row-level throughput still requires a person to drive the platform. At the highest stiffness values tested, success rates drop — the policy hasn't fully learned to push hard enough without losing contact. The bigger open problem is outdoor deployment. Wind causes the stalk to sway continuously, which breaks the locked push-point assumption and disrupts the force–deflection signal the GRU relies on. The current system was developed and evaluated in controlled indoor conditions; making it robust to field dynamics is the most significant gap between the lab and real agricultural use.
Citation
@mastersthesis{chawla2026grasp,
title = {GRASP: Generalizable Robotic Adaptation for Sim-to-Real Plant Interaction},
author = {Chawla, Yogesh},
school = {University of Nebraska--Lincoln},
year = {2026},
month = {June},
note = {M.S. Thesis, Biological Systems Engineering / Computer Science},
advisor = {Santosh Pitla},
}