Guidance on deploying Isaac Lab-trained policy to Isaac Sim

[smtbhd32-ABX]

Problem

A locomotion policy trained in Isaac Lab using the DeeproboticsLite3RoughEnvCfg config runs stably in Isaac Lab and also stably in MuJoCo, but becomes severely unstable (jittering, falling immediately) when deployed into Isaac Sim via our ROS2 bridge.

The bridge handles:

• Sensor streaming (IMU, joint states) from Isaac Sim → policy node
• Action commands from policy node → Isaac Sim articulation controller
• It follows the architecture described in the Isaac Sim ROS2 RL Controller tutorial

---

Environment

Component	Detail
Robot	DeepRobotics Lite3
Training framework	Isaac Lab (forked from DeepRoboticsLab/rl_training)
Deployment sim	Isaac Sim 5.1.0
Bridge	Custom ROS2 bridge (isaac_bridge)
ROS2 distro	Humble
Policy	DeeproboticsLite3RoughEnvCfg base config

Repos:

• Training fork: https://github.com/Airbotix-Technology-Pvt-Ltd/Lite3_rl_training
• Deployment SDK: https://github.com/Airbotix-Technology-Pvt-Ltd/sdk_deploy

---

Any guidance will be appreciated, thanks in advance

[smtbhd32-ABX]

Update — suspected issue: mesh terrain in training environment

After further investigation, I believe the root cause of the instability is an environment mismatch. The policy was trained on Isaac Lab's procedural geometric terrain, but the Isaac Sim deployment scene (demo3.usd) uses mesh-based geometry (warehouse floor, stairs, slopes).

To close this gap, I replaced Isaac Lab's geometric terrain with my USD mesh terrain for training. This introduced two new problems:

1. Performance degradation
Mesh terrain is significantly heavier than procedural terrain. I can now only run ~512 parallel environments before hitting memory limits, down from 4096. Training is much slower as a result and the mean reward stays negative — the policy is not converging.

2. Invalid spawn positions
Robots sometimes initialize inside the mesh geometry or on unstable surfaces (edges of stairs, sloped faces). This causes broken physics from the very start of an episode — robots are stuck or behaving erratically before the policy takes a single step.

---

For context: a flat-ground policy trained with the default config converges well and transfers stably into Isaac Sim. The problem is specific to stairs and slopes.

Any guidance on the recommended workflow for this scenario would be appreciated.

[smtbhd32-ABX]

Context — what we are trying to achieve

We are targeting autonomous navigation of the Lite3 robot in a warehouse environment. The robot needs to walk gracefully over warehouse floors, stairs, and slopes as part of a larger autonomy stack (Nav2 + SLAM).

For locomotion, we chose RL training because it handles uneven terrain better than traditional controllers. For deployment, we are using the official sdk_deploy from DeepRobotics specifically because the same setup transfers directly to the real robot without changing the software stack.

The current blocker is getting a stable locomotion policy for non-flat terrain. Flat ground works end-to-end in sim and on the real robot.

[bopeng5221]

I think your ros2 bridge problem is not caused by terrains. I think it's caused by mismatch of joint orders, scaling, observation/action order. This is a very careful step and you need to make sure everything is correct. Please double check, just look at our training and deploying code very carefully and compare the difference with your code.

[smtbhd32-ABX]

It’s not that the robot doesn’t work at all — the issue is that it jitters.

To give more context: when I use the default Isaac Lab terrain generator (your RL training code), I observe jitter in the robot’s DOF response — it shows noticeable vibrations. The same training code works correctly in Isaac Lab and MuJoCo (including stairs, slopes, and flat terrain), but in Isaac Sim the robot jitters even on flat ground and falls on stairs.

To isolate this, I ran a controlled test using my own USD:

I trained on a simple flat-ground USD (only a ground plane, lightweight).
Training converged properly.
The robot walks smoothly with zero jitter.

However, when I tried the same USD-import approach with a stair terrain:

Training became very slow
The policy did not converge
The setup became significantly heavier computationally

So while the flat-ground case works well, I’m not able to replicate the same result for stairs.

Because the flat-ground experiment behaves correctly, I believe the API and pipeline are being used properly. I suspect the issue is related to how the policy interacts with different terrain representations — specifically, a difference between imported USD ground meshes and the default terrain generator.

As suggested, I will go through my code again thoroughly and check for any issues on my side. I’ll follow up with more detailed findings once I’ve verified everything.

Thanks again for your response — your tutorials have been very helpful

[bopeng5221]

Thank you for your thoughtful response. It seems you are taking this very seriously and have been trying hard to get this working. I'm glad to help you. For jitters, I would need more specific information like videos of different scenarios on different angles to analyze the problem. Since this doesn't involve real robot deployment, this has nothing to do with sim-to-real transfer. One possible reason is that the collision mesh and visualization mesh of your own usd is different and this leads to unreasonable jitterring. You can join our discord channel for faster response and easier communication. We can even mmake a short phone call within discord

[smtbhd32-ABX]

Hi @bopeng5221, thanks a lot for the kind and encouraging response — I really appreciate it.

I’ve joined the Discord channel and recorded videos from different scenarios and angles. Could you please let me know the best place to share them for your review?

Regarding your point about a possible mismatch between visualization and collision meshes — could you suggest how to systematically debug or verify this in Isaac Sim? Any specific tools, checks, or recommended workflow would be very helpful.

For reference, here are the components I’m currently using:

• ROS2 bridge (UDP ↔ Isaac ROS messages):
  https://github.com/Airbotix-Technology-Pvt-Ltd/sdk_deploy/tree/main/src/isaac_bridge/src

• USD environment used in Isaac Sim:
  https://github.com/Airbotix-Technology-Pvt-Ltd/sdk_deploy/blob/main/src/isaac_bridge/isaacsim/demo3.usd

This bridge is what connects Isaac Sim with the SDK, so both systems can communicate during deployment.

I’ve also attached a screenshot of the current setup for context.

Thanks again for your support!