FluxVLA Engine is a full-stack, end-to-end engineering platform for deploying embodied intelligence applications. Built on the core design principles of unified configuration, standardized interfaces, module decoupling, and deployability, it creates a complete engineering loop from data to real-device deployment. With the goal of providing a standardized industry–academia–research foundation, it significantly lowers the engineering barrier for VLA research and development.
- Cabinet:- PnPBottleToCabinetClose+- PnPWineToCabinetClose.
- Drawer:- PnPCanToDrawerClose+- PnPCupToDrawerClose.
- Microwave:- PnPMilkToMicrowaveClose+- PnPPotatoToMicrowaveClose.
- Generalization: the remaining 18 post-train novel tasks.
- The RoboCasa results are evaluated with 50 trials per task.
[2026/09/09] 🔥 FluxVLA now supports checkpoint-free GPT-6 Astra evaluation on LIBERO through the OpenAI Responses API, including multi-view observations, native LIBERO action control, environment-configured credentials, and organized rollout artifacts.
[2026/09/09] 🔥 FluxVLA-native GR00T N1.7 is now open sourced, including training and inference workflows for LIBERO, RoboCasa, and Franka robots.
[2026/08/25] 🔥 FluxVLA-native DiT4DiT support is now available, including training and inference workflows for LIBERO and RoboCasa.
[2026/08/20] 🔥 FluxVLA-native Cosmos3 support is now available for Nano, Super, and Edge, including post-training and inference workflows.
[2026/08/13] 🔥 FastWAM world-action model support is now available.
[2026/08/10] 🔥 FluxVLA deployment on NVIDIA Jetson Orin is now supported, with edge inference acceleration reaching 7.4 Hz for GR00T-N1.5. See orin_flashing.md for initial flashing and orin_docker_runtime.md to start the FluxVLA Docker setup.
[2026/06/30] 🔥 Single-arm and dual-arm Franka real-robot inference is now supported, including joint/eepose control configs and a deployment guide. See docs/franka.md.
[2026/06/25] 🔥 GR00T-RTC-accelerated inference is now supported, achieving 45 Hz on an RTX 5090.
[2026/06/22] 🔥 A minimal Oli humanoid whole-body (loco-manipulation) real-robot inference path (operator + runner + example config) is now available. See docs/oli_whole_body.md.
[2026/06/17] 🔥 ARM reward modeling with RA-BC/AW-BC reweighting is now supported. See docs/arm.md for setup and usage.
[2026/06/10] 🔥 RoboCasa GR1 simulation tasks with GR00T are now supported.
[2026/06/04] 🔥 Triton backend for Pi0.5-RTC is now supported, see inference_acceleration.
[2026/05/28] 🔥 FluxDAgger is now released: a model-decoupled DAgger pipeline for dual-arm manipulation, making it easy to integrate different VLAs and reward models.
[2026/05/28] 🔥 The embodied manipulation simulation benchmark FluxBisim is now released.
[2026/05/09] 🔥 SmolVLA is now supported.
[2026/04/24] 🔥 Pi0.5-RTC is now supported.
[2026/04/22] 🔥 ZMQ-based remote inference framework is now supported.
[2026/04/15] 🔥 DreamZero WAM is now supported.
[2026/04/08] 🔥 FluxVLA has been open-sourced.
Choose one of the following installation paths:
- Recommended one-command installer: use this for normal training, simulation evaluation, and real-robot inference setups.
- Update an existing FluxVLA environment: use this if you installed an earlier FluxVLA release and only need to refresh changed packages.
- Manual installation from scratch: use this only when you need to control every package install step yourself.
conda create -n fluxvla python=3.10 -y
conda activate fluxvla
# Choose one mode: sim-only, real-only, or full.
bash scripts/install_env.sh sim-only
# bash scripts/install_env.sh real-only
# bash scripts/install_env.sh fullIf the installer has issues: check modes and CUDA profile selection
sim-only installs simulation / LIBERO / RoboCasa runtime dependencies plus
the pinned RoboCasa source checkouts under ./src, real-only installs
real-robot and remote-inference dependencies, and full installs both. Pass
--skip-robocasa if you do not need the RoboCasa checkouts.
RoboCasa simulator assets are downloaded by default whenever the installer
installs the RoboCasa source checkouts (sim-only, full, or real-only --with-robocasa). The installer calls scripts/download_robocasa_assets.py
and uses FLUXVLA_ROBOCASA_ASSET_ENDPOINT (default: HF_ENDPOINT, then
https://hf-mirror.com). Use --skip-robocasa-assets to skip only the assets,
or --skip-robocasa to skip both the source checkouts and the assets.
The installer selects a CUDA PyTorch profile automatically from the current
CUDA toolkit / nvcc version first: CUDA >= 12.8 selects cu128, otherwise it
selects cu124. If no toolkit is visible, it falls back to driver-reported
CUDA and then GPU generation. Override it with --profile cu128 or
--profile cu124.
After PyTorch is installed, the FlashAttention wheel is selected from the
actual Python tag, PyTorch version, CUDA major version, C++ ABI, and CPU
architecture. If your platform has no matching prebuilt wheel, set
FLASH_ATTN_WHEEL_URL explicitly or pass --skip-flash-attn.
av is installed from the pip wheel first by default to avoid slow conda
dependency resolution; if no wheel is available, the installer falls back to
conda. Set FLUXVLA_AV_INSTALLER=conda if you specifically want the
conda-forge package.
On Linux x86_64, the installer also installs ffmpeg=7 from conda-forge
before TorchCodec. TorchCodec needs FFmpeg shared libraries such as
libavutil.so.59; the imageio-ffmpeg executable and the PyAV wheel do not
replace those libraries. To repair an existing Torch 2.8 environment, run:
conda install -y -c conda-forge "ffmpeg=7"
python -m pip install --force-reinstall "torchcodec==0.7.0"
python -c "from torchcodec.decoders import VideoDecoder; print('TorchCodec OK')"Real-robot runners still require the system ROS installation itself. On ROS Noetic machines, source ROS before launching inference:
source /opt/ros/noetic/setup.bashIf the installer has issues: use a cached or mirrored FlashAttention wheel
FlashAttention wheels are large, so GitHub release downloads can dominate a
fresh install on slow networks. For repeated installs, put the exact wheel file
in ./wheelhouse/, ./wheels/, or ~/.cache/fluxvla/wheels/; the installer
will use it before any network request. You can also point at a local file or
an internal mirror:
FLASH_ATTN_WHEEL_FILE=/path/to/flash_attn-2.8.3.post1+cu12torch2.8cxx11abiTRUE-cp310-cp310-linux_x86_64.whl \
bash scripts/install_env.sh sim-only --profile cu128
FLASH_ATTN_WHEEL_BASE_URLS="https://your-mirror.example.com/fluxvla/wheels" \
bash scripts/install_env.sh sim-only --profile cu128If the installer has issues: customize pip mirrors and timeouts
The installer respects your existing pip configuration first. If that index is missing a package, or if no pip index is configured, it probes PyPI plus several common mirrors and retries by response time instead of pinning one mirror globally. For slow or unstable networks, customize the candidate list and timeouts:
PIP_INDEX_CANDIDATES="https://mirrors.aliyun.com/pypi/simple https://mirrors.cloud.tencent.com/pypi/simple https://pypi.tuna.tsinghua.edu.cn/simple https://pypi.org/simple" \
PIP_INSTALL_TIMEOUT=7200 \
PIP_NETWORK_TIMEOUT=900 \
GH_PROXY=https://ghfast.top \
bash scripts/install_env.sh fullIf you already cloned and installed FluxVLA(v0.1.0), you do not need to recreate the conda environment. Pull the latest code and update only the packages whose versions changed for the current simulation / model stack:
bash scripts/update_env.shUse --skip-pull if you already updated the checkout yourself, and
--skip-project if you do not want to reinstall FluxVLA in editable mode.
The updater refreshes the complete unified base dependency set, including the
DiT4DiT-compatible Diffusers revision, peft==0.19.1, and av==14.2.0.
Equivalent manual commands
git pull
conda install -y -c conda-forge "ffmpeg=7"
python -m pip install --upgrade -r requirements-base.txt
python -m pip install --upgrade --only-binary=:all: "av==14.2.0"
python -m pip install --upgrade "torchcodec==0.7.0" # Torch 2.8; use 0.2.1 for Torch 2.6
python -m pip install "mujoco==3.2.6" gymnasium lxml bddl==1.0.1 hydra-core==1.2.0 robomimic==0.2.0
python -m pip install --force-reinstall --no-deps "libero @ git+https://github.com/yinchimaoliang/LIBERO.git@058fda1ddebe92918af091cb6816759ca6d003f0"
python -m pip install --force-reinstall --no-deps "robosuite @ git+https://github.com/yinchimaoliang/robosuite.git@e293cc32ff3c48957a4ebcad09952432b0dc9049"
python -m pip install --no-build-isolation -e .
python -c "import av, diffusers, peft, transformers; from diffusers import Cosmos2_5_PredictBasePipeline; print(av.__version__, diffusers.__version__, peft.__version__, transformers.__version__)"RoboCasa GR00T support is still optional. The installer manages the Isaac-GR00T
and RoboCasa GR1 local checkouts under ./src for sim-only and full; use
--skip-robocasa if you do not use RoboCasa configs.
The update helper does not reinstall PyTorch or FlashAttention. Existing
flash-attn==2.5.5 environments can keep using it only if it still imports
against the installed PyTorch/CUDA build:
python - <<'PY'
import torch, flash_attn
from flash_attn.flash_attn_interface import flash_attn_func, flash_attn_varlen_func
print("torch", torch.__version__, "cuda", torch.version.cuda)
print("flash-attn", flash_attn.__version__)
PYIf you upgrade PyTorch with the current installer or the commands below,
reinstall a matching FlashAttention wheel as well. The installer currently
defaults to flash-attn==2.8.3.post1 for the supported PyTorch profiles.
Use the manual path only if you are not using scripts/install_env.sh.
Install PyTorch first, then FlashAttention, then the remaining FluxVLA
dependencies.
1. Create a conda environment
conda create -n fluxvla python=3.10 -y
conda activate fluxvla2. Install PyTorch (CUDA version)
Important: Before running
pip install -r requirements.txt, you must install PyTorch from the official CUDA index first. The default PyPI index cannot fetch CUDA-enabled builds.
# CUDA 12.8
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128For other CUDA versions, replace cu128 with the corresponding value (e.g., cu118, cu121). See: https://pytorch.org/get-started/locally/ and https://pytorch.org/get-started/previous-versions/.
3. Install flash-attention
The one-command installer downloads a prebuilt FlashAttention wheel from the official release assets. For manual installation, install the wheel matching your Python, PyTorch, and C++ ABI instead of building from source:
PYTAG=$(python - <<'PY'
import sys
print(f"cp{sys.version_info.major}{sys.version_info.minor}")
PY
)
ABI=$(python - <<'PY'
import torch
print(str(torch._C._GLIBCXX_USE_CXX11_ABI).upper())
PY
)
pip install --no-deps \
"https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.3.post1/flash_attn-2.8.3.post1+cu12torch2.8cxx11abi${ABI}-${PYTAG}-${PYTAG}-linux_x86_64.whl"If you installed PyTorch 2.6, replace torch2.8 in the wheel URL with
torch2.6.
FlashAttention wheels are tied to the installed Python, PyTorch, CUDA, and C++
ABI. flash-attn==2.5.5 is not forbidden, but it is only safe to keep when it
was built for the exact PyTorch/CUDA stack you are still using. After any
PyTorch upgrade, reinstall a matching FlashAttention wheel.
4. Install FFmpeg and av
conda install -c conda-forge "ffmpeg=7" av=14.2.05. Install fluxvla and other dependencies
pip install -r requirements.txt
pip install --no-build-isolation -e .Note:
requirements.txtnow composesrequirements-base.txt,requirements-sim.txt, andrequirements-real.txt. It does not install PyTorch; install CUDA PyTorch first or usescripts/install_env.sh. TorchCodec is also installed by the environment scripts because its version must match PyTorch. For a manual x86_64 install, usetorchcodec==0.7.0with Torch 2.8 ortorchcodec==0.2.1with Torch 2.6. TorchCodec also requires the conda/system FFmpeg shared libraries; PyAV andimageio-ffmpegalone are not sufficient. Linux aarch64 uses the PyAV fallback.
Jetson Orin Docker configuration
For Jetson Orin setup, see docs/orin_flashing.md for initial flashing and JetPack setup, and docs/orin_docker_runtime.md for the validated FluxVLA Docker runtime workflow.
The validated Orin runtime is published as a Docker image:
docker pull fluxvla/fluxvla:fluxvla-orin-1.0.0
scripts/run_docker.shscripts/run_docker.sh uses this image by default and mounts the current repository to /workspace/FluxVLA. Runtime details are documented in docs/orin_docker_runtime.md.
Optional: RoboCasa GR00T source checkouts
GR00T N1.5 and N1.7 configs are organized under configs/gr00tn15/ and configs/gr00tn17/, respectively. RoboCasa GR00T N1.5 configs such as configs/gr00tn15/gr00tn15_eagle_3b_robocasa_30_eps_full_finetune.py require the pinned Isaac-GR00T and RoboCasa GR1 task checkouts. The one-click installer handles them for sim-only and full by default and places them under ./src:
bash scripts/install_env.sh sim-onlyUse FLUXVLA_ROBOCASA_SRC_ROOT=/path/to/src to choose another checkout root, --skip-robocasa to skip these source installs, and --with-robocasa to force them in real-only mode. Runtime dependencies and the patched robosuite build are installed from requirements-sim.txt.
If you are not using the installer, the equivalent manual commands are:
pip install "mujoco==3.2.6" gymnasium lxml
pip install "robosuite @ git+https://github.com/yinchimaoliang/robosuite.git@e293cc32ff3c48957a4ebcad09952432b0dc9049"
git clone https://github.com/NVIDIA/Isaac-GR00T.git ./src/Isaac-GR00T
git -C ./src/Isaac-GR00T checkout 4af2b622892f7dcb5aae5a3fb70bcb02dc217b96
pip install --no-deps -e ./src/Isaac-GR00T
git clone https://github.com/robocasa/robocasa-gr1-tabletop-tasks.git \
./src/robocasa-gr1-tabletop-tasks
git -C ./src/robocasa-gr1-tabletop-tasks checkout 4840e671596f93ca03651524b9f72ffb1aadfeff
pip install --no-deps -e ./src/robocasa-gr1-tabletop-tasks--no-deps is recommended for editable installs so the RoboCasa packages do not replace the pinned FluxVLA model stack dependencies. RoboCasa assets and datasets are covered in Data & Assets Preparation.
Optional: LIBERO / MuJoCo EGL setup for online evaluation
If you want to evaluate LIBERO on devices that do not support ray tracing (e.g., A100), please refer to EGL Device GPU Rendering Configuration.
scripts/install_env.sh sim-only and scripts/install_env.sh full now probe MuJoCo EGL automatically. If EGL devices are not visible, the installer tries to install the system packages below, creates the NVIDIA GLVND vendor file, and writes a conda activation hook for MUJOCO_GL=egl. Use FLUXVLA_EGL_SETUP=always to make this check strict, or --skip-egl-setup to skip it.
Install system dependencies
export MUJOCO_GL=egl
export PYOPENGL_PLATFORM=egl
sudo apt-get update
sudo apt-get install -y libegl1 libglvnd0 libopengl0 libegl-dev libgl1-mesa-dev libx11-dev libglew-dev libosmesa6-devEnvironment checks
Make sure /proc/1/environ contains the following environment variables:
- NVIDIA_DRIVER_CAPABILITIES=all
- NVARCH=x86_64
- NVIDIA_REQUIRE_CUDA=cuda>=12.4
- brand=teslaand- driver>=470
Create an EGL configuration file
Create file /usr/share/glvnd/egl_vendor.d/10_nvidia.json with the following content:
{
"file_format_version": "1.0.0",
"ICD": {
"library_path": "libEGL_nvidia.so.0"
}
}Then launch eval with __EGL_VENDOR_LIBRARY_FILENAMES=/usr/share/glvnd/egl_vendor.d/10_nvidia.json unless your environment already exports it.
Configure pre-commit hooks (optional but recommended)
To ensure code quality and consistency (especially for C++/CUDA code), install pre-commit hooks:
pip install pre-commit
pre-commit installThis will automatically check and format code before every commit.
Configure Weights & Biases (wandb)
Weights & Biases is used for experiment tracking and visualization. Configure it as follows:
- Install wandb (included in requirements.txt):
pip install wandb- Log in to your wandb account:
wandb login- Set environment variables:
export WANDB_PROJECT=fluxvla # project name (default: fluxvla)
export WANDB_ENTITY=your-team-name # team name or username (default: None)
export WANDB_MODE=online # online, offline, or disabled (default: online)- If you want to disable wandb logging during training, set:
export WANDB_MODE=disabledNote: all wandb configuration is read from environment variables; no additional settings are needed in config files.
Configure TensorBoard (optional)
TensorBoard is supported as an optional logging backend for experiment metric visualization. Configure it as follows:
- Add 'tensorboard'toactive_trackersin your config file:
metric=dict(
type='VLAMetric',
active_trackers=('jsonl', 'wandb', 'tensorboard'),
...
)Alternatively, enable it via command line without modifying the config file:
--cfg-options 'runner.metric.active_trackers=[jsonl,wandb,tensorboard]'- After training, launch TensorBoard to view metrics:
tensorboard --logdir work_dirs/tensorboardNote: event files are saved to {work_dir}/tensorboard/{run_id}/ per run, enabling automatic comparison across experiments. If the TENSORBOARD_LOG_PATH environment variable is set, it will be used directly as the log directory.
Use the datasets we prepared directly
Download the required datasets and place them under ./datasets. Download only the datasets you need according to your configuration.
For example, download the libero-10 dataset:
huggingface-cli download limxdynamics/FluxVLAData --repo-type dataset --include "libero_10_no_noops_lerobotv2.1/*" --local-dir ./datasetsReplace libero_10_no_noops_lerobotv2.1 with the corresponding folder name of the dataset you want to download.
For RoboCasa GR00T training with the released 30-demo subset, download the
dataset under ./datasets:
huggingface-cli download limxdynamics/FluxVLAData \
--repo-type dataset \
--include "robocasa_gr1_24tasks_first30ep/*" \
--local-dir ./datasetsFor full-data RoboCasa GR1 training, replace the include pattern with
robocasa_lerobot_V2.1/*.
Compute transformed normalization statistics
Recompute normalization statistics whenever the training data, robot action semantics, action horizon, or terminal-padding policy changes. The automatic training path and the command-line tool call the same implementation, so they produce the same statistics when given the same profile and dataset settings.
For configs that opt in with auto_compute_statistics, the startup priority
is:
- use inline dataset_statisticswhen present;
- otherwise use dataset_statistics_pathwhen present;
- otherwise compute transformed statistics once on rank 0 and save
dataset_statistics.jsonplusdataset_statistics_metadata.jsonin the work directory.
The PI0.5 UR3, dual-Franka, and Tron2 training configs use this automatic
path. The ALOHA configs use the official OpenPI PI0.5 Trossen statistics from
gs://openpi-assets/checkpoints/pi05_base/assets/trossen/norm_stats.json for
both training normalization and action denormalization. RoboCasa keeps
checked-in dataset-specific statistics to avoid scanning the full dataset at
every startup. To run a dataset-specific calculation manually from the
repository root:
conda activate fluxvla
# UR3: six relative joints and an absolute gripper.
python tools/compute_transformed_dataset_stats.py /path/to/ur3 \
--profile ur3 --action-horizon 50 \
--variable-name _PI05_UR3_STATS --output /tmp/ur3_stats.py
# Dual Franka joint-position actions: relative joints and absolute grippers.
python tools/compute_transformed_dataset_stats.py /path/to/franka \
--profile franka-qpos --action-horizon 50 \
--variable-name _PI05_FRANKA_QPOS_STATS \
--output /tmp/franka_qpos_stats.py
# Dual Franka Cartesian poses: fully absolute actions.
python tools/compute_transformed_dataset_stats.py /path/to/franka \
--profile franka-eepose --action-horizon 50 \
--variable-name _PI05_FRANKA_EEPOSE_STATS \
--output /tmp/franka_eepose_stats.py
# Tron2: fully absolute arm, head, and gripper qpos targets.
python tools/compute_transformed_dataset_stats.py /path/to/tron2 \
--profile tron2 --action-horizon 50 \
--variable-name _TRON2_STATS --output /tmp/tron2_stats.py
# RoboCasa GR1: relative arm/waist joints and absolute Fourier-hand commands.
python tools/compute_transformed_dataset_stats.py /path/to/robocasa_lerobot_V2.1 \
--profile robocasa-joint-delta --action-horizon 16 \
--statistic-name robocasa_gr1_24tasks_joint_delta \
--variable-name _PI05_ROBOCASA_STATS \
--output /tmp/pi05_robocasa_joint_delta_stats.pyThe tool applies configured robot coordinate/sign transforms first, converts
only selected action dimensions to deltas second, and computes statistics
last. Fully absolute policies can use --profile absolute or --no-delta;
for example, a GR00T policy whose action column already contains absolute qpos
can use auto_compute_statistics=dict(profile='absolute'). Override
state_key or action_key when the dataset uses different columns.
The output contains mean, std, min, max, q01, and q99, so it can
serve mean/std, min/max, or PI0.5 quantile normalization. Automatic computation
inherits action_window_size, window_start_idx,
supervise_terminal_padding, and statistic_name directly from the training
config.
For OpenPI parity, do not regenerate the ALOHA statistics from the local
training split. The checked-in _PI05_ALOHA_STATS is a direct copy of the
official PI0.5 Trossen asset. If a different ALOHA calibration or data domain
intentionally requires dataset-specific statistics, generate a replacement
with:
python tools/compute_transformed_dataset_stats.py /path/to/aloha \
--profile aloha --action-key observation.state \
--gripper-input-range=-0.01,0.08 --action-horizon 50 \
--variable-name _PI05_ALOHA_STATS --output /tmp/aloha_stats.pyWhen overriding the official asset, use the same replacement dictionary for
both train_dataloader.dataset.dataset_statistics and
inference.denormalize_action.norm_stats.
The legacy tools/compute_pi05_norm_stats.py command remains available as a
compatibility wrapper around the generic tool.
Terminal padding is included by default. Add --exclude-terminal-padding if
the config masks padded actions from the loss. The default action-window start
is 0; set --window-start-index only when the config intentionally uses a
different offset. For large datasets, use --temp-dir /path/with/free-space
to place temporary memory-mapped files on a disk with sufficient capacity.
Run python tools/compute_transformed_dataset_stats.py --help for the
available profiles and overrides such as custom state/action keys and delta
masks.
ARM datasets
The built-in ARM example config configs/arm/arm_clip_aloha_example.py expects a progress-labeled LeRobot v3.x dataset at ./datasets/ARM_manual_test_10Episodes_lerobotv3.0.
Download the released example dataset to the expected location with:
huggingface-cli download limxdynamics/FluxVLAData \
--repo-type dataset \
--include "ARM_manual_test_10Episodes_lerobotv3.0/*" \
--local-dir ./datasetsARM training reads the progress column directly from this dataset. For RA-BC / AW-BC on policy or DAgger datasets that do not already contain progress, first train or load an ARM checkpoint, then generate arm_progress.parquet with scripts/compute_arm_awbc_progress.py. See docs/arm.md and tools/arm_awbc/README.md.
Prepare assets
Use the FluxVLA asset downloader below as the supported path for RoboCasa GR1 tabletop tasks. The table lists the upstream archives used by the script; manually downloading and extracting those archives is not sufficient for this stack because the script also fixes the directory layout and normalizes Objaverse XML metadata for the pinned RoboCasa GR1 checkout.
When using scripts/install_env.sh, this downloader runs by default together
with the RoboCasa source checkouts unless --skip-robocasa or
--skip-robocasa-assets is passed. For manual installation or refreshing the
assets, run this command from the FluxVLA repository root. It downloads the
required archives through the selected Hugging Face endpoint, extracts them
into the RoboCasa asset directory, and normalizes the Objaverse XML metadata:
python scripts/download_robocasa_assets.py --endpoint https://hf-mirror.comIf the archives or extracted assets already exist locally, still run this
script so the XML compatibility step is applied. For assets that have already
been extracted into ./src/robocasa-gr1-tabletop-tasks/robocasa/models/assets,
you can run only the validation and XML normalization step:
python scripts/download_robocasa_assets.py --normalize-onlySymlinks are not required; they are only a convenience when the assets already live on another local disk or shared storage.
SARM datasets
FluxVLA SARM workflows accept standard LeRobot v2.1 or v3.x datasets. Besides the usual observation / action fields, the dataset must carry SARM subtask annotations in episodes metadata.
Published SARM example datasets on Hugging Face:
- LeRobot v3.x manual sparse+dense annotations for training / inference: limxdynamics/FluxVLAData/SARM_manual_test_10Episodes_lerobotv3.0
- LeRobot v3.x unlabeled dataset kept for manual or VLM labeling: limxdynamics/FluxVLAData/SARM_vlm_test_10Episodes_lerobotv3.0
- New LeRobot v2.1 manual conversion for training / inference and legacy-tool compatibility: limxdynamics/FluxVLAData/SARM_manual_test_10Episodes_lerobotv2.1
- New LeRobot v2.1 unlabeled conversion for manual or VLM labeling workflows: limxdynamics/FluxVLAData/SARM_vlm_test_10Episodes_lerobotv2.1
Download them under ./datasets with:
huggingface-cli download limxdynamics/FluxVLAData --repo-type dataset --include "SARM_manual_test_10Episodes_lerobotv3.0/*" --local-dir ./datasets
huggingface-cli download limxdynamics/FluxVLAData --repo-type dataset --include "SARM_vlm_test_10Episodes_lerobotv3.0/*" --local-dir ./datasets
huggingface-cli download limxdynamics/FluxVLAData --repo-type dataset --include "SARM_manual_test_10Episodes_lerobotv2.1/*" --local-dir ./datasets
huggingface-cli download limxdynamics/FluxVLAData --repo-type dataset --include "SARM_vlm_test_10Episodes_lerobotv2.1/*" --local-dir ./datasetsUse the manual_* datasets directly for training / inference. Use the vlm_* datasets as clean starting points for manual stage writing or VLM auto-annotation. Prefer the v2.1 pair when another tool expects meta/episodes.jsonl plus per-episode videos; prefer the v3.0 pair when you want to keep native LeRobot v3.x metadata layout.
Before using a LeRobot v3.x SARM dataset, sanity-check the video metadata:
-
LeRobot v3.x allows either many episodes in one MP4 or one MP4 per episode.
-
If many episodes share one MP4, each episode that points to that file must use correct from_timestamp/to_timestampoffsets.
-
If videos are already split as file-000.mp4,file-001.mp4, ..., each episode should point to its ownfile_index, andfrom_timestampwill usually reset to0.0.
-
If the directory contains multiple MP4 files but all episodes still point to file-000.mp4, the dataset metadata is malformed and should be fixed before use.
-
For ready-to-use SARM dataset structure, annotation columns, and progress inference usage, see docs/sarm.md.
-
For writing manual stages or generating VLM-based annotations, see tools/sarm_annotate/README.md.
Private dataset directory structure
If you train with fluxvla on private datasets, you need to convert your raw data (e.g., HDF5 files collected by ALOHA robots) into the LeRobot Dataset v2.1 format. For a step-by-step conversion guide, see Data Conversion Guide.
For SARM specifically, FluxVLA supports both LeRobot v2.1 and v3.x datasets as long as the required SARM annotation columns are present. The SARM-specific metadata contract is documented in docs/sarm.md.
The converted dataset should follow this directory structure:
├── data
│ └── chunk-000
│ │ └── episode_000000.parquet
│ │ └── episode_000001.parquet
│ │ └── ... (more parquet files)
│ │ └── episode_00000N.parquet
│ └── chunk-001
│ └── ... (more chunks)
│ └── chunk-00N
├── meta
│ └── episodes.jsonl
│ └── episodes_stats.jsonl
│ └── info.json
│ └── tasks.jsonl
├── videos
│ └── chunk-000
│ │ └── camera name 0
│ │ │ └── episode_000000.mp4
│ │ │ └── episode_000001.mp4
│ │ │ └── ...(more mp4 files)
│ │ │ └── episode_00000N.mp4
│ │ └── camera name 1
│ │ └── ...(more cameras)
│ │ └── camera name N
│ └── chunk-001
│ └── ... (more chunks)
│ └── chunk-00N
Download the required pretrained checkpoints and place them under ./checkpoints. Download only the checkpoints you need based on your configuration.
For ARM and SARM workflows, you typically need a CLIP checkpoint for training / inference. SARM VLM-based annotation also needs the Qwen3-VL checkpoint used by the official SARM workflow. Detailed usage is documented in docs/arm.md and docs/sarm.md.
VLA models
Download the PI0 RoboCasa full-data checkpoint while preserving the path expected by its config:
hf download limxdynamics/FluxVLAEngine \
--include "pi0_paligemma_robocasa_full_data_full_finetune_bs256/*" \
--local-dir ./checkpointsDiT4DiT requires the
Cosmos-Predict2.5-2B
Diffusers checkpoint. Download the exact revision and directory layout expected by
configs/dit4dit/dit4dit_libero_all_full_finetune.py:
hf download nvidia/Cosmos-Predict2.5-2B \
--revision diffusers/base/post-trained \
--local-dir ./checkpoints/Cosmos-Predict2.5-2BDownload the FluxVLA DiT4DiT LIBERO checkpoint linked in the performance table:
hf download limxdynamics/FluxVLAEngine \
--include "dit4dit_libero_all_full_finetune_bs256/*" \
--local-dir ./checkpointsThe official DiT4DiT LIBERO checkpoint
is optional for the default train-from-Cosmos recipe. Download it when reproducing the released model or
when setting model.pretrained_name_or_path to _official_dit4dit_ckpt:
hf download mondo-robotics/dit4dit-model \
--include "dit4dit_libero/*" \
--local-dir ./checkpoints/dit4dit-modelVision-Language Models (VLM)
Large Language Models (LLM)
Vision backbone networks
Tip: You can speed up downloads with
huggingface-cli download <model-name> --local-dir ./checkpoints/<model-name>.
For the built-in ARM and SARM configs, place the CLIP files under ./checkpoints/clip-vit-base-patch32:
huggingface-cli download openai/clip-vit-base-patch32 --local-dir ./checkpoints/clip-vit-base-patch32If you use VLM-based SARM annotation, place the official SARM VLM under ./checkpoints/Qwen3-VL-30B-A3B-Instruct.
All-in-one: One configuration file manages the full workflow
- Manage key parameters for data, models, training, evaluation, inference, and deployment through a single config file (easier to reproduce and deploy).
Supports different VLA models
- Supports OpenVLA, LlavaVLA, Gr00t, Pi0, Pi0.5, FastWAM, and DiT4DiT.
Supports different modules
- Supports Llama, Gemma, and Qwen-family LLM backbones.
- Supports DINOv2 and SigLIP vision backbones.
- Supports PaliGemma and Qwen-VL VLM backbones.
Supports reward modeling workflows
- Supports SARM training, annotation, and progress inference on LeRobot v2.1/v3.x datasets. See docs/sarm.md for details.
- Supports ARM reward modeling, progress reconstruction, and RA-BC / AW-BC sample reweighting. See docs/arm.md for details.
Supports different training strategies
- Supports FSDP together with DDP, and supports LoRA training mode.
- Supports eval-after-train.
- Supports resuming training from checkpoints.
Data and weight formats
- Supports Parquet datasets and loading LeRobot-format data.
- Supports model weights in safetensors format.
Evaluation and inference capabilities
- Supports multi-GPU evaluating libero on devices without ray tracing.
- Supports uploading LIBERO and RoboCasa evaluation summaries to Feishu Sheets; see Feishu Evaluation Reporting.
- Supports remote inference infrastructure with ZMQ-based server/client architecture, enabling GPU-offloaded inference for resource-constrained edge devices. See Remote Inference Serving.
- Supports RTC (Real-Time Chunking) to improve cross-chunk trajectory continuity.
- Supports accelerated inference for GR00T and PI0.5; see Inference Acceleration, including Triton fused kernels, CUDA Graph capture, and CUDA custom operators.
- Provides a minimal Oli humanoid whole-body (loco-manipulation) real-robot inference path (rospy sensor input + WebSocket control; base/hand commands are robot-SDK integration points). See docs/oli_whole_body.md.
Local debugging
/root/miniconda3/envs/fluxvla/bin/torchrun --standalone --nnodes 1 --nproc-per-node [NUM_GPUS] scripts/train.py --config [CONFIG_PATH] --work-dir [WORK_DIR] --cfg-options train_dataloader.per_device_batch_size=[PER_DEVICE_BATCH_SIZE]
Example:
export WANDB_MODE=disabled
/root/miniconda3/envs/fluxvla/bin/torchrun --standalone --nnodes 1 --nproc-per-node 2 scripts/train.py --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py --work-dir ./checkpoints/pi05_paligemma_libero_10_full_finetune --cfg-options train_dataloader.per_device_batch_size=2
RoboCasa GR00T smoke training example:
WANDB_MODE=disabled TOKENIZERS_PARALLELISM=false \
torchrun --standalone --nnodes 1 --nproc-per-node 1 scripts/train.py \
--config configs/gr00tn15/gr00tn15_eagle_3b_robocasa_30_eps_full_finetune.py \
--work-dir work_dirs/smoke_groot_robocasa_train \
--cfg-options \
runner.type=FSDPTrainRunner \
runner.sharding_strategy=no-shard \
train_dataloader.per_device_batch_size=1 \
runner.enable_gradient_checkpointing=False \
runner.max_steps=2 \
runner.save_iter_interval=1 \
runner.max_keep_ckpts=2 \
"runner.metric.active_trackers=('jsonl',)"Local evaluation
/root/miniconda3/envs/fluxvla/bin/torchrun --standalone --nnodes 1 --nproc-per-node [NUM_GPUS] scripts/eval.py --config [CONFIG_PATH] --ckpt-path [CKPT_PATH] --cfg-options [CFG_OPTIONS]
Example:
export WANDB_MODE=disabled
/root/miniconda3/envs/fluxvla/bin/torchrun --standalone --nnodes 1 --nproc-per-node 2 scripts/eval.py --config configs/pi05/pi05_paligemma_libero_10_full_finetune.py --ckpt-path checkpoints/pi05_paligemma_libero_10_full_finetune_bs64/checkpoints/step-028548-epoch-18-loss=0.0111.safetensors
RoboCasa GR00T evaluation example:
MUJOCO_GL=egl WANDB_MODE=disabled TOKENIZERS_PARALLELISM=false \
PYTHONHASHSEED=7 \
torchrun --standalone --nnodes 1 --nproc-per-node 1 scripts/eval.py \
--config configs/gr00tn15/gr00tn15_eagle_3b_robocasa_30_eps_full_finetune.py \
--ckpt-path work_dirs/gr00t_eagle_3b_robocasa_gr1_24x30_finetune_bs64/checkpoints/step-010000.safetensors \
--cfg-options \
eval.norm_stats_path=work_dirs/official_groot_gr1_dataset_statistics.json \
eval.output_dir=work_dirs/gr00t_eagle_3b_robocasa_eval \
eval.num_trials_per_task=50 \
eval.seed=7eval.seed controls the RoboCasa episode seeds and stochastic GR00T action
sampling seeds during evaluation. PYTHONHASHSEED is independent and must be
set before Python starts; using the same value is recommended when reproducing
reported RoboCasa results.
Cluster training
export WANDB_MODE=disabled
bash scripts/train.sh [CONFIG] [WORK_DIR] --cfg-options train_dataloader.per_device_batch_size=[PER_DEVICE_BATCH_SIZE] train_dataloader.batch_size=[GLOBAL_BATCH_SIZE] runner.max_steps=[MAX_STEPS] runner.save_interval=[SAVE_INTERVAL] runner.max_keep_ckpts=[MAX_KEEP_CKPTS] --eval-after-train
Resume training from a checkpoint
To resume training from a checkpoint, use the --resume-from argument to specify the checkpoint file path. Training will continue from the saved global step, epoch, model state, and optimizer state.
Local training example:
export WANDB_MODE=disabled
/root/miniconda3/envs/fluxvla/bin/torchrun --standalone --nnodes 1 --nproc-per-node 2 scripts/train.py \
--config configs/pi05/pi05_paligemma_libero_10_full_finetune.py \
--work-dir ./work_dirs/pi05_paligemma_libero_10_full_finetune \
--resume-from ./work_dirs/pi05_paligemma_libero_10_full_finetune/checkpoints/checkpoint_epoch_5.pt \
--cfg-options train_dataloader.per_device_batch_size=2
Cluster training example:
export WANDB_MODE=disabled
bash scripts/train.sh [CONFIG] [WORK_DIR] \
--resume-from [CHECKPOINT_PATH] \
--cfg-options train_dataloader.per_device_batch_size=[PER_DEVICE_BATCH_SIZE] runner.max_steps=[MAX_STEPS]
Cluster evaluation
export WANDB_MODE=disabled
bash scripts/eval.sh [CONFIG] [CKPT_PATH] --cfg-options [CFG_OPTIONS]
Real-robot inference
When running inference on a real robot, first install the environment on the robot side, and then run:
python scripts/inference_real_robot.py --config [CONFIG] -- ckpt-path [CKPT_PATH]
Q: Problems connecting to Hugging Face when downloading models or datasets.
A: If you encounter Hugging Face connectivity issues (e.g., slow downloads, timeouts, or connection refused), set the following environment variable before running the command and use hf-mirror:
export HF_ENDPOINT="https://hf-mirror.com"Q: conda install av is very slow at resolving the environment.
A: You can use the libmamba solver to speed up dependency resolution:
conda install -c conda-forge av=14.2.0 --solver=libmambaQ: GR00T evaluation on LIBERO is unstable.
A: This is expected. GR00T's performance on LIBERO is sensitive to random seeds, the hardware environment, and the number of training epochs. Small changes in these factors may cause noticeable fluctuations in evaluation results. It is recommended to run experiments with multiple random seeds and select the best checkpoint based on evaluation performance.
Q: When running pip install -r requirements.txt, building egl_probe fails with RuntimeError: CMake must be installed.
A: egl_probe needs CMake to build. Install it via conda (recommended) or apt:
conda install -c conda-forge cmake
# or
sudo apt install cmakeNote: Do not use
pip install cmake. The pip package is a Python wrapper and may fail because pip isolates the build environment.
Q: egl_probe build fails and reports Compatibility with CMake < 3.5 has been removed from CMake.
A: This is usually because your CMake version is too new for the egl_probe CMakeLists.txt. Set the following environment variable before installing:
CMAKE_POLICY_VERSION_MINIMUM=3.5 pip install -r requirements.txtQ: After installation, I get NumPy version errors (e.g., RuntimeError: Numpy is not available or version incompatibility warnings).
A: During installation, some dependencies may overwrite the pinned NumPy version. Reinstall the correct version directly:
pip install numpy==1.26.4Please see the contribution workflow and guidelines in docs/CONTRIBUTING.md.
Quick conventions:
- Discuss first: for new features/models or other large changes, please open a GitHub Issue to align on scope and design.
- Branch from upstream: create your branch from upstream/mainand use prefixes likefeat/,fix/,docs/, etc. (details in the contributing guide).
- Run checks before PR: make sure local pre-commit passes and CI is green.
- Commit messages: we recommend Conventional Commits (examples in the contributing guide).
If you encounter any issues while using this repository, feel free to contact us. You can reach us directly at mason@limxdynamics.com and wayne@limxdynamics.com, or open a GitHub issue for help.
If you use FluxVLA in your research or projects, please cite the relevant works as:
@software{FluxVLA2026,
author = {Li, Yinhao and Mao, Weixin and Lan, Zihan and Rong, Jikun and Zhu, Minzhao and Mao, Yiming and Shen, Bowen and Huang, Xu},
title = {{FluxVLA Engine: A One-Stop VLA Engineering Platform for Embodied Intelligence}},
year = {2026},
month = apr,
version = {1.0.0},
doi = {10.5281/zenodo.20049506},
url = {https://github.com/FluxVLA/FluxVLA},
license = {Apache-2.0},
}
@InProceedings{Mao_2026_CVPR,
author = {Mao, Yiming and Yu, Zixi and Mao, Weixin and Li, Yinhao and Hu, Qirui and Lan, Zihan and Zhu, Minzhao and Chen, Hua},
title = {ARM: Advantage Reward Modeling for Long-Horizon Manipulation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2026},
pages = {4468-4477}
}
@article{Huang_2026_IROS,
title={Long-Term Memory for VLA-based Agents in Open-World Task Execution},
author={Huang, Xu and Mao, Weixin and Li, Yinhao and Chen, Hua and Zhao, Jiabao},
journal={arXiv preprint arXiv:2604.15671},
year={2026}
}Acknowledgements: This project benefits from the following open-source projects and community efforts. Thanks to: LeRobot, NVIDIA Isaac GR00T, DreamZero (code), OpenVLA, OpenPI (pi0), LLaVA, DeepSpeed, Qwen, Triton, RTC, Training RTC, and Realtime-VLA. If we missed your project or contribution, please open an issue or pull request so we can properly acknowledge it.
- Support more vision backbone networks.
- Support more VLM backbones.
- Support more VLA methods.
- Support training with VLM data or reasoning-chain-of-thought (CoT) data.
- Full implementation of the logger feature.
- Support Isaac Sim.