feat(policies): Add remote http policy, to benchmark custom models

[grach0v]

What this does

Adds policy, that queries HTTP server for actions.
It's needed if you have your custom policy, that is hard to integrate to lerobot repository, but you want to run benchmarks available in lerobot.

How it was tested

You can install additional dependencies by doing

pip install -e ".[server]"

And run an example server with

uvicorn examples.remote.remote_policy_server:app --host 0.0.0.0 --port 8000

Then you can run benchmark for example

pip install -e ".[libero]"

lerobot-eval \
    --output_dir=/logs/ \
    --env.type=libero \
    --env.task=libero_spatial \
    --eval.batch_size=1 \
    --eval.n_episodes=3 \
    --policy.type=remote \
    --policy.server_url=http://localhost:8000 \
    --policy.timeout=30.0 \
    --policy.attempts=3 \
    --policy.additional_args='{"dataset_info":{"action_type":"eef","robot_embodiment":"single_arm","robot_type":"franka","stereo_replace_depth":false,"handheld":false,"no_state":false,"obs_dof":8,"action_dof":7},"inference_config":{"n_actions":6,"n_inference_steps":10}}' \
    --policy.n_action_steps=10 \
    --output_dir=./eval_logs_libero_spatial/ \
    --env.max_parallel_tasks=1 \
    --rename_map='{"observation.images.image":"observation.images.static1", "observation.images.image2": "observation.images.wrist1"}'

How to checkout & try? (for the reviewer)

Examples:

pip install -e ".[server]"
uvicorn examples.remote.remote_policy_server:app --host 0.0.0.0 --port 8000

pip install -e ".[libero]"
lerobot-eval \
    --output_dir=/logs/ \
    --env.type=libero \
    --env.task=libero_spatial \
    --eval.batch_size=1 \
    --eval.n_episodes=3 \
    --policy.type=remote \
    --policy.server_url=http://localhost:8000 \
    --policy.timeout=30.0 \
    --policy.attempts=3 \
    --policy.n_action_steps=10 \
    --output_dir=./eval_logs_libero_spatial/ \
    --env.max_parallel_tasks=1 

[Copilot]

Pull Request Overview

This PR adds a remote policy type that allows external models to be evaluated through LeRobot's benchmarking infrastructure. It enables users with custom policies served via HTTP to run LeRobot benchmarks without integrating their model code directly into the repository.

Key changes:

• Remote policy implementation that communicates with HTTP servers for action prediction
• MessagePack-based serialization utilities for efficient tensor transmission
• Example FastAPI server demonstrating the protocol
• Integration with LeRobot's existing evaluation pipeline

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/lerobot/utils/messaging.py	Adds msgpack-based serialization/deserialization for tensors and numpy arrays
src/lerobot/policies/remote/configuration_remote.py	Configuration class defining remote policy parameters (URL, timeout, retry attempts)
src/lerobot/policies/remote/modeling_remote.py	Remote policy implementation with HTTP client and action queue management
src/lerobot/policies/remote/processor_remote.py	Pre/post-processing pipeline for remote policy observations
src/lerobot/policies/remote/init.py	Module exports for remote policy components
src/lerobot/policies/factory.py	Factory integration to enable remote policy instantiation
src/lerobot/policies/init.py	Adds RemoteConfig to public API exports
src/lerobot/async_inference/constants.py	Adds "remote" to supported policies list
pyproject.toml	Adds server extra dependencies (fastapi, uvicorn, msgpack)
examples/remote/remote_policy_server.py	Example FastAPI server returning zero actions
README.md	Documentation for remote policy evaluation workflow

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Duplicate imports of Any, Dict on lines 2 and 7. Consolidate imports into a single from typing statement.

[Copilot]

Commented-out decorator @timeit(logger) should be removed if not needed, or uncommented and properly configured if timing is required.

Suggested change

# @timeit(logger)

[Copilot]

Corrected spelling of 'inert' to 'inert' (though 'inert' is correct, the article should be 'an inert' not 'a inert').

Suggested change

	"""Remote policy is inference-only; return a inert preset for API compatibility."""
	"""Remote policy is inference-only; return an inert preset for API compatibility."""

[Copilot]

Potential NoneType error if B is not inferred from any tensor/array in obs_input. Add a fallback default or raise a descriptive error if B remains None.

Suggested change

	if B is None:
	raise ValueError(
	"Could not infer batch size 'B' from obs_input. "
	"Ensure that obs_input contains at least one array-like value (torch.Tensor or np.ndarray) with a batch dimension."
	)

[helper2424]

Lerobot has Async inference which work over Grpc.

[grach0v]

@helper2424
Correct me if I am wrong, but you can only run inference with remote policy lerobot.async_inference.robot_client
you can't run lerobot-eval with such policy.

[helper2424]

Yep, but it's extendable.

[helper2424]

I mean, you have create additional http policy server - it could be done with Async remote server, as purpose is the same

[grach0v]

Oh, okay
I will adjust remote policy server so it uses async remote server

[helper2424]

Very cool, if u need help - ping me

[grach0v]

Hey, it took me some time to understand how grpc in lerobot works
But now it works
You can start remote server and then run benchmark with something like

lerobot-eval \
    --output_dir=./eval_logs_libero_spatial/ \
    --env.type=libero \
    --env.task=libero_spatial \
    --env.max_parallel_tasks=1 \
    --eval.batch_size=1 \
    --eval.n_episodes=3 \
    --policy.type=remote \
    --policy.server_address=localhost:8080 \
    --policy.request_timeout=30.0 \
    --policy.retries=3 \
    --policy.n_action_steps=10 \
    --policy.remote_policy_type=pi05 \
    --policy.remote_pretrained_name_or_path=lerobot/pi05_libero_finetuned \
    --policy.remote_policy_device=cuda \
    --rename_map='{"observation.images.empty_camera_0":"observation.images.image"}'
    ```
  
   For example

[grach0v]

@helper2424 Hey
Can you please check it out, do I have to edit something else?