Swarm-Steward: commanding a robot swarm with natural language

June 25, 2026 · Presented at IEEE ICUAS 2026, Corfu

Imagine telling a fleet of drones what to do the same way you'd brief a colleague — "search the northern sector, keep an eye on the tank farm, and orbit building 21" — and the swarm just… does it. That's the idea behind Swarm-Steward, the NAMUR project's flagship system, presented at IEEE ICUAS 2026 in Corfu, Greece.

Real-world validation with a DJI Mini 4 Pro swarm

Swarm-Steward hierarchical LLM architecture

From running chilly winter field experiments in snowy Denmark to presenting under the summer sun in Corfu, this work is the project's most complete demonstration yet of LLM-driven multirobot control: Swarm-Steward: Scalable and Reliable Natural-Language Coordination of Autonomous Aerial and Ground Robots.

The intent-to-control gap

Multi-robot systems offer strong advantages for tasks that benefit from coverage, redundancy, and parallelism — including search and rescue and infrastructure inspection. In practice, however, operating a swarm remains difficult: high-level intent must be translated into platform-specific commands, operators must track robot identifiers and constraints, and plans must be continuously adapted as state changes (battery, position, connectivity, environment). This intent-to-control gap limits who can effectively use swarm systems and how quickly they can be deployed.

Swarm-Steward closes that gap through a conversational interface. Figure 1 shows the kind of interaction the system supports: the operator creates named groups, adjusts altitudes, reasons about spatial relationships, and requests coordinated behaviours — all in plain language.

Swarm-Steward natural-language GUI: group creation, altitude control, spatial reasoning — Figure 1 — Natural-language interaction with *Swarm-Steward* showing group creation, altitude control, spatial reasoning, and coordinated behaviors across multiple drones.

A hierarchical LLM multi-agent system

Swarm-Steward is a platform-agnostic architecture for conversational swarm control. The operator defines named groups and commands them through natural language to perform coordinated behaviours such as formation manoeuvring, orbiting, path traversal, and area coverage. The system separates concerns across four layers:

UI layer — natural-language commands, mission previews, and confirmations.
LLM-MAS layer — a coordinator and specialized sub-agents that plan and ground requests.
Action-execution layer — deterministic, schema-constrained group actions with a safety gate.
Platform layer — the actual drones and robots, reached through a bridge (ROS 2/DDS or MQTT).

Figure 2 summarizes the end-to-end pipeline: natural-language commands are decomposed by a Coordinator into staged sub-tasks executed by specialized agents, then dispatched as deterministic group actions.

Swarm-Steward architecture pipeline from operator input to action execution — Figure 2 — Pipeline from operator input to action execution via bridge and telemetry feedback.

Specialized sub-agents strictly separate planning, grounding, and execution — see Figure 3. A key invariant is that only the Swarm agent may modify world state; the remaining agents are non-actuating and restricted to context gathering and synthesis. This keeps the LLM in the role of a smart operator rather than a risky driver.

Swarm-Steward sub-agents and their tool interfaces — Figure 3 — Sub-agents and their tool interfaces.

Targeted retrieval-augmented grounding

All benchmarks below ran on the hardware summarized in Table 1.

	Type	CPU	RAM	GPU (VRAM)	OS
PC-A	Desktop	Ryzen 9 5950X	32 GB	RTX 4090 (24 GB)	Ubuntu 22.04
PC-B	Laptop	Ultra 9 275HX	64 GB	RTX 5090 (24 GB)	Ubuntu 25.10

Table 1 — Hardware configurations used in the experiments.

PC-A ran the simulation/model benchmarks; PC-B the real-world validation.

To keep reasoning scalable, Swarm-Steward applies dual retrieval-augmented generation over both map features (Feature RAG) and telemetry variables (State RAG), injecting only relevant candidates at each step. Feature RAG grounds free-form geographic references by embedding map-feature descriptions and retrieving the top-k candidates most similar to the user query.

We stress-tested it with up to 10,000 map features (in-domain synthetic noise included), across four embedding models — two cloud (OpenAI, Gemini) and two local (BGE-M3, E5-large-v2). Table 2 reports the results: all four saturate exact-name, partial-name, and category retrieval at or near 100% HR@10, so the main discriminator is descriptive queries, where Gemini and the local models reach 85–86% while OpenAI trails at 79%. Crucially, search latency stays below 30 ms even at the 10,000-feature pool, so scaling the map never inflates the LLM context.

Model	Dim	Exact HR	Exact MRR	Partial HR	Partial MRR	Desc. HR	Desc. MRR	Mean HR	Mean MRR	Embed (ms)	Search (ms)
BGE-M3 (local)	1024	100	0.87	93	0.71	86	0.76	95	0.83	15.7	12.2
OpenAI (cloud)	1536	100	0.86	100	0.74	79	0.78	95	0.84	133.1	14.2
Gemini (cloud)	3072	100	0.87	100	0.74	86	0.84	97	0.86	380.3	24.7
E5-large-v2 (local)	1024	100	0.90	100	0.68	85	0.82	96	0.85	18.1	12.3

Table 2 — Feature RAG with in-domain noise at a 10,000-feature pool (HR@10 / MRR, averaged over three scenarios, three repetitions). Best per column in bold.

Category queries (always 100% HR / ≥0.98 MRR) omitted. (cloud) = API; (local) = GPU of PC-A.

Action management and lifecycle control

World-modifying actions are tracked and executed under an explicit lifecycle managed by an Action Manager and an Action State Machine (ASM), shown in Figure 4. The Action Manager converts tool calls into tracked actions, maintains per-group priority queues, and dispatches actions sequentially per group; the ASM computes waypoints (including formation offsets and altitude adaptation), sends waypoint-level commands through the bridge, tracks progress from telemetry, and enforces runtime safety constraints — geofencing, altitude and separation limits — with optional operator preview before execution.

Swarm-Steward action state machine: execution and monitoring lifecycle — Figure 4 — Action execution and monitoring. The *Action Manager* converts tool calls into tracked actions, maintains per-group priority queues, and dispatches actions sequentially per group. An *Action State Machine* (ASM) computes waypoints (including formation offsets and altitude adaptation), sends waypoint-level commands through the bridge, tracks progress from telemetry, and enforces runtime safety constraints.

Does it scale? 5 to 500 drones

We evaluated the full orchestration pipeline in a controlled benchmark spanning two orders of magnitude: 5, 10, 25, 50, 100, 200, and 500 simulated drones, each configuration executed four times (28 sessions, 280 prompts total), using Llama 3.3 70B over an MQTT bridge. Table 3 reports the results.

Drones	C	Success (%)	LLM time (s)	Tokens (×10³)	Payload (KB)
5	2,2,2,2	100.0	4.40 ± 0.10	600 ± 6	3.4
10	2,3,3,4	100.0	4.70 ± 0.42	619 ± 13	6.7
25	2,3,5,6	97.5	4.42 ± 0.28	619 ± 8	16.4
50	2,3,7,8	97.5	4.31 ± 0.06	623 ± 2	32.9
100	2,3,9,12	90.0	4.60 ± 0.28	655 ± 2	66.2
200	2,4,7,16	95.0	5.40 ± 0.36	728 ± 20	132.2
500	2,3,18,26	70.0	7.92 ± 0.70	899 ± 32	330.9

Table 3 — Scalability benchmark results (mean ± std, 4 repetitions per size). C: number of groups, matching Prompt P2.

Overall: 92.9% success, 5.10 s mean LLM time, 678 ×10³ mean tokens, ~0.66 KB payload per drone. Llama 3.3 70B via Groq; MQTT bridge on PC-A.

📈 Near-constant LLM token cost from 5 to 500 drones (600–899 ×10³ tokens) — the group-level abstraction means larger fleets mainly grow the state payload (~0.66 KB per drone), not the LLM context.
🎯 92.9% task success across 280 distinct prompts; all configurations up to 50 drones stay at ≥97.5%.
✔️ The degradation appears mainly at N≥100 and stems from prompt-specific scaling limits (tool-call-budget constraints, context-window overflow) — not from incorrect coordinator planning, which stays accurate for every prompt where the model responds.

Simulation to real world, unchanged logic

We validated sim-to-real transfer by executing the same scripted mission in simulation and on a physical 5× DJI Mini 4 Pro swarm at the SDU cricket field (55.367866° N, 10.435056° E). A seven-step operator script — multi-drone takeoff, dynamic grouping, formation traversal, building orbit, area coverage, sequential inspection, and return-to-home — ran through the same natural-language interface in both environments.

Real-world validation: DJI Mini 4 Pro swarm coordinated takeoff — Figure 5 — Real-world validation with DJI Mini 4 Pro drones: coordinated takeoff of the swarm, a DJI Mini 4 Pro drone ready for takeoff, and the operator GUI showing the virtual industrial map with silos and buildings, with group Bravo traversing a path and group Charlie moving to orbit a building.

Simulated climbs and descents took slightly longer (e.g., takeoff to 30 m: 10.1 s sim vs. 9.6 s real), while path traversal and orbit maneuvers ran faster in reality (20.9 s vs. 16.1 s; 28.7 s vs. 25.3 s). Overall timing deviations stayed within 25%, primarily due to wind, GNSS noise, and simplified simulator dynamics — the same control interface worked unchanged in both worlds. A demonstration video is available at youtu.be/eXD3l9EsT4M.

Ultimately, this is a step toward making autonomous systems accessible to non-experts without sacrificing the robustness and auditability required for field deployment.

Swarm-Steward: Scalable and Reliable Natural-Language Coordination of Autonomous Aerial and Ground Robots
Jarabo-Peñas, Bravo-Arrabal, Rolland, Christensen · IEEE ICUAS 2026, pp. 796–804

IEEE Xplore · DOI · ResearchGate · Watch the system in action (video)

Supported by the Independent Research Fund Denmark (grant 10.46540/4264-00105B, NAMUR project), Innovation Fund Denmark through DIREC (PERSIST, U07), and the EU Horizon Europe WildDrone project (Marie Skłodowska-Curie grant 101071224).

Swarm-StewardICUAS 2026IEEE Xplore