Can AI Truly Run a Laboratory? USTC Delivers the First Rigorous Physical-World Stress Test
How do we determine whether artificial intelligence is truly capable of conducting scientific research — achieving end-to-end autonomous scientific discovery?
A groundbreaking study from the University of Science and Technology of China (USTC) moves this question beyond theoretical planning and knowledge retrieval. For the first time, it places AI as the central cognitive agent controlling a fully automated physical laboratory — transforming AI from a “thought engine” into an active experimental operator.
A Fully Integrated Machine Scientist Lab
The research team constructed a modular autonomous catalytic laboratory, comprising 45 integrated workstations covering:
- Chemical synthesis
- Material characterization
- Catalytic performance testing
To bridge AI reasoning with physical execution, the team encoded each laboratory capability as machine-readable skills — standardized, constraint-aware APIs that enforce real-world operational limits (e.g., temperature ranges, reagent compatibility, safety protocols). This enables AI agents to directly invoke hardware, while respecting physical, chemical, and procedural boundaries.
Figure 1. Architecture of the AI-interpretable machine scientist laboratory for catalysis research.
Figure 2. End-to-end workflow: translating high-level scientific intent into executable robotic instructions.
Systematic Real-World Evaluation Across 48 Configurations
The platform served as a rigorous benchmark, evaluating 48 distinct AI agent configurations — combinations of:
- 6 agent frameworks (e.g., ReAct, Plan-and-Execute, Reflexion)
- 9 large language models (including GPT-5.5, Claude Opus 4.7, Codex)
Testing spanned 32 expert-defined catalysis research tasks, resulting in 4,608 individual evaluation trials. Crucially, assessment went beyond plan generation:
✅ Validity: Does the proposed workflow pass formal verification?
✅ Dispatchability: Can it be successfully sent to the robot control layer?
✅ Executability: Does it run fully autonomously — without human intervention or repair?
Figure 3. AI agent performance across planning, validation, dispatch, and execution stages.
Harsh Reality: Only 3.3% Fully Autonomous Execution Rate
Results revealed a stark gap between linguistic fluency and physical reliability:
- Only 151 out of 4,608 workflows (3.3%) executed without manual correction.
- Best-performing configuration (Claude Code + Claude Opus 4.7) achieved 28.1% executability.
- Codex + GPT-5.5 reached 19.8%.
Parameter Tuning ≠ Scientific Redesign
In a five-round closed-loop experiment, Codex/GPT-5.5 demonstrated ability to adjust parameters (e.g., catalyst loading, reaction time) based on experimental feedback — but failed at higher-order reasoning:
- ✅ Adjusted material ratios and thermal profiles
- ❌ Retained original workflow skeleton across all rounds
- ❌ Never redesigned analytical methods
- ❌ Consistently omitted critical components (e.g., electrode binders, chromogenic reagents)
Key insight: Interpreting experimental evidence and tuning parameters is not equivalent to diagnosing flawed research strategy — nor does it imply capacity for scientific re-planning.
Long-Horizon Planning Remains a Critical Bottleneck
While some agents generated workflows up to 44 steps (validated by domain experts), only 3 workflows exceeded 30 steps across all tests. As task complexity increases, maintaining logical coherence and physical feasibility degrades significantly — exposing fundamental limitations in long-context reasoning and constraint-aware composition.
Rethinking AI’s Scientific Competence
The study proposes a tripartite framework to disambiguate commonly conflated capabilities:
| Level | Description | Current State |
|---|---|---|
| L1: Plan Generation | Linguistically coherent experimental protocol | ✅ Strong (LLMs excel here) |
| L2: Physical Execution | Workflow executable on real lab hardware | ⚠️ Limited (3.3% success rate) |
| L3: Strategy Revision | Redesigning core hypotheses & methodology post-evidence | ❌ Not observed |
Language-model-based planning does not guarantee reliable physical execution — and local optimization does not equate to scientific re-conceptualization.
From Testbed to Training Ground
This machine scientist infrastructure functions as both:
🔹 Testbed: Quantifying AI’s scientific competence via reproducible, physics-grounded metrics.
🔹 Training Ground: Feeding real-world failures, successful workflows, instrument telemetry, and expert annotations back into model training — enabling iterative alignment of AI agents with scientific practice.
The resulting “Plan → Execute → Observe → Replan” loop creates a self-improving cycle grounded in empirical reality — shifting AI-for-Science from speculative prototyping toward verifiable, evidence-driven autonomy.
Figure 4. Five-cycle iterative loop between AI agent and machine scientist on open-ended scientific problems.
Reference: arXiv:2607.23045
Article adapted from XinZhiYuan (New Intelligence Era).