How do AI agents perform when the task is real?
GEPHYRA is a family of benchmarks that measure AI agents in real-world settings, in the two ways people use them: reactive, when someone asks the agent to do something, and proactive, when the agent has to notice a problem and act without being asked.
The name is the Greek word for bridge (say JEF-ih-ruh): Grounded Evaluation of Proactive, Helpful Yet Reactive Agents.
First benchmark: GEPHYRA-Select
Organizations such as hospitals can train better models together than alone, but only with the right partners. A partner can look suitable on paper and still make your model worse. GEPHYRA-Select asks an AI agent to choose partners from the summaries organizations can share, and scores each choice against what actually happens when the models are trained.
More tracks, covering proactive behavior, multi-agent collaboration and security, will be announced as they are released.
Status
The first benchmark, GEPHYRA-Select, is being built. Code, data and results will be released here, on Hugging Face and on PyPI.