Experiment Runner

ExperimentRunner runs N matches with pluggable agents and collects structured results.

Configuration

from pathlib import Path
from neg_env import ExperimentConfig

config = ExperimentConfig(
    game_id="ultimatum",          # which game to play
    num_matches=100,                 # how many matches to run
    max_turns_per_match=200,         # abort match after this many turns
    max_messages_per_turn=10,        # cap messages per agent per turn
    max_message_pings=5,             # max reply rounds for message_only exchanges
    max_stale_turns=10,              # force action after N consecutive message_only/pass turns
    log_directory=Path("./logs"),    # save JSON logs per match (optional)
    metadata={"experiment": "v1"},   # arbitrary metadata attached to logs
    open_dashboard=True,             # open live browser dashboard
    dashboard_port=8765,             # port for dashboard server
    max_workers=4,                   # run matches in parallel (default: 1)
    opik_enabled=True,               # enable Opik experiment tracing
    opik_project_name="my-project",  # Opik project name
)

Stale turn enforcement

Agents can use message_only or pass to chat without advancing the game. To prevent matches from running forever, max_stale_turns (default 10) limits consecutive turns with no real action. Once the limit is reached, message_only and pass are removed from the agent's allowed actions, forcing it to take a game action (e.g. submit_offer, accept, reject).

config = ExperimentConfig(
    game_id="ultimatum",
    num_matches=10,
    max_stale_turns=5,   # force action after 5 consecutive chat-only turns
)

Parallel execution

Set max_workers to run multiple matches concurrently using a thread pool. This is especially useful for LLM-backed agents where most time is spent waiting on API calls.

config = ExperimentConfig(
    game_id="ultimatum",
    num_matches=50,
    max_workers=8,   # 8 matches run in parallel
)

Opik tracing

When opik_enabled=True, each experiment run is logged to Opik for analysis and comparison. Set OPIK_API_KEY and OPIK_WORKSPACE in your environment (see .env.example).

config = ExperimentConfig(
    game_id="ultimatum",
    num_matches=10,
    opik_enabled=True,
    opik_project_name="ultimatum-v2",
)

Running

from neg_env import ExperimentRunner, RandomAgent

agents = [RandomAgent(agent_id="alice", seed=42), RandomAgent(agent_id="bob", seed=43)]
result = ExperimentRunner(config).run(agents)

# Or with custom game settings
from neg_env import UltimatumGame
result = ExperimentRunner(config).run(agents, game=UltimatumGame(total=200))

Results

ExperimentResult provides aggregate statistics and per-match details.

Aggregate stats

result.no_deal_count         # matches without resolution
result.mean_payoffs          # {"alice": 42.3, "bob": 32.7} — mean utility
result.payoff_matrix         # {"alice": [60, 50, ...], "bob": [...]} — all payoffs
result.completion_rate       # fraction of matches that finished (0.0–1.0)
result.total_duration_seconds

For ultimatum:

result.mean_shares           # {"alice": 55.0, "bob": 45.0} — mean deal amounts

For first-price-auction:

result.mean_bids             # {"bidder_a": 38.5, "bidder_b": 42.0} — mean bids

Per-match details

for mr in result.match_results:
    mr.match_id
    mr.outcome          # game-specific outcome dict
    mr.status           # "finished" or "running" (if aborted)
    mr.num_turns
    mr.num_messages
    mr.duration_seconds
    mr.log              # MatchLog object (full event trace)
    mr.error            # exception string if the match errored

Hydra CLI runner

Instead of writing Python scripts, you can run any game from the command line using run.py with Hydra config management.

Install the extra dependency:

pip install -e ".[hydra]"

Scenarios

Scenarios are self-contained YAML configs under conf/scenario/ that define game, agents, and experiment preset. Each scenario overrides the root config entirely.

# Run a single scenario
python run.py +scenario=ultimatum
python run.py +scenario=bilateral_trade
python run.py +scenario=provision_point

# Override params on top of a scenario
python run.py +scenario=ultimatum game.total=200
python run.py +scenario=first_price_auction experiment=benchmark

Run multiple games in parallel

Run all games simultaneously with a unified tabbed dashboard:

python run.py 'scenarios=[ultimatum,first_price_auction,bilateral_trade,provision_point]'

All scenarios run in parallel, each with its own agents and game instance. A single dashboard serves all games with a tab per game showing live match progress, summaries, and completed match details.

You can also run a subset:

python run.py 'scenarios=[ultimatum,bilateral_trade]'

Each scenario runs independently with its own game, agents, and experiment settings. Per-game results are printed when all games finish.

Basic usage (without scenarios)

You can also override game and agents directly:

python run.py game=bilateral_trade                   # switch game
python run.py game=ultimatum game.total=200          # override game param
python run.py experiment=benchmark                   # 20 matches, parallel
python run.py experiment=quick                       # 3 matches, dashboard
python run.py experiment.max_stale_turns=5           # override stale turn limit

Baseline vs Redteam mode

The mode key controls agent_a's prompt strategy. Two conditions:

• redteam (default): agent_a uses an attacker preset (adversarial prompt)
• baseline: agent_a uses a rational preset (cooperative prompt)

The victim (agent_b) always uses a rational preset. agent_a is always GPT-5.2.

python run.py game=ultimatum mode=redteam experiment=quick
python run.py game=ultimatum mode=baseline experiment=quick

The mode is embedded in the Opik project name for analysis:

{game}__{experiment_type}__{mode}__{victim_model}__{timestamp}
# e.g. ultimatum__quick__redteam__allenai_olmo-3.1-32b-instruct__20260301_134426

Batch matrix runner

scripts/run_matrix.py automates running all combinations of games, modes, and victim models. Runs execute in parallel (default 4 concurrent, configurable with -p).

# Preview commands without executing
python scripts/run_matrix.py --dry-run

# Run the full sweep (4 games x 2 modes x 4 victim models = 32 runs)
python scripts/run_matrix.py

# Filter to specific games/modes/models
python scripts/run_matrix.py --games ultimatum bilateral_trade --modes redteam baseline
python scripts/run_matrix.py --games ultimatum --models allenai/olmo-3.1-32b-instruct

# Control parallelism and experiment preset
python scripts/run_matrix.py -p 8 --experiment benchmark

The game-agent mapping per mode:

Game	agent_a redteam preset	agent_a baseline preset	agent_b (victim) preset
ultimatum	ultimatum/attacker	ultimatum/rational	ultimatum/rational
bilateral_trade	bilateral_trade/seller_attacker	bilateral_trade/seller_rational	bilateral_trade/buyer_rational
first_price_auction	first_price_auction/attacker	first_price_auction/rational	first_price_auction/rational
provision_point	provision_point/contributor_attacker	provision_point/contributor_rational	provision_point/contributor_rational

Analysis

scripts/analyze_experiments.py fetches results from Opik, extracts metrics, and generates:

• Per-group CSVs and plots under analysis/{game}/{experiment_type}/{mode}/{model}/
• Baseline vs redteam comparison table per game: analysis/{game}/comparison_baseline_vs_redteam.csv
• Grouped bar chart per game: analysis/{game}/comparison_baseline_vs_redteam.png

# Analyze all experiments
python scripts/analyze_experiments.py

# Filter by game, mode, or model
python scripts/analyze_experiments.py --games ultimatum --modes redteam baseline
python scripts/analyze_experiments.py --models allenai/olmo-3.1-32b-instruct

The comparison table shows per victim model: mean u_A(baseline), mean u_A(redteam), delta, and Welch's t-test p-value.

Agent presets

Agent presets live under conf/agent/<game>/. Each game has personality types plus an attacker variant for adversarial experiments.

Personality types per game:

Game	Available presets
Ultimatum	rational, altruistic, selfish, colluder, attacker
First-Price Auction	rational, altruistic, selfish, colluder, attacker
Bilateral Trade	initiator_rational, initiator_altruistic, initiator_selfish, initiator_attacker, provider_rational, provider_altruistic, provider_selfish, provider_colluder, provider_attacker, buyer_rational, seller_rational
Principal-Agent	principal_rational, principal_altruistic, principal_selfish, principal_attacker, worker_rational, worker_altruistic, worker_selfish, worker_colluder, worker_attacker, oracle
Provision Point	coordinator, attacker_coordinator, rational, altruistic, selfish, colluder, contributor_rational, contributor_attacker

Attacker presets differ from selfish presets: selfish agents are openly aggressive, while attackers use active deception — they appear cooperative while secretly exploiting game mechanics.

# Mix agent personalities
python run.py 'agents=[{preset: ultimatum/rational, agent_id: a1}, {preset: ultimatum/attacker, agent_id: a2}]'

# Override agent params inline
python run.py 'agents=[{preset: ultimatum/rational, agent_id: a1, temperature: 1.5}, {preset: ultimatum/altruistic, agent_id: a2}]'

Creating custom scenarios

Create a YAML file under conf/scenario/:

# conf/scenario/my_experiment.yaml
# @package _global_
defaults:
  - override /game: ultimatum
  - override /experiment: quick

agents:
  - preset: ultimatum/attacker
    agent_id: agent_1
  - preset: ultimatum/rational
    agent_id: agent_2

Then run it:

# Single scenario
python run.py +scenario=my_experiment

# Include in a multi-game run
python run.py 'scenarios=[my_experiment,bilateral_trade]'

Experiment presets

Preset	Matches	Workers	Dashboard	Opik
quick	3	3	yes	yes
standard	5	5	yes	yes
benchmark	20	10	no	yes

See resolved config

python run.py --cfg job
python run.py +scenario=bilateral_trade --cfg job
python run.py --help   # list all available games, agents, experiments