Examples Overview

Real-world examples of using SkyDiscover across different optimization domains

SkyDiscover includes ~200 benchmarks across math, systems, algorithms, and reasoning domains. Each benchmark demonstrates how to set up and run evolutionary search for different types of optimization problems.

Example Categories

Math Optimization

Circle packing, Heilbronn problems, autocorrelation inequalities, and geometric optimization

Systems Optimization

Cloud scheduling, load balancing, model placement, and database optimization

Algorithm Design

Competitive programming problems from Frontier-CS benchmark (172 tasks)

Custom Problems

Learn how to create your own benchmarks with custom evaluators

Quick Start

All benchmarks follow a consistent structure:

cd benchmarks/math/circle_packing

uv run skydiscover-run \
  initial_program.py \
  evaluator.py \
  -c config.yaml \
  -s adaevolve \
  -i 100

Replace adaevolve with your search algorithm: evox, openevolve, gepa, or shinkaevolve

Benchmark Structure

Every benchmark contains three core files:

1. Initial Program

   The starting solution with an EVOLVE-BLOCK marking the code to be evolved:

   initial_program.py

   # EVOLVE-BLOCK-START
   def solve(input_data):
       # Your initial solution here
       return result
   # EVOLVE-BLOCK-END
   
2. Evaluator

   A scoring function that returns a combined_score (higher is better):

   evaluator.py

   def evaluate(program_path: str) -> dict:
       # Load and run the program
       # Compute performance metrics
       return {"combined_score": 0.73, ...}
   
3. Configuration

   System prompt and search settings:

   config.yaml

   system_prompt: "Optimize the circle packing algorithm..."
   language: python
   diff_based_generation: true
   

Available Benchmarks

Domain	Benchmarks	Example Problems
Math	14 tasks	Circle packing, Erdos problems, Heilbronn triangle
Systems	5 tasks	Cloud routing, MoE load balancing, GPU scheduling
GPU	4 tasks	Triton kernel optimization (vecadd, matmul)
Algorithms	172 tasks	Competitive programming (Frontier-CS)
Reasoning	Multiple	ARC-AGI visual reasoning
Prompts	1 task	Natural language prompt evolution (HotPotQA)

Installation

Install dependencies based on which benchmarks you want to run:

Base

uv sync

Some benchmarks may have additional requirements.txt files in their directories. Install these with:

uv pip install -r benchmarks/<task>/requirements.txt

Environment Setup

Set your API key before running:

export OPENAI_API_KEY="sk-..."

Next Steps

Math Examples

Explore mathematical optimization problems

Systems Examples

Learn about systems optimization tasks

Create Custom

Build your own benchmark

View Benchmarks

Browse all benchmarks on GitHub