Methodology

Evaluation scope

ICLayout-Bench evaluates how AI agents turn circuit requirements into physical layouts. Each task defines the supplied circuit, physical and electrical requirements, operating conditions and scoring rule.

Each dataset has a core collection used for its main score. Public and private scores are reported separately, with no combined score. The public circuit library also includes tasks outside the core evaluation.

Scores

Each circuit receives a score from 0 to 100 under its declared scoring rule. Higher values indicate better layout quality. Physical and electrical requirements act as gates; a valid circuit can still have room to improve in electrical performance or area.

Model summaries use the selected experiment results, grouped by harness, model and effort. Recorded zero scores are included. Complete configurations are compared on the same core tasks with known scores, following the manuscript analysis. The same score is used throughout the overview, leaderboard, model profile and comparison pages.

Coverage shows how many core tasks have a recorded score. A configuration with incomplete coverage can show its available scores, but remains unranked. Ties share a rank. Public and private ranking eligibility are independent.

The selected experiment data already identifies the results used for these summaries. Historical result pages remain available as evidence; browsing them does not change the main score.

Validity

Validity is the percentage of selected core tasks whose recorded outcome is pass. It measures how often a configuration satisfies the task’s required checks, independently of how high its quality score is.

Validity is shown when all selected tasks have finished. A recorded failure stays in the denominator.

Cost & Tokens

Cost per task is the mean recorded USD cost over the same tasks used for the dataset score. Output tokens per task is the mean recorded output token count on those tasks. These values describe the selected experiments, rather than current model list prices.

Resource averages require a recorded value for every included task. A missing value stays unavailable; it is not replaced with zero. Costs in other currencies are not converted. Public and private resource summaries remain separate.

In the score-versus-resource charts, higher scores and lower resource use are preferable. Positive costs use a logarithmic axis; recorded zero costs use a linear axis so zero stays visible. Public and private views share the same axis range for the current model selection. The dashed Pareto frontier connects complete configurations for which no other complete configuration in the visible selection has an equal or higher score at an equal or lower resource use, with at least one strict improvement. The frontier updates with the dataset, metric and model filters. Select a point to inspect its model configuration.

Result states

Scores, result states and evaluation modes retain their recorded meanings. The website does not rerun the evaluator or infer missing scores.

Evaluation conditions

Model and effort labels identify the evaluated configuration. Harness labels distinguish the agent environment. A leaderboard row uses that same configuration for both datasets.

Evaluation labels describe how a result was produced; they do not certify the remote model’s identity. Result records and downloadable assets are reviewed separately.