Schema Selection Evaluator · 可复现实验

方法	平均必要列召回	平均候选列数	完整策略	参考 SQL 结果匹配	边界状态符合预期
仅候选检索	58.2%	4.8	1 / 5	1 / 5	1 / 4
检索 + 路径补全	86.0%	8.4	4 / 5	4 / 5	4 / 4

orders_zh

按客户编号汇总下单时间为上海9月1日至7日、排除取消订单的未发金额和件数

联合租户/订单键补全；时间半开区间、取消与粒度

仅候选检索

ready · 6 列

必要列召回 60.0% · 完整策略否

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "order_lines",
    "orders"
  ],
  "columns": [
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.customer_id",
    "orders.status"
  ],
  "relationships": [],
  "candidate_columns": 6,
  "notes": [],
  "coverage": {
    "strategy": "aggregate-lines-per-order",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 0.6,
    "missing_columns": [
      "order_lines.order_id",
      "order_lines.tenant_id",
      "orders.order_id",
      "orders.tenant_id"
    ],
    "relationships_recall": 0.0,
    "missing_relationships": [
      "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)"
    ],
    "element_recall": 0.6153846153846154,
    "complete": false,
    "extra_columns": []
  },
  "reference_query": {
    "status": "not_run_incomplete_context",
    "expected_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ]
  }
}

路径补全后

ready · 10 列

必要列召回 100.0% · 完整策略是

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "order_lines",
    "orders"
  ],
  "columns": [
    "order_lines.order_id",
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.tenant_id",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.customer_id",
    "orders.order_id",
    "orders.status",
    "orders.tenant_id"
  ],
  "relationships": [
    "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)"
  ],
  "candidate_columns": 10,
  "notes": [
    "Added FK endpoints: order_lines.order_id, order_lines.tenant_id, orders.order_id, orders.tenant_id"
  ],
  "coverage": {
    "strategy": "aggregate-lines-per-order",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 1.0,
    "missing_columns": [],
    "relationships_recall": 1.0,
    "missing_relationships": [],
    "element_recall": 1.0,
    "complete": true,
    "extra_columns": []
  },
  "reference_query": {
    "status": "matched",
    "expected_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ],
    "actual_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ]
  }
}

orders_en

By customer id, sum remaining amount and quantity for order date September 1–7 Shanghai, excluding cancelled orders

同一业务问题的英文词表覆盖

仅候选检索

ready · 6 列

必要列召回 60.0% · 完整策略否

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "order_lines",
    "orders"
  ],
  "columns": [
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.customer_id",
    "orders.status"
  ],
  "relationships": [],
  "candidate_columns": 6,
  "notes": [],
  "coverage": {
    "strategy": "aggregate-lines-per-order",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 0.6,
    "missing_columns": [
      "order_lines.order_id",
      "order_lines.tenant_id",
      "orders.order_id",
      "orders.tenant_id"
    ],
    "relationships_recall": 0.0,
    "missing_relationships": [
      "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)"
    ],
    "element_recall": 0.6153846153846154,
    "complete": false,
    "extra_columns": []
  },
  "reference_query": {
    "status": "not_run_incomplete_context",
    "expected_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ]
  }
}

路径补全后

ready · 10 列

必要列召回 100.0% · 完整策略是

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "order_lines",
    "orders"
  ],
  "columns": [
    "order_lines.order_id",
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.tenant_id",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.customer_id",
    "orders.order_id",
    "orders.status",
    "orders.tenant_id"
  ],
  "relationships": [
    "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)"
  ],
  "candidate_columns": 10,
  "notes": [
    "Added FK endpoints: order_lines.order_id, order_lines.tenant_id, orders.order_id, orders.tenant_id"
  ],
  "coverage": {
    "strategy": "aggregate-lines-per-order",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 1.0,
    "missing_columns": [],
    "relationships_recall": 1.0,
    "missing_relationships": [],
    "element_recall": 1.0,
    "complete": true,
    "extra_columns": []
  },
  "reference_query": {
    "status": "matched",
    "expected_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ],
    "actual_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ]
  }
}

customer_product

按客户名称和商品名称汇总下单时间为上海9月1日至7日、排除取消的未发金额

沿客户—订单—明细—商品补齐关系与键

仅候选检索

ready · 7 列

必要列召回 41.2% · 完整策略否

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "customers",
    "order_lines",
    "orders",
    "products"
  ],
  "columns": [
    "customers.name",
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.status",
    "products.name"
  ],
  "relationships": [],
  "candidate_columns": 7,
  "notes": [],
  "coverage": {
    "strategy": "customer-order-line-product",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 0.4117647058823529,
    "missing_columns": [
      "customers.customer_id",
      "customers.tenant_id",
      "order_lines.order_id",
      "order_lines.product_id",
      "order_lines.tenant_id",
      "orders.customer_id",
      "orders.order_id",
      "orders.tenant_id",
      "products.product_id",
      "products.tenant_id"
    ],
    "relationships_recall": 0.0,
    "missing_relationships": [
      "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)",
      "order_lines(tenant_id,product_id)->products(tenant_id,product_id)",
      "orders(tenant_id,customer_id)->customers(tenant_id,customer_id)"
    ],
    "element_recall": 0.4583333333333333,
    "complete": false,
    "extra_columns": []
  },
  "reference_query": {
    "status": "not_run_incomplete_context",
    "expected_rows": [
      [
        "Acme",
        "Bolt",
        7000
      ],
      [
        "Beta",
        "Nut",
        4000
      ]
    ]
  }
}

路径补全后

ready · 17 列

必要列召回 100.0% · 完整策略是

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "customers",
    "order_lines",
    "orders",
    "products"
  ],
  "columns": [
    "customers.customer_id",
    "customers.name",
    "customers.tenant_id",
    "order_lines.order_id",
    "order_lines.ordered_qty",
    "order_lines.product_id",
    "order_lines.shipped_qty",
    "order_lines.tenant_id",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.customer_id",
    "orders.order_id",
    "orders.status",
    "orders.tenant_id",
    "products.name",
    "products.product_id",
    "products.tenant_id"
  ],
  "relationships": [
    "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)",
    "order_lines(tenant_id,product_id)->products(tenant_id,product_id)",
    "orders(tenant_id,customer_id)->customers(tenant_id,customer_id)"
  ],
  "candidate_columns": 17,
  "notes": [
    "Added FK endpoints: customers.customer_id, customers.tenant_id, order_lines.order_id, order_lines.product_id, order_lines.tenant_id, orders.customer_id, orders.order_id, orders.tenant_id, products.product_id, products.tenant_id"
  ],
  "coverage": {
    "strategy": "customer-order-line-product",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 1.0,
    "missing_columns": [],
    "relationships_recall": 1.0,
    "missing_relationships": [],
    "element_recall": 1.0,
    "complete": true,
    "extra_columns": []
  },
  "reference_query": {
    "status": "matched",
    "expected_rows": [
      [
        "Acme",
        "Bolt",
        7000
      ],
      [
        "Beta",
        "Nut",
        4000
      ]
    ],
    "actual_rows": [
      [
        "Acme",
        "Bolt",
        7000
      ],
      [
        "Beta",
        "Nut",
        4000
      ]
    ]
  }
}

stock

列出商品名称和库存

单表问题不需要路径扩展

仅候选检索

ready · 2 列

必要列召回 100.0% · 完整策略是

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "products"
  ],
  "columns": [
    "products.available_qty",
    "products.name"
  ],
  "relationships": [],
  "candidate_columns": 2,
  "notes": [],
  "coverage": {
    "strategy": "scoped-stock",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 1.0,
    "missing_columns": [],
    "relationships_recall": 1.0,
    "missing_relationships": [],
    "element_recall": 1.0,
    "complete": true,
    "extra_columns": []
  },
  "reference_query": {
    "status": "matched",
    "expected_rows": [
      [
        "Bolt",
        30
      ],
      [
        "Nut",
        40
      ]
    ],
    "actual_rows": [
      [
        "Bolt",
        30
      ],
      [
        "Nut",
        40
      ]
    ]
  }
}

路径补全后

ready · 2 列

必要列召回 100.0% · 完整策略是

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "products"
  ],
  "columns": [
    "products.available_qty",
    "products.name"
  ],
  "relationships": [],
  "candidate_columns": 2,
  "notes": [],
  "coverage": {
    "strategy": "scoped-stock",
    "tables_recall": 1.0,
    "missing_tables": [],
    "columns_recall": 1.0,
    "missing_columns": [],
    "relationships_recall": 1.0,
    "missing_relationships": [],
    "element_recall": 1.0,
    "complete": true,
    "extra_columns": []
  },
  "reference_query": {
    "status": "matched",
    "expected_rows": [
      [
        "Bolt",
        30
      ],
      [
        "Nut",
        40
      ]
    ],
    "actual_rows": [
      [
        "Bolt",
        30
      ],
      [
        "Nut",
        40
      ]
    ]
  }
}

underspecified

未发金额

故意缺少分组/日期/状态描述，图扩展不能补出业务口径

仅候选检索

ready · 3 列

必要列召回 30.0% · 完整策略否

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "order_lines"
  ],
  "columns": [
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.unit_price_cents"
  ],
  "relationships": [],
  "candidate_columns": 3,
  "notes": [],
  "coverage": {
    "strategy": "aggregate-lines-per-order",
    "tables_recall": 0.5,
    "missing_tables": [
      "orders"
    ],
    "columns_recall": 0.3,
    "missing_columns": [
      "order_lines.order_id",
      "order_lines.tenant_id",
      "orders.created_at",
      "orders.customer_id",
      "orders.order_id",
      "orders.status",
      "orders.tenant_id"
    ],
    "relationships_recall": 0.0,
    "missing_relationships": [
      "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)"
    ],
    "element_recall": 0.3076923076923077,
    "complete": false,
    "extra_columns": []
  },
  "reference_query": {
    "status": "not_run_incomplete_context",
    "expected_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ]
  }
}

路径补全后

ready · 3 列

必要列召回 30.0% · 完整策略否

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "order_lines"
  ],
  "columns": [
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.unit_price_cents"
  ],
  "relationships": [],
  "candidate_columns": 3,
  "notes": [],
  "coverage": {
    "strategy": "aggregate-lines-per-order",
    "tables_recall": 0.5,
    "missing_tables": [
      "orders"
    ],
    "columns_recall": 0.3,
    "missing_columns": [
      "order_lines.order_id",
      "order_lines.tenant_id",
      "orders.created_at",
      "orders.customer_id",
      "orders.order_id",
      "orders.status",
      "orders.tenant_id"
    ],
    "relationships_recall": 0.0,
    "missing_relationships": [
      "order_lines(tenant_id,order_id)->orders(tenant_id,order_id)"
    ],
    "element_recall": 0.3076923076923077,
    "complete": false,
    "extra_columns": []
  },
  "reference_query": {
    "status": "not_run_incomplete_context",
    "expected_rows": [
      [
        "C1",
        2,
        10,
        7000
      ],
      [
        "C2",
        1,
        2,
        4000
      ]
    ]
  }
}

denied_salary

列出员工工资 salary

禁止表在排序前被排除

仅候选检索

no_candidates · 0 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "no_candidates",
  "tables": [],
  "columns": [],
  "relationships": [],
  "candidate_columns": 0,
  "notes": [],
  "status_matches_expectation": true
}

路径补全后

no_candidates · 0 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "no_candidates",
  "tables": [],
  "columns": [],
  "relationships": [],
  "candidate_columns": 0,
  "notes": [],
  "status_matches_expectation": true
}

ambiguous_warehouse

列出物流跟踪号和仓库名称

起始仓与目的仓两条关系需要澄清

仅候选检索

ready · 2 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "shipments",
    "warehouses"
  ],
  "columns": [
    "shipments.tracking_number",
    "warehouses.name"
  ],
  "relationships": [],
  "candidate_columns": 2,
  "notes": [],
  "status_matches_expectation": false
}

路径补全后

ambiguous_path · 2 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ambiguous_path",
  "tables": [
    "shipments",
    "warehouses"
  ],
  "columns": [
    "shipments.tracking_number",
    "warehouses.name"
  ],
  "relationships": [],
  "candidate_columns": 2,
  "notes": [
    "shipments(tenant_id,origin_id)->warehouses(tenant_id,warehouse_id)",
    "shipments(tenant_id,destination_id)->warehouses(tenant_id,warehouse_id)"
  ],
  "status_matches_expectation": true
}

disconnected_invoice

按客户名称汇总发票金额

没有维护过的关系时不按同名字段猜连接

仅候选检索

ready · 2 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "customers",
    "invoices"
  ],
  "columns": [
    "customers.name",
    "invoices.amount_cents"
  ],
  "relationships": [],
  "candidate_columns": 2,
  "notes": [],
  "status_matches_expectation": false
}

路径补全后

no_authorized_path · 2 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "no_authorized_path",
  "tables": [
    "customers",
    "invoices"
  ],
  "columns": [
    "customers.name",
    "invoices.amount_cents"
  ],
  "relationships": [],
  "candidate_columns": 2,
  "notes": [
    "customers → invoices: no approved, authorized path"
  ],
  "status_matches_expectation": true
}

denied_join_key

按客户编号汇总下单时间为上海9月1日至7日、排除取消订单的未发金额和件数

补全不能重新引入被禁止的联合键列

仅候选检索

ready · 6 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "ready",
  "tables": [
    "order_lines",
    "orders"
  ],
  "columns": [
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.customer_id",
    "orders.status"
  ],
  "relationships": [],
  "candidate_columns": 6,
  "notes": [],
  "status_matches_expectation": false
}

路径补全后

no_authorized_path · 6 列

边界用例：单独验收状态

查看候选、遗漏、路径与 SQL 结果

{
  "status": "no_authorized_path",
  "tables": [
    "order_lines",
    "orders"
  ],
  "columns": [
    "order_lines.ordered_qty",
    "order_lines.shipped_qty",
    "order_lines.unit_price_cents",
    "orders.created_at",
    "orders.customer_id",
    "orders.status"
  ],
  "relationships": [],
  "candidate_columns": 6,
  "notes": [
    "order_lines → orders: no approved, authorized path"
  ],
  "status_matches_expectation": true
}

实验边界

[
  "Hand-authored aliases and fixtures; no model or embedding retrieval, no held-out benchmark.",
  "Fixed reference SQL only: result matches are not generated-SQL accuracy.",
  "Graph expansion checks approved connectivity, not automatic business intent or cardinality inference.",
  "The English and Chinese order cases are two language versions of the same business task.",
  "underspecified uses a known target as a diagnostic; its missing intent cannot be inferred from the question."
]

完整机器可读结果：report.json；复现：python3 evaluate.py --top-k 8