Test whether retrieved fields support valid joins, when composite keys need completion, and how permissions and multiple relationships should be handled. Contributed the design to an open-source handbook, then tested it with Python / SQLite.
For “sum unshipped amounts by customer,” retrieval may find quantity and unit price but miss tenant keys, order joins, dates, or statuses. The context is incomplete before SQL generation begins.
Goal: check whether retrieved tables, columns, and relationships support a valid query strategy, and explain what is missing.
Added Chinese and English guidance on schema selection and path evaluation to awesome-ai-roadmap; the PR was revised and merged after maintainer review.
Built an evaluator that reads SQLite schemas, retrieves candidate columns, completes approved relationships, and exports HTML / JSON comparisons with fixed reference-query results.
Development used AI assistance. The evaluator was published as a separate follow-up implementation.
Separate candidate selection, foreign-key path completion, and SQL-result checks to locate the failing stage. Permission filtering and ambiguity handling constrain execution; extra candidates add context cost.
Before merging, the maintainer clarified permission preservation during expansion, business semantics and cardinality checks, and scoring each valid strategy separately. These requirements also informed the implementation.
| Metric | Candidate retrieval only | Retrieval + path completion |
|---|---|---|
| Required-column recall, macro average | 58.2% | 86.0% |
| Mean candidate columns | 4.8 | 8.4 |
| Complete strategy coverage | 1 / 5 | 4 / 5 |
| Fixed reference SQL result match | 1 / 5 | 4 / 5 |
Run recorded September 29, 2026. Four additional permission, ambiguity, and disconnected-path cases returned the expected statuses after completion. Full data: JSON report 。
This tests a mechanism using curated aliases, synthetic data, and fixed reference SQL. Chinese and English order inputs represent the same business question. These figures are not model-generated SQL accuracy or real business impact.
The Chinese order question initially retrieved six columns. Path completion retained ten, covering all required columns for one strategy. The fixed reference query returned C1: 2 orders, 10 units, 7,000 cents; C2: 1 order, 2 units, 4,000 cents.
Composite keys are handled as complete relationships to avoid joining matching order IDs across tenants.
Explore candidates and query results →Two declared relationships connect logistics and warehouse tables. The system returns both candidate paths and an ambiguity status for clarification. Forbidden composite keys remain excluded during completion.
View the multiple-path case → · View the permissions case →Graph completion cannot infer date windows, status rules, or grouping. The case retains missing fields and a not-executed status. Lowering top-k to three reduced complete strategy coverage to 1/5, showing the effect of insufficient initial candidates.
View the incomplete-coverage case →The September 29, 2026 local verification used Python 3.11.9 and its standard library, with the commands below. This public page provides HTML and JSON reports.
python3 evaluate.py
python3 -m unittest -v Reports are generated in reports/ . Source includes synthetic data for eight tables, retrieval and graph expansion, labels, reference SQL, and 34 tests. No API key or database service is required.