Repository navigation
ai: add Hybrid Search with SQL, full-text plus vector search in one query - #24202
Conversation
ti-chi-bot
Bot
added
contribution
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Title check |  |
The title clearly and concisely describes the main change: adding SQL-based hybrid search with full-text and vector search in one query. |
| Description check | |
The description covers the required change summary, affected TiDB version, reference link, AI involvement, and change-category checklist. It also provides detailed testing and reviewer context. |
| Docstring Coverage | |
No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0… |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
✨ Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository: pingcap/docs/.coderabbit.yaml
- Review profile: ASSERTIVE
- Plan: Advanced
- Run ID:
9f464575-05a6-454b-b4bb-794cbcab0def
📒 Files selected for processing (3)
TOC-ai.mdai/guides/vector-search-hybrid-search-sql.mdai/guides/vector-search-hybrid-search.md
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 2 remain after this review.
qiancai
added
translation/no-need
ti-chi-bot
Bot
removed
the
missing-translation-status
[LGTM Timeline notifier]Timeline:
|
|
|
||
| The following query runs both searches, ranks each result list, and merges them with Reciprocal Rank Fusion (RRF). Each row scores `1 / (60 + rank)` for every list it appears in, and the scores are added. A row that ranks well in both lists rises to the top. The constant `60` is the value commonly used for RRF; a larger value reduces the advantage of top-ranked rows over lower-ranked ones. | ||
|
|
||
| ```sql |
There was a problem hiding this comment.
The outer query starts from docs and LEFT JOINs the two Top-K result sets:
FROM docs d
LEFT JOIN vec ON vec.id = d.id
LEFT JOIN fts ON fts.id = d.idAlthough both search branches return at most 20 candidates, the outer query might still scan the entire docs table before filtering out non-matching rows.
This could become a significant performance bottleneck on large datasets, even when both search branches use their respective indexes.
I suggest rewriting the SQL as follows:
WITH
vec AS (
SELECT id,
ROW_NUMBER() OVER (ORDER BY distance, id) AS r
FROM (
SELECT /*+ READ_FROM_STORAGE(TIFLASH[docs]) */
id,
VEC_EMBED_COSINE_DISTANCE(
content_vector,
'vector index error 8200 partitioned table'
) AS distance
FROM docs
ORDER BY distance
LIMIT 20
) v
),
fts AS (
SELECT id,
ROW_NUMBER() OVER (ORDER BY score DESC, id) AS r
FROM (
SELECT id,
fts_match_word(
'vector index error 8200 partitioned table',
content
) AS score
FROM docs
WHERE fts_match_word(
'vector index error 8200 partitioned table',
content
)
ORDER BY score DESC
LIMIT 20
) f
),
hits AS (
SELECT id,
1.0 / (60 + r) AS contribution,
r AS vector_rank,
NULL AS fulltext_rank
FROM vec
UNION ALL
SELECT id,
1.0 / (60 + r),
NULL,
r
FROM fts
),
top_hits AS (
SELECT id,
SUM(contribution) AS rrf_score,
MAX(vector_rank) AS vector_rank,
MAX(fulltext_rank) AS fulltext_rank
FROM hits
GROUP BY id
ORDER BY rrf_score DESC, id
LIMIT 3
)
SELECT d.id, d.content,
t.rrf_score,
t.vector_rank,
t.fulltext_rank
FROM top_hits t
JOIN docs d ON d.id = t.id
ORDER BY t.rrf_score DESC, t.id;With this approach, the RRF fusion operates on at most 40 candidate rows. The final Top-K selection is performed logically before joining back to docs to retrieve document content.
This avoids the need to use the full docs table as the driving input for fusion.
|
@zhaoshangzi: adding LGTM is restricted to approvers and reviewers in OWNERS files. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@yongman: adding LGTM is restricted to approvers and reviewers in OWNERS files. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@JinheLin: adding LGTM is restricted to approvers and reviewers in OWNERS files. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
`WHERE fts_match_word(...) > 0` fails with ERROR 1221, because TiDB requires fts_match_word() to appear on its own in the WHERE clause. Without `> 0`, the query runs, uses both the vector index and the full-text index, and returns the documented output (verified on TiDB v8.5.3-serverless).
|
@qiancai: Your lgtm message is repeated, so it is ignored. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: qiancai The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
5 participants
What is changed, added or deleted? (Required)
Adds
ai/guides/vector-search-hybrid-search-sql.md, "Hybrid Search with SQL". The existing Hybrid Search guide uses thepytidbSDK, which fuses the two result lists in Python, and says you "can perform a search using SQL directly" without showing how. This page shows it: one table with an Auto Embedding column, a full-text index, and a vector index; vector search by text; full-text search; and both merged with Reciprocal Rank Fusion in a single SQL statement. It mirrors the existing "Full-Text Search with SQL" and "Full-Text Search with Python" pair.Also proposes one
TOC-ai.mdline under Hybrid Search. The Doc team decides placement.Also adds one sentence to the existing Hybrid Search guide (
ai/guides/vector-search-hybrid-search.md), where it already says you "can perform a search using SQL directly", linking to the new page.Measured on TiDB Cloud Starter,
8.0.11-TiDB-v8.5.3-serverless, AWS Oregon (us-west-2), 2026-10-06. Every SQL block was run verbatim and in order on a fresh database, with no statements in between, and each output table on the page is copied from that run.EXPLAINon the Step 5 query showsannIndex:on the vector search andtextSearch:(top20 ...)on the full-text search, so the plan uses both indexes. On a table this small the indexes had not been built yet (information_schema.tiflash_indexesreported no indexed stable rows), so "uses" here means the optimizer chose the index, not that rows were served from a built index.Reviewer note on the
READ_FROM_STORAGE(TIFLASH[docs])hint in Step 5. On a 6-row table, whether the optimizer used the vector index without the hint depended on the statistics state. Right after insert, with pseudo statistics, it used the index for both the Step 3 query and the Step 5 subquery. Once a real row count was recorded, an independent review saw it read from TiKV and skip the vector index for both, the top-level Step 3 query included. With the hint, every plan used both indexes. The page explains the hint and says it applies to any vector search on a small table. If the vector search owners prefer a different recommendation, the hint is the line to change.The same review saw full-text scores change a few minutes after
ANALYZE TABLE, once TiFlash moved the rows to its stable layer. With the original query text, the filler word "on" then let an unrelated row move up in the full-text ranking. I could not reproduce that shift in a second run, so the page now uses a query without filler words and adds a sentence saying scores can change on a freshly loaded table, which can reorder rows with similar scores. The new query gave the page's output in every run, but none of those runs reproduced the shift, so it is not proven to hold after one. In the review's shifted state the old query matched an unrelated row only through the word "on", which the new query does not contain.The availability wording in Prerequisites matches the existing Hybrid Search guide and Auto Embedding overview, except that "we are continuously rolling it out" became "is being rolled out", following the writing style guide's rule against "we".
Which TiDB version(s) do your changes apply to? (Required)
What is the related PR or file link(s)?
AI agent involvement
The text and the measurements were produced by an AI agent (Claude). I reviewed the changes and the evidence before opening this PR.
Do your changes match any of the following descriptions?
Summary by CodeRabbit