docs: add the 15.8 content chunker and semantic search guide (7 languages) - #463
Open
marevol wants to merge 18 commits into
Open
docs: add the 15.8 content chunker and semantic search guide (7 languages)#463marevol wants to merge 18 commits into
marevol wants to merge 18 commits into
Conversation
- Add ja/15.8/config/search-semantic.rst covering the core content chunk + vector (semantic) search feature: overview, prerequisites, config key reference (system.properties vs fess_config.properties channels), setup with the built-in OpenSearch ML Commons provider, reindex and scheduler job steps, search integration (exact/ann, min_score, limitations), AI search (RAG) chunk selection, migration from the deprecated fess-webapp-semantic-search plugin, and operational notes. - Register the new page in the config index toctree. - Update rank-fusion.rst and llm-overview.rst to point to the built-in semantic searcher instead of the deprecated plugin. - Mark fess-webapp-semantic-search as deprecated in webapp-plugin.rst.
The content_chunker.embedding.name default changed from ollama to the built-in opensearch (ML Commons) provider in fess core.
Update the Japanese content-chunk / semantic search guide to match the new static index-mapping design: reindexing is no longer required for fresh installs, the embedded OpenSearch now bundles a lucene-only kNN plugin, all content_chunker.* settings live in system.properties, and the scheduled indexer job auto-registers at startup. Reorganize the upgrade path into four explicit scenarios (fresh install, existing install without vectors, existing install adding vectors, and former fess-webapp-semantic-search plugin users), document previously-missing config keys (content_chunker.job.retry_failed, corrected defaults for embedding.dimension and job.max_documents_per_run), and remove the knn.param.m/ef_construction keys that are no longer configurable. Also add a pointer from the upgrade guide to the semantic search migration steps, since chunk-vector search requires an explicit reindex that a plain recrawl does not perform.
The JVM system property example still recommended -Drank.fusion.searchers=default,semantic, the exact leftover value that now triggers a startup warning: the core searcher registers as semantic_chunk, not semantic (a name owned by the retired fess-webapp-semantic-search plugin), so this value silently excludes core semantic chunk search from Rank Fusion. Replace the example with a working one, document how searcher names are derived, and add a warning explaining the semantic vs semantic_chunk distinction and how to remove the leftover setting.
The multi-node note recommended pinning the Content Chunk Vector Indexer job to one node but never said how. Add the two required, non-obvious steps verified against FessProp.isSchedulerTarget and ScriptExecutorJob.process(): setting a distinct scheduler.target.name on the node that should run it, and changing the job's target field from "all" to that same name (the field ignores scheduler.target.name entirely while set to "all"). Cross-reference the scheduler guide for the target field itself.
RST does not support nesting simple inline markup, so the double-backtick code spans for ``all`` and ``scheduler.target.name`` inside the **bold** warning sentence rendered as literal backtick characters instead of <code> spans. Rewrite the sentence so the bold span and the code spans no longer overlap. Verified with a docutils doctree walk (no strong/emphasis node contains a literal backtick) and by rendering the html5 output directly across all three files touched in this branch.
Port the Japanese 15.8 semantic search / content chunking documentation to English: new search-semantic.rst covering configuration, setup, processing status, Rank Fusion behavior, AI search mode integration, and the four migration paths from 15.7 (or from the deprecated fess-webapp-semantic-search plugin). Update rank-fusion.rst (correct searcher name to semantic_chunk, add a migration warning), llm-overview.rst, webapp-plugin.rst, upgrade.rst, and config/index.rst to cross-reference the new page.
The Maintenance screen (Re-indexing) is under the System Info sidebar category, not System, and the visible system.properties content is the "App Properties" panel under System Info > Config Info, not a "System Properties" screen under General (that panel actually shows JVM system properties, a different dataset). Fix all four occurrences in search-semantic.rst and the one in upgrade.rst to match the actual admin UI labels.
- ja/15.8/config/search-semantic.rst: fix a sixth occurrence of the
same wrong-panel reference ("system properties screen") that the
previous commit corrected in four other spots; this one lacked the
"admin screen > ..." prefix so it wasn't caught by the earlier grep.
- en/15.8/config/search-semantic.rst: correct "millions to tens of
millions" back to the Japanese source's actual bound of several
million to 10 million documents, so operators don't read the
guidance as applying only well past 10M documents.
…ed value The several-million-to-10-million-document range for raising jvm.chunk.options -Xmx is an extrapolation from a rough per-pending-ID memory estimate, not a measured threshold; actual usage depends on document ID length, which varies by site since Fess IDs are URL-derived. Add "を目安に" / "As a guideline" so readers don't treat the number as validated, matching how this docset already hedges similar sizing guidance (e.g. install-linux.rst's memory recommendation). Numbers unchanged.
Port the German translation of the chunk-vector (semantic chunk search) documentation from the reviewed English source: new search-semantic.rst, a toctree entry, an llm-overview.rst wording update, the rank-fusion.rst searcher-name/allowlist correction with its new warning block, a webapp-plugin.rst deprecation note, and an upgrade.rst signpost to the 15.7-to-15.8 migration section.
Port the Spanish translation of the chunk-vector (semantic chunk search) documentation from the reviewed English source: new search-semantic.rst, a toctree entry, an llm-overview.rst wording update, the rank-fusion.rst searcher-name/allowlist correction with its new warning block, a webapp-plugin.rst deprecation note, and an upgrade.rst signpost to the 15.7-to-15.8 migration section.
Port the French translation of the chunk-vector (semantic chunk search) documentation from the reviewed English source: new search-semantic.rst, a toctree entry, an llm-overview.rst wording update, the rank-fusion.rst searcher-name/allowlist correction with its new warning block, a webapp-plugin.rst deprecation note, and an upgrade.rst signpost to the 15.7-to-15.8 migration section.
Port the Korean translation of the chunk-vector (semantic chunk search) documentation from the reviewed English source: new search-semantic.rst, a toctree entry, an llm-overview.rst wording update, the rank-fusion.rst searcher-name/allowlist correction with its new warning block, a webapp-plugin.rst deprecation note, and an upgrade.rst signpost to the 15.7-to-15.8 migration section.
Port the Simplified Chinese translation of the chunk-vector (semantic chunk search) documentation from the reviewed English source: new search-semantic.rst, a toctree entry, an llm-overview.rst wording update, the rank-fusion.rst searcher-name/allowlist correction with its new warning block, a webapp-plugin.rst deprecation note, and an upgrade.rst signpost to the 15.7-to-15.8 migration section.
…ch guide Final review pass across all seven languages for the chunk-vector/semantic search guide: - Correct the k-NN prerequisites table: RPM/DEB packages always connect to a separately installed external OpenSearch and never embed one, so they belong in the external-OpenSearch row (full faiss/nmslib support), not the embedded row (lucene-only, silent data loss on faiss/nmslib). The embedded row is now scoped to bin/fess and the TAR.GZ/ZIP default. - Document that re-indexing during migration runs as an asynchronous background operation with no completion notification, that the old index is retained and must be deleted manually, and that index disk usage roughly doubles until it is. Reorder the migration steps so the indexer job is only started after the alias swap is confirmed. - Note that with a metered embedding provider (openai, gemini), the first indexer run bills the entire existing corpus at once unless a finite content_chunker.job.max_documents_per_run is set. - Clarify that the opensearch-provider username/password fallback to the search engine's own credentials only applies while api.url is left unconfigured, matching the shipped fess_config.properties documentation.
…ion guide The overview paragraph in zh-cn/15.8/config/rank-fusion.rst had two bold spans immediately adjacent to CJK characters with no escaping. docutils requires an inline end-string to be followed by whitespace or specific punctuation; a bare CJK ideograph does not qualify, so the closing markup silently failed to match. The parser instead extended a single strong span across both intended bold runs, leaving the interior "**" and the |Fess| substitution as literal, unsubstituted text -- with no docutils diagnostic at all. Fixed with the same backslash-space escape already used elsewhere in this branch for CJK-adjacent markup.
…mantic search guide Second review pass across all seven languages: - Replace the alias-swap verification guidance: _cat/indices only shows that the new index exists, not which index the fess.search/fess.update aliases point to. Point operators at _cat/aliases instead, and stop presenting a quiet Fess log as proof of success -- the reindex code only logs a warning on failure, so silence only means no known failure occurred, matching the existing convention already used in admin-index-backup.rst. - Align the k-NN prerequisites table and configuration reference with the ANN allow-list the indexer now enforces: content_chunker.search.knn.engine accepts only lucene/faiss (nmslib is never valid for this nested field and is rejected regardless of deployment), content_chunker.search.knn.method accepts only hnsw, content_chunker.search.knn.space_type accepts only cosinesimil/innerproduct/l2, and content_chunker.embedding.dimension is capped at 16000. Any other value now falls back to its default with a warning instead of silently reaching the mapping. - Document the minimum external OpenSearch version (2.19.0): the shipped index settings always send knn.derived_source.enabled, which older external clusters do not recognize, causing index creation to fail independent of whether the k-NN plugin itself is installed.
marevol
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the content chunker and semantic (vector) search feature being integrated into Fess core in codelibs/fess#3184, in all seven documentation languages.
This started as a Japanese-only draft. The core implementation has changed substantially since then — most importantly, a fresh installation no longer needs a reindex — so this PR both corrects the original Japanese page and ports the result to the remaining six languages.
Changes
Per language (
ja,en,de,es,fr,ko,zh-cn):<lang>/15.8/config/search-semantic.rst— overview and processing flow; prerequisites by deployment shape; configuration reference; setup walkthrough (ML Commons model register/deploy, reindex, the Content Chunk Vector Indexer job,content_chunk_statusvalues); how semantic search behaves (exact/ANN selection,min_score, query-syntax limitations, BM25 degradation guarantee); AI search (RAG) integration; a four-pattern migration section; notes on dimension changes, disk usage and chunk-only modeconfig/index.rst— toctree registration next torank-fusionconfig/rank-fusion.rst,config/llm-overview.rst— replace stalefess-webapp-semantic-searchplugin instructions with pointers to the built-in featuredev/webapp-plugin.rst— mark the plugin as deprecated (integrated into core in 15.8)install/upgrade.rst— signpost the migration sectionCorrections to the original Japanese draft
The draft was written against the earlier design, in which the index definitions were spliced in at runtime. These statements were wrong at head and are now fixed:
content_chunker.job.max_documents_per_runnow defaults to unlimited, not10000content_chunker.embedding.dimensionhas a default (768)content_chunker.*keys are read fromsystem.properties. The reference previously split them by channel, andcontent_chunker.embedding.opensearch.*— includingmodel.id— was documented as living infess_config.properties, where the admin System Properties screen silently ignored itsearch.knn.param.mandsearch.knn.param.ef_constructionare fixed in the shipped index definition and are no longer configurableindex.knnis now set on every index created by 15.8, not only when semantic search is enabledMigration guidance
Four patterns: a fresh installation (nothing to do), an existing installation not using vector search (nothing to do), an existing installation enabling it, and a user of the deprecated
fess-webapp-semantic-searchplugin.Points an operator can otherwise miss, each now documented:
index.knnis a final setting and mappings are applied put-if-absent, so the shipped definitions never reach an existing index_cat/aliasesfor the alias swap before starting the indexer job, and expect roughly twice the index disk during the transition-Drank.fusion.searchers=default,semanticforward silently excludes the built-insemantic_chunksearcher, because the plugin's searcher was namedsemanticdefault_pipelinesurvives jar removal and must be detached explicitlyscheduler.target.nameand a change to the job's target fieldPrerequisites by deployment shape
bin/fessfrom tar.gz/zip withSEARCH_ENGINE_HTTP_URLunset):luceneengine only, because the bundled k-NN plugin ships without its JNI librariesluceneandfaissindex.knn.derived_source.enabledVerification
fess_config.propertiesand the source constants:doc:/:ref:resolution. The rendered-HTML scan matters because nested inline markup and CJK-adjacent markup both fail silently with no parser diagnostic; several real instances were found and fixed this way