learningcircuit--local-deep-research
7a0da7932b
OSV-Scanner (Scheduled) / scan-scheduled (push) Failing after 0s
Create Release / test-gate (push) Has been cancelled
Create Release / release-gate (push) Has been cancelled
Create Release / ci-gate (push) Has been cancelled
Create Release / version-check (push) Has been cancelled
Create Release / e2e-test-gate (push) Has been cancelled
Create Release / responsive-test-gate (push) Has been cancelled
Create Release / compat-test-gate (push) Has been cancelled
Create Release / compose-integration-gate (push) Has been cancelled
Create Release / vulture-gate (push) Has been cancelled
Create Release / build (push) Has been cancelled
Create Release / provenance (push) Has been cancelled
Create Release / prerelease-docker (push) Has been cancelled
Create Release / publish-docker (push) Has been cancelled
Create Release / create-release (push) Has been cancelled
Create Release / cleanup-changelog (push) Has been cancelled
Create Release / trigger-pypi (push) Has been cancelled
Create Release / monitor-pypi (push) Has been cancelled
Create Release / Clean up orphan prerelease tags and signatures (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-form] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-metrics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [research-workflow] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-core] (push) Has been cancelled
CodeQL Advanced / Analyze (javascript-typescript) (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [history-news] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [library] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [link-analytics] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-core] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [chat-lifecycle] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [error-benchmark] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [settings-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) (push) Has been cancelled
Docker Tests (Consolidated) / Accessibility Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Unit Tests (push) Has been cancelled
Docker Tests (Consolidated) / LLM Example Tests (push) Has been cancelled
Docker Tests (Consolidated) / Production Image Smoke Test (push) Has been cancelled
Docker Tests (Consolidated) / Infrastructure Tests (push) Has been cancelled
OSSF Scorecard / OSSF Security Scorecard Analysis (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [mobile] (push) Has been cancelled
Backwards Compatibility / Verify Encryption Constants (push) Has been cancelled
Backwards Compatibility / PyPI Version Compatibility (push) Has been cancelled
Backwards Compatibility / Database Migration Tests (push) Has been cancelled
CodeQL Advanced / Analyze (python) (push) Has been cancelled
Docker Tests (Consolidated) / detect-changes (push) Has been cancelled
Docker Tests (Consolidated) / Build Test Image (push) Has been cancelled
Docker Tests (Consolidated) / All Pytest Tests + Coverage (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [accessibility] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [api-crud] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-login] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-pages] (push) Has been cancelled
Docker Tests (Consolidated) / UI Tests (Puppeteer) [auth-register] (push) Has been cancelled
155 行
4.3 KiB
Markdown
155 行
4.3 KiB
Markdown
# Elasticsearch 搜索引擎
|
|
|
|
这个文档介绍如何在 Local Deep Research 项目中使用和配置 Elasticsearch 搜索引擎。
|
|
|
|
## 概述
|
|
|
|
Elasticsearch 搜索引擎允许你搜索 Elasticsearch 索引中的文档。这对于需要搜索大量结构化或非结构化文本数据的场景非常有用。
|
|
|
|
## 前提条件
|
|
|
|
1. 运行中的 Elasticsearch 服务器(本地或远程)
|
|
2. Elasticsearch Python 客户端库:`elasticsearch>=8.10.0`
|
|
|
|
## 安装
|
|
|
|
确保你已经安装了 `elasticsearch` Python 包。如果你是从源代码安装 Local Deep Research,可以运行:
|
|
|
|
```bash
|
|
pip install elasticsearch>=8.10.0
|
|
```
|
|
|
|
## 基本用法
|
|
|
|
### 在代码中使用
|
|
|
|
```python
|
|
from local_deep_research.web_search_engines.engines.search_engine_elasticsearch import ElasticsearchSearchEngine
|
|
|
|
# 创建搜索引擎实例
|
|
es_search = ElasticsearchSearchEngine(
|
|
hosts=["http://localhost:9200"], # Elasticsearch 服务器地址
|
|
index_name="my_index", # 要搜索的索引名称
|
|
username="user", # 可选:认证用户名
|
|
password="pass", # 可选:认证密码
|
|
max_results=10 # 返回结果数量
|
|
)
|
|
|
|
# 执行搜索
|
|
results = es_search.run("你的搜索查询")
|
|
|
|
# 处理结果
|
|
for result in results:
|
|
print(f"标题: {result.get('title')}")
|
|
print(f"片段: {result.get('snippet')}")
|
|
print(f"内容: {result.get('content')}")
|
|
```
|
|
|
|
### 高级搜索
|
|
|
|
ES搜索引擎支持多种高级搜索方式:
|
|
|
|
```python
|
|
# 使用 Elasticsearch 查询字符串语法
|
|
results = es_search.search_by_query_string("title:关键词 AND content:内容")
|
|
|
|
# 使用 Elasticsearch DSL(领域特定语言)
|
|
results = es_search.search_by_dsl({
|
|
"query": {
|
|
"bool": {
|
|
"must": {"match": {"content": "搜索词"}},
|
|
"filter": {"term": {"category": "技术"}}
|
|
}
|
|
}
|
|
})
|
|
```
|
|
|
|
## 配置说明
|
|
|
|
### 搜索引擎参数
|
|
|
|
| 参数 | 类型 | 默认值 | 说明 |
|
|
|------|------|--------|------|
|
|
| hosts | List[str] | ["http://localhost:9200"] | Elasticsearch 服务器地址列表 |
|
|
| index_name | str | "documents" | 要搜索的索引名称 |
|
|
| username | Optional[str] | None | 认证用户名 |
|
|
| password | Optional[str] | None | 认证密码 |
|
|
| api_key | Optional[str] | None | API 密钥认证 |
|
|
| cloud_id | Optional[str] | None | Elastic Cloud ID |
|
|
| max_results | int | 10 | 最大结果数 |
|
|
| highlight_fields | List[str] | ["content", "title"] | 要高亮显示的字段 |
|
|
| search_fields | List[str] | ["content", "title"] | 要搜索的字段 |
|
|
| filter_query | Optional[Dict] | None | 可选的过滤查询 |
|
|
| llm | Optional[BaseLLM] | None | 用于相关性过滤的语言模型 |
|
|
| max_filtered_results | Optional[int] | None | 过滤后的最大结果数 |
|
|
|
|
## 索引数据
|
|
|
|
为了便于使用,我们提供了 `ElasticsearchManager` 工具类来帮助索引数据:
|
|
|
|
```python
|
|
from local_deep_research.utilities.es_utils import ElasticsearchManager
|
|
|
|
# 创建 ES 管理器
|
|
es_manager = ElasticsearchManager(
|
|
hosts=["http://localhost:9200"]
|
|
)
|
|
|
|
# 创建索引
|
|
es_manager.create_index("my_index")
|
|
|
|
# 索引单个文档
|
|
es_manager.index_document(
|
|
index_name="my_index",
|
|
document={
|
|
"title": "文档标题",
|
|
"content": "文档内容...",
|
|
}
|
|
)
|
|
|
|
# 批量索引文档
|
|
documents = [
|
|
{"title": "文档1", "content": "内容1"},
|
|
{"title": "文档2", "content": "内容2"}
|
|
]
|
|
es_manager.bulk_index_documents("my_index", documents)
|
|
|
|
# 索引文件(自动提取内容)
|
|
es_manager.index_file("my_index", "path/to/document.pdf")
|
|
|
|
# 索引整个目录中的文件
|
|
es_manager.index_directory(
|
|
"my_index",
|
|
"path/to/docs",
|
|
file_patterns=["*.pdf", "*.docx", "*.txt"]
|
|
)
|
|
```
|
|
|
|
## 示例
|
|
|
|
查看 `examples/elasticsearch_search_example.py` 获取完整的使用示例。
|
|
|
|
## 运行示例
|
|
|
|
确保 Elasticsearch 正在运行,然后执行:
|
|
|
|
```bash
|
|
python examples/elasticsearch_search_example.py
|
|
```
|
|
|
|
## 故障排除
|
|
|
|
### 无法连接到 Elasticsearch
|
|
|
|
- 确保 Elasticsearch 服务器正在运行
|
|
- 检查主机地址和端口是否正确
|
|
- 验证认证凭据(如果需要)
|
|
- 检查网络连接和防火墙设置
|
|
|
|
### 搜索结果为空
|
|
|
|
- 确保索引存在并且包含数据
|
|
- 检查搜索字段是否正确
|
|
- 尝试使用更简单的查询
|
|
- 检查 Elasticsearch 日志获取更多信息
|