文件历史

22 次代码提交

作者 SHA1 备注 提交日期
rayx ec8a78ee08 refactor(diskann): decouple from libaio via dlopen (#532)
Co-authored-by: Zefeng Yin <yinzefeng.yzf@alibaba-inc.com>
2026-07-16 17:58:25 +08:00
Cuiys d25f3a4552 fix(python): install the _zvec extension inside the zvec package (#511) 2026-06-24 11:23:52 +08:00
Jalin Wang 65e2deec45 Bump CMake minimum version to 3.26 (#516) 2026-06-23 15:25:00 +08:00
Cuiys 5627e950e8 fix(python): only bundle the DiskANN plugin where it is supported (#510) 2026-06-23 10:08:24 +08:00
Cuiys 3a15a9d5b7 fix(python): exclude C++ SDK headers and libs from the wheel (#509)
The wheel shipped ~94 public headers under `include/` and 7 static/shared
libraries under `lib/` (libzvec*.a, libzvec*.dylib). These come from the
`cc_library(PACKED)` install rules, which are meant for a standalone C++
`make install`, not the Python wheel. `_zvec.so` links the zvec libraries
statically (verified via otool: it depends only on system libs), so none of
those `lib/` / `include/` files are used at runtime — they are pure bloat.

Tag the three runtime install rules (_zvec, the DiskANN plugin, the jieba
dict) with `COMPONENT python`, and set `install.components = ["python"]` in
pyproject.toml so scikit-build-core installs only that component into the
wheel. The SDK install rules (default component) are excluded.

The standalone `make install` is unaffected (it still installs every
component).

Verified on macOS arm64: wheel no longer contains any `include/`, `lib/`,
`.a`, or `.h` entries; `import zvec` and `from zvec import Query` still work.
2026-06-22 19:41:05 +08:00
feihongxu0824 841b1ed3f9 fix: refactor parquet buffer cache ownership (#492) 2026-06-16 10:47:07 +08:00
feihongxu0824 4679a8f39d feat: support zvec core-only build (#481) 2026-06-09 23:45:42 +08:00
rayx e720c1fd20 feat: add diskann index (#369) 2026-06-04 20:52:43 +08:00
egolearner e91b7802f3 feat(ci): enable sccache on Windows and propagate compiler launcher to ExternalProject (#439)
feat(ci): enable sccache on Windows and propagate compiler launcher to ExternalProject

Windows CI builds were ~6-9x slower than other platforms (35-40 min vs
4-7 min) due to missing compiler cache. This commit enables sccache for
Windows and ensures all ExternalProject_Add dependencies (Arrow, lz4)
receive the compiler launcher on every platform.

Changes:

MSVC /Zi → /Z7 compatibility:
- Replace /Zi with /Z7 in root CMakeLists.txt for DEBUG and
  RELWITHDEBINFO configurations (MSVC cache variable level)
- Patch rocksdb, antlr4, and googletest source CMakeLists to replace
  hardcoded /Zi with /Z7 at the source, eliminating D9025 warnings
- Clear COMPILE_PDB_NAME on googletest targets to prevent CMake from
  adding /Fd<path>, which causes sccache to look for nonexistent .pdb
- Remove /FS (forced synchronous PDB writes) from Arrow ExternalProject
  MSVC flags — unnecessary with /Z7

Windows CI workflow:
- Add mozilla-actions/sccache-action@v0.0.10
- Set SCCACHE_GHA_ENABLED=true to use GitHub Actions Cache backend
- Pass CMAKE_C/CXX_COMPILER_LAUNCHER=sccache via --config-settings

Compiler launcher propagation:
- Propagate CMAKE_C/CXX_COMPILER_LAUNCHER to Arrow ExternalProject on
  all platforms (Windows, Linux, macOS, Android, iOS)
- Propagate to lz4 ExternalProject on Windows via CMAKE_ARGS, using
  CMP0141=NEW + CMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded for /Z7
2026-06-02 16:23:37 +08:00
egolearner 3e98314d7c feat(ci): speed up clang-tidy CI workflow (#438)
- Add `clang_tidy_deps` CMake target that only builds generated headers
  (protobuf, Arrow, glog, gflags, lz4) instead of full `zvec_db`
- Run clang-tidy in parallel via `xargs -P $(nproc)` with per-file logs
- Split actions/cache into restore/save to ensure cache is saved even
  when clang-tidy fails, without caching incomplete build artifacts
- Shallow-clone submodules with `--depth 1`
- Bump ccache size from 100M to 500M
- Add `--quiet` to suppress noisy clang-tidy summary lines
- Remove redundant concurrency block from reusable workflow

Performance comparison (3 runs, same repo):

| Step            | Before  | After (cold) | After (warm) |
|-----------------|---------|--------------|--------------|
| Submodule clone | 66s     | 49s          | 28s          |
| Build deps      | 70s     | 21s          | 0s (skipped) |
| Cache restore   | 1s      | 1s           | 12s          |
| Total           | 3m03s   | 2m34s        | 1m46s        |

Dominant win: reduced build target (~49s). Headers cache adds ~9s net.
`--depth 1` contributes ~15s (with network variance).
2026-06-02 10:28:05 +08:00
egolearner 02bfb31cf5 feat: add fts support (#408)
Add BM25-based full-text search with CJK (jieba) tokenization, supporting
  query_string and match_string syntax, phrase queries, boolean operators
  (AND/OR/NOT/MUST), and hybrid retrieval with existing vector search.

  ## Core
  - BitPacked posting format with block-max WAND pruning
  - Tokenizer pipeline: jieba (cut/cut_for_search/hmm/full), whitespace,
    lowercase, with extensible pipeline composition
  - Query parser: boolean operators, phrase queries, field scoping,
    boost, MUST(+) modifier inside OR (ES query_string semantics)
  - AST rewriter: dedup repeated terms with linear boost aggregation,
    flatten same-type composites, canonicalize OR-with-must_not into AND
    wrapper, empty-node propagation, contradiction detection
  - FTS reduce/merge integrated into Optimize compaction
  - Multi-segment score-descending sort
  - Auto-register bundled jieba dict on SDK import

  ## Performance
  - Block-max WAND with cached block_max_info_for (single binary search)
  - AVX2/SSE bitpacked encoding with cross-arch scalar fallback
  - MultiGet for batch posting retrieval and phrase position verification
  - HashSkipList memtable for posting writes
  - PinnableSlice zero-copy reads
  - Filter pushdown into composite iterators (Disjunction/Conjunction/Phrase)
  - Candidate-driven (brute-force) evaluation for selective invert filters
  - Precomputed BM25 IDF weights, cached SIMD dispatch pointers
  - Shortest-list anchor for phrase position matching
  - Single-open per-term posting iterator

  ## Query
  - Tokenize query terms through the same pipeline as indexing
  - EmptyNode for zero-token queries (all stop-words / punctuation)
  - Backslash unescape after lexing in query parser
  - Schema allows collections without vector fields (FTS-only use case)
  - Create/Drop Index validates supported index types
  - FTS fields disallowed in SQL filter expressions

  ## Bindings
  - C API: fts query params, brute-force ratio config
  - Python SDK: FTS search, jieba dict auto-registration

  ## Internals
  - Bypass cppjieba::Jieba to drop KeywordExtractor (~12MB fewer required files)
  - Hide tokenizer pipeline from public header (Pimpl-style FtsState)
  - ListColumnFamilies to avoid double-open on segment load
  - Reorganized fts_column into tokenizer/, posting/, iterator/ subdirs
2026-06-01 15:02:54 +08:00
feihongxu0824 30a20e14b0 feat: add iOS build support (#321)
* support ios build

* fix: ci

* fix: update build_ios.sh

* fix: pr

* fix: cmake minimum version

* fix: cmake minimum version

* fix: cmake minimum version in ci

* fix: add all test in ci

* fix: ci
2026-04-09 16:14:26 +08:00
Jalin Wang ca4b893755 fix: misc fix for Windows(BUILD_SHARED_LIBS, CI) (#296) 2026-04-02 09:23:35 +08:00
lichen2015 ed8c010cfd feat: add c language support (#167)
* init branch

Add GitHub Actions release workflow for C API

add release linux-arm64

remove c api version

refact some code

add api reference

* feat(c-api): add nullable/doc-result APIs for agency migration (#234)


* Flattened index parameters structure

* refact c api code

* remove RAII guard

* refact use opaque pointer pattern

* refact version info

* refact use pure opaque pointers

* use typedef instead of enum

* fix set_last_error

* fix build yml

* fix doc.h

* remove wheel exclude

* fix c msvc link

* remove release yml

* fix c link libstdc++

* fix c link libgcc

---------

Co-authored-by: Donny/강동윤 <kdy.1997.dev@gmail.com>
2026-03-31 18:00:08 +08:00
Jalin Wang 8182cfff93 feat: windows support (#216)
Preliminary support for Windows with remaining TODOs
2026-03-30 20:47:34 +08:00
rayx b49833bf56 Refactor/add scalar dist function and setup cmake with detailed cpu flag (#238)
* fix: add scalar

* fix: add avx512fp16

* fix: cmake config
2026-03-20 09:53:48 +08:00
egolearner e5ba11b6fe feat: add hnsw-rabitq support (#69)
* feat: add hnsw-rabitq

* undefine transform

* support config rotator type

* support sample_count

* fix ut

* refactor: update interface

* refactor: update interface

* update rabitq index params

* fix interface update

* fix searcher test

* fix streamer test

* streamer support bf and add more ut

* rm env check

* add collection ut

* add schema check

* add rabitq query param binding

* add integration test

* fix local_builder

* cleanup dist calculator

* add RaBitQ-Library submodule

* cleanup rabitq converter/reformer

* add files

* disable build on mac

* disable Feature_Optimize_HNSW_RABITQ

* disable python/tests/test_collection_hnsw_rabitq.py:13

* check avx2/avx512

* fix refine

* fix mac ci

* add dimension check

* fix mac ci

* fix ci

* add missing lib

* fix centroids selection for Cosine/InnerProduct

* rename hnsw-rabitq to hnsw_rabitq

* address comments

* fix compile

* fix rabitqlib name

* fix mac compile

* check avx2/avx512 support

* runtime check again compile-time

* check AUTO_DETECT_ARCH

* rm debug log

* fix

* rabitq use avx2 by default

* add missing file

* fix search_bf/group_by dist

* address comments

* fix typo

* address comments

* rabitq support disable id_map

* address comments
2026-03-19 17:21:42 +08:00
feihongxu0824 b10c3fc316 chore: default not use oss and use beijing-oss (#207) 2026-03-09 17:10:57 +08:00
feihongxu0824 fb0b900a80 fix(build): replace CMAKE_SOURCE_DIR with PROJECT_ROOT_DIR for subproject builds (#195) 2026-03-04 20:01:52 +08:00
feihongxu0824 83c03ed0e4 chore(build): accelerate third-party downloads using OSS mirror URLs (#194) 2026-03-04 11:19:10 +08:00
feihongxu0824 e957442355 feat: refact cpp sdk (#27)
* refact cpp sdk

---------

Co-authored-by: zhouqinren.zqr <zhouqinren.zqr@alibaba-inc.com>
2026-01-21 20:05:42 +08:00
sanyi 6524fabd80 Initial commit 2025-12-30 11:02:17 +08:00