GitHub now restricts /stargazers to admins/collaborators, so the Actions
GITHUB_TOKEN 403s. Read STAR_HISTORY_TOKEN instead and surface a clearer error.
Co-authored-by: Cursor <cursoragent@cursor.com>
Generate light and dark variants and use a <picture> element so the README chart follows the viewer's GitHub theme. The daily workflow now regenerates both.
Co-authored-by: Cursor <cursoragent@cursor.com>
The anonymous star-history.com API is rate-limited and returned an empty chart for this repo, so the embedded image showed no data. Render the chart ourselves (scripts/gen_star_history.py) by sampling stargazer timestamps with a token, output a PNG, and have the daily workflow regenerate it. Replaces the empty SVG.
Co-authored-by: Cursor <cursoragent@cursor.com>
The README embedded the live star-history.com SVG API, which is rate-limited and intermittently fails to render (GitHub Camo caches the error). Commit a static assets/star-history.svg and point the README at it, plus a daily workflow that regenerates the image, with a guard so a rate-limited response never overwrites a good chart.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Declare sentencepiece as a dependency: the eagerly-imported Baichuan
tokenizer needs it, so a fresh `pip install airllm` + `import airllm`
crashed with ModuleNotFoundError on 3.0.0.
- Make per-model-family imports in __init__ defensive so a missing
optional dependency for one niche family only warns instead of breaking
the whole package; the core AutoModel/AirLLMBaseModel path always loads.
- Bump version to 3.0.1.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the build (and the build-only dry-run) out of the protected `pypi`
environment so dispatching from main no longer fails environment branch
rules. Only the separate, environment-gated publish job runs on real
releases and uploads via OIDC.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Add .github/workflows/release.yml: build + publish to PyPI on GitHub
Release via Trusted Publishing (OIDC), with a tag/version guard, twine
check, and a build-only dry-run option.
- setup.py reads the top-level README.md (with fallback) so the package
always ships the current README; refresh the short description.
- Remove the stale air_llm/README.md duplicate and gitignore it (now a
build-time artifact synced from the root README).
Co-authored-by: Cursor <cursoragent@cursor.com>
- Refresh hero line and Updates with DeepSeek-V3 (671B, ~12GB) and v3.0.
- Add the (previously missing) Supported Models section: model families plus a
"tiny GPU, huge models" VRAM table.
- Modernize the Quickstart example to Qwen3-32B with bigger one-line swaps.
Co-authored-by: Cursor <cursoragent@cursor.com>
AirLLM hardcoded float16 for streaming. On very deep models (e.g. Qwen3-235B-A22B's
94 layers) fp16's narrow range overflows to inf/NaN and silently corrupts the
output (repeated "!"), while shallower models happened to stay in range. Most
modern checkpoints are natively bfloat16, whose wider range avoids this.
dtype now defaults to None and is resolved from config.torch_dtype (handling both
torch.dtype and string forms), falling back to float16 only if unavailable. Users
can still override with dtype=.
Validated: Qwen3-235B-A22B-FP8 produced garbage in fp16 and correct output once
running in its native bf16 (peak ~2.5GB via fp8 streaming).
Co-authored-by: Cursor <cursoragent@cursor.com>
split_and_save_layers advanced the source shard by only one per layer
("if max(shards) > shard: shard += 1"), so a layer whose tensors span a jump of
more than one shard was persisted missing some tensors. For fp8 checkpoints this
silently dropped the companion weight_scale_inv block scales, producing garbage
output (e.g. Qwen3-235B-A22B-FP8 / Qwen3-30B-A3B-FP8). Single-file checkpoints
(Qwen3-0.6B-FP8) and the on-demand path were unaffected, which masked the bug.
Load every shard up to the highest one a layer references (while-loop) so each
saved layer shard is complete.
Validated on Qwen3-30B-A3B-FP8 (sharded fp8 MoE): correct output, ~0.65GB peak.
Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-quantized fp8 weights (FineGrainedFP8, e.g. DeepSeek-V3, Qwen3 *-FP8) flow
through the normal load path because the quantizer reports param_needs_quantization
== False for them. The old streaming code force-cast every loaded tensor to the
runtime dtype and folded "*weight_scale_inv" into "*weight", which dropped both the
fp8 quantization and the per-block scales and produced garbage output.
- move_layer_to_device now places fp8 weights and their weight_scale_inv scales
verbatim (no dtype cast), and only casts ordinary high-precision tensors.
- only call create_quantized_param when the quantizer actually needs it; add a
param_needs_quantization/check_quantized_param compatibility shim.
- _param_names_from_state_dict only collapses bitsandbytes companion tensors
(.4bit.*/.8bit.*), keeping fp8 weight + weight_scale_inv as distinct params.
Validated on Qwen3-0.6B-FP8 / Qwen3-8B-FP8 (same block-fp8 scheme as DeepSeek-V3):
correct output at ~0.3-1.3GB peak VRAM.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Load config/model with trust_remote_code=False first, falling back to the
model's bundled remote code only when transformers doesn't know the arch.
Vendored remote code (e.g. DeepSeek-V2's modeling_deepseek.py) is often
pinned to an old transformers and breaks on removed cache APIs like
DynamicCache.seen_tokens. This makes DeepSeek-V2 (native deepseek_v2) work.
- When sdpa is unavailable, request attn_implementation="eager" explicitly so
transformers doesn't silently re-select sdpa and error again.
- Read real shard filenames from the index instead of reconstructing them;
repos vary in zero-padding width (DeepSeek: model-00001-of-000004.safetensors)
and extension.
- test harness: thread through --delete-original for large-model runs.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Splitter now drops layers with no weights in the checkpoint (tied lm_head),
fixing both the empty-shard save crash and split-completion detection.
- AirLLMBaseModel detects tied input/output embeddings: it keeps the embedding
resident on GPU, re-ties lm_head to it, and streams only the decoder layers
and final norm. Prefetch skips non-streamed indices.
- auto_model now routes any standard *ForCausalLM to the generic
AirLLMBaseModel and keeps an override table only for legacy custom-code archs
(ChatGLM, legacy QWen, Baichuan, InternLM). New architectures work with no
code changes.
Validated bit-exact vs full-load on the GPU box:
- TinyLlama-1.1B (Llama, non-tied): MATCH, 141MB peak.
- Qwen2.5-0.5B (Qwen2, tied): MATCH, 314MB peak.
Co-authored-by: Cursor <cursoragent@cursor.com>
Replace the hand-rolled GenerationMixin forward loop with a thin wrapper that
lets transformers own forward/generate and only streams each big module's
weights disk->GPU->meta via forward hooks (with prefetch). This decouples
AirLLM from per-architecture attention/rotary/cache internals, which had broken
against modern transformers (e.g. missing _is_stateful, position_embeddings,
bare-tensor layer outputs).
Validated on the GPU box (RTX 4090, transformers 4.57.3):
- TinyLlama-1.1B: bit-exact vs full-load reference, 141MB peak.
- Qwen2.5-7B (4 shards) capped to 4GB VRAM: correct output, 1.1GB peak.
Also adds a reusable test harness (tests/test_streaming_gpu.py) with a
--max-vram-gb cap and an optional full-load correctness comparison.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Drop optimum BetterTransformer dependency (removed upstream; crashed on
import with transformers>=4.49). Use built-in attn_implementation="sdpa".
- Remove post-install transformers auto-upgrade and pin a tested dependency
range (transformers>=4.49,<5.13) so a plain pip install is reproducible.
- Handle single-file checkpoints (model.safetensors / pytorch_model.bin with
no index.json) in the splitter, and download the single weight file on
demand instead of skipping it.
- Bump version to 3.0.0.
Co-authored-by: Cursor <cursoragent@cursor.com>
lyogavin