Jan v0.8.6 hands-on: what RTX 5000 Ada test numbers mean for local LLM BOMs

StorageReview's Jan v0.8.6 retest on an RTX 5000 Ada workstation logged 123 tok/s for Jan V3.5 4B and 82 tok/s for Jan V2 VL High, framing Windows local-LLM design choices for engineers.

已發布 已更新 6 分鐘閱讀 ✓ 支援 AI 分享

TL;DR — 三句話摘要

  • StorageReview tested Jan v0.8.6 on Windows 11 Pro for Workstations with an Intel Xeon w5-2465X, 128GB of RAM, and an RTX 5000 Ada (32GB VRAM). (S1)
  • Median decode was 123 tokens/sec for Jan V3.5 4B Q4_K_XL and 82 tokens/sec for Jan V2 VL High Q4_K_M at an 8,192-token context with 100 GPU layers and a batch size of 2,048. (S1)
  • Peak VRAM use was 4,839 MiB for the 4B model and 8,197 MiB for Jan V2 VL High during the RAID 5 vs RAID 6 prompt. (S1)

取自 key_takeaways （實際欄位，最多 3 筆）。事實與分析分別標示。

Jan v0.8.6 hands-on: what RTX 5000 Ada test numbers mean for local LLM BOMsEditorial cover for a component sourcing brief. Memory motif; headline "Jan v0.8.6 hands-on: what RTX 5000 Ada test numbers mean for local LLM BOMs". Reported key numbers: 128 GB; 4,839 MiB; 8,197 MiB. 1 cited source. A one-time entrance animation stops under reduced motion.Jan v0.8.6 hands-on: what RTX 5000 Ada testnumbers mean for local LLM BOMsREPORTED KEY NUMBERS128 GB4,839 MiB8,197 MiB1 sourceXAIPART
Cover image generated from data already shown on this page: the category motif, the headline, and reported key numbers. No decorative illustration; no third-party image.

您來自 AI 回答 — 這是可驗證的原始內容

AI 摘要可能遺漏了其主張的依據。原始預測、受影響零件及每一項引用皆在本頁，且均可追溯至其來源。

XAIPART 預測

本簡報包含可證偽的預測。

跳至完整預測及其證偽規則 ›
受影響零件

此報導無目錄相符項目 — 寧可不顯示，也不推斷相符。

查看判定方式 ›
正在尋找零件？送出詢價 → RFQ

Reported by StorageReview SSD/NVMe/存储评测与行业新闻 on . Summarized and analyzed by XAIPART Editorial.

What happened

StorageReview published a hands-on review of Jan v0.8.6, an open-source desktop application marketed as a local ChatGPT alternative that runs downloaded GGUF models through llama.cpp and can also call hosted AI providers.S1

The test platform was an HP Z4 Rack G5 workstation running Windows 11 Pro for Workstations, equipped with an Intel Xeon w5-2465X (16 cores), 128GB of system RAM, and an NVIDIA RTX 5000 Ada Generation GPU with 32GB of VRAM, paired with NVIDIA driver 597.06 and CUDA 13.2 (nvidia-smi reading).S1 StorageReview installed Jan from scratch on that machine to evaluate setup flow, model management, GPU-accelerated inference, web search, multimodal input, and the local OpenAI-compatible API.S1

Jan v0.8.6 bundles llama.cpp directly inside the application (the review shows llama.cpp 0.5.0, build b11146), changes GPU-offload configuration, adjusts parallel-sequence defaults, and adds on-disk conversation prompt caching compared with earlier v0.8.5 and pre-0.8.5 builds.S1 StorageReview reran every performance test on v0.8.6 because of those changes.S1 The Windows installer required no account to use a local model, and on first launch Jan offered to download its recommended Jan V3.5 model, listed at 2.8GB.S1 The Hub also surfaced Jan V3.5 4B at 2.5GB in Q4_K_M and Jan V2 VL High at 4.7GB with multimodal and tools tags.S1

Using a RAID 5 vs RAID 6 prompt at an 8,192-token context, 100 GPU layers, and a batch size of 2,048, StorageReview measured a median decode rate of 123 tokens/sec for Jan V3.5 4B Q4_K_XL (503 output tokens, 4,839 MiB peak VRAM) and 82 tokens/sec for Jan V2 VL High Q4_K_M (1,728 output tokens, 8,197 MiB peak VRAM).S1 In a multimodal retest, Jan V2 VL High read six values from an nvidia-smi screenshot (GPU model, driver 597.06, 32,760 MiB VRAM, 1% utilization, 35°C, 4W of 250W) after 21 seconds of reasoning and 1,979 reported output tokens.S1 A built-in OpenAI-compatible server ran at http://127.0.0.1:1337/v1.S1

Key numbers from the StorageReview Jan v0.8.6 hands-on
Test GPU / VRAMNVIDIA RTX 5000 Ada, 32GB VRAM[S1]
System RAM128GB[S1]
Jan V3.5 4B Q4_K_XL decode123 tok/s (median of 3 runs)[S1]
Jan V3.5 4B peak VRAM4,839 MiB[S1]
Jan V2 VL High Q4_K_M decode82 tok/s (median of 3 runs)[S1]
Jan V2 VL High peak VRAM8,197 MiB[S1]
Jan V3.5 4B context size used8,192 tokens; 100 GPU layers; batch size 2,048[S1]
Bundle of llama.cppv0.5.0, build b11146[S1]
Local API endpointhttp://127.0.0.1:1337/v1[S1]

Background timeline

  1. - StorageReview publishes its Jan v0.8.6 hands-on on an HP Z4 Rack G5 with an RTX 5000 Ada, rerunning tests after v0.8.5 and v0.8.6 landed shortly before publication. [S1]
  2. - Jan v0.8.5 and v0.8.6 introduce bundled llama.cpp, revised GPU offload, adjusted parallel-sequence defaults, and on-disk conversation prompt caching. [S1]

XAIPART take

Technical impact

The published numbers show a small language model (4B parameters, Q4_K_XL) sitting comfortably inside 5GB of VRAM headroom on the RTX 5000 Ada, while a multimodal variant (Q4_K_M) used about 8GB at the same 8K context. That mapping is useful when sizing workstation GPUs for engineering teams that want on-device inference: a single 16GB RTX-class card can host the 4B model with room for context growth, while a 24GB card is the practical floor for the multimodal variant at long context. Bundling llama.cpp inside Jan also removes a class of DLL/CUDA runtime mismatches that procurement teams usually have to triage, which simplifies deployment images across a fleet. [XAIPART analysis]

The dialog around prompt caching is the second technical watchpoint: the review explicitly chose not to publish a prefill median because the cache was warm in repeated runs. For designers building internal tools, that means first-turn latency will be substantially higher than the steady-state decode numbers imply, and any internal benchmark should record cold vs. warm passes separately. [XAIPART analysis]

Market impact

The HP Z4 Rack G5 plus RTX 5000 Ada combination is positioned as a development workstation, and the StorageReview results act as a reference BOM for procurement: one Ada-generation 32GB GPU was enough to run both the text-only and multimodal variants tested. Because Jan is open source and the local API is OpenAI-compatible, design teams can prototype against http://127.0.0.1:1337/v1 and swap in hosted providers (OpenAI, Anthropic, Gemini, Groq, Mistral) without rewriting client code. For sourcing, the implication is that on-prem evaluation hardware can be justified at 32GB VRAM per seat for small-model local inference, rather than defaulting to cloud spend for every prompt. [XAIPART analysis]

The cited Jan V3.5 download at 2.8GB and Jan V2 VL High at 4.7GB indicate that even a single high-end workstation SSD can stage the model library; no specialty storage is required. That keeps the local-LLM workstation BOM close to a standard EDA or CAD seat, which matters for bulk purchasing decisions. [XAIPART analysis]

Action

Engineers evaluating local LLMs should treat the 123 tok/s and 82 tok/s figures as steady-state decode baselines on an RTX 5000 Ada and budget cold-context latency separately, since conversation prompt caching was enabled. Procurement should treat 32GB VRAM Ada-class GPUs as the practical reference point for teams planning to run both text-only and multimodal 4B-class models at 8K context. Before deployment, confirm that any external web-search provider (the Exa MCP service in the review hit its free rate limit) is on a paid tier or has documented headroom, and validate that the target client code works against Jan's OpenAI-compatible endpoint at port 1337 to avoid lock-in to a single vendor. [XAIPART analysis]

XAIPART Forecast

XAIPART expects that if StorageReview runs the same Jan V3.5 4B Q4_K_XL text-generation test on the RTX 5000 Ada within 6 months of this brief's publication (by 2027-04-14) and does not post a median decode rate at or above 123 tokens/sec at an 8,192-token context, then our forecast that the RTX 5000 Ada sets the Ada-generation local-LLM baseline will be wrong; if it clears 123 tok/s, the Ada 32GB card remains the procurement reference for 4B-class models at the same context size. The threshold (123 tok/s at 8,192 tokens) is taken directly from this brief's Key numbers, and the proper noun anchor is the NVIDIA RTX 5000 Ada Generation. [XAIPART analysis]

Facts are as reported by the cited source. The design-in, sourcing and forward-looking statements above are XAIPART's analysis, not the source's claims. Uncertainty: the 82 tok/s and 123 tok/s figures are single-source, single-hardware; Jan V2 VL High's output-token count was flagged by the source as including reasoning tokens and should not be compared directly with the 4B model; prefill performance was not published because of caching; and the Web Search feature could not complete in the retest because Exa's free MCP rate limit was hit.

Corrections: none at publication. This brief will be updated with a timestamp if the source corrects its report.

替代品與生命週期

已建檔的替代品——目錄範圍，並非市場排名
零件關係生命週期依據
— 檔案中無替代品 需要資料 — 需要 match_type 在任何前代標籤出現之前

「前代」與「替代」標籤僅來自已記錄的相符類型、生命週期狀態或來源文字。未記錄相符項目時，不顯示任何標籤。我們自家的刊登內容不構成市場參考。

受影響零件

無目錄相符項目。 此消息未對應到 XAIPART 目前上架的任何零件。我們不會推測相符項目，也不會顯示佔位清單。

各維度成本壓力

Source: FRED · Generated · reference output z and composite are algorithm outputs, not a fetch snapshot

Cost pressure by dimension — composite stateDiverging bars for three dimensions around a zero midpoint. Input Cost has no reading; Supply Demand z -1.4711, notable; Fx Energy z 0.1243, neutral. Composite index -10.1, band neutral.0Input costno readingSupply/demandz −1.4711 · notable46 dFX & energyz +0.1243 · neutral5 dcomposite −10.1 · neutral−100 · −40 · −15 · 0 · +15 · +40 · +100
Underlying values — copyable for verification.
DimensionFRED series IDUnitLagAs of
Input costPCOPPUSDM, PALUMUSDM, PNICKUSDMUSD/mt52 d
Supply/demandPCU334413334413, IPG3344Sindex(1982=100) / index(2017=100)46 d
FX & energyDEXCHUS, DCOILBRENTEU, DHHNGSPrate / USD/bbl / USD/MMBtu5 d

No free authoritative interface for: gold · silver · rare earth · polysilicon · lithium carbonate · market share — listed rather than approximated.

本內容的產生方式

流程證據 — 擷取、來源合理性、模型與閘門
此簡報的稽核軌跡
Source fetched StorageReview SSD/NVMe/存储评测与行业新闻 · www.storagereview.com · 27,227 chars · fetch status success
Source sanity (S1) ok — passed before any model call (no s1_ block recorded)
Model / prompt MiniMax-M3 · prompt Blog Agent - News Summary (v2.8-final) · 14,530 tokens · 41.2s · run completed
Gate verdict pass · 6/6 applicable checks passed · spec v2.9 · C10 ratio 0.00425 · C10 rounds 0
Time contract 3 <time datetime> values in this brief — 0 violations (+08:00 or date-only)

上述每一列皆由流程產生，並非人工撰寫。

編輯評分

確定性閘門結果 — 純 PHP，判定過程無模型參與
閘門評分明細 6 項適用檢查， 6通過： 100 百分比. No hold items or mandatory failures. Gate spec: v2.9 (S0 + S1 + C1–C20) · Source affinity (C20): 0.8718. 適用檢查已通過 6 / 6 = 100% · 判定： pass No hold items or mandatory failures Gate spec: v2.9 (S0 + S1 + C1–C20) · Source affinity (C20): 0.8718 是否發布由審核關卡決定。獨立稽核分數僅供參考。

相關報導

發現錯誤？ 告訴我們哪裡有誤 ——更正會與原文一同發布並記錄。我們不會默默修改文章。

留言

成為第一個留言的人。

來源

  1. Jan v0.8.6 Hands-On: Running Local LLMs on Windows — https://www.storagereview.com/review/jan-v0-8-6-hands-on-running-local-llms-on-windows · rss · 存取於

事實標記為 S1/S2，並可追溯至連結來源。標示為 分析 是 XAIPART 自身的推理，與報導事實分開保留。來源連結會被追蹤，而非 nofollow ——署名為刻意為之。