omiGlass 多视觉语言模型图像描述评测实践从 img_23 看一条 Ollama 视觉评测流水线【免费下载链接】FriendAI that sees your screen, listens to your conversations and tells you what to do项目地址: https://gitcode.com/GitHub_Trending/fr/Friend本指南以 omiGlass/prompts/series_1/img_23.md 为切入点剖析 omiGlass 智能眼镜项目中同一张图片、四个视觉语言模型并行描述的评测数据集是如何生成的。你将掌握该评测文件的结构规范、底层 Ollama 推理调用链imageDescription.ts 与 ollama.ts、四模型输出的差异分析方法以及如何在本地复现与扩展这条评测流水线。一、img_23.md 是什么一份多模型图像描述评测记录在 omiGlass 仓库中omiGlass/prompts/series_1/ 目录存放着一批img_N.jpeg原图与一一对应的img_N.md评测文件。img_23.md就是其中一页它不包含任何人工标注而是由四个不同视觉语言模型VLM对同一张 600×800 的建筑室内照片生成的描述文本以####模型名####作为分节标题依次排列。其完整格式如下即原文档全部内容####Description####默认模型的输出。####Description (llava-llama3)####llava-llama3 的输出。####Description (llava:34b-v1.6)####llava:34b-v1.6 的输出。####Description (moondream:1.8b-v2-fp16)####显式指定 moondream 模型的输出。这种标题 描述文本的扁平格式由评测脚本直接拼接生成见下文 generate.ts本质上是一份可读、可 diff、可被后续 LLM 检索引用的评测语料而非人写文档。二、原文档完整内容四个模型的原始输出为保证事实准确以下完整摘录 img_23.md 的四段模型输出未做任何删改####Description####默认模型 moondream:1.8b-v2-fp16The image features a large, open space with an industrial-style ceiling. The ceiling is made of metal and has a grid pattern on it, giving the area a unique appearance. A person can be seen in the background, possibly observing or working within this space.####Description (llava-llama3)####The image captures a unique perspective of the interior of an architectural structure. The dominant color in the frame is yellow, which is the exterior wall of the building. This wall is made up of small square panels arranged in a grid-like pattern, giving it a sense of uniformity and order.Two large metal beams run diagonally across the image from the top left to the bottom right corner. These beams, coupled with the triangular shape of the roof, create an interesting geometric contrast against the yellow wall.On the right side of the image, theres a blurred face partially visible in the background. This adds a touch of mystery and intrigue to the scene, making one wonder about the identity of this person and what they might be doing in this setting.Overall, the image presents an intriguing blend of architectural elements and subtle human presence, set against a vibrant yellow backdrop. Its a snapshot that invites viewers to explore further and unravel its story.####Description (llava:34b-v1.6)####The image appears to be taken from a low angle, looking upwards towards what could be the interior of a building with an exposed metal framework. The framework consists of rafters or beams that intersect at various angles, forming a geometric pattern against the sky. There are no texts visible in this image. The lighting suggests its daytime as sunlight is streaming through the structure, casting shadows and highlighting the metallic surfaces. It seems like a candid shot capturing an architectural detail rather than a posed composition.####Description (moondream:1.8b-v2-fp16)####显式指定The image features a high ceiling with metal beams that are crossed, creating an interesting architectural design. A person can be seen looking down from the top of the building through these beams, adding a sense of scale and perspective to the scene. The background is not described in detail but adds depth to the overall composition, suggesting the presence of other buildings or structures nearby.三、评测流水线generate.ts 如何批量产出这些文件这些 .md 文件并非手工整理而是由 omiGlass/prompts/generate.ts 一次性批量生成。该脚本的逻辑非常清晰读取prompts目录下的所有子目录即series_1等系列收集其中全部.jpeg图片并预先规划好输出路径——把.jpeg替换为.md见 generate.ts。定义runTest(title, test)工具函数依次对每张图片执行测试将输出以#### title ####\n作为分节头累加到outputs字符串中见 generate.ts。依次运行四轮测试见 generate.tsDescription调用imageDescription(img)使用默认模型Description (llava-llama3)显式传入llava-llama3Description (llava:34b-v1.6)显式传入llava:34b-v1.6Description (moondream:1.8b-v2-fp16)显式传入moondream:1.8b-v2-fp16。将拼接好的多段结果整体写入对应的.md文件见 generate.ts。从源码结构可以推断一个值得注意的细节第一轮Description与第四轮moondream:1.8b-v2-fp16使用的是同一个模型因为imageDescription的默认参数就是moondream:1.8b-v2-fp16见 imageDescription.ts。而 img_23.md 中这两段输出内容并不相同——这说明在非确定性采样下同一 VLM 对同一张图重复推理也可能给出措辞与细节不同的描述。评测时留意这一点有助于区分模型间差异与模型内随机性。四、底层推理链路imageDescription → Ollama HTTP 客户端生成 .md 的核心函数是 imageDescription.ts 中的imageDescription(src, model)。其工作方式如下默认模型为moondream:1.8b-v2-fp16可传入KnownModel联合类型中的任意模型名。构造两条消息后交给ollamaInference见 imageDescription.tssystem 消息You are a very advanced model and your task is to describe the image as precisely as possible. Transcribe any text you see.——即尽可能精确描述并转写画面中出现的任何文字。llava:34b-v1.6 输出中主动声明 There are no texts visible in this image正是对该提示词指令的响应。user 消息Describe the scene并附带以Uint8Array形式传入的图片像素数据。再往下是真正的推理调用 ollama.ts 中的ollamaInference将每条消息的图片通过toBase64()转为 base64 字符串见 ollama.ts这是 Ollama Chat API 对多模态消息的标准要求。通过 axios 向keys.ollama发起 POST请求体为{ stream: false, model, messages }见 ollama.tsstream: false表示等待完整响应而非流式返回。整个请求被包在backoff()中见 ollama.ts失败时按退避策略重试提升批量评测的稳定性。最终从响应response.message.content提取文本并经trimIdent去除多余缩进后返回。服务地址与密钥统一收敛在 keys.tsEXPO_PUBLIC_OLLAMA_API_URLOllama 服务地址README 中默认http://localhost:11434/api/chatEXPO_PUBLIC_GROQ_API_KEYGroq 密钥供llamaFind问答阶段使用EXPO_PUBLIC_OPENAI_API_KEYOpenAI 密钥供语音模块使用。五、四模型输出对比从 img_23 读出的评测要点将四段输出与 img_23.jpeg 原图暖黄色面板屋顶、金属斜梁框架、玻璃天窗透出日光对照可以提炼出这套评测语料的典型分析维度1. 结构共识强。四个模型都正确识别出室内/工业风格空间 金属网格或斜交梁结构这一核心要素说明在主体结构描述上1.8B 量级的 moondream 与 34B 量级的 llava 都能给出可靠结论。2. 主色调捕捉差异明显。llava-llama3 明确点出 The dominant color in the frame is yellow与原图暖黄色面板高度吻合其余三个模型则把注意力放在金属框架上未突出色彩。从源码看各模型使用的提示词完全相同因此这种差异可归因于模型自身的注意力偏好。3. 人物/小目标识别是最大分歧点。llava-llama3 声称右侧有 blurred face默认 moondream 认为背景 a person can be seen显式 moondream 更进一步描述为 looking down from the top而 llava:34b-v1.6 完全没有提及人物。对画面中模糊、占比较小的目标各 VLM 存在过度解释与选择性忽略两种截然不同的行为——这正是评测语料最有价值的信息。4. 额外信息的主动性。llava:34b-v1.6 是唯一主动报告画面无文字的模型体现了其对 OCR 指令的遵循度llava-llama3 则额外给出了构图层面的主观评价adds a touch of mystery说明其输出风格更偏向叙事性描述。六、从评测到产品图像描述如何支撑眼镜问答这套评测并非孤立实验。在 Agent.ts 中imageDescription被直接用于产品逻辑眼镜端拍照后逐张调用imageDescription(p)生成描述并存入内存队列当用户提问时将所有照片描述拼接交给llamaFind(question, combined)见 Agent.ts后者调用 Groq 上的llama3-70b-8192大模型在只依据图片描述回答、不得泛化的约束下给出答案见 groq-llama3.ts。因此评测文件中模型描述的准确性、一致性直接决定了眼镜端看图回答问题的体验上限——这也解释了为何项目要把多模型描述沉淀为可复现的评测语料。七、本地复现与扩展评测结合 README.md 与源码复现该评测的完整路径如下准备环境进入omiGlass目录执行npm install或yarn install本机安装并启动 Ollama。配置密钥复制.env.template为.env填入EXPO_PUBLIC_OLLAMA_API_URL默认http://localhost:11434/api/chat如需跑llamaFind再配置 Groq/OpenAI 密钥见 keys.ts。拉取模型至少执行ollama pull moondream:1.8b-v2-fp16要复现完整四模型评测还需准备llava-llama3与llava:34b-v1.6。运行评测脚本执行 generate.ts脚本会自动遍历prompts下所有子目录的.jpeg跑完四轮后把结果写回同名.md。扩展新模型在 ollama.ts 的KnownModel联合类型中追加模型名并在generate.ts中新增一轮runTest调用即可无需改动其他代码。八、局限与注意点从源码与输出反推这套评测存在几点需要读者注意的边界无 ground truth 标注.md中只有模型输出、没有人工标准答案评测结论依赖读者自行对照原图研判同一模型重复运行默认轮与显式 moondream 轮是同一模型不宜把两段差异当作模型间对比使用依赖本地 Ollama批量评测的吞吐与稳定性取决于 Ollama 服务backoff重试虽能缓解偶发失败但长队列耗时仍然可观描述并非绝对事实如人物识别所示小目标与模糊区域易出现幻觉生产环境应对imageDescription的输出保持审慎。对想要构建拍照→理解→问答类视觉 Agent 的开发者而言img_23.md 及其背后的流水线提供了一个轻量、可复现的多模型评测范式一份脚本、一套提示词、数行输出即可横向评估候选 VLM 对同一场景的描述能力。【免费下载链接】FriendAI that sees your screen, listens to your conversations and tells you what to do项目地址: https://gitcode.com/GitHub_Trending/fr/Friend创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考