AI可观测性平台：LLM调用的专属Metrics、Tracing与Logging设计

AI可观测性平台LLM调用的专属Metrics、Tracing与Logging设计一、为什么传统可观测性不适用于LLM应用微服务的可观测性体系Metrics/Tracing/Logging三支柱已经非常成熟——Prometheus Grafana监控QPS和延迟Jaeger/Zipkin追踪调用链路ELK/Loki收集日志。但将这些方案直接套用到LLM应用上时会出现明显的能力缺口。核心差异在于LLM调用引入了传统API调用不存在的维度——Token消耗、模型推理时间、缓存命中率、Prompt/Response的内容质量。这些维度不仅是性能指标更直接关联到成本和用户体验。一个LLM请求的生命周期包含多个专属观测点graph LR subgraph 请求生命周期 A[用户请求] -- B[Prompt组装br/含RAG检索结果] B -- C{命中语义缓存?} C --|是| D[返回缓存结果br/Token消耗0] C --|否| E[模型推理] E -- F[流式输出br/Token-by-Token] F -- G[Response后处理] G -- H[返回用户] end subgraph 观测点 O1[Prompt Token数] O2[缓存命中率] O3[TTFT首Token延迟] O4[Token生成速率] O5[总Token消耗/$成本] O6[Response质量指标] end B -.- O1 C -.- O2 E -.- O3 F -.- O4 F -.- O5 G -.- O6二、AI专属指标体系2.1 核心指标定义与传统微服务相比LLM应用需要监控以下专属指标public class LLMMetrics { // Token相关 private final Counter promptTokensTotal; // 累计Prompt Token消耗 private final Counter completionTokensTotal; // 累计Completion Token消耗 private final Histogram tokensPerRequest; // 每次请求的总Token数 // 延迟相关 private final Histogram timeToFirstToken; // 首Token延迟TTFT private final Histogram timePerOutputToken; // 每个输出Token的生成延迟 private final Histogram totalRequestLatency; // 请求总延迟 // 缓存相关 private final Counter cacheHits; // 语义缓存命中次数 private final Counter cacheMisses; // 语义缓存未命中次数 private final Gauge cacheHitRate; // 缓存命中率 // 成本相关 private final Counter totalCost; // 累计费用美元 private final Histogram costPerRequest; // 每次请求的费用 // 质量相关 private final Histogram responseTokenCount; // Response token数量 private final Counter streamingConnections; // 活跃的流式连接数 private final Counter rateLimitErrors; // 限流错误次数 }2.2 指标的Prometheus采集实现Configuration public class LLMMetricsConfig { Bean public MeterRegistry meterRegistry() { return new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); } Bean public LLMObservabilityInterceptor llmInterceptor(MeterRegistry registry) { return new LLMObservabilityInterceptor(registry); } } Aspect Component public class LLMObservabilityInterceptor { private final MeterRegistry registry; Around(annotation(llmCall)) public Object observeLLMCall(ProceedingJoinPoint pjp, LLMCall llmCall) { Timer.Sample sample Timer.start(registry); long startTime System.nanoTime(); LLMResponse response; try { response (LLMResponse) pjp.proceed(); recordSuccess(llmCall.model(), response, sample, startTime); } catch (Exception e) { recordFailure(llmCall.model(), e); throw e; } return response; } private void recordSuccess(String model, LLMResponse resp, Timer.Sample sample, long startTime) { // Token指标 registry.counter(llm.tokens.prompt, model, model) .increment(resp.getPromptTokens()); registry.counter(llm.tokens.completion, model, model) .increment(resp.getCompletionTokens()); registry.counter(llm.cost.total, model, model) .increment(calculateCost(model, resp)); // 延迟指标 double ttftMs (resp.getFirstTokenTime() - startTime) / 1_000_000.0; registry.timer(llm.latency.ttft, model, model) .record((long) ttftMs, TimeUnit.MILLISECONDS); registry.timer(llm.latency.total, model, model) .record(System.nanoTime() - startTime, TimeUnit.NANOSECONDS); // 输出速率 double tps resp.getCompletionTokens() / ((System.nanoTime() - resp.getFirstTokenTime()) / 1_000_000_000.0); registry.summary(llm.tokens_per_second, model, model) .record(tps); } private double calculateCost(String model, LLMResponse resp) { // GPT-4: prompt $0.03/1K, completion $0.06/1K return switch (model) { case gpt-4 - resp.getPromptTokens() * 0.03 / 1000 resp.getCompletionTokens() * 0.06 / 1000; case gpt-3.5-turbo - resp.getPromptTokens() * 0.0015 / 1000 resp.getCompletionTokens() * 0.002 / 1000; default - 0; }; } }2.3 Grafana告警规则groups: - name: llm_alerts rules: - alert: LLMCostSpike expr: rate(llm_cost_total[5m]) 10 # 每分钟费用超过$10 for: 5m annotations: summary: LLM cost spike: {{ $value }}/min - alert: HighTTFT expr: histogram_quantile(0.99, rate(llm_latency_ttft_bucket[5m])) 3000 for: 5m annotations: summary: P99 TTFT exceeds 3s - alert: CacheHitRateDrop expr: rate(llm_cache_hits_total[5m]) / (rate(llm_cache_hits_total[5m]) rate(llm_cache_misses_total[5m])) 0.5 for: 10m annotations: summary: Cache hit rate dropped below 50% - alert: RateLimitErrors expr: rate(llm_rate_limit_errors_total[5m]) 1 for: 2m annotations: summary: Rate limit errors detected三、Prompt→Response完整链路追踪3.1 Tracing设计LLM调用的链路追踪需要记录从用户请求到模型响应的完整路径包括Prompt的组装过程含RAG检索的文档块public class LLMTracer { private final Tracer tracer; public LLMResponse traceLLMCall(LLMRequest request) { Span span tracer.spanBuilder(llm.completion) .setAttribute(model.name, request.getModel()) .setAttribute(model.provider, request.getProvider()) .setAttribute(llm.prompt.tokens, request.getPromptTokens()) .setAttribute(llm.max_tokens, request.getMaxTokens()) .setAttribute(llm.temperature, request.getTemperature()) .startSpan(); try (Scope scope span.makeCurrent()) { // 如果有RAG检索记录子Span if (request.hasRAGContext()) { Span retrievalSpan tracer.spanBuilder(rag.retrieval) .setAttribute(rag.documents.retrieved, request.getRAGDocuments().size()) .setAttribute(rag.top_k, request.getRAGTopK()) .startSpan(); retrievalSpan.end(); } LLMResponse response callModel(request); span.setAttribute(llm.completion.tokens, response.getCompletionTokens()); span.setAttribute(llm.stop_reason, response.getStopReason()); span.setAttribute(llm.cost, calculateCost(request.getModel(), response)); span.setStatus(StatusCode.OK); return response; } catch (Exception e) { span.setStatus(StatusCode.ERROR, e.getMessage()); span.recordException(e); throw e; } finally { span.end(); } } }3.2 链路追踪的可视化在Jaeger中一次LLM请求的完整链路如下llm.completion (3.2s) ───────────────────────── ├── rag.retrieval (120ms) ── │ ├── vector_db.query (95ms) ── │ └── rerank (23ms) ── ├── llm.prefill (680ms) ─────── └── llm.decode (2.4s) ───────────────────────── ├── token_1 (80ms) ├── token_2 (78ms) ├── ... └── token_30 (72ms)每个Span携带的Tagsllm.prompt.tokens: 输入token数llm.completion.tokens: 输出token数llm.ttft_ms: 首Token延迟llm.cost_usd: 本次调用费用rag.documents.retrieved: 检索文档数四、成本归因与开源方案对比4.1 多维度成本归因graph TB subgraph 成本归因维度 D1[按模型br/GPT-4 vs Claude vs Llama] D2[按业务线br/客服 vs 文档 vs 代码] D3[按用户br/VIP用户 vs 免费用户] D4[按功能br/RAG检索 vs 纯生成] end subgraph 成本分析 C1[日/周/月消费趋势] C2[单次请求成本分布] C3[Token使用效率br/输出/输入比] C4[缓存节省的成本] end D1 -- C1 D2 -- C1 D3 -- C2 D4 -- C3 D4 -- C4成本归因SQL示例基于ClickHouse存储-- 按业务线统计每日LLM成本 SELECT toDate(timestamp) AS date, business_line, model, sum(prompt_tokens) AS total_prompt_tokens, sum(completion_tokens) AS total_completion_tokens, sum(cost_usd) AS total_cost_usd, count() AS request_count, avg(ttft_ms) AS avg_ttft_ms FROM llm_observability WHERE date today() - 7 GROUP BY date, business_line, model ORDER BY date DESC, total_cost_usd DESC; -- 识别高成本低价值调用 SELECT user_id, business_line, avg(completion_tokens) AS avg_output_tokens, sum(cost_usd) AS total_cost, count() AS call_count, sum(cost_usd) / count() AS avg_cost_per_call FROM llm_observability WHERE timestamp now() - INTERVAL 24 HOUR AND completion_tokens 10 -- 输出极短但调用了昂贵模型 AND model LIKE gpt-4% GROUP BY user_id, business_line HAVING call_count 100 ORDER BY total_cost DESC LIMIT 20;4.2 开源方案对比方案核心能力部署方式适用场景优劣势LangfuseTracing Prompt管理 评估自建/Cloud全栈LLM可观测功能全面但自建运维成本高Helicone代理层 日志 缓存Cloud优先API网关型监控轻量但定制能力有限Phoenix (Arize)Tracing 向量分析自建调试与质量分析向量分析强但指标少Weights Biases实验追踪 模型评估Cloud模型训练推理训练侧强推理侧弱自建PrometheusJaeger完全可控自建定制化需求灵活但开发成本高4.3 Langfuse集成示例from langfuse import Langfuse from langfuse.decorators import observe, langfuse_context langfuse Langfuse( secret_keysk-lf-..., public_keypk-lf-..., hosthttps://cloud.langfuse.com ) observe(as_typegeneration) async def llm_completion(prompt: str, model: str gpt-4): langfuse_context.update_current_observation( modelmodel, usage_details{prompt_tokens: len(prompt) // 4} ) response await openai_client.chat.completions.create( modelmodel, messages[{role: user, content: prompt}] ) langfuse_context.update_current_observation( usage_details{ completion_tokens: response.usage.completion_tokens, total_tokens: response.usage.total_tokens, }, cost_details{ total: calculate_cost(model, response.usage) } ) return response observe() async def rag_pipeline(query: str): # 自动追踪RAG检索Span documents await vector_search(query, top_k5) langfuse_context.update_current_observation( metadata{retrieved_docs: len(documents)} ) prompt build_prompt(query, documents) return await llm_completion(prompt)4.4 自建 vs 开源的选型决策选择自建的条件需要对指标维度做深度定制如与业务系统的成本结算对接数据安全合规要求不允许外部SaaS已有成熟的Prometheus/Grafana/Jaeger运维体系选择开源的条件团队对LLM可观测领域经验不足需要开箱即用需要Prompt版本管理和AB测试能力需要面向非工程团队产品/运营的Dashboard推荐策略先用Langfuse快速建立可观测能力1周内完成再逐步将核心指标接入自建的Prometheus体系1~2个月渐进迁移。五、总结LLM应用的可观测性是一个仍在快速演进的领域。与已经标准化的微服务可观测性不同LLM的可观测性需要覆盖新的维度Token级别指标不仅衡量延迟和QPS更要追踪Token消耗与成本的实时变化语义缓存命中率直接影响模型调用量和响应延迟是需要重点监控的核心指标Prompt→Response全链路Tracing从RAG检索文档到最终生成的每个token都应该被追踪多维度成本归因按模型/业务线/用户/功能维度的成本分析是控制LLM支出的前提开源方案的合理利用Langfuse适合快速启动但长期的指标沉淀和告警体系应建立在自建基础设施上一个实用的启动路径第一周接入Langfuse建立Tracing第二周在Prometheus中补齐AI专属指标第三周搭建成本Dashboard和告警规则。可观测性不是一次性的工程而是在每次模型升级、Prompt迭代和流量增长中持续演进的能力。