Crawlee 如何用 OpenTelemetry 配合 Jaeger 追踪每个请求的处理链路【免费下载链接】crawleeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.项目地址: https://gitcode.com/GitHub_Trending/cr/crawlee如果你用 Crawlee 跑爬虫想看到每个请求在run、handleRequest、requestHandler等阶段各花了多少时间可以这样组合Crawlee 官方的crawlee/otel包负责自动插桩爬虫的请求处理管线并转发日志OpenTelemetry SDK 负责把 span 通过 OTLP 导出Jaeger 作为后端接收并可视化这些 trace。OpenTelemetry 本身不提供可视化Jaeger 只是其中一个兼容后端的选项。以下流程在本地用 Docker 容器运行 Jaeger验证方式是最终能在 Jaeger UI 里查到每个请求对应的 span 链。准备环境用 Docker 启动 Jaeger在项目里创建一个docker-compose.yml端口分别是 16686Jaeger UI、4317OTLP gRPC、4318OTLP HTTP并通过环境变量COLLECTOR_OTLP_ENABLEDtrue开启 OTLP 接收services: jaeger: image: jaegertracing/all-in-one:1.53 container_name: jaeger ports: # Jaeger UI - 16686:16686 # OTLP gRPC - 4317:4317 # OTLP HTTP - 4318:4318 environment: - COLLECTOR_OTLP_ENABLEDtrue restart: unless-stopped然后启动容器docker compose up -d启动后可以在浏览器访问http://localhost:16686确认 Jaeger UI 是否可用。安装依赖插桩 Crawlee 需要安装crawlee/otel以及 OpenTelemetry SDK 相关包npm install crawlee/otel opentelemetry/api opentelemetry/api-logs opentelemetry/sdk-node opentelemetry/sdk-trace-base opentelemetry/resources opentelemetry/semantic-conventions opentelemetry/exporter-trace-otlp-grpc两个版本前提来自 packages/otel/package.json 与 packages/otel/src/constants.tscrawlee/otel要求 Node.js22.0.0该插桩声明适配的 Crawlee 版本范围是4.0.0-0 5.0.0-0即 Crawlee v4在范围内的版本中缺失的方法会被跳过并给出警告而不是让模块加载失败。创建插桩文件OpenTelemetry 插桩必须在导入 Crawlee 或任何被插桩模块之前完成。由于 Crawlee 以 ECMAScript 模块发布自动插桩只能通过 Node 的 module hook 来 patch 爬虫类所以要用--import把两个 setup 文件预加载到主代码之前。模块 hook 文件Crawlee 以 ESM 发布自动插桩只能靠 Node module hook 完成。单独建一个文件注册 hook它必须最先被预加载import { register } from node:module; import { pathToFileURL } from node:url; // Installs the OpenTelemetry module hook, which the automatic instrumentation needs in order to patch the Crawlee // classes as they are imported. This file must be preloaded before the OpenTelemetry setup and before the crawler. register(opentelemetry/instrumentation/hook.mjs, pathToFileURL(./));SDK setup 文件setup 文件初始化 OpenTelemetry 并注册CrawleeInstrumentation。exporter 会缓冲数据因此这个文件同时也要负责在进程退出前 flush 掉缓冲区里的 telemetryimport { CrawleeInstrumentation } from crawlee/otel; import { OTLPTraceExporter } from opentelemetry/exporter-trace-otlp-grpc; import { resourceFromAttributes } from opentelemetry/resources; import { NodeSDK } from opentelemetry/sdk-node; import { BatchSpanProcessor } from opentelemetry/sdk-trace-base; import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from opentelemetry/semantic-conventions; // Create a resource that identifies your service const resource resourceFromAttributes({ [ATTR_SERVICE_NAME]: my-crawler, [ATTR_SERVICE_VERSION]: 1.0.0, deployment.environment: development, }); // Configure exporters to send data to Jaeger via OTLP // The gRPC exporter takes the collector endpoint without a signal path - unlike the HTTP one, // which would use http://localhost:4318/v1/traces. const traceExporter new OTLPTraceExporter({ url: http://localhost:4317, }); // Create the Crawlee instrumentation const crawleeInstrumentation new CrawleeInstrumentation(); // Initialize the OpenTelemetry SDK export const sdk new NodeSDK({ resource, spanProcessors: [new BatchSpanProcessor(traceExporter)], instrumentations: [crawleeInstrumentation], }); // Start the SDK sdk.start(); console.log(OpenTelemetry initialized); // This file is preloaded before the crawler, so it also owns flushing the buffered telemetry on the way out. let shuttingDown: Promisevoid | undefined; const shutdown () { // Every handler below can fire, and the SDK must only be shut down once. shuttingDown ?? sdk.shutdown(); return shuttingDown; }; // beforeExit covers a script that simply runs to completion. The flush it starts is async work, so Node keeps the // process alive for it and then fires beforeExit once more - hence on rather than once, and hence shutdown // having to be idempotent. process.on(beforeExit, () { void shutdown(); }); // Signals have to be handled separately, as they do not emit beforeExit. SIGINT is the one you send by pressing // Ctrl-C, so without it a local run loses whatever the exporter had not sent yet. for (const signal of [SIGINT, SIGTERM] as const) { process.once(signal, () { void shutdown().then(() process.exit(0)); }); }注意 gRPC exporter 的url只填 collector 端点http://localhost:4317不带信号路径如果改用 HTTP exporter则对应http://localhost:4318/v1/traces。主爬虫文件创建你的爬虫即可CrawleeInstrumentation会自动插桩核心爬虫方法import { CheerioCrawler } from crawlee; const crawler new CheerioCrawler({ maxRequestsPerCrawl: 10, async requestHandler({ request, $, enqueueLinks, log }) { const title $(title).text(); log.info(Crawled ${request.url}, { title }); await enqueueLinks({ include: [https://crawlee.dev/**], }); }, }); await crawler.run([https://crawlee.dev]); // The setup file flushes the telemetry on exit. console.log(Crawl complete. View traces at http://localhost:16686);示例中的目标站点https://crawlee.dev和maxRequestsPerCrawl: 10直接取自仓库内文档示例替换成你自己的起始 URL 即可。运行爬虫并确认顺序带 setup 文件运行爬虫npx tsx --import ./src/register-hook.ts --import ./src/setup.ts ./src/main.ts两个--import按顺序在任何你自己的代码之前执行先装 module hook再启动 OpenTelemetry SDK最后爬虫模块才被加载并 patch。顺序错了插桩就不生效。如果你是在 Crawlee 仓库的 checkout 里跑文档自带的示例从仓库根目录执行路径见 docs/guides/trace-and-monitor-crawlers.mdxpnpm exec tsx --import ./docs/guides/trace_and_monitor_register_hook.ts \ --import ./docs/guides/trace_and_monitor_setup.ts \ ./docs/guides/trace_and_monitor_basic.ts在 Jaeger UI 中查看请求链路打开http://localhost:16686可以在 Jaeger UI 里搜索不同 trace、应用过滤、比较 trace、查看详细属性和时间分布。当requestHandlingInstrumentation开启默认时以下方法会被自动插桩CrawlerMethodSpan NameBasicCrawlerruncrawlee.crawler.runBasicCrawlerhandleRequestcrawlee.crawler.handleRequestBasicCrawlerrunRequestHandlercrawlee.crawler.runRequestHandlerBasicCrawlerrequestFunctionErrorHandlercrawlee.crawler.requestFunctionErrorHandlerBasicCrawlerhandleFailedRequestHandlercrawlee.crawler.handleFailedRequestHandlerHttpCrawlermakeHttpRequestcrawlee.http.makeHttpRequestBrowserCrawlernavigatecrawlee.browser.navigateAdaptivePlaywrightCrawlerrunRequestHandlercrawlee.crawler.runRequestHandler所有爬虫都继承BasicCrawler的方法所以这张表覆盖了全部AdaptivePlaywrightCrawler单列是因为它用自己的实现替换了runRequestHandler但它一次运行仍然为每个请求产生一个crawlee.crawler.runRequestHandlerspan。crawlee.http.makeHttpRequest和crawlee.browser.navigate记录为 client span因为它们是会离开本进程的调用其余是 internal span。自动 span 上还会带这些属性方便在 Jaeger 里检索每个自动插桩的 span 都带code.function.namecrawlee.crawler.run额外带crawlee.crawler.type正在运行的爬虫类名接收爬虫上下文的方法request handler、navigation handler、error handler额外带url.full请求 URL、http.request.method请求方法、crawlee.request.idCrawlee 请求 ID、crawlee.request.retry_count重试次数。其中url.full和http.request.method是稳定的 OpenTelemetry 语义约定Crawlee 专有数据则保留crawlee.前缀。可选用 wrapWithSpan 给自定义代码加 span自动插桩覆盖的是爬虫核心方法如果你想在请求处理器、hooks 或 error handler 里放更细的 span可以用wrapWithSpan。span 名和属性都可以从被包装函数收到的参数动态推导示例取自 docs/guides/trace_and_monitor_wrap_with_span.ts节选import { wrapWithSpan } from crawlee/otel; import { context, trace } from opentelemetry/api; const crawler new CheerioCrawler({ // Wrap the request handler with a custom span requestHandler: wrapWithSpan( async ({ request, $, enqueueLinks, log }: CheerioCrawlingContext) { // Access the current span to add custom attributes const span trace.getSpan(context.active()); const title $(title).text(); if (span) { span.setAttribute(page.title, title); } await enqueueLinks({ include: [https://crawlee.dev/**], }); }, { // Dynamic span name based on the request spanName: ({ request }: CheerioCrawlingContext) scrape ${request.url}, }, ), }); await crawler.run([https://crawlee.dev]);wrapWithSpan接受三个选项spanName字符串或接收 handler 参数并返回 span 名的函数、spanOptions静态SpanOptions或返回它的函数可带 attributes、tracer自定义 tracer 实例默认用已注册CrawleeInstrumentation的 tracer没有注册时用全局 provider 的 tracer。在包装函数内部也可以用trace.getSpan(context.active())拿到当前 span追加span.setAttribute(...)或span.addEvent(...)。注意 span 只携带你显式设置的属性自动插桩 span 才带code.function.name。可选自定义插桩范围CrawleeInstrumentation的配置项见 docs/guides/trace-and-monitor-crawlers.mdxOptionDefaultDescriptionenabledtrue整体启用或禁用插桩requestHandlingInstrumentationtrue插桩爬虫核心请求处理方法logInstrumentationtrue把 Crawlee 日志转发为 OpenTelemetry 日志customInstrumentation[]需要额外插桩的自定义类方法数组如果你想完全接管插桩哪些方法可以关掉自动请求处理插桩改用customInstrumentation精确指定类和方法例如只给BasicCrawler.run和runRequestHandler建 span完整示例见 docs/guides/trace_and_monitor_custom.tsimport { CrawleeInstrumentation } from crawlee/otel; const crawleeInstrumentation new CrawleeInstrumentation({ // Disable default request handling instrumentation requestHandlingInstrumentation: false, // Disable log forwarding to OpenTelemetry logInstrumentation: false, // Define custom methods to instrument customInstrumentation: [ { moduleName: crawlee/basic, className: BasicCrawler, methodName: run, spanName: crawler.run, }, { moduleName: crawlee/basic, className: BasicCrawler, methodName: runRequestHandler, spanName(context: any) { return request ${context.request.url}; }, }, ], });排查与已知限制没有 span 产出时先检查 hook。crawlee/otel的 README 明确说明不注册 module hook 时爬虫照常运行但不会产生任何 span用--experimental-loaderopentelemetry/instrumentation/hook.mjs虽然能 patch 类但 span 传不出来。所以必须按上面的方式用node:module的register()从预加载文件注册opentelemetry/instrumentation/hook.mjs。更具体的 ESM 环境问题文档指向 OpenTelemetry 官方的 ESM support 说明。进程退出前必须 flush。exporter 缓冲 telemetry如果 setup 文件里没有beforeExit/信号处理的 shutdown 逻辑批处理的 span 会在进程退出时丢失本地运行按 Ctrl-C 发的是SIGINT不处理它同样会丢掉 exporter 尚未发送的数据。Jaeger 不收 OpenTelemetry 日志。logInstrumentation默认开启时Crawlee 日志会以 OTLP log record 发出但只有给 SDK 配置 log record processor 且后端实现了 OTLP logs 服务时数据才有去处。Jaeger 是纯 tracing 后端把 log exporter 指向jaegertracing/all-in-one容器时每个日志批次都会以UNIMPLEMENTED: unknown service opentelemetry.proto.collector.logs.v1.LogsService失败。日志要发往 OpenTelemetry Collector 或能摄取日志的后端如果不想让 Crawlee 日志进入 OpenTelemetry把logInstrumentation设为false即可。日志级别的映射关系是SOFT_FAIL和WARNING都变成WARNPERF变成DEBUG日志行级别过滤交给底层日志库所有消息都会转发需要在 OpenTelemetry pipeline 里过滤。版本边界。插桩声明适配 Crawlee4.0.0-0 5.0.0-0crawlee/otel要求 Node.js22.0.0如果你的 Crawlee 不在这个范围插桩不生效。完整路径回顾最短主路径是docker compose up -d起 Jaeger → 安装crawlee/otel和 OpenTelemetry SDK 依赖 → 写好register-hook.ts、setup.ts、main.ts三个文件 → 用npx tsx --import按 hook、setup、main 的顺序运行 → 在http://localhost:16686的 Jaeger UI 中按url.full、crawlee.request.id等属性查每个请求的 span 链。需要更细粒度时再叠加wrapWithSpan和customInstrumentation。更多细节可直接对照 docs/guides/trace-and-monitor-crawlers.mdx 与 packages/otel/README.md。【免费下载链接】crawleeCrawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.项目地址: https://gitcode.com/GitHub_Trending/cr/crawlee创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考