• English

      • Observability

      Luna DevOps exports traces, metrics, and structured logs through OpenTelemetry. Prepare an OpenTelemetry Collector with OTLP HTTP support and, when required, Prometheus, Loki, and Tempo.

      Export telemetry

      Configure the same Collector endpoint for API, Worker, and Agent:

      OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=production,k8s.cluster.name=main

      Leaving the endpoint empty disables export. If the Collector requires authentication, inject OTEL_EXPORTER_OTLP_HEADERS from a Secret instead of storing credentials in public configuration.

      After restart, confirm that the Collector receives:

      • luna-devops-api
      • luna-worker
      • luna-agent when AI Assistant is enabled

      Read and search logs

      Log records are always structured, while terminal rendering and OTel export are independent. Local processes use LOG_FORMAT=auto by default: interactive terminals render one console record per line, while redirected output switches to JSON. Docker Compose keeps auto and therefore emits JSON in non-interactive containers, while production Helm explicitly sets LOG_FORMAT=json. LOG_COLOR=auto|always|never affects console output only, and NO_COLOR always disables color; JSON and OTel records never contain ANSI sequences.

      Failure records use stable event.name, operation, outcome, and error.code fields. error.message keeps the complete credential-redacted error chain, and contextual records also include trace_id, span_id, request_id, and resource IDs. Start incident searches with the response requestId or traceId, then inspect the complete dependency error. Internal addresses, file paths, SQLSTATE values, and resource IDs remain available for diagnosis; actual token, Authorization, Cookie, password, API key, and private-key values are redacted.

      Production API error responses contain only a stable code, a generic message localization key, and available requestId / traceId values. They do not expose database errors, internal addresses, or stacks. With APP_ENV=development, responses also contain a credential-redacted developerDetail.

      Configure the Agent observability page

      Open Global Settings → AI Assistant → Advanced AI Settings and enter:

      • Prometheus query URL, such as http://prometheus:9090
      • Loki query URL, such as http://loki:3100
      • Tempo query URL, such as http://tempo:3200
      • Optional Tenant IDs and bearer tokens

      Enable Agent Observability after all three URLs are present and test each connection. Only Luna API receives these query URLs and tokens; they are not sent to browsers. Use Operations → Agent Observability to view model usage, tool calls, turns, and traces.

      Open a turn to see its complete Trace ID prominently in the header. Click the Trace ID block to copy it for searching the same request chain in Tempo, a log platform, or a diagnostic ticket. The Agent observability page displays Trace IDs and Span IDs as lowercase hexadecimal strings regardless of the transport encoding used by Tempo query responses.

      Interpret model token usage

      The conversation overview, turn list, and turn detail use the Provider's official usage as their source of truth:

      • Input tokens are the complete input count and already include cache-read and cache-write tokens. Output tokens already include reasoning tokens. Cache and reasoning values are breakdowns and must not be added to input or output again.
      • A — for cache read, cache write, or reasoning means the Provider did not report that detail. A displayed 0 means it explicitly reported zero.
      • Agent model spans use gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_write.input_tokens, and gen_ai.usage.reasoning.output_tokens. When usage is unavailable, inspect luna.gen_ai.usage.status and luna.gen_ai.usage.unavailable_reason.
      • Input and output throughput use the standard gen_ai.client.token.usage Histogram, with gen_ai.token.type set to input or output. Cache and reasoning breakdowns remain available from spans or the turn APIs.

      Cache breakdown availability depends on the model Provider's official response. Missing cache-write usage does not prove that a cache write occurred or that its size was zero.

      Analyze prompt-cache hit rate

      1. Select the period to analyze in the Agent observability overview. The overview hit rate aggregates the official model usage included in that period.
      2. When you open a trace detail, narrow the scope to model calls marked assistant in the current trace. Title generation and context summarization are marked title and summary, so their Tokens do not affect assistant-response usage or cache hit rate.
      3. At both levels, sum the token counts first and then calculate the official weighted hit rate: Σ cache-read input tokens / Σ input tokens. Do not take the arithmetic mean of per-request hit rates.

      Cache-write tokens are already included in input tokens, so they belong in the denominator but never in the cache-hit numerator. A missing standalone cache-write breakdown does not change the total-input denominator. If any included official usage omits its cache-read breakdown, or the summed input-token count is zero, display the hit rate as — (null in the API). Display 0% only when the summed input-token count is positive and cache-read usage was explicitly reported as zero.

      Prometheus scraping

      To scrape API metrics directly, configure:

      METRICS_ENABLED=true
METRICS_ADDR=:9090
METRICS_PATH=/metrics

      Expose this listener only to a controlled monitoring network. Worker and Agent metrics continue to use OTLP.

      Sensitive content capture

      AI_OBSERVABILITY_CAPTURE_CONTENT=true writes model inputs, outputs, tool arguments, and results as provided to controlled traces. Ordinary natural-language content is not heuristically redacted by keywords or regular expressions; fields explicitly declared through tool-catalog sensitivePaths are still replaced with [REDACTED]. If one content field exceeds 128 KiB of UTF-8 JSON, the entire field is omitted and the span receives luna.ai.content.truncated=true plus a luna.ai.content.omitted event. Correlated logs retain event metadata only, not content. Keep this setting disabled in production. Enable it only during a controlled diagnostic window after restricting Tempo access and retention, then restore false and restart Agent.

      Verify

      1. Make one successful API request and start one build or Release task.
      2. Confirm that traces, logs, and metrics for the same operation can be correlated.
      3. Stop the Collector temporarily and confirm that business requests continue and export resumes later.
      4. Check that telemetry contains no tokens, cookies, Secrets, request bodies, or model prompts.

      Production environments should enable TLS, authentication, least-privilege access, sampling, and retention policies for the Collector and backends.