#转录处理器 Runbook

last-verified: 2026-07-14（对照 infra PR #1459;发现和代码不符以代码为准）

RingCentral Call Log single-fetch、recording readiness、429 bounded redrive 与最新 structured log contract，见 RingCentral Call Log、录音就绪与 Rate Limit 调查。该调查以 infra PR #2035 的 live code 和 AWS test logs 为基线；本 runbook 中较早的 line number、固定 rate limit 与 reconciliation 描述不得覆盖该结论。

Function: TranscribeProcessorTS Runtime: Node.js 20 Memory: 256 MB Timeout: 15 minutes Reserved Concurrency: 2 (prevents RingCentral 429 rate limit storms) Purpose: Download call recordings from RingCentral → Upload to S3 → 用 Deepgram 同步转录（可疑结果由 Gemini 二次重写）

#Table of Contents

• Architecture Overview
• Transcription Providers (Deepgram + Gemini)
• Processing Flow (6 Phases)
• Critical Dependencies
• Common Issues
  • Issue 1: DLQ Messages
  • Issue 2: 429 Rate Limit Errors
  • Issue 3: 404 Not Found Errors
  • Issue 4: IAM Permission Denied
  • Issue 5: Deepgram 转录失败与二次重写
• Monitoring & Metrics
• CloudWatch Queries
• Recovery Procedures

#Architecture Overview

#Dual-Table DynamoDB Architecture (V2)

TranscribeProcessor uses atomic dual-table updates via TransactWriteItems:

call-events (Processing State)
├── PK: telephonySessionId (alphanumeric like "baa82aa2e94c46f29b2b3be2adac14ea")
├── Boolean Flags: transcriptionStarted, callLogFetched, recordingAvailable
└── Purpose: Prevent duplicate processing, track state

call-analysis (Dashboard Data)
├── PK/SK: orgId / timestamp
├── GSI: telephonySessionId-index
├── Fields: fromPhoneNumber, toPhoneNumber, hasRecording, s3AudioPath, etc.
└── Purpose: Query by client/time for dashboard, track business metrics

Atomic Operations (lambda/transcribe-processor/src/infrastructure/event-repository.ts):

• markCallLogFetchedAtomic() - Updates both tables in single transaction
• markTranscriptionStartedAtomic() - Sets transcriptionStarted flag + analysis metadata
• updateRecordingAvailableAtomic() - Handles delayed recording availability

#Transcription Providers (Deepgram + Gemini)

⚠️ AWS Transcribe 已不在转录路径里（PR #905 起移除）。本 runbook 里凡出现 "AWS Transcribe job" 的旧描述都已废弃 —— 当前转录是同步 Deepgram，可疑结果由 Gemini 二次重写，没有异步 Transcribe job、没有 job status 轮询。以 lambda/transcribe-processor/src/infrastructure/transcription-orchestrator.ts 为准。

#Provider 分工（现状）

没有 AWS Transcribe fallback。Deepgram 失败时不会切到别的 provider —— 而是 throw error → SQS 重投（原因：Deepgram outage 通常几分钟内恢复，SQS retry 比 AWS Transcribe fallback 便宜 5.6×，见 infra .claude/rules/transcribe-processor.md + PR #905）。

#三种策略（TRANSCRIPTION_AUDIO_STRATEGY 环境变量）

选择逻辑在 record-processor.ts:1848（default deepgram-first）：

Gemini 重写的 diarization 回退保护：若 Deepgram 检出 ≥2 个 speaker 而 Gemini 只找到 1 个，重写被拒绝、保留 Deepgram 结果（防止 Gemini 丢失 speaker 区分，见 transcription-orchestrator.ts:437）。

#Deepgram 配置

#Deepgram 失败在日志里长什么样

• 单次失败：logger.warn('Deepgram failed, will retry via the transcription pipeline', { error, jobName, consecutiveDeepgramFallbacks })（fallback-alert.ts）。这是可自愈的重试信号，不 page 人。
• 持续失败告警：连续 10 次 Deepgram 失败（FALLBACK_ALERT_THRESHOLD）且距上次告警 >1h 时，emit 一条 logger.error('DEEPGRAM_SUSTAINED_FAILURE: ...')。logger.error 只写 CloudWatch + Sentry，不发 Lark（见 Monitoring & Metrics 的告警语义说明）。
• Gemini 重写失败：logger.warn('Gemini rewrite failed, keeping Deepgram result', ...) —— 非 fatal，保留 Deepgram 结果继续。

注：fallback-alert.ts 源码顶部注释还写着 "Lark + Discord + Sentry" —— 那是过时注释；实际调 logger.error，只到 CloudWatch + Sentry。

CloudWatch Logs Query（Deepgram → Gemini provider 分布）：

fields @timestamp, jobName, provider, model
| filter @message like /Transcription completed/
| stats count() as jobs by provider, model

#Processing Flow (6 Phases)

#Phase 1: Webhook Parsing & Event History Tracking

Code: lambda/transcribe-processor/src/record-processor.ts:103-145

// Extract webhook from SQS message
const [clientId, webhookPayload] = parseWebhookFromSqsRecord(record, logger);

// Extract telephony event details
const eventDetails = extractTelephonyEventDetails(webhookPayload, logger);

// Track events: Setup, Answered, Disconnected

Key Behaviors:

• Tracks ALL events (Setup, Answered, Disconnected) but only processes recordings on Disconnected
• Extracts telephonySessionId (alphanumeric DynamoDB key like "baa82aa2e94c46f29b2b3be2adac14ea")
• Extracts sessionId (numeric RingCentral ID used for S3 paths like "4867428010")
• Validates webhook UUID for deduplication

Why Both IDs?

• telephonySessionId → DynamoDB PK (globally unique across all clients)
• sessionId → S3 path convention (matches RingCentral's numeric session ID)

Dashboard Metrics (Row 2):

• Widget 1: TranscribeProcessor Errors (should be <1%)
• Widget 2: Duration (should be <120 seconds)

CloudWatch Logs Query (Q3 - SQS Processing Latency):

fields @timestamp, @message, client_id, telephonySessionId
| filter @message like /Starting SQS Record Transcription Processing/
| sort @timestamp desc
| limit 20

#Phase 2: UUID Deduplication

Code: lambda/transcribe-processor/src/record-processor.ts:148-174

// Check existing call record
const existingCall = await eventRepo.getCallEvent(telephonySessionId);

if (existingCall) {
  const eventHistory = existingCall.eventHistory ?? [];
  const processedUuids = eventHistory.map(e => e.uuid).filter(uuid => uuid);

  if (webhookUuid && processedUuids.includes(webhookUuid)) {
    // Only skip if processing succeeded (s3AudioPath exists)
    if (existingCall.s3AudioPath) {
      logger.info('Webhook already processed successfully, skipping');
      return;
    }

    // UUID found but no s3AudioPath = processing failed, retry allowed
    logger.warn('Webhook found but processing incomplete, retrying');
  }
}

Why This Matters:

• RingCentral may send duplicate webhooks (network retries, failover)
• Prevents duplicate S3 uploads and Transcribe job starts
• Allows retries if previous processing failed (no s3AudioPath = incomplete)

CloudWatch Logs Query (Q7 - Duplicate Processing Detection):

fields @timestamp, webhookUuid, telephonySessionId, @message
| filter @message like /already processed/ or @message like /processing incomplete/
| stats count() as duplicates by webhookUuid
| sort duplicates desc

#Phase 3: API Enrichment (Call Log Fetch)

Code: lambda/transcribe-processor/src/record-processor.ts:280-405

Goal: Fetch authoritative call metadata from RingCentral Call Log API

// Wrap enrichment call with 404/429 error handling
const callLogResponse = await withRingCentralErrorHandling(
  () => ringcentralClient.fetchCallLog(telephonySessionId, secretName),
  telephonySessionId,
  callStartTime,
  sqsClient,
  queueUrl,
  receiptHandle,
  approximateReceiveCount,
  eventRepo,
  franchise,
  siteId,
  logger,
);

What Gets Enriched:

• callStartTime (authoritative, overwrites webhook timestamp)
• callDuration (only if call confirmed completed)
• fromPhoneNumber, fromName, toPhoneNumber, toName, toLocation
• callDirection, callResult, callType, callAction
• recordingType (OnDemand, Automatic)

Smart Guard Check (Delayed Recording Scenario):

// RingCentral may finish encoding AFTER Disconnected webhook arrives
const needsFirstFetch = existingCall?.callLogFetched !== true;
const needsRecordingUpgrade =
  existingCall?.callLogFetched === true &&
  recordingExists &&
  existingCall?.recordingAvailable !== true;

if (needsFirstFetch) {
  await eventRepo.markCallLogFetchedAtomic(...);
} else if (needsRecordingUpgrade) {
  await eventRepo.updateRecordingAvailableAtomic(...);
}

Dashboard Metrics (RingCentral API Health):

• RateLimitCapacity / RateLimitRemaining 以 runtime response headers 为准
• ApiCallsPerMinute 需要按 AccountId 与 API surface 结合 structured logs 判断，不硬编码统一的 per-client threshold

CloudWatch Logs Query (Q1 - API Rate Limit Detection):

fields @timestamp, client_id, @message, error
| filter @message like /429/ or @message like /rate limit/
| stats count() as rate_limit_errors by client_id
| sort rate_limit_errors desc

#Phase 4: 404/429 Error Handling

Code: lambda/transcribe-processor/src/record-processor.ts:522-629

#404 Not Found (Call Log Not Ready)

Root Cause: RingCentral Call Log API has 1-2 minute propagation delay after call ends

Retry Strategy:

if (statusCode === 404) {
  const elapsed = Date.now() - Date.parse(eventTime);
  const fifteenMinutes = 15 * 60 * 1000;

  if (elapsed < fifteenMinutes) {
    // Transient failure - retry via SQS
    throw error; // SQS will redeliver after visibility timeout
  } else {
    // Permanent failure - mark as failed
    await eventRepo.markCallLogFailedAtomic(telephonySessionId, '404_timeout');
    return null;
  }
}

Behavior:

• Elapsed < 15 minutes: Throw error → SQS retry (exponential backoff)
• Elapsed ≥ 15 minutes: Mark as failed permanently, return null

Dashboard Impact:

• Row 1: DLQ Message Count will increase if 15-minute timeout exceeded
• Row 2: Lambda Error Rate will spike during retry period

CloudWatch Logs Query (Q2 - Lambda Cold Starts & 404 Errors):

fields @timestamp, telephonySessionId, @message, error
| filter @message like /404 Not Found/ or error like /404/
| stats count() as errors_404 by bin(1h)
| sort @timestamp desc

#429 Too Many Requests (Rate Limit)

Root Cause: RingCentral 按 API group、authenticated user/app 等维度执行 rate limit，也可能存在 customized 或额外 protection policy。Call Log 属于 Heavy API；实际 capacity 以 runtime X-Rate-Limit-* headers 为准，不能把 10 req/min/client 当成所有 account 的固定规则。

Current bounded clone-and-redrive pattern (Retry-After + wide random spread):

if (statusCode === 429) {
  // Extract Retry-After header (usually 60 seconds)
  const retryAfterSeconds = extractRetryAfter(errorMessage) ?? 60;

  // CRITICAL: Add WIDE random spread (2-10 minutes) to prevent thundering herd
  const minAdditionalSpreadSeconds = 2 * 60;  // 2 minutes
  const maxAdditionalSpreadSeconds = 10 * 60; // 10 minutes
  const additionalSpread = minAdditionalSpreadSeconds +
    Math.floor(Math.random() * (maxAdditionalSpreadSeconds - minAdditionalSpreadSeconds));

  const jitteredDelaySeconds = retryAfterSeconds + additionalSpread;

  // PR #2035: send a new delayed message and carry a persistent redriveCount.
  // After 5 clones, stop cloning and return to normal SQS retry / DLQ handling.
  await sqsClient.send(new SendMessageCommand({
    QueueUrl: queueUrl,
    MessageBody: originalBody,
    DelaySeconds: Math.min(jitteredDelaySeconds, 900),
    MessageAttributes: { ...attributes, redriveCount: nextRedriveCount },
  }));
}

Why Wide Spread?

• 事故中同步重试持续撞到同一 account 的 429 window；不要假设快速重试会恢复 quota
• Without spread: All messages retry at same time → burst → more 429s
• With 2-10 min spread: Messages spread across wide window → avoids thundering herd

Clone Delay Examples (with Retry-After = 60s):

• Attempt 1: 60s + random(120-600s) = 180-660s (3-11 minutes)
• Attempt 2: 60s + random(120-600s) = 180-660s (3-11 minutes)
• 最多 5 次 clone；之后 429 走普通 SQS batchItemFailure

Dashboard Metrics (Row 13):

• Capacity metrics 只有在 response 同时提供合法 remaining / limit headers 时才 emit
• 429 必须结合 AccountId + Source metric series 和 ringcentral.rate_limited structured event 排查

CloudWatch Logs Query (RingCentral 429):

fields @timestamp, account_id, telephony_session_id, api_surface,
       rate_limit_group, retry_after_seconds, client_error_action
| filter message = "ringcentral.rate_limited"
| stats count() as throttles by account_id, api_surface, rate_limit_group
| sort throttles desc

#Phase 5: Recording Download & S3 Upload

Code: lambda/transcribe-processor/src/record-processor.ts:648-916

Steps:

1. Extract contentUri from RingCentral Call Log API response
2. Download recording MP3 to /tmp/{telephonySessionId}.mp3
3. Upload to S3: s3://recordings/{franchise}/{siteId}/{YYYY}/{MM}/{DD}/{sessionId}.mp3
4. Clean up /tmp/ file

S3 Path Structure (UTC-based) —— 以 core/path-builder.ts 为准：

s3://call-analytics-recordings/
  └── orangeTheory/                   ← franchise
      └── 2c9fc00886d14b9a9a24a12d337c438c/  ← siteId
          └── 2025/
              └── 11/
                  └── 30/
                      ├── 4867428010.mp3              ← Recording
                      ├── 4867428010-call-log.json    ← RingCentral API response
                      └── transcripts/
                          └── {siteId}-{telephonySessionId}/
                              └── transcription.json  ← Deepgram/Gemini transcript（同步写入）

Transcript 输出路径是 {franchise}/{siteId}/{YYYY}/{MM}/{DD}/transcripts/{safeSiteId}-{safeTelephonySessionId}/transcription.json（buildTranscriptOutputKey()），siteId / telephonySessionId 已剥掉所有非字母数字字符。这条路径 hardcode 进了 studio-api 读取逻辑，改动需跨 repo 协调。

Code Snippet (lambda/transcribe-processor/src/record-processor.ts:769-815):

const localPath = `/tmp/${telephonySessionId}.mp3`;

try {
  await withRingCentralErrorHandling(
    () => ringcentralClient.downloadRecording(contentUri, localPath, secretName),
    ...
  );

  s3Uri = await s3Repo.uploadFile(s3Paths.recordingKey, localPath);
  logger.info('Recording uploaded to S3', { s3Uri });
} finally {
  // Clean up temporary file
  await unlink(localPath);
}

Common Errors:

• IAM S3:PutObject denied: Lambda role missing S3 write permissions
• 429 Rate Limit: RingCentral recording download endpoint also subject to rate limits
• Disk space: /tmp/ has 10 GB limit (1000+ recordings max)

Dashboard Metrics (Row 3 - Lambda Duration):

• Widget 1: TranscribeProcessor Duration (spikes to >120s during download)
• Normal: <60s (API calls only)
• Slow: 60-120s (large recordings)
• Critical: >120s (near timeout)

CloudWatch Logs Query (Q5 - AI Analysis Performance):

fields @timestamp, @duration, telephonySessionId, @message
| filter @message like /Recording uploaded to S3/
| stats avg(@duration) as avg_duration_ms, max(@duration) as max_duration_ms by bin(1h)
| sort @timestamp desc

#Phase 6: Deepgram/Gemini 转录 + DynamoDB 更新

Code: lambda/transcribe-processor/src/record-processor.ts:1847-2016（转录编排在 infrastructure/transcription-orchestrator.ts）

⚠️ 这里不是 AWS Transcribe。转录是同步完成的：Deepgram（可疑结果由 Gemini 重写）在 Lambda 内直接产出 transcript 并写入 S3，然后才 markTranscriptionStartedAtomic。没有异步 job、没有 ConflictException、没有 job status 轮询。Provider / 策略细节见 Transcription Providers。

Steps:

1. Build 内部 job name：{env}-{region}-{siteId}-{telephonySessionId}（buildTranscribeJobName() —— 只是 transcript 的内部标识符 + 日志关联键，不是 AWS Transcribe job）
2. Build output key：{franchise}/{siteId}/{YYYY}/{MM}/{DD}/transcripts/{siteId}-{telephonySessionId}/transcription.json（buildTranscriptOutputKey()）
3. 选 strategy（TRANSCRIPTION_AUDIO_STRATEGY，default deepgram-first），调 transcribe() 同步转录 —— transcript JSON 在这一步就已写入 S3
4. 成功后 markTranscriptionStartedAtomic 原子更新 call-events + call-analysis 两张表

PII redaction：Deepgram 侧 redact=pci（只脱敏支付卡类，phone/name/SSN 按设计保留可见）。旧文档里那份 pii_entity_types JSON（CREDIT_DEBIT_NUMBER / SSN / BANK_ROUTING 等）是 AWS Transcribe 的 config，已不适用。

转录失败：transcribe() 返回 success:false 或 Deepgram throw 时，record-processor rethrow error → handler 的 Record processing failed catch 把 messageId 加进 batchItemFailures → SQS 重投（不 fallback 到别的 provider）。

Atomic DynamoDB Update (call-events + call-analysis)：

// record-processor.ts:2002 — transcript 此时已同步写入 S3
await eventRepo.markTranscriptionStartedAtomic(
  telephonySessionId,
  result.jobName ?? jobName,                          // 内部标识符，非 AWS job
  s3Uri,                                              // s3://bucket/path/to/recording.mp3
  s3Paths.recordingKey,                               // franchise/siteId/YYYY/MM/DD/sessionId.mp3
  result.synchronous ? outputKey : s3Paths.transcriptKey, // Deepgram/Gemini → 已写好的 outputKey
  s3Paths.callLogKey,                                 // franchise/siteId/YYYY/MM/DD/sessionId-call-log.json
);

What Gets Updated:

• call-events.transcriptionStarted → true (prevents duplicate processing)
• call-events.transcriptionJobName → 内部 job name（日志关联 / lookup 用）
• call-analysis.s3AudioPath → S3 recording path (for dashboard)
• call-analysis.s3TranscriptPath → transcript 实际路径（已写好，供 ai-analysis 读取）
• call-analysis.transcriptionJobName → 内部 job name (for troubleshooting)

CloudWatch Logs Query（Q6 - 成功转录 by provider）：

fields @timestamp, session, provider, jobName, @message
| filter @message like /Transcription job started successfully/
| stats count() as successful_jobs by provider
| sort successful_jobs desc

#Critical Dependencies

#Environment Variables

Code: lambda/transcribe-processor/src/handler.ts:88-140

CALL_EVENTS_TABLE_NAME      // call-events DynamoDB table
CALL_ANALYSIS_TABLE_NAME    // call-analysis DynamoDB table
CONFIG_TABLE_NAME            // client-configuration table
S3_BUCKET_NAME               // call-analytics-recordings-{env}
SQS_QUEUE_URL                // transcribe-processor-queue-multi-tenant
AWS_REGION                   // us-east-1 (prod) or us-east-2 (dev)
ENVIRONMENT                  // prod, dev, test

Validation: Strict environment validation on cold start (lines 105-140)

• Missing variables → Lambda throws Error on startup
• Prevents data corruption from misconfiguration

CloudWatch Logs Query (Environment Validation Errors):

fields @timestamp, @message, error
| filter @message like /Missing required environment variables/
| sort @timestamp desc
| limit 10

#IAM Permissions

Required Policies（IAM role 在 lib/stacks/iam-stack.ts 的 transcribeProcessorRole，Lambda 装配在 lib/stacks/lambda-stack.ts）：

// DynamoDB — call-events + call-analysis (读写) + config (只读)
dynamodbTable.grantReadWriteData(transcribeProcessor);
analysisTable.grantReadWriteData(transcribeProcessor);
configTable.grantReadData(transcribeProcessor);

// S3 — 录音 + transcript 输出
recordingsBucket.grantReadWrite(transcribeProcessor);

// Secrets Manager — Deepgram API key（secret 名由 DEEPGRAM_SECRET_NAME 指定）
// 注：RingCentral OAuth token 不走直连 Secrets Manager，而是 Lambda invoke
// GET_CREDENTIALS_FUNCTION_NAME（credentialsClient.getCredentials），见 Credential Access

// SQS — 429 重试时延长 visibility timeout
transcribeProcessor.addToRolePolicy(new PolicyStatement({
  actions: ['sqs:ChangeMessageVisibility'],
  resources: [queueArn],
}));

⚠️ 旧文档这里列了 transcribe:StartTranscriptionJob / GetTranscriptionJob / ListTranscriptionJobs 的 IAM policy。已废弃 —— 当前 CDK 里没有任何 transcribe:* action（grep -rn "transcribe:" lib/ 返回空），因为转录不再用 AWS Transcribe。

Common IAM Errors:

• Access Denied on S3:PutObject → Missing s3:PutObject permission
• Access Denied on Secrets Manager → Deepgram/RC secret 不匹配 policy 或不存在
• Access Denied on DynamoDB → Missing dynamodb:PutItem 或 dynamodb:TransactWriteItems

CloudWatch Logs Query (IAM Permission Errors):

fields @timestamp, @message, error, client_id
| filter error like /Access Denied/ or error like /AccessDeniedException/
| stats count() as denied_errors by error
| sort denied_errors desc

#SQS Configuration

Queue Name: transcribe-processor-queue-multi-tenant Visibility Timeout: 960 seconds (16 minutes) Message Retention: 14 days Reserved Concurrency: 2 (prevents RingCentral 429 storms)

Why 960s Visibility Timeout?

• Lambda timeout: 900s (15 minutes)
• Buffer: 60s for SQS message processing overhead
• If message visibility < Lambda timeout → duplicate processing risk

Dead Letter Queue (DLQ):

• Queue: transcribe-processor-dlq-multi-tenant
• Max Receives: 3（这是 clone budget 用尽后的普通 SQS receive path；不要与 MAX_RATE_LIMIT_REDRIVE_COUNT=5 混淆）
• Retention: 14 days

SQS Batch Item Failures Pattern (lambda/transcribe-processor/src/handler.ts:227-283):

const batchItemFailures: { itemIdentifier: string }[] = [];

for (const record of event.Records) {
  try {
    await processSingleRecord(...);
  } catch (error) {
    logger.error('Record processing failed', { error });
    batchItemFailures.push({ itemIdentifier: record.messageId });
  }
}

return { batchItemFailures };

Why This Matters:

• Without batchItemFailures: Entire batch fails, all messages retry
• With batchItemFailures: Only failed messages retry, successful ones deleted
• Prevents re-processing successful messages in large batches

Dashboard Metrics (Row 8 - SQS Queue Depth):

• Widget 1: Messages Visible (should be <10)
• Widget 2: Messages in Flight (should be <5)

CloudWatch Logs Query (Q3 - SQS Processing Latency):

fields @timestamp, @message, messageId, approximateReceiveCount
| filter @message like /Record processing failed/
| stats count() as retries by approximateReceiveCount
| sort retries desc

#Common Issues

#Issue 1: DLQ Messages

Symptom: Dashboard Row 1 shows DLQ message count >0

Investigation:

1. Check DLQ contents (AWS Console or CLI):

aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --max-number-of-messages 10

2. Parse message to extract client_id and telephonySessionId:

# Example message body:
{
  "client_id": "orangeTheory-2c9fc00886d14b9a9a24a12d337c438c",
  "telephonySessionId": "baa82aa2e94c46f29b2b3be2adac14ea"
}

3. Check CloudWatch Logs for error details:

fields @timestamp, @message, error, telephonySessionId
| filter telephonySessionId = "baa82aa2e94c46f29b2b3be2adac14ea"
| sort @timestamp desc
| limit 50

Common Causes:

#A. IAM Permission Denied

Error: Access Denied on S3, DynamoDB, Secrets Manager, or Transcribe

Fix:

1. Check Lambda execution role in IAM Console
2. Verify resource permissions match required policies (see IAM Permissions)
3. Redeploy CDK stack if permissions missing:

cd /path/to/callytics-infrastructure
cdk diff --context environment=prod
cdk deploy --context environment=prod

#B. Missing Secret in Secrets Manager

Error: Secret not found: ringcentral-api-tokens-{franchise}-{siteId}

Fix:

1. Check secret exists:

aws secretsmanager describe-secret --secret-id ringcentral-api-tokens-orangeTheory-2c9fc00886d14b9a9a24a12d337c438c

2. If missing, create secret:

aws secretsmanager create-secret \
  --name ringcentral-api-tokens-orangeTheory-2c9fc00886d14b9a9a24a12d337c438c \
  --secret-string '{"access_token":"xxx","refresh_token":"yyy","expires_at":1234567890}'

#C. Malformed Webhook Payload

Error: Cannot parse webhook payload or Missing required field: telephonySessionId

Fix:

1. Check DLQ message structure
2. Verify webhook source (RingCentral API Gateway format expected)
3. If payload is invalid, purge DLQ message (cannot be processed)

#D. 转录持续失败（Deepgram）

Error: Deepgram failed, will retry via the transcription pipeline / DEEPGRAM_SUSTAINED_FAILURE

转录失败会 rethrow → SQS 重投，5 次仍失败才进 DLQ。没有 AWS Transcribe job 冲突这回事（旧文档的 ConflictException + -retry-{timestamp} 已废弃）。

Fix:

• 排查见 Issue 5: Deepgram 转录失败与二次重写
• 确认 Deepgram secret / 服务状态；短暂 outage 会自愈，DLQ 里的走下方 redrive

Recovery (see DLQ Recovery Procedure):

#Issue 2: 429 Rate Limit Errors

Symptom: Dashboard Row 13 shows Rate Limit Capacity % dropping to 0%

Understanding the Issue:

• RingCentral enforces ~10 API calls/minute per client (medium rate limit)
• TranscribeProcessor makes 2 API calls per webhook:
  1. Call Log fetch (GET /restapi/v1.0/account/~/call-log)
  2. Recording download (GET /restapi/v1.0/account/~/recording/{id}/content)
• 429 Trap: Every 429 request RESETS the penalty clock (extends cooldown)

Automatic Recovery (lambda/transcribe-processor/src/record-processor.ts:574-619):

1. Detect 429 error
2. Parse Retry-After header (typically 60 seconds)
3. Add random spread: 2-10 minutes (120-600 seconds)
4. Extend SQS visibility timeout to Retry-After + spread
5. Throw error → SQS retry after delay

Retry Progression (with Retry-After = 60s):

• Message 1: Retry in 60s + 342s = 402s (6.7 minutes)
• Message 2: Retry in 60s + 187s = 247s (4.1 minutes)
• Message 3: Retry in 60s + 521s = 581s (9.7 minutes)
• Result: Messages spread across 3-11 minute window, no burst

CloudWatch Logs Pattern:

429 Rate Limit - spreading retry to prevent thundering herd
{
  session: "baa82aa2e94c46f29b2b3be2adac14ea",
  receiveCount: 2,
  retryAfterSeconds: 60,
  additionalSpreadSeconds: 342,
  totalDelaySeconds: 402,
  delayMinutes: "6.7",
  pattern: "Retry-After + Random(2-10min) spread"
}

Manual Intervention (if 429 errors persist):

1. Check reserved concurrency (should be 2):

aws lambda get-function-concurrency --function-name TranscribeProcessorTS-prod

2. Reduce concurrency further (temporary mitigation):

aws lambda put-function-concurrency --function-name TranscribeProcessorTS-prod --reserved-concurrent-executions 1

3. Monitor recovery (Row 13 - Rate Limit Capacity %):

• Normal: >50%
• Recovery: 20-50%
• Critical: <20%

4. Check per-client 429 errors (Q1 - API Rate Limit Detection):

fields @timestamp, client_id, @message
| filter @message like /429 Rate Limit/
| stats count() as rate_limit_errors by client_id
| sort rate_limit_errors desc

Permanent Fix (if one client constantly hits rate limit):

• Increase reconciliation window (reduce API call frequency)
• Contact RingCentral support to request rate limit increase
• Implement per-client rate limiting in CDK (future enhancement)

#Issue 3: 404 Not Found Errors

Symptom: Dashboard Row 2 shows TranscribeProcessor Errors increasing

Understanding the Issue:

• RingCentral Call Log API has 1-2 minute propagation delay after call ends
• Disconnected webhook arrives BEFORE call log is available
• TranscribeProcessor retries for 15 minutes before giving up

Expected Behavior (lambda/transcribe-processor/src/record-processor.ts:546-571):

Elapsed < 15 minutes:

404 Not Found - call log not ready yet, will retry
{
  session: "baa82aa2e94c46f29b2b3be2adac14ea",
  elapsedMs: 87000,         // 1.45 minutes
  thresholdMs: 900000       // 15 minutes
}

• Throw error → SQS retry (exponential backoff)
• Normal: Succeeds on 2nd-3rd attempt (2-5 minutes elapsed)

Elapsed ≥ 15 minutes:

404 timeout exceeded - marking call as failed
{
  session: "baa82aa2e94c46f29b2b3be2adac14ea",
  elapsedMs: 920000         // 15.3 minutes
}

• Mark as failed permanently in call-events table
• Update call-analysis.processingStatus → 404_timeout
• Message moved to DLQ

CloudWatch Logs Query (404 Analysis):

fields @timestamp, telephonySessionId, elapsedMs, @message
| filter @message like /404 Not Found/ or @message like /404 timeout/
| stats count() as errors_404, avg(elapsedMs) as avg_elapsed_ms by bin(1h)
| sort @timestamp desc

Investigation:

1. Check if call exists in RingCentral:

# Manual API call (requires access token from Secrets Manager)
curl -H "Authorization: Bearer $ACCESS_TOKEN" \
  "https://platform.ringcentral.com/restapi/v1.0/account/~/call-log?telephonySessionId=baa82aa2e94c46f29b2b3be2adac14ea"

2. Check DynamoDB for partial processing:

aws dynamodb get-item \
  --table-name call-events-prod \
  --key '{"telephonySessionId": {"S": "baa82aa2e94c46f29b2b3be2adac14ea"}}'

Common Causes:

#A. Call Deleted from RingCentral

• RingCentral retains call logs for 90 days
• After 90 days, Call Log API returns 404 permanently
• Fix: Cannot recover, purge DLQ message

#B. Webhook Timestamp Incorrect

• If webhook timestamp is wrong, elapsed time calculation is incorrect
• Example: Webhook timestamp = 1 day ago → immediate 404 timeout
• Fix: Investigate webhook source, contact RingCentral support

#C. RingCentral API Outage

• If RingCentral Call Log API is down, all calls return 404
• Fix: Check RingCentral Service Status (https://status.ringcentral.com)
• Wait for service restoration, DLQ messages can be redriven

Recovery:

• If call exists in RingCentral: Redrive DLQ message (see DLQ Recovery Procedure)
• If call deleted permanently: Purge DLQ message

#Issue 4: IAM Permission Denied

Symptom: Dashboard Row 2 shows TranscribeProcessor Errors, logs show Access Denied

Common Permission Errors:

#A. S3:PutObject Denied

Access Denied: User: arn:aws:sts::ACCOUNT:assumed-role/TranscribeProcessorRole-prod/TranscribeProcessorTS-prod
is not authorized to perform: s3:PutObject on resource:
arn:aws:s3:::call-analytics-recordings-prod/orangeTheory/2c9fc00886d14b9a9a24a12d337c438c/2025/11/30/4867428010.mp3

Root Cause: Lambda role missing S3 write permissions

Fix (lib/stacks/lambda-stack.ts):

recordingsBucket.grantReadWrite(transcribeProcessor);

Deployment:

cdk diff --context environment=prod
cdk deploy --context environment=prod

#B. Secrets Manager Access Denied

Access Denied: User: arn:aws:sts::ACCOUNT:assumed-role/TranscribeProcessorRole-prod/TranscribeProcessorTS-prod
is not authorized to perform: secretsmanager:GetSecretValue on resource:
arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:ringcentral-api-tokens-orangeTheory-2c9fc00886d14b9a9a24a12d337c438c

Root Cause: Secret name doesn't match IAM policy patterns

Fix: Verify secret name matches one of 3 patterns:

• ringcentral-api-tokens-*
• RINGCENTRAL@*
• RingCentral/*

If secret name is different, update IAM policy (lib/stacks/lambda-stack.ts):

transcribeProcessor.addToRolePolicy(new PolicyStatement({
  actions: ['secretsmanager:GetSecretValue'],
  resources: [
    `arn:aws:secretsmanager:${region}:${account}:secret:ringcentral-api-tokens-*`,
    `arn:aws:secretsmanager:${region}:${account}:secret:RINGCENTRAL@*`,
    `arn:aws:secretsmanager:${region}:${account}:secret:RingCentral/*`,
  ],
}));

#C. DynamoDB TransactWriteItems Denied

Access Denied: User: arn:aws:sts::ACCOUNT:assumed-role/TranscribeProcessorRole-prod/TranscribeProcessorTS-prod
is not authorized to perform: dynamodb:TransactWriteItems on resource:
arn:aws:dynamodb:us-east-1:ACCOUNT:table/call-events-prod

Root Cause: Lambda role missing DynamoDB write permissions

Fix (lib/stacks/lambda-stack.ts):

dynamodbTable.grantReadWriteData(transcribeProcessor);
analysisTable.grantReadWriteData(transcribeProcessor);

#Issue 5: Deepgram 转录失败与二次重写

旧版这里是 "Transcribe Job Name Conflicts"（AWS Transcribe ConflictException + -retry-{timestamp}）。已废弃 —— 转录是同步 Deepgram/Gemini，没有异步 job 名冲突。重复 webhook 由 Phase 2 UUID Deduplication 挡，不靠 job name。

Symptom: 日志出现 Deepgram failed, will retry via the transcription pipeline，或 DLQ 有消息且 CloudWatch 显示转录相关 error。

转录失败怎么处理（代码行为）：

1. 单次 Deepgram 失败 → logger.warn，error 被 rethrow → SQS 重投这条 message（不 fallback 到别的 provider）。多数 Deepgram 抖动在几分钟内自愈。
2. 可疑单 speaker 结果 → 触发 Gemini 重写（deepgram-first）；dual 策略下每通非 voicemail 都过 Gemini。Gemini 重写失败是非 fatal 的，保留 Deepgram 结果继续。
3. Gemini diarization 回退 → Deepgram ≥2 speaker 但 Gemini 只 1 speaker 时，重写被拒、保留 Deepgram（transcription-orchestrator.ts:437）。
4. 持续失败 → 连续 10 次 Deepgram 失败触发一条 DEEPGRAM_SUSTAINED_FAILURE logger.error（CloudWatch + Sentry，不 page）。

CloudWatch Logs Pattern（持续失败）：

DEEPGRAM_SUSTAINED_FAILURE: all transcriptions have failed for an extended period
{
  consecutiveDeepgramFallbacks: 10
}

投递失败在哪里 page：转录失败本身不直接 page 人。真正接人工告警的是下游 CloudWatch alarm —— 消息最终进 Transcribe DLQ（TranscribeDLQAlarm，阈值 1）才 page。见 Monitoring & Metrics + Issue 1: DLQ Messages。

Manual Investigation:

1. 确认是 Deepgram 侧还是我们侧：

fields @timestamp, session, provider, jobName, error, errorCause.message
| filter @message like /Deepgram failed/ or @message like /Gemini rewrite failed/ or @message like /DEEPGRAM_SUSTAINED_FAILURE/
| sort @timestamp desc
| limit 50

2. 看 provider 分布（Deepgram 成功率是否骤降 / Gemini 是否在大量兜底）：

fields @timestamp, provider
| filter @message like /Transcription job started successfully/
| stats count() as jobs by provider, bin(1h)

3. 确认 Deepgram secret 存在（DEEPGRAM_SECRET_NAME，default deepgram/api-key）：

aws secretsmanager describe-secret --secret-id deepgram/api-key

Recovery:

• Deepgram 短暂 outage：无需动作，SQS 重投会自愈；消息进 DLQ 才走 DLQ Recovery Procedure。
• Deepgram API key 失效 / secret 缺失：修好 secret 后 redrive DLQ。
• 持续失败告警：确认 Deepgram 服务状态（https://status.deepgram.com），必要时临时切 TRANSCRIPTION_AUDIO_STRATEGY=gemini-only（需部署环境变量改动，走 Park & Notify）。

#Monitoring & Metrics

#Dashboard Metrics (Row 2 - TranscribeProcessor)

Widget 1: Lambda Errors

• Metric: AWS/Lambda - Errors
• Dimension: FunctionName: TranscribeProcessorTS-prod
• Healthy: <1%
• Warning: 1-5%
• Critical: >5%

Widget 2: Lambda Duration

• Metric: AWS/Lambda - Duration
• Dimension: FunctionName: TranscribeProcessorTS-prod
• Healthy: <60 seconds (API calls only)
• Warning: 60-120 seconds (large recordings)
• Critical: >120 seconds (approaching timeout)

Widget 3: Lambda Invocations

• Metric: AWS/Lambda - Invocations
• Dimension: FunctionName: TranscribeProcessorTS-prod
• Pattern: Spikes correlate with call volume

#Custom Metrics (CallAnalytics/RingCentral Namespace)

Metric 1: Rate Limit Capacity %

// Code: lambda/transcribe-processor/src/utils/metrics.ts
metrics.addMetric('RateLimitCapacity', MetricUnit.Percent, capacityPercent);

• Healthy: >50%
• Warning: 20-50%
• Critical: <20% (approaching rate limit)

Metric 2: API Calls Per Minute

metrics.addMetric('ApiCallsPerMinute', MetricUnit.Count, callsPerMinute);

• Healthy: <8/min per client
• Warning: 8-10/min
• Critical: >10/min (rate limit risk)

Metric 3: Processing Duration

emitProcessingDurationMetrics(duration, success);

• Tracks successful vs failed processing time
• Success: Average <60s
• Failed: Average >120s (timeouts)

#告警语义（先读这个再看下面的 alarm）

日志级别 ≠ 告警级别。三者分开：

PROD 页人不靠 Lambda logger.error，靠 CloudWatch alarm。TranscribeProcessor 相关的可执行 PROD 告警只有 DLQ 这一路。以 lib/stacks/monitoring-stack.ts 为准。

#CloudWatch Alarms（lib/stacks/monitoring-stack.ts）

⚠️ 旧文档写的 TranscribeProcessorErrorRate（>10% error rate page P2）和 TranscribeProcessorDuration（≥14min page P3）在当前 stack 里不存在或不 page。Duration 只在 dashboard Row 3 可视化,没有独立 page 告警。

收到告警怎么办 → 告警响应 step（DLQ / queue-age / ingestion 等）统一在 infra repo 的 Pipeline Monitoring Runbook 维护，这里不重复。

#CloudWatch Queries

#Q1: API Rate Limit Detection (429 Errors)

fields @timestamp, client_id, @message, error, retryAfterSeconds, totalDelaySeconds
| filter @message like /429/ or @message like /rate limit/
| stats count() as rate_limit_errors by client_id
| sort rate_limit_errors desc

Purpose: Identify clients hitting RingCentral rate limits

Expected Output:

client_id                                                     | rate_limit_errors
orangeTheory-2c9fc00886d14b9a9a24a12d337c438c                | 45
planetFitness-5d8e91c997f25eab1b35b23e448d549d               | 12

#Q2: 404 Not Found Analysis

fields @timestamp, telephonySessionId, elapsedMs, thresholdMs, @message
| filter @message like /404 Not Found/ or @message like /404 timeout/
| stats count() as errors_404, avg(elapsedMs) as avg_elapsed_ms by bin(1h)
| sort @timestamp desc

Purpose: Analyze 404 error patterns and elapsed times

Expected Output:

bin(1h)              | errors_404 | avg_elapsed_ms
2025-11-30 14:00:00  | 23         | 92000          ← Normal (< 2 min)
2025-11-30 13:00:00  | 2          | 915000         ← Timeout (> 15 min)

#Q3: SQS Processing Latency

fields @timestamp, messageId, approximateReceiveCount, @duration
| filter @message like /Starting SQS Record Transcription Processing/
| stats avg(@duration) as avg_duration_ms, max(@duration) as max_duration_ms by approximateReceiveCount
| sort approximateReceiveCount asc

Purpose: Correlate retry count with processing duration

Expected Output:

approximateReceiveCount | avg_duration_ms | max_duration_ms
1                       | 4500            | 8200            ← First attempt
2                       | 6200            | 12000           ← Retry 1
3                       | 9800            | 18000           ← Retry 2

#Q4: Duplicate Processing Detection

fields @timestamp, webhookUuid, telephonySessionId, @message
| filter @message like /already processed/ or @message like /processing incomplete/
| stats count() as duplicates by webhookUuid
| sort duplicates desc

Purpose: Identify duplicate webhooks from RingCentral

Expected Output:

webhookUuid                                | duplicates
b7f2e3c4-5d6a-4f8b-9c0e-1a2b3c4d5e6f      | 3

#Q5: Recording Download Performance

fields @timestamp, @duration, telephonySessionId, s3Uri
| filter @message like /Recording uploaded to S3/
| stats avg(@duration) as avg_duration_ms, max(@duration) as max_duration_ms by bin(1h)
| sort @timestamp desc

Purpose: Track recording download and S3 upload duration

Expected Output:

bin(1h)              | avg_duration_ms | max_duration_ms
2025-11-30 14:00:00  | 45000           | 87000           ← Normal
2025-11-30 13:00:00  | 92000           | 145000          ← Slow downloads

#Q6: Successful Jobs by Client

fields @timestamp, client_id, jobName, @message
| filter @message like /Transcription job started successfully/
| stats count() as successful_jobs by client_id
| sort successful_jobs desc

Purpose: Verify processing success rate per client

Expected Output:

client_id                                                     | successful_jobs
orangeTheory-2c9fc00886d14b9a9a24a12d337c438c                | 1247
planetFitness-5d8e91c997f25eab1b35b23e448d549d               | 892

#Q7: IAM Permission Errors

fields @timestamp, @message, error, resource
| filter error like /Access Denied/ or error like /AccessDeniedException/
| stats count() as denied_errors by resource
| sort denied_errors desc

Purpose: Identify missing IAM permissions

Expected Output:

resource                                                                          | denied_errors
arn:aws:s3:::call-analytics-recordings-prod/orangeTheory/.../4867428010.mp3      | 34
arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:ringcentral-api-tokens-...       | 12

#Recovery Procedures

#DLQ Recovery Procedure

Goal: Redrive messages from DLQ back to main queue for retry

Prerequisites:

• Root cause identified and fixed (IAM permissions, secret created, etc.)
• Messages have not expired (14-day retention)

Steps:

1. Verify DLQ message count:

aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --attribute-names ApproximateNumberOfMessages

2. Sample 1-2 messages to verify fixability:

aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --max-number-of-messages 2

3. Parse message to extract client_id and telephonySessionId:

{
  "Body": "{\"client_id\":\"orangeTheory-2c9fc00886d14b9a9a24a12d337c438c\",\"telephonySessionId\":\"baa82aa2e94c46f29b2b3be2adac14ea\"}"
}

4. Verify fix (test one message manually if possible):

   • IAM permissions added? → Check Lambda role in IAM Console
   • Secret created? → Check Secrets Manager
   • 404 timeout? → Check if call still exists in RingCentral

5. Redrive all messages (AWS Console or CLI):

AWS Console:

• Navigate to SQS > DLQ queue
• Click "Start DLQ redrive"
• Select destination: transcribe-processor-queue-multi-tenant
• Click "Redrive messages"

AWS CLI:

aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:ACCOUNT:transcribe-processor-dlq-multi-tenant \
  --destination-arn arn:aws:sqs:us-east-1:ACCOUNT:transcribe-processor-queue-multi-tenant

6. Monitor recovery (Dashboard Row 1):

• DLQ message count should decrease to 0
• Main queue depth will spike temporarily (messages being reprocessed)
• Lambda invocations will increase (processing redriven messages)

7. Verify success (CloudWatch Logs):

fields @timestamp, telephonySessionId, @message
| filter @message like /Transcription job started successfully/
| stats count() as reprocessed_jobs by bin(5m)
| sort @timestamp desc

Expected Timeline:

• Small DLQ (<10 messages): 5-10 minutes
• Medium DLQ (10-100 messages): 30-60 minutes (limited by concurrency = 2)
• Large DLQ (>100 messages): 2-4 hours

#Purge Invalid DLQ Messages

When to Purge:

• Malformed webhook payload (cannot be parsed)
• Call deleted from RingCentral (permanent 404)
• Duplicate messages (already processed successfully)

Steps:

1. Backup DLQ messages (for audit):

aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --max-number-of-messages 10 > dlq-backup-$(date +%Y%m%d-%H%M%S).json

2. Purge DLQ:

aws sqs purge-queue \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant

WARNING: Purge is permanent and cannot be undone!

#Related Documentation

• AI Analysis Processor Runbook - Transcription → AI 分析（DeepSeek V4 Flash via OpenRouter，非 Bedrock）
• Reconciliation Runbook - Missing call recovery
• Pipeline Monitoring Runbook（infra repo） - 告警响应 step + 跨 4-Lambda trace（"收到告警怎么办" 的入口）
• On-Call Guide - Emergency response procedures

Last Updated: 2026-07-14（对照 infra PR #1459） Version: 2.1（Deepgram/Gemini 同步转录 + Dual-table DynamoDB） Owner: DevOps Team Next Review: 2026-10-14 (Quarterly)