> For AI agents: the complete documentation index is available at /llms.txt, the full documentation bundle is available at /llms-full.txt.

# 转录处理器 Runbook

> **last-verified: 2026-07-14**（对照 infra PR #1459;发现和代码不符以代码为准）
>
> RingCentral Call Log single-fetch、recording readiness、429 bounded redrive 与最新 structured log contract，见 [RingCentral Call Log、录音就绪与 Rate Limit 调查](/system-design/ringcentral-call-log-readiness-and-rate-limits.md)。该调查以 infra PR #2035 的 live code 和 AWS test logs 为基线；本 runbook 中较早的 line number、固定 rate limit 与 reconciliation 描述不得覆盖该结论。

**Function**: TranscribeProcessorTS
**Runtime**: Node.js 20
**Memory**: 256 MB
**Timeout**: 15 minutes
**Reserved Concurrency**: 2 (prevents RingCentral 429 rate limit storms)
**Purpose**: Download call recordings from RingCentral → Upload to S3 → 用 Deepgram 同步转录（可疑结果由 Gemini 二次重写）

***

## Table of Contents

- [Architecture Overview](#architecture-overview)
- [Transcription Providers (Deepgram + Gemini)](#transcription-providers-deepgram--gemini)
- [Processing Flow (6 Phases)](#processing-flow-6-phases)
- [Critical Dependencies](#critical-dependencies)
- [Common Issues](#common-issues)
  - [Issue 1: DLQ Messages](#issue-1-dlq-messages)
  - [Issue 2: 429 Rate Limit Errors](#issue-2-429-rate-limit-errors)
  - [Issue 3: 404 Not Found Errors](#issue-3-404-not-found-errors)
  - [Issue 4: IAM Permission Denied](#issue-4-iam-permission-denied)
  - [Issue 5: Deepgram 转录失败与二次重写](#issue-5-deepgram-转录失败与二次重写)
- [Monitoring & Metrics](#monitoring--metrics)
- [CloudWatch Queries](#cloudwatch-queries)
- [Recovery Procedures](#recovery-procedures)

***

## Architecture Overview

### Dual-Table DynamoDB Architecture (V2)

TranscribeProcessor uses **atomic dual-table updates** via `TransactWriteItems`:

```
call-events (Processing State)
├── PK: telephonySessionId (alphanumeric like "baa82aa2e94c46f29b2b3be2adac14ea")
├── Boolean Flags: transcriptionStarted, callLogFetched, recordingAvailable
└── Purpose: Prevent duplicate processing, track state

call-analysis (Dashboard Data)
├── PK/SK: orgId / timestamp
├── GSI: telephonySessionId-index
├── Fields: fromPhoneNumber, toPhoneNumber, hasRecording, s3AudioPath, etc.
└── Purpose: Query by client/time for dashboard, track business metrics
```

**Atomic Operations** (lambda/transcribe-processor/src/infrastructure/event-repository.ts):

- `markCallLogFetchedAtomic()` - Updates both tables in single transaction
- `markTranscriptionStartedAtomic()` - Sets transcriptionStarted flag + analysis metadata
- `updateRecordingAvailableAtomic()` - Handles delayed recording availability

***

## Transcription Providers (Deepgram + Gemini)

> ⚠️ **AWS Transcribe 已不在转录路径里**（PR #905 起移除）。本 runbook 里凡出现 "AWS Transcribe job" 的旧描述都已废弃 —— 当前转录是**同步** Deepgram，可疑结果由 Gemini 二次重写，没有异步 Transcribe job、没有 job status 轮询。以 `lambda/transcribe-processor/src/infrastructure/transcription-orchestrator.ts` 为准。

### Provider 分工（现状）

| Provider                                       | 角色                                                  | 何时用                       |
| ---------------------------------------------- | --------------------------------------------------- | ------------------------- |
| **Deepgram**                                   | Primary（同步转录 + speaker diarization + PII redaction） | 每通有录音的通话默认走这里             |
| **Gemini**（`gemini-flash-lite` via OpenRouter） | 二次重写 / rewrite                                      | 对 Deepgram 结果做二次判断（见下方策略） |

**没有 AWS Transcribe fallback**。Deepgram 失败时不会切到别的 provider —— 而是 throw error → SQS 重投（原因：Deepgram outage 通常几分钟内恢复，SQS retry 比 AWS Transcribe fallback 便宜 5.6×，见 infra `.claude/rules/transcribe-processor.md` + PR #905）。

### 三种策略（`TRANSCRIPTION_AUDIO_STRATEGY` 环境变量）

选择逻辑在 `record-processor.ts:1848`（default `deepgram-first`）：

| Strategy                  | 行为                                                                                                         |
| ------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `deepgram-first`（default） | 先 Deepgram；只对**可疑单 speaker** 结果触发 Gemini 重写（`shouldRewriteWithGemini()` 启发式）                               |
| `dual`                    | 同 `deepgram-first` 输入，但**每通非 voicemail、非空** transcript 都过一遍 Gemini 二次判断；Gemini 成功则采用，失败则保留 Deepgram（#1340） |
| `gemini-only`             | 跳过 Deepgram，直接 Gemini 转写                                                                                   |

**Gemini 重写的 diarization 回退保护**：若 Deepgram 检出 ≥2 个 speaker 而 Gemini 只找到 1 个，重写被拒绝、保留 Deepgram 结果（防止 Gemini 丢失 speaker 区分，见 `transcription-orchestrator.ts:437`）。

### Deepgram 配置

| 项             | 值 / 来源                                                                                                  |
| ------------- | ------------------------------------------------------------------------------------------------------- |
| 默认 model      | `nova-2-phonecall`（`DEFAULT_DEEPGRAM_MODEL`，低带宽电话调优）                                                    |
| A/B 实验 model  | `nova-3-general`（issue #674，仅 `test` env + account allowlist；`core/deepgram-model-routing.ts`，A/B 结束后删） |
| Diarization   | 始终开（`enableDiarization: true`）                                                                          |
| PII redaction | 始终开（`enableRedaction: true`；`redact=pci` —— 只脱敏支付卡类，phone/name/SSN 按设计保留可见）                             |
| API key       | Secrets Manager，secret 名由 `DEEPGRAM_SECRET_NAME` 指定（default `deepgram/api-key`）                         |
| 输出格式          | 归一化成 AWS Transcribe JSON 格式，下游消费方无需关心是哪个 provider 转的                                                    |

### Deepgram 失败在日志里长什么样

- **单次失败**：`logger.warn('Deepgram failed, will retry via the transcription pipeline', { error, jobName, consecutiveDeepgramFallbacks })`（`fallback-alert.ts`）。这是可自愈的重试信号，不 page 人。
- **持续失败告警**：连续 **10** 次 Deepgram 失败（`FALLBACK_ALERT_THRESHOLD`）且距上次告警 >1h 时，emit 一条 `logger.error('DEEPGRAM_SUSTAINED_FAILURE: ...')`。`logger.error` 只写 CloudWatch + Sentry，**不发 Lark**（见 [Monitoring & Metrics](#monitoring--metrics) 的告警语义说明）。
- **Gemini 重写失败**：`logger.warn('Gemini rewrite failed, keeping Deepgram result', ...)` —— 非 fatal，保留 Deepgram 结果继续。

> 注：`fallback-alert.ts` 源码顶部注释还写着 "Lark + Discord + Sentry" —— 那是过时注释；实际调 `logger.error`，只到 CloudWatch + Sentry。

**CloudWatch Logs Query**（Deepgram → Gemini provider 分布）：

```
fields @timestamp, jobName, provider, model
| filter @message like /Transcription completed/
| stats count() as jobs by provider, model
```

***

## Processing Flow (6 Phases)

### Phase 1: Webhook Parsing & Event History Tracking

**Code**: `lambda/transcribe-processor/src/record-processor.ts:103-145`

```typescript
// Extract webhook from SQS message
const [clientId, webhookPayload] = parseWebhookFromSqsRecord(record, logger);

// Extract telephony event details
const eventDetails = extractTelephonyEventDetails(webhookPayload, logger);

// Track events: Setup, Answered, Disconnected
```

**Key Behaviors**:

- Tracks ALL events (Setup, Answered, Disconnected) but only processes recordings on Disconnected
- Extracts `telephonySessionId` (alphanumeric DynamoDB key like "baa82aa2e94c46f29b2b3be2adac14ea")
- Extracts `sessionId` (numeric RingCentral ID used for S3 paths like "4867428010")
- Validates webhook UUID for deduplication

**Why Both IDs?**

- `telephonySessionId` → DynamoDB PK (globally unique across all clients)
- `sessionId` → S3 path convention (matches RingCentral's numeric session ID)

**Dashboard Metrics** (Row 2):

- Widget 1: TranscribeProcessor Errors (should be \<1%)
- Widget 2: Duration (should be \<120 seconds)

**CloudWatch Logs Query** (Q3 - SQS Processing Latency):

```
fields @timestamp, @message, client_id, telephonySessionId
| filter @message like /Starting SQS Record Transcription Processing/
| sort @timestamp desc
| limit 20
```

***

### Phase 2: UUID Deduplication

**Code**: `lambda/transcribe-processor/src/record-processor.ts:148-174`

```typescript
// Check existing call record
const existingCall = await eventRepo.getCallEvent(telephonySessionId);

if (existingCall) {
  const eventHistory = existingCall.eventHistory ?? [];
  const processedUuids = eventHistory.map(e => e.uuid).filter(uuid => uuid);

  if (webhookUuid && processedUuids.includes(webhookUuid)) {
    // Only skip if processing succeeded (s3AudioPath exists)
    if (existingCall.s3AudioPath) {
      logger.info('Webhook already processed successfully, skipping');
      return;
    }

    // UUID found but no s3AudioPath = processing failed, retry allowed
    logger.warn('Webhook found but processing incomplete, retrying');
  }
}
```

**Why This Matters**:

- RingCentral may send duplicate webhooks (network retries, failover)
- Prevents duplicate S3 uploads and Transcribe job starts
- Allows retries if previous processing failed (no `s3AudioPath` = incomplete)

**CloudWatch Logs Query** (Q7 - Duplicate Processing Detection):

```
fields @timestamp, webhookUuid, telephonySessionId, @message
| filter @message like /already processed/ or @message like /processing incomplete/
| stats count() as duplicates by webhookUuid
| sort duplicates desc
```

***

### Phase 3: API Enrichment (Call Log Fetch)

**Code**: `lambda/transcribe-processor/src/record-processor.ts:280-405`

**Goal**: Fetch authoritative call metadata from RingCentral Call Log API

```typescript
// Wrap enrichment call with 404/429 error handling
const callLogResponse = await withRingCentralErrorHandling(
  () => ringcentralClient.fetchCallLog(telephonySessionId, secretName),
  telephonySessionId,
  callStartTime,
  sqsClient,
  queueUrl,
  receiptHandle,
  approximateReceiveCount,
  eventRepo,
  franchise,
  siteId,
  logger,
);
```

**What Gets Enriched**:

- `callStartTime` (authoritative, overwrites webhook timestamp)
- `callDuration` (only if call confirmed completed)
- `fromPhoneNumber`, `fromName`, `toPhoneNumber`, `toName`, `toLocation`
- `callDirection`, `callResult`, `callType`, `callAction`
- `recordingType` (OnDemand, Automatic)

**Smart Guard Check** (Delayed Recording Scenario):

```typescript
// RingCentral may finish encoding AFTER Disconnected webhook arrives
const needsFirstFetch = existingCall?.callLogFetched !== true;
const needsRecordingUpgrade =
  existingCall?.callLogFetched === true &&
  recordingExists &&
  existingCall?.recordingAvailable !== true;

if (needsFirstFetch) {
  await eventRepo.markCallLogFetchedAtomic(...);
} else if (needsRecordingUpgrade) {
  await eventRepo.updateRecordingAvailableAtomic(...);
}
```

**Dashboard Metrics** (RingCentral API Health):

- `RateLimitCapacity` / `RateLimitRemaining` 以 runtime response headers 为准
- `ApiCallsPerMinute` 需要按 `AccountId` 与 API surface 结合 structured logs 判断，不硬编码统一的 per-client threshold

**CloudWatch Logs Query** (Q1 - API Rate Limit Detection):

```
fields @timestamp, client_id, @message, error
| filter @message like /429/ or @message like /rate limit/
| stats count() as rate_limit_errors by client_id
| sort rate_limit_errors desc
```

***

### Phase 4: 404/429 Error Handling

**Code**: `lambda/transcribe-processor/src/record-processor.ts:522-629`

#### 404 Not Found (Call Log Not Ready)

**Root Cause**: RingCentral Call Log API has 1-2 minute propagation delay after call ends

**Retry Strategy**:

```typescript
if (statusCode === 404) {
  const elapsed = Date.now() - Date.parse(eventTime);
  const fifteenMinutes = 15 * 60 * 1000;

  if (elapsed < fifteenMinutes) {
    // Transient failure - retry via SQS
    throw error; // SQS will redeliver after visibility timeout
  } else {
    // Permanent failure - mark as failed
    await eventRepo.markCallLogFailedAtomic(telephonySessionId, '404_timeout');
    return null;
  }
}
```

**Behavior**:

- **Elapsed \< 15 minutes**: Throw error → SQS retry (exponential backoff)
- **Elapsed ≥ 15 minutes**: Mark as failed permanently, return null

**Dashboard Impact**:

- Row 1: DLQ Message Count will increase if 15-minute timeout exceeded
- Row 2: Lambda Error Rate will spike during retry period

**CloudWatch Logs Query** (Q2 - Lambda Cold Starts & 404 Errors):

```
fields @timestamp, telephonySessionId, @message, error
| filter @message like /404 Not Found/ or error like /404/
| stats count() as errors_404 by bin(1h)
| sort @timestamp desc
```

***

#### 429 Too Many Requests (Rate Limit)

**Root Cause**: RingCentral 按 API group、authenticated user/app 等维度执行 rate limit，也可能存在 customized 或额外 protection policy。Call Log 属于 Heavy API；实际 capacity 以 runtime `X-Rate-Limit-*` headers 为准，不能把 `10 req/min/client` 当成所有 account 的固定规则。

**Current bounded clone-and-redrive pattern** (`Retry-After` + wide random spread):

```typescript
if (statusCode === 429) {
  // Extract Retry-After header (usually 60 seconds)
  const retryAfterSeconds = extractRetryAfter(errorMessage) ?? 60;

  // CRITICAL: Add WIDE random spread (2-10 minutes) to prevent thundering herd
  const minAdditionalSpreadSeconds = 2 * 60;  // 2 minutes
  const maxAdditionalSpreadSeconds = 10 * 60; // 10 minutes
  const additionalSpread = minAdditionalSpreadSeconds +
    Math.floor(Math.random() * (maxAdditionalSpreadSeconds - minAdditionalSpreadSeconds));

  const jitteredDelaySeconds = retryAfterSeconds + additionalSpread;

  // PR #2035: send a new delayed message and carry a persistent redriveCount.
  // After 5 clones, stop cloning and return to normal SQS retry / DLQ handling.
  await sqsClient.send(new SendMessageCommand({
    QueueUrl: queueUrl,
    MessageBody: originalBody,
    DelaySeconds: Math.min(jitteredDelaySeconds, 900),
    MessageAttributes: { ...attributes, redriveCount: nextRedriveCount },
  }));
}
```

**Why Wide Spread?**

- 事故中同步重试持续撞到同一 account 的 429 window；不要假设快速重试会恢复 quota
- Without spread: All messages retry at same time → burst → more 429s
- With 2-10 min spread: Messages spread across wide window → avoids thundering herd

**Clone Delay Examples** (with Retry-After = 60s):

- Attempt 1: 60s + random(120-600s) = 180-660s (3-11 minutes)
- Attempt 2: 60s + random(120-600s) = 180-660s (3-11 minutes)
- 最多 5 次 clone；之后 429 走普通 SQS `batchItemFailure`

**Dashboard Metrics** (Row 13):

- Capacity metrics 只有在 response 同时提供合法 remaining / limit headers 时才 emit
- 429 必须结合 `AccountId + Source` metric series 和 `ringcentral.rate_limited` structured event 排查

**CloudWatch Logs Query** (RingCentral 429):

```
fields @timestamp, account_id, telephony_session_id, api_surface,
       rate_limit_group, retry_after_seconds, client_error_action
| filter message = "ringcentral.rate_limited"
| stats count() as throttles by account_id, api_surface, rate_limit_group
| sort throttles desc
```

***

### Phase 5: Recording Download & S3 Upload

**Code**: `lambda/transcribe-processor/src/record-processor.ts:648-916`

**Steps**:

1. Extract `contentUri` from RingCentral Call Log API response
2. Download recording MP3 to `/tmp/{telephonySessionId}.mp3`
3. Upload to S3: `s3://recordings/{franchise}/{siteId}/{YYYY}/{MM}/{DD}/{sessionId}.mp3`
4. Clean up `/tmp/` file

**S3 Path Structure** (UTC-based) —— 以 `core/path-builder.ts` 为准：

```
s3://call-analytics-recordings/
  └── orangeTheory/                   ← franchise
      └── 2c9fc00886d14b9a9a24a12d337c438c/  ← siteId
          └── 2025/
              └── 11/
                  └── 30/
                      ├── 4867428010.mp3              ← Recording
                      ├── 4867428010-call-log.json    ← RingCentral API response
                      └── transcripts/
                          └── {siteId}-{telephonySessionId}/
                              └── transcription.json  ← Deepgram/Gemini transcript（同步写入）
```

> Transcript 输出路径是 `{franchise}/{siteId}/{YYYY}/{MM}/{DD}/transcripts/{safeSiteId}-{safeTelephonySessionId}/transcription.json`（`buildTranscriptOutputKey()`），siteId / telephonySessionId 已剥掉所有非字母数字字符。这条路径 hardcode 进了 `studio-api` 读取逻辑，改动需跨 repo 协调。

**Code Snippet** (lambda/transcribe-processor/src/record-processor.ts:769-815):

```typescript
const localPath = `/tmp/${telephonySessionId}.mp3`;

try {
  await withRingCentralErrorHandling(
    () => ringcentralClient.downloadRecording(contentUri, localPath, secretName),
    ...
  );

  s3Uri = await s3Repo.uploadFile(s3Paths.recordingKey, localPath);
  logger.info('Recording uploaded to S3', { s3Uri });
} finally {
  // Clean up temporary file
  await unlink(localPath);
}
```

**Common Errors**:

- **IAM S3:PutObject denied**: Lambda role missing S3 write permissions
- **429 Rate Limit**: RingCentral recording download endpoint also subject to rate limits
- **Disk space**: `/tmp/` has 10 GB limit (1000+ recordings max)

**Dashboard Metrics** (Row 3 - Lambda Duration):

- Widget 1: TranscribeProcessor Duration (spikes to >120s during download)
- Normal: \<60s (API calls only)
- Slow: 60-120s (large recordings)
- Critical: >120s (near timeout)

**CloudWatch Logs Query** (Q5 - AI Analysis Performance):

```
fields @timestamp, @duration, telephonySessionId, @message
| filter @message like /Recording uploaded to S3/
| stats avg(@duration) as avg_duration_ms, max(@duration) as max_duration_ms by bin(1h)
| sort @timestamp desc
```

***

### Phase 6: Deepgram/Gemini 转录 + DynamoDB 更新

**Code**: `lambda/transcribe-processor/src/record-processor.ts:1847-2016`（转录编排在 `infrastructure/transcription-orchestrator.ts`）

> ⚠️ 这里**不是** AWS Transcribe。转录是**同步**完成的：Deepgram（可疑结果由 Gemini 重写）在 Lambda 内直接产出 transcript 并写入 S3，然后才 `markTranscriptionStartedAtomic`。没有异步 job、没有 `ConflictException`、没有 job status 轮询。Provider / 策略细节见 [Transcription Providers](#transcription-providers-deepgram--gemini)。

**Steps**:

1. Build 内部 job name：`{env}-{region}-{siteId}-{telephonySessionId}`（`buildTranscribeJobName()` —— 只是 transcript 的内部标识符 + 日志关联键，**不是** AWS Transcribe job）
2. Build output key：`{franchise}/{siteId}/{YYYY}/{MM}/{DD}/transcripts/{siteId}-{telephonySessionId}/transcription.json`（`buildTranscriptOutputKey()`）
3. 选 strategy（`TRANSCRIPTION_AUDIO_STRATEGY`，default `deepgram-first`），调 `transcribe()` 同步转录 —— transcript JSON 在这一步就已写入 S3
4. 成功后 `markTranscriptionStartedAtomic` 原子更新 call-events + call-analysis 两张表

**PII redaction**：Deepgram 侧 `redact=pci`（只脱敏支付卡类，phone/name/SSN 按设计保留可见）。旧文档里那份 `pii_entity_types` JSON（`CREDIT_DEBIT_NUMBER` / `SSN` / `BANK_ROUTING` 等）是 AWS Transcribe 的 config，已不适用。

**转录失败**：`transcribe()` 返回 `success:false` 或 Deepgram throw 时，`record-processor` rethrow error → handler 的 `Record processing failed` catch 把 messageId 加进 `batchItemFailures` → SQS 重投（不 fallback 到别的 provider）。

**Atomic DynamoDB Update** (call-events + call-analysis)：

```typescript
// record-processor.ts:2002 — transcript 此时已同步写入 S3
await eventRepo.markTranscriptionStartedAtomic(
  telephonySessionId,
  result.jobName ?? jobName,                          // 内部标识符，非 AWS job
  s3Uri,                                              // s3://bucket/path/to/recording.mp3
  s3Paths.recordingKey,                               // franchise/siteId/YYYY/MM/DD/sessionId.mp3
  result.synchronous ? outputKey : s3Paths.transcriptKey, // Deepgram/Gemini → 已写好的 outputKey
  s3Paths.callLogKey,                                 // franchise/siteId/YYYY/MM/DD/sessionId-call-log.json
);
```

**What Gets Updated**:

- `call-events.transcriptionStarted` → `true` (prevents duplicate processing)
- `call-events.transcriptionJobName` → 内部 job name（日志关联 / lookup 用）
- `call-analysis.s3AudioPath` → S3 recording path (for dashboard)
- `call-analysis.s3TranscriptPath` → transcript 实际路径（已写好，供 ai-analysis 读取）
- `call-analysis.transcriptionJobName` → 内部 job name (for troubleshooting)

**CloudWatch Logs Query**（Q6 - 成功转录 by provider）：

```
fields @timestamp, session, provider, jobName, @message
| filter @message like /Transcription job started successfully/
| stats count() as successful_jobs by provider
| sort successful_jobs desc
```

***

## Critical Dependencies

### Environment Variables

**Code**: `lambda/transcribe-processor/src/handler.ts:88-140`

```typescript
CALL_EVENTS_TABLE_NAME      // call-events DynamoDB table
CALL_ANALYSIS_TABLE_NAME    // call-analysis DynamoDB table
CONFIG_TABLE_NAME            // client-configuration table
S3_BUCKET_NAME               // call-analytics-recordings-{env}
SQS_QUEUE_URL                // transcribe-processor-queue-multi-tenant
AWS_REGION                   // us-east-1 (prod) or us-east-2 (dev)
ENVIRONMENT                  // prod, dev, test
```

**Validation**: Strict environment validation on cold start (lines 105-140)

- Missing variables → Lambda throws Error on startup
- Prevents data corruption from misconfiguration

**CloudWatch Logs Query** (Environment Validation Errors):

```
fields @timestamp, @message, error
| filter @message like /Missing required environment variables/
| sort @timestamp desc
| limit 10
```

***

### IAM Permissions

**Required Policies**（IAM role 在 `lib/stacks/iam-stack.ts` 的 `transcribeProcessorRole`，Lambda 装配在 `lib/stacks/lambda-stack.ts`）：

```typescript
// DynamoDB — call-events + call-analysis (读写) + config (只读)
dynamodbTable.grantReadWriteData(transcribeProcessor);
analysisTable.grantReadWriteData(transcribeProcessor);
configTable.grantReadData(transcribeProcessor);

// S3 — 录音 + transcript 输出
recordingsBucket.grantReadWrite(transcribeProcessor);

// Secrets Manager — Deepgram API key（secret 名由 DEEPGRAM_SECRET_NAME 指定）
// 注：RingCentral OAuth token 不走直连 Secrets Manager，而是 Lambda invoke
// GET_CREDENTIALS_FUNCTION_NAME（credentialsClient.getCredentials），见 Credential Access

// SQS — 429 重试时延长 visibility timeout
transcribeProcessor.addToRolePolicy(new PolicyStatement({
  actions: ['sqs:ChangeMessageVisibility'],
  resources: [queueArn],
}));
```

> ⚠️ 旧文档这里列了 `transcribe:StartTranscriptionJob` / `GetTranscriptionJob` / `ListTranscriptionJobs` 的 IAM policy。**已废弃** —— 当前 CDK 里没有任何 `transcribe:*` action（`grep -rn "transcribe:" lib/` 返回空），因为转录不再用 AWS Transcribe。

**Common IAM Errors**:

- `Access Denied` on S3:PutObject → Missing `s3:PutObject` permission
- `Access Denied` on Secrets Manager → Deepgram/RC secret 不匹配 policy 或不存在
- `Access Denied` on DynamoDB → Missing `dynamodb:PutItem` 或 `dynamodb:TransactWriteItems`

**CloudWatch Logs Query** (IAM Permission Errors):

```
fields @timestamp, @message, error, client_id
| filter error like /Access Denied/ or error like /AccessDeniedException/
| stats count() as denied_errors by error
| sort denied_errors desc
```

***

### SQS Configuration

**Queue Name**: `transcribe-processor-queue-multi-tenant`
**Visibility Timeout**: 960 seconds (16 minutes)
**Message Retention**: 14 days
**Reserved Concurrency**: 2 (prevents RingCentral 429 storms)

**Why 960s Visibility Timeout?**

- Lambda timeout: 900s (15 minutes)
- Buffer: 60s for SQS message processing overhead
- If message visibility \< Lambda timeout → duplicate processing risk

**Dead Letter Queue** (DLQ):

- Queue: `transcribe-processor-dlq-multi-tenant`
- Max Receives: 3（这是 clone budget 用尽后的普通 SQS receive path；不要与 `MAX_RATE_LIMIT_REDRIVE_COUNT=5` 混淆）
- Retention: 14 days

**SQS Batch Item Failures Pattern** (lambda/transcribe-processor/src/handler.ts:227-283):

```typescript
const batchItemFailures: { itemIdentifier: string }[] = [];

for (const record of event.Records) {
  try {
    await processSingleRecord(...);
  } catch (error) {
    logger.error('Record processing failed', { error });
    batchItemFailures.push({ itemIdentifier: record.messageId });
  }
}

return { batchItemFailures };
```

**Why This Matters**:

- Without `batchItemFailures`: Entire batch fails, all messages retry
- With `batchItemFailures`: Only failed messages retry, successful ones deleted
- Prevents re-processing successful messages in large batches

**Dashboard Metrics** (Row 8 - SQS Queue Depth):

- Widget 1: Messages Visible (should be \<10)
- Widget 2: Messages in Flight (should be \<5)

**CloudWatch Logs Query** (Q3 - SQS Processing Latency):

```
fields @timestamp, @message, messageId, approximateReceiveCount
| filter @message like /Record processing failed/
| stats count() as retries by approximateReceiveCount
| sort retries desc
```

***

## Common Issues

### Issue 1: DLQ Messages

**Symptom**: Dashboard Row 1 shows DLQ message count >0

**Investigation**:

1. **Check DLQ contents** (AWS Console or CLI):

```bash
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --max-number-of-messages 10
```

2. **Parse message to extract client\_id and telephonySessionId**:

```bash
# Example message body:
{
  "client_id": "orangeTheory-2c9fc00886d14b9a9a24a12d337c438c",
  "telephonySessionId": "baa82aa2e94c46f29b2b3be2adac14ea"
}
```

3. **Check CloudWatch Logs for error details**:

```
fields @timestamp, @message, error, telephonySessionId
| filter telephonySessionId = "baa82aa2e94c46f29b2b3be2adac14ea"
| sort @timestamp desc
| limit 50
```

**Common Causes**:

#### A. IAM Permission Denied

**Error**: `Access Denied` on S3, DynamoDB, Secrets Manager, or Transcribe

**Fix**:

1. Check Lambda execution role in IAM Console
2. Verify resource permissions match required policies (see [IAM Permissions](#iam-permissions))
3. Redeploy CDK stack if permissions missing:

```bash
cd /path/to/callytics-infrastructure
cdk diff --context environment=prod
cdk deploy --context environment=prod
```

#### B. Missing Secret in Secrets Manager

**Error**: `Secret not found: ringcentral-api-tokens-{franchise}-{siteId}`

**Fix**:

1. Check secret exists:

```bash
aws secretsmanager describe-secret --secret-id ringcentral-api-tokens-orangeTheory-2c9fc00886d14b9a9a24a12d337c438c
```

2. If missing, create secret:

```bash
aws secretsmanager create-secret \
  --name ringcentral-api-tokens-orangeTheory-2c9fc00886d14b9a9a24a12d337c438c \
  --secret-string '{"access_token":"xxx","refresh_token":"yyy","expires_at":1234567890}'
```

#### C. Malformed Webhook Payload

**Error**: `Cannot parse webhook payload` or `Missing required field: telephonySessionId`

**Fix**:

1. Check DLQ message structure
2. Verify webhook source (RingCentral API Gateway format expected)
3. If payload is invalid, purge DLQ message (cannot be processed)

#### D. 转录持续失败（Deepgram）

**Error**: `Deepgram failed, will retry via the transcription pipeline` / `DEEPGRAM_SUSTAINED_FAILURE`

转录失败会 rethrow → SQS 重投，5 次仍失败才进 DLQ。**没有** AWS Transcribe job 冲突这回事（旧文档的 `ConflictException` + `-retry-{timestamp}` 已废弃）。

**Fix**:

- 排查见 [Issue 5: Deepgram 转录失败与二次重写](#issue-5-deepgram-转录失败与二次重写)
- 确认 Deepgram secret / 服务状态；短暂 outage 会自愈，DLQ 里的走下方 redrive

**Recovery** (see [DLQ Recovery Procedure](#dlq-recovery-procedure)):

***

### Issue 2: 429 Rate Limit Errors

**Symptom**: Dashboard Row 13 shows Rate Limit Capacity % dropping to 0%

**Understanding the Issue**:

- RingCentral enforces \~10 API calls/minute per client (medium rate limit)
- TranscribeProcessor makes 2 API calls per webhook:
  1. Call Log fetch (`GET /restapi/v1.0/account/~/call-log`)
  2. Recording download (`GET /restapi/v1.0/account/~/recording/{id}/content`)
- **429 Trap**: Every 429 request RESETS the penalty clock (extends cooldown)

**Automatic Recovery** (lambda/transcribe-processor/src/record-processor.ts:574-619):

1. Detect 429 error
2. Parse `Retry-After` header (typically 60 seconds)
3. Add random spread: 2-10 minutes (120-600 seconds)
4. Extend SQS visibility timeout to `Retry-After + spread`
5. Throw error → SQS retry after delay

**Retry Progression** (with Retry-After = 60s):

- Message 1: Retry in 60s + 342s = 402s (6.7 minutes)
- Message 2: Retry in 60s + 187s = 247s (4.1 minutes)
- Message 3: Retry in 60s + 521s = 581s (9.7 minutes)
- **Result**: Messages spread across 3-11 minute window, no burst

**CloudWatch Logs Pattern**:

```
429 Rate Limit - spreading retry to prevent thundering herd
{
  session: "baa82aa2e94c46f29b2b3be2adac14ea",
  receiveCount: 2,
  retryAfterSeconds: 60,
  additionalSpreadSeconds: 342,
  totalDelaySeconds: 402,
  delayMinutes: "6.7",
  pattern: "Retry-After + Random(2-10min) spread"
}
```

**Manual Intervention** (if 429 errors persist):

1. **Check reserved concurrency** (should be 2):

```bash
aws lambda get-function-concurrency --function-name TranscribeProcessorTS-prod
```

2. **Reduce concurrency further** (temporary mitigation):

```bash
aws lambda put-function-concurrency --function-name TranscribeProcessorTS-prod --reserved-concurrent-executions 1
```

3. **Monitor recovery** (Row 13 - Rate Limit Capacity %):

- Normal: >50%
- Recovery: 20-50%
- Critical: \<20%

4. **Check per-client 429 errors** (Q1 - API Rate Limit Detection):

```
fields @timestamp, client_id, @message
| filter @message like /429 Rate Limit/
| stats count() as rate_limit_errors by client_id
| sort rate_limit_errors desc
```

**Permanent Fix** (if one client constantly hits rate limit):

- Increase reconciliation window (reduce API call frequency)
- Contact RingCentral support to request rate limit increase
- Implement per-client rate limiting in CDK (future enhancement)

***

### Issue 3: 404 Not Found Errors

**Symptom**: Dashboard Row 2 shows TranscribeProcessor Errors increasing

**Understanding the Issue**:

- RingCentral Call Log API has 1-2 minute propagation delay after call ends
- Disconnected webhook arrives BEFORE call log is available
- TranscribeProcessor retries for 15 minutes before giving up

**Expected Behavior** (lambda/transcribe-processor/src/record-processor.ts:546-571):

**Elapsed \< 15 minutes**:

```
404 Not Found - call log not ready yet, will retry
{
  session: "baa82aa2e94c46f29b2b3be2adac14ea",
  elapsedMs: 87000,         // 1.45 minutes
  thresholdMs: 900000       // 15 minutes
}
```

- Throw error → SQS retry (exponential backoff)
- Normal: Succeeds on 2nd-3rd attempt (2-5 minutes elapsed)

**Elapsed ≥ 15 minutes**:

```
404 timeout exceeded - marking call as failed
{
  session: "baa82aa2e94c46f29b2b3be2adac14ea",
  elapsedMs: 920000         // 15.3 minutes
}
```

- Mark as failed permanently in `call-events` table
- Update `call-analysis.processingStatus` → `404_timeout`
- Message moved to DLQ

**CloudWatch Logs Query** (404 Analysis):

```
fields @timestamp, telephonySessionId, elapsedMs, @message
| filter @message like /404 Not Found/ or @message like /404 timeout/
| stats count() as errors_404, avg(elapsedMs) as avg_elapsed_ms by bin(1h)
| sort @timestamp desc
```

**Investigation**:

1. **Check if call exists in RingCentral**:

```bash
# Manual API call (requires access token from Secrets Manager)
curl -H "Authorization: Bearer $ACCESS_TOKEN" \
  "https://platform.ringcentral.com/restapi/v1.0/account/~/call-log?telephonySessionId=baa82aa2e94c46f29b2b3be2adac14ea"
```

2. **Check DynamoDB for partial processing**:

```bash
aws dynamodb get-item \
  --table-name call-events-prod \
  --key '{"telephonySessionId": {"S": "baa82aa2e94c46f29b2b3be2adac14ea"}}'
```

**Common Causes**:

#### A. Call Deleted from RingCentral

- RingCentral retains call logs for 90 days
- After 90 days, Call Log API returns 404 permanently
- **Fix**: Cannot recover, purge DLQ message

#### B. Webhook Timestamp Incorrect

- If webhook `timestamp` is wrong, elapsed time calculation is incorrect
- Example: Webhook timestamp = 1 day ago → immediate 404 timeout
- **Fix**: Investigate webhook source, contact RingCentral support

#### C. RingCentral API Outage

- If RingCentral Call Log API is down, all calls return 404
- **Fix**: Check RingCentral Service Status ([https://status.ringcentral.com](https://status.ringcentral.com))
- Wait for service restoration, DLQ messages can be redriven

**Recovery**:

- If call exists in RingCentral: Redrive DLQ message (see [DLQ Recovery Procedure](#dlq-recovery-procedure))
- If call deleted permanently: Purge DLQ message

***

### Issue 4: IAM Permission Denied

**Symptom**: Dashboard Row 2 shows TranscribeProcessor Errors, logs show `Access Denied`

**Common Permission Errors**:

#### A. S3:PutObject Denied

```
Access Denied: User: arn:aws:sts::ACCOUNT:assumed-role/TranscribeProcessorRole-prod/TranscribeProcessorTS-prod
is not authorized to perform: s3:PutObject on resource:
arn:aws:s3:::call-analytics-recordings-prod/orangeTheory/2c9fc00886d14b9a9a24a12d337c438c/2025/11/30/4867428010.mp3
```

**Root Cause**: Lambda role missing S3 write permissions

**Fix** (lib/stacks/lambda-stack.ts):

```typescript
recordingsBucket.grantReadWrite(transcribeProcessor);
```

**Deployment**:

```bash
cdk diff --context environment=prod
cdk deploy --context environment=prod
```

#### B. Secrets Manager Access Denied

```
Access Denied: User: arn:aws:sts::ACCOUNT:assumed-role/TranscribeProcessorRole-prod/TranscribeProcessorTS-prod
is not authorized to perform: secretsmanager:GetSecretValue on resource:
arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:ringcentral-api-tokens-orangeTheory-2c9fc00886d14b9a9a24a12d337c438c
```

**Root Cause**: Secret name doesn't match IAM policy patterns

**Fix**: Verify secret name matches one of 3 patterns:

- `ringcentral-api-tokens-*`
- `RINGCENTRAL@*`
- `RingCentral/*`

**If secret name is different, update IAM policy** (lib/stacks/lambda-stack.ts):

```typescript
transcribeProcessor.addToRolePolicy(new PolicyStatement({
  actions: ['secretsmanager:GetSecretValue'],
  resources: [
    `arn:aws:secretsmanager:${region}:${account}:secret:ringcentral-api-tokens-*`,
    `arn:aws:secretsmanager:${region}:${account}:secret:RINGCENTRAL@*`,
    `arn:aws:secretsmanager:${region}:${account}:secret:RingCentral/*`,
  ],
}));
```

#### C. DynamoDB TransactWriteItems Denied

```
Access Denied: User: arn:aws:sts::ACCOUNT:assumed-role/TranscribeProcessorRole-prod/TranscribeProcessorTS-prod
is not authorized to perform: dynamodb:TransactWriteItems on resource:
arn:aws:dynamodb:us-east-1:ACCOUNT:table/call-events-prod
```

**Root Cause**: Lambda role missing DynamoDB write permissions

**Fix** (lib/stacks/lambda-stack.ts):

```typescript
dynamodbTable.grantReadWriteData(transcribeProcessor);
analysisTable.grantReadWriteData(transcribeProcessor);
```

***

### Issue 5: Deepgram 转录失败与二次重写

> 旧版这里是 "Transcribe Job Name Conflicts"（AWS Transcribe `ConflictException` + `-retry-{timestamp}`）。**已废弃** —— 转录是同步 Deepgram/Gemini，没有异步 job 名冲突。重复 webhook 由 [Phase 2 UUID Deduplication](#phase-2-uuid-deduplication) 挡，不靠 job name。

**Symptom**: 日志出现 `Deepgram failed, will retry via the transcription pipeline`，或 DLQ 有消息且 CloudWatch 显示转录相关 error。

**转录失败怎么处理（代码行为）**：

1. **单次 Deepgram 失败** → `logger.warn`，error 被 rethrow → SQS 重投这条 message（**不** fallback 到别的 provider）。多数 Deepgram 抖动在几分钟内自愈。
2. **可疑单 speaker 结果** → 触发 Gemini 重写（`deepgram-first`）；`dual` 策略下每通非 voicemail 都过 Gemini。Gemini 重写失败是**非 fatal** 的，保留 Deepgram 结果继续。
3. **Gemini diarization 回退** → Deepgram ≥2 speaker 但 Gemini 只 1 speaker 时，重写被拒、保留 Deepgram（`transcription-orchestrator.ts:437`）。
4. **持续失败** → 连续 10 次 Deepgram 失败触发一条 `DEEPGRAM_SUSTAINED_FAILURE` `logger.error`（CloudWatch + Sentry，不 page）。

**CloudWatch Logs Pattern**（持续失败）：

```
DEEPGRAM_SUSTAINED_FAILURE: all transcriptions have failed for an extended period
{
  consecutiveDeepgramFallbacks: 10
}
```

**投递失败在哪里 page**：转录失败本身不直接 page 人。真正接人工告警的是下游 CloudWatch alarm —— 消息最终进 Transcribe DLQ（`TranscribeDLQAlarm`，阈值 1）才 page。见 [Monitoring & Metrics](#monitoring--metrics) + [Issue 1: DLQ Messages](#issue-1-dlq-messages)。

**Manual Investigation**:

1. **确认是 Deepgram 侧还是我们侧**：

```
fields @timestamp, session, provider, jobName, error, errorCause.message
| filter @message like /Deepgram failed/ or @message like /Gemini rewrite failed/ or @message like /DEEPGRAM_SUSTAINED_FAILURE/
| sort @timestamp desc
| limit 50
```

2. **看 provider 分布**（Deepgram 成功率是否骤降 / Gemini 是否在大量兜底）：

```
fields @timestamp, provider
| filter @message like /Transcription job started successfully/
| stats count() as jobs by provider, bin(1h)
```

3. **确认 Deepgram secret 存在**（`DEEPGRAM_SECRET_NAME`，default `deepgram/api-key`）：

```bash
aws secretsmanager describe-secret --secret-id deepgram/api-key
```

**Recovery**:

- Deepgram 短暂 outage：无需动作，SQS 重投会自愈；消息进 DLQ 才走 [DLQ Recovery Procedure](#dlq-recovery-procedure)。
- Deepgram API key 失效 / secret 缺失：修好 secret 后 redrive DLQ。
- 持续失败告警：确认 Deepgram 服务状态（[https://status.deepgram.com），必要时临时切](https://status.deepgram.com），必要时临时切) `TRANSCRIPTION_AUDIO_STRATEGY=gemini-only`（需部署环境变量改动，走 Park & Notify）。

***

## Monitoring & Metrics

### Dashboard Metrics (Row 2 - TranscribeProcessor)

**Widget 1: Lambda Errors**

- **Metric**: `AWS/Lambda - Errors`
- **Dimension**: `FunctionName: TranscribeProcessorTS-prod`
- **Healthy**: \<1%
- **Warning**: 1-5%
- **Critical**: >5%

**Widget 2: Lambda Duration**

- **Metric**: `AWS/Lambda - Duration`
- **Dimension**: `FunctionName: TranscribeProcessorTS-prod`
- **Healthy**: \<60 seconds (API calls only)
- **Warning**: 60-120 seconds (large recordings)
- **Critical**: >120 seconds (approaching timeout)

**Widget 3: Lambda Invocations**

- **Metric**: `AWS/Lambda - Invocations`
- **Dimension**: `FunctionName: TranscribeProcessorTS-prod`
- **Pattern**: Spikes correlate with call volume

***

### Custom Metrics (CallAnalytics/RingCentral Namespace)

**Metric 1: Rate Limit Capacity %**

```typescript
// Code: lambda/transcribe-processor/src/utils/metrics.ts
metrics.addMetric('RateLimitCapacity', MetricUnit.Percent, capacityPercent);
```

- **Healthy**: >50%
- **Warning**: 20-50%
- **Critical**: \<20% (approaching rate limit)

**Metric 2: API Calls Per Minute**

```typescript
metrics.addMetric('ApiCallsPerMinute', MetricUnit.Count, callsPerMinute);
```

- **Healthy**: \<8/min per client
- **Warning**: 8-10/min
- **Critical**: >10/min (rate limit risk)

**Metric 3: Processing Duration**

```typescript
emitProcessingDurationMetrics(duration, success);
```

- Tracks successful vs failed processing time
- Success: Average \<60s
- Failed: Average >120s (timeouts)

***

### 告警语义（先读这个再看下面的 alarm）

日志级别 ≠ 告警级别。三者分开：

| 机制                                                          | 去哪                             | 会 page 人吗                           |
| ----------------------------------------------------------- | ------------------------------ | ----------------------------------- |
| `logger.warn`                                               | CloudWatch only                | ❌                                   |
| `logger.error`                                              | CloudWatch + Sentry            | ❌ **不发 Lark**                       |
| `logger.alert`                                              | CloudWatch + Sentry + **Lark** | ✅（只用于有 owner/action/runbook 的可执行事件） |
| CloudWatch Alarm → SNS → alarm-dispatcher Lambda → **Lark** | Lark                           | ✅ **PROD 页人的真正来源**                  |

**PROD 页人不靠 Lambda `logger.error`**，靠 CloudWatch alarm。TranscribeProcessor 相关的可执行 PROD 告警只有 **DLQ** 这一路。以 `lib/stacks/monitoring-stack.ts` 为准。

### CloudWatch Alarms（`lib/stacks/monitoring-stack.ts`）

| Alarm                             | Trigger           | Pages 人？ | 说明                                                                                                                                                |
| --------------------------------- | ----------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **TranscribeDLQAlarm**            | Call Log DLQ ≥1 条 | ✅ **是**  | 唯一直接 page 的 transcribe 告警。webhook 处理 3 次重试后仍失败才进 DLQ → SnsAction → alarm-dispatcher → Lark。Runbook：[Issue 1: DLQ Messages](#issue-1-dlq-messages) |
| **TranscribeProcessorErrorAlarm** | 5 分钟内 ≥5 errors   | ❌ 否      | PR #1459 起**去 action 化** —— 只进 Sentry + dashboard 供诊断，不 page（源码注释 "Diagnostic alarm ... no direct page"）。error 不代表客户受影响                           |

> ⚠️ 旧文档写的 `TranscribeProcessorErrorRate`（>10% error rate page P2）和 `TranscribeProcessorDuration`（≥14min page P3）在当前 stack 里**不存在或不 page**。Duration 只在 dashboard Row 3 可视化,没有独立 page 告警。

**收到告警怎么办** → 告警响应 step（DLQ / queue-age / ingestion 等）统一在 infra repo 的 [Pipeline Monitoring Runbook](https://github.com/retaintive/callytics-infrastructure/blob/main/docs/observability/runbook-pipeline-monitoring.md#dlq) 维护，这里不重复。

***

## CloudWatch Queries

### Q1: API Rate Limit Detection (429 Errors)

```
fields @timestamp, client_id, @message, error, retryAfterSeconds, totalDelaySeconds
| filter @message like /429/ or @message like /rate limit/
| stats count() as rate_limit_errors by client_id
| sort rate_limit_errors desc
```

**Purpose**: Identify clients hitting RingCentral rate limits

**Expected Output**:

```
client_id                                                     | rate_limit_errors
orangeTheory-2c9fc00886d14b9a9a24a12d337c438c                | 45
planetFitness-5d8e91c997f25eab1b35b23e448d549d               | 12
```

***

### Q2: 404 Not Found Analysis

```
fields @timestamp, telephonySessionId, elapsedMs, thresholdMs, @message
| filter @message like /404 Not Found/ or @message like /404 timeout/
| stats count() as errors_404, avg(elapsedMs) as avg_elapsed_ms by bin(1h)
| sort @timestamp desc
```

**Purpose**: Analyze 404 error patterns and elapsed times

**Expected Output**:

```
bin(1h)              | errors_404 | avg_elapsed_ms
2025-11-30 14:00:00  | 23         | 92000          ← Normal (< 2 min)
2025-11-30 13:00:00  | 2          | 915000         ← Timeout (> 15 min)
```

***

### Q3: SQS Processing Latency

```
fields @timestamp, messageId, approximateReceiveCount, @duration
| filter @message like /Starting SQS Record Transcription Processing/
| stats avg(@duration) as avg_duration_ms, max(@duration) as max_duration_ms by approximateReceiveCount
| sort approximateReceiveCount asc
```

**Purpose**: Correlate retry count with processing duration

**Expected Output**:

```
approximateReceiveCount | avg_duration_ms | max_duration_ms
1                       | 4500            | 8200            ← First attempt
2                       | 6200            | 12000           ← Retry 1
3                       | 9800            | 18000           ← Retry 2
```

***

### Q4: Duplicate Processing Detection

```
fields @timestamp, webhookUuid, telephonySessionId, @message
| filter @message like /already processed/ or @message like /processing incomplete/
| stats count() as duplicates by webhookUuid
| sort duplicates desc
```

**Purpose**: Identify duplicate webhooks from RingCentral

**Expected Output**:

```
webhookUuid                                | duplicates
b7f2e3c4-5d6a-4f8b-9c0e-1a2b3c4d5e6f      | 3
```

***

### Q5: Recording Download Performance

```
fields @timestamp, @duration, telephonySessionId, s3Uri
| filter @message like /Recording uploaded to S3/
| stats avg(@duration) as avg_duration_ms, max(@duration) as max_duration_ms by bin(1h)
| sort @timestamp desc
```

**Purpose**: Track recording download and S3 upload duration

**Expected Output**:

```
bin(1h)              | avg_duration_ms | max_duration_ms
2025-11-30 14:00:00  | 45000           | 87000           ← Normal
2025-11-30 13:00:00  | 92000           | 145000          ← Slow downloads
```

***

### Q6: Successful Jobs by Client

```
fields @timestamp, client_id, jobName, @message
| filter @message like /Transcription job started successfully/
| stats count() as successful_jobs by client_id
| sort successful_jobs desc
```

**Purpose**: Verify processing success rate per client

**Expected Output**:

```
client_id                                                     | successful_jobs
orangeTheory-2c9fc00886d14b9a9a24a12d337c438c                | 1247
planetFitness-5d8e91c997f25eab1b35b23e448d549d               | 892
```

***

### Q7: IAM Permission Errors

```
fields @timestamp, @message, error, resource
| filter error like /Access Denied/ or error like /AccessDeniedException/
| stats count() as denied_errors by resource
| sort denied_errors desc
```

**Purpose**: Identify missing IAM permissions

**Expected Output**:

```
resource                                                                          | denied_errors
arn:aws:s3:::call-analytics-recordings-prod/orangeTheory/.../4867428010.mp3      | 34
arn:aws:secretsmanager:us-east-1:ACCOUNT:secret:ringcentral-api-tokens-...       | 12
```

***

## Recovery Procedures

### DLQ Recovery Procedure

**Goal**: Redrive messages from DLQ back to main queue for retry

**Prerequisites**:

- Root cause identified and fixed (IAM permissions, secret created, etc.)
- Messages have not expired (14-day retention)

**Steps**:

1. **Verify DLQ message count**:

```bash
aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --attribute-names ApproximateNumberOfMessages
```

2. **Sample 1-2 messages to verify fixability**:

```bash
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --max-number-of-messages 2
```

3. **Parse message to extract client\_id and telephonySessionId**:

```json
{
  "Body": "{\"client_id\":\"orangeTheory-2c9fc00886d14b9a9a24a12d337c438c\",\"telephonySessionId\":\"baa82aa2e94c46f29b2b3be2adac14ea\"}"
}
```

4. **Verify fix** (test one message manually if possible):
   - IAM permissions added? → Check Lambda role in IAM Console
   - Secret created? → Check Secrets Manager
   - 404 timeout? → Check if call still exists in RingCentral

5. **Redrive all messages** (AWS Console or CLI):

**AWS Console**:

- Navigate to SQS > DLQ queue
- Click "Start DLQ redrive"
- Select destination: `transcribe-processor-queue-multi-tenant`
- Click "Redrive messages"

**AWS CLI**:

```bash
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:ACCOUNT:transcribe-processor-dlq-multi-tenant \
  --destination-arn arn:aws:sqs:us-east-1:ACCOUNT:transcribe-processor-queue-multi-tenant
```

6. **Monitor recovery** (Dashboard Row 1):

- DLQ message count should decrease to 0
- Main queue depth will spike temporarily (messages being reprocessed)
- Lambda invocations will increase (processing redriven messages)

7. **Verify success** (CloudWatch Logs):

```
fields @timestamp, telephonySessionId, @message
| filter @message like /Transcription job started successfully/
| stats count() as reprocessed_jobs by bin(5m)
| sort @timestamp desc
```

**Expected Timeline**:

- Small DLQ (\<10 messages): 5-10 minutes
- Medium DLQ (10-100 messages): 30-60 minutes (limited by concurrency = 2)
- Large DLQ (>100 messages): 2-4 hours

***

### Purge Invalid DLQ Messages

**When to Purge**:

- Malformed webhook payload (cannot be parsed)
- Call deleted from RingCentral (permanent 404)
- Duplicate messages (already processed successfully)

**Steps**:

1. **Backup DLQ messages** (for audit):

```bash
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant \
  --max-number-of-messages 10 > dlq-backup-$(date +%Y%m%d-%H%M%S).json
```

2. **Purge DLQ**:

```bash
aws sqs purge-queue \
  --queue-url https://sqs.us-east-1.amazonaws.com/ACCOUNT/transcribe-processor-dlq-multi-tenant
```

**WARNING**: Purge is permanent and cannot be undone!

***

## Related Documentation

- **[AI Analysis Processor Runbook](/operations/runbooks/ai-analysis.md)** - Transcription → AI 分析（DeepSeek V4 Flash via OpenRouter，非 Bedrock）
- **[Reconciliation Runbook](/operations/runbooks/reconciliation.md)** - Missing call recovery
- **[Pipeline Monitoring Runbook（infra repo）](https://github.com/retaintive/callytics-infrastructure/blob/main/docs/observability/runbook-pipeline-monitoring.md)** - 告警响应 step + 跨 4-Lambda trace（"收到告警怎么办" 的入口）
- **[On-Call Guide](/operations/on-call-guide.md)** - Emergency response procedures

***

**Last Updated**: 2026-07-14（对照 infra PR #1459）
**Version**: 2.1（Deepgram/Gemini 同步转录 + Dual-table DynamoDB）
**Owner**: DevOps Team
**Next Review**: 2026-10-14 (Quarterly)