Background video OCR
Frames are extracted on-device at the configured fps, uploaded to S3 via presigned URLs, OCR’d in parallel by λ₄, aggregated by λ₅, polished by λ₆ (Gemini) — then λ₈ orchestrates completion, λ₉ publishes via SNS→APNS, and the iOS app persists the polished text.
Cloud pipeline
Click any node for URL, handler file, IAM notes, and iOS call site. Every AWS service touched at every step is rendered — Lambda URLs, S3 buckets, Rekognition, DynamoDB, SNS → APNS — plus the external Gemini API.
Prompts used
frame_ocr
gemini-2.5-flash qr_reader_v1/frame_ocr_lambda.py:31 · temp 0 sha256 66326cc5be6b…
Perform OCR on this image and return plain text only. Do not describe the image.
video_polish
gemini-2.5-flash qr_reader_v1/video_ocr_polishing_lambda_rest.py:102 · temp 0.0
Please stitch together the OCRs that are taken from individual screenshots/frames from a video. There should be overlapping lines which can be used as a marker for when one frame ends and another begins. Please do not alter the content in any way besides stitching the content together and please add correct indentation. Do not alter or add any content.
Your final output must be a single, valid JSON object and nothing else. The JSON object must conform to the following structure:
{
"stitched_response": "(string) This key must hold the final, fully reconstructed and cleaned document text.",
"additional_notes": "(string) Use this field to briefly describe your process. Mention any significant noise you filtered out (e.g., 'Removed text from a Save As dialog box') or any ambiguities you encountered during the reconstruction."
}
Here is the raw OCR text from video frames:
{raw_text} Model mix
last 30 runs · 0 post-F1
gemini-2.5-flash 30 · 100%
Model attribution comes from the Lambda's self-reported modelId (post-F1). Older runs fall back to runSettings.geminiModel from app_defaults.yaml.
Recent runs for this flow
Polish
additional_notes Gemini's self-reported edits during the polish stage. Tag color = concern tier: pink = fidelity (silent source edits), yellow = determinism (variant picks between runs), green = neutral (stylistic cleanup). Tag rules live in scripts/polish_notes_categorize.py.
▸ 2026-04-23T12-27-49Z__iter_03 2026-04-23 12:27 565 chars Regex variant selectionTypo fix (name swap)
Would have corrected `vscode.ViewColumn.one` to `vscode.ViewColumn.One` (Frame 3 vs Frame 1/4). Would have corrected `nickBaseFilenameNoExtension` to `pickBaseFilenameNoExtension` (Frame 3 vs Frame 4). Would have corrected `panel.webview.onDidChangeMessage` to `panel.webview.onDidReceiveMessage` (Frame 12 vs Frame 13). Would have corrected `getWebViewContent` to `getWebviewContent` (Frame 17/18 vs Frame 19). Would have corrected the regex `lastPart.replace(/["^]*$/, "")` in Frame 21 to `lastPart.replace(/\.[^.]*$/, "")` as the former was clearly OCR-degraded.
▸ 2026-04-23T12-24-30Z__iter_03 2026-04-23 12:24 310 chars Regex variant selection
Would have added a semicolon to `console.error("Failed to init session with server:", err)`.
Would have corrected the regex `lastPart.replace(/["^]*$/, "")` to `lastPart.replace(/\.[^.]*$/, "")` or `lastPart.replace(/\.[^/.]*$/, "")`.
Would have corrected `return` on its own line to `return `<!DOCTYPE html>`. ▸ 2026-04-23T12-18-46Z__iter_02 2026-04-23 12:18 5076 chars Indentation normalizationRegex variant selectionTypo fix (name swap)
1. **Frame 2:** The initial `// src/extension.ts` was extracted. The rest of the frame was heavily concatenated and OCR-degraded, so its overlapping content was ignored in favor of Frame 1's clear formatting. I would have corrected the formatting if not in strict mode.
2. **Frame 3 vs Frame 1:** `vscode.ViewColumn.one` (Frame 3) was preferred over `vscode.ViewColumn.One` (Frame 1) as Frame 3 was chronologically later. I would have corrected to `vscode.ViewColumn.One` in normal mode for standard casing.
3. **Frame 4 vs Stitched:**
* `vscode.ViewColumn.One` (Frame 4) was preferred over `vscode.ViewColumn.one` (stitched from Frame 3). This reverted the previous change. I would have corrected to `vscode.ViewColumn.One` in normal mode for standard casing.
* `console.error("Failed to init session with server:", err)` (Frame 4, no semicolon) was preferred over `console.error("Failed to init session with server:", err);` (stitched). I would have added a semicolon in normal mode.
* `pickBaseFilenameNoExtension` (Frame 4) was preferred over `nickBaseFilenameNoExtension` (stitched from Frame 1). I would have corrected `nickBase` to `pickBase` if it was a typo, but here `pickBase` is the chronologically later version, so it was preferred as-is.
4. **Frame 5 vs Stitched:** Frame 5's `vscode.window.createWebviewPanel(` was clearly truncated, so the stitched version was preferred for that line. I would have corrected the truncation if not in strict mode.
5. **Frame 6 vs Stitched:** The `catch` block format `}).catch((err) => console.error("Failed to init session with server:", err));` (Frame 6) was preferred over the braced version from previous frames. I would have formatted with braces in normal mode for consistency.
6. **Frame 7 vs Stitched:** The `catch` block format `}).catch((err) => { console.error("Failed to init session with server:", err) });` (Frame 7) was preferred over the single-line version from Frame 6. I would have added a semicolon after `err` in normal mode.
7. **Frame 11 vs Stitched:** Frame 11's `panel.webview.onDidReceiveMessage` block was structurally degraded (missing `if` and `try` blocks), so its overlapping content was ignored, and only the closing `catch` block and final `}); }` were appended. I would have corrected the structural degradation if not in strict mode.
8. **Frame 12 vs Stitched:** Frame 12's `panel.webview.onDidChangeMessage` was clearly OCR-degraded (missing `if` and `try` blocks), so it was not used to replace the `onDidReceiveMessage` block. I would have corrected `onDidChangeMessage` to `onDidReceiveMessage` in normal mode.
9. **Frame 13 vs Stitched:** Frame 13's `panel.webview.onDidReceiveMessage` block was structurally degraded, so its overlapping content was ignored. I would have corrected the structural degradation if not in strict mode.
10. **Frame 14 vs Stitched:**
* Frame 14's `await editor.edit((editBuilder) => {});` was clearly truncated/degraded, so the stitched version was preferred. I would have corrected the empty `editBuilder` in normal mode.
* The multiline `throw new Error(...)` from Frame 14 was preferred over the single-line version from Frame 13.
11. **Frame 15 vs Stitched:** Frame 15's `panel.webview.onDidReceiveMessage` block was severely degraded, so its overlapping content was ignored. The single-line `throw new Error(...)` from Frame 15 was not preferred over the stitched multiline version as it was less robust.
12. **Frame 16 vs Stitched:**
* The JSDoc for `pickBaseFilenameNoExtension` in Frame 16 included an extra `-> "22_generate_parenthesis.py"`, which was preferred.
* The regex `/\[^.]*$/` (Frame 16) was preferred over `/\[^/\.]*$/` (stitched from Frame 15).
13. **Frame 17 vs Stitched:** `getWebViewContent` (Frame 17) was preferred over `getWebviewContent` (stitched). This introduced an inconsistency with the call site `panel.webview.html = getWebviewContent(sessionId);`. I would have corrected `getWebViewContent` to `getWebviewContent` in normal mode for consistency.
14. **Frame 18 vs Stitched:** `getWebviewContent` (Frame 18) was preferred over `getWebViewContent` (stitched from Frame 17), correcting the inconsistency.
15. **Frame 19 vs Stitched:** Frame 19 was heavily degraded (lost indentation, line breaks), so its overlapping content was ignored in favor of the stitched version's formatting. I would have corrected the formatting if not in strict mode.
16. **Frame 21 vs Stitched:** The `return` statement for `getWebviewContent` in Frame 21 had an extra indentation, which was preferred.
17. **Frame 22 vs Stitched:** Indentation differences in the HTML/script block were resolved by preferring Frame 22's version.
18. **Frame 23 vs Stitched:** Indentation differences in the HTML/script block were resolved by preferring Frame 23's version.
19. **Frame 24 vs Stitched:** Frame 24 was heavily degraded (lost indentation, line breaks), so its overlapping content was ignored in favor of the stitched version's formatting. I would have corrected the formatting if not in strict mode. ▸ 2026-04-23T12-17-13Z__iter_02 2026-04-23 12:17 40 chars uncategorized
Would have corrected 'gooddd' to 'good'.
▸ 2026-04-23T12-13-13Z__iter_02 2026-04-23 12:13 907 chars Regex variant selectionTypo fix (name swap)
Would have corrected `console.error("Failed to init session with server:", err)` to include a semicolon in frames 4, 7, and 8. Would have corrected the truncated `vscode.window.createWebviewPanel` call in frame 5. Would have corrected `panel.webview.onDidChangeMessage` to `panel.webview.onDidReceiveMessage` in frame 11. Would have corrected the empty `editBuilder` in `await editor.edit((editBuilder) => {});` in frame 13. Would have corrected the regex `lastPart.replace(/\\.[^.\/]*$/, "")` in frame 19 to `lastPart.replace(/\.[^.]*$/, "")` (removing the extra backslash and `/` from the character class, preferring the latest non-degraded version from frame 18). Would have corrected `function getWebviewContent(sessionId: string): string [` to `function getWebviewContent(sessionId: string): string {` in frame 19. Would have corrected `return <!DOCTYPE html>` to `return `<!DOCTYPE html>` in frame 22. ▸ 2026-04-23T12-11-49Z__iter_01 2026-04-23 12:11 133 chars uncategorized
Would have added a space between "Kiley" and "Add" in the line "Well Made by KileyAdd the dry ingredients to the wet." from Frame 12.
▸ 2026-04-23T12-09-16Z__iter_01 2026-04-23 12:09 622 chars Regex variant selectionTypo fix (name swap)
Would have corrected `vscode.ViewColumn.one` (Frame 3) to `vscode.ViewColumn.One`. Would have corrected `console.error("Failed to init session with server:", err)` (Frame 4, 5, 8) to include a semicolon. Would have corrected `nickBaseFilenameNoExtension` (Frame 1, 2, 3) to `pickBaseFilenameNoExtension` (Frame 4 onwards). Would have corrected `panel.webview.onDidChangeMessage` (Frame 11) to `panel.webview.onDidReceiveMessage` (Frame 12 onwards). Would have corrected `getWebViewContent` (Frame 16, 17) to `getWebviewContent` (Frame 18 onwards). Would have corrected the degraded regex `["^]*$` (Frame 20) to `\.[^.]*$`. ▸ 2026-04-23T12-08-04Z__iter_01 2026-04-23 12:08 477 chars uncategorized
The punctuation in "(~1 1/2 cups;" (Frames 1, 2, 3) was updated to "(~1 1/2 cups," (Frame 4) as Frame 4 was chronologically later and not degraded. Frame 12 showed significant OCR degradation in formatting (concatenated lines, missing line breaks); the content from this section was instead taken from Frame 13, which presented the same content with correct formatting. UI elements and truncation markers (e.g., "less") were excluded as they are not part of the recipe content.
▸ 2026-04-23T12-04-16Z__iter_01 2026-04-23 12:04 420 chars Regex variant selectionTypo fix (name swap)
Would have corrected the regex `/["^]*$/` in `pickBaseFilenameNoExtension` (from Frame 17) to a more robust pattern like `/\.[^/.]*$/` if not in strict mode. Would have corrected the typo `untiledFile` to `untitledFile` (from Frame 17) if not in strict mode. Would have corrected `string [` to `string {` in `getWebviewContent` function signature (from Frame 16) if not in strict mode, as `[` appears to be an OCR error. ▸ 2026-04-23T12-00-24Z__iter_03 2026-04-23 12:00 192 chars uncategorized
Removed UI elements from frames 13 and 15. Corrected a semicolon to a comma in the banana measurement based on a later frame. Completed truncated lines by preferring content from later frames.
▸ 2026-04-23T11-57-48Z__iter_03 2026-04-23 11:57 560 chars Indentation normalizationRegex variant selectionTypo fix (name swap)
Stitched multiple overlapping frames. Handled conflicting tokens by preferring the version from the chronologically later frame (e.g., `vscode.ViewColumn.One` over `one`, `pickBaseFilenameNoExtension` over `nickBaseFilenameNoExtension`, specific regex variants, `getWebviewContent` over `getWebViewContent`, and the full `vscode.postMessage` with `s3Url`). Corrected indentation based on the latest frame's style. Ignored clearly OCR-degraded frames or lines (e.g., Frame 2, Frame 12's `onDidChangeMessage`, Frame 19/20's zero-indentation and malformed regex).
▸ 2026-04-23T11-55-21Z__iter_03 2026-04-23 11:55 267 chars Indentation normalization
Stitched content from multiple frames, removing transient UI elements like status bars and footers. Corrected minor OCR variations (e.g., 'less' to 'Mill', semicolon to comma in ingredient list, and adjusted line breaks for better readability where OCR was degraded).
▸ 2026-04-23T11-52-32Z__iter_03 2026-04-23 11:52 951 chars Indentation normalizationRegex variant selectionScope / nesting correctionTypo fix (name swap)
Stitched content from multiple frames, preferring the latest non-degraded version for overlapping lines. Corrected the structural placement of the `panel.dispose()` statement and its associated `catch` block within the `onDidReceiveMessage` handler, as several frames showed them at incorrect indentation levels or outside their logical scope. Handled OCR-degraded frames (e.g., Frame 2, 7, 19, 21) by preferring more robust versions or previous frames' content. Resolved conflicting versions of `vscode.ViewColumn` (preferring `One`), `nickBaseFilenameNoExtension` vs `pickBaseFilenameNoExtension` (preferring `pickBaseFilenameNoExtension`), and `getWebViewContent` vs `getWebviewContent` (preferring `getWebviewContent`). Updated the regex for `pickBaseFilenameNoExtension` to `/\[^.]*$/` based on the latest non-degraded frame. Added implied closing backtick for the HTML template literal and the closing brace for the `getWebviewContent` function.
▸ 2026-04-23T11-51-10Z__iter_02 2026-04-23 11:51 330 chars uncategorized
Stitched multiple overlapping frames, de-duplicated lines, and preferred the latest frame's version for minor differences (e.g., semicolon vs. comma in banana measurement). Removed transient UI elements like '10:36 LIVE STEM Explore Local Following Shop For You' and '+ 1310%AT&T Wi-FiNot PlayingFocus' from the stitched response.
▸ 2026-04-23T11-47-24Z__iter_02 2026-04-23 11:47 597 chars Indentation normalizationRegex variant selectionTypo fix (name swap)
Stitched content by identifying overlapping lines and preferring the latest non-degraded version. Corrected `vscode.ViewColumn.one` to `vscode.ViewColumn.One`, `nickBaseFilenameNoExtension` to `pickBaseFilenameNoExtension`, and updated the regex in `pickBaseFilenameNoExtension`. Ignored OCR degradations such as `onDidChangeMessage` (kept `onDidReceiveMessage`), `string [` (kept `string {`), `untiledFile` (kept `untitledFile`), truncated code blocks, and incorrect regex patterns. Applied consistent indentation and added missing closing backtick and brace for the `getWebviewContent` function. ▸ 2026-04-23T11-46-02Z__iter_02 2026-04-23 11:46 250 chars uncategorized
Corrected 'less' to 'Mill' for @Bob's Red flour. Corrected 'cups;' to 'cups,'. Restored line breaks and formatting for a section of instructions where OCR was degraded in one frame. Removed UI elements and timestamps from the final stitched response.
▸ 2026-04-23T11-42-14Z__iter_02 2026-04-23 11:42 718 chars Redundant-brace filteringRegex variant selectionTypo fix (name swap)
Stitched content by incrementally adding new lines and resolving conflicts based on the 'prefer latest unless degraded' rule. Key changes included updating `vscode.ViewColumn.One` to `vscode.ViewColumn.one`, `nickBaseFilenameNoExtension` to `pickBaseFilenameNoExtension`, refining the `pickBaseFilenameNoExtension` regex and JSDoc, and updating the `getWebviewContent` function name case. Degraded OCR frames (e.g., Frame 2, 11, 19, 21) were ignored for overlapping content, preserving the more robust versions from earlier frames. The `catch` block format for fetch calls was updated to the latest non-degraded version. HTML string backticks and closing braces were carefully managed to maintain structural integrity.
▸ 2026-04-23T11-40-51Z__iter_01 2026-04-23 11:40 313 chars uncategorized
Stitched multiple overlapping frames, de-duplicated lines, and preserved formatting. Preferred the latest frame's version for minor punctuation differences (e.g., changed ';' to ',' in banana measurement and '.' to ',' in baking instructions). Removed transient UI elements from the top and bottom of some frames.
▸ 2026-04-23T11-38-14Z__iter_01 2026-04-23 11:38 722 chars Regex variant selectionTypo fix (name swap)
Stitched content by de-duplicating lines and preferring the latest frame's version for overlapping content, including minor formatting differences (e.g., `vscode.ViewColumn.One` vs `one`, `catch` block formatting, `throw new Error` formatting). Handled structural degradation in some frames by reconstructing the logical flow and inserting content from later frames into the correct structural position (e.g., `panel.webview.onDidReceiveMessage` block, `getWebviewContent` return statement). Corrected `nickBaseFilenameNoExtension` to `pickBaseFilenameNoExtension` and `getWebViewContent` to `getWebviewContent` based on later, more robust OCR results. Used the latest non-degraded regex for `pickBaseFilenameNoExtension`.
▸ 2026-04-23T11-36-53Z__iter_01 2026-04-23 11:36 435 chars uncategorized
Removed UI elements like timestamps, navigation bars, and 'less' indicators. Corrected a minor OCR artifact where 'Well Made by Kiley' was concatenated with the following line in one frame, by referring to earlier frames for consistent line breaks. Used the latest frame's content to complete truncated lines and add missing details (e.g., 'Mill' for flour, closing parenthesis for chocolate chips, full baking instructions, hashtags).
▸ 2026-04-23T11-35-04Z__iter_01 2026-04-23 11:35 451 chars Redundant-brace filteringIndentation normalizationtry/catch structure inferenceTypo fix (name swap)
Stitched multiple frames, resolving conflicts by preferring the latest frame's content unless clearly truncated or OCR-degraded. Handled inconsistent indentation by adopting the latest valid style. Reconstructed the `onDidReceiveMessage`'s `try...catch` block, as several frames showed it syntactically degraded or misplaced. Corrected minor OCR errors like `onDidChangeMessage` to `onDidReceiveMessage` and `getWebViewContent` to `getWebviewContent`.
▸ 2026-04-23T11-25-03Z__iter_03 2026-04-23 11:25 101 chars uncategorized
Removed transient UI elements (timestamps, navigation bars, Wi-Fi status) from the stitched response.
▸ 2026-04-23T11-22-15Z__iter_03 2026-04-23 11:22 616 chars Indentation normalizationRegex variant selectionTypo fix (name swap)
Preferred the latest frame's version for conflicting lines, except when clearly truncated or OCR-degraded. Specifically, reverted `panel.webview.createWebviewPanel` (Frame 5), `editBuilder.insert` (Frame 13), `getWebviewContent` function signature (Frame 19), and `getWebviewContent` return statement (Frame 22) due to truncation. Reverted `panel.webview.onDidChangeMessage` (Frame 11) and `lastPart.replace(/\\.[^.\/]*$/, "")` (Frame 19) to previous versions due to clear OCR degradation (typo and incorrect regex escaping, respectively). Indentation was preserved from the latest non-degraded frame for each block.
▸ 2026-04-23T11-20-55Z__iter_03 2026-04-23 11:20 237 chars uncategorized
Stitched content by de-duplicating overlapping lines and preferring the chronologically latest frame's version for minor OCR differences (e.g., semicolon vs. comma, missing parenthesis). Removed UI elements and 'less' truncation markers.
▸ 2026-04-23T11-17-41Z__iter_03 2026-04-23 11:17 762 chars Regex variant selectionTypo fix (name swap)
Stitched multiple frames, preferring the latest version for overlapping content. Ignored heavily OCR-degraded or structurally problematic frames (e.g., Frame 2, 7, 11, 12, 13, 14, 15, 16, 21, 26) for their content, but extracted new lines or formatting if not degraded. Specifically, updated `vscode.ViewColumn.One` (Frame 4), `catch` block formatting (Frame 6, 8), `pickBaseFilenameNoExtension` function name (Frame 4), `throw new Error` formatting (Frame 14, 15), `pickBaseFilenameNoExtension` regex (Frame 16, 18, 19, 21 - preferring the last non-degraded `/\[^.]*$/` from Frame 19), `getWebviewContent` function name (Frame 17, 19 - preferring the last `getWebviewContent` from Frame 19), and the `return` statement formatting for the HTML string (Frame 23).
▸ 2026-04-23T11-16-19Z__iter_02 2026-04-23 11:16 82 chars uncategorized
Removed transient UI elements and truncated text markers from the raw OCR results.
▸ 2026-04-23T11-13-52Z__iter_02 2026-04-23 11:13 780 chars Redundant-brace filteringRegex variant selectiontry/catch structure inferenceTypo fix (name swap)
Stitched multiple overlapping frames. Handled OCR degradation in Frame 2 and Frame 20 (incorrect regex, typos). Resolved inconsistencies in casing (e.g., `vscode.ViewColumn.One` vs `one`, `getWebviewContent` vs `getWebViewContent`), function names (`nickBaseFilenameNoExtension` vs `pickBaseFilenameNoExtension`), regex patterns (`/\.[^/\.]*$/` vs `/\.[^.]*$/`), and the presence of semicolons. Reconstructed the `try...catch` block and `panel.dispose()` placement within the `onDidReceiveMessage` callback for logical correctness, as several frames showed misplaced or truncated versions. Ensured the `vscode.postMessage` included `s3Url` and the `checkForScan` function's `else` and `catch` blocks included `setTimeout` and status updates, preferring the most complete versions.
▸ 2026-04-23T11-12-30Z__iter_02 2026-04-23 11:12 118 chars uncategorized
Removed transient UI elements such as timestamps, navigation bars, and network information from the stitched response.
▸ 2026-04-23T11-10-08Z__iter_02 2026-04-23 11:10 498 chars Regex variant selectionTypo fix (name swap)
Stitched multiple frames, resolving minor variations in casing (e.g., `ViewColumn.One` vs `ViewColumn.one`), function names (`nickBaseFilenameNoExtension` vs `pickBaseFilenameNoExtension`), and `catch` block formatting by preferring the chronologically latest non-degraded version. Corrected OCR errors such as `onDidChangeMessage` to `onDidReceiveMessage` and malformed regex in `pickBaseFilenameNoExtension`. Incorporated additional details like `s3Url` in `vscode.postMessage` from later frames.
▸ 2026-04-23T11-08-46Z__iter_01 2026-04-23 11:08 245 chars uncategorized
Stitched content from multiple frames, removing UI elements like timestamps, navigation bars, and network indicators. Minor OCR variations (e.g., semicolon vs. comma) were resolved by preferring the version from the chronologically latest frame.
▸ 2026-04-23T11-06-19Z__iter_01 2026-04-23 11:06 718 chars Regex variant selectionTypo fix (name swap)
Resolved conflicts by preferring chronologically later frames unless they were clearly truncated or OCR-degraded. Specific resolutions include: preferring `vscode.ViewColumn.One` (correct casing) over `vscode.ViewColumn.one` (OCR degradation); preferring `pickBaseFilenameNoExtension` over `nickBaseFilenameNoExtension` from a later frame; preferring `onDidReceiveMessage` (correct API) over `onDidChangeMessage`; preferring the more complete and syntactically correct `catch` block structures; and preferring the regex `/\[^.]*$/` for filename extraction as it was robust and consistently well-formed. Heavily degraded frames were largely ignored for conflicting content, only contributing unique, non-degraded lines.
▸ 2026-04-23T11-04-51Z__iter_01 2026-04-23 11:04 197 chars uncategorized
Stitched multiple overlapping frames, de-duplicated lines, and preferred the latest frame's version for minor text variations (e.g., 'cups;' vs 'cups,'). Ignored UI elements present in some frames.
▸ 2026-04-23T11-00-53Z__iter_01 2026-04-23 11:00 456 chars Redundant-brace filteringRegex variant selectionTypo fix (name swap)
Stitched multiple overlapping OCR frames. Handled OCR degradations (e.g., single-line frames, typos like 'onDidChangeMessage', 'untiledFile', 'getWebViewContent', truncated code, incorrect closing braces, degraded regex) by preferring the latest non-degraded version. Resolved conflicting casing for `vscode.ViewColumn.One` by preferring the latest frame's version. Corrected the closing backtick and semicolon for the `getWebviewContent` return statement.