Keyboard and captions tracks

[richiemcilroy]

Adds keyboard press recording and display, and migrates captions and keyboard presses to interactive timeline tracks for improved editing and visualization.

---

 

Greptile Summary

This PR introduces keyboard press recording and display, and migrates both captions and keyboard presses to interactive timeline tracks for improved editing and visualization.

Major Changes:

• Keyboard recording during screen capture using device_query crate, capturing key down/up events with timestamps
• New crates/project/src/keyboard.rs module implementing intelligent grouping of key events (handles rapid typing, modifiers, special keys, backspace, and command combinations)
• New WGPU-based keyboard rendering layer (crates/rendering/src/layers/keyboard.rs) with fade animations and bounce effects
• Timeline-based caption and keyboard tracks in the editor UI, enabling visual segment editing (drag, resize, split)
• New KeyboardTab UI for configuring keyboard display settings (font, size, colors, animation timing, grouping behavior)
• Migration of captions from legacy structure to timeline segments for consistent editing experience
• TypeScript and Rust type definitions aligned across the stack (KeyboardSettings, KeyboardData, KeyboardTrackSegment)

Architecture:

• Recording: cursor.rs polls keyboard state via device_query, writes keyboard.json per segment
• Processing: generate_keyboard_segments command aggregates events and groups them using configurable thresholds
• Rendering: KeyboardLayer uses glyphon for text rendering with background boxes, positioned relative to captions
• UI: Timeline tracks provide interactive segment manipulation; settings tabs control appearance and behavior

Confidence Score: 5/5

• This PR is safe to merge with no critical issues found
• The implementation is well-architected with proper separation of concerns (recording, processing, rendering, UI). The Rust code follows workspace lint rules, includes comprehensive tests for the grouping logic, and handles edge cases properly. The TypeScript changes integrate cleanly with existing patterns. Type definitions are aligned between Rust and TypeScript. No security issues, memory leaks, or logic errors detected.
• No files require special attention

Important Files Changed

Filename	Overview
crates/project/src/keyboard.rs	New keyboard event processing module with grouping logic and segment generation; includes comprehensive tests
crates/rendering/src/layers/keyboard.rs	New rendering layer for keyboard segments using WGPU and glyphon. Implements fade animations, bounce effects, and proper text rendering with background. Well-structured.
apps/desktop/src/routes/editor/KeyboardTab.tsx	New UI tab for keyboard settings with sliders, toggles, and segment generation. Follows established patterns and integrates properly with editor context.
apps/desktop/src/routes/editor/Timeline/KeyboardTrack.tsx	New timeline track component for keyboard segments. Implements drag-to-move, resize handles, and split functionality. Consistent with existing track patterns.
apps/desktop/src/routes/editor/Timeline/CaptionsTrack.tsx	New timeline track for captions, matching keyboard track pattern. Enables visual editing of caption timing on timeline. Solid implementation.
crates/recording/src/cursor.rs	Extended to record keyboard events using device_query. Adds comprehensive keycode mapping and keyboard event flushing. Integrates well with existing cursor recording.
apps/desktop/src/routes/editor/context.ts	Extended with caption and keyboard track types, segment definitions, and timeline actions. Adds split/delete operations for new segment types. Properly normalized.
crates/project/src/configuration.rs	Adds KeyboardData, KeyboardSettings, CaptionTrackSegment, and keyboard_segments to timeline. Default values are sensible. Schema evolution handled cleanly.
apps/desktop/src-tauri/src/lib.rs	Adds generate_keyboard_segments command to aggregate and group keyboard events from recording segments. Proper error handling and sorting.
crates/rendering/src/layers/captions.rs	Refactored to support timeline-based caption segments. Uses timeline data when available. Some complexity in dual-mode support.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Recording Start] --> B[cursor.rs: Poll keyboard via device_query]
    B --> C[Track key down/up events with timestamps]
    C --> D[Flush to keyboard.json per segment]
    D --> E[Recording Stop]
    
    E --> F[Editor: Load keyboard.json from segments]
    F --> G[generate_keyboard_segments command]
    G --> H[Group events by threshold]
    H --> I[Apply modifier/special key logic]
    I --> J[Create KeyboardTrackSegments]
    
    J --> K[Timeline: Display as interactive track]
    K --> L[User edits: drag, resize, split]
    
    J --> M[Render Pipeline]
    M --> N[KeyboardLayer.prepare]
    N --> O[Find active segment for current time]
    O --> P[Calculate fade and bounce]
    P --> Q[Render text with glyphon]
    Q --> R[Render background box with WGPU]
    R --> S[Final video frame]

Loading

_{Last reviewed commit: 572802f}

[cursor]

Cursor Agent can help with this pull request. Just @cursor in comments and I'll start working on changes in this branch.
_{Learn more about Cursor Agents}