AI builder journey now tracked with telemetry events
A full telemetry system has been added to capture every step of the AI builder experience—from user messages through workflow creation, execution, and completion. Developers can now analyze exactly how users interact with the AI builder.
The AI builder now has eyes everywhere. A comprehensive telemetry system has been added to track the entire user journey through the instance AI feature, giving developers visibility into how users interact with the builder from first message to published workflow.
The system now tracks both user actions and AI builder responses at every stage. On the backend, events fire when the builder sends messages, creates or modifies workflows, executes them, and publishes results. On the frontend, user actions are captured—when someone sends a message to the builder, completes a setup wizard, answers questions, or approves a plan.
The telemetry uses a consistent linking key—thread_id—so events can be correlated to reconstruct complete conversations. An additional inputThreadId connects related events within confirmation sessions, such as all the steps of a setup wizard or a multi-question form.
Type safety improvements were made alongside the telemetry work. The InstanceAiConfirmation type was extracted from an inline definition and made reusable. The PendingConfirmationItem type now guarantees that confirmation objects exist, eliminating scattered non-null assertions throughout the codebase.
The changes span both the CLI backend and the editor-ui frontend, touching confirmation panels, setup wizards, and store logic across the AI builder feature.
View Original GitHub DescriptionFact Check
Summary
- Add comprehensive telemetry events to track the full user journey through the instance AI builder, using
thread_idas the linking key - Backend events: Builder sent message, created/modified/executed/published workflow, asked for input, satisfied user intent
- Frontend events: User sent builder message, User finished providing input (with provided_inputs/skipped_inputs per step)
- Extract
InstanceAiConfirmationas a named type, strengthenPendingConfirmationItemto guaranteeconfirmationis present (removes!assertions) - Add
inputThreadIdto confirmation payloads to link input-related events - Includes temporary
console.debug/logger.debuglogging for testing (to be removed before merge)
Test plan
- Send a message in instance AI → verify "User sent builder message" fires with correct
thread_id,is_first_message - Wait for builder response → verify "Builder sent message" and "Builder satisfied user intent" fire
- Trigger workflow creation → verify "Builder created workflow" fires with
workflow_id - Trigger workflow modification → verify "Builder modified workflow" fires
- Trigger workflow execution → verify "Builder executed workflow" fires with correct
exec_type - Trigger workflow publish → verify "Builder published workflow" fires
- Trigger setup wizard (credentials) → verify "Builder asked for input" fires, then "User finished providing input" fires on Continue/Later with correct
provided_inputs/skipped_inputs - Trigger approval confirmation → verify "User finished providing input" fires with approve/deny choice
- Trigger questions confirmation → verify "User finished providing input" fires with all answers
- Check browser console for
[Telemetry]debug logs on all frontend events - Check backend logs for
[Telemetry]debug entries on all backend events
🤖 Generated with Claude Code