Conversation
There was a problem hiding this comment.
Pull request overview
This PR extends the chat Agent Debug Logs pipeline to support a new resolved debug-event detail kind that summarizes customization (instructions/skills/agents/hooks) resolution, including duration and per-item resolution logs, and wires it through both the debug detail UI and the “resolve debug event details” builtin tool.
Changes:
- Adds a new resolved debug content kind (
customizationSummary) and rendering/formatting support (UI + tool output). - Collects structured per-item resolution logs + duration during automatic instruction collection and exposes the latest collection event via
IPromptsService. - Logs and resolves customization summary debug events from
PromptsDebugContribution.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/vs/workbench/contrib/chat/common/tools/builtinTools/resolveDebugEventDetailsTool.ts | Adds plain-text formatting for resolved customizationSummary debug event content. |
| src/vs/workbench/contrib/chat/common/promptSyntax/service/promptsServiceImpl.ts | Stores the latest instructions collection event on the prompts service. |
| src/vs/workbench/contrib/chat/common/promptSyntax/service/promptsService.ts | Introduces InstructionsCollectionEvent debug fields and exposes lastInstructionsCollectionEvent on IPromptsService. |
| src/vs/workbench/contrib/chat/common/promptSyntax/computeAutomaticInstructions.ts | Captures duration + per-item debug details during collection; omits debug-only fields from telemetry payload. |
| src/vs/workbench/contrib/chat/common/chatDebugService.ts | Extends the resolved debug content union with customizationSummary and related types. |
| src/vs/workbench/contrib/chat/common/chatDebugEvents.ts | Documents the new generic debug-event category/content for customization resolution. |
| src/vs/workbench/contrib/chat/browser/promptsDebugContribution.ts | Logs customization summary events and provides resolve support; adds session disposal cleanup. |
| src/vs/workbench/contrib/chat/browser/chatDebug/chatDebugDetailPanel.ts | Renders customizationSummary in the detail panel and supports copy/open-in-editor text. |
| src/vs/workbench/contrib/chat/browser/chatDebug/chatCustomizationDiscoveryRenderer.ts | Adds rich rendering + plain-text serialization for customization summary content. |
|
|
||
| // Log resolved customizations from the last instructions collection. | ||
| const collectionEvent = this.promptsService.lastInstructionsCollectionEvent; | ||
| if (!isFirstInvocation && collectionEvent) { |
There was a problem hiding this comment.
lastInstructionsCollectionEvent is global state on IPromptsService and isn’t scoped to the session/request. In multi-session or concurrent ComputeAutomaticInstructions.collect() scenarios (e.g. subagent tool + regular chat), this can be overwritten between the collection and onWillInvokeAgent, causing the logged customization summary to reflect the wrong request/session. Consider storing the collection event keyed by sessionResource (or passing it through the request pipeline) instead of a single “last” value; also, the !isFirstInvocation gate means the first request in a session never logs a customization summary even when collection info exists.
| if (!isFirstInvocation && collectionEvent) { | |
| if (collectionEvent) { |
| let resolvedHooks: ChatRequestHooks | undefined; | ||
| try { | ||
| const hooksInfo = await this.promptsService.getHooks(new CancellationTokenSource().token); | ||
| resolvedHooks = hooksInfo?.hooks; |
There was a problem hiding this comment.
A CancellationTokenSource is created inline for getHooks(...) but never disposed. Even though this is “just” for the token, the CTS still holds resources and should be disposed (or avoid creating one and use CancellationToken.None).
| localize('customizationsResolved', 'Resolve Customizations'), | ||
| details, | ||
| undefined, | ||
| { id: customizationEventId, category: 'customization' }, |
There was a problem hiding this comment.
This event is logged with category 'customization', but the Agent Debug Logs “Chat Customization” filter currently only toggles generic events with category 'discovery' (see chatDebugFilters.ts logic). As-is, these customization events will remain visible even when the toggle is off. Either log under the existing 'discovery' category or update the filter logic to include 'customization' as well.
| { id: customizationEventId, category: 'customization' }, | |
| { id: customizationEventId, category: 'discovery' }, |
| return { | ||
| kind: 'customizationSummary', | ||
| resolutionLogs: logs, | ||
| durationInMillis: collectionEvent.durationInMillis, | ||
| counts: { | ||
| instructions: logs.filter(e => e.category === 'applying' || e.category === 'referenced').length, | ||
| skills: logs.filter(e => e.category === 'skill').length, | ||
| agents: logs.filter(e => e.category === 'custom-agent').length, | ||
| hooks: logs.filter(e => e.category === 'hook').length, | ||
| skipped: logs.filter(e => e.category === 'skipped').length, | ||
| }, | ||
| }; |
There was a problem hiding this comment.
counts.instructions is computed from resolutionLogs entries, but resolutionLogs currently does not include anything for listedInstructionsCount (and could also contain multiple log entries per instruction file in the future). This can make the rendered summary counts disagree with collectionEvent.totalInstructionsCount / the header summary. Prefer deriving instruction counts directly from collectionEvent’s count fields (applying/referenced/agent/listed) or add an explicit log category for “listed” items and include it consistently.
| this._register(chatDebugService.registerProvider({ | ||
| provideChatDebugLog: async () => undefined, | ||
| resolveChatDebugLogEvent: async (eventId) => { | ||
| return this._resolveDiscoveryEvent(eventId); | ||
| return this._resolveDiscoveryEvent(eventId) ?? this._resolveCustomizationEvent(eventId); | ||
| } | ||
| })); |
There was a problem hiding this comment.
New behavior adds a resolvable customizationSummary event kind via _resolveCustomizationEvent(...), but the existing PromptsDebugContribution unit tests only cover discovery → fileList resolution. Add tests that (1) log a customization event and (2) verify resolveEvent returns kind: 'customizationSummary' with expected counts/log entries (including hooks when present).
|
|
||
| if (!pattern) { | ||
| this._logService.trace(`[InstructionsContextComputer] No pattern (applyTo / paths) found: ${uri}`); | ||
| telemetryEvent.debugDetails.push({ category: 'skipped', name: basename(uri).toString(), uri, reason: localize('debugDetail.noPattern', 'no applyTo pattern') }); | ||
| continue; |
There was a problem hiding this comment.
debugDetails entries (including localized reason strings) are collected unconditionally during instruction collection. This adds per-file allocation and localization overhead on every request even when debug logging isn’t enabled/consumed. Consider gating population of telemetryEvent.debugDetails (and maybe durationInMillis) behind the relevant debug-log setting(s), or collecting these details only when a debug consumer is registered.
🤖 QA Agent — Auto-generated Pytest Testsimport pytest
def localize(key, default, count):
return default.format(count)
def getCategorySectionTitle(category, count):
switcher = {
'applying': localize('chatDebug.customization.instructions', "Instructions ({0})", count),
'referenced': localize('chatDebug.customization.referenced', "Referenced ({0})", count),
'skill': localize('chatDebug.customization.skill', "Skills ({0})", count),
'custom-agent': localize('chatDebug.customization.customAgent', "Agents ({0})", count),
'hook': localize('chatDebug.customization.hook', "Hooks ({0})", count),
'skipped': localize('chatDebug.customization.skipped', "Skipped ({0})", count)
}
return switcher.get(category, "")
def test_getCategorySectionTitle_applying():
# Verifies that the title for the 'applying' category is correctly formatted
assert getCategorySectionTitle('applying', 3) == "Instructions (3)"
def test_getCategorySectionTitle_referenced():
# Verifies that the title for the 'referenced' category is correctly formatted
assert getCategorySectionTitle('referenced', 5) == "Referenced (5)"
def test_getCategorySectionTitle_skill():
# Verifies that the title for the 'skill' category is correctly formatted
assert getCategorySectionTitle('skill', 2) == "Skills (2)"
def test_getCategorySectionTitle_custom_agent():
# Verifies that the title for the 'custom-agent' category is correctly formatted
assert getCategorySectionTitle('custom-agent', 4) == "Agents (4)"
def test_getCategorySectionTitle_hook():
# Verifies that the title for the 'hook' category is correctly formatted
assert getCategorySectionTitle('hook', 1) == "Hooks (1)"
def test_getCategorySectionTitle_skipped():
# Verifies that the title for the 'skipped' category is correctly formatted
assert getCategorySectionTitle('skipped', 0) == "Skipped (0)"
def test_getCategorySectionTitle_unknown_category():
# Verifies that an unknown category returns an empty string
assert getCategorySectionTitle('unknown', 10) == "" |
Uh oh!
There was an error while loading. Please reload this page.