Skip to content

fix(js/plugins/google-cloud): Add input and output logs for prompt rendering. #3124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix(js/plugins/google-cloud): Add input and output logs for prompt rendering. #3124

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
079049d
Add input and output logs for prompt rendering.
bryanatkinson Jun 23, 2025
a6430ad
Write logs for all action types except indexer and embedder
bryanatkinson Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 7 additions & 19 deletions js/plugins/google-cloud/src/gcpOpenTelemetry.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,27 +351,15 @@ class AdjustingTraceExporter implements SpanExporter {
featuresTelemetry.tick(span, this.logInputAndOutput, this.projectId);
// Set root status explicitly
span.attributes['genkit:rootState'] = span.attributes['genkit:state'];
} else {
if (type === 'action' && subtype === 'model') {
// Report generate metrics () for all model actions
generateTelemetry.tick(span, this.logInputAndOutput, this.projectId);
}
if (type === 'action' && subtype === 'tool') {
// TODO: Report input and output for tool actions
}
if (
type === 'action' ||
type === 'flow' ||
type == 'flowStep' ||
type == 'util'
) {
// Report request and latency metrics for all actions
actionTelemetry.tick(span, this.logInputAndOutput, this.projectId);
}
}
if (type === 'userEngagement') {
} else if (type === 'action' && subtype === 'model') {
// Report generate metrics () for all model actions
generateTelemetry.tick(span, this.logInputAndOutput, this.projectId);
} else if (type === 'userEngagement') {
// Report user acceptance and feedback metrics
engagementTelemetry.tick(span, this.logInputAndOutput, this.projectId);
} else {
// Report request and latency metrics for all actions
actionTelemetry.tick(span, this.logInputAndOutput, this.projectId);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block is no longer hit if [type=action, subtype=model]. Is that intended?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep! The action telemetry now only writes logs, and only for specific action types (prompt, generate action wrapper, and tools), and so although this was previously called for the model actions it was a no-op. We write logs for those actions above with the 'generateTelemetry'.

}
}

Expand Down
74 changes: 36 additions & 38 deletions js/plugins/google-cloud/src/telemetry/action.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,48 +31,46 @@ class ActionTelemetry implements Telemetry {
logInputAndOutput: boolean,
projectId?: string
): void {
if (!logInputAndOutput) {
const attributes = span.attributes;
const type = attributes['genkit:type'] as string;
if (!logInputAndOutput || type === 'embedder' || type === 'indexer') {
return;
}
const attributes = span.attributes;
const actionName = (attributes['genkit:name'] as string) || '<unknown>';
const subtype = attributes['genkit:metadata:subtype'] as string;

if (subtype === 'tool' || actionName === 'generate') {
const path = (attributes['genkit:path'] as string) || '<unknown>';
const input = truncate(attributes['genkit:input'] as string);
const output = truncate(attributes['genkit:output'] as string);
const sessionId = attributes['genkit:sessionId'] as string;
const threadName = attributes['genkit:threadName'] as string;
let featureName = extractOuterFeatureNameFromPath(path);
if (!featureName || featureName === '<unknown>') {
featureName = actionName;
}
const actionName = (attributes['genkit:name'] as string) || '<unknown>';
const path = (attributes['genkit:path'] as string) || '<unknown>';
const input = truncate(attributes['genkit:input'] as string);
const output = truncate(attributes['genkit:output'] as string);
const sessionId = attributes['genkit:sessionId'] as string;
const threadName = attributes['genkit:threadName'] as string;
let featureName = extractOuterFeatureNameFromPath(path);
if (!featureName || featureName === '<unknown>') {
featureName = actionName;
}

if (input) {
this.writeLog(
span,
'Input',
featureName,
path,
input,
projectId,
sessionId,
threadName
);
}
if (output) {
this.writeLog(
span,
'Output',
featureName,
path,
output,
projectId,
sessionId,
threadName
);
}
if (input) {
this.writeLog(
span,
'Input',
featureName,
path,
input,
projectId,
sessionId,
threadName
);
}
if (output) {
this.writeLog(
span,
'Output',
featureName,
path,
output,
projectId,
sessionId,
threadName
);
}
}

Expand Down
47 changes: 46 additions & 1 deletion js/plugins/google-cloud/tests/logs_test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -395,7 +395,6 @@ describe('GoogleCloudLogs', () => {
await getExportedSpans();

const logs = await getLogs(1, 100, logLines);
expect(logs.length).toEqual(9);
const logObjectMessages = getStructuredLogMessages(logs);
expect(logObjectMessages).toContain(
'Config[testFlow > sub1 > sub2 > generate > testModel, testModel]'
Expand Down Expand Up @@ -524,6 +523,52 @@ describe('GoogleCloudLogs', () => {
expect(logObjectMessages).toContain('UserAcceptance[flowName]');
});

it('writes prompt template input and output logs', async () => {
const testPrompt = ai.definePrompt(
{
name: 'testPrompt',
input: {
schema: z.object({ name: z.string() }),
},
},
async (input) => {
return {
messages: [
{ role: 'user', content: [{ text: `Hello, ${input.name}` }] },
],
};
}
);
const testFlow = createFlowWithInput(ai, 'testFlow', async (input) => {
return await testPrompt.render({ name: input });
});

await testFlow('test');

await getExportedSpans();
const logs = await getLogs(1, 100, logLines);
const logObjectMessages = getStructuredLogMessages(logs);
expect(logObjectMessages).toContain('Input[testFlow > render, testFlow]');
expect(logObjectMessages).toContain('Output[testFlow > render, testFlow]');
// Ensure the structured data is as expected
logs.map((log) => {
const structuredLog = JSON.parse(log as string);
if (structuredLog.message === 'Input[testFlow > render, testFlow]') {
expect(JSON.parse(structuredLog.content)).toEqual({ name: 'test' });
}
if (structuredLog.message === 'Output[testFlow > render, testFlow]') {
expect(JSON.parse(structuredLog.content)).toEqual({
messages: [
{
content: [{ text: 'Hello, test' }],
role: 'user',
},
],
});
}
});
});

it('writes tool input and output logs', async () => {
const echoTool = ai.defineTool(
{ name: 'echoTool', description: 'echo' },
Expand Down