Skip to content

feat(imessage): receive mini-app content #236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Tom Tang (qwerzl) wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(imessage): receive mini-app content #236

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
da65c25
feat(imessage): receive mini-app content
qwerzl Aug 13, 2026
893e447
chore(imessage): require mini-app decoder
qwerzl Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions bun.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions packages/core/src/authoring.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ export {
setLogLevel,
} from "@photon-ai/otel";
// Content factories, schemas, and the inbound-record type (from `content/`).
export { asResolvedApp } from "./content/app";
export { asAttachment } from "./content/attachment";
export { avatarSchema } from "./content/avatar";
export { asContact } from "./content/contact";
Expand Down
55 changes: 43 additions & 12 deletions packages/core/src/content/app.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,18 @@ import type { ContentBuilder } from "./types";
* `image` and `imageTitle` must be set together; `imageSubtitle` requires
* `image`.
*/
export const appLayoutSchema = z
.object({
caption: z.string().nonempty().optional(),
subcaption: z.string().nonempty().optional(),
trailingCaption: z.string().nonempty().optional(),
trailingSubcaption: z.string().nonempty().optional(),
image: z.instanceof(Uint8Array).optional(),
imageTitle: z.string().nonempty().optional(),
imageSubtitle: z.string().nonempty().optional(),
summary: z.string().nonempty().optional(),
})
const resolvedAppLayoutSchema = z.object({
caption: z.string().nonempty().optional(),
subcaption: z.string().nonempty().optional(),
trailingCaption: z.string().nonempty().optional(),
trailingSubcaption: z.string().nonempty().optional(),
image: z.instanceof(Uint8Array).optional(),
imageTitle: z.string().nonempty().optional(),
imageSubtitle: z.string().nonempty().optional(),
summary: z.string().nonempty().optional(),
});

export const appLayoutSchema = resolvedAppLayoutSchema
.refine(
(layout) =>
layout.caption !== undefined ||
Expand Down Expand Up @@ -63,7 +64,7 @@ export type AppLayout = z.infer<typeof appLayoutSchema>;
const urlAccessor = z.function({ input: [], output: z.promise(z.url()) });
const layoutAccessor = z.function({
input: [],
output: z.promise(appLayoutSchema),
output: z.promise(resolvedAppLayoutSchema),
});

export const appSchema = z.object({
Expand Down Expand Up @@ -192,6 +193,36 @@ export const asApp = (url: AppUrl, options: AppOptions = {}): App => {
});
};

/**
* Construct app content from an already-decoded URL and visible layout.
* Provider adapters use this for inbound native app cards so reading the
* content never refetches or invents metadata that the sender did not supply.
*
* @param url - The decoded URL delivered by the native app card.
* @param layout - The decoded layout or a lazy resolver for native media.
* @param options - Optional app rendering behavior.
* @returns Validated app content backed only by the decoded native fields.
*/
export const asResolvedApp = (
url: string,
layout: AppLayout | (() => Promise<AppLayout>),
options: AppOptions = {}
): App => {
const resolvedUrl = z.url().parse(url);
const getLayout = memoize(async () =>
resolvedAppLayoutSchema.parse(
typeof layout === "function" ? await layout() : layout
Comment on lines +206 to +214

@cursor cursor Bot Aug 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the key root-cause fix, not a workaround: inbound native cards already have a decoded layout, so reusing asApp (URL scrape / Open Graph) would invent or overwrite sender-provided card data.

One intentional tradeoff here: this path validates with the looser base schema and skips the stricter authoring refinements (visible slot required, image/title pairing). That seems right for “preserve what Apple sent,” but inbound app layouts may be weaker than outbound ones.

)
);

return appSchema.parse({
type: "app",
url: () => Promise.resolve(resolvedUrl),
layout: getLayout,
...options,
});
};

/**
* Construct an app card from a URL.
*
Expand Down
2 changes: 1 addition & 1 deletion packages/imessage/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@
},
"dependencies": {
"@grpc/grpc-js": "^1.14.4",
"@photon-ai/advanced-imessage": "^2.0.2",
"@photon-ai/advanced-imessage": "^2.1.0",
"@photon-ai/otel": "^3.3.0",
"lru-cache": "^11.0.0",
"marked": "^18.0.5",
Expand Down
36 changes: 36 additions & 0 deletions packages/imessage/src/remote/inbound.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
asContact,
asCustom,
asReply,
asResolvedApp,
asText,
asVoice,
createLogger,
Expand Down Expand Up @@ -252,12 +253,47 @@ const buildOrderedPartMessage = async (
const unsupportedMessageContent = (): Content =>
asCustom({ imessage_type: "unsupported-message" });

const toMiniAppContent = (
client: AdvancedIMessage,
message: AppleMessage
): Content | undefined => {
const miniApp = message.content.miniApp;
if (!(miniApp?.url && miniApp.layout)) {
return;
}

const imageAttachment = messageAttachments(message).find((attachment) =>
normalizeAppleAttachmentMimeType(attachment).startsWith("image/")
);
Comment on lines +256 to +267

@cursor cursor Bot Aug 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good receive mapping overall: require both url and layout before creating renderable app content, and leave identity-only mini-apps as metadata.

Minor edge case: image bytes come from the first image/* attachment. Fine for typical mini-app cards; worth watching if Apple ever attaches extra images alongside the card art.

const layout = imageAttachment
? async () => ({
...miniApp.layout,
image: Uint8Array.from(
await downloadPrimaryAttachment(client, imageAttachment.guid)
),
})
: miniApp.layout;

return asResolvedApp(miniApp.url, layout, {
live: miniApp.live,
});
};

const buildUnwrappedContentMessage = async (
client: AdvancedIMessage,
base: RemoteMessageBase,
message: AppleMessage,
messageGuidStr: string
): Promise<IMessageMessage> => {
const miniAppContent = toMiniAppContent(client, message);
if (miniAppContent) {
return {
...base,
id: messageGuidStr,
content: miniAppContent,
};
}

const attachments = messageAttachments(message);
const voiceAttachmentGuid = message.isAudioMessage
? attachments.find((attachment) => appleAudioMimeType(attachment))?.guid
Expand Down
30 changes: 30 additions & 0 deletions packages/imessage/src/remote/message-metadata.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ import type {
MessageMention,
MessagePlacedSticker,
MessageReaction,
MiniAppContent,
MiniAppLayoutInfo,
SingleServiceAddressInfo,
StickerPlacement,
TextFormat,
Expand All @@ -13,6 +15,8 @@ import type {
IMessageAppliedReaction,
IMessageAttachmentMetadata,
IMessageMention,
IMessageMiniAppContent,
IMessageMiniAppLayoutInfo,
IMessageNativeMessageMetadata,
IMessagePlacedSticker,
IMessageReaction,
Expand All @@ -34,6 +38,29 @@ const toMention = (mention: MessageMention): IMessageMention => ({
start: mention.start,
});

const toMiniAppLayout = (
layout: MiniAppLayoutInfo
): IMessageMiniAppLayoutInfo => ({
caption: layout.caption,
imageSubtitle: layout.imageSubtitle,
imageTitle: layout.imageTitle,
subcaption: layout.subcaption,
summary: layout.summary,
trailingCaption: layout.trailingCaption,
trailingSubcaption: layout.trailingSubcaption,
});

const toMiniAppContent = (miniApp: MiniAppContent): IMessageMiniAppContent => ({
appName: miniApp.appName,
appStoreId: miniApp.appStoreId,
extensionBundleId: miniApp.extensionBundleId,
layout: miniApp.layout ? toMiniAppLayout(miniApp.layout) : undefined,
live: miniApp.live,
sessionId: miniApp.sessionId,
teamId: miniApp.teamId,
url: miniApp.url,
});

const toAttachmentMetadata = (
attachment: AttachmentInfo
): IMessageAttachmentMetadata => ({
Expand Down Expand Up @@ -134,6 +161,9 @@ export const toMessageMetadata = (
sendErrorCode: native.sendErrorCode,

nativeText: native.content?.text,
miniApp: native.content?.miniApp
? toMiniAppContent(native.content.miniApp)
: undefined,
formatting: native.content?.formatting?.map(toTextFormat) ?? [],
mentions: native.content?.mentions?.map(toMention) ?? [],
subject: native.subject,
Expand Down
28 changes: 28 additions & 0 deletions packages/imessage/src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,31 @@ const miniAppCardSessionSchema = z.object({
targetMessageGuid: z.string(),
});

const miniAppLayoutInfoSchema = z
.object({
caption: z.string().optional(),
imageSubtitle: z.string().optional(),
imageTitle: z.string().optional(),
subcaption: z.string().optional(),
summary: z.string().optional(),
trailingCaption: z.string().optional(),
trailingSubcaption: z.string().optional(),
})
.readonly();

const miniAppContentSchema = z
.object({
appName: z.string().optional(),
appStoreId: z.number().int().positive().optional(),
extensionBundleId: z.string(),
layout: miniAppLayoutInfoSchema.optional(),
live: z.boolean(),
sessionId: z.string().optional(),
teamId: z.string(),
url: z.url().optional(),
})
.readonly();

const textFormatSchema = z
.object({
effectName: z.string().optional(),
Expand Down Expand Up @@ -178,6 +203,7 @@ export const nativeMessageMetadataSchema = z.object({
sendErrorCode: z.number().int(),

nativeText: z.string().optional(),
miniApp: miniAppContentSchema.optional(),
formatting: z.array(textFormatSchema).readonly(),
mentions: z.array(mentionSchema).readonly(),
subject: z.string().optional(),
Expand Down Expand Up @@ -231,6 +257,8 @@ export type IMessageAttachmentMetadata = z.infer<
typeof attachmentMetadataSchema
>;
export type IMessageMention = z.infer<typeof mentionSchema>;
export type IMessageMiniAppContent = z.infer<typeof miniAppContentSchema>;
export type IMessageMiniAppLayoutInfo = z.infer<typeof miniAppLayoutInfoSchema>;
export type IMessageNativeMessageMetadata = z.infer<
typeof nativeMessageMetadataSchema
>;
Expand Down
Loading
Loading