feat: integrate OpenAI Moderation API for listings and messages#40
feat: integrate OpenAI Moderation API for listings and messages#40
Conversation
- Add moderationResults table to schema - Create moderation.ts with moderateContent internalAction - Refactor createListing/updateListing from mutations to actions - Refactor sendMessage from mutation to action - Add ConvexError responses for flagged content - Update all test files for action-based API with fetch mocks - Disable ts-jest diagnostics for convex-test ID serialization - Document OPENAI_API_KEY in .env.example Resolves: POLY-31, POLY-36, POLY-37
📝 WalkthroughWalkthroughAdds an OpenAI moderation service and integrates it into listings and messaging flows by converting key mutations into actions that run moderation and persist via new internal queries/mutations; tests, frontend hooks, schema, and env updated to support moderation and action-based APIs. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant CreateAction as "createListing Action"
participant Moderation as "moderateContent (OpenAI)"
participant InternalMutation as "internalCreateListing"
participant DB as Database
Client->>CreateAction: call action(title, description, ...)
CreateAction->>CreateAction: validate inputs, get profile
CreateAction->>Moderation: internal.moderation.moderateContent(text, contentType, userId)
Moderation->>Moderation: call OpenAI Moderation API
alt Not flagged
Moderation-->>CreateAction: {flagged: false, categories: {}}
CreateAction->>InternalMutation: internal.listings.internalCreateListing(persist payload)
InternalMutation->>DB: insert listing, log moderation result
InternalMutation-->>CreateAction: listingId
CreateAction-->>Client: return listingId
else Flagged
Moderation-->>CreateAction: {flagged: true, categories: {...}}
CreateAction-->>Client: throw Error (content flagged)
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (9)
jest.config.js (1)
22-22: Disabling TS diagnostics hides type errors during test runs.This is a reasonable workaround for convex-test serialization issues, but be aware that genuine type errors in test files will no longer be caught by Jest. Consider adding a separate
tsc --noEmitstep in CI to retain type safety.backend/convex/schema.ts (1)
123-134: Consider adding an index oncontentIdfor per-item moderation lookups.If you ever need to retrieve moderation history for a specific listing or message (e.g., to show admins how many times a piece of content was flagged), the current indexes won't cover that query efficiently. A compound index like
by_contentType_contentIdwould help.Also,
inputTextstores the full user-submitted text. Consider documenting a retention/purge policy for this table, especially for flagged content that may contain harmful material.Suggested index addition
.index('by_userId', ['userId']) - .index('by_contentType', ['contentType', 'createdAt']), + .index('by_contentType', ['contentType', 'createdAt']) + .index('by_contentType_contentId', ['contentType', 'contentId']),backend/convex/listings.ts (3)
588-599: Moderation runs on every create regardless of text content, which is correct. But consider: thecontentIdis not passed for new listings.For audit trail completeness, consider logging the
contentIdafter creation by callinglogModerationResultagain with the newly createdlistingId. Currently the moderation log for new listings won't have acontentIdto correlate back to the listing.
717-732: Moderation runs on every update even when only non-text fields change.When a user updates only
price,images,condition, orcategory, the action still calls the moderation API with the existing title + description. This adds unnecessary latency to non-text updates.Skip moderation when no text fields changed
- // Screen updated text content via moderation - const titleToCheck = (update.title as string) ?? listing.title; - const descToCheck = (update.description as string) ?? listing.description; - - const moderationResult = await ctx.runAction(internal.moderation.moderateContent, { - text: titleToCheck + ' ' + descToCheck, - contentType: 'listing', - userId: identity.subject, - contentId: args.id, - }); - - if (moderationResult.flagged) { - throw new ConvexError( - 'Your listing contains content that violates our community guidelines. Please revise and try again.' - ); - } + // Screen updated text content via moderation (only if text fields changed) + if (update.title !== undefined || update.description !== undefined) { + const titleToCheck = (update.title as string) ?? listing.title; + const descToCheck = (update.description as string) ?? listing.description; + + const moderationResult = await ctx.runAction(internal.moderation.moderateContent, { + text: titleToCheck + ' ' + descToCheck, + contentType: 'listing', + userId: identity.subject, + contentId: args.id, + }); + + if (moderationResult.flagged) { + throw new ConvexError( + 'Your listing contains content that violates our community guidelines. Please revise and try again.' + ); + } + }
672-673: Consider typing theupdateobject instead of usingRecord<string, unknown>with casts.Using a typed object would eliminate the need for all the
ascasts at lines 738-750 and make the code safer.Example typed approach
- const update: Record<string, unknown> = {}; + const update: { + title?: string; + description?: string; + price?: number; + images?: string[]; + condition?: 'new' | 'used' | 'refurbished'; + category?: 'textbooks' | 'electronics' | 'furniture' | 'tickets' | 'other'; + tags?: string[]; + } = {};Then the cast block at lines 735-752 simplifies to:
await ctx.runMutation(internal.listings.internalUpdateListing, { id: args.id, - update: { - title: update.title as string | undefined, - description: update.description as string | undefined, - ... - }, + update, });backend/convex/__tests__/messages.test.ts (1)
40-45: No test for moderation rejection insendMessage.All tests mock moderation to pass (
flagged: false). Consider adding a test where the fetch mock returnsflagged: trueto verify thatsendMessagecorrectly rejects flagged content with the expected error message.Example test for flagged content
+ it('rejects message when content is flagged by moderation', async () => { + const t = createConvexTest(); + + const buyer = await createTestUser(t, 'buyer@calpoly.edu', 'Buyer'); + const seller = await createTestUser(t, 'seller@calpoly.edu', 'Seller'); + const listingId = await createTestListing(t, seller.id); + const conversationId = await createTestConversation(t, listingId, buyer.id, seller.id); + + // Mock moderation to flag content + (global.fetch as jest.Mock).mockResolvedValueOnce({ + ok: true, + json: async () => ({ + results: [{ flagged: true, categories: { harassment: true }, category_scores: {} }], + }), + }); + + const asBuyer = t.withIdentity(buyer.identity); + + await expect(async () => { + await asBuyer.action(api.messages.sendMessage, { + conversationId, + body: 'Flagged content', + }); + }).rejects.toThrow('community guidelines'); + });backend/convex/__tests__/listings-pagination.test.ts (1)
16-39: Duplicated test setup: modules map and fetch mock are repeated across test files.This file defines its own
modulesobject and fetch mock instead of reusingcreateConvexTest()and a shared mock helper fromtestUtils.ts. The same duplication exists inlistings.test.ts. Consider extracting the fetch mock intotestUtils.tsalongside the existingcreateConvexTest()to reduce boilerplate.backend/convex/__tests__/listings.test.ts (1)
289-312: No test for moderation rejection increateListingorupdateListing.Similar to the messages tests, all listing tests mock moderation as passing. Consider adding at least one test that verifies flagged content is rejected for
createListingandupdateListing.backend/convex/moderation.ts (1)
24-27: Graceful-degradation paths skip audit logging — consider logging skipped/failed moderation too.All four early-return paths (
!apiKey, non-OK response, unexpected shape, catch) return{ flagged: false, categories: {} }without writing tomoderationResults. If you ever need to audit how often moderation was bypassed (outage, missing key, etc.), there will be no record. Consider callinglogModerationResultwith a distinguishing flag (e.g.,flagged: false+ askippedorerrorstatus) on at least the error/catch paths.Also applies to: 45-51, 56-59, 63-67
| export const logModerationResult = internalMutation({ | ||
| args: { | ||
| contentType: v.union(v.literal('listing'), v.literal('message')), | ||
| contentId: v.optional(v.string()), | ||
| inputText: v.string(), | ||
| flagged: v.boolean(), | ||
| categories: v.optional(v.string()), | ||
| userId: v.string(), | ||
| }, | ||
| handler: async (ctx, args) => { | ||
| await ctx.db.insert('moderationResults', { | ||
| contentType: args.contentType, | ||
| contentId: args.contentId, | ||
| inputText: args.inputText, | ||
| flagged: args.flagged, | ||
| categories: args.categories, | ||
| userId: args.userId, | ||
| createdAt: Date.now(), | ||
| }); | ||
| }, | ||
| }); |
There was a problem hiding this comment.
Audit table stores full user-submitted text — note PII/compliance implications.
inputText captures the complete message body or listing description. Depending on your compliance posture (GDPR right-to-erasure, CCPA), ensure there is a plan to purge or anonymize old moderationResults rows. This is especially relevant since messages may contain personal information.
… in frontend Frontend was still using useMutation for createListing and updateListing, which are now actions after moderation integration. This caused TypeScript errors: FunctionReference<"action"> not assignable to FunctionReference<"mutation">.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
frontend/app/listings/new.tsx (1)
122-124:⚠️ Potential issue | 🟠 MajorSame
ConvexErrorextraction issue as inedit.tsx.The moderation rejection message is stored in
ConvexError.data, not.message. Apply the same fix here:Proposed fix
+ import { ConvexError } from 'convex/values'; ... } catch (error: unknown) { - const errorMessage = error instanceof Error ? error.message : 'Failed to create listing'; + const errorMessage = + error instanceof ConvexError ? (error.data as string) : + error instanceof Error ? error.message : + 'Failed to create listing'; Alert.alert('Error', errorMessage); }frontend/app/listings/[id]/edit.tsx (1)
119-121:⚠️ Potential issue | 🟠 MajorCheck
ConvexError.datainstead oferror.messagefor custom error messages.When the backend throws
new ConvexError('Your listing contains content that violates...'), Convex stores the string in the error's.dataproperty. On the client, accessingerror.messagewill not contain the custom message. SinceConvexErrorextendsError, the current code matches theinstanceof Errorcheck and attempts to read.message, which fails to surface the user-facing moderation message.Proposed fix
+ import { ConvexError } from 'convex/values'; ... } catch (error: unknown) { - const errorMessage = error instanceof Error ? error.message : 'Failed to update listing'; + const errorMessage = + error instanceof ConvexError ? (error.data as string) : + error instanceof Error ? error.message : + 'Failed to update listing'; Alert.alert('Error', errorMessage); }
ConvexError stores the custom error string in .data, not .message. Without this fix, moderation rejection messages would not surface to the user in the Alert dialog.
2 participants
Linked Issues
Linear: POLY-31, POLY-36, POLY-37
Summary
Integrate the OpenAI Moderation API to screen user-generated content (listings and messages) before persistence, with graceful degradation and audit logging.
Backend changes:
moderation.tsmoderateContentinternalAction — calls OpenAI Moderation API, returns{ flagged, categories }logModerationResultinternalMutation — persists results tomoderationResultstable for auditOPENAI_API_KEYis unset or API fails, content is allowed through with a console warningschema.tsmoderationResultstable withcontentType,contentId,inputText,flagged,categories,userId,createdAtby_userId,by_contentTypelistings.tscreateListingfrom mutation → action withinternalCreateListingmutationupdateListingfrom mutation → action withinternalUpdateListingmutationinternalGetListingquery for ownership verification within actionsmoderateContentbefore persistenceConvexError("Your listing contains content that violates our guidelines...")on flagged contentmessages.tssendMessagefrom mutation → action withinternalSendMessagemutationmoderateContentbefore persistenceConvexError("Your message contains content that violates our community guidelines...")on flagged contentjest.config.jsdiagnostics: falseto ts-jest — required becauseconvex-testserializes action return values to plain strings, losing brandedIdtypesTest changes:
listings.test.ts—.mutation()→.action()forcreateListing/updateListing, added fetch mock + moderation modulemessages.test.ts—.mutation()→.action()forsendMessage, added fetch mock + moderation modulelistings-pagination.test.ts—.mutation()→.action()forcreateListing, added fetch mock + moderation modulereports.test.ts—.mutation()→.action()forcreateListing, added fetch mock + moderation moduletestUtils.ts— Added moderation module to shared test modulesEnvironment variables:
OPENAI_API_KEY— New, documented in.env.examplewith placeholder value. Moderation skips if unset. I will have the key added to our shared .env fileHow to Test
Steps to verify locally:
Manual flow:
npm run dev:backend(in terminal A)npm run dev(in terminal B)Verify the change:
moderationResultstable in Convex dashboard → results are logged for each moderation checkuseMutationtouseActionforcreateListing,updateListing, andsendMessagein a follow-up PR.Checklist
.env.exampleupdated withOPENAI_API_KEY).env.exampleuses placeholder values only)Summary by CodeRabbit
New Features
Security / Behavior
Chores