Content sources as data processing pipelines

[schickling]

Note: This issue is WIP and needs to be expanded.

Notes

• Differentiate between input data and output data
• Input data validation could be done via Zod
  • Question: How do define the output schema? (based on the input schema)
  • Other schema formats: Leveraging JSON Schema #163
  • Add Union Field Type #250
  • Also see this comment
• This would also unlock Add support for multiple sources #84
• Make it easy to let the user define what they want to access (e.g. pre-rendered HTML vs raw MD source)

Markdown/MDX processing pipeline

• This could allow for a smoother unified/remark/rehype content processing integration
• Investigate table of content use case #137

Related

• Consider renaming field option default to fallback #120
• Leveraging JSON Schema #163
• Pass both frontmatter and body field for md and mdx content #106
• Add support for multiple sources #84
• Consider renaming required to nullable #160
• Combine fields and computedFields #91
  • Provide a resolve key on fields to generate a default value when one is not provided #275
• Fields of type reference should be able to embed their data #86
• Add support for named scopes ("computed collection") #71
• Allow me to define fields I don't want in the output #148
• Provide more utility functions (pick, ...) #24
• Support for nested field types in computed fields #149
• Support referencing TS type definitions for type: 'json' fields #26
• Provide a way to populate field values from Markdown/MDX processing pipeline #216
• Consider removing the distinction between document and nested types #225
• Preprocessing and Postprocessing Hooks? #325
• Private Notion doc