Skip to content

feat(pgpm/export): loop-based export + auto-discover tables#1137

Open
pyramation wants to merge 1 commit into
mainfrom
chore/auto-discover-export-tables
Open

feat(pgpm/export): loop-based export + auto-discover tables#1137
pyramation wants to merge 1 commit into
mainfrom
chore/auto-discover-export-tables

Conversation

@pyramation
Copy link
Copy Markdown
Contributor

Summary

Addresses constructive-planning#814.

Phase 1 — Loop-based export: Replaces ~50 individual queryAndParse() calls in both export-meta.ts (SQL) and export-graphql-meta.ts (GraphQL) with loops over META_TABLE_ORDER. Both flows now use the same array as the single source of truth, reducing update points from 4 to 2 when adding a new table.

Phase 2 — Auto-discover (SQL flow): After exporting all configured tables, the SQL flow queries information_schema.tables for the 3 export schemas (metaschema_public, services_public, metaschema_modules_public) and auto-exports any new tables not already in META_TABLE_ORDER. Field types are auto-inferred from information_schema.columns via mapPgTypeToFieldType(). EXPORT_BLACKLIST excludes tables that should not be exported (node_type_registry, blueprint*).

GraphQL flow remains config-based (it needs field names to build query fragments and has no direct DB access), but the loop-based approach means adding a table to GraphQL export only requires updating META_TABLE_CONFIG + META_TABLE_ORDER (2 places, down from 4).

New exports from export-utils.ts:

  • EXPORT_SCHEMAS — the 3 schemas that participate in export
  • EXPORT_BLACKLIST — tables excluded from auto-discovery
  • mapPgTypeToFieldType() — now public, used by auto-discovery

Net effect for adding a new table:

  • SQL export: zero code changes (auto-discovered)
  • GraphQL export: add to META_TABLE_CONFIG + META_TABLE_ORDER (2 places)
  • To blacklist a table: add to EXPORT_BLACKLIST (1 place)

Review & Testing Checklist for Human

Medium risk — behavioral refactor of export logic, but no new data models.

  • Run pgpm export against a real database to verify SQL output matches previous behavior
  • Verify GraphQL export still produces identical output (the loop groups by schema and runs in parallel, same as before)
  • Check that auto-discovered tables get correct field types (especially image/upload/url overrides — these only apply to configured tables, auto-discovered ones get standard type inference)
  • Confirm EXPORT_BLACKLIST tables are not present in export output

Notes

  • The export-graphql-meta.ts file went from ~210 lines to ~135 lines (net -80 lines of repetitive calls)
  • The export-meta.ts file went from ~217 lines to ~209 lines, but gained auto-discovery logic
  • Test updates: parity tests now verify both files iterate META_TABLE_ORDER (structural check) instead of regex-matching individual queryAndParse('table_name') calls
  • Auto-discovered tables only get standard type inference (uuid, text, boolean, jsonb, int, etc.). Special type hints (image, upload, url) still require an entry in META_TABLE_CONFIG with the override

Link to Devin session: https://app.devin.ai/sessions/f42f42528fb0465684159227efbef017
Requested by: @pyramation

Phase 1: Replace ~50 individual queryAndParse() calls in both SQL and
GraphQL flows with loops over META_TABLE_ORDER. Both flows now use the
same array as the single source of truth, eliminating 2 of 4 update
points when adding new tables.

Phase 2: SQL flow auto-discovers tables via information_schema.tables
for the 3 export schemas. New tables are automatically exported without
any config changes. EXPORT_BLACKLIST excludes tables that should not be
exported (node_type_registry, blueprint*, etc).

GraphQL flow remains config-based (needs field names for query fragments)
but the loop-based approach means adding a table only requires updating
META_TABLE_CONFIG + META_TABLE_ORDER (2 places, down from 4).

Closes constructive-io/constructive-planning#814
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant