AI & Automation

Gong Transcript Sync: A Self-Healing Pipeline on GitHub Actions and Supabase

Marq records its sales calls. A copy of every transcript is supposed to land in a database so the team's AI tools can read them, but the old copier had quietly stopped working. I rebuilt it as a small program that runs on a schedule, checks its own work against the source every day, and fixes any gap itself. Then I gave the AI tools a safe way to read and download the calls.

Client: Marq

Key Metrics

10 days Ask to Live
4,735 of 4,735 Transcripts Backfilled
21 of 21 Hourly Runs Green on Day One
1,709 calls in 18 s Largest Export Verified
Architecture diagram: Gong API feeds a single GitHub Actions job (hourly sync, daily reconcile, self-healing backfill) writing to Supabase Postgres with row-level security, read by MCP tools and AI agents

Project Details

Marq's Head of Growth opened with one line: make sure the database is always up to date with Gong transcripts, start there first. The existing sync was two automations pasted into a browser-based workflow tool, with no version control and no review. Both had stopped writing without an error: a step read a key the step before it never produced, a fallback swallowed the empty value, and every run reported nothing to sync, which looks exactly like a healthy run. Only a fraction of the company's calls had reached the warehouse. Ten days after the ask, a single committed TypeScript job was live on a GitHub Actions schedule, writing to Supabase, and had backfilled every one of the 4,735 calls Gong had on record, an exact match. Since then it has gone hourly, grown a daily reconcile that lists every call ID since 2020 and proves none is missing, and gained MCP tools that let Claude and the sales team's agents search, read and batch-download transcripts without the transcript text ever passing through the model.

Challenge

  • A failure that looks like success. The old sync's bug produced a clean "nothing to sync" on every run. The replacement had to fail loudly, and had to be reviewable in a pull request so a crossed wire could never land unseen again.
  • Someone else's database. The target was a Supabase project that already served the company's analytics authentication. The schema, row-level security and storage had to go in without touching what was there, and before a single row landed.
  • Access without the keys. The role I was granted could run SQL from the dashboard but could not reset the database password, create a scoped key, or change pooler settings. The one credential the job needed was the one nobody could see.
  • Shipping safely from a hostile default. Scheduled workflows only run from the default branch, so the runner could not be tested before merging. GitHub schedules also silently disable after 60 days without commits, the same silent-death shape as the bug being replaced.

Approach

One writer, the database as the state

The design is deliberately small. One GitHub Actions job owns the Gong credentials and is the only thing that ever writes. There is no cursor and no sync log: on every run the job lists calls from Gong, diffs them against what the warehouse already holds, and fetches only the missing ones, so a daily run and a full historical backfill are the same code path with a different window. Each call is written in its own transaction with an idempotent upsert, the raw API payload is stored in a private bucket with a SHA-256 checksum, and any unexpected response shape ends the run with a red X. Row-level security was enabled before any data existed: default deny, read-only for company accounts, writes impossible even for authenticated users after I revoked the grants Supabase adds by default.

Secrets never entered the repository. The script reads its environment at run time, the workflow holds five Actions secrets, and the Supabase pooler's private certificate authority was committed as a public file and trusted through Node's extra-CA setting, rather than disabling TLS verification on a connection carrying the whole transcript corpus.

Proving it before the password arrived

While the database password was still with its owner, I verified everything that did not need it. A throwaway harness run through the SQL editor checked that the ported row builder fit the live schema on a real call: all 63 columns present, every not-null column covered, JSON stored as JSON, a double insert leaving one row, and the foreign-key cascade deleting party rows with their call. I also read the live catalog to show that nothing else connected with the password at all, which turned a nervous "can we reset it" into a confident ask, and then it turned out the owner had it saved. The first live run upserted seven calls with real transcripts and CRM links, and an immediate re-run reported zero missing.

A review that found three ways to hide missing data

The pull request came back from the Head of Growth with findings from an automated review, three of which could recreate the exact failure the job existed to remove: a call listed before Gong had produced its transcript was written once and never revisited; a missing payload was skipped but still counted toward completion, so the backfill could declare itself done with calls absent; and a free-form dispatch input was interpolated into a shell step that held secrets. All three were closed before the backfill ran. Completeness is now tracked separately from row existence, with calls awaiting a transcript retried for 14 days. Skipped calls stay in the remaining count and the completion marker cannot fire while anything was skipped. The input is validated. The review also made a standing rule: every change to this repo now passes an automated simplify and multi-lens review before the pull request opens.

Merged on day ten, the backfill drained in six runs: 4,735 of 4,735 calls, back to August 2023, 52 of them awaiting transcripts inside the retry window. Two weeks later the count stood at 98.9 percent with transcripts, the rest being calls Gong itself never transcribed, reported to the stakeholder as the permanent gap rather than left to read as a discrepancy.

From warehouse to tools the agents can use

With the data layer live, the warehouse went onto the company's hosted MCP server: search calls, get a transcript, and the one the Head of Growth asked for after testing the first two, a single call that exports a whole date range as a zip assembled server-side, a manifest to skim and one Markdown file per call with YAML frontmatter. The tool response carries no transcript text at all, which was the actual requirement: an agent that reads the files from disk quotes verbatim instead of reconstructing a call from memory. The review gate on that build caught a counted call that could vanish from a zip when a row landed between the count and the drain, fixed by draining by identity, and a CSV formula injection path through call titles, which an external participant controls. Raising the export cap to 2,000 calls meant finding that the database API silently truncates every response at 1,000 rows, measuring memory at 425 MB peak rather than guessing, and proving it in production with a 1,709-call export in 18 seconds.

Hourly, then a daily proof that nothing is missing

In September the VP of Operations asked how hard hourly would be, because reps were building AI skills that act on calls from earlier the same day. It went live four days later with a three-hour lookback and 21 of 21 runs green on the first day. The cutover itself opened a gap: calls between the last daily run and the first hourly one were never listed, and one of them was noticed a week later. I repaired it with a single backfill dispatch, 11 calls, and built the permanent answer the stakeholder had asked for: a third schedule that lists every call ID since 2020, diffs it against the warehouse and ingests whatever is missing, with a guard that fails the run if a schedule ever matches none of the known crons, so an edited cron cannot quietly demote the reconcile. Its first run reported zero missing.

Results

  • Live ten days after the ask, with every historical call backfilled to an exact match against Gong's own total.
  • The silent-failure shape is gone. A green run means the inventory matched; a missing payload, a wrong response shape or an unknown schedule all end red.
  • Hourly sync plus a daily reconcile, so the sales team's agents see calls from the same morning and a gap can survive at most a day.
  • Transcripts reachable by agents without entering the model's context, through search, single-transcript and batch-export MCP tools, the export verified at 1,709 calls in 18 seconds.
  • Nothing sensitive in the repo: secrets in Actions, TLS verified against a committed public CA, row-level security proven against the live API with an anonymous request rejected.
  • The pattern was reused for the company's second transcript source and became the foundation for an AI questionnaire layer over every call.

Tools Used

GitHub Actions, TypeScript, Supabase (PostgreSQL, storage, row-level security), Gong API, Model Context Protocol, Node.js, Docker on AWS ECS

GitHub Actions Supabase PostgreSQL TypeScript Gong API MCP Data Pipeline Row-Level Security Backfill Reliability Sales Enablement automation

Get Started

Looking to Hire?

I bring 19+ years of cross-functional experience to every role. Connect on LinkedIn to learn more.

Need a Consultant?

Start with a one-hour session on the decision in front of you, or send me the brief for a longer engagement.