Sync
How Uptend keeps profiles current from a mapped database, how often each kind of read runs, and how to run one now.
Once a source has a backing table, Uptend reads it on a schedule and turns its rows into profiles. Nothing is pushed from your side and nothing is written to your database; every read runs inside the same read-only transaction as everything else Uptend sends.
The Sync card on a source's page under Settings → Data → Data sources shows what runs, how often, what the last runs did, and lets you run one now.
Four kinds of read
| Read | What it does | When it runs |
|---|---|---|
| Full | Reads every row of the backing table through the row filter, a thousand at a time, and creates or updates a profile for each. | The first sync, then nightly. |
| Incremental | Reads only the rows whose cursor column moved since the last read. | Every 15 minutes when the cursor is trustworthy or usable; hourly when it is unindexed. |
| Discovery | Reads only rows that are new since the last read, using the signed-up column. Snapshot mode's answer to "who signed up in the last fifteen minutes" when there is no cursor column. | Every 15 minutes, when the mapping has no usable cursor and Signed up is a column on the table. |
| Refresh | Re-reads the profiles Uptend already holds, by key, and recomputes every joined, first-row and list value and every usage metric for each. A profile whose row is gone from the table, or no longer passes the row filter, is marked deleted in source. | On the Refresh derived values cadence, for the primary source. |
Runs against one source never overlap. If a scheduled read and a manual one are asked for together, they run one after the other.
Why derived values have their own cadence
An incremental read sees a row when that row changes. An account's "active users this week" changes when a user row changes, and the account row does not move. So the derived shapes — joined value, first related row, list — cannot ride the change stream. They are recomputed for every profile on the Refresh derived values cadence, which you set on the Sync card: every 15 minutes up to daily, one hour by default.
The cost is one statement per derived property per thousand profiles, run in your database. A heavier cadence is fine on an indexed schema and worth checking against the cost verdict on an expensive one.
Usage metrics ride the same refresh. A metric's windows — the last 7, 30 and 90 days, the previous windows, the weekly series — move even when nothing about the profile changed, so its whole family is recomputed for every profile on this cadence, and for the rows a full or incremental read touches. Metrics that share a table and a path cost one pair of statements per thousand profiles however many of them there are.
A metric that is not indexed in your database holds the refresh to daily. When a metric's join column has no index, its statement walks the related table for every batch, and that is the slowest thing a refresh can do. Until the index exists the source refreshes at most once a day, whatever cadence you chose; the Sync card says so under the cadence. Each such metric carries a ready-to-run CREATE INDEX CONCURRENTLY recipe; add the index, re-read the schema, and the cadence you chose applies again.
Changes, deletes, and what a read can't see
- A row that reappears is un-deleted. A profile marked deleted in source comes back the next time its key is read.
- An incremental read cannot see a delete. A row that no longer exists is not in "rows that changed". The refresh read is what notices, by re-reading the profiles Uptend holds and marking the missing ones.
- A deleted-at column is honoured on every read. If the mapping names one, a row with a value there (or
true, for a boolean column) is marked deleted rather than updated. - Only the primary source deletes. An additional source's rows say nothing about whether a person is still a customer, so its reads only enrich.
- Reads start a little behind. Every incremental and discovery read begins five minutes before the point the last one reached and re-reads the overlap. That is deliberate: a transaction can stamp its
updated_atand commit later, and without the overlap that row would never be seen. Re-reading is harmless because the same row always updates the same profile.
Sync now
Click Sync now on the Sync card. What it runs depends on the source:
- The first time, it runs a full read. The first thousand rows are read before the page comes back, so there are profiles to look at immediately; the rest continues in the background and the card shows progress.
- After that, it runs the change stream (incremental, or discovery in snapshot mode) and then, for the primary source, a refresh — so a property or usage metric you just added is computed without waiting for its cadence. The Usage page offers the same button when its metrics are waiting on a refresh.
The button is disabled while a run is in flight.
Reading the card
Schedule says what the cursor verdict allows. Recent runs lists the last runs with their kind, whether they were scheduled or manual, and what they did: rows read, profiles created, updated and deleted, and how many issues they reported. A failed run shows its reason on the row.
Issues from the latest run lists what the last run could not use, with the row's key in your table so you can find it: a value that would not coerce to the property's type, a row an additional source could not match to a profile, or an expression or usage metric that could not run. A usage metric that timed out and is not indexed in your database reports the index recipe with the error, so the fix is on the card rather than in a log. Up to five hundred issues are kept per run.
The profiles list says Data as of the last successful run, because a synced copy is only ever as current as its last read.
What a source sees
Every read connects as the role you gave Uptend, from a single connection, one batch at a time, with a sixty-second statement timeout. pg_stat_activity shows the connections as uptend. If you can point Uptend at a read replica rather than your primary, do — it removes the question of load entirely.
A run that stops reporting progress for three hours is marked failed. After any failure the source is left alone for an hour before the schedule tries again. Pausing the source stops every read until you resume it.