Sync a resource

A sync walks resource.each_page and records a Reference per object. No bytes move.

Check it first

{ "name": "check_resource", "arguments": { "id": "..." } }

Resource#check calls the type’s check! — a HEAD or equivalent, not a listing — and writes checked_at and check_error. list_resources reports both.

Sync it

{ "name": "sync_resource", "arguments": { "id": "..." } }

Resource#sync! raises ArgumentError unless syncable?, which is respond_to?(:each_page). Six types implement it directly and two inherit it — see Resources.

Then it takes the lock, starts a Run, and enqueues SyncResourceJob:

def sync!
  raise ArgumentError, "#{self.class.sti_name} is not syncable" unless syncable?
  return false unless claim_sync!

  Run.start!(kind: "sync", resource: self).tap do |run|
    SyncResourceJob.perform_later(tenant_id, id, run.id)
  end
end

Two items make it safe to call repeatedly:

• claim_sync! is a conditional UPDATE on sync_started_at. A resource already syncing returns false rather than starting twice. The claim goes stale after SYNC_ABANDONED_AFTER, 6 hours.
• The job resumes at its cursor. SyncResourceJob#build_enumerator passes the cursor straight into each_page and yields the resource’s own next cursor alongside each object.

Each iteration calls Reference.discover! and enqueues AnalyzeItemJob for anything not yet analyzed.

Cursors are per type

each_page(cursor:) { |page, next_cursor| … } — the cursor’s shape belongs to the dialect:

Type	Cursor
s3	continuation token
webdav, caldav, carddav	the last entry’s path
imap	UID, valid only within a uidvalidity
rss	—

Watch it

{ "name": "list_runs", "arguments": { "kind": "sync" } }

cancel_run sets status to cancelled. The iteration notices on its next halted? check rather than being killed, so it stops within a few objects and whatever was recorded stays recorded.

On a schedule

A syncable resource may hold a sync_interval (minimum 1 minute) and a next_sync_at. ScheduleSyncsJob runs every minute, walks every tenant, and calls sync! on each Resource.due_for_sync.

release_sync! computes the next next_sync_at from the interval rather than from the finish time, so a slow sync does not drift the schedule.

When it fails

Resource::Failed is retried five times with polynomial backoff, then fail_run marks the run failed. See Jobs.