Jobs and failure

Solid Queue, in a second database (config.solid_queue.connects_to), in development as well as production. bin/jobs runs it, bin/dev keeps it up, and /jobs mounts Mission Control.

Two pools

config/queue.yml:

Pool	Queues	Threads	Processes
bulk	`sync`, `export`, `default`	5	`BULK_CONCURRENCY`, default 1
analysis	`analysis`	`ANALYSIS_THREADS`, default 4	`ANALYSIS_CONCURRENCY`, default 1

Splitting them keeps a large sync from occupying the workers analysis needs. Within the analysis pool, AnalyzeItemJob caps concurrency per tenant with limits_concurrency (ANALYSIS_PER_TENANT, default 2).

Recurring

config/recurring.yml:

Job	Queue	Schedule
`ScheduleSyncsJob`	`sync`	every minute
`SweepAuditEventsJob`	`sync`	daily at 4am
`SolidQueue::Job.clear_finished_in_batches`	—	hourly at :12, production only

ScheduleSyncsJob walks every tenant and calls sync! on each Resource.due_for_sync.

Iteration

Every job over an unbounded number of items includes JobIteration::Iteration and TrackedRun:

Job	Queue	Gate key
`SyncResourceJob`	`sync`	`sync`
`ExportItemsJob`	`export`	`export`
`AnalyzeItemsJob`	`default`	`analyze`
`ReindexItemsJob`	`default`	`reindex`
`ProposeMergesJob`	`default`	`dedupe`

Three rules follow from job-iteration:

each_iteration enqueues rather than computes. Iterations must finish quickly for graceful shutdown to work; AnalyzeItemsJob#each_iteration is one perform_later call.
The cursor shape belongs to the type. SyncResourceJob wraps whatever resource.each_page yields as its next cursor.
Code after yield in a custom enumerator is not guaranteed to run.

ReindexItemsJob and ProposeMergesJob iterate pages rather than rows, cursoring on the last id of each page, because one request per document made a rebuild a request storm.

Runs

app/models/run.rb. Every bulk operation creates one, and TrackedRun updates it.


`kind`	`sync`, `export`, `analyze`, `reindex`, `dedupe`
`status`	`queued`, `running`, `done`, `failed`, `cancelled`, `gated`
`processed`	incremented by `progressed!`
`deadline`	optional; passing it cancels the run

Run#halted? re-reads the status from the database and returns true if the row is gone, the run was cancelled, or its deadline passed. That is the only way to stop a run — a bulk job holds no token anything can revoke, so cancellation is a flag the iteration reads.

gated! is distinct from cancelled!: nobody asked for it to stop.

Gates

app/models/gate.rb. A gate is a row keyed by an iterator name, optionally scoped to one record.

Gate.decide(key: "sync", reference: resource, enabled: true, live: true)
# => Decision(enabled:, live:)

Most specific wins: a gate on the reference, then the key-wide gate, then what the iterator declared. Absence means the iterator’s own default, not “off”.

live: false is a dry run — the iteration walks and counts but does nothing.

URIS_ITERATORS_DISABLED shuts every iterator everywhere. It is an operator switch; per-tenant and per-resource switches are gates rows.

Failure policy

The error class decides. Both SyncResourceJob and ExportItemsJob carry:

retry_on Resource::Failed, wait: :polynomially_longer, attempts: 5 do |job, error|
  job.fail_run(error)
end

and AnalyzeItemJob additionally discard_on Analyzer::Failed.

	Is	Policy
`Analyzer::Failed`	a file that cannot be read	discarded
`Resource::Failed`	a resource that cannot be reached	retried, five attempts

Adapters translate vendor errors, so nothing above Resource names an SDK.

test/jobs/failure_policy_test.rb asserts both halves; test/jobs/gated_iteration_test.rb covers gates.