A user launches an investigation on Arbiter: a query, a set of topics, a set of platforms, and a date range. Arbiter goes out to each platform, collects the public posts that match those topics, indexes them, and runs analysis over what came back. The user then works through that analysis by asking an agent questions about it. Every platform is a separate API with its own rate limits, so collection takes between five and forty-five minutes. The platforms are the ones setting that pace, not us. Between the first post arriving and the last, we could watch the pipeline move but not the investigation itself, and the previous post explained why. This one is about what we changed.
Arbiter began with two platforms and a dozen users. It now runs across ten platforms for over 250 users who have run more than 450 investigations, and a single investigation can reach four platforms over a range of two weeks to six months or more. That growth turned two properties we could live without early on into requirements. First, the system has to be observable, meaning it can say which stage an investigation is at, how many posts each platform returned, and which topics returned no posts, while the run is still ongoing. Second, it has to be fault-tolerant, meaning that when collection from one platform fails midway through, for reasons outside our control, the user does not lose their progress. At two weeks a failed platform is a cheap retry; at six months it is months of coverage to collect again.
Those two requirements drove the redesign, and it landed in two places: the workflow orchestration and the application state.
One job at a time, per platform
One instance per platform handled that platform’s collection, working through every topic across the whole date range in sequence. That was enough when an investigation meant two platforms and a short range. Three things then grew together: the platforms we support, the number of topics per investigation, and the time span each investigation covers. The work inside one instance is the product of the three, so it became the bottleneck, and no amount of parallelism further down the pipeline makes up for a front door that only admits one job at a time.
The standard answer is to break that sequence into many small units, one topic for one week. What kept us from it was that we could not track the pieces. Once units are independent, each one can fail on its own, and our logging has to surface the topic, the platform, the week, and the time of the failures. The tracking we had was built for the unit we had, one instance per platform, so fanning out would have multiplied potentially blind failures without giving us the means to record and fix them.
That missing tracking was not only our problem. One question dominated the user interviews: how do I know whether collection has finished. Three requirements followed:
- One owner per investigation that could report what stage it had reached, instead of leaving us to infer it from counts in a status table.
- Fan-out down to a unit small enough to retry cheaply, without putting more concurrent requests at a platform than that platform allows.
- Failures that stop at their own boundary. One topic failing in one week should not cost that platform its other weeks, and the user should be told which topic-weeks failed.
Moving workflow orchestration to Trigger
Writing our own coordination layer was a reasonable trade when the workflow had one shape. The cost of it grew with everything else: every platform we added, every stage we inserted, and every failure mode we learned about was maintenance we owned, and it was costing us engineering time rather than money. We looked at AWS Step Functions, Temporal, Inngest, and Trigger.dev. All four can run a workflow of this shape, with fan-out, retries, and durable state, so capability was not the deciding factor. Two things decided it: how much infrastructure we would still be operating ourselves afterwards, and how quickly an engineer could change a workflow and watch it run.
On AWS a change would have to be deployed before it could be tested at all, because parallel functions behind queues do not run on a laptop, and at its worst a single change would take thirty to forty minutes to reach somewhere we could watch it work.
Measured that way, Step Functions would have kept us authoring state machines as JSON and deploying to test them. Temporal’s workers run in your own environment whether or not you use their cloud, and that is the operational burden we were trying to remove rather than relocate. Trigger.dev came out ahead on both: workflows are TypeScript files in the same repository as the product, and they run on Trigger’s infrastructure, so there are no workers of ours to keep alive, and no one of ours on call for them.
Breaking collection into smaller, retryable units
Collection is now one workflow run, from launch to finished analysis:
Retrieval runs first, against our own index and not the platforms. Checking coverage before collecting is not new, and the legacy system did it too. What changed is that the result is now part of the run’s state instead of an internal detail that only we could see. Within the first few minutes, before any collection starts, a user sees how much coverage we already hold for each platform and each week. Those counts keep filling in as collection runs, including the topics that returned nothing, and why. They can still change the scope at that point, dropping a platform, widening the date range, or rewriting a topic.
Collection then runs only for the weeks that came back short, with topics collected in parallel instead of one after another. Each platform caps how many of those run at once, which is what keeps the new parallelism from turning straight into rate-limit errors. Smaller units are also what make the run fault-tolerant. A retry re-runs one topic for one week rather than a whole platform, and because raw collector output is written to durable storage as it arrives, it collects only the gaps instead of paying a provider twice.
The last step is one child run per platform, covering its own collection, analysis, and reporting, with the parent waiting on all of them together. That is the change a user notices most directly. A slow platform now delays only its own results, so someone can start reading what has landed, and asking the agent about it, while the rest of the run continues. Usable results now reach a user within five to twenty minutes, even for investigations covering roughly twice as many topics over twice the time range. In the legacy system a single analysis step would run over everything once collection was declared finished, so the slowest platform set the pace for the whole investigation.
That is the answer to the problem the previous post described: analysis running on partial data, with nothing in the system reporting it.
Moving application state to Convex
The workflow layer gave us the fault tolerance. The other half was getting that state in front of a user, and DynamoDB, which had served us well at the earlier scale, did not serve this one.
An investigation is relational, with platforms, topics, posts, and analyses nested under it, and almost every read either walks those relationships or has to update live as the run progresses. DynamoDB is fast when you know your access patterns in advance, and serving a new kind of read later usually means adding an index for it. That is a fair trade for a product whose read patterns have settled. Ours had not, because each new feature tended to ask something of an investigation that no earlier feature had. Size was the second problem: roughly a quarter of the payloads an investigation produces exceed DynamoDB’s 400 KB per-item limit, and the largest reach tens of megabytes, so we had written our own scheme for splitting them across items and putting them back together on read.
We moved to Convex, a database that keeps its schema and queries in TypeScript and pushes changes straight to subscribed clients. Convex has a document limit of its own, so the point was never that the limit went away. It was that we no longer had to design our own way around it, and that showing a user where their investigation stands became a query rather than a system we had to build and keep running.
The stack, before and after
This redesign moved a fair amount of code, but what it really moved was responsibility. Almost every component below used to be something we operated ourselves, inside one AWS account and one CloudFormation stack:
| Component | Before | Now | Note |
|---|---|---|---|
| collectors | ECS / Fargate | Trigger | our workflow code |
| processing stages | Lambda | Trigger | our workflow code |
| work tracking | SQS + status tables | Trigger | nothing of ours |
| application state | DynamoDB | Convex | our schema |
| progress to browser | Streams + websocket | Convex | subscriptions |
| frontend + deploys | separate repo + CDK | Vercel | one repo |
| reranking, classifying | third-party APIs | Modal + AWS | self-hosted |
| search | Elasticsearch | Elasticsearch | still ours |
| raw data + artifacts | S3 | S3 | still ours |
Almost every row follows the same rule. Infrastructure that needed one of our engineers watching it went to managed services, and the parts that decide what Arbiter is, the workflow definitions, the schema, the search index, and the raw data, stayed with us. Deployment is where that shows up daily. The frontend, the workflows, and the database schema are now one TypeScript repository, with the app deploying to Vercel and the workflows to Trigger, so a change that crosses the boundary between them is a type error, where it used to be three separate edits in two languages with nothing checking across them.
What we run ourselves, and what we do not
We decided each of those moves the same way, by asking whether running the thing ourselves would tie up an engineer who should be doing something else. Writing our own orchestration would have, and we had just spent a long time learning what that costs. Handing it to Trigger put that time back into the work only we can do, which is prototyping features and benchmarking what the system actually returns. What we accepted in exchange is a ceiling we cannot raise ourselves. Concurrent-run limits are ours to tune but not ours to remove, and they are what we expect to hit first when several long investigations start together, and where we expect to ask this question again.
There is one part of this we will not hand to anyone. A user’s trust sits on how good the evidence behind an investigation is, and on how plainly the system reports what it did not find. If we are careless there, none of the rest of this post matters.
Being careful there means knowing what a user asked for, what came back, and what they did with it. Knowing that the system is healthy tells us almost none of that. Watching infrastructure and watching how a product gets used are different jobs, and we ended up building for them separately. That is what the next post is about.
If you are working on large-scale data collection, distributed processing, search infrastructure, NLP systems, social intelligence, narrative analysis, or information ecosystems, we would like to compare notes.