Cora: letting the model hold the pen.

Throughout my collaboration with Kieran on Cora I managed to transform my practice to be not only AI-friendly but AI-synergetic. It is an honour and a tremendous challenge to work with one of the leading AI adopters in the SaaS space — Every — and a great test of one’s abilities just to keep up with things. But it remains incredibly rewarding.

The collaboration started on an incredible premise, with Kieran asking:

Can we try working in a way where it’s the LLM that writes the software, and we just steward, guide (and flog!) it sometimes?

At the time it sounded preposterous — I want to hand-manoeuvre my precious code, chisel it using my precious byte-dremel wheels, to the very last sigil of perfection. But I took the plunge, and it has been the best experience I’ve had.

And along the way, I helped Cora get to new heights.

Transactional workflows, at scale

A lot of what Cora does comes down to transactional workflows, applied at a rather extreme scale — on the order of millions of workflows per day. Cora was the first adopter of geneva_drive with outstanding results. Every workflow that gets run performs anywhere from just a couple to a dozen steps, with quite sophisticated retry semantics.

Transient errors are bread and butter with this workload, so I had to design some refined error handling for Cora. Most of that work has been instrumental for making geneva_drive’s exception handling policies what they are now. As a teaser, here is how refined we had to be — and this is not even the entire unit:

# Ruby / stdlib network errors
on_exception Net::ReadTimeout, Net::OpenTimeout, **TRANSIENT_POLICY
on_exception OpenSSL::SSL::SSLError, SocketError, Faraday::SSLError, **TRANSIENT_POLICY
on_exception Errno::ECONNRESET, Errno::ETIMEDOUT, Errno::ECONNREFUSED, **TRANSIENT_POLICY

# Faraday (used by RubyLLM, Google API client, etc.)
on_exception Faraday::TimeoutError, Faraday::ConnectionFailed, **TRANSIENT_POLICY

# Down gem (used for fetching remote resources like images/attachments)
on_exception "Down::ConnectionError", **TRANSIENT_POLICY

# RubyLLM transient errors. RateLimitError is included here so that
# upstream quota exhaustion (e.g. Vertex AI RESOURCE_EXHAUSTED) backs off
# and reattempts instead of surfacing the exception on every call. Steps
# that want a shorter/jittered wait still rescue RubyLLM::RateLimitError
# inline (see EmailProcessingWorkflow#classify_email, #generate_summary).
on_exception RubyLLM::OverloadedError, RubyLLM::ServerError, RubyLLM::ServiceUnavailableError,
  RubyLLM::RateLimitError,
  **TRANSIENT_POLICY

# ApplicationClient (ClaudeClient) transient errors
on_exception ApplicationClient::ServerError, **TRANSIENT_POLICY

# Google API transient errors
on_exception OAuth2::Error, Signet::AuthorizationError, Google::Apis::ServerError, Google::Apis::TransmissionError,
  **TRANSIENT_POLICY

# Google rate limit errors (HTTP 429 RateLimitError, semaphore timeouts, quota throttle timeouts,
# and HTTP 403 quota-exceeded ClientError). Uses a custom === matcher so that only quota-exceeded
# ClientErrors are retried, not all ClientErrors (e.g. 404 Not Found).
on_exception GmailToolkit::RateLimitErrorMatcher, **TRANSIENT_POLICY

At the time of writing, Cora has executed over a quarter of a billion geneva_drive workflows, and here’s to many more!

Your own private Gmail

My experience tells me that if most of your product value is integration with one or more vendors, you are likely to find the vendor’s tooling lacking. Moreover, if you need to go to a vendor for every tiny little thing to try, your work is going to turn into hell. Want to test your email sorting with 2000 messages? Good luck doing that with your test account. Want to figure out how you display a mailbox that has 126 custom labels, with names conflicting with your product’s own? Slim chance.

So — for Cora I have built a full Gmail emulation layer, including a fake UI. This layer allows us to hydrate a fake Gmail account with as many email messages as we like, to set up any labels we may want, and to then access it using the same Google-issued SDK calls. When we set up a development environment, we don’t even have to connect a Gmail account — a test account gets pre-configured for you with a sufficiently large corpus of messages ready to go.

And, needless to say, we even generate correct message and label IDs which conform to the Gmail formats.

But a good emulation layer does not end here! Since we also perform searches against GMail, I have built a full parser for the GMail search syntax, which converts queries into SQL. In our case, the queries are then performed against a SQLite database, one per fake account.

One does not simply observe a rate limit

A challenge of working with Gmail is the whole number of rate limits and restrictions that it imposes on the callers. They are not quite documented, they are burst-sensitive, and they get applied in layers. A lot of them you won’t even perceive until you hit them — and once you do, you hit them hard.

To deal with this I had to design a comprehensive set of wrappers that keep track of all SDK calls Cora makes to Gmail, for every specific user, and get the requisite throttles applied automatically. From the point of view of the calling code, the modules it is dealing with are the same Gmail SDK objects with exactly the same methods.

Fun fact: doing Gmail at scale requires no less than 8 (you read that right, eight) layers of wrappers on top of the stock SDK.

You can just write an autoscaler

Cora rides Render and does so big time. Again: no Kubernetes here, and we don’t want any. However, we do have varying workloads — and we needed an autoscaling solution to accommodate those workloads.

To that end, I wrote skaler — a highly sophisticated parametric autoscaler, which supports Render but is easily extensible to other cloud platforms. skaler has configurable surge quotas, has back-off, and is based off a tried-and-tested beating heart in the form of a PID controller.

Unlike nearly all autoscalers on the market including Kubernetes HPA, it also intelligently observes resource limits, across an unlimited number of resource bands:

• It will not scale you beyond your available Postgres connections (including your deploy surge)
• It will not scale you beyond your available Redis connections
• It will observe any and all custom constraints you may add

And because an autoscaler is a signal processor and needs to have a solid record of the feedback coming from the system, it carefully persists its work tables into Redis.

skaler has been autoscaling Cora background job clusters for a long time now, with the number of instances varying from 1 to 30.

And the best part: it lives inside the Cora Rails app — so no extra services to manage, no background processes to monitor and no YAML to write. Just an ActiveJob.

Slop reduction directive

Kieran is bullish on LLMs, and I am grateful to learn from him. However, my essential role within Cora is also slop containment — I examine large, sweeping changes done to the codebase, and subject them to stringent evaluations:

• What are the race conditions that will occur here?
• Can we do the same, but with half of the code?
• Could there be a simpler architecture we could apply?

This way of working is fine and allows us to increase the value of code reviews dramatically. Instead of nitpicking over insignificant details to increase influence, we both can paint in broad strokes, and deal with the consequences as they come along. And since we know that most of the architecture is an LLM extrusion, there is little worry that one of us will be upset at a rework of “their baby”.

This creates an unprecedented and very powerful work dynamic. Nothing is sacred, little is touchy-feely, and we can execute at speed, with the best results. Your LLM went too sloppy? Nobody will regret taking a cleaver to its output, because it’s just that — an intermediate artifact in the fabrication of software.