Money Deducted, Inventory Not: How Distributed Transactions Clean Up the Mess

Imagine an order-placement endpoint. The order service first inserts an order into its own database, then calls the inventory service to deduct stock. That call times out. The order service has no idea whether the other side actually deducted anything — all it sees is a timeout. Retry, and it might deduct twice. Don’t retry, and it might never deduct at all, while the order is already committed.

A local transaction can only manage its own database. Two tables in the same database can be wrapped in one BEGIN ... COMMIT. Switch to two services and two databases, and no ready-made transaction can wrap both sides together.

Most of the approaches in the sections below accept a brief inconsistency and rely on retries and compensation to pull things back into line. That goal is called eventual consistency. Let’s lay the problem out first.

sequenceDiagram
    participant O as Order Service
    participant S as Inventory Service
    O->>O: insert order in local transaction
    O->>S: call to deduct inventory
    S--xO: timeout
    Note over O,S: order committed, inventory not deducted

The most direct idea is to copy the database’s two-phase commit. The coordinator first asks every participant to prepare; if all nod, it commits; if one refuses, it rolls everything back. XA at the database layer is exactly this protocol. Its trouble starts after prepare, when the data rows stay locked until the final commit or rollback releases them. If the coordinator crashes at that point, the participants keep their hands raised forever. The price of this path is a long transaction stretching across the network, which is painful under high concurrency.

Split One Big Transaction into Three Business Actions (TCC)

TCC takes a different angle: instead of asking the database to lock resources for it, the business reserves them itself. One transaction splits into three phases — Try, Confirm, Cancel.

Try checks and reserves. For a transfer of 100, Try only freezes the 100 in account A; the balance stays put for now. If the freeze succeeds, the later steps for that money will almost certainly go through.

Confirm confirms. It actually deducts the frozen 100, and the transaction ends.

Cancel cancels. It unfreezes the 100 back into the balance.

Take freezing inventory for an order. Try subtracts 1 from available inventory and adds 1 to frozen inventory. Confirm subtracts 1 from frozen inventory, and the goods are effectively shipped out. Cancel subtracts 1 from frozen inventory and adds 1 back to available inventory, and the goods return to the shelf.

sequenceDiagram
    participant TC as Transaction Coordinator
    participant Order as Order Service
    participant Stock as Inventory Service

    TC->>Order: Try reserve
    TC->>Stock: Try reserve
    Order-->>TC: success
    Stock-->>TC: success
    alt all succeeded
        TC->>Order: Confirm
        TC->>Stock: Confirm
    else one step failed
        TC->>Order: Cancel
        TC->>Stock: Cancel
    end

The reserve step settles the question of “can this be done?” up front, so the later Confirm almost never fails. What holds the resource is a business field, and the database’s row lock no longer plays a part at this layer. That is also where this approach intrudes on the business: every participant has to implement all three interfaces, Try, Confirm, and Cancel.

Three pitfalls deserve their own section; they cause trouble more easily than the interfaces themselves.

“Hanging” is the nastiest one. Cancel has already finished, and only then does the late Try freeze the resource. Once frozen, nobody sends another Confirm or Cancel, so the resource hangs there forever.

Write the Message and the Business into the Same Local Transaction (Local Message Table)

Another approach is more plain-spoken. It finishes the order first, records “we still owe inventory a notification” in a local ledger, and pays it back slowly afterward.

Step one: inside the order service’s local transaction, do two things — insert the order, and insert a pending message into a local message table. Both writes live in the same database and the same transaction, so they either both succeed or both fail.

Step two: an independent polling job reads the unsent messages from that table and sends them downstream one by one, or publishes them to an MQ.

Step three: once the downstream processes a message successfully, mark that row as sent.

Retries are covered by the polling. If a send fails, it tries again next time until it succeeds. As a result, the downstream may receive duplicate messages. Put a business unique ID in the message, and the downstream blocks the second copy with a unique index or a deduplication table.

sequenceDiagram
    participant Biz as Order Service
    participant DB as Local Database
    participant Job as Polling Job
    participant MQ as Message Queue
    participant Down as Downstream Service

    Biz->>DB: insert order + pending message in one transaction
    DB-->>Biz: commit
    Job->>DB: read unsent messages
    Job->>MQ: publish
    Job->>DB: mark as sent
    MQ->>Down: deliver
    Down-->>MQ: processed successfully (idempotent)

The upside: it depends on no special middleware, an ordinary database is enough, and you fully control the retry logic. The costs come in three places. The message table and the business table sit in the same database, so the two concerns are tied together. Polling has an interval, so the downstream usually receives the message a few hundred milliseconds to a few seconds later. And the message table grows with business volume, so you need to clean up the sent history on a regular basis.

Let the MQ Hold the Message (MQ Transactional Message)

Writing the local message table yourself is a bit tedious. MQs like RocketMQ build the same job into the protocol, under the name transactional message.

The producer first sends a half message to the broker. The broker stores it, and consumers cannot see it.

Only after the broker confirms receipt does the producer run the local transaction, such as inserting the order.

If the local transaction succeeds, the producer sends commit to the broker, and the half message becomes visible to consumers. If the local transaction fails, it sends rollback, and the message is discarded.

If the broker waits too long without a commit or rollback, it turns around and checks back with the producer, asking whether that local transaction actually succeeded, then decides to commit or discard. This step exists for the case where the producer crashes.

sequenceDiagram
    participant P as Producer
    participant MQ as Broker
    participant DB as Local Database
    participant C as Consumer

    P->>MQ: send half message
    MQ-->>P: stored, invisible to consumers
    P->>DB: run local transaction
    alt local transaction succeeded
        P->>MQ: commit
        MQ->>C: message visible, delivered
    else local transaction failed
        P->>MQ: rollback
        MQ->>MQ: discard message
    end
    Note over MQ,P: producer crashes, broker checks back on the local transaction
    MQ->>P: check back transaction status
    P-->>MQ: success or failure

The half message being invisible to consumers, plus the check-back — together these two things bind “the local transaction” and “the message being sent” into a single whole. The business code no longer needs to build its own message table and polling.

It ties you to a specific MQ. RocketMQ supports this semantics; Kafka’s transactions are a different matter, mainly solving atomic reads and writes inside Kafka and multi-partition commits, and you still have to cover the cross-external-database part yourself. The check-back interface is yours to implement too. And the downstream still needs idempotency.

How to Choose

None of the three approaches escapes idempotency. TCC’s Confirm and Cancel get called repeatedly, and consumers in the message approaches receive duplicate messages. Without idempotency, the more complete your coordination layer is, the more neatly duplicates and mix-ups show up.