The quiet ways an integration fails

A loud failure is cheap. The connection drops, an alert fires and someone fixes it before lunch. The quiet failure costs money. Every message returns a success code, every dashboard is green and the two systems drift apart.

You find it weeks later, far from the cause. A customer disputes an invoice. Month-end does not balance. By then the bad records have piled up and someone has to repair them by hand.

Values that change on the way across

Truncated fields. Suppose the source allows 100 characters for a company name and the target allows 40. The target accepts the record and cuts the rest. Weeks later a legal name on an invoice is wrong, or two customers look identical.

Shifted time zones. One side sends local time with no offset. The other reads it as UTC. Every timestamp moves by hours, and a payment made late on the last day of the month lands in the next one. It surfaces as a cut-off dispute or interest that is off by a day.

Rounding in money. Suppose the source stores amounts to four decimal places and the target stores two. Each line rounds by a fraction of a cent. Invoice totals differ by a few cents, the ledger stops balancing and finance spends days hunting the difference.

The checks that catch these on the day:

• Compare field lengths on both sides before go-live. Reject or flag any value that is too long. Never cut it silently.
• Send every timestamp with an explicit offset. Test with a record created one minute before midnight on the last day of a month.
• Agree one rounding rule and the one place it is applied. Compare the day's total on both sides, to the cent.

Records that arrive twice or only in part

Duplicates from retries. The sender posts a payment, the reply times out and the sender retries. The first request had succeeded, so the receiver now holds two payments. With no idempotency key, it cannot tell a retry from a new request. You find out when a customer is charged twice.

Partial batches. Suppose a file of 500 records goes in. The receiver accepts 497, skips three with bad data and returns success for the batch. The three rejects sit in a log nobody reads. Weeks later three orders have not shipped and nobody knows why.

• Attach a unique key to every write. The receiver stores it and returns the original result if the key comes again.
• Count records sent, accepted and rejected for every batch. Any gap opens a ticket the same day.
• Decide in writing whether a batch is all or nothing. If it is not, send the rejects somewhere a person will look.

Messages that never arrive

The missing file. A nightly file from a bank or a supplier does not come. Nothing fails, because nothing ran, so no error alert fires. The gap shows up days later as unmatched payments or stock figures that stopped moving.

Rate limits at month end. The integration stays inside the other side's limit for most of the month. On the last day volume jumps and the other side refuses requests. If the adapter logs the refusal and moves on, those records are gone. You see a few missing invoices in the busiest week.

• Set a heartbeat for every scheduled transfer. If the file has not arrived by the agreed time, a named person gets an alert.
• Queue and retry after a rate-limit refusal. Never skip.
• Test at month-end volume, not at average volume.

The other side changes without telling you

Changed enums. The other system adds an order status, renames a code or sends a value in a different case. The worst adapters map unknown values to a default, so a new status such as "on hold" quietly becomes "open". Reports drift until the figures stop making sense.

Rule of thumb: an unknown value is an error, not a default. Stop the record, keep it and tell someone.

The check on the day is strict validation. List the allowed values for every coded field and send anything else to a queue for review. Subscribe to the other side's change notices, and re-read their interface documentation on a schedule.

Defences that cover all of them

Each check above handles one failure. These seven habits cover the whole class, including failures you have not met yet.

1. A field-level mapping document. For every field that crosses the boundary: source, target, type, length, conversion rule and what happens when the value is missing or malformed.
2. Idempotent writes. Any message can be sent twice without creating two records.
3. Reconciliation on both sides. Record counts and control totals, compared daily.
4. A dead-letter queue. A message that cannot be processed is kept, with the reason, until a person deals with it.
5. Alerts that name a person. An alert sent to a shared inbox belongs to nobody.
6. A log of every message that crosses the boundary. Without it you cannot prove which side is wrong.
7. A heartbeat for scheduled work. Alert on silence, not only on errors.

Reconciliation finds what the other checks miss

If you can afford only one defence, choose reconciliation. It does not care how the failure happened, only whether both sides agree. Count the records and sum the money on each side, every day, for the same period. A difference of one record or one cent is a finding.

Warning: a green dashboard proves that messages moved. It does not prove that the data is right.

Keep the daily results. When a dispute arrives months later, they show the last day the two systems agreed.

What to ask for before go-live

Whoever builds your integration, ask to see these before you accept it:

• The mapping document, signed by the data owners on each side.
• A test that sends the same message twice and shows one record.
• Yesterday's reconciliation report from the staging environment.
• The name of the person each alert reaches, and proof that a test alert arrived.

Stack9 writes a field-level mapping document before the first adapter. Our adapters retry on failure, queue when the other side is down, alert a named person and log every message that crosses the boundary.