Real-time vs Batch Integration
When to use real-time integration and when batch processing is the better choice. A practical guide to choosing the right integration approach.
When to use real-time integration and when batch processing is the better choice. A practical guide to choosing the right integration approach.
Data moves between systems immediately (or within seconds) of an event occurring. When a customer places an order on your website, your warehouse system knows about it within moments.
True real-time typically uses webhooks or event streaming - the source system pushes data as events happen rather than waiting to be asked.
Data syncs frequently - every few minutes - but not instantaneously. Often implemented by polling (checking for changes on a schedule) rather than push-based events.
Data accumulates and moves in scheduled batches - hourly, nightly, or weekly. All changes since the last batch are processed together.
E-commerce inventory: When stock is low, real-time sync prevents overselling. A customer should not be able to buy the last unit while another customer's order is in a batch queue.
Fraud detection: Checking transactions against fraud rules must happen before the transaction completes - batching is too late.
Customer support context: When a customer calls, the agent needs to see their recent orders immediately, not from last night's batch.
Payroll processing: Salary calculations need complete data. Processing in real-time would mean incomplete calculations as timesheet entries arrive throughout the period.
Data warehouse loading: Analytics queries run on yesterday's complete data, not constantly changing current data.
Month-end reporting: Financial close processes need all transactions finalised before processing - real-time would create moving targets.
| Factor | Real-time | Batch |
|---|---|---|
| Data freshness | Seconds/minutes | Hours/days |
| Complexity | Higher - event handling, error recovery | Lower - straightforward ETL |
| Infrastructure cost | Higher - always-on, scalable | Lower - runs periodically |
| Error handling | Must handle immediately | Can review before retry |
| System coupling | Tighter (with sync calls) | Looser (files/staging) |
| Testing difficulty | Higher - timing/sequencing issues | Lower - deterministic |
Most real-world integrations use a mix. You might process orders in real-time but sync product catalog changes in nightly batches. Some patterns that combine approaches:
A real-time event starts a batch process. An "end of day" event triggers nightly processing. A "file received" event starts a batch import.
Very frequent small batches - every 5 minutes - provide near real-time freshness with batch-style processing. Simpler than full event streaming, fresher than traditional batches.
Prioritise real-time for the 20% of data that matters most. Order status updates might be real-time while customer preference changes sync overnight.
Principle: Start with batch unless there is a clear business reason for real-time. It's easier to add real-time capabilities later than to simplify an overly complex real-time architecture.
Real-time integration sounds impressive, but batch processing often provides the best balance of simplicity, cost, and reliability. The right choice depends on your specific requirements - how fresh data needs to be, what systems can support, and what complexity your team can manage.
Do not default to real-time because it seems modern. Default to the simplest approach that meets your actual requirements. If you want help making that call, our custom system integration services start with exactly this question.
Real-time integration moves data between systems within seconds of an event, usually through webhooks or event streaming where the source system pushes data as it happens. Batch integration lets data build up and moves it on a schedule, such as hourly, nightly or weekly, processing all changes since the last batch together. Near real-time sits between them, syncing every few minutes, often by polling.
Real-time integration is justified when users expect immediate feedback, data has a short shelf life, such as stock levels or pricing, downstream processes depend on it, or regulations require immediate reporting. Examples include online store inventory, where real-time sync prevents overselling the last unit, fraud checks that must happen before a transaction completes, and support staff who need a caller's recent orders immediately.
Batch integration is better when data does not lose value over time, such as reporting and analytics, processing is resource-intensive, systems have maintenance windows, errors need human review, or cost is a constraint. Payroll suits batch because salary calculations need complete data, and month-end reporting needs every transaction finalised. Batch is usually cheaper than real-time infrastructure and easier to test.
Micro-batching runs very frequent small batches, for example every 5 minutes, to give near real-time freshness with batch-style processing. It is simpler than full event streaming and fresher than traditional batches. Micro-batching is one of several hybrid patterns, alongside event-triggered batches, where an event such as end of day starts a batch, and real-time for the most critical data with batch for the rest.
Start with batch integration unless there is a clear business reason for real-time, because adding real-time later is easier than simplifying an overly complex real-time design. To decide, ask what breaks if the data is an hour or a day old, who is affected by delays, what the extra complexity costs, and whether the source system can push events or only respond to polling.
Ask the author
Ask it here and it comes straight to the founder. No sales call, no obligation, and a real answer even if the answer is that you do not need us.
Kasun Wijayamanna
Founder, replies within one business day
Tell us what you are working on. We will come back with a practical recommendation and clear next steps.
Built here. Your data stays here.
Thanks for reaching out. We will get back to you within one business day.
See what else we do