"We need everything in real-time" is a common request. But real-time isn't always better—it's more complex, often more expensive, and sometimes completely unnecessary. Batch processing, the old workhorse of enterprise integration, remains the right choice for many scenarios.
This guide helps you decide when each approach makes sense and how to design integrations that match your actual requirements rather than perceived needs.
Defining the Terms
Real-time Integration
Data moves between systems immediately (or within seconds) of an event occurring. When a customer places an order on your website, your warehouse system knows about it within moments.
True real-time typically uses webhooks or event streaming—the source system pushes data as events happen rather than waiting to be asked.
Near Real-time
Data syncs frequently—every few minutes—but not instantaneously. Often implemented by polling (checking for changes on a schedule) rather than push-based events.
Batch Processing
Data accumulates and moves in scheduled batches—hourly, nightly, or weekly. All changes since the last batch are processed together.
When Real-time Makes Sense
Real-time Is Justified When
- Users expect immediate feedback: Customer-facing systems where delays are visible and frustrating
- Data has a short shelf life: Stock levels, pricing changes, availability status
- Downstream processes depend on it: Warehouse pick operations waiting for orders
- Regulatory or contractual requirements: Some industries require immediate reporting
- Competitive advantage: Faster response than competitors matters
Real-time Examples
E-commerce inventory: When stock is low, real-time sync prevents overselling. A customer shouldn't be able to buy the last unit while another customer's order is in a batch queue.
Fraud detection: Checking transactions against fraud rules must happen before the transaction completes—batching is too late.
Customer support context: When a customer calls, the agent needs to see their recent orders immediately, not from last night's batch.
When Batch Is Better
Batch Processing Works Well When
- Data doesn't change value with time: Historical records, analytics, reporting
- Processing is resource-intensive: Complex transformations, aggregations, validations
- Systems aren't always available: Legacy systems with maintenance windows
- Errors need human review: Data quality issues require investigation before committing
- Cost is a constraint: Batch is usually cheaper than real-time infrastructure
Batch Examples
Payroll processing: Salary calculations need complete data. Processing in real-time would mean incomplete calculations as timesheet entries arrive throughout the period.
Data warehouse loading: Analytics queries run on yesterday's complete data, not constantly changing current data.
Month-end reporting: Financial close processes need all transactions finalised before processing—real-time would create moving targets.
Comparison
| Factor | Real-time | Batch |
|---|---|---|
| Data freshness | Seconds/minutes | Hours/days |
| Complexity | Higher—event handling, error recovery | Lower—straightforward ETL |
| Infrastructure cost | Higher—always-on, scalable | Lower—runs periodically |
| Error handling | Must handle immediately | Can review before retry |
| System coupling | Tighter (with sync calls) | Looser (files/staging) |
| Testing difficulty | Higher—timing/sequencing issues | Lower—deterministic |
Hybrid Approaches
Most real-world integrations use a mix. You might process orders in real-time but sync product catalog changes in nightly batches. Some patterns that combine approaches:
Event-Triggered Batches
A real-time event starts a batch process. An "end of day" event triggers nightly processing. A "file received" event starts a batch import.
Micro-batching
Very frequent small batches—every 5 minutes—provide near real-time freshness with batch-style processing. Simpler than full event streaming, fresher than traditional batches.
Real-time for Critical, Batch for Rest
Prioritise real-time for the 20% of data that matters most. Order status updates might be real-time while customer preference changes sync overnight.
Making the Decision
Ask the Right Questions
- What breaks if data is 1 hour old? 1 day old? If nothing critical breaks, batch might be fine.
- Who is impacted by delays? Customers notice more than back-office staff.
- What's the cost of complexity? Real-time requires more sophisticated error handling, monitoring, and recovery.
- What are the source system's capabilities? Can it push events, or only respond to polling?
- What's the data volume pattern? Steady flow suits real-time. Spikes might overwhelm it.
Principle: Start with batch unless there's a clear business reason for real-time. It's easier to add real-time capabilities later than to simplify an overly complex real-time architecture.
Summary
Real-time integration sounds impressive, but batch processing often provides the best balance of simplicity, cost, and reliability. The right choice depends on your specific requirements—how fresh data needs to be, what systems can support, and what complexity your team can manage.
Don't default to real-time because it seems modern. Default to the simplest approach that meets your actual requirements.