ENGINEERING / FIELD NOTES
Resilience by Design
Keep critical systems working through disruption.
Failure is a design input. Architecture, implementation and recovery testing determine how a system behaves when its dependencies stop behaving as expected.
Lessons from supporting an existing client team in a complex banking environment.
By Javier Sevilla
CTO · Software Architect · Lead Developer
DESIGN FOR THE FAILURE PATH
A dependency slows down
A customer journey is at risk.
↓ detect & contain
Bound the impact
Timeouts · circuit breakers · isolated capacity
↓ choose a safe response
Degrade or pause deliberately
Protect correctness. Explain the service state.
↓ restore & verify
Recover with evidence
Check service health and data integrity.
Conceptual failure path; not a client deployment diagram.
THE CONTEXT
Reinforcing the team already doing the work.
One of our clients operated a complex banking technology estate, with established engineering teams, internal standards and a wider resilience programme. We joined as additional architecture and engineering support, working alongside the people responsible for those systems.
Our contribution supported the practical work of translating resilience requirements into implementation and validation. The useful lesson was the connection between business priorities, application behaviour and evidence from testing.
This article explores those engineering principles. The client remains anonymous, and the diagrams illustrate concepts rather than its internal infrastructure.
01 / START WITH THE SERVICE
Define what must keep working.
Begin with a critical customer journey and map its dependencies: application code, data, identity, networks, external providers and operational procedures. Agree the acceptable disruption and the recovery priorities before selecting a deployment topology.
SERVICE LEVEL OBJECTIVE
SLO
A target for user-visible service quality over a defined measurement window, such as successful requests or latency.
RECOVERY TIME OBJECTIVE
RTO
The target time to restore a workload after disruption. Validate it through recovery exercises.
RECOVERY POINT OBJECTIVE
RPO
The maximum acceptable data loss, expressed as a period of time. Validate it against the recovered data.
These objectives are related, but they measure different things. Setting a timeout or an availability target does not establish a recovery capability. For banks, tolerance for disruption frames the impact that critical operations must be prepared to withstand.
02 / CONTAIN THE FAILURE
Choose patterns around the failure mode.
Resilience can be improved in existing systems. Addressing it early makes dependencies and trade-offs easier to reason about; existing platforms need incremental changes and validation.
Isolate capacity
Bulkheads separate resource pools so one overloaded dependency does not consume everything. Circuit breakers temporarily limit calls to a failing dependency. Combine them with deadlines and bounded retries; each solves a different part of the failure path.
Make asynchronous work recoverable
Queues can decouple execution, but introduce duplicate delivery, ordering and backlog concerns. Design idempotent handlers, bounded retries and dead-letter recovery. Sagas coordinate compensating actions where a distributed transaction needs them.
Degrade without misleading users
Keep useful, safe functionality available when a dependency fails. Cached or read-only views need clear freshness rules. A payment or ledger operation must not appear successful when its authoritative outcome is unknown.
Test the data recovery strategy
Choose active-passive or active-active operation according to consistency requirements, replication behaviour and cost. Exercise failover and failback, inspect replication lag, and verify data integrity. Multiple regions alone do not establish a recovery guarantee.
Tooling may include Istio or Resilience4j for traffic and application controls, Kafka or RabbitMQ for messaging, and Redis for caching. The pattern, configuration and application semantics matter more than the product name.
03 / PUT THE CHECKS INTO DELIVERY
Turn requirements into reviewable evidence.
Automated delivery checks make agreed standards repeatable. They support architecture review and operational judgement; static configuration checks alone cannot detect every single point of failure.
01 / INSPECT
Configuration
Check health probes, resource limits, deployment placement and timeout policies against the service requirements.
02 / EXERCISE
Load behaviour
Exercise representative traffic and saturation. Observe tail latency, error rates, queues and recovery after load drops.
03 / DISRUPT
Failure response
Inject a scoped dependency fault. Check the customer-visible response, retry behaviour and recovery path.
04 / REVIEW
Release evidence
Retain test results, policy versions, exceptions and owners. Use explicit criteria for release decisions.
Configuration → Load → Failure response → Release evidence
Policy engines such as Open Policy Agent can produce decision logs for auditing. Protect sensitive inputs and preserve traceability. A passing rule proves that particular check passed; it does not establish complete regulatory compliance.
References: Open Policy Agent: decision logs
04 / CONTROLLED FAILURE TESTING
Test a hypothesis. Measure the recovery.
Define the expected behaviour, target resources, blast radius, stop conditions and recovery procedure before an experiment. Start in a suitable test environment. Any production exercise needs a risk-based scope and the client’s operational approval.
SCENARIO A
Slow dependency
Inject latency or packet loss. Check deadlines, bounded retries and the intended fallback. Rerouting helps only when an alternative is available and safe.
SCENARIO B
Unavailable zone
Simulate the effects of zone loss on selected compute, networking and services. Observe capacity and failover; a test does not switch off a provider’s actual Availability Zone.
SCENARIO C
Database failover
Exercise loss of a primary and the supported recovery path. Measure restoration time and verify recovered data, consistency and client reconnection.
A successful experiment provides evidence for the tested conditions. It does not guarantee zero downtime under every failure. Automated stop conditions help limit an experiment; recovery still needs to be verified.
05 / GOVERNANCE THAT CONNECTS TO ENGINEERING
Keep ownership and evidence clear.
Work within the client’s operating model.
Engineering and operations own the implementation and day-to-day controls. Risk and compliance provide oversight and challenge. Independent internal audit supplies a separate assurance function. Business continuity specialists help connect recovery plans to critical operations.
Reusable templates, infrastructure modules and documented exceptions help teams apply standards consistently. Policy results and recovery exercises feed those conversations; they do not replace them.
REGULATORY CONTEXT
Be precise about the framework.
EU DORA has applied since 17 January 2025 to the financial entities within its scope. For covered financial entities identified as essential or important under NIS2, DORA is the sector-specific Union legal act under NIS2 Article 4. Applicability depends on the entity and jurisdiction.
The Basel Committee’s Principles for operational resilience (2021) are the relevant reference here. They describe a principles-based approach to sustaining critical operations through disruption.
06 / LEARN AND IMPROVE
Make each incident improve the next design.
Use incidents and exercises to expose assumptions. Review contributing factors, agree the changes, assign owners and verify that the changes work. Some findings belong in application code; others belong in architecture, monitoring or a recovery procedure.
01
Observe
Collect service behaviour and customer impact.
02
Understand
Review contributing factors and failed assumptions.
03
Improve
Change the relevant code, control or procedure.
04
Validate
Repeat the scenario and review the evidence.
Observe → Understand → Improve → Validate → Observe
Track measures with a defined baseline and period: service quality, time to recover, data recovery, repeat incidents and unresolved findings. Share reviewed patterns through documentation and engineering champions, with room for justified exceptions.
Resilience is an ongoing engineering capability.
The goal is to make failure behaviour explicit, reduce avoidable impact and recover with evidence. That work continues as the system and its dependencies change.
BACORE / ARCHITECTURE + IMPLEMENTATION
Add engineering capacity where you need it.
We can reinforce an existing team with a focused architecture review, implementation support or a broader development engagement. Start with the system, the constraints and the decisions you need to make.
Technical and regulatory references reviewed on 5 October 2026. Public sources explain the principles discussed; client context comes from the author’s account.