BACORE / SYSTEM DOCUMENTATION 03 / 04
SaaS Release Lab
Find the cause. Fix the bug. Check the release.
SaaS Release Lab follows a bug from a repeatable example to its cause, a code fix and checks on the deployed version.
Reproduction • Root-cause analysis • Correction • Release verification
FAILURE INVESTIGATION • EXAMPLE
A retry should not duplicate the work.
REPRODUCTION CASE
Deliver the same event twice
Failure condition: duplicate processing
CORRECTION STRATEGY
Persistent idempotency key
Check the same case before and after the change
Reproduce → Diagnose → Fix → Verify
Example investigation: duplicate event delivery, a persistent idempotency key and a repeatable regression check.
SOFTWARE RELIABILITY / HANDS-ON DELIVERY
01 / REPRODUCE
Repeatable failure
02 / DIAGNOSE
Evidence & root cause
03 / CORRECT
Code & regression test
04 / RELEASE
Verification & rollback
THE PROBLEM
“It works here” is an incomplete release decision.
A defect can disappear during manual testing and return under retries, concurrency or a different environment. Release work needs a repeatable case, a clear explanation of the cause and checks that exercise the conditions that originally failed.
EXAMPLE SCENARIO
The lab uses an intentionally introduced duplicate-event defect in a small SaaS application. Replaying the same inputs before and after a correction connects the release decision to a repeatable check.
SYSTEM WORKFLOW
How the system works.
01
Build a minimal reproduction
Capture the request, event sequence, environment and expected behaviour. Reduce the failure to a case another engineer can run.
02
Trace the cause
Follow the request across API, worker and persistence boundaries. Use logs and targeted instrumentation to distinguish symptoms from causes.
03
Correct the behaviour
Implement a focused change and add a regression test for the failing case. Check related behaviour, including concurrency where relevant.
04
Verify the deployment
Run staging checks, verify the deployed version and inspect health signals. Keep rollback conditions and recovery steps alongside the release.
ENGINEERING DECISIONS
Key technical decisions.
A controlled test environment
Versioned fixtures and repeatable environment setup keep the reproduction independent of changing external data. Each investigation records the inputs and configuration needed to rerun the failure.
Evidence attached to the change
Each investigation records the failing example, its cause, the code change and the before/after results. Another engineer can follow the reasoning and repeat the checks.
Release and rollback checks
CI checks, smoke tests and deployment markers support the release decision. Recovery conditions and rollback steps sit alongside the deployment procedure.
Technology: C# / .NET · PostgreSQL · Background workers · Automated tests · Containers · CI/CD
VERIFICATION
How to verify it.
Use these checks to verify normal operation, failure handling and recovery.
Duplicate delivery under concurrency
Deliver the same event repeatedly and concurrently. Verify that the correction preserves one business outcome.
A worker stops during processing
Stop a worker between processing steps. Restart it and verify that recovery preserves a consistent business outcome.
The deployed version fails a smoke test
Deploy a version that fails a smoke test. Verify that release checks flag the failure, then exercise the recovery procedure.
SYSTEM SCOPE
What is included.
The lab includes a small application, fixed test data, environment setup and a release procedure. Its example follows a duplicate-event bug from reproduction to verification. The same steps apply to interrupted workers, access-control bugs and API regressions.
Explore MetaSuite for another BACore software systems project. →
ARCHITECTURE + HANDS-ON IMPLEMENTATION
Bring the bug that keeps coming back.
BACore can investigate the failure, implement the correction and verify the deployment. A useful starting point is a reproducible case, access to the relevant code and a clear expected outcome.
EXPLORE MORE SYSTEMS