Platform
Eight microservices across three network isolated trust zones. Seven PostgreSQL databases, one per service, no shared tables. Kafka carrying payment events. Redis for sessions and idempotency.
CASE STUDY · FINTECH PAYMENTS · AWS
Payments infrastructure built for payday morning,
a partner bank outage, and a bad release.
Meridian Pay is a UK electronic money institution handling 1.2 million business accounts. I designed and built the platform end to end: eight services across three trust zones, infrastructure defined entirely in Terraform, and a deployment path where going backwards is as fast as going forwards.
Design, infrastructure, pipeline, deployment, and evidence capture. Two months. Built to production standards, documented as a public engineering reference.
8services, three network isolated zones
7databases, one per service
3critical CVEs blocked before merge
8001msisolation timeout from the blocked path
01THE CONSTRAINT
One bank account. 1.2 million customers.
That is not a mistake. It is how UK payments companies work when they do not hold a banking licence.
Meridian Pay is an electronic money institution. It moves money, but it is not a bank, which means it cannot hold customer funds itself. Every customer's money sits pooled in a single safeguarding account at a partner bank. One account, 1.2 million businesses.
What divides that pool is a ledger. A list. If the list is wrong, the company cannot prove to a regulator which pound belongs to which business, and that is not a defect you fix in the next sprint. It is an existential problem for the licence.
Every architectural decision on this platform traces back to that one constraint.
02SCOPE
What I built
Designed, built and deployed end to end, then destroyed the same day.
Infrastructure
Every resource defined in Terraform. VPC across three availability zones, EKS, RDS sized by criticality tier, ElastiCache, ECR with immutable tags. Remote state with locking.
Delivery
GitHub Actions with four security gates. GitOps deployment through ArgoCD. Helm charts generated from one template across all services. Rollback as a single command.
Operations
Prometheus and Grafana measuring customer experience, not server health. Alertmanager routing to Slack by severity. Load scenarios written as code, run inside the cluster.
03ARCHITECTURE
Three trust zones, enforced by network policy
The money zone, which holds payments and the ledger, sits behind a default deny network policy. Nothing reaches it unless a rule explicitly permits it. This is not a convention anyone can forget to follow, it is enforced at the network layer by the cluster.
The decision came from a real failure pattern: an analytics query run directly against a production ledger once degraded payment latency for an hour, because everything in the network could reach everything else.
EDGE ZONE
meridian-edgeTLS termination, WAF, rate limiting and JWT validation. The gateway holds no data of its own. A gateway that stores state becomes both a bottleneck and a single point of failure.
APPLICATION ZONE
meridian-appAuthentication, accounts, FX, fraud scoring and notifications. Fraud analysis learns about payments through Kafka events rather than querying the money zone directly, which is what keeps it outside that boundary.
MONEY ZONE
meridian-moneyPayment orchestration and the append only ledger. Default deny on both ingress and egress. Only the gateway may reach payment-service, and only payment-service may reach the ledger.
04DECISIONS
Five decisions and what they cost
Every one of these is written up as an architecture decision record in the repository, including the alternatives I rejected and the cases where I would now decide differently.
The ledger has no balance column
A balance is the sum of its entries, derived on read. Storing one creates the possibility of it disagreeing with the entries that produced it, and in a financial ledger that disagreement is the failure mode you cannot recover from.
Every movement writes two rows sharing a transaction ID, one debit and one credit, summing to zero. A transaction whose legs do not balance is rejected before it reaches the database. That is an integrity control, not error handling.
Amounts are stored as integers in minor units. Never floating point. Rounding error in a ledger is not a bug you fix later.
Cost: balance reads are more expensive. Mitigated with a cached projection in account-service that carries an explicit staleness marker, and is never used for anything that moves money.
A partner bank outage queues payments, it does not reject them
The partner bank had two outages in six months, half an hour each. The original behaviour was to reject the payment and tell the customer it failed.
That sounds reasonable until you think about what the customer does next. They submit again. Now there are two payments for one invoice and someone unpicks it by hand.
Payments are accepted and queued. The customer is never told something failed. When the bank recovers, the queue drains in order.
Cost: the platform now owns a queue it must monitor. An alert fires if the oldest queued payment exceeds ten minutes, because a queue nobody watches is worse than a rejection.
Databases are sized by what breaks when they are down
Payment and ledger run Multi AZ with seven day backups. Notification runs the smallest available instance, one day of backups, and no pod disruption budget at all.
That last part is deliberate. Notification is explicitly permitted to be unavailable during a node drain. A delayed confirmation email is an inconvenience. A delayed payment is not.
Criticality and load are different axes. Account-service is tier two by criticality but carries the highest request volume on the platform, at roughly six balance reads for every payment, so it scales hardest.
Cost: more infrastructure to reason about than one shared database. Bought in exchange for blast radius: a notification outage cannot degrade the money path.
I did not weaken IMDS hardening to satisfy a storage driver
The launch template restricts the instance metadata hop limit for IMDSv2 hardening, which stops containers reaching the metadata endpoint. The EBS CSI driver needed credentials from exactly that endpoint, and crash looped.
The easy fix was to raise the hop limit. I removed the dependency instead and moved Kafka to ephemeral storage, which was acceptable because broker data does not need to survive a restart in this environment.
Security controls interact with each other. When one blocks something, the question is whether you actually need the thing it is blocking.
Cost: Kafka loses its data on restart. For a production deployment the correct answer is a scoped IAM role for the driver through IRSA, which is recorded in the decision log as the next change.
Managed secrets with IRSA, not a self hosted Vault
Discovery found database passwords in a shared spreadsheet and hardcoded in service configuration, accessible to people who had left the company.
Terraform now generates a 32 character password per database and writes it to AWS Secrets Manager. No human sees or types it. Applications retrieve credentials at runtime through IAM Roles for Service Accounts, so nothing is stored in the cluster at all.
I considered HashiCorp Vault and rejected it. Vault offers dynamic short lived credentials, which is stronger, but it adds an operational component including unsealing for a benefit this platform does not currently need.
Cost: automatic rotation is not configured. That requires a rotation Lambda and is a genuine gap, recorded in the decision log rather than left unstated.
05THE PIPELINE
Four gates, and what they actually caught
Nothing merges until it passes secret scanning across full git history, dependency and image scanning, infrastructure scanning, and static analysis.
The gates block on findings that are both critical and have an available fix. Not on everything. A gate that blocks everything is a gate teams learn to route around, and I have watched that happen.
One run is deliberately captured red. A pipeline that has never blocked anything has not been proven to work.
CRITICAL
Eight credentials committed to the repository
Found by scanning full git history, not just the working tree. Several dated from months earlier and survived an attempted removal.
CRITICAL
Authentication bypass in the JWT library
CVE in PyJWT, present across all eight Python services. On a payments platform this means forging a token and acting as any customer.
CRITICAL
Remote code execution in the Go standard library
Present in the ledger binary, the single most sensitive service on the platform.
HIGH
Instance metadata service not hardened
Worker nodes allowed IMDSv1, which lets a compromised container read node credentials. Fixed with a launch template enforcing IMDSv2 and a hop limit of one.
HIGH
EKS secrets not encrypted with a customer managed key
Kubernetes secrets were stored without envelope encryption. Fixed with a dedicated KMS key and cluster encryption configuration.
BASELINE
Security rating E, 74 open issues
Starting state from static analysis before remediation. Recorded because the starting point is what makes the improvement meaningful.
06ISOLATION
The money zone rejects what it should, and only what it should
A policy that has been applied is not a policy that is being enforced. On EKS the two are genuinely different, because the VPC CNI accepts network policy objects and ignores them unless enforcement is enabled through a managed addon.
So I tested it by trying.
BLOCKED
curl: (28) Connection timed out after 8001 milliseconds
A service in the application zone attempting to reach the ledger directly. The packet is dropped, not rejected. A rejection would confirm to an attacker that the service exists.
PERMITTED
{"status":"ok","service":"payment-service"}
The gateway reaching payment-service on the permitted path. Isolation that blocks everything is not isolation, it is an outage.
09EVIDENCE
Screenshots from the running platform
Captured from my own screen, on real AWS infrastructure, during two capture days. Console URLs are visible where natural.
10LIMITATIONS
What is not here, and why
Every engineering project has a line where scope stopped. Here is mine, stated rather than quietly omitted.
Test coverage is thin
Static analysis flags zero percent coverage on new code, and that assessment is correct. The platform was built to demonstrate infrastructure and delivery practice, and the test suite is the honest casualty of that focus. On a client engagement this would be the first thing scheduled.
Secret rotation is not automated
Credentials are generated by Terraform and stored in Secrets Manager, never typed or committed. But rotation requires a Lambda rotation function that I did not build. The credentials are strong and unique per database. They are not rotating.
Secrets Manager rather than Vault
The original design named HashiCorp Vault. I used AWS Secrets Manager with IRSA instead, which achieves the outcome that matters, no credential stored in the cluster or the repository, without running and unsealing a secrets store. Defensible, but a departure from the original plan, and worth saying so.
No dynamic application security testing
The pipeline runs four static gates. A DAST stage against a running environment would catch a different class of issue and is not there.
Rolling updates, not blue green
Rollback is a single command and is measured. It is not a blue green switch with two environments running in parallel. For this platform the recovery time was good enough that the added complexity was not justified, but it is a real difference from the original design.
Kafka on ephemeral storage
Broker data does not survive a restart. This was a deliberate trade to avoid weakening IMDS hardening for the storage driver, and it is acceptable in a non production environment. A production deployment needs a scoped IAM role for the CSI driver instead.
11STACK
Everything used
Cloud
Infrastructure
Containers
Delivery
Security
Observability
Messaging
Load
Services
What this means if you are hiring
The work on this page is the standard I build to when nobody is telling me what to cut.
If your deployments are stressful, if your pipeline does not catch anything, if you are mid migration to Kubernetes, or if you cannot currently answer what happens when the thing you depend on goes offline, that is the work I do.
I am a senior DevOps engineer with three years on platforms where the cost of being wrong was measured in money rather than inconvenience. Previously at Barclays, IBM and RM Education.
Platform engineeringKubernetes in production, infrastructure as code, GitOps delivery
DevSecOpsPipelines that block real findings without blocking the team
ResilienceFailure drills, measured recovery, observability that answers business questions