How We Worked Embedded System: Infrastructure Ownership from Day One We took complete ownership of the infrastructure environment. Not consulting at arm's length — direct access to the AWS account, the EC2 instances, the container configurations, and the monitoring layer.

The first step was visibility.

Before changing anything, we needed to understand exactly where the pressure was being applied. We introduced real-time performance monitoring across all 18 sites — CPU, memory, request latency, error rates. Not just checking whether sites were up, but understanding the pattern of how they degraded under load.

This monitoring revealed the traffic distribution problem: during peak periods, two or three sites were responsible for over 80% of the load, but all 18 shared the same resources. The high-traffic sites were starving the low-traffic sites, which were in turn destabilising the high-traffic sites.

Law 1: Distance is the enemy of speed. Without visibility, the infrastructure team was always reacting to outages after the fact. Monitoring moved the response from reactive to anticipatory.

Recovery Phase

Phase 1 Architecture Redesign (Weeks 1-3): Eliminate the single point of failure

The core architectural change: move from 18 containers on one instance to isolated environments per application.

Each site deployed within its own isolated container environment. Traffic surge on one site cannot destabilise others. A code issue on one country's booking flow doesn't take Australian checkout offline.

This is not a complex architectural principle — isolated environments are standard infrastructure practice. The previous architecture simply hadn't been designed by anyone thinking about failure modes under load.

Phase 2 Auto-Scaling Implementation (Weeks 3-5): Let the platform expand and contract automatically with demand.

Auto-scaling rules configured: when CPU or memory crosses defined thresholds on any site's environment, additional compute spins up automatically. When demand falls, it scales back down.

The business implication: peak season promotional bursts — the moments when Ozsnow ran its most important marketing campaigns — would no longer be the moments of highest failure risk. The platform would scale to meet the traffic.

Cost implication: Auto-scaling with proper rightsizing meant OzSnow was no longer paying for peak-capacity compute year-round. Resources expand when needed, contract when not. Monthly AWS spend reduced approximately 40% compared to the over-provisioned single-instance model.

Phase 3 Traffic Routing Simplification (Weeks 5-7): Reduce infrastructure management overhead

The 18-site architecture had accumulated multiple entry points, separate load balancers, and routing complexity that was difficult to maintain. We redesigned so all incoming traffic was handled through a single unified gateway — one load balancer, one routing layer, 18 applications behind it.

Simpler to manage. Fewer failure points. Faster diagnostic path when something goes wrong.

Phase 4: Monitoring and Alerting (Ongoing): Never be surprised by a failure again.

Alert thresholds configured for each site individually. Notification path defined: infrastructure anomaly → Halcrow alerted immediately → OzSnow notified within 15 minutes of anything that affects customer experience.

The monitoring layer we built became the operational foundation for ongoing maintenance. Problems caught at threshold, not at crash.

The Outcome Metric (Before / After)

• Infrastructure model: 18 containers on 1 EC2 instance / Isolated environments with auto-scaling

• Monitoring: None / Real-time across all 18 sites

• Peak season uptime: Frequent crashes during surges / 100% uptime through the full ski season

• Traffic spike response: Manual (or none) / Automatic scaling

• Blast radius: Single-site failure → all 18 affected / Each site isolated

• Monthly AWS cost: Over-provisioned baseline / ~40% reduction via auto-scaling

• Incident response: Reactive (customer reports) / Proactive (threshold alerts)

The result that mattered most: OzSnow went through a full peak ski season — their entire annual revenue window — with zero infrastructure outages during major traffic periods. The platform was stable when it needed to be stable.