I Moved My Queue Workers Off The VPS. Now A Job Spike Can't Touch The Web Server Or The GPS Ingestor.

I run a GPS tracking platform out of Dakar. For a long time the whole thing lived on a single DigitalOcean droplet: 2 vCPU, 8 GB RAM. That box ran the web app, the API, the Traccar daemon that ingests GPS positions, and 17 queue:work processes burning through SQS messages all day. It worked until it didn't.

This is the story of why I pulled the queue workers off that box, how the auto-scaler actually works, and the part I underestimated going in: physical isolation between workloads. A queue backlog can no longer touch the web server. A batch export can no longer wobble the GPS ingestor. Each tier scales on its own signal.

The squeeze

The capacity model I keep in docs/architecture/capacity.md is brutally simple. Sustained throughput is the minimum of six terms: FPM workers, app CPU, Postgres pool, Postgres CPU, MySQL pool, MySQL CPU. Whichever is smallest wins. Everything else is slack.

On the current 2 vCPU box, app CPU wins. Sustained ceiling is roughly 40 req/sec. Plenty for the web tier on a normal day. The problem is that the web tier is not the only thing on the box.

Three workloads were sharing two cores:

• Web + API: 30 PHP-FPM workers, average request around 150 ms, CPU-hot when traffic peaks.
• Traccar: a Java daemon listening on a TCP port, parsing GPS packets from a few hundred devices, dumping them into SQS. Bursty but cheap most of the time.
• 17 queue:work processes: positions queue, batches queue, CRM, emails, updates. Some short-lived and CPU-bound (positions), some long-lived and memory-hungry (batch exports).

The first sign of trouble was memory, not CPU. Top would show 6.4 / 7.8 GiB used + 1.1 GiB swap with 24 queue:work processes resident. Postgres was off-box on a separate EC2 instance, so the binding constraint on the droplet was the workers, not the database.

Then a batch export would kick off, RAM would tighten, swap would creep up, and the next request my user made got the latency of a box paging to disk. It wasn't a thundering herd. It was the noisy neighbor problem, and I was every neighbor.

The decision: evict the queue

Two options.

Option A: resize the box. 4 vCPU, 16 GB. Doubles every term in the capacity formula that I cared about. Costs more, and it kicks the problem six months down the road instead of solving it.

Option B: separate the workloads. Keep the web tier and Traccar on the VPS. Pull the queue workers onto their own infrastructure. Let each tier scale on its own signal.

I went with B for a reason that wasn't about cost. Bundling unrelated workloads onto one box means they share a fate. A queue spike steals CPU from a page request. A heavy job pushes Traccar into swap. A memory leak in any one of them is a memory leak for all of them. The blast radius of every problem is "the whole product."

Once you split them, the blast radius is bounded by hardware. A runaway batch worker on EC2 can fill its own RAM and crash. The droplet doesn't notice. A traffic spike that maxes FPM doesn't slow down the GPS ingestor, because the GPS ingestor isn't on the same CPU as FPM anymore.

That's the part of the story I want to make explicit, because it's the structural win and it's invisible in any cost spreadsheet:

• A queue backlog can no longer steal CPU from a page request. Web latency is no longer correlated with SQS depth.
• A busy batch job can no longer push Traccar into swap. GPS packet ingestion is no longer correlated with queue memory pressure.
• Each tier scales on its own signal. Web tier on req/sec. GPS ingestor on device count. Queue fleet on SQS depth. Three independent levers instead of one tangled one.

The mechanism

The auto-scaler is small. Four pieces, all Laravel.

WorkerMonitorCommand runs every minute on production

// bootstrap/app.php
$schedule->command('workers:monitor')->everyMinute()->runInBackground();
$schedule->command('workers:scale-in')->everyFiveMinutes();

It iterates every SQS queue, asks for the ApproximateNumberOfMessages attribute, appends a row to storage/logs/queue-metrics.jsonl, and fires a QueueBusy event for any queue whose depth is at or above the threshold.

Append-only JSONL was a deliberate choice. No state inside the command, just snapshots. Three days later I can cat queue-metrics.jsonl | jq and see exactly what happened. That turned out to matter, more below.

QueueBusy dispatches ScaleOutWorkerAction immediately

// AppServiceProvider
Event::listen(QueueBusy::class, ScaleOutWorkerAction::class);

ScaleOutWorkerAction looks up the per-queue spec, checks how many EC2 instances are already tagged for that queue, and either launches one or backs off if the queue is at its max_count. The instance comes up from a pre-baked AMI with Supervisor and the CloudWatch agent already installed. User-data injects the queue-specific Supervisor config so the box knows which queue to drain. Within about 90 seconds of launch, workers are processing messages.

Every instance is tagged Role=app-worker, Queue={queue-name}, Project=traxelio. The tags are not decorative. They are how scale-in stays safe.

WorkerScaleInCommand runs every 5 minutes with a grace counter

For each queue, if depth is below the scale-in threshold, increment a counter. If depth is above, reset it. When the counter hits scale_in_grace_checks (default 3, so 15 minutes of sustained calm), terminate the oldest EC2 instance tagged to that queue.

The grace period is the difference between a stable system and one that thrashes. Without it you'd spin instances up and down on every minor blip in SQS depth, which costs money in instance-launch overhead and is operationally noisy. With it, scale-in only fires when the queue has genuinely cooled off.

Termination is tag-gated at the IAM level, not just at the application level:

Action: ec2:TerminateInstances
Condition: ec2:ResourceTag/Role = app-worker
       AND ec2:ResourceTag/Project = traxelio

The scaler's IAM principal physically cannot terminate the web VPS or any Traccar instance, even if the application logic had a bug. The blast radius is bounded by tag, not by trust.

Per-queue right-sizing

This is the part I think is worth stealing if you build something similar. Different queues have different shapes. Treating them identically is leaving money or performance on the table.

// config/neptune-worker.php
"positions" => [
    "instance_type" => "c6i.large",  // 2 vCPU / 4 GB, compute optimized
    "processes" => 8,
    "max_count" => 5,
],
"batches" => [
    "instance_type" => "r6i.large",  // 2 vCPU / 16 GB, memory for large exports
    "disk_gb" => 40,                 // extra scratch space for CSV builds
    "processes" => 2,                // jobs are heavy, keep concurrency low
    "max_count" => 2,
],
"crm" => [
    "instance_type" => "t3.medium",  // burstable, low average load
    "processes" => 4,
    "max_count" => 2,
],

GPS positions: short-lived, CPU-bound, can spike to thousands of messages/sec when a hundred devices wake up at once. Compute-optimized box, eight workers per instance, up to five instances.

Batch exports: long-running, memory-hungry, single large CSV per job. Memory-optimized box, only two workers (high concurrency on memory-hungry jobs is a footgun), capped at two instances because they're rare.

CRM and emails: low average load with bursts. Burstable t3.medium is exactly what burstable instances are for.

The c6i.xlarge default catches anything I haven't profiled yet. Conservative on purpose.

The tuning loop

This is where the JSONL snapshots paid off.

I set WORKER_SCALE_OUT_THRESHOLD=20000 on day one. That number was a guess. Three days later I jq-ed the metrics file and found out the guess was bad:

batches:    max 9,912, mean 952, ≥1000 19% of the time
positions:  max 152
crm:        max 38
emails:     max 12
updates:    max 4

The 20k threshold was twice the actual peak on the busiest queue. It would never fire on routine load, which meant the auto-scaler was effectively decorative.

Two diagnostic notes worth recording.

First, the positions queue showing zero in the metrics is healthy, not broken. SocketListenCommand dispatches ProcessPositionsBatchJob (viaQueue → positions) as Traccar publishes, and the workers drain it as fast as it arrives. End-to-end latency is in the low milliseconds. Zero backlog means the system is keeping up, not that it's idle.

Second, memory was the binding constraint on the VPS, not CPU, when I checked. 6.4 / 7.8 GiB resident with 24 workers on the droplet. That's what I needed EC2 help to absorb, not CPU. So the threshold needed to be tight enough to actually pull batches off the VPS when load grew, not just sit there as insurance.

I lowered it to 12,000:

.env.production:
- WORKER_SCALE_OUT_THRESHOLD=20000
+ WORKER_SCALE_OUT_THRESHOLD=12000

12k beats 10k and 20k for three reasons:

• Well above the 9.9k peak so it won't fire on routine load.
• Low enough that a 25% growth in batch volume pulls in EC2 help.
• Not so aggressive that minor noise churns instances.

That's the kind of decision you cannot make without metrics. The cost of the JSONL file is roughly nothing. The value of three days of it was the difference between a threshold that worked and one that didn't.

What AWS would have given me for free, and why I built it myself anyway

This is the part I want to be honest about, because the obvious question reading any of the above is "Babacar, why didn't you just use AWS Auto Scaling Groups?" And the answer is: I could have. I didn't. Here's the 1:1, primitive by primitive.

EC2 Auto Scaling Groups with target tracking. The managed equivalent of what I built. You define a target metric (SQS backlog per instance is the canonical one for queue workers), AWS adjusts the desired capacity to keep the metric at the target. Scale-in protection, lifecycle hooks, instance refresh, mixed-instance policies. This is the boring, well-paved road. If you only need one queue and you want the least code to maintain, this is the right answer.

What I did instead: dispatch on a QueueBusy event, look up a per-queue spec, call RunInstances directly with tags. The "one ASG per queue" managed equivalent would have been five ASGs, five target tracking policies, five sets of scale-out and scale-in step configs. The control plane lives in AWS instead of in my Laravel app.

Application Auto Scaling. Same shape as ASG target tracking, but for ECS service tasks on Fargate. Per-task pricing. Faster cold start than a fresh EC2 boot. The right fit for short-lived bursty workloads that need to spin up in seconds. The wrong fit for sustained, multi-hour workloads where per-vCPU-hour pricing on dedicated EC2 wins. My positions queue runs hot all day; Fargate would cost more for that shape.

Spot via mixed-instances policy. This is the one I should genuinely be running and am not. Queue workers are the textbook Spot use case: jobs are idempotent (a position re-published is a no-op, a batch export retry produces the same CSV), interruptions are recoverable. Capacity-optimized-prioritized allocation, price-capacity-optimized strategy. AWS quotes 50 to 70 percent EC2 cost reduction for typical workloads. I haven't done it yet because I wanted to validate the basic auto-scaler first, but it's first on the list of things to add.

Predictive scaling. This one is the most interesting and the most likely to disappoint. Predictive scaling uses machine learning (literally, AWS Forecast underneath) to anticipate load and pre-scale before a metric crosses a threshold. The pitch is "no more cold starts, your capacity arrives before the spike." Reactive scaling waits for the queue to fill, then provisions. Predictive scaling provisions on a forecast.

Where it shines: diurnal traffic patterns. A consumer web app that wakes up at 8am every weekday. Streaming services with a clear evening peak. Anything with a repeating, learnable shape.

Where it doesn't: event-driven workloads with no diurnal signal. My batches queue spikes when a user clicks "export." That click is not on a forecast model's calendar. My positions queue follows vehicles, which roughly follow daylight hours but with enough irregularity (a fleet leaving at 5am on a Saturday for a long haul) that I'd rather react to actual load than guess at modeled load.

The honest take is: my workload probably has some forecastable shape (positions does follow a daily curve in Senegal), but the spikes that actually matter for capacity (batch exports, position bursts when devices reconnect after a network blip) are event-driven and not on any forecast. Predictive scaling would correctly identify the diurnal baseline and miss the spikes that actually hurt. I'll do a Forecast-Only mode trial on positions someday for the curiosity value, but I don't expect it to graduate to production.

EventBridge + Lambda + RunInstances. The interesting one. This is the AWS-native equivalent of what I built, point for point. CloudWatch alarm on SQS depth fires an EventBridge rule, the rule triggers a Lambda, the Lambda calls RunInstances with the right tags. Functionally identical to my WorkerMonitorCommand → QueueBusy event → ScaleOutWorkerAction chain, except the dispatcher lives in Lambda instead of Laravel. I would have built it almost exactly the same shape if I'd gone that route.

Karpenter. Kubernetes-native node autoscaler. Beautiful piece of engineering. Completely irrelevant to me because I'm not on EKS. Mentioning only for completeness, because every "how AWS does autoscaling in 2026" survey ends with Karpenter and you should know what it is.

So why did I build my own? Three reasons, in order of how honest they are.

Control. The scaler is 400 lines of Laravel I can read in an afternoon. Every decision (which queue triggers scale-out, what AMI to launch, how grace counters increment, when to refuse a launch because we're at max) is in code I own. When something behaves unexpectedly, I Log::info it and watch it happen on the next run. With ASGs, the same debugging is done through CloudWatch alarms, scaling activities, and lifecycle hook event payloads, which is fine but is its own ecosystem to learn.

Per-queue specs that vary by hardware shape. This is the load-bearing one. ASGs assume one launch template per group. My positions queue wants compute-optimized, my batches queue wants memory-optimized, my CRM queue wants burstable. The managed equivalent is "one ASG per queue with its own launch template," which is fine, but at that point I'm orchestrating five ASGs and the management overhead starts to rival the custom scaler.

Learning. I wanted to understand what an auto-scaler actually does. Reading AWS docs explains the abstractions; writing the thing explains the trade-offs. The grace counter, the tag-gated IAM, the JSONL metrics file, the per-queue max_count ceiling, the decision to use raw events instead of polling, the choice to keep the dispatcher in the app instead of in a Lambda. Every one of those was a question I had to answer because I was writing the code. ASG would have given me good defaults for all of them and hidden the question.

Would I make the same call again with full knowledge of the AWS primitives? Yes. The scaler is one of the parts of Traxelio I understand most thoroughly, precisely because I built it. That's an asset, not just for this scaler but for the next time I need to reason about distributed work, instance lifecycle, IAM tag-gating, or operational metrics. The features I'd add to it (per-queue thresholds, Spot, killswitch) are smaller than the migration to AWS-managed primitives would be, and they keep the understanding in my head.

The argument for AWS-managed Auto Scaling is real and I'd recommend it to anyone who wants the boring, well-paved road. The argument for building your own is also real, and the dividend is paid in understanding. Pick the one that matches what you want out of the project.

What I'd do differently

Three things, ordered by how much I'd actually do them.

Per-queue scale-out thresholds. Right now the threshold is global. A 12k scale-out makes sense for batches because its peak is 9.9k. It doesn't make sense for positions because positions peaks at 152. The fix is small: move the threshold into the per-queue spec next to instance_type. I haven't done it because the global number isn't actively hurting me yet, but it's on the list.

Cost dashboards before instance dashboards. I have CloudWatch agent on every worker streaming logs to /traxelio/worker* log groups. What I don't have yet is a single view of "how much did the auto-scaler cost me this week." The data is in Cost Explorer; I just haven't built the daily-roll dashboard. Coming.

A killswitch. Tag-gated IAM is good. A second-layer human killswitch (an env var that makes ScaleOutWorkerAction no-op) would be better, especially during incidents. Two minutes of work I keep not doing.

The bigger lesson

The interesting part of this project wasn't the auto-scaler. The auto-scaler is roughly 400 lines of Laravel and an AMI bake. The interesting part was the structural choice: stop sharing a CPU between workloads with fundamentally different shapes.

Web is latency-sensitive, low average CPU, sensitive to spikes. GPS ingest is throughput-sensitive, mostly idle, sensitive to memory pressure. Queue work is bursty, sometimes CPU-bound, sometimes memory-bound, opportunistic.

You can run all three on the same box. I did, for a long time, and it worked. But the day they start fighting each other for the same resource, you have a problem with no good local fix. You can't make the queue politely yield CPU when the web tier needs it. You can't make a batch export politely free RAM when Traccar needs to allocate a buffer.

The local fix is "tune the workers' nice value." The structural fix is "give them their own hardware." The structural fix scales. The local one doesn't.

If you're running a small SaaS on a single VPS today and it's starting to wobble, this is the question worth asking before you resize: which of these workloads is hurting which other workload, and is that something I want to keep paying for?

Six weeks in, the answer for Traxelio was "all three are hurting each other, and no, I don't want to keep paying for that." Pulling the queue off the box was a one-week project. Six weeks of decoupled web latency and stable GPS ingest later, it's the cheapest infrastructure decision I've made this year.