February, 22, 2024 archives

I am either really good at making Oracle Cloud Infrastructure instances fall over, or the VM.Standard.E2.1.Micro shape is even more under-powered than I expected. I had been using the Ubuntu “minimal” image as my base, so I thought I would try the Oracle Linux 8 image and I couldn’t even get it to run yum check-update without that process getting killed. That seems like a less-than-ideal experience out of the box.

What seems to happen on the instances (with Ubuntu) that I am using to host this site is that if something does too much I/O, the load average spikes, and things slowly grind through before recovering. The problem is that something like “running composer” seems to be too much I/O, which makes it awkward to deploy code.

Another thing that seems to get out of control quickly is when I reindex the site with Meilisearch. Considering there is very little data being indexed, that obviously shouldn’t be causing any sort of trouble. I have two instances spun up now, so I can play with the settings on one without temporarily choking off the live site. It’s probably just a matter of setting the maximum indexing memory in Meilisearch’s configuration or constraining the memory on that container.

I also added a OCI Flexible Network Load Balancer in front of my instance so I can quickly switch things over to another without waiting on any DNS propagation. Maybe if Ampere instances ever become available in my region I will play around with splitting the deployment across multiple instances.

Playing around with my OCI instances some more, I looked more closely at what was going on when I was able to trigger the load to go out of control, which seemed to be anything that did a fair amount of disk I/O. What quickly stuck out thanks to htop is that there were a lot of Oracle Cloud Agent processes that were blocking on I/O.

So in the time-honored tradition of troubleshooting by shooting suspected trouble, I removed Oracle Cloud Agent.

After doing that, I can now do the things that seemed to bring these instances to their knees without them falling over, so I may have found the culprit.

I also enabled PHP’s OPcache and some rough-and-dirty testing with good ol’ ab says I took the homepage from 6r/s to about 20r/s just by doing that. I am sure there’s more tuning that I could be doing. (Requesting a static file gets about 200 r/s.)

By the way, the documentation for how to remove Oracle Cloud Agent on Ubuntu systems is out of date. It is now a Snap package, so it has to be removed with sudo snap remove oracle-cloud-agent. And then I also removed snapd because I’m not using it and I’m petty like that.