My OpenSearch Docker blew up my hard drive

Hard drive filling up with ...?

I always thought I had plenty of disk space with an SSD of 500 Gb (of which only 70% was used). But after some upgrades in my Docker environment - well, simply wiping out everything, so I needed to re-download all images - I suddenly noticed that my hard-drive was filling up. Up to 100% so that even a simple vi would not work (because the swap file could not be created). What's up?

By using du --max-depth=1 -h on various folders, I was able to investigate that somehow Docker was to blame. Next, docker system df showed me that one of the containers went berserk. And finally, with docker container ls -s, I saw that my OpenSearch container (based on opensearchproject/opensearch:2) was to blame.

Diving into the OpenSearch container

Logging into the OpenSearch container (docker exec -it opensearch bash) I found that tools like du or df were not available. But I had a hunch, so navigated into the log folder /usr/share/opensearch/logs/ right away. And I was right, there was a logfile PerformanceAnalyzer.log that was increased with about 10Mb per second (up to the point that the logfile was 37G large), adding nothing more than the following message:

ERROR org.opensearch.performanceanalyzer.PerformanceAnalyzerApp - Error in ReaderMetricsProcessor...restarting, ExceptionCode: ReaderRestartProcessing

Mtf, how nice. I did reckognize the performance analyzer plugin - played around with it in the past a bit, but definitely had it on my wishlist to disable the thing when in a local development environment. Googling around I found that the error message might be related to the fact that the /tmp folder was mounted with the noexec flag - which definitely rang a bell because I mounted /tmp via tmpfs. But instead of undoing this, I decided I wanted to get rid of the plugin once and for all.

Disabling the performance analyzer

Unfortunately, there is no environment variable or configuration flag for disabling the performance analyzer in OpenSearch (like there is for the security plugin). In the docs, one hint was to disable the setting via the HTTP API:

curl -XPOST localhost:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabled": false}'

But this did nothing for me - the logfile just kept on growing. To my understanding, the flag did not disable the plugin in the overall OpenSearch configuration, but just started up the plugin to then disable itself. Anyway, did not work, moving on.

Removing the performance analyzer plugin

One approach I considered was to simply symlink the logfile /usr/share/opensearch/logs/PerformanceAnalyzer.log to /dev/null, but this felt ugly. A better approach for me was to remove the plugin, before OpenSearch was even being started. There is a CLI for this:

/usr/share/opensearch/bin/opensearch-plugin remove opensearch-performance-analyzer

And earlier in another docker-compose.yml file I created, I implemented something similar for this, not for the performance analyser but for the security plugin:

services:
  opensearch:
    image: opensearchproject/opensearch:2
    command: sh -c "opensearch-plugin remove opensearch-security && exec /usr/share/opensearch/opensearch-docker-entrypoint.sh"

But to extend upon this command with multiple subcommands to remove individual plugins (resulting in a huge one-liner) did not feel good. Instead, a custom Docker build offered a cleaner approach:

services:
  opensearch:
    build: ./docker-compose/opensearch

with a file docker-compose/opensearch/Dockerfile like the following:

FROM opensearchproject/opensearch:2
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-performance-analyzer

Restarting the cluster with docker-compose up -d --build confirmed that this new approach indeed fixed the issue.

While we are at it ...

It made me wonder what other plugins were enabled by default, while I actually did not need them. Diving into the docs and the currently enabled plugins, I found numerous other plugins that were of no use to me, when developing Magento or Shopware. I added them all to my custom Dockerfile:

FROM opensearchproject/opensearch:2
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-performance-analyzer
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security-analytics
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-reports-scheduler
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-cross-cluster-replication
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-notifications
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-notifications-core
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-alerting
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-anomaly-detection
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-ml
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-asynchronous-search
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-neural-search
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-knn
RUN /usr/share/opensearch/bin/opensearch-plugin remove opensearch-geospatial

There might be more plugins to disable. Running /usr/share/opensearch/bin/opensearch-plugin list allows you to analyse which plugins are enabled.

Retrospect

In retrospect, I have had far more issues with tuning bits in OpenSearch, than I had with ElasticSearch. Maybe this is not necessarily due to the product itself, but much more with the Docker image they provide. It seems the OpenSearch image is offered much more as a demo, than an image to use for local development. Anyway, I tuned it, the performance seems nice and at least my hard drive is not filling up anymore.