fix: infinite cache growth

See the discussion here: #984 And the upstream PR here: openedx/edx-platform#34210 The tl;dr is that the Redis course structure cache was growing without bounds. While the upstream fix should resolve that issue, we decided that Tutor should have a maxmemory limit and an eviction policy set for operational safety. Thus, Redis now has a 4gb maxmemory. If you need more memory on your instance, you should implement the "redis-conf" patch. To manually expire existing keys, run: tutor local run cms ./manage.py cms shell -c "from django.core.cache import caches; c = caches['course_structure_cache']; [c.expire(key, 604800) for key in c.keys('*')]"
overhangio · Feb 19, 2024 · 59b1987 · 59b1987
1 parent 208937d
commit 59b1987
Show file tree

Hide file tree

Showing 5 changed files with 22 additions and 1 deletion.
diff --git a/changelog.d/20240208_111236_regis_infinite_course_cache.md b/changelog.d/20240208_111236_regis_infinite_course_cache.md
@@ -0,0 +1,4 @@
+- 💥[Bugfix] Prevent infinite growth of course structure cache in Redis. (by @regisb)
+  - Redis is now configured with a maximum memory size of 4GB. If this is too low for your platform, you should increase this value using the new "redis-conf" patch.
+  - Make sure that course structure cache keys have an actual timeout.
+- [Feature] Introduce the "redis-conf" patch. (by @regisb)
diff --git a/docs/reference/patches.rst b/docs/reference/patches.rst
@@ -376,6 +376,13 @@ File: ``apps/openedx/settings/lms/production.py``
 
 Python-formatted LMS settings in production. Values defined here override the values from :patch:`openedx-lms-common-settings`.
 
+``redis-conf``
+==============
+
+File: ``apps/redis/redis.conf``
+
+Implement this patch to override hard-coded Redis configuration values. See the `Redis configuration reference <https://redis.io/docs/management/config-file/>`__`.
+
 ``uwsgi-config``
 ================
 

diff --git a/tutor/templates/apps/openedx/settings/partials/common_all.py b/tutor/templates/apps/openedx/settings/partials/common_all.py
@@ -73,7 +73,7 @@
     },
     "course_structure_cache": {
         "KEY_PREFIX": "course_structure",
-        "TIMEOUT": 7200,
+        "TIMEOUT": 604800, # 1 week
         "BACKEND": "django_redis.cache.RedisCache",
         "LOCATION": "redis://{% if REDIS_USERNAME and REDIS_PASSWORD %}{{ REDIS_USERNAME }}:{{ REDIS_PASSWORD }}{% endif %}@{{ REDIS_HOST }}:{{ REDIS_PORT }}/{{ OPENEDX_CACHE_REDIS_DB }}",
     },

diff --git a/tutor/templates/apps/redis/redis.conf b/tutor/templates/apps/redis/redis.conf
@@ -39,3 +39,10 @@ auto-aof-rewrite-percentage 100
 auto-aof-rewrite-min-size 64mb
 aof-load-truncated yes
 aof-use-rdb-preamble yes
+
+############################## MEMORY MANAGEMENT ################################
+
+maxmemory 4gb
+maxmemory-policy allkeys-lru
+
+{{ patch("redis-conf") }}
diff --git a/tutor/templates/build/openedx/Dockerfile b/tutor/templates/build/openedx/Dockerfile
@@ -51,6 +51,9 @@ RUN git config --global user.email "tutor@overhang.io" \
 {{ patch("openedx-dockerfile-git-patches-default") }}
 {%- else %}
 # Patch edx-platform
+# Prevent course structure cache infinite growth
+# https://github.com/openedx/edx-platform/pull/34210
+RUN curl -fsSL https://github.com/openedx/edx-platform/commit/ad201cd664b6c722cbefcbda23ae390c06daf621.patch | git am
 {%- endif %}
 
 {# Example: RUN curl -fsSL https://github.com/openedx/edx-platform/commit/<GITSHA1>.patch | git am #}