MySQL Data Too Long Error During Forum Migration from MongoDB to MySQL

[Abdul-Muqadim-Arbisoft]

While migrating forum data from MongoDB to MySQL using the following command:
tutor local run lms ./manage.py lms forum_migrate_course_from_mongodb_to_mysql --no-toggle all

I encountered a Data Too Long for Column 'title' error. It seems MySQL is not accepting longer titles, and we might need to increase the column length.

Error Log:

2023-08-16 07:41:24.513000) while time zone support is active.
  warnings.warn(

Traceback (most recent call last):
  File "/openedx/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.11/site-packages/MySQLdb/connections.py", line 261, in query
    _mysql.connection.query(self, query)
MySQLdb.DataError: (1406, "Data too long for column 'title' at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/openedx/edx-platform/./manage.py", line 106, in <module>
    execute_from_command_line([sys.argv[0]] + django_args)
  File "/openedx/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/openedx/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/openedx/venv/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/openedx/venv/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/forum/management/commands/forum_migrate_course_from_mongodb_to_mysql.py", line 47, in handle
    migrate_content(db, course_id)
  File "/openedx/venv/lib/python3.11/site-packages/forum/migration_helpers.py", line 70, in migrate_content
    create_or_update_thread(content)
  File "/openedx/venv/lib/python3.11/site-packages/forum/migration_helpers.py", line 85, in create_or_update_thread
    thread = CommentThread.objects.create(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
    obj.save(force_insert=True, using=self.db)
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
    self.save_base(
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
    results = self._do_insert(
              ^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
    return manager._insert(
           ^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
    cursor.execute(sql, params)
  File "/openedx/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/openedx/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/openedx/venv/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/django/db/backends/mysql/base.py", line 75, in execute
    return self.cursor.execute(query, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/openedx/venv/lib/python3.11/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/openedx/venv/lib/python3.11/site-packages/MySQLdb/connections.py", line 261, in query
    _mysql.connection.query(self, query)
django.db.utils.DataError: (1406, "Data too long for column 'title' at row 1")

Steps to Reproduce:

• Run the forum migration command as shown above.
• Observe the error in the logs.

Expected Behavior:
The migration should successfully transfer data from MongoDB to MySQL without truncation or errors.

Actual Behavior:
The migration fails due to a MySQL DataError (1406), indicating that the title field length is exceeding the allowed limit.

[taimoor-ahmed-1]

The problem originates from the fact that mongodb doesn't enforce limits on fields individually, whereas in MySQL we have that limit. Currently, the limit is 255 (Charfield). Should we update the title fields to TextField in light of this issue?

@regisb @Ali-Salman29 what do you guys suggest?

[regisb]

I might be naive, but I think it does make sense to keep a limit on title lengths. Otherwise any user could fill the database with arbitrary long fields.

Here's what I suggest:

1. increase the title length limit to something like 1024 characters
2. trim longer titles on migration from mongodb to mysql
3. make sure that creating forum posts with too long titles does not result in a 500 error

[DawoudSheraz]

Does this require any action in tutor-forum? This looks like an upstream issue rather than tutor-forum.

[regisb]

Yes, this is an issue for openedx/forum. @taimoor-ahmed-1 can you please migrate or copy the issue there?

[Abdul-Muqadim-Arbisoft]

Created the issue over openedx/forum

[taimoor-ahmed-1]

I have assigned this to myself over at forum, will open a PR soon

[DawoudSheraz]

Closing this issue as this is an upstream issue that is being tracked on openedx/forum#165