"sqlite3.OperationalError: no such table" when doing make

[roperi]

I'm getting whilst running the migrations just after bootstraping with make.
make

Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying music.0001_initial... OK
  Applying music.0002_song... OK
  Applying music.0003_video... OK
  Applying music.0004_auto_20140826_0130... OK
  Applying music.0005_video_preview_url... OK
  Applying music.0006_release_player_code... OK
  Applying music.0007_song_player_code... OK
  Applying music.0008_auto_20160204_1335... OK
  Applying music.0009_auto_20160215_0146... OK
  Applying music.0010_press... OK
  Applying music.0011_auto_20160328_0129... OK
  Applying music.0012_auto_20160704_1559... OK
  Applying music.0013_video_priority... OK
  Applying music.0014_auto_20200125_1148... OK
  Applying news.0001_initial... OK
  Applying news.0002_auto_20140826_2342... OK
  Applying news.0003_auto_20141101_2348... OK
  Applying news.0004_auto_20160118_0159... OK
  Applying sessions.0001_initial... OK
  Applying shows.0001_initial... OK
  Applying shows.0002_auto_20140826_2342... OK
  Applying shows.0003_venue... OK
  Applying shows.0004_remove_gig_city... OK
  Applying shows.0005_auto_20140828_1258... OK
  Applying shows.0006_auto_20140828_1433... OK
  Applying shows.0007_remove_gig_slug... OK
  Applying shows.0008_auto_20160803_0130... OK
/home/user/workspace/myproject/venv/bin/python manage.py loaddata hth/jahhills.json
Traceback (most recent call last):
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: no such table: music_release__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 115, in loaddata
    connection.check_constraints(table_names=table_names)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 286, in check_constraints
    column_name, referenced_column_name,
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: Problem installing fixtures: no such table: music_release__old
make: *** [Makefile:24: update] Error 1

If I run make again I get No migrations to apply but then fails again with:

Running migrations:
  No migrations to apply.
/home/user/workspace/myproject/venv/bin/python manage.py loaddata hth/jahhills.json
Traceback (most recent call last):
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
sqlite3.OperationalError: no such table: shows_venue__old

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/core/management/commands/loaddata.py", line 115, in loaddata
    connection.check_constraints(table_names=table_names)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 286, in check_constraints
    column_name, referenced_column_name,
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/django/db/backends/sqlite3/base.py", line 326, in execute
    return Database.Cursor.execute(self, query)
django.db.utils.OperationalError: Problem installing fixtures: no such table: shows_venue__old
make: *** [Makefile:24: update] Error 1

Then make test fails

$ make test
/home/user/workspace/myproject/venv/bin/flake8 hth
/home/user/workspace/myproject/venv/bin/pytest --cov --cov-report=html --cov-report=term
Traceback (most recent call last):
  File "/home/user/workspace/myproject/venv/bin/pytest", line 5, in <module>
    from pytest import main
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/pytest/__init__.py", line 6, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/_pytest/assertion/__init__.py", line 9, in <module>
    from _pytest.assertion import rewrite
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 24, in <module>
    from _pytest.assertion import util
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/_pytest/assertion/util.py", line 14, in <module>
    import _pytest._code
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/_pytest/_code/__init__.py", line 2, in <module>
    from .code import Code  # noqa
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/_pytest/_code/code.py", line 28, in <module>
    import pluggy
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/pluggy/__init__.py", line 16, in <module>
    from .manager import PluginManager, PluginValidationError
  File "/home/user/workspace/myproject/venv/lib/python3.7/site-packages/pluggy/manager.py", line 11, in <module>
    import importlib_metadata
ModuleNotFoundError: No module named 'importlib_metadata'
make: *** [Makefile:33: test] Error 1

[bhrutledge]

@roperi I was able to bootstrap a brand-new environment by following the "Getting Started" instructions (which I tweaked slightly in 8feb843).

It looks like the issue is that manage.py loaddata hth/jahhills.json failed (after a successful manage.py migrate). In the output, I see references to things like music_release__old, but the __old prefix isn't anywhere in the codebase. Did you add that?

Also, what's your goal with cloning this repo? It's very specific to https://www.hallelujahthehills.com/. For example, jahhills.json contains all of the content.

[roperi]

Hi @bhrutledge ! Thanks for the project. I wanted to set up a site for my sister's band using django and thought your project was the best choice among several I also checked. I also find it very interesting how it is structured (compared to other django projects i have seen) so I'm also learning from it so I can leverage your experience for future projects (especially the deployment bit). I forked it first but since I'm a still learning Github and git I decided to clone it and made it private instead so I don't screw up yours by some errant git push. At the moment i'm reading about how to manage remote branches because for some reason my local and origin repos ended up in the datasette branch.

I only notice the manage.py loaddata hth/jahhills.json today. I did a quick search in the jahhills.json file but couldn't find any reference to music_release__old. And no, I didn't add it. I haven't pushed any changes yet not even to my copy. In fact, I haven't even made any changes to any of the code!

[roperi]

By the way, the make test error was because my venv was getting my system's environment variables instead of the project ones. So once I got that fixed I run the make test again got errors and warnings.

30 failed, 85 passed, 25 skipped, 41 warnings in 16.27s

I saw some of these:
ValueError: Missing staticfiles manifest entry for 'css/normalize.css'

Should I open a new issue about this ?

[bhrutledge]

@roperi Cool! I'm flattered! It's been a great learning project for me (and helped me get a job as software engineer).

One big caveat: it's currently using Django 1.11, which is no longer supported, and there's a lot of good stuff to use in the newer versions. One of the reasons for the static deployment was to avoid the security risks without having to put in the work to upgrade Django (which I've captured in a GitHub project).

That said, I'm happy to help you get it working as an example.

Re: the test failures, I was able to reproduce that locally by running make test before manage.py collectstatic had been run. I haven't encountered that before, because I regularly run make, which runs collectstatic. I suspect you encountered it because of the initial failures when running make. To avoid this gotcha, I tweaked the settings during testing; see ff58801. So, thanks for reporting that!

To get it working locally, I recommend updating your fork (or recreating it) to get my latest changes, and using a fresh clone of that to go through the updated Getting started instructions (which are quite sparse, because they were only intended for me). Using a fork is the most common way to customize someone else's work on GitHub. I also wouldn't worry about it being public.

Feel free to continue commenting on this issue with your progress. And, out of curiosity, what operating system are you using?

[roperi]

Hi! Well, I think this project looks very professional and and I'm not surprised you got a job because of it! Especially the TDD approach (that's the definite sign of a serious programmer, i think).

I'm using Debian 10. Gotta say I'm not a software developer but I have developed some python and django apps for my own professional use. So please bare with me as I'm still slow!

By the way, I didn't encounter problems when running make serve. I can see the site on my browser at 127.0.0.1:8000.

Thanks so much for offering help. I'll try again once I fully understand your comments above.

[roperi]

@bhrutledge,

I tried again the make test and I think it went swimmingly 115 passed, 25 skipped, 44 warnings in 5.36s.

I managed to merge your latest changes. I'm basically following this Create a private fork of a public repository guide. So far so good and I can still do pull requests.

At the moment I'm checking the code. Specifically the makefiles and the app configuration, models, admin and views.

I don't get when we should use the make clean. I noticed you just added it.

[bhrutledge]

Nice! The Makefile would definitely benefit from some more commenting. 😉 make clean just deletes all the files that are created by make dist, to make it easier to change/debug the build process. It's not necessary to use on a regular basis.

[roperi]

Appreciate your help! I have some -perhaps very naive- questions, if you don't mind answering.

1) I'm planning to use nginx + gunicorn + django. is there anything I should consider/change/adjust/gotchas to consider ? I have some experience deploying using this setup

2) I'm very inexperienced with frontend (im more of a backend kind of person). When, during development process, do i have to run make css? Also, for deployment purposes, do I have to run make css as well?

3) How do I build or rebuild the media and static files without using Makefile ( eg. so i can copy their folders to /var/www/html/ and later serve them via AWS S3) ?

Thanks again!

EDIT

I now know answer to question 2.

UPDATE

I think I get everything now. I'll attempt to deploy it during this weekend. Sorry for posting these questions.

[bhrutledge]

All good! Thanks for the update. Let me know how it goes. :)

[roperi]

Hi @bhrutledge ! I did it! I deployed it successfully! Now I'm going to edit the HTML/CSS. Thanks again for the project! I had fun and learned quite a lot!

[bhrutledge]

@roperi Hooray! Nicely done! Where you did you deploy it? Also, I’d love to see the site and your repo. Can you share those?

[roperi]

@bhrutledge ! Sure. I made it public at roperi/coctelduo. I worked on the develop branch where I stripped out many things to build from there while I learn a bit more about make and testing. Site is at coctelduo.com but it is still a work in progress. It's basically a clone as I haven't touched the html/Css yet.

[bhrutledge]

@roperi Thanks for sharing! The site is looking great! I also took a look at the repo earlier, and was going to make some suggestions by opening issues, but it looks like you took it down. Do you mind making it public again? Or adding me as a collaborator?

Since I think we've resolved your original issue, I'm going to close this, but feel free to add more comments.

[roperi]

Thanks, @bhrutledge ! I forgot to mention I credited you as the site creator. And yes, I put it back to private again when I realised I pushed the settings.py file with private details by mistake. I don't know what to do now to revert those changes.

[bhrutledge]

@roperi Have you read through https://docs.github.com/en/github/authenticating-to-github/removing-sensitive-data-from-a-repository? That seems to give you a few options. Alternatively, if you don't care about the commit history, you could start over by deleting the repository on GitHub, removing the .git directory from your local repository, running git init in your local repository, and pushing it to a new GitHub repository.

Either way, you should change the private details that were committed. For example, if it was an API key, you should generate a new one.

[roperi]

Alright, @bhrutledge . Good idea! I'll do this this weekend. Thanks!

[roperi]

Hi @bhrutledge . I pushed it here

[bhrutledge]

@roperi Good stuff! As you probably saw, I created a handful of issues based on a quick scan of the repo. I'm guessing some of those were already on your radar. I've found it useful to track my todo's that way, but if feel free to close them if it's not useful for you.