ValueError: dictionary update sequence element #0 has length 1; 2 is required

[mrmch]

Seeing this issue:
https://github.com/niwibe/django-orm-extensions/issues/19

Using djorm-ext-hstore 0.4.3
pg 9.1.9
django 1.4.3

It's very sporadic and unpredictable.

[niwinz]

I can not reproduce it in my test environment with stantard unittest.
Also, I use it in production and all works properly.

This issue seems happens on testing environment with nose. I have investigated a lot of about this bug and I can not reproduce this error in normal test envirnoment (without nose) .

[ntucker]

Adding this made it work. I think someone mentioned it somewhere. I use django-discover-runner.

def setUp(self):
    psycopg2.extras.register_hstore(connection.cursor(), globally=True)

[mrmch]

I could predictably reproduce it while using a django environment with multiple database servers (both pg 9.1.9).

From a django shell (./manage.py shell), querying one database and then another would cause the issue.

ex:

# is fine, 167 valid id, data is DictionaryField
Customer.objects.get(id=167).data
Customer.objects.using('production').get(id=167).data
ValueError: dictionary update sequence element #0 has length 1; 2 is required

[niwinz]

I'm going to work on it in few hours. Thanks for all reports!

[niwinz]

I have added some proposed test and I have investigated about it some time and I can not reproduce these errors.

Using basic unit testing and with these variables:

• hstore installed in template1 of postgresql
• with tests with multiple databases
• with django 1.4 and 1.5
• with psycopg2 (2.3, 2.4, 2.4.1, 2.5)
• with python2 and python3

And all tests are successful. The problem seems out of djorm-ext-hstore. I would like to help for solve this issue but I don't know how if i can not reproduce it on my environments. (I have this plugin in production in other environments also with tests and all works correctly...)

:(

[tejinderss]

I confirm that I get this error from time to time on Django 1.5 and latest hstore extension module.

[ntucker]

niwibe, did you try using django-discover-runner? It might not be this module's fault, but at least that would help determine what is happening.

[niwinz]

I currently use django-discover-runner in other project with djorm-ext-hstore successful.

Are you put djorm-ext-core as installed app? Without it, this extension does not works ...
I don't know but all possible and reasonable test environments that I test, it works correctly. I have it on production environments. I don't understand this bug :S

[niwinz]

https://github.com/niwibe/djorm-ext-hstore/blob/master/testing/settings.py#L33 this line is mandatory for make hstore extension works properly...

[mrmch]

Hi niwibe,

As the instructions don't mention adding 'djorm_core', 'djorm_expressions', and 'djorm_hstore' to INSTALLED_APPS, I hadn't taken that step.

Thanks for clarifying that -- I'll test this against my environment.

[ntucker]

I added
'djorm_core',
'djorm_expressions',
'djorm_hstore',
to my app list

This didn't change anything. Are they supposed to be listed before/after something?

[niwinz]

Not, the order is not important. Can you make me a simple test project that on your environment does not works? Because on my environments I can not reproduce this error...

[mrmch]

I added the apps and ran syncdb, no avail.

I will try to setup a multi server example later today.

% ./manage.py shell_plus
From 'customers' autoload: Customer, CustomerDataKeys
Python 2.7.3 (default, Aug  1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> Customer.objects.all()
[<Customer:>, '...(remaining elements truncated)...']
>>> Customer.objects.using('production').all()
Traceback (most recent call last):
 File "<console>", line 1, in <module>
File "/srv/virtualenvs/betawithus/local/lib/python2.7/site-packages/django/db/models/query.py", line 72, in __repr__
 data = list(self[:REPR_OUTPUT_SIZE + 1])
File "/srv/virtualenvs/betawithus/local/lib/python2.7/site-packages/django/db/models/query.py", line 87, in __len__
 self._result_cache.extend(self._iter)
File "/srv/virtualenvs/betawithus/local/lib/python2.7/site-packages/django/db/models/query.py", line 301, in iterator
 obj = model(*row[index_start:aggregate_start])
File "/srv/virtualenvs/betawithus/local/lib/python2.7/site-packages/django/db/models/base.py", line 300, in __init__
 setattr(self, field.attname, val)
File "/srv/virtualenvs/betawithus/local/lib/python2.7/site-packages/djorm_hstore/fields.py", line 38, in __set__
 value = self.field._attribute_class(value, self.field, obj)
File "/srv/virtualenvs/betawithus/local/lib/python2.7/site-packages/djorm_hstore/fields.py", line 21, in __init__
 super(HStoreDictionary, self).__init__(value, **params)
 ValueError: dictionary update sequence element #0 has length 1; 2 is required

[boronine]

Hey guys, I'm having the same issue. Looks like DictionaryField gets serialized as a JSON string, but deserialization expects an actual JSON object. So whose fault is it, the serializer's or the deserializer's?

[niwinz]

Please. For solve it I need some test project for reproduce it. On my tests all works fine... See comment: #11 (comment)

[boronine]

(See #14 for my fix)

[fdemmer]

i ran into this today while writing tests for my app and running tests for djorm-hstore fail too, as mentioned in #12. i dug around, added debug output and at first i thought the singleton ConnectionCreateHandler does not work, because the handler callback vanished from the dictionary of handlers after the first call. But that is what it is supposed to do when using unique=True!

After setting unique=False (the default) my tests run fine.

So i am now wondering why is unique set True in the first place?!
https://github.com/niwibe/djorm-ext-hstore/blob/master/djorm_hstore/models.py#L104

[niwinz]

It set to true because in theory it should registred only once.
But if setting it to False solves this strange bug, I can put it to False.

But the strangest thing is that on my environments works perfectly with unique=True... remember, I have been said that I use it on many projects with unique=True and works all fine... :s

[fdemmer]

i cannot fully explain it either. i have a seemingly completely similar dev environment on another computer where tests run fine with unique=True.

i think, that it should register once per connection. at least to my limited understanding register_hstore_handler needs to be called for every newly created connection, so every time backend_signals.connection_created signals. with unique=True this line del self.unique_handlers[connection.vendor] deletes the callback doing register_hstore(connection.cursor(), globally=True).

well, let's apply the patch and see if it helps or hurts anyone else. it should not hurt imho.

[niwinz]

I agree with you!

[fdemmer]

i looked at this again last night...

i saw, but somehow totally ignored the globally=True on the register_hstore(connection.cursor(), globally=True) call. of course once should be enough with that argument, you are right!

and i have seen it work with unique=True/globally=True and break with unique=True/globally=False (which would be correct behaviour). unfortunately there is still a bug somewhere, as i also see unique=True/globally=True broken in the other environment.

setting unique=False in #17 certainly is only a workaround and as far as i understand the register_hstorecode, it hits the database each time to get the hstore oid. so that might be a negative side effect.

the question now is, why does globally=True not always work. or does it and something else is broken with the dictionary adapter...?

i am not sure how much time i can spend on investigating further, but i think an issue should remain open, to at least track further findings and warn users of the potential problem.

[fdemmer]

i think i got it... the oid is the problem and how and when it is determined... and maybe a little how the tests are set up... and generally it is a multi-db issue.

in test settings.py two databases are configured, but while running tests of course django creates new "test__" databases (from "template1"). however initially django connects to one of the configured "test" or "test2" databases. i don't know which. but on this initial connect the connect-signal triggers the extension registration, which determines the hstore oid and uses it to register the extension *globally_.

so the extension is now registerd with the oid from "test" or "test2". if one had added the hstore extension to "template1" before creating "test" and "test2" this would be no problem, as both would have the hstore oid copied from "template1".

so in my case i did not have "template1" set up on my notebook when i created the test databases and both have different hstore oids and tests fail. i did have it set up on my pc and the test databases have the same hstore oid and test work.

so, what does it mean!?

using unique=False works around this problem by reloading the oid for every connection. the more i think about it, this is a very, very ugly workaround.

there needs to be a way to register hstore not per connection, but per database, because databases can have different oids for the extension.

btw: this came to me while brushing my teeth, which shows once again how important getting up from in front of the screen is for solving problems :)

[niwinz]

A lot of thanks for your research. And yeah, these ideas often appear in the shower jeje.
Again, thanks! I used this text as quote on the readme for explain this bug.

[binarin]

I think there is another thing with unique=True and multiple databases, when only some of them use hstore. So when first connection is made to hstore-less databases, hstore is not registered altogether - unique handler is wasted doing nothing:

https://github.com/niwibe/djorm-ext-hstore/blob/master/djorm_hstore/models.py#L98