strip unicode whitespace #4

gaborbernat · 2018-07-18T10:22:58Z

At the moment the parser uses the built-in string strip function (https://github.com/RonnyPfannschmidt/iniconfig/blob/master/iniconfig.py#L134). Nowadays is more and more common that people inadvertently introduce Unicode whitespaces in their configuration. The string strip function does not remove this. When we detect such we should at least warn (especially on the left-hand side), or fail. Failing to strip such characters is never the desired effect, as it effectively generates a new key that visually seems the same leaving the user in confusion of why his config does not work.

Note the py package uses this package vendored, and via that pytest/tox too (https://github.com/pytest-dev/py/blob/master/py/_vendored_packages/iniconfig.py).

Issues generated by this omission:

Using two sets of {a, b, ...} in envlist doesnt show enviroments in tox -l tox-dev/tox#915

Detecting unicode spaces https://stackoverflow.com/questions/8921365/in-python-how-to-list-all-characters-matched-by-posix-extended-regex-space/37903375#37903375

RonnyPfannschmidt · 2018-07-18T10:36:25Z

as is iniconfig is not unicode aware, so this one is a really tricky one to fix
its def not on my own roadmap

gaborbernat · 2018-07-18T10:41:23Z

@RonnyPfannschmidt what about doing strip as (in case the content read from the file is of type unicode):

>>> from unicodedata import name
>>> import sys
>>> import re
>>> spaces = u''.join(re.findall(r'\s', u''.join(unichr(c) for c in xrange(sys.maxunicode+1)), re.UNICODE))
>>> 'envlist '.strip(spaces)
u'envlist'
>>> name(u'\xA0')
'NO-BREAK SPACE
>>> u'envlist\xA0'.strip(spaces)
u'envlist'
>>>

gaborbernat · 2018-07-18T10:45:18Z

An alternative solution would be to try to avoid such configs by either warning when such lines are detected.

RonnyPfannschmidt · 2018-07-18T10:55:26Z

i am fine with any consistent solution - i just don't dont have the time/motivation to do the complete dance of ensuring its consistent

right now iniconfig is not unicode aware and works in terms of native strings and ascii whitespace

note that your example generates the spaces in a pretty expensive manner

This package supports Python 3.x support, so allow it accordingly. It is required for an upcoming www/py-autobahn update (Python 2/3 compatible). During QA, a UnicodeDecodeError was observed running tests under Python 3: File "/usr/local/lib/python3.6/site-packages/py/_vendored_packages/iniconfig.py", line 82, in _parse for lineno, line in enumerate(line_iter): File "/usr/local/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 165: ordinal not in range(128) pytest uses the py package, which vendors the iniconfig package, which isn't unicode aware [1][2][3]. Patch out unicode characters from setup.cfg accordingly until it's resolved. While investigating the cause of the above issue, a fix for setup.cfg's encoding was identified, which removes the need to set the locale via USE_LOCALE so remove it accordingly. While I'm here: - Pet portlint, spurious space at end of line in pkg-descr - Add LICENSE_FILE/TEST_DEPENDS/test target/NO_ARCH [1] pytest-dev/pytest#3799 [2] pytest-dev/iniconfig#5 [3] pytest-dev/iniconfig#4 portlint: OK (looks fine.) porttest: OK (poudriere: 12amd64{py36,py27}) maketest: 215 passed, 1 skipped in 3.29 seconds (Python 2.7) maketest: 209 passed, 7 skipped in 3.07 seconds (Python 3.6) Approved by: portmgr (blanket: ports/framework compliance) MFH: 2019Q2 git-svn-id: svn+ssh://svn.freebsd.org/ports/head@501964 35697150-7ecd-e111-bb59-0022644237b5

This package supports Python 3.x support, so allow it accordingly. It is required for an upcoming www/py-autobahn update (Python 2/3 compatible). During QA, a UnicodeDecodeError was observed running tests under Python 3: File "/usr/local/lib/python3.6/site-packages/py/_vendored_packages/iniconfig.py", line 82, in _parse for lineno, line in enumerate(line_iter): File "/usr/local/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 165: ordinal not in range(128) pytest uses the py package, which vendors the iniconfig package, which isn't unicode aware [1][2][3]. Patch out unicode characters from setup.cfg accordingly until it's resolved. While investigating the cause of the above issue, a fix for setup.cfg's encoding was identified, which removes the need to set the locale via USE_LOCALE so remove it accordingly. While I'm here: - Pet portlint, spurious space at end of line in pkg-descr - Add LICENSE_FILE/TEST_DEPENDS/test target/NO_ARCH [1] pytest-dev/pytest#3799 [2] pytest-dev/iniconfig#5 [3] pytest-dev/iniconfig#4 portlint: OK (looks fine.) porttest: OK (poudriere: 12amd64{py36,py27}) maketest: 215 passed, 1 skipped in 3.29 seconds (Python 2.7) maketest: 209 passed, 7 skipped in 3.07 seconds (Python 3.6) Approved by: portmgr (blanket: ports/framework compliance) MFH: 2019Q2

This package supports Python 3.x support, so allow it accordingly. It is required for an upcoming www/py-autobahn update (Python 2/3 compatible). During QA, a UnicodeDecodeError was observed running tests under Python 3: File "/usr/local/lib/python3.6/site-packages/py/_vendored_packages/iniconfig.py", line 82, in _parse for lineno, line in enumerate(line_iter): File "/usr/local/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 165: ordinal not in range(128) pytest uses the py package, which vendors the iniconfig package, which isn't unicode aware [1][2][3]. Patch out unicode characters from setup.cfg accordingly until it's resolved. While investigating the cause of the above issue, a fix for setup.cfg's encoding was identified, which removes the need to set the locale via USE_LOCALE so remove it accordingly. While I'm here: - Pet portlint, spurious space at end of line in pkg-descr - Add LICENSE_FILE/TEST_DEPENDS/test target/NO_ARCH [1] pytest-dev/pytest#3799 [2] pytest-dev/iniconfig#5 [3] pytest-dev/iniconfig#4 portlint: OK (looks fine.) porttest: OK (poudriere: 12amd64{py36,py27}) maketest: 215 passed, 1 skipped in 3.29 seconds (Python 2.7) maketest: 209 passed, 7 skipped in 3.07 seconds (Python 3.6) Approved by: portmgr (blanket: ports/framework compliance) MFH: 2019Q2 git-svn-id: svn+ssh://svn.freebsd.org/ports/head@501964 35697150-7ecd-e111-bb59-0022644237b5

This package supports Python 3.x support, so allow it accordingly. It is required for an upcoming www/py-autobahn update (Python 2/3 compatible). During QA, a UnicodeDecodeError was observed running tests under Python 3: File "/usr/local/lib/python3.6/site-packages/py/_vendored_packages/iniconfig.py", line 82, in _parse for lineno, line in enumerate(line_iter): File "/usr/local/lib/python3.6/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 165: ordinal not in range(128) pytest uses the py package, which vendors the iniconfig package, which isn't unicode aware [1][2][3]. Patch out unicode characters from setup.cfg accordingly until it's resolved. While investigating the cause of the above issue, a fix for setup.cfg's encoding was identified, which removes the need to set the locale via USE_LOCALE so remove it accordingly. While I'm here: - Pet portlint, spurious space at end of line in pkg-descr - Add LICENSE_FILE/TEST_DEPENDS/test target/NO_ARCH [1] pytest-dev/pytest#3799 [2] pytest-dev/iniconfig#5 [3] pytest-dev/iniconfig#4 portlint: OK (looks fine.) porttest: OK (poudriere: 12amd64{py36,py27}) maketest: 215 passed, 1 skipped in 3.29 seconds (Python 2.7) maketest: 209 passed, 7 skipped in 3.07 seconds (Python 3.6) Approved by: portmgr (blanket: ports/framework compliance) Approved by: ports-secteam (joneum, blanket: ports/framework compliance)

RonnyPfannschmidt · 2023-01-07T11:20:28Z

@gaborbernat with the new release unicode support is in but unicode whitespace is not yet considered

gaborbernat mentioned this issue Jul 18, 2018

Using two sets of {a, b, ...} in envlist doesnt show enviroments in tox -l tox-dev/tox#915

Closed

RonnyPfannschmidt added the help wanted label Jul 18, 2018

RonnyPfannschmidt mentioned this issue Aug 10, 2018

Decode error in pytest.ini with chinese characters #5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

strip unicode whitespace #4

strip unicode whitespace #4

gaborbernat commented Jul 18, 2018

RonnyPfannschmidt commented Jul 18, 2018

gaborbernat commented Jul 18, 2018 •

edited

Loading

gaborbernat commented Jul 18, 2018

RonnyPfannschmidt commented Jul 18, 2018

RonnyPfannschmidt commented Jan 7, 2023

strip unicode whitespace #4

strip unicode whitespace #4

Comments

gaborbernat commented Jul 18, 2018

RonnyPfannschmidt commented Jul 18, 2018

gaborbernat commented Jul 18, 2018 • edited Loading

gaborbernat commented Jul 18, 2018

RonnyPfannschmidt commented Jul 18, 2018

RonnyPfannschmidt commented Jan 7, 2023

gaborbernat commented Jul 18, 2018 •

edited

Loading