File tree Expand file tree Collapse file tree 3 files changed +5
-2
lines changed Expand file tree Collapse file tree 3 files changed +5
-2
lines changed Original file line number Diff line number Diff line change @@ -565,7 +565,7 @@ defined within the ``TOK`` class:
565
565
| S_END | 11002 | End of sentence | |
566
566
+---------------+---------+---------------------+---------------------------+
567
567
568
- (*) The token types marked with an asterisk are reserved for the Reynir package
568
+ (*) The token types marked with an asterisk are reserved for the Greynir package
569
569
and not currently returned by the tokenizer.
570
570
571
571
To obtain a descriptive text for a token kind, use
@@ -752,6 +752,8 @@ can be found in the file ``test/toktest_normal_gold_expected.txt``.
752
752
Changelog
753
753
---------
754
754
755
+ * Version 2.0.6: Fixed handling of abbreviations such as *m.v. * (*miðað við *)
756
+ that should not start a new sentence even if the following word is capitalized
755
757
* Version 2.0.5: Fixed bug where single uppercase letters were erroneously
756
758
being recognized as abbreviations, causing prepositions such as 'Í' and 'Á'
757
759
at the beginning of sentences to be misunderstood in ReynirPackage
Original file line number Diff line number Diff line change @@ -57,7 +57,7 @@ def read(*names, **kwargs):
57
57
58
58
setup (
59
59
name = "tokenizer" ,
60
- version = "2.0.5" ,
60
+ version = "2.0.6" , # Also update src/tokenizer/__init__.py
61
61
license = "MIT" ,
62
62
description = "A tokenizer for Icelandic text" ,
63
63
long_description = u"{0}\n {1}" .format (
Original file line number Diff line number Diff line change 42
42
from .abbrev import Abbreviations , ConfigError
43
43
44
44
__author__ = u"Miðeind ehf"
45
+ __version__ = u"2.0.6" # Also update setup.py
You can’t perform that action at this time.
0 commit comments