expand_to_neighboring_punct¶

scripts.fix_tokenization.expand_to_neighboring_punct(i, is_puncts)[source]¶

>>> expand_to_neighboring_punct(0, [False, True, True])
(0, 3)
>>> expand_to_neighboring_punct(2, [True, True, False])
(0, 3)
>>> expand_to_neighboring_punct(1, [False, False, False])
(1, 2)