split_apostrophe_unanalyzable

scripts.fix_tokenization.split_apostrophe_unanalyzable(i, terminals, preterminals, preterminal_parents, state)[source]

Split apostrophe as unanalyzable. x’xxx -> x’ xxx. use when the original token is unanalyzable.