util.tokenize module

module_dependencies.util.tokenize.tokenize(tokens: str) → Tuple[str, ...]

Convert e.g. "nltk.tokenize" into ('nltk', 'tokenize').

Parameters:

tokens (str) – Dot-separated tokens in a string.

Return Variable:

Tuple of tokens.

Return type:

Tuple[str, …]

module_dependencies.util.tokenize.detokenize(variable: Tuple[str, ...]) → str

Convert e.g. ('nltk', 'tokenize') into "nltk.tokenize".

Parameters:

variable (Variable) – Tuple of tokens.

Return Variable:

Dot-separated tokens in a string.

Return type:

str