Whitespace Tokenizer

class py_stringmatching.tokenizer.whitespace_tokenizer.WhitespaceTokenizer(return_set=False)

Segments the input string using whitespaces then returns the segments as tokens.

Currently using the split function in Python, so whitespace character refers to the actual whitespace character as well as the tab and newline characters.

Parameters:return_set (boolean) – A flag to indicate whether to return a set of tokens instead of a bag of tokens (defaults to False).

return_set

An attribute to store the flag return_set.

Type:boolean

get_delim_set()

Gets the current set of delimiters.

Returns:A Python set which is the current set of delimiters.

get_return_set()

Gets the value of the return_set flag.

Returns:The boolean value of the return_set flag.

set_delim_set(delim_set)

Sets the current set of delimiters.

Parameters:delim_set (set) – A set of delimiter strings.

set_return_set(return_set)

Sets the value of the return_set flag.

Parameters:return_set (boolean) – a flag to indicate whether to return a set of tokens instead of a bag of tokens.

tokenize(input_string)

Tokenizes input string based on white space.

Parameters:input_string (str) – The string to be tokenized. Returns:A Python list, which is a set or a bag of tokens, depending on whether return_set is True or False. Raises:TypeError – If the input is not a string.

Examples

>>> ws_tok = WhitespaceTokenizer()
>>> ws_tok.tokenize('data science')
['data', 'science']
>>> ws_tok.tokenize('data        science')
['data', 'science']
>>> ws_tok.tokenize('data   science')
['data', 'science']
>>> ws_tok = WhitespaceTokenizer(return_set=True)
>>> ws_tok.tokenize('data   science data integration')
['data', 'science', 'integration']