View all sister analysis

How to use the sister.tokenizers function in sister

To help you get started, we’ve selected a few sister examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

tofunlp / sister / tests / test_tokenizers.py View on Github

def setUp(self):
        self.tokenizer = tokenizers.SimpleTokenizer()

tofunlp / sister / tests / test_tokenizers.py View on Github

def test_tokenize_not_implemented(self):
        class Dummy(tokenizers.Tokenizer):
            pass
        with self.assertRaises(NotImplementedError):
            sentence = 'I am a dog.'
            Dummy().tokenize(sentence)