How to use the fasttext.FastText.tokenize function in fasttext

To help you get started, we’ve selected a few fasttext examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github Kaggle / docker-python / tests / test_fasttext.py View on Github external
def test_tokenize(self):
        tokens = fasttext.FastText.tokenize("Hello World")

        self.assertEqual(["Hello", "World"], tokens)