How to use the ginza.sudachi_tokenizer.SudachiTokenizer function in ginza

To help you get started, we’ve selected a few ginza examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

def create_tokenizer(cls, nlp=None):
        return SudachiTokenizer(nlp)