How to use the blingfire.text_to_ids function in blingfire
To help you get started, we’ve selected a few blingfire examples, based on popular ways it is used in public projects.
Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.
microsoft / BlingFire / ldbsrc / bert_base_tok / test_bling.py View on Github 
# from blingfiretok import *
# load bert base tokenizer model, note one model can be used by multiple threads within the same process
# h = load_model("./bert_base_tok.bin")
h = blingfire.load_model(os.path.join(os.path.dirname(blingfire.__file__), "bert_base_tok.bin"))
for line in sys.stdin:
line = line.strip()
print(line)
#line = text_to_words(line)
#print(line)
ids = blingfire.text_to_ids(h, line, 128, 100)
print(ids)
blingfire.free_model(h)blingfire
Python wrapper of lightning fast Finite State Machine based NLP library.
