How to use the wordfreq.TOKEN_RE.search function in wordfreq
To help you get started, we’ve selected a few wordfreq examples, based on popular ways it is used in public projects.
Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.
LuminosoInsight / wordfreq / wordfreq_builder / wordfreq_builder / tokenizers.py View on Github 
def lowercase_text_filter(token):
"""
If this looks like a token that we want to count, return it, lowercased.
If not, filter it out by returning None.
"""
if TOKEN_RE.search(token):
return token.lower()
else:
return Nonewordfreq
Look up the frequencies of words in many languages, based on many sources of data.
Package Health Score
56 / 100Popular wordfreq functions
- wordfreq.available_languages
- wordfreq.cB_to_freq
- wordfreq.config
- wordfreq.config.DB_FILENAME
- wordfreq.config.RAW_DATA_DIR
- wordfreq.get_frequency_dict
- wordfreq.iter_wordlist
- wordfreq.language_info.get_language_info
- wordfreq.lossy_tokenize
- wordfreq.preprocess.preprocess_text
- wordfreq.print_wordfreq
- wordfreq.simple_tokenize
- wordfreq.TOKEN_RE.search
- wordfreq.tokenize
- wordfreq.tokens.lossy_tokenize
- wordfreq.top_n_list
- wordfreq.util.standardize_word
- wordfreq.word_frequency
- wordfreq.wordfreq
- wordfreq.zipf_frequency