View all t5 analysis

How to use the t5.data.sentencepiece_vocabulary.SentencePieceVocabulary function in t5

To help you get started, we’ve selected a few t5 examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

google-research / text-to-text-transfer-transformer / t5 / data / test_utils.py View on Github

def sentencepiece_vocab(extra_ids=0):
  return sentencepiece_vocabulary.SentencePieceVocabulary(
      os.path.join(TEST_DATA_DIR, "sentencepiece", "sentencepiece.model"),
      extra_ids=extra_ids)

google-research / text-to-text-transfer-transformer / t5 / data / utils.py View on Github

def get_vocabulary(self):
    """Returns a SentencePieceVocabulary object using the Task's model."""
    return sentencepiece_vocabulary.SentencePieceVocabulary(
        self.sentencepiece_model_path)