How to use the scrubadub.utils.Lookup function in scrubadub

To help you get started, we’ve selected a few scrubadub examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github datascopeanalytics / scrubadub / scrubadub / filth / base.py View on Github external
class Filth(object):
    """This is the base class for all ``Filth`` that is detected in dirty dirty
    text.
    """

    # this allows people to customize the output, especially for placeholder
    # text and identifier replacements
    prefix = u'{{'
    suffix = u'}}'

    # the `type` is used when filths are merged to come up with a sane label
    type = None

    # the `lookup` is used to keep track of all of the diffent types of filth
    # that are encountered across all `Filth` types.
    lookup = utils.Lookup()

    def __init__(self, beg=0, end=0, text=u''):
        self.beg = beg
        self.end = end
        self.text = text

    @property
    def placeholder(self):
        return self.type.upper()

    @property
    def identifier(self):
        # NOTE: this is not an efficient way to store this in memory. could
        # alternatively hash the type and text and do away with the overhead
        # bits of storing the tuple in the lookup
        i = self.lookup[(self.type, self.text.lower())]