How to use the normality.encoding.normalize_result function in normality

To help you get started, we’ve selected a few normality examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github occrp-attic / ingestors / ingestors / support / encoding.py View on Github external
def read_file_decoded(self, file_path):
        encoding = self.result.encoding
        with open(file_path, 'rb') as fh:
            body = fh.read()
            if encoding is None:
                result = chardet.detect(body)
                encoding = normalize_result(result, self.DEFAULT_ENCODING)

        try:
            body = body.decode(encoding)
            if encoding != self.DEFAULT_ENCODING:
                log.info("Decoding [%s] as: %s", self.result, encoding)
            return body
        except UnicodeDecodeError as ude:
            raise ProcessingException('Error decoding file as %s: %s' %
                                      (encoding, ude))
github occrp-attic / ingestors / ingestors / support / encoding.py View on Github external
def detect_list_encoding(self, items, default=DEFAULT_ENCODING):
        detector = chardet.UniversalDetector()
        for text in items:
            if not isinstance(text, bytes):
                continue
            detector.feed(text)
            if detector.done:
                break

        detector.close()
        return normalize_result(detector.result, default)
github alephdata / aleph / services / ingest-file / ingestors / support / encoding.py View on Github external
def detect_list_encoding(self, items, default=DEFAULT_ENCODING):
        detector = chardet.UniversalDetector()
        for text in items:
            if not isinstance(text, bytes):
                continue
            detector.feed(text)
            if detector.done:
                break

        detector.close()
        return normalize_result(detector.result, default)