How to use the elasticsearch5.helpers.streaming_bulk function in elasticsearch5

To help you get started, we’ve selected a few elasticsearch5 examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github cisco / mindmeld / mindmeld / components / _elasticsearch_helpers.py View on Github external
logger.warning(
                "Elasticsearch index '%s' for application '%s' already exists!",
                index_name,
                app_namespace,
            )
            logger.info("Loading index %r", index_name)
        else:
            create_index(
                app_namespace, index_name, mapping, es_host=es_host, es_client=es_client
            )

        count = 0
        # create the progess bar with docs count
        pbar = tqdm(total=docs_count)

        for okay, result in streaming_bulk(
            es_client,
            docs,
            index=scoped_index_name,
            doc_type=doc_type,
            chunk_size=50,
            raise_on_error=False,
        ):

            action, result = result.popitem()
            doc_id = "/%s/%s/%s" % (index_name, doc_type, result["_id"])
            # process the information from ES whether the document has been
            # successfully indexed
            if not okay:
                logger.error("Failed to %s document %s: %r", action, doc_id, result)
            else:
                count += 1