hiq-python

🦉 Observability And Optimization In Modern AI Era

> 🔥 HiQ now supports GPU profiling, DNN model visualization and tracing for DNN libraries like pyTorch, transformers, LAVIS, and LLMs like LLaMA, OPT, Bloom, T5 and GPT2 in addition to Onnxruntime, FastAPI and Flask.

HiQ is a declarative, non-intrusive, dynamic and transparent tracking system for both monolithic application and distributed system. It brings the runtime information tracking and optimization to a new level without compromising with speed and system performance, or hiding any tracking overhead information. HiQ applies for both I/O bound and CPU bound applications. In addition to latency tracking, HiQ provides memory, disk I/O and Network I/O tracking out of the box. The output can be saved in form of normal line by line log file, or HiQ tree, or span graph.

HiQ's philosophy is to decouple observability logic from business logic. We don't have to enter the black hole to observe it. Do you like the idea? Leave a ⭐ if you enjoy the project and welcome to say Hi to us on Slack 👋

📜HiQ Pape: A Declarative, Non-intrusive, Dynamic and Transparent Observability and Optimization System

Installation

• Basic Installation

pip install hiq-python

• HiQ also supports extra installation

pip install hiq-python[fastapi] # To support fastapi web server online tracing
pip install hiq-python[gpu]     # To support GPU tracing, which will install pynvml
pip install hiq-python[lavis]   # To support Salesforce LAVIS Vision Language models
pip install hiq-python[transformers] # To support tracing Hugging Face's transformers library
pip install hiq-python[full]         # To support all the cases, and this will install all the dependency libraries

Get Started

Let start with a simplest example by running HiQ against a simple monolithic python code 📄 main.py:

# this is the main.py python source code
import time

def func1():
    time.sleep(1.5)
    print("func1")
    func2()

def func2():
    time.sleep(2.5)
    print("func2")

def main():
    func1()

if __name__ == "__main__":
    main()

In this code, there is a simple chain of function calls: main() -> func1 -> func2.

Now we want to trace the functions without modifying its code. Let's run the following:

git clone https://github.com/oracle-samples/hiq.git
cd hiq/examples/quick_start
python main_driver.py

If everything is fine, you should be able to see the output like this:

From the screenshot we can see the timestamp and the latency of each function:

HiQ just traced the main.py file running without touching one line of its code.

Documentation

HTML: 🔗 HiQ Online Documents | PDF: Please check 🔗 HiQ User Guide.

Logging: https://hiq.readthedocs.io/en/latest/4_o_advanced.html#log-monkey-king
Tracing: https://hiq.readthedocs.io/en/latest/5_distributed.html

• Zipkin: https://hiq.readthedocs.io/en/latest/5_distributed.html#zipkin
• Jaeger: https://hiq.readthedocs.io/en/latest/5_distributed.html#jaeger

Metrics:

• Prometheus: https://hiq.readthedocs.io/en/latest/7_integration.html#prometheus

Streaming:

• Kafka: https://hiq.readthedocs.io/en/latest/7_integration.html#oci-streaming

DNN Model Observability & Visualization

HiQ can visualize DNN model. To get the following BERT model's structure, you can just run:

python -m hiq.vis

The graph is self-explantory. There are several conventions:

• ❄️ means frozen layer, where requires_grad is false.
• 📈 means gradient exists for that model parameter, which usually happens after backpopulation.
• +, bold font, and underscored dotted line mean the displayed layer is a folded version of multiple layers with the same structure.

What you need to do is just calling print_model(model) in your code. Refer to: here for how to use it.

HiQ Web UI

• Main Page

• Latency Details

Jupyter NoteBook

HiQ was originally developed to find Onnxruntime performance bottleneck in DNN inference, and it works well for other computation intensive applications too. The following are two examples.

Add Observability to PaddlePaddle (PaddleOCR)

• Latency

• Memory

• Latency Gantt Chart

• HiQ Call Graph

Add Observability to Onnxruntime (AlexNet)

• Latency
• Intrusive

Examples

Please check 🔗 examples for usage examples.

Contributing

HiQ welcomes contributions from the community. Before submitting a pull request, please review our contribution guide](./CONTRIBUTING.md).

Security

Please consult the 🔗 security guide for our responsible security vulnerability disclosure process.

License

Copyright (c) 2022, 2023 Oracle and/or its affiliates. Released under the Universal Permissive License v1.0 as shown at .

Presentation and Demos

• Introduction to Observability with HiQ