PyFunc Server

PyFunc Server is Kserve extension for deploying user-defined python function model. It leverages mlflow.pyfunc model for model loading.

Usage

HTTP Server

Run following command to load sample echo-model model and start HTTP server:

PROMETHEUS_MULTIPROC_DIR=prometheus \
python -m pyfuncserver --model_dir echo-model/model

This will start http server at port 8080 which you can test using curl command

curl localhost:8080/v1/models/model-1:predict -H "Content-Type: application/json" -d '{}'

UPI V1 Server

Run following command to load sample echo-model model and start UPI v1 server:

PROMETHEUS_MULTIPROC_DIR=prometheus \
CARAML_PROTOCOL=UPI_V1 \
WORKERS=2 python -m pyfuncserver --model_dir echo-model/model

Since UPI v1 interface is gRPC then you can use grpcurl to send request

grpcurl -plaintext -d '{}'  localhost:9000 caraml.upi.v1.UniversalPredictionService/PredictValues

Development

Requirements:

• pipenv

To install locally (including test dependency):

pipenv shell
make setup

To run all test

make test

To run benchmark

make benchmark

Configuration

Pyfunc server can be configured via following environment variables

Directory Structure

├── benchmark              <- Benchmarking artifacts
├── docker                 <- Dockerfiles and environment files
    ├── Dockerfile         <- Dockerfile that will be used by kaniko to build user image in the cluster
    ├── base.Dockerfile    <- Base docker image that will be used by `Dockerfile`
├── examples               <- Examples of PyFunc models implementation
├── test                   <- Test package
├── pyfuncserver           <- Source code of this workflow
│   ├── __main__.py        <- Entry point of pyfuncserver
│   ├── config.py          <- Pyfuncserver configurations
│   ├── server.py          <- Main server that orchestrate the initialization of all server within pyfuncserver
│   ├── metrics            <- Module related to pubishing prometheus metrics
│   ├── utils              <- Utility package
│   ├── model              <- Model package
│   └── protocol           <- Protocol implementations
│       └── rest           <- Server implementation for HTTP_JSON protocol
│       └── upi            <- Server implementation for UPI_V1 protocol
├── .gitignore
├── Makefile               <- Makefile
├── README.md              <- The top-level README for developers using this project.
├── requirements.txt       <- pyfuncserver dependencies
├── setup.py               <- setup.py
├── run.sh                 <- Script to activate `merlin-model` environment and run pyfuncserver when `docker run` is invoked