etl-pipeline-runner

etl-pipeline-runner

Extract Transform Load (ETL) pipeline runner is a simple, yet effective python package to run ETL-Pipelines for Data sceince projects. The sole purpose of this package is to:

Extract data from a source --> Transform the data according to the necessity --> Load data to a Database

Installation

Install the library with pip:

    pip install etl-pipeline-runner

Usage

Run an ETL Pipeline that extracts data from kaggle and stores it in a SQLite Database.

Data source: https://www.kaggle.com/datasets/edenbd/150k-lyrics-labeled-with-spotify-valence

Destination: Under songs table of project.sqlite Database. Suppose the database is located or will be created in /data directory.

Example code:

1. Import the following services from etl_pipeline_runner

from etl_pipeline_runner.services import (
    ETLPipeline,
    DataExtractor,
    CSVHandler,
    SQLiteLoader,
    ETLQueue,
)

2. Create a function that defines the transformation you want to perform on the dataset before loading to the Database. The function signature must match the following. Here pd refers to pandas.

    def transform_songs(data_frame: pd.DataFrame):
        data_frame = data_frame.drop(columns=data_frame.columns[0], axis=1)
        data_frame = data_frame.rename(columns={"seq": "lyrics"})
        return data_frame

3. Create an object of the SQLiteLoader.

    DATA_DIRECTORY = os.path.join(os.getcwd(), "data")
    songs_loader = SQLiteLoader(
        db_name="project.sqlite",
        table_name="song_lyrics",
        if_exists=SQLiteLoader.REPLACE,
        index=False,
        method=None,
        output_directory=DATA_DIRECTORY,
    )

4. Create an object of the CSVHandler.

    songs_dtype = {
        "#": "Int64",
        "artist": str,
        "seq": str,
        "song": str,
        "label": np.float64,
    }

    songs_csv_handler = CSVHandler(
        file_name="labeled_lyrics_cleaned.csv",
        sep=",",
        names=None,
        dtype=songs_dtype,
        transformer=transform_songs,
        loader=songs_loader,
    )

5. Create an object of the DataExtractor.

    songs_extractor = DataExtractor(
        data_name="Song lyrics",
        url="https://www.kaggle.com/datasets/edenbd/150k-lyrics-labeled-with-spotify-valence",
        type=DataExtractor.KAGGLE_ARCHIVE,
        file_handlers=(songs_csv_handler,),
    )

6. Create an object of ETLPipeline.

    songs_pipeline = ETLPipeline(
        extractor=songs_extractor,
    )

7. Finally run the pipeline:

    if __name__ == "__main__":
        ETLQueue(etl_pipelines=(songs_pipeline,)).run()

Setting-up credentials for KAGGLE Datasource

If your data source is kaggle, you need api key to download the dataset. etl-pipeline-runner uses opendatasets for donwloading dataset from Kaggle.
Following step will guide you to setup kaggle credentials.

1. Go to https://kaggle.com/me/account (sign in if required).
2. Scroll down to the "API" section and click "Create New API Token".
3. This will download a file kaggle.json with the following contents:

    {"username":"YOUR_KAGGLE_USERNAME","key":"YOUR_KAGGLE_KEY"}

4. You can either put the credentials in your root directory as kaggle.json or enter your username and key in terminal when asked.

Services explained

1. SQLiteLoader

Parameters description:

2. CSVHandler

Parameters description:

3. DataExtractor

Parameters description:

4. ETLPipeline

Parameters description:

5. ETLQueue

Parameters description:

Contributing

This is an open source project and I welcome contributions. Please create an issue first and make a feature branch assiciated to the issue.

Local Development Setup

1. Clone the repository:

    git clone git@github.com:prantoamt/etl-pipeline-runner.git

2. Install pdm package manager based on your local environment: https://pdm-project.org/latest/

3. Go to the project directory and install the requirements using pdm:

    pdm install

4. Open up the project in VS code, make your changes and create a pull request with proper description.