To help you get started, we've collected the most common ways that gokart is being used within popular public projects.

m3dev / gokart / gokart / testing / check_if_run_with_empty_data_frame.py View on Github

def _test_run_with_empty_data_frame(cmdline_args: List[str], test_run_params: test_run):
    from unittest.mock import patch

    try:
        gokart.run(cmdline_args=cmdline_args)
    except SystemExit as e:
        assert e.code == 0, f'original workflow does not run properly. It exited with error code {e}.'

    with CmdlineParser.global_instance(cmdline_args) as cp:
        all_tasks = _get_all_tasks(cp.get_task_obj())

    if test_run_params.namespace is not None:
        all_tasks = [t for t in all_tasks if t.task_namespace == test_run_params.namespace]

    with patch('gokart.TaskOnKart.load_data_frame', new=lambda *args, required_columns=None, **kwargs: pd.DataFrame(columns=required_columns)):
        with patch('gokart.TaskOnKart.dump', new=lambda *args, **kwargs: None):
            test_status_list = [_run_with_test_status(t) for t in all_tasks]

    test_logger.info('gokart test results:\n' + '\n'.join(s.format() for s in test_status_list))
    if any(s.fail() for s in test_status_list):
        exit(1)

View more ways to use gokart

gokart

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.

Documentation for the latest release is hosted on readthedocs.

About gokart

Here are some good things about gokart.

• The following meta data for each Task is stored separately in a pkl file with hash value
  • task output data
  • imported all module versions
  • task processing time
  • random seed in task
  • displayed log
  • all parameters set as class variables in the task
• Automatically rerun the pipeline if parameters of Tasks are changed.
• Support GCS and S3 as a data store for intermediate results of Tasks in the pipeline.
• The above output is exchanged between tasks as an intermediate file, which is memory-friendly
• pandas.DataFrame type and column checking during I/O
• Directory structure of saved files is automatically determined from structure of script
• Seeds for numpy and random are automatically fixed
• Can code while adhering to SOLID principles as much as possible
• Tasks are locked via redis even if they run in parallel

All the functions above are created for constructing Machine Learning batches. Provides an excellent environment for reproducibility and team development.

Here are some non-goal / downside of the gokart.

• Batch execution in parallel is supported, but parallel and concurrent execution of task in memory.
• Gokart is focused on reproducibility. So, I/O and capacity of data storage can become a bottleneck.
• No support for task visualize.
• Gokart is not an experiment management tool. The management of the execution result is cut out as Thunderbolt.
• Gokart does not recommend writing pipelines in toml, yaml, json, and more. Gokart is preferring to write them in Python.

Getting Started

Within the activated Python environment, use the following command to install gokart.

pip install gokart

Quickstart

A minimal gokart tasks looks something like this:

import gokart

class Example(gokart.TaskOnKart):
    def run(self):
        self.dump('Hello, world!')

task = Example()
output = gokart.build(task)
print(output)

gokart.build return the result of dump by gokart.TaskOnKart. The example will output the following.

Hello, world!

This is an introduction to some of the gokart. There are still more useful features.

Please See Documentation .

Have a good gokart life.

Achievements

Gokart is a proven product.

• It's actually been used by m3.inc for over 3 years
• Natural Language Processing Competition by Nishika.inc 2nd prize : Solution Repository

Thanks

gokart is a wrapper for luigi. Thanks to luigi and dependent projects!

• luigi