How to use the genie-toolkit.DatasetSplitter function in genie-toolkit

To help you get started, we’ve selected a few genie-toolkit examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github stanford-oval / almond-cloud / training / tasks / prepare-training-set.js View on Github external
noQuoteExpandFactor: 10,

            ppdbFile: ppdb,

            locale: this._language,
            rng: this._rng,
            debug: this._options.debug,
        });

        const train = new Genie.DatasetStringifier();
        const eval_ = new Genie.DatasetStringifier();
        const promises = [];
        promises.push(StreamUtils.waitFinish(train.pipe(this._options.train)));
        promises.push(StreamUtils.waitFinish(eval_.pipe(this._options.eval)));

        const splitter = new Genie.DatasetSplitter({
            rng: this._rng,
            locale: this._language,

            train,
            eval: eval_,

            evalProbability: this._options.evalProbability,
            forDevices: this._forDevices,
            splitStrategy: this._options.splitStrategy,
            useEvalFlag: true
        });

        source.pipe(augmenter).pipe(splitter);

        await Promise.all(promises);
    }