DataPoint

> JavaScript Utility for collecting, processing and transforming data.

DataPoint helps you reason with and streamline your data processing layer. With it you can collect, process, and transform data from multiple sources, and deliver the output in a tailored format to your end consumer.

Prerequisites

Node v8 LTS or higher

Installing

npm install --save data-point

Table of Contents

Getting Started

DataPoint provides the following mechanisms for transforming data:

Reducers - these are the simplest transformations; think of them as DataPoint "primitives"
Entities - these are more complex transformations that are defined using one or more reducers
Middleware - middleware functions give the user more control when resolving entities; they're useful to implement caching and other metatasks

The following examples demonstrate some of these concepts. For detailed API documentation, you can jump into the DataPoint.create section and move from there.

Additionally, there is a Hello World YouTube tutorial that explains the basics of DataPoint.

Hello World Example

Trivial example of transforming a given input with a function reducer.

const DataPoint = require('data-point')
// create DataPoint instance
const dataPoint = DataPoint.create()

// function reducer that concatenates
// accumulator.value with 'World'
const reducer = (input) =&gt; {
  return input + ' World'
}

// applies reducer to input
dataPoint
  .resolve(reducer, 'Hello')
  .then((output) =&gt; {
    // 'Hello World'
    console.log(output) 
  })

Example at: examples/hello-world.js

Fetching remote services

Based on an initial feed, fetch and aggregate results from multiple remote services.

Example

Using the amazing swapi.co service, the example below gets information about a planet and the residents of that planet.

const DataPoint = require('data-point')

// create DataPoint instance
const dataPoint = DataPoint.create()

const {
  Request,
  Model,
  Schema,
  map
} = DataPoint

// schema to verify data input
const PlanetSchema = Schema('PlanetSchema', {
  schema: {
    type: 'object',
    properties: {
      planetId: {
        $id: '/properties/planet',
        type: 'integer'
      }
    }
  }
})

// remote service request
const PlanetRequest = Request('Planet', {
  // {value.planetId} injects the
  // value from the accumulator
  // creates: https://swapi.co/api/planets/1/
  url: 'https://swapi.co/api/planets/{value.planetId}'
})

const ResidentRequest = Request('Resident', {
  // check input is string
  inputType: 'string',
  url: '{value}'
})

// model entity to resolve a Planet
const ResidentModel = Model('Resident', {
  inputType: 'string',
  value: [
    // hit request:Resident
    ResidentRequest,
    // extract data
    {
      name: '$name',
      gender: '$gender',
      birthYear: '$birth_year'
    }
  ]
})

// model entity to resolve a Planet
const PlanetModel = Model('Planet', {
  inputType: PlanetSchema,
  value: [
    // hit request:Planet data source
    PlanetRequest,
    // map result to an object reducer
    {
      // map name key
      name: '$name',
      population: '$population',
      // residents is an array of urls
      // eg. https://swapi.co/api/people/1/
      // where each url gets mapped
      // to a model:Resident
      residents: ['$residents', map(ResidentModel)]
    }
  ]
})

const input = {
  planetId: 1
}

dataPoint.resolve(PlanetModel, input)
  .then((output) =&gt; {
    console.log(output)
    /*
    output -&gt; 
    { 
      name: 'Tatooine',
      population: 200000,
      residents:
      [ 
        { name: 'Luke Skywalker', gender: 'male', birthYear: '19BBY' },
        { name: 'C-3PO', gender: 'n/a', birthYear: '112BBY' },
        { name: 'Darth Vader', gender: 'male', birthYear: '41.9BBY' },
        ...
      ] 
    }
    */
  })

Example at: examples/full-example-instances.js

API

create

Static method that creates a DataPoint instance.

SYNOPSIS

DataPoint.create([options])

Arguments

Argument	Type	Description
options	`Object` (optional)	This parameter is optional, as are its properties (values, entities, and entityTypes). You may configure the instance later through the instance's API.

The following table describes the properties of the options argument.

Property	Type	Description
values	`Object`	Hash with values you want exposed to every Reducer
entities	`Object`	Application's defined entities
entityTypes	`Object`	Custom Entity Types

RETURNS

DataPoint instance.

DataPoint.create example

Create a DataPoint object without configuring options

const DataPoint = require('data-point')
const dataPoint = DataPoint.create()

Create the DataPoint object and set options argument:

const DataPoint = require('data-point')
const dataPoint = DataPoint
  .create({
    values: {
      foo: 'bar'
    },
    entities: {
      'reducer:HelloWorld': (input) =&gt; {
        return `hello ${input}!!`
      }
    }
  })

createReducer

Static method that creates a reducer, which can be executed with resolve or transform.

SYNOPSIS

DataPoint.createReducer(source:*, [options:Object]):Reducer

ARGUMENTS

Argument	Type	Description
source	`*`	Spec for any of the supported Reducer types
options	`Object`	Optional config object

Options

Property	Type	Description
default	`*`	Default value for the reducer. Setting this value is equivalent to using the withDefault reducer helper.

resolve

Execute a Reducer against an input value. This function supports currying and will be executed when at least the first 2 parameters are provided.

SYNOPSIS

dataPoint.resolve(reducer:Reducer, input:*, options:Object):Promise(output:*)

This method returns a Promise with the final output value.

ARGUMENTS

Argument	Type	Description
reducer	Reducer	Reducer that manipulates the input.
input	`*`	Input value that you want to transform. If none, pass `null` or empty object `{}`.
options	`Object`	Options within the scope of the current transformation. More details available here.

EXAMPLES:

Hello World example.
With options example.

transform

This method is similar to dataPoint.resolve. The differences between the methods are:

.transform() accepts an optional third parameter for node style callback.
.transform() returns a Promise that resolves to the full Accumulator object instead of accumulator.value. This may come in handy if you want to inspect other values from the transformation.

SYNOPSIS

// as promise
dataPoint.transform(reducer:Reducer, input:*, options:Object):Promise(acc:*)
// as nodejs callback function
dataPoint.transform(reducer:Reducer, input:*, options:Object, done:Function)

This method will return a Promise if done is omitted.

ARGUMENTS

Argument	Type	Description
reducer	Reducer	Reducer that manipulates the input.
input	`*`	Input value that you want to transform. If none, pass `null` or empty object `{}`.
options	`Object`	Options within the scope of the current transformation
done	`function` (optional)	Error-first Node.js style callback with the arguments `(error, accumulator)`. The second parameter is an Accumulator object where `accumulator.value` is the actual result of the transformation.

transform options

The following table describes the properties of the options argument.

Property	Type	Description
locals	`Object`	Hash with values you want exposed to every reducer. See example.
trace	`boolean`	Set this to `true` to trace the entities and the time each one is taking to execute. Use this option for debugging.

resolveFromAccumulator

Execute a Reducer from a provided Accumulator. This function will attempt at resolving a reducer providing an already constructed Accumulator Object. It will take the value provided in the Accumulator object to use as the input.

SYNOPSIS

dataPoint.resolveFromAccumulator(reducer:Reducer, acc:Accumulator):Promise(acc:Accumulator)

This method returns a Promise with the final output value.

ARGUMENTS

Argument	Type	Description
reducer	Reducer	Reducer that manipulates the input.
acc	Accumulator	Reducer's accumulator Object. The main property is `value`, which is the value the reducer will use as its input.

addEntities

This method adds new entities to a DataPoint instance.

SYNOPSIS

When defining new entities, must refer to either a [built-in type](#entity-types) like `'model'` or a [custom entity type](#custom-entity-types). should be unique for each type; for example, model:foo and hash:foo can both use the foo ID, but an error is thrown if model:foo is defined twice.

dataPoint.addEntities({
  ':': { ... },
  ':': { ... },
  ...
})

OPTIONS

Part	Type	Description
EntityType	`string`	valid entity type
EntityId	`string`	unique entity ID

addValue

Stores any value to be accessible via Accumulator.values. This object can also be set by passing a values property to DataPoint.create.

SYNOPSIS

dataPoint.addValue(objectPath, value)

ARGUMENTS

Argument	Type	Description
objectPath	`string`	object path where you want to add the new value. Uses _.set to append to the values object
value	`*`	anything you want to store

Accumulator

This object is passed to reducers and middleware callbacks; it has contextual information about the current transformation or middleware that's being resolved.

The accumulator.value property is the current input data. This property should be treated as a read-only immutable object. This helps ensure that your reducers are pure functions that produce no side effects. If the value is an object, use it as your initial source for creating a new object.

Properties exposed:

Key	Type	Description
value	`Object`	Value to be transformed.
initialValue	`Object`	Initial value passed to the entity. You can use this value as a reference to the initial value passed to your Entity before any reducer was applied.
values	`Object`	Access to the values stored via dataPoint.addValue.
params	`Object`	Value of the current Entity's params property. (for all entities except Reducer)
locals	`Object`	Value passed from the `options` argument when executing dataPoint.transform.
reducer	`Object`	Information relative to the current Reducer being executed.
debug	`Function`	debug method with scope `data-point`

Reducers

Reducers are used to transform values asynchronously. DataPoint supports the following reducer types:

Path Reducer

A path reducer is a string that extracts a path from the current Accumulator value (which must be an Object). It uses lodash's _.get behind the scenes.

SYNOPSIS

'$[.|..|

Most popular functions for data-point

To help you get started, we’ve selected a few data-point examples, based on popular ways it is used in public projects.

data-point.create

data-point.Model