Using Neural Networks to solve Regression Problems

The regression Neural Networks task builds a feed-forward neural network (NN) for regression. NN is a model that includes a certain number of elementary perceptrons organized in layers. 

Each perceptron returns sigmoidal function (activation function) that depends on a weighted sum of its input. To retrieve the optimal weight vector for each perceptron a back-propagation algorithm is employed.

The output of the task is a model, consisting in a weight matrix ,that can be employed by an Apply Model task to perform the NN forecast on a set of examples.


Prerequisites

Additional tabs

  • The Monitor tab, where it is possible to view the temporal evolution of some quantities related to the network optimization. In particular, the behavior of error, gradient and misclassification is reported as a function of the number of iterations. These plots can be viewed during and after computation operations. 

  • The Weights tab, where it is possible to view the weight matrix for each hidden layer. The layer to be displayed can be selected from the Chose Layer drop down list. Each row in the spreadsheet represents a neuron, while each column corresponds to an input attribute.

  • The Results tab, where statistics on the NN computation are displayed, such as the execution time.


Procedure

  1. Drag the Neural Networks task onto the stage.

  2. Connect a task, which contains the attributes from which you want to create the model, to the new task.

  3. Double click the Neural Networks task. The left-hand pane displays a list of all the available attributes in the dataset, which can be ordered and searched as required.

  4. Configure the options described in the table below.

  5. Save and compute the task.

Neural Networks options

Parameter Name

Description

Number of hidden layers

The number of hidden layers in the network.

Neurons for each hidden layer

The number of neurons in each hidden layer. This option is enabled only if the value specified for the Number of hidden layers option is greater than zero.

Learning rate

Indicate the coefficient of the gradient descent. For high values it is likely that the algorithm converges more quickly, but it may lack in precision. Alternatively, with low learning rates, the algorithm will converge more slowly, but with more accuracy.

Maximum number of iterations

The (overall) maximum number of iterations for the gradient descent algorithm.

Momentum term

Specify the momentum of the back-propagation algorithm. Momentum adds a fraction (between 0 and 1) of the previous weight update to prevent the system from converging to a local minimum.

Number of trials

Select the number of trials, i.e. the repeated execution of back-propagation algorithm, with different starting conditions.

Normalization for input variables

The type of normalization to use when treating ordered (discrete or continuous) variables.

Possible methods are:

  • None: no normalization is performed (default)

  • Normal: data are normalized according to the Gaussian distribution, where μ is the average of and σ is its standard deviation: 

     

  • Minmax [0,1]: data are normalized to be comprised in the range [0,1]:

     

  • Minmax [-1, 1]: data are normalized to be included in the range [-1, 1]:

     

Every attribute can have its own value for this option, which can be set in the Data Manager task. These choices are preserved if Attribute is selected in the Normalization of input variables option; otherwise any selections made here overwrite previous selections made.

Maximum number of iterations for any trial

The maximum number of iterations allowed for the gradient descent algorithm for each trial.

Normalization of output variables

Select the method for normalizing output variables, in the same way as the Normalization for input variables

Use symmetric activation function

If selected, use an activation function in the range (-1,1) instead of (0,1).

Aggregate data before processing

If selected, identical patterns are aggregated and considered as a single pattern during the training phase.

Initialize random generator with seed

If selected, a seed, which defines the starting point in the sequence, is used during random generation operations. Consequently using the same seed each time will make each execution reproducible. Otherwise, each execution of the same task (with same options) may produce dissimilar results due to different random numbers being generated in some phases of the process.

Append results

If selected, the results of this computation are appended to the dataset, otherwise they replace the results of previous computations.

Input attributes

Drag and drop here the input attributes you want to use to build the network. Instead of manually dragging and dropping attributes, they can be defined via a filtered list.

Output attributes

Drag and drop here the attributes you want to use to build the network. Instead of manually dragging and dropping attributes, they can be defined via a filtered list.


Example

The following example uses the Adult dataset.

Description

Screenshot

  • After having imported the dataset with the Import from Text File task and splitting the dataset into test and training sets (30% test and 70% training) with the Split Data task, add a Neural Networks Regression to the flow and configure the following options:

    • Number of hidden layers: 1

    • Neurons for each hidden layer: 5

    • Maximum number of iterations: 500

    • Maximum number of iterations for any trial: 500

    • Input attributes: age, workclass, education, relationship, race

    • Output attributes: hours-per-week

  • Save and compute the task.

The execution of the task can be monitored in the Monitor tab of the NN task. 

In these plots the behavior of the error as a function of the iteration is showed.

Analogous plots can be visualized for covering and error by clicking on the corresponding tabs. 

Clicking on the Weights tab at the end of the execution, displays the weights obtained through the optimization procedure.

The layer to be visualized can be selected in the Chose layer drop-down list:

Each row corresponds to a neuron of that layer and contains the weights relative to each input attribute. The first column contains the bias, i.e. the constant term in the linear combination of inputs.

Add an Apply Model task to the stage to apply the model to data

The forecast ability of the generated model can be viewed by adding an Apply Model task to the NN task, and computing with default options.

The forecast produced can be analyzed by right-clicking the Apply Model task and selecting Take a look.

In the data table the following columns relative to the results of NN elaboration have been added:

  • the NN output forecast: pred(hours-per-week)

  • the error, i.e. the difference between the real output and the predicted output; if the actual output is missing, the field is left empty: err(hours-per-week)