The Structure

This is a column of numbers which represents the number of neurones for each layer. The minumum is 2 rows, input and output. The most common is 3 rows, with a single middle layer. In theory there can be any number of middle layers, and there can be any number of neurones in each layer. The following general approach can be used, although each network is unique, and some trial and error may be necessary

• The larger the number of input and output, the greater the complexity of patterns in the training data, the more where data values are away from 0 and 1, and the more where similar patterns of input values are related to different outcome values, then the larger number of neurones (in terms of layers or neurones per layer) is required.
• Where the model is similar to regression, with linear relationships between inputs and outputs, only 2 layers (input and outputs) are necessary.
• Where a limited number of cause and effect patterns are clearly represented in the training data, one, or at most 2 middle layers should suffice. Where many or unrecognized patterns exists in the training data, such as training a network to predict share prices, many layers and neurones, requiring high speed computers with dedicated processing over a long time, are required
• The number of neurones in the middle layers can be determined by trial and error. although not obligatory, it is useful to have at least the same number of neurone as inputs in the first middle layer, and more neurones than the number of outputs in the last middle layer.
• In our example (a simple XOR simulator plus a switch), there is 1 middle layer which contains 4 neurone, 1 more than the number of inputs and 3 more than the outputs. The example net will still train with 2 or 3 neurones in the middle layer, but requires many more iterations to reach the same precision.

The values representing the structure are placed in the Network Structure text area. In our example, there are 3 inputs and one output. The middle layer contains 4 neurones.

Data : Input and Output

The data is a table of numbers representing the desired pattern. It can have any number of rows, but the number of columns is the number of inputs (first row in structure) plus the number of outputs (last row in the structure). For training, the number of columns must conformed to input + output. To use a trained net to interpret a set of data, only the input columns are required.

All data used by the backpropagation, be they input parameters or result output, represents Fuzzy Logic, and numerically represented as values between 0 for false, and 1 for True. Real data will therefore need to be edited to conform with this. Fuzzy Logic is discussed in the introduction panel, and will not be elaborated here.

• The simplest binary groups (no/yes, false/true, female/male). 0 for false and 1 for true can used
• For multiple groups, the easiest is to have an input for each group. For example 1 0 0 0 for group A, 0 1 0 0 for group B, 0 0 1 0 for group C and 0 0 0 1 for group D.
  A more abbreviated set of dummy variable can also be used. For example 0 0 for group A, 0 1 for group B, 1 0 for group C and 1 1 for group D. This makes for a smaller neural net with shorter training runs, but the results are intuitively more difficult to use, as group names will need to be firstly converted into a different format
• Measurements must be transformed to a conceptual value between 0 and 1 for false and true. We will use height of a person to demonstrate 3 common methods of transformation, using 155cms for short, 170cm for tall. One of the following options can be used
  • The use of a cut off to transform into two input values. Those with 155cms or less would be 1 0, 170cms or more 0 1, in between 0 0, and 1 1 does not exist.
  • Using a straight line gradient as a single value. 155cms or less = 0, 160cms or more = 1, and the rest (ht-155) / (170-155)
  • Conversion to Fuzzy Logic values using logistic transformation, which clusters values near the extremes and stretching the distances between value in between, producing a bi-modal distribution of probability values between 0 and 1. In our height example, 155cms is given the probability of 0.05 and 170cms 0.95 for the transform. Logic of Fuzzy Logic is discussed in the Introduction panel of this page, and will not be further elaborated here.

Training Schedule These are parameters that controls the speed and precision of training the neural network. They do not matteer much when the training data is brief, conceptually clear, and all the values are close to 0 and 1 (as in the example). They become increasingly important when the training data is large, where the values are heterogenous, and when the same set of inputs are linked to different outcomes

• The learning rate is a value between 0 where no learning occurs and 1 when the weights in the neurones are fully corrected by the values of the error found. When the training set is simple, the same training rate can be used throughout. When the training data is complex, with values not close to 0 and 1, and where the same input are related to different outcomes, there is a need to reduce the learning rate as training progresses, so that the result would converge better. In most cases users should adjust these parameters by trial and error. They do not affect the final outcome of training, but governs the speed (number of iterations) required.
  • Maximum learning rate is the rate set for the start of the training.The value 1 can be used, but this tends to over-correct and thus prolong training. In most cases it should begins at about 0.8 (the default setting).
  • Minimum rate is the smallest training rate used, and should be the same or lower value as the acceptable error
  • Decrement is the increment of decrease (proportion of the current training rate) as the training progresses. The value is usually set at 0.5 to 0.1, although smaller values can also be used.
  • Number of iteration (of the training data) per decrement is the rate at which the training rate is reduced. Given that most backprogagation training requires 500 to 50,000 cycles, this can be set at about 1/10th of the expected number of cycles required.
• Acceptable Error is the error acceptable to end the training. Neural net are base on Fuzzy Logic, where false(0) and true (1) are unattainable extremes, so the user has to determine how close to 0 and 1 they would accept. The default is set to 0.05, meaning that outcomes >= 0.95 is accepteable as 1 and <= 0.05 is accepted as 0. In many cases, especially when the training data is complex, this level of precision is unattainable. Also there is a need to avoid over-training, as the neural net will then model the trivial variations in the data. For practical reasons, a precision closer than 0.2 is considered workable and 0.1 as precise.
• Maximum iteration. Training will cease when the acceptable error is attained or when the maximum number of iterations is reached. Maximum iteration is required to stop training, if the required precision is not be attainable, or the duration of training exceeds the time allowed by the browser.

The neural net text area contains the neural net, a table of coefficients, each row containing the coefficients of a neurone. When the program begins, this area is blanked, as no neural network as yet exists, and the program creates the neural net using random numbers.

If the neural net already exists, either because the user paste it in the text area, or if some training has already occurs, the coefficients are used and further modified if more training is performed

The program produces coefficients to 10 decimal places, in excess to precision requirements in most cases, but allowing users to truncate them as preferred. In general, the number of decimal places should be 1 or 2 more than the precision of results required by the user. In our example, truncation coefficients to 3 decimal places will produce the same results.

The Default Example

The default example is a backpropagation network with 3 layers

• In the Structure box, the 3 rows are 3 for 3 inputs, 4 for 4 neurones in a single middle layer, and 1 for a single output
• In the Data Matrix text area is the training data, which demonstrates a decision making algorithm based on the XOR pattern, which cannot be otherwise computed numerically.
  • There are 4 columns. The first 3 are the inputs, and the last the output
  • In the first 4 rows, where the value in the third volumn is 0, if both the values in the first two columns are both 0 or both 1, the network should return the value of 0. If the values of the first two columns are different (0 1 or 1 0), then the return value is 1
  • In the last 4 rows, where the value in the third column is 1, the return values are reversed.
  • This the return values produced depends 2 patterns,
    • Are A and B both true or both false(1 1 or 0 0) or are they opposite to each other (0 1 or 1 0)
    • Is the third input represent true (1) or false (0)

Suggestions for training The following schedules are suggested to help users not familiar with the program

1. Make sure the neural net structure and the training data are compatible, the the number of columns in the training data is the number of input plus the number of outputs.
2. Leave the default settings for training, but initially set a low value (e.g. 1000) for maximum iterations
3. Click the Commence Training button to train the network
4. At the end of initial traing, examine the neural net produced. Adjust training parameters, and click the Commence Training button again. This will use the existing neural net and further modifying it.
5. Repeat adjusting and re-training until the required solution is obtains, or when no further improvement is possible

Using the Trained Neural Net This page also provides a platform for using the Bckpropagation network once it is trained.

• Requirements The structure of the network must be stated in the Network Structure text area, and the trained network in the Neural Net Matrix text area. The two sets of numbers must be compatiable, in that the number of inputs and outputs are the same
• Clicking the Produce Program button will produce a Jaascript function that will calculate the output from a row of inputs. This program can be copied and pasted into any web page that acts as an interpreter of the neural net, or as a basis to write a function in any other computer language in another application.
• To Calculate Results a set of data is required in the Data Matrix text box. For interpretation, only the input values are required (the rest of the columns will be ignored). Clicking the Calculate Results will produce a table , each row containing the input values, followed by the result output values.

Exporting the Trained Neural Net The trained neural net can be exported as a html program

• Requirements The structure of the network must be stated in the Network Structure text area, and the trained network in the Neural Net Matrix text area. The two sets of numbers must be compatiable, in that the number of inputs and outputs are the same
• Clicking the Export Trained Neuralnet button will produce the source code for a complete html web page, including the input/output interface and the Javascript program representing the trained neural network.
• The source code can be copied as pasted into a text editor, and saved as an html file. The html file can then be used via a web browser to interpret future data.

Data Matrix
0 0 0 0 0 1 0 1 1 0 0 1 1 1 0 0 0 0 1 1 0 1 1 0 1 0 1 0 1 1 1 1

Neural Net Matrix

Using the Trained neural Net

Calculate output from information in Structure, input data, and neural net text areas

Translate Neural Net as Javascript Program