This menu window deals with making a network, adding to it, turning weights and units on and off and listing values on the network for individual patterns. Normally the only values you want to see are the output values for patterns and you can get these values elsewhere however here you can also list input and hidden layer unit values.
If you have a three layer network then clicking this button will add a unit to the hidden layer with random initial values for the new weights. If you have more than a three layer network the program will give you an entry box where you need to enter which hidden layer to add to. (The input layer is 1, the first hidden layer is 2, the second hidden layer is 3, ... .) The typed command to add a unit to layer 2 (first hidden) of the network is:
ah 2
Each button will bring up a series of 4 entry boxes and in each box you have to type in either the layer number or the unit number for each node necessary to specify the weight you want to turn on or off. The order of the boxes is:
The typed version of the command to turn on the weight from layer 1 unit 2 to layer 3 unit 4 is:
onw 1 2 3 4Likewise, to turn the weight off the command is:
ofw 1 2 3 4
When you press this button two entry boxes will come up, one to give the layer number of the unit to turn off and then one to give the unit number in that layer. The typed command to turn the fifth unit in layer two off is:
ofu 2 5while the typed command to turn it back on is:
onu 2 5
Once upon a time I failed to count up how many weights are effectively turned off by turning off a unit so the number that are effectively turned off will not be registered in the menu area, maybe someday I will get to that.
The button will bring up a series of 4 entry boxes and in each box you have to type in either the layer number or the unit number for each node necessary to specify the weight you want to add. The order of the boxes is:
The typed version of the command to turn on the weight from layer 1 unit 2 to layer 3 unit 4 is:
ac 1 2 3 4 * ac is for add connection
The options available are to make a two, three or four layer network. (In fact you can make a network with any number of layers by typing in the right command but four layers is rarely useful and more than four is very rarely done so there are no menu entries for making more than a four layer network). Click whichever configuration network you want, this gives new windows where you can select sizes and other options.
Whichever size network you choose fill in the entry boxes with the number of units you want in each layer.
If you want direct input to output connections in a three or four layer network click the button that changes the setting. These direct connections are often of some benefit in problems where there are not too many inputs and outputs and especially in function approximation problems where the mapping is nearly linear. In these problems the output layer units are linear. The non-linear hidden layer units can then make a small non-linear correction. This may also mean you need fewer hidden layer units.
If you want a recurrent network, life is complicated. IF THE NETWORK IS A RECURRENT NETWORK AND USES "H" DO NOT include the number of short term memory units when you input the number of input units if you are using H to stand for all the hidden layer units. Thus for the poetry problem tell the program you want 25 input units (not 45, the 45 comes from 25 normal input units plus the 20 more short term units whose values come from the hidden layer). The program will ultimately output a make command that looks like "m 25+20 20 25" so you will end up with 45 input units for the network.
IF THE NETWORK IS A RECURRENT NETWORK AND USES "h" you do count however many h units you have as input units and you DO NOT click on the recurrent network button. So if you use this data:
0.00000 hh 0.15636 0.15636 hh 0.30887 0.30887 hh 0.45378 0.45378 hh 0.58753 0.58753 hh 0.70683 0.70683 hh 0.80874 0.80874 hh 0.89075 0.89075 hh 0.95086the number of input units should be 3 and you DO NOT click on the recurrent network button.
This version also allows you to make what I describe as a pseudo-cascade correlation type network. This is very nearly the cascade correlation style network described by Fahlman and Lebiere. A cascade correlation network looks like:
o o o
h3
h2
h1
i i i i i i i
Every hidden unit (h) connects with all the input units (i) and
all the output units (o) plus the h3 unit connects to the two
h units below it and the h2 unit connects to the h1 unit below
it. The idea is that as h1 finds important features h2 can
use this complex feature as well as the input unit features.
Then h3 can use whatever features the h1 and h2 units find.
To make a network that looks like this use the command:
m 7 3 3 x pwhere the x must come before the p. In this implementation all the h units count as being in a single hidden layer whereas in discussions of the cascade correlation network each hidden unit is considered to be in a layer by itself. Moreover in the implementation here the errors are not passed back as you might expect. Unit h3 does not pass back errors to h2 and h2 does not pass back errors to h1.
To finally make the network, click the "Make" button at the bottom of the window or click "Exit" to exit the window without making a network.
This button brings up a new menu window where you can set the parameters for adding hidden units to a network as the training proceeds. The main idea is that adding hidden layer units as the training proceeds can get you out of a local minimum.
When you select this button you get an entry box where you type in an input pattern, the pattern is then run through the network and the output layer is printed out. The typed command alternative is to simply type in the pattern in the main window entry box.
When you select this button an entry box pops up where you type in the training set pattern number you want the network to evaluate. An example of the typed alternative is:
p 5 * list training set pattern number 5
When you select this button an entry box pops up where you type in the test set pattern number you want the network to evaluate. An example of the typed alternative is:
t 5 * list test set pattern number 5
Output format is controlled with the `f' command as in:
f or * output node values using real (the C %f) format f oc * output node values using compressed format f oa * output node values using analog compressed format f oe * output values with e notation f os * output nodes values scaled up to their normal stateThe first sets the output to real numbers. The second sets the output to be compressed mode where the value printed will be a `1' when the unit value is greater than 1.0 - tolerance, a `^' when the value is above 0.5 but less than 1.0 - tolerance, a `v' when the value is less than 0.5 but greater than the tolerance. Below the tolerance value a `0' is printed. The tolerance can be changed using the `t' command (not a part of the format command). For example, to make all values greater than 0.8 print as `1' and all values less than 0.2 print as `0' use:
t 0.2Of course this same tolerance value is also used to check to see if all the patterns have converged. The third output format is meant to give "analog compressed" output. In this format a `c' (c for close) is printed when a value is close enough to its target value. Otherwise, if the answer is close to 1, a `1' is printed, if the answer is close to 0, a `0' is printed, if the answer is above the target but not close to 1, a `^' is printed and if the answer is below the target but not close to 0, a `v' is printed. This output format is designed for problems where the output is a real number, as for instance, when the problem is to make a network learn sin(x). The format "e" writes out node values using exponential notation with four places to the right of the decimal point.
After some pattern has been submitted to the network you can use this command to take a look at the units on any layer including a hidden layer and the input layer. An entry box comes up where you type in the layer number. Layer 1 is the input layer. This can be useful at times especially if you want to see if hidden layer values are being copied down to the input layer in a recurrent network. An example of the typed alternative is:
l 2 * list unit values on layer 2