Demo: Error Backpropagation Neural Networks

Developer	sonots
First Edition	04/2005
Last Modified	05/2006
Language	Matlab

Table of Contents

Abstract
ebpnn.m
ebpnn2.m
ebpnnTest
Discussion
References

This page is written in both English and Japanese.

Abstract

This is an experiment rather than a software of error backpropagation NN. We use Neural Network toolbox.

In ebpnn.m, we set the # of neurons of output layer only 1, and we used a characteristic which error backpropagation NN outputs real values, that is, we classified a class if the ouput values is almost equal to the class interger label.

In ebpnn2.m, we set the # of neurons of output layer = # of classes like perceptron neural networks, and we classified a class if it's associated neuron is fired. Actually, because the output value is a real value, we defined 'the fired neuron' as a neuron which output the nearest value to 1.

ebpnn.m

% ebpnn: Classification using Error backpropagation Neural Networks
%
%  [C] = ebpnn(data, proto, protoClass)
%
%  Input and output arguments ([]'s are optional):
%   data        (matrix) of size NxD. N is the number of data (classifiee)
%                vectors, and D is the dimension of each vector. 
%   proto       (matrix) of size PxD. P is the number of prototype vectors,
%                and D is the dimension of each vector. 
%   protoClass  (vector) of size Px1 that contains integer class labels of 
%                prototypes. protoClass(j) is the class of jth prototype.
%   [NNH]       (vector) # of neurons in each hidden layer. [30 30] means
%                two hidden layers with 30 nurons for both. 
%   [EPOCHS]    (scalar) # of iterations of training
%   [GOAL]      (scalar) continue iterations until error ratio reaches to
%                GOAL. 
%   [FUNC]      (cell array) transfer functions for each layer (including
%                output layer) such as {'tansig', 'tansig'} or {'logsig'
%                'purelin'}
%   C           (vector) of size Nx1: integers indicating the class 
%                decision for data items. C(i) is the classification for
%                data item i.
%
% Rquirement: Neural Network toolbox
%
% Author : Naotoshi Seo
% Date   : April, 2005
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function C = ebpnn(data, proto, protoClass, NNH, EPOCHS, GOAL, FUNC)
 if nargin < 4 | isempty(NNH),
     NNH = [30]; % # of neurons in hidden layers (default: 1 layer)
 end
 if nargin < 5 | isempty(EPOCHS),
     EPOCHS = 50; % # of epochs (iterations) of training
 end
 if nargin < 6 | isempty(GOAL),
     GOAL   = 0.01; % stop threshold of training
 end
 if nargin < 7 | isempty(FUNC),
     FUNC   = repmat({'purelin'}, 1, length(NNH)+1);
     % NN toolbox' s default is 'tansig',
     % but it does not work well usaully. ??
 end
 if length(NNH) ~= length(FUNC) - 1
     error('# of layers of NNH must be # of layers of FUNC - 1.');
 end
 if size(data,2) ~= size(proto,2)
     error('Dimension of data vectors and prototype vectors do not match.');
 end
 if size(proto,1) ~= size(protoClass,1)
     error('Row # of prototypes and prototype class vector do not match.');
 end

 proto = proto'; % arguments for nn toolbox is reverse. (DxN)
 protoClass = protoClass';
 data  = data';

 % error backpropagation nn can handle real values not only boolean,
 % so one output neuron is fine (about 1 => class 1, about 2 => class 2). 
 % BUT, it means class label [10 20 30] might work better than [1 2 3].
 % It is wierd. 
 net = newff(minmax(proto),[NNH 1], FUNC);
 net.trainParam.epochs = EPOCHS;
 net.trainParam.goal = GOAL;
 net = train(net, proto, protoClass);
 [dataClassified,Pf,Af,E,perf] = sim(net, data);
 C = round(dataClassified)';

ebpnn2.m

% ebpnn2: Classification using Error backpropagation Neural Networks
%  In contrast to ebpnn, ebpnn2 converts interger class labels into
%  boolean expression labels and uses # of classes as # of neurons
%  at output layer. Inputs and outputs are same with ebpnn. 
%
%  [C] = ebpnn(data, proto, protoClass)
%
%  Input and output arguments ([]'s are optional):
%   data        (matrix) of size NxD. N is the number of data (classifiee)
%                vectors, and D is the dimension of each vector. 
%   proto       (matrix) of size PxD. P is the number of prototype vectors,
%                and D is the dimension of each vector. 
%   protoClass  (vector) of size Px1 that contains integer class labels of 
%                prototypes. protoClass(j) is the class of jth prototype.
%   [NNH]       (vector) # of neurons in each hidden layer. [30 30] means
%                two hidden layers with 30 nurons for both. 
%   [EPOCHS]    (scalar) # of iterations of training
%   [GOAL]      (scalar) continue iterations until error ratio reaches to
%                GOAL. 
%   [FUNC]      (cell array) transfer functions for each layer (including
%                output layer) such as {'tansig', 'tansig'} or {'logsig'
%                'purelin'}
%   C           (vector) of size Nx1: integers indicating the class 
%                decision for data items. C(i) is the classification for
%                data item i.
%
% Rquirement: Neural Network toolbox
%
% Author : Naotoshi Seo
% Date   : April, 2005
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
function C = ebpnn2(data, proto, protoClass, NNH, EPOCHS, GOAL, FUNC)
 if nargin < 4 | isempty(NNH),
     NNH = [30]; % # of neurons in hidden layers (default: 1 layer)
 end
 if nargin < 5 | isempty(EPOCHS),
     EPOCHS = 50; % # of epochs (iterations) of training
 end
 if nargin < 6 | isempty(GOAL),
     GOAL   = 0.01; % stop threshold of training
 end
 if nargin < 7 | isempty(FUNC),
     FUNC   = repmat({'purelin'}, 1, length(NNH)+1);
     % NN toolbox' s default is 'tansig',
     % but it does not work well usaully. ??
 end
 if length(NNH) ~= length(FUNC) - 1
     error('# of layers of NNH must be # of layers of FUNC - 1.');
 end
 if size(data,2) ~= size(proto,2)
     error('Dimension of data vectors and prototype vectors do not match.');
 end
 if size(proto,1) ~= size(protoClass,1)
     error('Row # of prototypes and prototype class vector do not match.');
 end

 proto = proto'; % arguments for nn toolbox is reverse. (DxN)
 protoClass = protoClass';
 data  = data';

 % Find all class labels
 classLabel = unique(protoClass);
 nClass     = length(classLabel); % # of neurons at output layer
                                  % Assign one neuron for one class
 protoClass = int2bool(protoClass); 

 % NN 
 net = newff(minmax(proto),[NNH nClass], FUNC);
 net.trainParam.epochs = EPOCHS;
 net.trainParam.goal = GOAL;
 net = train(net, proto, protoClass);
 [dataClassified,Pf,Af,E,perf] = sim(net, data);
 C = bool2int(dataClassified, classLabel);
 C = C';

% convert interger class label into bool expression
% as [1 2 4 2] => [1 0 0 0]
%                 [0 1 0 1]
%                 [0 0 1 0]  
function boolClass=int2bool(class);
 % Find all class labels
 classLabel = unique(class);
 nClass     = length(classLabel);
 for i=1:nClass
     boolClass(i, :) = (class == classLabel(i));
 end

% convert bool expression class label (members are real values)
% into interger label
% as [0.1  0.8  1.9 1.1] => [0 1 0 1] => [2 1 4 1]
%    [0.9 -0.1  0.1 0.2]    [1 0 0 0]
%    [0.2 -0.8  0.7 0.8]    [0 0 1 0]
function class=bool2int(boolClass, classLabel);
 [mini, minIndex] = min(abs(boolClass - 1)); % closest to 1.
 for i=1:length(minIndex)
     class(i) = classLabel(minIndex(i));
 end

ebpnnTest

Source Codes

function ebpnnTest
 proto = [
    0.6213    0.7373
    0.5226    0.8939
    0.9797    0.6614
    0.9568    0.0118
    0.8801    0.1991
    0.8757    0.0648 
    0.1730    0.2987
    0.2714    0.2844
    0.2523    0.4692
    ];
 protoClass = [
     1
     1
     1
     2
     2
     2 
     3
     3
     3
     ];
 data = [
    0.9883    0.4329
    0.5828    0.2259
    0.4235    0.5798
    0.5155    0.7604
    0.3340    0.5298 
    ];

 % main
 dataClass = ebpnn(data, proto, protoClass, 30, 50, 0.01, {'purelin', 'purelin'});
 %dataClass = ebpnn2(data, proto, protoClass, 30, 50, 0.01, {'purelin', 'purelin'});

 % plot proto
 figure(2);
 classLabel = unique(protoClass);
 nClass     = length(classLabel);
 plotLabel = {'r*', 'g*', 'b*'};
 for i=1:nClass
     A = proto(protoClass == classLabel(i), :);
     plot(A(:, 1), A(:, 2), plotLabel{i});
     hold on;
 end

 % plot data
 plotLabel = {'ro', 'go', 'bo'};
 for i=1:nClass
     A = data(dataClass == classLabel(i), :);
     plot(A(:, 1), A(:, 2), plotLabel{i});
     hold on;
 end
 legend('1: proto','2: proto', '3: proto', '1: data', '2: data', '3: data');
 title('Error Backpropagation NN');

Results

By ebpnn.m

By ebpnn2.m

Discussion

ebpnn2 was better. This is actually as we expected.

We guessed as ebpnn2 is better because, in the case of ebpnn, class label [10 20 30] might work better than [1 2 3] because their differences as value are larger. It is wierd. Even if we make the differences larger and larger, it lasts forever, however, ebpnn2 equals to ebpnn which has infinite differences.

Discussion

In our experience, compared with knn, error backpropagation NN takes dramatically much time, but does not result dramatically better results. Knn might work enough for many situations. Theoritically, knn equals to perceptron NN, so the more complex error backpropagation NN should superior to knn in terms of recognition rate, hopefuly.

First of all, the backpropagation NN is an optimization algorithm to approximate output values of output layers to supervised values, and it is not a exact algorithm to separate feature spaces optimally. This means that the backpropagation is proper to be used for approximation of non-linear functions than non-linear separation of feature spaces (pattern recognition). SVM should be better for pattern recognition.

References

[1] Bishop, C. M. (1995a). Neural Networks for Pattern Recognition. Oxford University Press. (Amazon)
newff (Neural Network Toolbox)
Error Backpropagation

Naotoshi Seo

Announce