• Introduction
• Neocognitron Architecture
• Neocognitron data processing
• Training Weights on the S-Layers

Introduction

Artificial neural network architectures such as backpropagation tend to have general applicability. We can use a single network type in many different applications by changing the network’s size, parameters, and training sets. In contrast, the developers of the neocognitron set out to tailor architecture for a specific application: recognition of handwritten characters. Such a system has a great deal of practical application, although, judging from the introductions to some of their papers, Fukushima and his coworkers appear to be more interested in developing a model of the brain .To that end, their design was based on the seminal work performed by Hubel and Weisel elucidating some of the functional architecture of the visual cortex.

We could not begin to provide a complete accounting of what is knownabout the anatomy and physiology of the mammalian visual system. Nevertheless, we shall present a brief and highly simplified description of some of that system’s features as an aid to understanding thejbasis of the neocognitron design.

Figure A: Visual pathways from the eye to the primary visual cortex
are shown. Some nerve fibers from each eye cross over into
the opposite hemisphere of the brain, where they meet nerve
fibers from the other eye at the LGN. From the LGN, neurons
project back to area 17. From area 17, neurons project into
other cortical areas, other areas deep in the brain, and also
back to the LGN.
Figure shows the main pathways for neurons leading from the retina back to the area of the brain known as the viiual, or striate, cortex. This area is also known as area 17. The optic nerve ii made up of axons from nerve cells called retinal ganglia. The ganglia receive stimulation indirectly from the light-receptive rods and cones through several intervening neurons. Hubel and Weisel used an amazing technique to discern the function of the various nerve cells in the visual system. They used microelectrodes to record the response of individual neurons in the cortex while stimulating the retina with light. By applying a variety of patterns and shapes, they were able to determine the particular stimulus to which a neuron was most sensitive.

The retinal ganglia and the cells of the lateral geniculate nucleus (LGN) appear to have circular receptive fields.

They respond most strongly to circular spots of light of a particular size on a particular part of the retina. The part of the retina responsible for stimulating a particular ganglion cell is called the receptive field of the ganglion. Some of these receptive fields give an excitatory response to a centrally located spot of light, and an inhibitory response to a larger, more diffuse spot of light. These fields have an on-center off-surround response characteristic. Other receptive fields have the opposite characteristic, with an inhibitory response to the centrally located spot—an off-center on-surround response characteristic.

The visual cortex itself is composed of six layers of neurons. Most of the neurons from the LGN terminate on cells in layer IV. These cells have circularly symmetric receptive fields like the retinal ganglia and the cells of the LGN. Further along the pathway, the response characteristic of the cells begins to increase in complexity. Cells in layer IV project to a group of cells directly above called simple cells. Simple cells respond to line segments having a particular orientation. Simple cells project to cells called complex cells. Complex cells respond to lines having the same orientation as their corresponding simple cells, although complex cells appear to integrate their response over a wider receptive field. In other words, complex cells are less sensitive to the position of the line on the retina than are the simple cells. Some complex cells are sensitive to line segments of a particular orientation that are moving in a particular direction.

Cells in different layers of area 17 project to different locations of the brain. For example, cells in layers II and III project to cells in areas 18 and 19. These areas contain cells called hypercomplex cells. Hypercomplex cells respond to lines that form angles or corners and that move in various directions across the receptive field.

The picture that emerges from these studies is that of a hierarchy of cells with increasingly complex response characteristics. It is not difficult to extrapolate this idea of a hierarchy into one where further data abstraction takes place at higher and higher levels. The neocognitron design adopts this hierarchical structure in a layered architecture, as illustrated schematically in Figure
Figure B: The neocognitron hierarchical structure is shown. Each box
represents a level in the neocognitron comprising a simplecell
layer, usi, and a complex-cell layer, Ua, where i is
the layer number. U0 represents signals originating on the
retina. There is also a suggested mapping to the hierarchical
structure of the brain. The network concludes with single
cells that respond to complex visual stimuli. These final cells
are often called grandmother cells after the notion that there
may be some cell in your brain that responds to complex
visual stimuli, such as a picture of your grandmother.

We remind you that the description of the visual system that we have presented
here is highly simplified. There is a great deal of detail that we have
omitted. The visual system does not adhere to a strict hierarchical structure
as presented here. Moreover, we do not subscribe to the notion that grandmother
cells per se exist in the brain. We know from experience that strict
adherence to biology often leads to a failed attempt to design a system to perform
the same function as the biological prototype: Flight is probably the most
significant example. Nevertheless, we do promote the use of neurobiological
results if they prove to be appropriate. The neocognitron is an excellent example
of how neurobiological results can be used to develop a new network
architecture.

NEOCOGNITRON ARCHITECTURE

The neocognitron design evolved from an earlier model called the cognitron,
and there are several versions of the neocognitron itself. The one that we shall
describe has nine layers of PEs (Processing elements), including the retina layer. The system was
designed to recognize the numerals 0 through 9, regardless of where they are
placed in the field of view of the retina. Moreover, the network has a high degree
of tolerance to distortion of the character and is fairly insensitive to the size of
the character. This first architecture contains only feedforward connections.

The PEs of the neocognitron are organized into modules that we shall refer to
as levels. A single level is shown in Figure 10.3. Each level consists of two
layers: a layer of simple cells, or S-cells, followed by a layer of complex
cells, or C-cells. Each layer, in turn, is divided into a number of planes,
each of which consists of a rectangular array of PEs. On a given level, the
S-layer and the C-layer may or may not have the same number of planes.
All planes on a given layer will have the same number of PEs; however, the
number of PEs on the S-planes can be different from the number of PEs on
the C-planes at the same level. Moreover, the number of PEs per plane can
vary from level to level. There are also PEs called Vs-cells and Vc-cells that
are not shown in the figure. These elements play an important role in the
processing, but we can describe the functionality of the system without reference
to them.

We construct a complete network by combining an input layer, which we
shall call the retina, with a number of levels in a hierarchical fashion, as shown
in Figure

Figure C: A single level of a neocognitron is shown. Each level consists
of two layers, and each layer consists of a number of planes.
The planes contain the PEs in a rectangular array. Data pass
from the S-layer to the C-layer through connections that are
not shown here. In neocognitrons having feedback, there also
will be connections from the C-layer to the S-layer.

That figure shows the number of planes on each layer for the
particular implementation that we shall describe here. We call attention to the fact that there is nothing, in principle, that dictates a limit to the size of the
network in terms of the number of levels.
The interconnection strategy is unlike that of networks that are fully interconnected
between layers, such as the backpropagation network.

Figure D shows a schematic illustration of the way units are
connected in the neocognitron.

The figure D shows the basic organization of the neocognitron
for the numeral-recognition problem. There are nine layers,
each with a varying number of planes. The size of each layer,
in terms of the number of processing elements, is given below
each layer. For example, layer Uc2 has 22 planes of 7 x 7
processing elements arranged in a square matrix. The layer
of C-cells on the final level is made up of 10 planes, each
of which has a single element. Each element corresponds to
one of the numerals from 0 to 9. The identification of the
pattern appearing on the retina is made according to which
C-cell on the final level has the strongest response.

As we look deeper into the network, the S-cells respond to features at higher
levels of abstraction; for example, corners with intersecting lines at various angles and orientations. The C-cells integrate the responses of groups of Scells.
Because each S-cell is looking for the same feature in a different location,
the C-cells’ response is less sensitive to the exact location of the feature on the
input layer. This behavior is what gives the neocognitron its ability to identify
characters regardless of their exact position in the field of the retina. By the
time we have reached the final layer of C-cells, the effective receptive field of each cell is the entire retina. Figure 10.6 shows the character identification
process schematically.

Figure F: This diagram is a schematic representation of the
interconnection strategy of the neocognitron. (a) On the first
level, each S unit receives input connections from a small
region of the retina. Units in corresponding positions on all
planes receive input connections from the same region of
the retina. The region from which an S-cell receives input
connections defines the receptive field of the cell, (b) On
intermediate levels, each unit on an s-plane receives input
connections from corresponding locations on all C-planes in
the previous level. C-eelIs have connections from a region of
S-cells on the S level preceding it. If the number of C-planes
is the same as that of S-planes at that level, then each C-cell
has connections from S-cells on a single s-plane. If there
are fewer C-planes than S-planes, some C-cells may receive
connections from more than one S-plane.

Each cell in a plane on the first S-layer receives inputs from
a single input layer-namely, the retina. On subsequent layers, each S-cell
plane receives inputs from each of the C-cell planes immediately preceding
it. The situation is slightly different for the C-cell planes. Typically, each
cell on a C-cell plane examines a small region of S-cells on a single S-cell
plane. For example, the first C-cell plane on layer 2 would have connections
to only a region of S-cells on the first S-cell plane of the previous layer. Reference back to Figure D reveals that there is not necessarily a one-to-one
correspondence between C-cell planes and S-cell planes at each layer in the
system. This discrepancy occurs because the system designers found it advantageous
to combine the inputs from some S-planes to a single C-plane if the
features that the S-planes were detecting were similar. This tuning process
is evident in several areas of the network architecture and processing equations.

Figure G: This figure illustrates how the neocognitron performs
its character-recognition function. The neocognitron
decomposes the input pattern into elemental parts consisting
of line segments at various angles of rotation. The system
then integrates these elements into higher-order structures at
each successive level in the network. Cells in each level
integrate the responses of cells in the previous level over a
finite area. This behavior gives the neocognitron its ability to
identify characters regardless of their exact position or size
in the field of view of the retina. Source: Reprinted with
permission from Kunihiko Fukushima, “A neural network for
visual pattern recognition.” IEEE Computer, March 1988. ©
1988 IEEE.

The weights on connections to S-cells are determined by a training process. Unlike in many other network architectures
(such as backpropagation), where each unit has a different weight vector,
all S-cells on a single plane share the same weight vector. Sharing weights
in this manner means that all S-cells on a given plane respond to the identical
feature in their receptive fields, as we indicated. Moreover, we need to train only one S-cell on each plane, then to distribute the resulting weights to the
other cells.
The weights on connections to C-cells are not modifiable in the sense that
they are not determined by a training process. All C-cell weights are usually
determined by being tailored to the specific network architecture. As with Splanes,
all cells on a single C-plane share the same weights. Moreover, in some
implementations, all C-planes on a given layer share the same weights.

NEOCOGNITRON DATA PROCESSING

In this section we shall discuss the various processing algorithms of the neocognitron
cells. First we shall look at the S-cell data processing including the
method used to train the network. Then, we shall describe processing on the
C-layer.

S-Cell Processing

We shall first concentrate on the cells in a single plane of Js, as indicated in
Figure H

Figure H: The retina, layer U0, is a 19-by-19-pixel array, surrounded by
inactive pixels to account for edge effects as described in the
text. One of the S-planes is shown, along with an indication
of the regions of the retina scanned by the individual cells.
Associated with each S-layer in the system is a plane of vrcells.
These cells receive input connections from the same
receptive field as do the S-cells in corresponding locations in
the plane. The processing done by these Vc-cells is described
in the text.
We shall assume that the retina, layer Uo, is an array of 19 by 19
pixels. Therefore, each Usi plane will have an array of 19 by 19 cells. Each
plane scans the entire retina for a particular feature. As indicated in the figure,
each cell on a plane is looking for the identical feature but in a different location
on the retina. Each S-cell receives input connections from an array of 3 by 3
pixels on the retina. The receptive field of each S-cell corresponds to the 3 by 3
array centered on the pixel that corresponds to the cell’s location on the plane.
When building or simulating this network, we must make allowances for
edge effects. If we surround the active retina with inactive pixels (outputs always
set to zero), then we can automatically account for cells whose fields
of view are centered on edge pixels. Neighboring S-cells scan the retina array
displaced by one pixel from each otlier. In this manner, the entire image
is scanned from left to right and top to bottom by the cells in each Splane.

A
single plane of Vc-cells is associated with the S-layer, as indicated in
Figure H. The Vc-plane contains the same number of cells as does each Splane.
Vc-cells have the same receptive fields as the S-cells in corresponding
locations in the plane. The output of a Vc-cell goes to a single S-cell in every
plane in the layer. The S-cells that receive inputs from a particular Vc-cell
are those that occupy a position in the plane corresponding to the position of
the Vc-cell. The output of the Vc-cell has an inhibitory effect on the S-cells.
Figure I shows the details of a single S-cell along with its corresponding
inhibitory cell.


FIGURE I: A single 5-cell and corresponding inhibitory cell on the Us
layer are shown. Each unit receives the identical nine inputs
from the retina layer. The weights, a,, on the S-ce\ determine
the feature for which the cell is sensitive. Both the a, weights
on connections from the retina, and the weight, b, from the
Vr-cells are modifiable and are determined during a training
process, as described in the text.

Up to now, we have been discussing the first S-layer, in which cells receive
input connections from a single plane (in this case the retina) in the previous
layer. For what follows, we shall generalize our discussion to include the case of layers deeper in the network where an S-cell will receive input connections
from all the planes on the previous C-layer.
Let the index k refer to the kth plane on level 1. We can label each cell
on a plane with a two-dimensional vector, with n indicating its position on the
plane; then, we let the vector v refer to the relative position of a cell in the
previous layer lying in the receptive field of unit n. With these definitions, we can write the following equation for the output of any 5-cell:


where the function Φ is a linear threshold function given by

Let’s dissect these equations in some detail. The inner summation of
first equation is the usual sum-of-products calculation of inputs, UCi-1(ki-1,n+v),
and weights, ai(ki-1, v. ki). The sum extends over all units in the previous Clayer
that lie within the receptive field of unit n. Those units are designated
by the vector n + v. Because we shall assume that all weights and cell output
values are nonnegative, the sum-of-products calculation yields a measure of how
closely the input pattern matches the weight vector on a unit.2 We have labeled
the receptive field Ai, indicating that the geometry of the receptive field is the
same for all units on a particular layer. The outer summation of the equation
extends over all of the ki-1planes of the previous C-layer. In the case of Us1 ,
there would be no need for this outer summation.

The product, bi(ki) . Vc,(n), in the denominator of Equation, represents the
inhibitory contribution of the Vc-cell. The parameter ri, where 0 < ri < ∞, determines the cell's selectivity for a specific pattern. The factor ri/(1 + ri)
goes from zero to one as ri goes from zero to infinity. Thus, for small values
of r;, the denominator of Equation could be relatively small compared to the
numerator even if the input pattern did not exactly match the weight vector. This
situation could result in a positive argument to the Φ function. If ri were large,
then the match between the input pattern and the weights in the numerator of
the equation would have to be more exact to overcome the inhibitory effects of the
Vc-cell input. Notice also that this same ri parameter appears as a multiplicative
factor of the 0 function. If ri is small, and cell selectivity is small, this factor
ensures that the output from the cell itself cannot become very large.
We can view the function of r; in another way. We rewrite the argument
of the function in the equation as


where e is the net excitatory term and h is the net inhibitory term. According
to function , the S-cell output will be nonzero only if

or

Thus, the quantity ri determines the minimum relative strength of excitation
versus inhibition that will result in a nonzero output of the unit. As r; increases,
ri/(l + ri) → 1. Therefore, a larger ri requires a larger excitation relative to
inhibition for a nonzero output.

Notice that neither of the weight expressions, ai(ki-1,v,ki), or bi(ki) depend
explicitly on the position, n, of the cell. Remember that all cells on a
plane share the same weights, even the bi(ki) weights, which we did not discuss
previously.
We must now specify the output from the inhibitory nodes. The Vc-cell at
position n has an output value of

where ci(v) is the weight on the connection from a cell at position v of the
Vc -cell’s receptive field. These weights are not subject to training. They can
take the form of any normalized function that decreases monotonically as the
magnitude of v increases.

Training Weights on the S-Layers

In principle, training proceeds as it does for many
networks. First, an input pattern is presented at the input layer and the data are
propagated through the network. Then, weights are allowed to make incremental
adjustments according to the specified algorithm. After weight updates have
occurred, a new pattern is presented at the input layer, and the process is repeated
with all patterns in the training set until the network is classifying the input
patterns properly.
In the neocognitron, sharing of weights on a given plane means that only a
single cell on each plane needs to participate in the learning process. Once its
weights have been updated, a copy of the new weight vector can be distributed
to the other cells on the same plane. To understand how this works, we can
think of the 5-planes on a given layer as being stacked vertically on top of one
another, aligned so that cells at corresponding locations are directly on top of one
another. We can now imagine many overlapping columns running perpendicular
to this stack. These columns define groups of S-cells, where all of the members
in a group have receptive fields in approximately the same location of the input
layer.

Unsupervised training:With this model in mind, we now apply an input pattern and examine
the response of the S’-cells in each column. To ensure that each 5-cell provides
a distinct response, we can initialize the a; weights to small, positive
random values. The bi weights on the inhibitory connections can be initialized
to zero. We first note the plane and position of the S-cell whose response is
the strongest in each column. Then we examine the individual planes so that,
if one plane contains two or more of these S-cells, we disregard all but the
cell responding the strongest. In this manner, we will locate the S-cell on each
plane whose response is the strongest, subject to the condition that each of those
cells is in a different column. Those S-cells become the prototypes, or representatives,
of all the cells on their respective planes. Likewise, the strongest
responding Vf-cell is chosen as the representative for the other cells on the
Vc-plane.
Once the representatives are chosen, weight updates are made according to
the following equations:

where qi is the learning rate parameter, ci-1(v), is the monotonically decreasing
function as described in the previous section, and the location of the representative
for plane ki is A.
Notice that the largest increases in the weights occur on those connections
that have the largest input signal, Uci-1(ki-1, n + v). Because the 5-cell whose
weights are being modified was the one with the largest output, this learning
algorithm implements a form of Hebbian learning. Notice also that weights can
only increase, and that there is no upper bound on the weight value. Once the cells on a given plane begin to respond to a certain feature, they
tend to respond less to other features. lfter a short time, each plane will have
developed a strong response to a particular feature. Moreover, as we look deeper
into the network, planes will be responding to more complex features.

Other Learning Methods. The designers of the original neocognitron knew
to what features they wanted each level, and each plane on a level, to respond.
Under these circumstances, a set of training vectors can be developed for each
layer, and the layers can be trained independently.
It is also possible to select the representative cell for each plane in advance.
Care must be taken, however, to ensure that the input pattern is presented in the
proper location with respect to the representative’s receptive field. Here again,
some foreknowledge of the desired features is required.