Learning by Glueing

The feedback process that extends locally constant functions to larger domains is not limited to
supervised learning. The ‚SET‘ – input of the engine may be completely absent.
There are several ways to accomplish this:

1. Whenever the output is completely undefined, create a new symbol y and add it to Y. The germs are tightly packed.
2. Initialize memory with random symbols chosen from a fixed set Y. Reduce the feedback thresholds to start the feedback process.
3. Initialize memory with random symbols chosen from a fixed set Y. An additional table ( indexed by Y ) keeps track of the size of the fibers. Each vote for an y ∈ Y is weighted by a decreasing function of the size of the fiber ( e.g. exp( -γ·(n – n₀) with n₀ = #X/#Y ). Large fibers will not grow further, and small ones will expand at first opportunity. The fibers will compete for available memory space.
   This process works remarkably well for small |Y|, even for |Y| = 2 or 3! Of course, this is not usually large enough to discriminate letter shapes. In this case we use several engines in parallel and form products of their output spaces.

This border case corresponds to a single neuron with, say, 65536 modifiable synapses organized into 256 dendritic compartments that computes a boolean function.

The plot below shows what happens with this feedback rule.
The pattern engine is exposed to millions of letter shapes in sequence. Each one creates a majority decision, and some fiber sizes change. Memory is initialized at random, the fibers all start at size 44000. After a short transient, the engine settles into interesting oscillation patterns. Each step on the ordinal axis corresponds to an entire page with some thousand letters.

Other values of γ give different pictures. Some combinations of γ , #X , #Y and , of course, input data seem to suppress the oscillations. There is certainly much room for further investigation.

Post a Comment