Bias (of an Activation Function)

The bias of an activation function is the net input that serves as the threshold, or as an analog to the threshold, of that function. For instance, the bias of the sigmoid-shaped logistic function that is perhaps the most popular activation function is the value of net input that produces an activity of 0.5 (exactly half way between on and off).

The bias is a parameter that can be trained as a network learns to perform a task. This can be accomplished, for example, by including an extra input unit that is always on. The weight of the connection between this unit and another unit is equal to the bias of the receiving unit.

The use of a trainable bias in the perceptron also has important theoretical implications for modeling classical conditioning. In mathematical models of such learning (Rescorla & Wagner, 1972) it is typical to include an input that represents a context that is present for every trial in an experiment. Of course, one could interpret the bias as being equivalent to this context (e.g., the bias unit is the representation of a context that is always present). Thus, a perceptron with a trainable bias provides a learning context "for free" (Dawson, 2008).

References:

1. Dawson, M. R. W. (2008). Connectionism and classical conditioning. Comparative Cognition and Behavior Reviews, 3 (Monograph), 1-115.
2. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical Conditioning II: Current Research And Theory (pp. 64-99). New York, NY: Appleton-Century-Crofts.

(Added January 2010)