HN Reader

There is a mistake right in the beginning, not sure how it affects the conclusions yet. The variables given are S - System variable (some kind of disturbance), Z is the outcome ( a controlled variable) and R is the action of a controller. The causal relations between them are S affects Z, S affects R, and R affects Z.

> The archetypal example for this is something like a thermostat. The variable S represents random external temperature fluctuations. The regulator R is the thermostat, which measures these fluctuations and takes an action (such as putting on heating or air conditioning) based on the information it takes in. The outcome Z is the resulting temperature of the room, which depends both on the action taken by the regulator, and the external temperature.

The problem here is that the regulator R does not measure external temperature. It just measures the controlled variable - the temperature Z, so the causal arrow should go from Z to R too, and the arrow from S to R does not exist.

This is pedantic, but I don't like the formulation of entropy as sum of p log(1/p). I think of log(p) as information of a single event, for which log base 1/2 gives the answer in bits. This makes the negative sign unnecessary, and technically all these formulas should specify the base of log > 1. Everything is cleaner with log base 1/2 (instead of e.g. using the equivalent negative log base 2). This comes up in log likelihood all the time too. I guess it's a prejudice against fractional bases.