It would likely take us less than an average of two guesses per blank to figure out the sentence is “How are you doing my friend?”. It would be very bad if it took us 13.5*16=216 guesses to fill in the 16 blanks. Let us say we are given the first letter of every word in this sentence: Random guessing on average takes us 13.5 guesses to get the correct letter. not assigning probabilities of 1/26), we can be much more efficient. However, some letters are more common than other letters, and some letters appear often together, so through clever ‘guessing’ (i.e. The English language has 26 letters, if you assume each letter has a probability of 1/26 of being next, the language has an entropy of 4.7 bits. Some distributions and their entropies Example: English Language It is important to know that information storage and communication are almost the same thing, as you can think of storage as communication with a hard disk. The letters in your keyboard are stores in a ‘byte’, which is 8 bits, which allows for 2⁸ =256 combinations. In a digital form, information is stored in ‘bits’, or a series of numbers that can either be 0 or 1. We have all learned this lesson the hard way when we have forgotten to save a document we were working on. Information is only useful when it can be stored and/or communicated. Entropy is defined as ‘lack of order and predictability’, which seems like an apt description of the difference between the two scenarios. A formal way of putting that is to say the game of Russian roulette has more ‘entropy’ than crossing the street. This is partially because we pretty much know what will happen when I cross the street, but we don’t really know what will happen in Russian roulette.Īnother way of looking at this, is to say we gain less information observing the result of crossing the street than we do from Russian roulette. If you were to watch me cross the street, or watch me play Russian roulette, which one would be more exciting? The possibilities are the same- me living or dying, but we can all agree that the crossing of the street is a bit boring, and the Russian roulette… maybe too exciting. A layman’s introduction to information theory
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |