Information Theory (IT) was born in 1948 when Claude Shannon published its a Mathematical Theory of Communication. This seminal work uses linguistic examples, so Information Theory is related to language since its inception. There were many attempts to explain IT to the general audience, but Rényi’s Diary is outstanding with its unique style, clear prose and rigor. If you read only one book on IT, read this one!

The Diary is written in the name of a fictional student of Rényi who attended the profs course on IT. The student tries to digest the course material and express the main ideas of the lectures in his own terms. It keeps only the necessary formalism, but if you skip the “math heavy” parts, you can still understand the main points of the book.

The second best one

If you are really keen on IT and want to know more about its main ideas and applications, An Introduction to Information Theory by John R. Pierce is your book. It explains Shannon’s original ideas in a very clear manner. It introduces the main ideas of IT through pretty good, practical examples and it devotes chapters to multidisciplinary connections too. It is a little bit more math heavier, but it can be read in “equation free mode” too.