3480 University Street, Montreal, Quebec, CANADA

McGill University

Department of Electrical and Computer Engineering

Academic Integrity

 McGill University values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offences under the code of students conduct and disciplinary procedures (see academic integrity for more information).


Winter 2012


 General Information:




Prof. H. Leib, Tel. 398-8938,   Room 757 ENGMC
email :
office hours : Thursday 15:30 - 17:00



Text book:

Thomas M. Cover, Joy A. Thomas, Elements of Information Theory, 2nd ed., John Wiley & Sons Inc. 2006




Raymond W. Yeung, Information Theory and Network Coding, Springer 2008



R. G. Gallager, Information Theory and Reliable Communication, John Wiley & Sons Inc. 1968



James Gleick, The information: A History, A Theory, A Flood, Pantheon Books 2011


Selected papers.



Pre-requisites :

A course on random processes (such ECSE-509 or ECSE-510) and a course on communication systems

(such as ECSE-411, CSE-511, ECSE-521)



Final mark composition :

Assignments (20%), one midterm test (2h, open books, 30%), term project (50%)



Schedule / Location :

Tuesdays and Thursdays, 11:30 13:00, room ENGTR 2120



First Class: 

Tuesday, January 10, 2012




Course Outline:


Information Theory is a mathematical discipline that seeks to quantify the notion of information and its transmission capabilities. Information Theory was introduced in 1948 by Claude Shannon, and provides the foundations for modern communication systems. Subsequently, Information Theory found applications in many other fields, such as computer science and statistics. Nowadays, Information Theory is an active research discipline, attracting talented researchers from a variety of fields.

This is a first course in Information Theory. Its main objectives are to provide a solid basis in this field, as well as an ability of using Informatic Theoretic techniques for a variety of applications. Students will be encouraged to choose term projects in Information Theory that are related to their own research area. Guidelines for term projects

The course covers the following main topics :

1)      Introduction and overview.

2)      Information sources and channel models, entropy and mutual information.

3)      Lossless source coding (data compression).

4)      Source coding with a fidelity criterion (rate distortion theory).

5)      Performance limits for channel coding (channel capacity)

6)      Multiple-Input Multiple-Output (MIMO) channels.

7)      An introduction to multi-user information theory



Useful Links:

Shannon's pioneering work

Information Theory - Britannica online

Introduction to Probability Theory - book

Probability web

Entropy on the web

Entropy and Information Theory on the web




 Date : 

Friday, March 30, 2012

 Time : 


 Location : 

ENGTR 2120

 Material :

1) Class lectures: from the beginning of the term up to and including March 20, 2012 (including assigned reading material).


2) Text book : Chapter 1, Chapter 2 (without 2.9), Chapter 3, Chapter 4 (without 4.4, 4.5),

Chapter 7 (without 7.12), Chapter 8, Chapter 9 (without 9.6).


3) Assignments : 1,2,3.

 Type of test : 

Open books, open notes, all hand-held calculators allowed, notebook computers or any other computers not allowed.