Domestic Affairs

Substandard surveys are worse than no surveys

By Patrick Mattimore (chinadaily.com.cn)
Updated: 2010-08-18 13:38

Bloomberg News Service ran the following headline today: "At Least Half of Apartments in Shanghai, Beijing Are Vacant, Daily Reports."

Bloomberg also noted that "China Daily" had cited an online investigation by website Sina.com., in which volunteers canvassed 100 Chinese cities and 1000 real-estate projects to reach the conclusion that about 51 percent of Shanghai apartments, 66 percent of Beijing flats and more than 70 percent of units in Hainan are vacant.

Here's the kicker though. The survey was based on counting the number of apartments observed to have no lights on at night. That's right. Apparently if an apartments' lights are turned off it doesn't mean that the residents aren't home, that they have gone to sleep, or are in another room. It means no one lives in the apartment.

What is particularly distressing about this "survey", is that there is no indication that the counters checked the apartments more than once, when in the evening the apartments were checked, or anything else to substantiate vacancy numbers. For example, going door-to-door on other evenings to see if the "lights out" apartments might be occupied, would have made sense. One wit on the Web wondered whether the counters had turned off their own apartment lights when they had gone out to do the counting.

It is vital that China's citizens be heard on important topics and good survey information is a critical means by which people express opinions. By promulgating information about a population's attitudes, surveys can influence the behavior and attitudes of other people. Surveys have powerful ramifications and therefore it is crucial that they be conducted well.

The apartment survey is a particularly egregious example of bad survey methodology leading to poor data, but unfortunately we rarely pay attention to methodology. Even more traditional surveys have limitations and should come with caveats.