Economics should be open

June 4, 2009

Billing Data and Randomized Experiments in Energy Efficiency Evaluation a research survey

Filed under: California, Data Insights, Energy, Residential — howardchong @ 9:44 pm

I’m doing a research survey of empirical evaluations of energy efficiency using billing data. Much evaluation is done in the laboratory and these estimates are extrapolated to the field. I’m looking at whether field data has been used to test the laboratory assumptions. I found one by Dubin et al from 1986. I review why this is important and other related articles. This is part of my ongoing research so feedback, especially detailed and esoteric knowledge are greatly appreciated.

Joskow and Marron, especially their 1993 Electricity Journal article, have critiqued how utilities have calculated the cost of saving energy (called a “negawatt”). One of the reasons they cite is the engineering estimates, the basis of many calculations, overstate the energy savings. Brown and White (ORNL/CON-323) bear this out, as does Table 2 of Hewitt et al (1986 Energy and Buildings,  Measured versus predicted savings…). However, both these are prone to sample selection errors that may bias the results. To the extent that those with the most to gain (save electricity) select into the program, a treatment effect would be too generous in attributing savings. If we scaled the program up, we’d get less savings from newer participants. Hence, via sample selection, it’s possible that we’re being too nice to the engineering estimate.

To get around sample selection, you’d want to have a truly randomized experiment. There is only one study that does this:

Price Effects of Energy-Efficient Technologies: A Study of Residential Demand for Heating
and Cooling
Author(s): Jeffrey A. Dubin, Allen K. Miedema, Ram V. Chandran
Source: The RAND Journal of Economics, Vol. 17, No. 3 (Autumn, 1986), pp. 310-325
Published by: Blackwell Publishing on behalf of The RAND Corporation

Price Effects of Energy-Efficient Technologies: A Study of Residential Demand for Heating and Cooling

Author(s): Jeffrey A. Dubin, Allen K. Miedema, Ram V. Chandran

Source: The RAND Journal of Economics, Vol. 17, No. 3 (Autumn, 1986), pp. 310-325

Published by: Blackwell Publishing on behalf of The RAND Corporation

Stable URL: http://www.jstor.org/stable/2555713

The attached paper, which is not well-cited, is the main study I’ve seen that combines

(1) randomized experiment, and (2) billing data for the purpose of (3) evaluating energy efficiency. They even go so far as to install separate metering on the cooling equipment. In particular, their experimental assignment appears to be immune from common selection problems of voluntary programs.

“Florida Power and Light identified a large random sample of its residential customers who live in single-family dwellings and have central electric heat and central air condition- ing-the all-electric customers who account for almost half of its residential electricity usage. Then, four subgroups were randomly selected from this large group. One was assigned to be a control group and the other three to receive, at no charge, one of the three conservation technology combinations.” pg 311

As an alternative to random treatment, you could also use a before and after study if you are sure that there are no time-dependent effects. At a minimum, you would want your data generating process to be time independent. Lucas Davis (

Durable goods and residential demand for energy and water: Evidence from a field trial

RAND 2008) uses a data set on washing machines. I believe that washing machines are pretty time independent (on the month to month scale, of course they run more often when we are awake). In the underlying ORNL paper, their measure of savings is via a before-and-after setting.

Situations where one first recruits a sample of people and then randomly assigns within this group are not as good as a fully random sample. The first step is a volunteering step, and you are essentially running your treatment vs control conditional on volunteers. It might be ok that volunteers are not different from the random sample in ways that matter for electricity use, but you don’t have data. The Brown and White paper evaluated a Bonneville Power Administration weatherization program and used good evaluation techniques, but were limited because of the volunteering issue.

As an aside…

Of course, on the other side, a complete random assignment would underestimate the impact if you allow people to sort into choosing to take an energy efficiency step. As an example, a New York Times article talked about LED lightbulbs being put in Buckingham palace. One reason is because it takes several days of labor to change a lightbulb, so the longevity benefit is driving installation, not just the energy savings. But partly this is due to them having a lot of lightbulbs. Here, the sorting is that users with lots of lightbulbs that are on for long periods are more likely to want to convert to LED lightbulbs.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: