Is There a Trade-off with Transparency? A Deep Dive into Automated Home Energy Estimates

In the battle of human versus machine, is there a clean winner? No, we’re not talking about a future dystopian society—we’re talking about home energy estimates.

Rocky Mountain Institute’s new report, An MPG for Homes: Accuracy and Application of Automated Home Energy Estimates, builds on earlier research to compare algorithm-based home energy estimates to in-person, on-site assessments in an effort to scale the transparency of home energy performance.

Analysis of almost 8,000 homes across 27 US states revealed that remotely generated home energy estimates are just that—estimates—but they may be accurate enough to open up a variety of useful applications for homeowners, city leaders, service providers, and others. By simply providing information, they may help to make meaningful progress in improving home performance, affordability, and comfort, and reducing carbon emissions from the residential sector at a greater scale.

This is significant as more and more cities and states are considering policies and programs that require greater energy performance transparency, like Portland, Oregon’s, or Massachusetts’s proposed plan to start requiring home energy information be made available at the point when a property is listed.

Unpacking the Options

If homeowners want to better understand the energy performance of their homes, there are two possible paths.

The first is to engage customized on-site energy assessments like Home Energy Score (HES) or RESNET’s Home Energy Rating System (HERS). HES is a rating system and program launched by the US Department of Energy (DOE) in 2012 to estimate a home’s energy use and identify cost-saving upgrades using energy modeling software fed by a comprehensive set of data points collected on-site by a qualified assessor. On-site assessments provide valuable home energy performance insights and actionable recommendations but, by their very nature, they are more difficult to scale across the residential sector due to the time, cost, and initiative required. This is one reason residential energy performance information has been available only for the less than 3 percent of US single-family homes that have been subject to these types of assessments.

The second path is to use the services of ClearlyEnergy or UtilityScore. Their algorithms draw from public data to provide fast and free energy estimates for all homes nationwide (single-family homes only with UtilityScore), giving all homeowners a more holistic understanding of the full operating costs of a new home before purchasing, and ideas to lower their energy bills and improve the comfort of their homes at any point after purchasing.

RMI has written in the past about the significant opportunity for automated home energy estimates to improve the transparency of home energy performance at scale, and to change the way we think about, and act on, our homes’ energy use. But for these algorithms to reach their market potential, they must be accurate enough to be useful for multiple stakeholders, some of whom have questioned the reliability of home energy estimates generated remotely.

This led RMI to address three fundamental questions:

1. How do algorithm-based estimates stack up against on-site assessment estimates?
2. Is there a trade-off between customization and accuracy versus automation and scale?
3. How can this information be used by different stakeholders in the market?

Comparing Human-Made to Machine-Generated

RMI partnered with the DOE to obtain a dataset for several thousand homes across 27 states and multiple climate zones that underwent an HES assessment during 2017. RMI asked the participating data vendors, ClearlyEnergy and UtilityScore, to generate their own estimates for electricity use and heating fuel use, and to generate corresponding estimates of costs.

RMI’s analysis compared asset-rating estimates between the vendors and HES, focusing on homes’ underlying energy-related assets and mitigating some of the operational uncertainty introduced by actual utility billing data. While electricity and natural gas use estimates were evaluated in this way using HES as the baseline, energy cost estimates were evaluated differently given accuracy concerns built into the HES baseline.

Overall, RMI’s comparative analysis found that algorithm-based assessments are accurate enough in assessing total energy use and total energy costs:

• Total energy use: Estimates from both vendors showed a 20–30 percent average absolute difference from HES estimates. Nearly three-quarters of all homes analyzed were less than 30 percent different, nearly half of all homes analyzed were less than 20 percent different, and more than one-quarter of all homes analyzed were less than 10 percent different. Estimates from both vendors showed an overall average nonabsolute difference within +/- 10 percent of HES estimates.
• Total energy costs: Vendor algorithms can outperform on energy cost estimates given their ability to pull significantly more granular utility rate data (HES currently uses statewide average utility rates). In fact, when the vendors’ implied utility rates were used, converted HES energy cost estimates were 9–22 percent different from reported HES cost estimates.

Visible Value for Multiple Parties

The existence of reasonably accurate “first look” energy performance estimates for all homes in the US can not only start the conversation for more homeowners, but can also motivate them to take the next step—whether that means pursuing a more comprehensive on-site assessment like HES or HERS, purchasing a more efficient home, or installing valuable energy upgrades themselves. While RMI believes additional research into the accuracy of the algorithms’ inputs and upgrade recommendations would advance the conversation and further benefit the market, the results of this analysis suggest that automated home energy estimates may be sufficiently accurate for multiple stakeholders to start reaping their benefits.

While their accuracy can and should improve with time, these algorithms are ultimately making significant strides in closing the information gap and creating visible value across the residential energy performance market.

Dig deeper into our analysis by downloading the report here. RMI’s Residential Energy+ team is partnering with leading US cities that are interested in driving greater home energy. If you are a city representative interested in learning more from RMI on this topic, please contact us.

Image courtesy of iStock.