DOE Technical Assistance Program - PowerPoint Presentation

1. DOE Technical Assistance ProgramMarch 29, 2011, 2-3pm ESTDeveloping an Evaluation, Measurement and Verification Plan For Residential Retrofit ProgramsJulie Michals, Northeast Energy Efficiency Partnerships, Inc.Chris Neme, Energy Futures Group

2. What is TAP?DOE’s Technical Assistance Program (TAP) supports the Energy Efficiency and Conservation Block Grant Program (EECBG) and the State Energy Program (SEP) by providing state, local, and tribal officials the tools and resources needed to implement successful and sustainable clean energy programs.

3. How Can TAP Help You?TAP offers:One-on-one assistance Extensive online resource library, including: WebinarsEvents calendarTAP BlogBest practices and project resourcesFacilitation of peer exchangeOn topics including:Energy efficiency and renewable energy technologiesProgram design and implementationFinancingPerformance contractingState and local capacity building

4. The TAP Blog Provides a platform for state, local, and tribal government officials and DOE’s network of technical and programmatic experts to connect and share best practices on a variety of topics. Access the TAP Blog!http://www.eereblogs.energy.gov/tap/

5. Accessing TAP Resources3) Ask questions via our call center at 1-877-337-3827 or email us at solutioncenter@ee.doe.gov We encourage you to:1) Explore our online resources via the Solution Center2) Submit a request via the Technical Assistance Center

6. Who We AreProgram Design & Implementation/Technical Assistance Team

7. Today’s SpeakersJulie Michals, Director, EM&V Forum at Northeast Energy Efficiency Partnerships, Inc.Chris Neme, Principal, Energy Futures Group

8. OVERVIEWDefining Evaluation, Measurement & Verification (EM&V)DOE Guidance on EM&V and ReportingBenefits of EM&VDeveloping a plan for Residential Retrofit EE programsQ&A

9. What is EM&V? Evaluation: Impact evaluation: quantification of energy savings & other benefitsUsing M&V toolsCan be at multiple levels – program, market segment, measure, etc.Process evaluation: assessment of program design & proceduresTypically interviews, focus groups, etc.Typically focused on customers, trade allies, program staffMeasurement & Verification: Collection of data (pre- and post efficiency measure installation)…Product info (e.g. manufacturer efficiency ratings)Metering (i.e. on-site measurement of energy use, power draw, hours of use)Billing dataSimulation modeling (calibrated to building energy usage)…to support estimates of energy savings

10. DOE EM&V Guidance DOE requires reporting of energy savings, but not specific EM&V as part of Grantees award agreementEstimates can be calculated using Recovery Act Benefits Calculator, but not intended to replace more rigorous EM&V techniqueshttp://www1.eere.energy.gov/wip/solutioncenter/calculator/default.aspxGrantees with resources to conduct more sophisticated EM&V encouraged to conduct studies in accordance with Program Notice 10-017, and to share results with DOE through Project Officers. See http://www1.eere.energy.gov/wip/pdfs/eecbg_evaluation_guidelines_10_017.pdf

11. Detailed Data CollectionDetailed Data Collection - DOE Guidance for SEP Grantees:Contact Information of people served/impacted (name, company, address of contact, phone, email)Detail descriptions of services received: address of actions taken, recommendations from audits, measures taken, installation dates etc.e.g., CA Evaluation Protocols (April 2006, pg 205)http://www.calmac.org/events/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf

12. Reporting required on: Job Impacts Energy SavingsEnergy Costs ReductionsRenewable Energy Capacity and GenerationEmission ReductionsProcess Metrics: # buildings retrofitted, square footage, efficiency measures purchased, etc.EEBCG Program Guidance 10-07B - Reporting: http://www1.eere.energy.gov/wip/pdfs/eecbg_reporting_program_guidance_10_007b.pdfSEP Program Guidance 10-006BA – Reporting:http://www1.eere.energy.gov/wip/pdfs/sep_10-006a_arra_reporting_guidance.pdf DOE Reporting Requirements

13. Characteristics of a Leader

14. Best Practices• Identify and convene stakeholders• Establish a leadership team• Develop a common energy vision• Develop a community energy baseline• Based on the vision and baseline, develop energy goals• Evaluate supply & demand policy and program resource options • Find and secure funding sources• Compile the plan• Measure & evaluate – continuously improve plan Strategic Energy Planning

15. Why Conduct EM&V?Retrospective:How much: energy and money saved, pollution reduced, jobs created, etc.?Was the investment cost-effective?Prospective:How can the program be improved?What appealed to participants? Why didn’t others participate?Did some retrofit contractors do better than others? Why?Actual savings vs. forecasted savings“closing rates” (from audit to completed jobs)Comprehensiveness of treatmentWere opportunities for additional cost-effective savings missed? Why?Note: Can also benefit program as it is being delivered

16. Why Develop EM&V Plans?EM&V Plan should ideally be developed before program launch:Critical to ensure the right data are collectedClarity around data tracking & reporting needsClarity around responsibilities for dataEnsures communications w/utilities about leveraging start earlySome EM&V features can be program design featuresAllows for on-going program refinement – not just “after-the-fact”Highlights expectations regarding accountabilityProgram staffRetrofit contractorsEnsures adequate budget set aside

17. EM&V Plan – Approaches for Residential RetrofitImpact Assessment Options Deemed savings Building energy modeling Billing analysisProcess Evaluation Tracking database “mining” Expert drive-alongs with retrofit contractors Interviews – participants, non-participants, staff, contractorsAll of these can be done either independently or piggybacking on utility/other EM&V efforts (or combination of the two)…

18. EM&V Plan – Deemed SavingsIndependent ApproachIdentify likely common efficiency measuresDevelop engineering assumptions/algorithms tailored to local situationEnsure all data needed for calculations are collected for each jobRequires specialized efficiency expertiseCan be hiredNot necessarily expensive ($5k-$25k) for limited range of measuresNeeds periodic maintenance/updatingPiggybackingReach out to utility/others to identify existing assumptions/algorithmsDetermine whether refinements neededGet utility help to refine or add measuresNote: combination of two approaches possible as well

19. EM&V Plan – Deemed Savings (2)Numerous existing “local” tools:e.g.: CA, Northwest, OH, MI, NJ, Mid-Atlantic, VT, MA – partial listBut not all address building envelope measuresQuality variesEase of use variesThus, leveraging still requires some technical expertise or supportSee Slide #31 for Deemed Savings references/resources

20. EM&V Plan – Building ModelingIndependent ApproachIdentify modeling tool(s) that will be used“Home Energy Score” – U.S. DOE pilotMany others…Distribute and train retrofit contractors on their useCollect pre- and post-treatment modeling results for each homeCould also be done “after the fact”, if all necessary data are collectedPotential program design/delivery advantagesAlso serves as “sales tool”Gives participants a “leave behind” – energy rating/performance scorePiggybackingIf utility/others using this approach, get community-specific dataSummarize & synthesize community-specific dataNote: May need to combine w/ Deemed Savings for some measures

21. DOE’s Home Energy Score Tool

22. Earth Advantage EPS(using SIMPLE algorithms)

23. EM&V Plan – Billing AnalysisIndependent ApproachMost accurateNeed to collect participants’ energy billsImportant to get “releases” signed during service delivery – difficult later!Still not always without difficultiesShould have at least 12 months pre- and post-treatment infoRequires specialized technical expertise & statistical toolsNeeds to be hiredNot necessarily expensive (10-30k) if you have good data, find right firmPiggybackingIf utility/others using this approach, get community-specific dataRequires up-front collaboration on EM&V design w/utilitiesWill cost utility extra $, possibly necessitating community contributionNote: May need to combine w/Deemed Savings for some measures

24. EM&V Plan – Process EvaluationIndependent ApproachSome aspects can be conducted in-house:Participant surveysNon-participant surveysContractor interviews or focus groupsSome approaches require hiring specialized technical expertiseDrive-alongs with retrofit contractorsTracking system reviewsNot necessarily expensive ($5-$15k) for targeted scope & the right firmPiggybackingPossible to add extra, community-specific questions to surveysRequires up-front collaboration on EM&V design with utilities/othersWill cost utility extra $, possibly necessitating community contribution

25. Cost Varies depending on extent of EM&VNeed to consider volume of participation relative to costRequires technical expertise – likely not in-houseEven leveraging existing utility efforts likely to require expertiseLimited range of contractors willing to do “small” projectsData availabilityCritical to good evaluationEssential to minimize costs of hiring contractorsRequires on-going monitoring of what you are getting from the fieldRequires up-front planning – integration w/program deliveryCollecting right data as you goChoosing an accounting/tracking softwareEase of use for evaluation purposesProgram management tool too!Barriers to EM&V

26. EM&V Steps & Ideal Timeline

27. ADDITIONAL BACKGROUND INFO AND RESOURCES

28. EM&V Planning GuidanceInternational Performance Measurement & Verification Protocol (IPMVP Vol 1, 2010 www.evo-world.org)Detailed steps for comprehensive M&V planning:http://www1.eere.energy.gov/femp/pdfs/intro_mv.pdf

29. DOE Guidance on EM&V – for GranteesConducting 3rd Party EvaluationsGuidelines for States Conducting or Contracting Evaluations of ARRA Funded SEP Activities (using 3rd party contractors): http://www1.eere.energy.gov/wip/pdfs/evaluation_webinar_slides_june16_2010.pdfHigh level guidelines/standards on:Evaluation Metrics – energy/demand savings, carbon emission reductions, job creationIndependent Evaluations – by 3rd independent partyAttribution of Effects – net effects due to SEP funds, with guidance on allocation of effects for jointly funded projectsEvaluation Budgeting – recommends 5% or less of project budgetTiming of Evaluation – evaluation planning to start at same time as when projects are initiated, determine baseline approach, data collection and analysis efforts Continued…

30. DOE Guidance on EM&V – for GranteesConducting 3rd Party Evaluations cont.High level guidelines/standards cont:State of the Art Analysis – evaluation approach should use current state of the art evaluation approaches and analysis methodsEvaluation Rigor and Reliability: Study should be as reliable as possible within study approach and budget limitsStudy Design and Study Plan: Study methods/approach, tasks to be conducted, detailed data collection approach, detailed analysis approach for energy and demand savingsSampling and Statistical Significance: minimize bias and maximize representativeness of the population. Sample to be no less rigorous than 90% confidence level with +/- 10% precisionM&V Approaches: analytic approach, baseline and post-installation operation assessments should use IPMVP field data collection frameworks (discussed later)

31. Available Energy Savings DataGrantees can refer to existing state energy efficiency program administrator data assumptions and algorithms if project data is not all available/collected. These “Technical Reference Manuals” (TRMs) include a mix of stipulated data, calculations based on models, prior EM&V studies and/or manufacturer specs. Several should be considered before using these sourcesExisting resources include:NW Regional Technical Forum: http://www.nwcouncil.org/energy/rtf/ Other state savings assumptions documents for: CT, MA, ME, NJ, NY, VT, PA and multi-state (MD, DC, DE) available at: http://neep.org/emv-forum/emv-library/research-evaluation-studiesCalifornia DEER Database: http://www.energy.ca.gov/deer/ Other state TRMsCEE evaluation clearinghouse: http://www.cee1.org/eval/clearinghouse.php3

32. Approaches/methods range from simple and direct to complex and indirect, sometimes combined, where more complex methods generally require more detailed data and higher costGuidelines for EM&V measurement/analysis include: US DOE/EPA Model Energy Efficiency Program Impact Evaluation Guidehttp://www.epa.gov/cleanenergy/documents/suca/evaluation_guide.pdf Regional EM&V Forum Guidelines: http://neep.org/emv-forum/forum-products-and-guidelines NW Regional Technical Forum Protocols http://www.nwcouncil.org/energy/rtf/ U.S. FEMP M&V Guidelines: Measurement and Verification for Federal Energy Projects Version 3.0, 2008 http://mnv.lbl.gov/ASHRAE Guideline 14: Measurement of Energy and Demand Savings (2002) – updated version forthcoming 2011. www.ashrae.org CA Evaluation Protocols: http://www.calmac.orgMost of the above refer to IPMVP: The International Performance Measurement & Verification Protocol (IPMVP Vol 1, 2010 www.evo-world.org)3. The spread of high resolution usage data (AMI) and other new technologies provides the opportunity for better methods in the future – stay tuned!Which EM&V Approach to Use?

33. Q&AQuestions?

34. Questions?34CONTACTSVEIC: Dan Quinlan, dquinlan@veic.org, 802-488-7677 (Team Lead)MEEA: Steve Kismohr, skismohr@mwalliance.org, 312-784-7257NEEP: Ed Londergan, elondergan@neep.org, 781-860-9177NEEA: Elaine Blatt, eblatt@nwalliance.org, 503-688-5458SWEEP: Curtis Framel, cframel@swenergy.org, 303-447-0078SEEA: Scott Slusher, scott@seealliance.org, 480-239-4236ACEEE: Eric Mackres, emackres@aceee.org, 202-507-4038NRDC: Lara Ettenson, lettenson@nrdc.org, 415-875-6100EFG: Richard Faesy, rfaesy@energyfuturesgroup.com, 802-482-5001

35. Basic Benchmarking: Benchmarking Your Building’s Energy Use Using ENERGY STAR’s Portfolio ManagerHost: Courtney Smith, ICFDate: March 30, 2011Time: 12:00-1:30 EDT Upcoming WebinarsFor the most up-to-date information and registration links, please visit the Solution Center webcast page at www.wip.energy.gov/solutioncenter/webcasts Please join us again: