JULY 9, 2015 Laurie Corcoran - PowerPoint Presentation

1. JULY 9, 2015Laurie CorcoranSenior analyst demand resource strategySimplifying administration and maintaining baseline accuracyProposed Changes to the Demand Response Baseline Method

2. BackgroundWhy modify the baseline method?

3. BackgroundAs mentioned at the May 2015 Markets Committee meeting, the current 90/10 baseline methodology is administratively complexThe ISO is proposing to replace the 90/10 baseline methodology with a method which performs comparably or better in terms of accuracy, bias, variability, and reduces the administrative burdenAdditionally, the new methodology must calculate baselines for two additional day types at the time of full integration: Saturdays, and Sunday/Holidays3

4. Current Baseline MethodologyThe ISO’s current 90/10 baseline methodology:Estimates the expected energy consumption of an asset by simulating a 10-day rolling average of meter data from previous days on which the asset was not dispatchedUnder this approach, the ISO takes 90 percent of the previously calculated baseline for each 5-minute interval of an Operating Day, and then adds to each interval 10 percent of the interval meter data from the most recent day on which the asset was not dispatchedComprehensive analysis of baseline methodologies conducted by DNV GL/KEMA (“DNV”) for PJM in 2011 demonstrated that ISO’s 90/10 baseline methodology was more accurate, and had lower bias and variability compared to other known methodologies4

5. Administrative IssuesThe meter data used to compute the 90/10 baseline remains embedded in the calculated baseline forever – i.e., the 90/10 baseline methodology is said to have an “infinite tail”The ISO and participants must store and manage many months of data to potentially recalculate baselines if any meter data is found to be incorrect even though the impact of any set of data becomes de minimus over timeAdditionally, the 90/10 baseline methodology involves a recursive algorithm in which today’s calculated baseline is a function of previously calculated baselines, so that:Every time a data correction is made, every subsequent baseline, starting with the baseline for the first day that was computed using the corrected data, needs to be recalculated; Any data corrections during a resettlement period impact all future settlements; andWhen data issues are identified following the end of a data resubmission period, unwinding the impacts of such data is complex.5

6. Proposed baseline modifications

7. Baseline AnalysisIn 2014, the ISO analyzed alternative baseline methodologies in comparison to the ISO’s 90/10 baseline methodology to determine which one(s) would:Achieve comparable or better accuracy, bias, and variabilityAddress the administrative issues with the 90/10 methodologyThe analysis found that a “Mean 10 of 10” baseline using a 6 week look back period:Performs comparably to or better than the current 90/10 methodology in terms of accuracy, bias, and variability* Reduces the administrative burden by:Eliminating the infinite tailAddressing the recursive nature of the 90/10 methodologySimplifies the calculation of Saturday and Sunday/Holiday baselines at the time of full integration 7* Details on the analysis, including the metrics used to determine accuracy, bias, and variability are available in the Appendix section of this presentation.

8. The ISO Proposes to Replace the Current Baseline with the Mean 10 of 10 Baseline MethodologyThe Mean 10 of 10 baseline is a 10-day rolling average of meter data from previous days on which the asset was not dispatchedBaseline accuracy, bias, variability are preservedTo address the infinite tail issue, the historical look-back period is limited to 6 weeks*To address the recursive nature of the ISO 90/10, the baseline calculation uses only meter data from ten historical days – i.e., prior day baselines are not used to compute future baselinesBaselines for Saturdays and Sunday/Holidays are 6-day rolling averages of meter data* 8*The decisions to limit the historical look back period to 6 weeks, and calculate Saturday and Sunday/holiday baselines as 6-day rolling averages, are explained in the Appendix section of this presentation.

9. Tariff ChangesThere are two places where the Tariff must be changed to replace the 90/10 with the Mean 10 of 10 baseline methodology:Section III.8A of the Tariff The Mean 10 of 10 methodology will be used to calculate baselines for single day type (weekdays)Section III.8B of the Tariff For full integration (now proposed to be implemented on June 1, 2018), the Mean 10 of 10 methodology will be used to calculate baselines for three day types (weekdays, Saturdays, and Sunday/Holidays)9

10. Proposed Stakeholder ScheduleTimingActionJuly 2015Introduce a proposal to modify the ISO 90/10 baseline methodology.August 2015Proposed market rule changes to implement a Mean 10 of 10 baseline methodology are introduced for stakeholder review and consideration. September 2015Markets Committee vote on market rule changes.October 2015Participants Committee vote on market rule changes. Filing of rule changes with the Commission. 10

11. 11

12. appendixAlternative Baseline Analysis

13. ISO Analysis of Baseline Administration IssuesPossible options to address the 90/10 baseline administration issuesTo address the infinite tail issue, use a finite look back periodTo address the recursion issue, calculate baselines using only meter data – i.e., do not include prior day baselines in the baseline calculationBecause the ISO 90/10 baseline methodology is recursive by nature, other methodologies needed to be identified and analyzed13

14. Solutions to Baseline Administration IssuesConsider a finite look back period of approximately 6 weeksMultiple baseline analyses in the last 10 years have considered the length of the look back period, and periods of 30 to 60 days have been studiedInitially, the ISO considered a look back period of 30 calendar days, however, in a 30 day period there may be only 4 Saturdays and 4 Sunday/holidaysIn order to ensure more historical data for Saturdays and Sunday/holidays is available for the baseline calculation, while respecting a finite look back period, the ISO extended the look back to 30 non-holiday weekdays (approximately 6 weeks)For non-holiday weekdays, limit the look back to the most recent 30 days of that day typeFor Saturdays and Sunday/holidays, limit the look back to the most recent 6 days of that day typeConsider calculating baselines using only meter data, and not using previously computed baselines Avoids issue with ISO 90/10 where any changes to previous day baselines impact future baselines 14

15. Analysis of Alternatives to the Current ISO 90/10 Baseline MethodologyGiven the recursive nature of the ISO 90/10 methodology, analysis was conducted to identify a new approach that:Achieves comparable or better accuracy, bias, and variabilityAddresses the administrative issues with the 90/10 methodologyWith analytical support provided by DNV, the ISO examined:Six alternative baseline methodologies in comparison to ISO 90/10The impact of limiting the look back period to approximately 6 weeks, and calculating baselines using only meter dataBaselines for 3 day types:Non-holiday weekdays,Saturdays, andSunday/holidaysAdjusting baselines using the Real-Time Forecast Adjustment Method* (“dynamic baseline adjustment”) accepted by the FERC on 1/9/2015 (See ISO New England Inc. and New England Power Pool, 150 FERC ¶ 61,007)15* http://www.iso-ne.com/static-assets/documents/committees/comm_wkgrps/mrkts_comm/mrkts/mtrls/2014/may672014/a03_iso_presentation_05_06_14.pptx

16. Alternative Baseline MethodologiesMedian 10 of 10 baseline – Median of the ten most recent non-event, non-holiday, weekdaysMean 10 of 10 baseline – Simple average of the ten most recent non-event, non-holiday, weekdaysWeighted 10 of 10 baseline – Weighted average of the ten most recent non-event, non-holiday, weekdays. Maximum weight is given to the 1st day or closest day (15%) and the weight assigned to each next day is gradually decreased giving the least weight to the 10th day or the furthest day (6%)Median 90-10 baseline – This baseline is a combination of the Median 10 of 10 and the Weighted 10 of 10 baselinesTop 5 of 10 baseline – Simple average of top 5 of the ten most recent non-event, non-holiday, weekdays, where the top days are based on the daily average hourly load. If there are fewer than ten non-event days in the last 30 days, then fill in as needed with most recent event daysMiddle 2 of 10 baseline – Simple average of middle 2 (i.e., median) of the ten most recent non-event, non-holiday, weekdays. The two middle days of the 10 are determined based on the daily average hourly load16

17. Analysis of Alternative Baseline MethodologiesFor purposes of the analysis, the current ISO 90/10 baseline methodology, incorporated the following:The dynamic baseline adjustmentSaturday and Sunday/holiday baselines were calculated from 10 days of the same day typeFor all alternate baseline methodologies, the 10 most recent non-event, non-holiday weekdays were chosen from the look back period as inputs to the weekday baseline calculationWhere less than 10 non-event, non-holiday, weekdays were available, the most recent interruption weekdays were included as neededFor Saturday and Sunday/holiday baselines, the most recent 6 days of the same day type (whether non-event or event days) were chosen as inputs17

18. Comparison MetricsAlternate baseline methodologies were compared on aggregate metrics for accuracy, bias, and variabilityAccuracy is a measure of how closely the baseline predicts the asset’s actual loadBias is the systematic tendency of a baseline method to over-predict or under-predict actual loadVariability is a measure of how well the baseline predicts actual load under many different conditions (e.g., time of day, day of week) and across many different customersBaseline performance was measured during simulated hourly eventsEach baseline is adjusted as if the asset were asked to reduce for an hour, on the hour, for all hours included in the analysisThe difference between the adjusted baseline and actual load (error) in each hour serves as basis of comparisonErrors are calculated for each asset, for each 5-minute interval and then integrated to hourly values18

19. Comparison Metrics (cont.)Basis of comparisonActual Error: adjusted baseline minus actual loadRelative Error: actual error divided by average annual hourly loadMetrics (calculated across all assets)Mean actual errorMeasures bias: reflects the actual magnitude of asset load; may be affected by outliersMedian Relative ErrorMeasures bias: gives assets the same weight regardless of their actual sizeMedian Relative Root Mean Squared Error (“RRMSE”) or AccuracyCombined measure of bias and variability: weights large errors more heavily than small errors95th percentile of relative standard deviationMeasures variability: gives an estimate of variability (on a relative basis) at higher level19

20. Comparison Metrics (cont.)Additionally, metrics were analyzed based on various classificationsDay type (weekday, Saturdays, Sunday/Holiday)Season (Summer, Winter)20

21. Comparison of Alternative Baselines21Mean 10 of 10Median 10 of 10Weighted 10 of 10Median 90-10Top 5 of 10Middle 2 of 10ISO 90/10*Calculated across all assets and seasons between October 1, 2012-September 30, 2013*

22. 22Mean 10 of 10Median 10 of 10Weighted 10 of 10Median 90-10ISO 90/10**Calculated across all assets and seasons between October 1, 2012-September 30, 2013Comparison of Alternative Baselines – Excluding Lowest Performers

23. Outcome of Comprehensive AnalysisA group of three methodologies (Mean 10 of 10, Weighted 10 of 10, Median 90-10) consistently performed better than Median 10 of 10, Top 5 of 10, and Middle 2 of 10, and performed very similarly to each other and to ISO 90/10Out of the top performing alternatives, the ISO recommends the Mean 10 of 10 methodology to replace ISO 90/10 due to its simplicity23