November 2017 IA 20252 Earth Observing System Data and Information System Contents About CFI Group Introduction and Methodology Survey and Data Collection Summary Executive Summary Survey Results ID: 712670
Download Presentation The PPT/PDF document "2017 Customer Satisfaction Results" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
2017 Customer Satisfaction Results
November 2017IA# 20252
Earth Observing System Data and Information SystemSlide2
Contents
About CFI GroupIntroduction and Methodology
Survey and Data Collection Summary
Executive Summary
Survey Results
Customer Satisfaction Model Results
CSI by DAAC and Other Segments
Satisfaction Driver DetailSlide3
About CFI Group
Founded in 1988Founding partner of the ACSI
Patent holder of the modeling engine used to compute the ACSI
Predictive analytics software and professional services
Serving a global list of clients from 6 offices on 4 continents
Providing “actionable” customer feedback insights based on the science of the ACSI
CFI GROUP WORLDWIDE
USA – Ann Arbor, MI
ENGLAND – London
ITALY – Milan
CHINA – Shanghai
SWEDEN – StockholmSlide4
Introduction & MethodologySlide5
Introduction and Methodology
Measure customer satisfaction with NASA Earth Observing System Data and Information System (EOSDIS) at a national level for each Distributed Active Archive Center (DAAC).Identify the key areas that NASA can leverage across the DAACs to continuously improve its service to its customers.
Assess the trends in satisfaction with NASA EOSDIS specifically in the following areas:
Customer SupportProduct Selection and Order
Product Search
Product Documentation
Product Quality
DeliverySlide6
Survey and Data CollectionSlide7
Survey and Data Collection Summary
Questionnaire developed by NASA EOSDIS and CFI Group.Measured respondent satisfaction and their experiences with a specific DAAC
The survey was designed to allow users to skip over the questions not related to their experience with the specified DAAC.
Each DAAC was allowed the opportunity to utilize their own unique supplemental questions (outside of the ACSI model questions).
Data collection performed via the web.NASA EOSDIS provided multiple lists of email addresses, which after combining, cleaning, and deduping, CFI Group sent out 289,745 email invitations.
A total of 7,505 responses were received, for a response rate of 2.6%.
The online survey was available September 7
th
through October 4
th, 2017.Three survey reminder announcements were sent by CFI Group (September 21st, September 26th, and October 2nd).Slide8
Executive Summary Slide9
Executive Summary: CSI and Performance Outcomes
The average aggregate Customer Satisfaction Index (CSI) score for NASA EOSDIS over the last eleven years is 77. The 2017 score is 78 and represents performance that is generally strong and consistent with that average. Future Behaviors
are likewise consistent over the last eleven years. The 2017 score of 87 for Likelihood to Recommend is one point above the eleven year average of 86 while this year’s score of 89 for Likelihood to Use the Services Again in the Future is also one point above the eleven year average score of 88.
All drivers of satisfaction, were rated at 80 or above on the aggregate, which is a good indicator of consistently strong performance across the customer experience.
At the DAAC level, ORNL DAAC (82) rated the highest by respondents in this year’s study. NSIDC DAAC (79,+3) and SEDAC (75, +2) realized the largest score increases in CSI over the last year.
The aggregate CSI score of 78 outpaces the ASCI aggregate Government score of 68 and is just above the Q3 2017 ACSI score of 77 for overall U.S. satisfaction.Slide10
Executive Summary: Product Quality
Product Quality is the highest impact driver of CSI and increased two points from last year as it posted a score of 85 for 2017. This is the highest Product Quality score since the inception of the survey. Respondents were satisfied across all aspects of product quality as all driver attributes scored in the mid 80s. The majority of respondents (78%) used software tools or packages to work with the data and report a strong CSI of 81. The 27% who made their own tool using a programming language to work with data were similarly satisfied (81) so this reinforces the high ratings on the ease of using the data as it is delivered. There was a small group of respondents who couldn’t understand how to use the data. Obviously they were the least satisfied, with a CSI of 68. Because of its high impact on satisfaction, maintaining a
favorable Product Quality score is essential to maintaining high CSI scores. Slide11
Executive Summary: Product Selection and Order
Product Selection and Order has a strong influence on CSI (1.0) and a posts a score of 83 in 2017. This combination of scores and impact make Product Selection and Order a strength of the aggregate DAACs that should remain a priority for attention in the future.
Seventy-eight percent of respondents requested/acquired data products from a DAAC in the last year. Those that requested/acquired data had higher CSI scores (80) than those who did not (75).
While respondents were pleased with the ‘Ease of selecting data products’ (82) as well as the ‘Ease of requesting or ordering data products’ (83), they were most pleased with ‘Direct downloads’ (84).Slide12
Executive Summary: Other Key Drivers
Both Customer Support (87) and Product Documentation (80) have a moderately high impact of 0.9. Customer Support is the highest rated satisfaction driver in the NASA EOSDIS satisfaction model while Product Documentation is the lowest rated. Product Search (82) and Delivery (84) both score well. Ninety-four percent of respondents downloaded or received data, so any improvements in either of these areas will affect a majority of the users.
All driver scores in the aggregate either held steady or improved in 2017. Some DAACs had declines in specific areas. Since aggregate scores have been consistently strong, this may be a good opportunity to better understand any micro differences among the various DAACs.Slide13
Customer Satisfaction Model ResultsSlide14
14
2017 NASA EOSDIS – Customer Satisfaction Model
Likelihood to Use Service in the Future
89
3.4
Likelihood to Recommend
87
3.8
SATISFACTION DRIVERS
FUTURE BEHAVIORS
Scores represent your performance as rated by
your
customers
Driver Impacts show you which driver has the most/least leverage – where improvements matter most/least to
your
customers
Future Behavior Impacts represent the impact of CSI on the future behaviors of
your
customers
Overall Satisfaction 81
Compared to Expectations 77
Compared to Ideal 76
Customer
Satisfaction
Index
78
Product Documentation
80
0.9
Product Quality
85
1.1
Product Selection and Order
83
1.0
Product Search
82
0.8
Customer Support
87
0.9
Delivery
84
0.8Slide15
Priorities for NASA EOSDIS
Impact on SatisfactionDriver Score
Maintain
Strengths
Top Priority
Minimal ConcernSlide16
The Customer Satisfaction Index (CSI) is 78, which is consistent with performance over the last four years.
Likelihood to Recommend is very strong and remains steady at 87. Respondents indicate a very strong Likelihood to use Services in the Future (89).CSI and Performance Outcomes: Four-year TrendingSlide17
At 78, NASA is 1 point above the ACSI average (77), and 10 points above the Federal Government average (68).
Scores in green represent CSI for other Federal Government Agency information providers measured by CFI.BenchmarksSlide18
CSI by DAAC and Other SegmentsSlide19
CSI and Frequency by DAAC
LP DAAC was again the most frequently cited DAAC for evaluation (38%).ORNL DAAC (82), ASF SAR DAAC (80) and PO DAAC-JPL (80) were the highest-scoring DAACs.
NSIDC DAAC (+3) and SEDAC (+2) realized the largest score increases in CSI over the last year.
2016
2016
2016
2017
2017
2017
Data center evaluated
%
N
CSI
%
N
CSI
Data center evaluated
%
N
CSI
%
N
CSI
ASDC-LaRC
9%
640
76
7%
548
77
ASF SAR DAAC
6%
412
79
9%
688
80
CDDIS
3%
214
78
3%
191
77
GES DISC
9%
634
77
11%
830
77
GHRC
5%
362
74
5%
405
72
LP DAAC
40%
2,839
78
38%
2,829
79
MODAPS LAADS
12%
822
77
11%
847
78
NSIDC DAAC
6%
398
76
5%
396
79
OB.DAAC
2%
156
81
2%
124
76
ORNL DAAC
2%
145
81
2%
164
82
PO DAAC-JPL
3%
186
79
2%
187
80
SEDAC
5%
325
73
4%
296
75
Number of Respondents
7,133
7,133
7,133
7,505
7,505
7,505Slide20
CSI: Four-year Comparison by DAAC
CSI moved very slightly for all DAACs compared to a year ago.
ASF SAR DAAC, SEDAC, and ORNL DAAC have shown slight but consistent yearly improvement since 2014. Slide21
CSI is two points higher for domestic respondents, driven primarily by higher scores in Customer Support, although Product Quality, and Delivery also scored higher.
Domestic respondents also were more likely to recommend and/or use the service in the future.CSI and Driver Scores: USA vs. All Other Countries* Indicates a Significant Difference between scores at 90% confidence level
Sample Size
USA
All Others
Difference
Significant Difference
Sample Size
1,272
6,233
NA
NA
Product Search
80
82
2
*
Product Selection and Order
83
83
0
Delivery
85
84
-1
Product Quality
87
84
-3
*
Product Documentation
80
80
0
Customer Support
92
86
-6
*
Customer Satisfaction Index
80
78
-2
*
Likelihood to Recommend
90
87
-3
*
Likelihood to Use Services in Future
92
88
-4
*Slide22
Yearly CSI Trend by LocationSlide23
CSI and Frequency by Type of User
What type of user are you? Select all that apply.~ Multiple responses allowed
Fifty-four percent of users were either Graduate Students (29%), Professors (16%) or Undergraduate students (9%). Earth Science Researchers (32%) represent the single most common user type.
University Professors again report the highest level of satisfaction (81) while Decision Support Systems Analysts, the General Public, and University Undergraduates report the lowest levels of CSI (76).
2016
2016
2016
2017
2017
2017
Type of User~
%
N
CSI
%
N
CSI
Type of User~
%
N
CSI
%
N
CSI
General Public
14%
1,019
76
14%
1,037
76
Elementary, Middle, High School Teacher
1%
83
76
1%
86
77
University Professor
16%
1,129
80
16%
1,193
81
University Undergraduate Student
36%
2,550
76
9%
656
76
Other Education and Outreach
5%
349
79
5%
355
79
Earth Science Researcher
32%
2,304
79
32%
2,409
79
Earth Science Modeler
8%
574
78
9%
650
79
NASA-affiliated Scientist
2%
167
79
1%
102
80
Non-NASA-affiliated Scientist
4%
304
79
4%
320
78
NASA Science Team Member
7%
475
79
1%
68
80
Data Tool Developer/Provider
5%
359
77
5%
409
77
Decision Support Systems Analyst
5%
375
76
6%
429
76
University Graduate Student
0%
0
--
29%
2,204
77
Other User Type
8%
548
76
9%
656
77
Number of Respondents
7,133
7,133
7,133
7,505
7,505
7,505Slide24
Two-thirds (67%) of respondents indicate they use the data and services for Land. Atmosphere (29%), Biosphere (16%), and Ocean (14%) are the next most common uses.
There is little variation among satisfaction levels depending on areas/disciplines of use.Areas/Disciplines Need/Use Earth Science Data and Services
For which general areas/disciplines do you need or use Earth science data and services? Select all that apply.
~ Multiple responses allowed
2016
2016
2016
2017
2017
2017
General areas need or use Earth science data and services~
%
N
CSI
%
N
CSI
General areas need or use Earth science data and services~
%
N
CSI
%
N
CSI
Atmosphere
28%
1,962
77
29%
2,206
78
Biosphere
15%
1,058
78
16%
1,198
79
Calibrated radiance
8%
592
78
8%
629
79
Cryosphere
7%
526
78
7%
556
79
Human dimensions
12%
863
76
13%
984
77
Land
66%
4,742
77
67%
5,062
78
Near-real-time applications
12%
862
77
13%
1,008
79
Ocean
15%
1,042
79
14%
1,072
78
Space geodesy
9%
634
76
9%
692
77
Other general area
8%
551
76
8%
630
76
Not applicable
0%
24
78
0%
36
82
Number of Respondents
7,133
7,133
7,133
7,505
7,505
7,505Slide25
Driver Detail: Product QualitySlide26
Product Quality (85) improves two points from last year. This increase was driven by a two-point increase in all attribute items across the board.
With an impact of 1.1 on CSI, Product Quality emerges as the highest-impact driver in 2017.Product Quality
Impact = 1.1Slide27
Product Quality performance by DAAC has remained fairly steady over the years.
ORNL DAAC has consistently gained every year and is nine points above their score of 80 in 2014. Both CDDIS (89) and ORNL DAAC (89) had the largest score fluctuations this year as both increased four points from last year. Product Quality: Four-year Comparison by DAACSlide28
More than three-quarters (78%) reported using software tools/packages, while just over a quarter (27%) made their own tools using programming language. Both groups had identical CSI.
Python was the most popular programming language as it was preferred by 37% of the respondents.Software Tools/Packages Used to Work with Data
Did you use software tools/packages to work with the data?
2016
2016
2016
2017
2017
2017
%
N
CSI
%
N
CSI
Used software tools or packages to work with data ~
%
N
CSI
%
N
CSI
Yes, used software tools or packages to work with data
78%
3,474
80
78%
3,656
81
Yes, made own using a programming language
26%
1,181
80
27%
1,286
81
No, couldn´t find what I needed
2%
82
71
2%
89
70
No, couldn´t understand how to use it
2%
89
67
2%
83
68
No, did not need software tools
5%
227
82
6%
266
83
Number of Respondents
4,462
4,462
4,462
4,711
4,711
4,711
Preferred programming language
%
N
CSI
%
N
CSI
C
4%
147
82
3%
125
83
C++
6%
251
80
5%
194
81
C#
1%
57
79
1%
36
78
Fortran 77
2%
66
83
1%
50
84
Fortran 90
6%
225
83
5%
201
81
Java
5%
221
77
5%
194
79
Perl
1%
37
81
1%
31
84
PHP
0%
20
80
1%
23
85
Python
33%
1,354
80
37%
1,568
81
Other
12%
488
80
18%
790
81
Don´t know/Not applicable
29%
1,198
80
25%
1,061
81
Number of Respondents
4,064
4,064
4,064
4,273
4,273
4,273Slide29
ArcGIS is the most used software tool/package at 64%, followed by Quantum GIS (37%), ENVI (32%) and Excel (27%).
There only minimal differences in CSI among the most popular tools. Tools Used to Work with Data
Please select the tool or tools you used to work with the data from <DAAC>.
2017
2017
2017
Used software tools or packages to work with data ~~
%
N
CSI
Tool(s)
used to work with data~
%
N
CSI
ArcGIS
64%
2,349
81
Convert to Vector
4%
159
82
ENVI
32%
1,176
81
ERDAS/IMAGINE
21%
766
82
Excel
27%
980
81
Ferret
1%
26
82
Geomatica
4%
148
83
Global Mapper
15%
562
81
GrADS
3%
118
81
GRASS
12%
435
82
HDFLook
1%
51
83
HDFView
9%
311
82
HEG
2%
67
85
IDL
11%
400
82
IDV
1%
26
84
IDRISI
7%
248
84
MapReady
1%
41
87
MATLAB
18%
649
81
MODIS Reprojection Tool (MRT)
10%
355
84
NCL
3%
100
81
Panoply
5%
201
81
Quantum GIS (QGIS)
37%
1,344
82
R
19%
680
82
SeaDAS
3%
123
83
Other/open source
18%
676
81
Don´t know/Not applicable
1%
26
82
Number of Respondents
3,656
3,656
3,656
~ Multiple responses allowedSlide30
Driver Detail: Product Selection and OrderSlide31
Seventy-eight percent of respondents requested/acquired data products from a DAAC in the last year. Product Selection and Order has a strong influence on CSI (1.0), and with an equally strong score of 83 is considered a strength of the aggregate DAACs.
All attribute areas show very little score variation in the last four years.Product Selection and Order
Impact = 1.0
* Response option new in 2017Slide32
Product Selection and Order scores by DAAC have remained relatively consistent over the past several years.
CDDIS (88), ORNL DAAC (86) and ASF SAR DAAC (86) report the highest Product Selection and Order scores in 2017. DAAC scores range from 78 to 88.Product Selection and Order: Four-year Comparison by DAACSlide33
Driver Detail: Customer SupportSlide34
Sixteen percent of respondents indicated that they contacted a DAAC user services office or interacted with DAAC personnel in the past year, which is two percentage points lower than last year.
Customer Support increased 2 points from last year and is the highest performing driver in 2017. Due to its high score and high impact on CSI, Customer Support should be considered a strength of the program.Customer Support
Impact = 0.9Slide35
Customer Support scores particularly well as all DAACs either held steady or increased in 2017.
ORNL DAAC (96) and PO DAAC-JPL (92) score the highest among all DAACs.Customer Support: Four-year Comparison by DAACSlide36
Driver Detail: Product DocumentationSlide37
Seventy-two percent of respondents looked for or obtained documentation related to the data, which is consistent with last year. Scores for this driver have had a one-point statistically significant increase both this year and in 2016.
Product Documentation has a relatively strong degree of leverage on CSI, at 0.9. Product Documentation
Impact = 0.9Slide38
Product Documentation scores for all DAACs either held steady or increased in 2017.
DAAC scores are relatively consistent, and range from a high of 85 (ORNL DAAC) to a low of 78 (shared by three DAACs).Product Documentation: Four-year Comparison by DAACSlide39
Driver Detail: Product SearchSlide40
The Product Search driver increased two points in 2017 to score 82.
The Product Search score increase was powered by increases in both attributes: ‘Ease of using search capability’ (81, +2) and ‘How well the search results met your needs’ (82, +1).With an impact of 0.8, the impact of Product Search is moderate. Product Search
Impact = 0.8Slide41
Product Search performance is consistent across DAACs, with scores ranging from 78 to 84.
This aspect of the customer experience has a moderate impact on CSI (0.8). Product Search: Four-year Comparison by DAACSlide42
Driver Detail: DeliverySlide43
Ninety-four percent of respondents downloaded or received data.
All facets of Delivery have been consistent over the last four years.Delivery (0.8) has a moderate impact on CSI. Delivery
Impact = 0.8Slide44
Delivery scores remain strong across DAACs.
ORNL DAAC led all DAACs with a score of 89 while GHRC reported the lowest score at 80.
Delivery: Four-year Comparison by DAACSlide45