Beruflich Dokumente
Kultur Dokumente
Improving
the Quality of Customer Satisfaction Measurements of MUSA Method using Clustering Data Mining Techniques
Nikos Tsotsolas, Yiannis Siskos and Gerasimos Marketos University of Piraeus ntsotsol@unipi.gr
* Research partially supported by the HERACLETOS EPEAEK II Programme of the Greek Ministry of National Education and Religious Affairs, co-funded by the European Union.
Structure:
MUSA Method Research objectives Data mining approach The experiment Application of results Conclusions
2
MUSA Method
The main objective of the method is the aggregation of individual judgements into a collective value function assuming that clients global satisfaction depends on a set of n criteria representing service characteristic dimensions.
The MUSA method assesses global and partial satisfaction functions Y* and X*I respectively, given customers judgements Y and Xi.
n * * Y b X i i i 1 n b 1 i i 1
where the value functions Y* and X*I are normalised in the interval [0,100], and bi is the weight of the i-th criterion
MUSA Method
MUSA uses a preference disaggregation model. In the traditional aggregation approach, the criteria aggregation model is known a priori, while the global preference is unknown. On the contrary, the philosophy of disaggregation involves the inference of preference models from given global preferences.
Aggregation Model
aggregation
CRITERIA
GLOBAL PREFERENCE
disaggregation
Aggregation Model?
MUSA Method
Customer's global satisfaction
Y* y* y*m
...
y*2 y*1 Y y1 y2 ... ym ... y
...
...
...
x1*m
...
...
x1*2 x1*1 X1 x11 x12 ... x1k ... x11 xi*1 Xi xi1 xi2 ... xik ... xii
...
xi*2
xn*m
...
...
xnn
Xn
Research Objectives
Samples of customers participating in satisfaction surveys often appear to have a low degree of homogeneity The collective nature of MUSA method may result in poor quality measurements when it is used to analyse data of really farraginous samples
A methodological framework has been developed for the segmentation of an initial sample into more homogeneous subsets using clustering data mining techniques.
6
Research Objectives
Lets say that we have two equal divided, farraginous groups of customers in our sample. The first is consisted of demanding customers and the second one has non-demanding customers. MUSA will produce a result describing neutral customers.
Demanding
Non-demanding
Neutral
Research Objectives
In the case of different importance of the criteria (criteria wights) given by farraginous groups of customers will lead us to similar problems.
MUSA results
40%
35.0%
35.0%
40%
35.0%
35.0%
40%
30%
30%
30%
25.0%
25.0%
25.0%
25.0%
20%
15.0%
15.0%
20%
15.0%
15.0%
20%
10%
10%
10%
0%
0%
0%
Research Objectives
MUSA gives as internal measures evaluating the quality of its results. The reliability evaluation of the results is mainly related to the following quantitative measures:
the fitting level to the customer satisfaction data (Average Fitting Index AFI and Overall Prediction Level -OPL)
the stability of the near-optimality analysis results (Average Stability Index ASI).
Research Objectives
Predicted global satisfaction level ~ y1 y1 N11 C11 N21 C21 R21 R11 N12 C12 N22 C22 R22 ... ~ y2 R12 ... N1j C1j N2j C2j R2j ... ~ yj R1j ... ~ y N1 R1 C1 N2 R2 C2
The Overall Prediction Level (OPL) is based on the sum of the main diagonal cells of the prediction table, and it represents the percentage of correctly classified customers: : the number of customers that have declared to belong to global satisfaction level m1, while the model classifies them to level m2
N m1m2
y2
...
...
yi
Ni1 Ci1
Ri1
Ni2 Ci2
Ri2
...
Nij Cij
Rij
...
Ni Ci
Ri
actual global satisfaction level m1, that the model classifies to level m2 : the percentage of customers of estimated global satisfaction level m1, that have declared to belong to level m2
Cm1m2
...
...
Na1 Ca1
Ra1
Na2 Ca2
Ra2
...
Naj Caj
Raj
...
Na Ra Ca
OPL N m1m1
m1 1
m1 1 m2 1
m1m2
10
Data Mining
Data Preprocessing:
Transformations
.. .. ..
Clusters (2,, n)
Preprocessed Data
11
Transformations
Two kinds of transformation were developed in order to cover both cases: segments with different demanding level or with different given satisfaction criteria importance.
4 3 2 1 0 Total Cr 1 Cr 2 Cr 3 Cr 4
12
Transformations
For different Demand, DCr(i) nominal: If Cr(i)<(T-thr) Then DCr(i)=INT(Cr(i)-(T-thr)) If Cr(i)>(T+thr) Then DCr(i)=INT(Cr(i)-(T+thr)) If(T-thr)Cr(i)(T+thr) Then DCr(i)=0 where T is the declared total satisfaction of the customer, Cr(i) is the satisfaction regarding his/her satisfaction on i criterion and thr is a threshold. For different Criteria Weights, W1Cr(i), W2Cr(i), numeric: W1Cr(i)=ABS(Cr(i)-T) and W2Cr(i)= W1Cr(i)*(crc(a)-(ABS([(a-1)/2-T]/(a-1)/2)) where T is the declared total satisfaction of the customer, Cr(i) is the satisfaction regarding satisfaction on i criterion, crc(a) is a correction parameter and a is the number of global satisfaction levels
13
EM algorithm
A hard membership is adopted in the K-means algorithm, (i.e., a data pattern is assigned to one cluster only). This is not the case with the EM algorithm, where a soft membership is adopted, (i.e., the membership of each data pattern can be distributed over multiple clusters)
14
EM algorithm
Similarly to K-means, first select the cluster parameters (A, A, P(A)) or guess the classes of the instances, then iterate
Each cluster A is defined by a mean (A) and a standard deviation (A) Samples are taken from each cluster A with a specified probability of sampling P(A)
Adjustment needed: we know cluster probabilities, not actual clusters for each instance. So, we use these probabilities as weights For cluster A:
Stop when the difference between two successive iteration becomes negligible (i.e. there is no improvement of clustering quality).
15
EM algorithm
16
The experiment
For the development of the transformation procedure and for the evaluation of our research results we designed and we implemented an experiment.
Steps:
1. 2. 3. 4. Generation of synthetic data DataSet Generator Evaluate clusters generation WEKA DM tool Evaluate MUSA results on new segments Select the most appropriate transformations
17
18
2nd Data Set Produce two segments regarding different criteria weights:
Demand Sets (500) (500) Satisfaction Levels A (Global) a(i) (per criterion)
Criterion 1 Neutral
15% 35% 5 5
Criterion 2 Neutral
15% 35%
Criterion 3 Neutral
35% 15%
Criterion 4 Neutral
35% 15% 19
20
0
164 446
1
336 54
I II
0
323 175
1
177 325
I II
Customers
500
500
AFI
97.17%
96.29%
ASI
96.59%
95.68%
OPL
95.00%
95.40%
1000
390
91.29%
95.21%
88.77%
96.13%
56.50%
86.15%
Cluster 0 --> II
610
94.75%
94.63%
92.95%
Criteria Weights
Samples - Data Sets
Generator Data Set I
Generator Data Set II Generator Data Set I + II Cluster 1 --> I Cluster 0 --> II
Cr 1
Cr 2
Cr 3
25.04%
26.28% 25.34% 25.24% 25.54%
25.92%
25.43% 27.69% 26.23% 25.01%
24.61%
24.16% 23.61% 24.89% 24.86%
24.44%
24.13% 23.36% 23.64% 24.59%
-55.25%
41.82% -27.96% -52.34% 40.26%
-60.43%
59.962 -19.57% -57.61% 54.48%
-57.28%
65.41% -27.40% -57.92% 62.63%
-16.13%
30.55% -5.16% -20.09% 35.28%
-61.86%
12.99% -54.35% -65.90% 19.14%
22
Customers
500
500
AFI
96.45%
96.51%
ASI
97.64%
96.63%
OPL
89.60%
91.00%
1000
498
92.52%
94.35%
91.97%
94.85%
60.30%
80.92%
Cluster 1 --> II
502
95.32%
94.65%
93.43%
Criteria Weights
Cr 1
Cr 2
Cr 3
16.45%
34.31% 25.31% 19.78% 33.91%
16.34%
34.23% 25.08% 17.65% 34.14%
33.67%
16.50% 24.35% 35.35% 16.11%
33.53%
14.96% 25.27% 27.22% 15.83%
28.92%
29.80% 27.06% 14.56% -6.48%
26.97%
-36.15% -7.87% 24.30% -17.50%
51.05%
32.27% 37.47% -20.66% -20.52%
33.58%
-49.20% 23.23% 22.10% 15.42%
36.29%
46.51% 31.45% 3.70%
25.44%
23
Application of results
The clustering procedure was applied on two real world surveys in order to be further evaluated. The measure of success would be the improvement of MUSAs internal quality measures through the proper segmentation of the initial sample. Survey 1: Policemen Satisfaction in Greece (sample: 1508, criteria: 8) Survey 2: Tourists Satisfaction in Skopelos Island (sample: 599, criteria: 5)
24
Survey 1
Evaluation of segments production using DCr(i) transformation:
MUSA's Internal Quality Measures Samples - Data Sets Initial Sample Customers 1508 AFI 93.02% ASI 78.02% OPL 56.63%
Cluster 0
Cluster 1 Cluster 0 Cluster 1
537
971 526 835
92.76%
95.16% 94.98% 96.07%
82.65%
74.46% 66.42% 83.14%
77.09%
72.81% 82.32% 88.38%
Cluster 2
Cluster 0 Cluster 1 Cluster 2
147
720 105 494
85.03%
96.87% 82.10% 96.03%
81.96%
80.19% 79.15% 85.06%
52.38%
90.83% 40.95% 90.28%
Cluster 3
189
93.51%
67.67%
82.01%
25
Survey 1
Evaluation of segments production using W1Cr(i), W2Cr(i) transformations:
MUSA's Internal Quality Measures Samples - Data Sets Customers AFI ASI OPL
Initial Sample
Cluster 0 Cluster 1
1508
579 929
93.02%
90.39% 95.60%
78.02%
79.65% 75.63%
56.63%
34.72% 77.40%
Cluster 0
Cluster 1 Cluster 2 Cluster 0
533
440 535 374
94.59%
90.29% 96.30% 93.99%
84.22%
80.52% 76.01% 82.40%
54.60%
47.50% 90.84% 42.51%
Cluster 1
Cluster 2 Cluster 3
188
547 399
85.98%
94.96% 97.04%
83.47%
74.41% 78.24%
27.13%
73.13% 91.73%
26
Survey 1
Evaluation of segments production using DCr(i) transformation:
Demanding Indices Clusters Initial Sample Cluster 0 Cluster 1 Global
33.46% 12.32% 35.00%
Cr 3
36.00% 29.34% 65.00%
Cr 5
-2.80% 6.42% -22.76%
Cr 8
70.05% 74.14% 36.00%
Criteria Weights
Clusters
Initial Sample Cluster 0 Cluster 1
Cr 1
9.71% 9.16% 10.24%
Cr 2
10.63% 9.56% 11.78%
Cr 3
12.50% 11.32% 22.99%
Cr 4
9.07% 8.72% 9.05%
Cr 5
8.75% 8.55% 10.36%
Cr 6
10.14% 9.25% 10.59%
Cr 7
12.50% 12.50% 12.50%
Cr 8
26.71% 30.94% 12.50%
Labelling: More women belonging to the Greek Police forces seem to belong into Cluster 0. More lower-level officers seem to belong into Cluster 1.
27
Survey 2
Evaluation of segments production using DCr(i) transformation:
` Samples - Data Sets Initial Sample Customers 599 MUSA's Internal Quality Measures AFI 93.18% ASI 62.64% OPL 53.59%
Cluster 0
Cluster 1 Cluster 0 Cluster 1
214
385 186 342
89.48%
96.25% 89.04% 96.69%
60.51%
90.69% 59.65% 74.61%
57.48%
77.66% 56.45% 84.21%
Cluster 2
71
92.31%
93.86%
39.44%
28
Survey 2
Evaluation of segments production using W1Cr(i), W2Cr(i) transformations:
MUSA's Internal Quality Measures Samples - Data Sets Initial Sample Customers 599 AFI 93.18% ASI 62.64% OPL 53.59%
Cluster 0
Cluster 1 Cluster 0 Cluster 1
330
269 149 179
96.65%
89.84% 88.54% 92.60%
46.00%
68.95% 78.03% 54.06%
75.76%
29.74% 34.90% 34.64%
Cluster 2
271
97.50%
46.00%
76.01%
29
Survey 2
Evaluation of segments production using DCr(i) transformation:
Demanding Indices Clusters Initial Sample Cluster 0 Global -60.37% -64.50% Cr 1 -49.56% -54.00% Cr 2 -55.99% -54.00% Cr 3 -38.34% -54.00% Cr 4 -12.14% -54.00% Cr 5 -75.29% -54.00%
Cluster 1
-55.10%
-40.09%
-56.97%
-29.22%
-10.32%
-76.54%
Criteria Weights
Cr 1 17.84%
Cr 2 20.45%
Cr 3 14.60%
Cr 4 11.24%
Cr 5 36.87%
20.00%
15.36%
20.00%
21.38%
20.00%
13.00%
20.00%
10.26%
20.00%
40.01%
Labelling: Tourists staying in hotels turn to belong into Cluster 1 while on the contrary the ones chose to stay in rooms to let seem to belong in cluster 0.
30
Conclusions (1)
Conclusions
Data Mining Clustering procedure led to more homogeneous segments of customers both in synthetic datasets and in real world surveys results. DCr(i) transformation seems to work better than W1Cr(i), W2Cr(i) transformations.
The labelling of the produced clusters segments is not always obvious Maybe more attention should be paid during the designing of the survey to include more demographical information.
31
Conclusions (2)
Future research
Some improvements regarding the data mining procedure may include:
Further experiments using the dataset generator evaluating the results should be undertaken. Real world surveys should be used as well. Other MUSA internal quality measures, recently proposed, should be also considered. The transformations regarding the different criteria weights should be improved, if it is possible.
Other or new similarity metrics should be studied. The labelling procedure should be thoroughly examined.
32
Thank you
33