RELIABILITY - PDF document

IMPR O VING THE INITIALISA TION AND RELIABILITY OF THE SELF ORGANISING OSCILLA T OR NETW ORK S. A. Salem, L. B. J ac k, and A. K. Nandi Signal Processing and Communications Group, Department of Electrical Engineering and Electronics, The Uni v ersity of Li v erpool Bro wnlo w Hill, Li v erpool, UK, L69 3GJ (sameh.salem, a.nandi) @li v .ac.uk ABSTRA CT The Self-Or ganising Oscillator Netw ork (SOON) pro vides a no v el w ay for data clustering [1, 2]. The SOON is a distance based algorithm, meaning that clusters are determined by a distance parameter , rather than by density distrib ution, or a pre-dened number of clusters. Repeated e xperiments ha v e highlighted the sensiti vity of this algorithm to the initial se- lection of phase v alues and prototypes. In repeated e xper - iments, the SOON as proposed by Frigui is sho wn to ha v e a number of shortf alls in terms of its performance o v er re- peated clustering runs. This paper proposes impro v ements to the initialisation stage of the algorithm by comparing the dif- ference between random initialisation of the phase curv e and initialisation using the ordering obtained from a hierarchical clustering approach. This leads to impro v ed con v er gence of the algorithm and more rob ust repeatability . When compared against random generation of phases and prototypes as pub- lished by Frigui originally , the changes in initialisation are sho wn to gi v e signicant impro v ements in the performance of the algorithm. 1. INTR ODUCTION Data o v erload is an increasing problem in man y dif ferent ar - eas of science and engineering. The creation of v ast datasets in science, go v ernment and industry presents challenging an- alytical and statistical problems. Unfortunately , the ability of interested parties to analyse these datasets in a reason- able amount of time and at a reasonable cost has not k ept pace. Clustering is one such area where the automated anal- ysis of lar ge datasets is important, while also becoming a problem. Numerous dif ferent unsupervised clustering tech- niques are commonly in use; the K ohonen Self Or ganising Map (SOM) has been perhaps one of the most popular un- supervised clustering algorithms, and is used in man y dif- ferent applications [3, 4], hierarchical clustering techniques, K-means clustering [5, 6], and k-medoids [7] are also widely used. Other alternati v e techniques, such as fuzzy clustering [7, 8] and man y v ariations of v ector quantisation [9] are den- sity based and their main dra wback is that the y cannot map the distrib ution of data in areas where the density of the data is lo w [2]. Distance based clustering techniques attempt to alle viate this problem by utilising the premise that clusters are determined by a distance parameter , rather than by den- sity distrib ution. The basis is simply that for an y gi v en clus- ter centre, all data points re garded as being members of the cluster will f all within a preset distance. This will allo w clus- ters in sparsely populated areas of data space to be formed without af fecting clustering in more dense areas of the data. This paper e xamines the applicability and the reliability of the SOON algorithm as a ne w clustering technique on a data set from a real-w orld communication data problem. 2. THEOR Y The Self-Or ganising Oscillator Netw ork (SOON) is a com- parati v ely ne w clustering algorithm [1] that has recei v ed rel- ati v ely little attention so f ar . SOON is a concept with roots in biology; the algorithm is modelled on biological principles: a good e xample of the synchronising oscillator phenomenon w ould be that of reies, which start by ashing at random initially , ho we v er the groups that are physically close to each other will synchronise their ring. Groups that are separated by distance will re as disparate groups, each synchronised within itself. The beha viour of self-or ganisation of components with an oscillatory nature gi v es rise to the name of the algorithm - the Self Or ganising Oscillator Netw ork (SOON). W ith the SOON method, each object in the data, O j , is represented as an “Inte grate and Fire” oscillator , characterized by a phase f j and state c j , where: c j = f ( f j ) = 1 b l n [ 1 + ( e b � 1 ) f j ] : (1) where 0  f j  1 and 0  c j  1 for j = 1 : : : : : : n ; f ( f j ) is smooth, monotonically increasing function with f ( 0 ) = 0 and f ( 1 ) = 1; b is a constant determining ho w f ( f j ) is conca v e do wn (usually b = 3). Whene v er an oscillator' s state reaches the threshold ( c j = 1), it “res”, with the follo wing consequenses:  The oscillator phase and state, f j and c j are set to zero; and  The phases of all the other oscillators change by an amount e ( f i ) , for i = 1 : : : : : : n ; i 6 = j . Changing the phase of other oscillators has the ef fect of ei- ther e xciting or inhibiting them. An oscillator is e xcited by ha ving its phase increased, while it is inhibited by decreasing its phase. The precise amount of the change is determined by the coupling function e ( f i ) , which in turn depends on the dis- similarity between the tw o oscillators (equi v alent objects). A typical coupling function w ould be as follo ws: e ( f i ) = 8 � � : C E [ 1 � ( d i j d o ) 2 ] d i j  d o C I [ ( d i j � 1 d o � 1 ) 2 � 1 ] d i j � d o (2) where d i j = d ( O i ; O j ) is short-hand for the measure of dissimilarity between tw o objects i and j, and d o is a threshold dissimilarity that determines the cut of f for what is deemed “similar”; d o can be vie wed as a resolution parame- ter , as it af fects the number of groups created. C E and C I are the maximum e xcitory and inhibitory couplings permitted. The ring of an oscillator tends to e xcite a fe w oscillators, whilst inhibiting man y others. Thus an oscillator recei v es much more inhibition than e xcitation; hence C E  C I . Once a set of oscillators is synchronized, the w ay that members of the set interact with other oscillators not in the set must be made uniform. F or lar ger datasets, a set of prototypes can be used as a smaller number of initial cluster centres; these are normally either selected at random from the data, to create a subset of the data, alternati v ely the y may be generated in order to co v er the e xtents of data space, so that an e v en distrib ution of the centres o v er data space is made. 2.1 Stability and Rob ustness The algorithm, as originally published, called for the ran- dom initialisation of the phase v alues; this can potentially cause problems with the initial selection of cluster centres, as the rst data point to re within a certain group may not be best suited to be the centre of the cluster . Additionally , poor choice of the C I and C E parameters may cause incorrect formation of clusters, as data points are pushed of f of the end of the phase curv e. By changing the initialisation stages of both the phase and prototype v alues, it should be possible to impro v e this signicantly . P art of the con v er gence time of the algorithm is based on the principle that o v er time, the ran- dom phases will tak e time to mo v e from their random start- ing positions to clusters of the same phase v alues, attracted by the distance from the cluster centre. By using a hierarchi- cal clustering technique to order the data points in terms of their proximity to each other by initialising the phase v ector f by the permutation v ector of the node labels of the lea v es of the dendrogram, it should be possible to achie v e the same result in a much f aster period of time, as the initial phase curv e will much more closely match that of the actual distri- b ution of the data in terms of distance. Thus, less time will be spent w aiting for the algorithm to stabilise, as the algorithm will start of f much closer to con v er gence than a random ini- tialisation w ould allo w . W e propose to use the results of the hierarchical clustering to select the prototype centres which are e xpected to impro v e the clustering performance , as the centres will map the centres of the data more closely than selecting a random subset of the data as proposed by [1]. 3. D A T ASETS In order to e xamine the reliability of the SOON algorithm, we use a communication data set, which represents a real- w orld problem; in this specic case constellation diagrams from dif ferent communication modulation schemes are a good benchmark test problems for unsupervised clustering algorithms, primarily due to the clean delineation between dif ferent clusters within constellation at high SNR. Ho we v er , by lo wering the SNR of the signal, it is possible to gain less well-separated data, which can pose more of a problem for the clustering algorithm, as the data becomes non-separable. 4. EXPERIMENTS The ef fect of the initialisation of the phase v alues of the data is in v estigated for both random generation of phase v alues -1.5 -1 -0.5 0 0.5 1 1.5 -1.5 -1 -0.5 0 0.5 1 1.5 (a) �1.5 �1 �0.5 0 0.5 1 1.5 �1.5 �1 �0.5 0 0.5 1 1.5 (b) �2 �1.5 �1 �0.5 0 0.5 1 1.5 2 �2 �1.5 �1 �0.5 0 0.5 1 1.5 2 (c) �2.5 �2 �1.5 �1 �0.5 0 0.5 1 1.5 2 2.5 �2 �1.5 �1 �0.5 0 0.5 1 1.5 2 (d) Figure 1: The clustering performance for SNR, (a) 25 dB, (b) 20 dB, (c) 15 dB, (d) 10 dB. and using the hierarchical clustering ordering to determine the initial phase v alues. A second set of e xperiments e xamine the ef fect of a randomly selected subset of the dataset for the initial prototypes, as opposed to using a subset of the clusters generated by the hierarchical clustering approach. 4.1 Results & Discussions A series of e xperiments were carried out to e xamine the clus- tering performance on a QPSK signal for dif ferent SNR v al- ues using proper v alue of d o . Figure 1 depicts the clustering performance for dif ferent SNR v alues. The ef fect of random generation of prototypes is tested and e v aluated 100 times using prototypes ( set to one half of the total number of data points a v ailable for clustering). One reference run (best run) w as selected from the 100 runs using a reliable v alidity inde x. In this paper I inde x [10] is used. Figure 2 sho ws the analysis for lo w SNR v alues (15 and 10 dB). The tw o plots in Figure 2 contain three subplots, these subplots sho w (top) the dif fer - ence in magnitude between the size of reference run clusters ( M ) and the comparison clusters ( N ), sho wing the relati v e re- producibility of the results; (middle) the number of runs ( R ) that achie v e the common cluster membership for each clus- ter in the reference run, (bottom) the “gain” per run ( N = M ) and the weighted gain per run ( N = M )  R 2 . As can be seen, e xamining the plots for 10 dB and 15 dB sho w that there is a mark ed deterioration in the reproducibility of the clustering result, as the fraction of cluster members maintained (i.e. the gain - N = M ) across man y runs drops signicantly for the 10 dB SNR data. F or impro v ement of the clustering performance of the SOON algorithm, we propose to use the hierarchical clus- tering to set up and order the phases on the conca v e function curv e that e xpress the SOON model which leads to better performance as well as speeding up the algorithm. Figure 3 sho ws the ef fect of hierarchical clustering for phase genera- 1 2 3 4 0 50 100 150 Reference (M) Common (N) 1 2 3 4 0 50 100 R 1 2 3 4 0 0.5 1 1.5 Cluster index (reference run) N/M (N/M)*R 2 (a) 1 2 3 4 0 50 100 150 Reference (M) Common (N) 1 2 3 4 0 50 100 R 1 2 3 4 0 0.5 1 1.5 Cluster index (reference run) N/M (N/M)*R 2 (b) Figure 2: Analysis of the ef fect of noise on clustering be- ha viour , o v er 100 runs for SNR, (a) 15 dB, (b) 10 dB. tion on the clustering performance at 10 dB SNR. As sho wn in Figure 3, the clustering performance using hierarchical clustering for phase generation gi v es a signicant impro v e- ment, which is sho wn by the gain per run, as well as the impro v ement in the weighted gain. F or further impro v ement of the repeatability of the re- peatability of the clustering performance, the results from the hierarchical clustering can also be used for the generation of prototypes, in order to achie v e better distrib ution of the pro- totypes among the dataset. This reduces the ef fect of noise within the data, while also impro ving the initial cluster posi- tions that the algorithm accepts, by choosing centres that are implicitly well distrib uted throughout the data. The ef fect of this is v ery clear when using hierarchical clustering for phase generation. Figure 4 depicts the clustering performance of hierarchical prototype generation using random and hierar - chical phases. As can be seen, using the hierarchically de- ri v ed prototypes signicantly impro v es the repeatability of the algorithm, while also unifying the gain across all clus- 1 2 3 4 5 0 50 100 150 Reference (M) Common (N) 1 2 3 4 5 0 50 100 R 1 2 3 4 5 0 0.5 1 1.5 Cluster index (reference run) N/M (N/M)*R 2 Figure 3: Analysis of the ef fect of hierarchical phase gener - ation on repeatability and performance o v er 100 runs. ters. Additionally , the performance remains substantially the same as the number of prototypes increases or or decreases. In order to rely on the conclusion obtained abo v e, clus- tering v alidity methods [10] can be used for e v aluating and assessing the results of the proposed impro v ements of the SOON algorithm. From T able 1, v alidation indices [10] v al- ues, I, CH , and DBnc indicate that the hierarchical phase with hierarchical prototype selection is better than the hierarchical phase with random prototype selection, and the latter is bet- ter than the random phase with random prototype selection. Ho we v er , Dunn inde x [10] v alues do not agree with the abo v e conclusion, where it is v ery sensiti v e to the presence of noise in datasets [11]. Additionally , the con v er gence time (on Xeon 2.8 GHZ CPU with 512 MB ram using C code) is e xperimentally tested for 100 runs. As illustrated in T able 1, the con v er gence time of hierarchical phase with hierarchical prototype gener - ations is the lo west one compared against random prototype generations with random phases, or hierarchical phases. V alidation indices Random phase generation Hierar chical phase generation Hierar chical phase & pr otototypes I [10] 1.7 1.8 2.8 CH [10] 467 471 690 DBnc [10] 0.98 0.89 0.62 Dunn [10] 0.0071 0.0070 0.0031 Con v ergence time in sec 75 72 36 T able 1: Results of dif ferent v alidity indices & con v er gence times 1 2 3 4 0 100 200 Reference (M) Common (N) 1 2 3 4 0 50 100 R 1 2 3 4 0 0.5 1 1.5 Cluster index (reference run) N/M (N/M)*R 2 (a) 1 2 3 4 0 50 100 150 Reference (M) Common (N) 1 2 3 4 0 50 100 R 1 2 3 4 0 0.5 1 1.5 Cluster index (reference run) N/M (N/M)*R 2 (b) Figure 4: Analysis of the ef fect of structured prototype selec- tion on repeatability for , (a) Random phase with hierarchical selection of prototypes, (b) Hierarchical phase with hierar - chical selection of prototypes. One of the main adv antages of SOON algorithm is its rob ustness in terms of the input parameters, where all data points re garded as being members of the cluster will f all within a preset distance which actually describes the de gree of the similarity between objects compared against the hi- erarchical clustering algorithm or an y partitional clustering algorithms that depend on the number of clusters K as one of the input parameters. Therefore, it is v ery dif cult to get a f air comparison between radius based clustering algorithms and partitional clustering algorithms. 5. CONCLUSIONS T w o impro v ements to the initialisation of the SOON algo- rithm ha v e been proposed. As can be seen, the proposed use of hierarchically deri v ed initial phase and prototype v alues causes signicant impro v ements in the repeatability of the clustering performance of the algorithm. Additionally , the con v er gence of the algorithm is also signicantly increased as the phase curv e is already ordered in terms of similarity , meaning that fe wer iterations are required for the algorithm to reach con v er gence. REFERENCES [1] H. Frigui and M. B. H. Rhouma, “Self-Or ganization of Pulse-Coupled Oscillators with Application to Cluster - ing, ” IEEE T r ans . P attern Analysis and Mac hine Intel- lig ence , v ol. 23, pp. 180-195, Feb . 2001. [2] L. B. Jack and A. K. Nandi, “Microarray Data using The Self Or ganising Oscillator Netw ork, ” in Pr oc . EU- SIPCO 2004 , V ienna, Austria, September 6-10. 2004, pp. 2183-2186. [3] T . K ohonen, Self-Or ganising Maps , Springer -V erlag, 1997. [4] S. Zhang, R. Ganesan, and G. D. Xistris, “Self- or ganizing neural ne w orks for automated machinery monitoring systems, ” Mec hanical systems and Signal Pr ocessing , v ol. 10, pp. 517-532, 1996. [5] J. MacQueen. “Some methods for classication and analysis of multi v ariate observ ations, ” in Pr oc . the F ifth Berk ele y Symposium on Mathematical Statistics and Pr obability , Uni v ersity of California Press, 1967, v ol. 1, pp. 281-297. [6] C. W . Chen, J. B. Luo, and K. J. P ark er , “Im- age Se gmentation V ia Adapti v e K-Mean Cluster - ing And Kno wledge-Based Morphological Operations W ith Biomedical Applications, ” IP , v ol. 7, pp. 1673- 1683, Dec. 1998. [7] L. Kaufman and P . Rousseeuw , F inding Gr oups in data: an intr oduction to cluster analysis , John W ile y and Sons, Ne w Y ork, 1990. [8] N. J. Pizzi, M. Ale xander , R. V i v anco, and R. Somarjaj. “F ast non iterati v e re gistration of magnetic resonance images, ” in Pr oc . SPIE: Medical Ima ging 2001 , v ol. 4322, pp. 1599-1608. [9] W . Xu, A. K. Nandi, and J. Zhang, “No v el fuzzy re- inforcement learning v ector quantization algorithm and its application in image compression, ” IEE Pr oceedings on Ima g e and Signal Pr ocessing , v ol. 150, pp. 292-298, Oct. 2003. [10] U. Maulik and S. Bandyopadhyay , “Performance Ev al- uation of some Clustering Algorithms and V alidity In- dices, ” IEEE T r ans. P attern Analysis and Mac hine In- tellig ence , v ol. 24, pp. 1650-1654, Dec. 2002. [11] M. Haliki, Y . Batistakis, and M. V azir giannis, “Cluster V alidity Methods: P art II, ” SIGMOD Recor d , v ol. 31, Issue 3, Sep. 2002.