Philippe Fournier-Viger
Description: Philippe Fournier-Viger http:www.philippe-Fournier-viger.com Finding Correlated and Statistically Significant Patterns 1 Source code and datasets available in the SPMF library Todays topic Finding patterns in database that indicate a
Related Topics
Download Presentation
"Philippe Fournier-Viger" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Philippe Fournier-Viger http://www.philippe-Fournier-viger.com Finding Correlated and
Statistically Significant Patterns 1 Source code and datasets available in the SPMF library<br>
slide2. Today’s topic Finding patterns in database that indicate a strong correlation or are statistically significant.
Frequent itemset mining (FIM) 2<br>
slide3. Frequent itemset mining Let there be a positive number minsup, set by the user.
Frequent itemset mining (FIM) consists of enumerating all frequent itemsets, that is sets of values having a support greater or equal to minsup in a database D. 3<br>
slide4. Example 4 For minsup = 2, the frequent itemsets are:
{lemon}, {pasta}, {orange}, {cake}, {lemon, pasta}, {lemon, orange}, {pasta, orange}, {pasta, cake}, {orange, cake}, {lemon, pasta, orange}
These itemsets appear many times but should we “believe” them?<br>
slide5. No. Many itemsets are spurious! A frequent itemset may contain items that are weakly correlated (appear together more or less by chance).
e.g {pasta, cake} appears in 50% of the transactions.
But it does not mean much because pasta appears in all transactions! 5<br>
slide6. How to solve this problem? Solution 1: Evaluate the correlation between items in an itemset using functions such as:
Bond
All-confidence
Solution 2: Apply statistical testing
Solution 3: Find other types of patterns, e.g., association rules. 6<br>
slide7. The BOND mEASURE 7 Omiecinski, E.R.: Alternative interest measures for mining associations in databases. IEEE Transactions on Knowledge and Data Engineering 15(1), pp. 57-69 (2003)<br>
slide8. (Conjunctive) support 8<br>
slide9. Disjunctive support 9<br>
slide10. Bond 10<br>
slide11. 11<br>
slide12. 12<br>
slide13. 13 (bond is anti-monotonic)<br>
slide14. Proof sketch 14<br>
slide15. Finding correlated frequent itemsets (using the bond) 15<br>
slide16. Example 16 For minsup = 2 and minbond = 0.75, the correlated frequent itemsets are:
{lemon}, {pasta}, {orange}, {cake} bond = 1 {lemon, pasta}, bond = 0.75 {lemon, orange}, bond = 0.5 {pasta, orange}, bond = 0.75 {pasta, cake}, bond = 0.5 {orange, cake}, bond = 0.66 {lemon, pasta, orange} bond = 0.5<br>
slide17. Finding rare correlated frequent itemsets using the bond 17<br>
slide18. Example 18 For maxsup = 3 and minbond = 0.6, the rare correlated itemsets are:
{bread} support = 1 bond = 1 {cake} support = 2 bond = 1 {orange, cake} support = 2 bond = 0.66<br>
slide19. The CORI algorithm 19 CORI: Bouasker, S., Ben Yahia, S.: Key correlation mining by simultaneous monotone and anti-monotone constraints checking. In: Proc. 30th Symp. on Applied Computing, pp. 851-856, 2015.<br>
slide20. Two structures Each itemset X has two structures
TID-List (Transaction-ID list): list of transactions containing X.
DTID-List (Disjunctive Transaction-ID list): list of transactions containing at least one item from X.
Example 20<br>
slide21. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4}
TIDLIST({pasta,lemon}) = {T1, T2, T4}
TIDLIST({bread,orange}) = {T1}<br>
slide22. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4}
TIDLIST({pasta,lemon}) = {T1, T2, T4}
TIDLIST({bread,orange}) = {T1}
DTIDLIST({pasta}) = {T1, T2, T3, T4}
DTIDLIST({pasta,lemon}) = {T1, T2, T3,T4}
DTIDLIST({bread,orange}) = {T1,T3, T4}<br>
slide23. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4} sup(…) = 4
TIDLIST({pasta,lemon}) = {T1, T2, T4} sup(…) = 3
TIDLIST({bread,orange}) = {T1} sup(…) = 1
DTIDLIST({pasta}) = {T1, T2, T3, T4}
DTIDLIST({pasta,lemon}) = {T1, T2, T3,T4}
DTIDLIST({bread,orange}) = {T1,T3, T4}<br>
slide24. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4} sup(…) = 4
TIDLIST({pasta,lemon}) = {T1, T2, T4} sup(…) = 3
TIDLIST({bread,orange}) = {T1} sup(…) = 1
DTIDLIST({pasta}) = {T1, T2, T3, T4} dsup(…) = 4
DTIDLIST({pasta,lemon}) = {T1, T2, T3,T4} dsup(…) = 4
DTIDLIST({bread,orange}) = {T1,T3, T4} dsup(…) = 3 If we have these lists, we can calculate the bond!<br>
slide25. The CORI algorithm 25<br>
slide26. The All-CONFIDENCE mEASURE 26 Omiecinski, E.R.: Alternative interest measures for mining associations in databases. IEEE Transactions on Knowledge and Data Engineering 15(1), pp. 57-69 (2003)<br>
slide27. All-confidence 27<br>
slide28. All-confidence 28<br>
slide29. 29<br>
slide30. 30 (all-confidence is anti-monotonic)<br>
slide31. Algorithm It is easy to modify a frequent itemset mining algorithm such as Apriori and Eclat to calculate the all-confidence.
We just need to pre-calculate the support of each single item.
This is all we need to calculate the all-confidence. 31<br>
slide32. Two interesting observations 32<br>
slide33. Other correlation measures 33<br>
slide34. Other correlation measures 34<br>
slide35. Other correlation measures 35 Wu et al.: Re-examination of interestingness measures in pattern mining: a unified framework. DMKD 21:371-397 (2010) Note: the definitions are given the case of two items a and b.<br>
slide36. Statistically significant itemsets 36 Webb, G.I. and Vreeken, J., 2013. Efficient discovery of the most interesting associations. ACM Transactions on Knowledge Discovery from Data (TKDD), 8(3), pp.1-31.<br>
slide37. Introduction The various correlation measures help to ensure that items are correlated.
However, they do not ensure statistical significance.
We can raise the bar higher to select itemsets and use statistical tests.
This can be useful to analyze medical data and draw conclusions that are more likely to be valid… 37<br>
slide38. Non-redundant and productive itemsets Goal: Reduce the number of patterns found by identifying only those that are statistically significant.
Statistical significance is measured with Fisher Exact test. Productive itemsets Non-redundant Itemsets Webb, G. I., Vreeken, J.: Efficient discovery of the most interesting associations. ACM Transactions on Knowledge Discovery from Data. 8(3),15 (2014)<br>
slide39. Non-redundant itemsets An itemset is non-redundant (i.e., is a generator) if it has no proper subset having the same support (occurrence frequency)
Example:
sup({pregnant,heart disease,woman}) = sup({pregnant,heart disease})
Thus, {pregnant,heart disease,woman} is redundant.<br>
slide40. Productive itemsets Idea: Check if we can predict the support of an itemset by assuming that some of its partitions are independent.
An itemset is productive if all its partitions into two subsets are positively correlated with each other.
{a,b,c} two partitions: {a,b} and {c} another possibility: {a,c} and {b}
another possibility: {b,c} and {a}
Example:
{alchool, liver_cancer} is productive because alchool is positively correlated with liver_cancer<br>
slide41. Productive itemsets Example 2: {alchool, liver_cancer, black_hair}
{alchool, liver_cancer} is not positively correlated with {black_hair}.
Thus, {alchool, liver_cancer, black_hair} is not productive.<br>
slide42. Fisher Exact Test to evaluate statistical significance 0.001380 <0.05 {studying ,Men} is positively correlated? YES! =1 =11 =9 =3 =12 =12 =10 =14 Online calculator: www.statology.org/fishers-exact-test-calculator/<br>
slide43. How to find non-redundant productive itemsets? First algorithm: Opus-Miner (Webb, 2014)
It performs a depth-first search.
The user must set a parameter k.
The algorithm returns the top k non-redundant productive itemsets having the highest lift or leverage.
Other algorithms:
IDPI+: Allows to interactively search for productive itemsets (Fournier-Viger, et a., 2018, “Interactive Discovery of Statistically Significant Itemsets”) Query = X= {male, tobacco, Alzheimer} is productive?
…<br>
slide44. Conclusion We have discussed:
correlated patterns
bond
all-confidence
other correlation measures
the CORI algorithm
statistically significant patterns 44 Open source Java data mining software, 240 algorithms
http://www.phillippe-fournier-viger.com/spmf/<br>
Statistically Significant Patterns 1 Source code and datasets available in the SPMF library<br>
slide2. Today’s topic Finding patterns in database that indicate a strong correlation or are statistically significant.
Frequent itemset mining (FIM) 2<br>
slide3. Frequent itemset mining Let there be a positive number minsup, set by the user.
Frequent itemset mining (FIM) consists of enumerating all frequent itemsets, that is sets of values having a support greater or equal to minsup in a database D. 3<br>
slide4. Example 4 For minsup = 2, the frequent itemsets are:
{lemon}, {pasta}, {orange}, {cake}, {lemon, pasta}, {lemon, orange}, {pasta, orange}, {pasta, cake}, {orange, cake}, {lemon, pasta, orange}
These itemsets appear many times but should we “believe” them?<br>
slide5. No. Many itemsets are spurious! A frequent itemset may contain items that are weakly correlated (appear together more or less by chance).
e.g {pasta, cake} appears in 50% of the transactions.
But it does not mean much because pasta appears in all transactions! 5<br>
slide6. How to solve this problem? Solution 1: Evaluate the correlation between items in an itemset using functions such as:
Bond
All-confidence
Solution 2: Apply statistical testing
Solution 3: Find other types of patterns, e.g., association rules. 6<br>
slide7. The BOND mEASURE 7 Omiecinski, E.R.: Alternative interest measures for mining associations in databases. IEEE Transactions on Knowledge and Data Engineering 15(1), pp. 57-69 (2003)<br>
slide8. (Conjunctive) support 8<br>
slide9. Disjunctive support 9<br>
slide10. Bond 10<br>
slide11. 11<br>
slide12. 12<br>
slide13. 13 (bond is anti-monotonic)<br>
slide14. Proof sketch 14<br>
slide15. Finding correlated frequent itemsets (using the bond) 15<br>
slide16. Example 16 For minsup = 2 and minbond = 0.75, the correlated frequent itemsets are:
{lemon}, {pasta}, {orange}, {cake} bond = 1 {lemon, pasta}, bond = 0.75 {lemon, orange}, bond = 0.5 {pasta, orange}, bond = 0.75 {pasta, cake}, bond = 0.5 {orange, cake}, bond = 0.66 {lemon, pasta, orange} bond = 0.5<br>
slide17. Finding rare correlated frequent itemsets using the bond 17<br>
slide18. Example 18 For maxsup = 3 and minbond = 0.6, the rare correlated itemsets are:
{bread} support = 1 bond = 1 {cake} support = 2 bond = 1 {orange, cake} support = 2 bond = 0.66<br>
slide19. The CORI algorithm 19 CORI: Bouasker, S., Ben Yahia, S.: Key correlation mining by simultaneous monotone and anti-monotone constraints checking. In: Proc. 30th Symp. on Applied Computing, pp. 851-856, 2015.<br>
slide20. Two structures Each itemset X has two structures
TID-List (Transaction-ID list): list of transactions containing X.
DTID-List (Disjunctive Transaction-ID list): list of transactions containing at least one item from X.
Example 20<br>
slide21. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4}
TIDLIST({pasta,lemon}) = {T1, T2, T4}
TIDLIST({bread,orange}) = {T1}<br>
slide22. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4}
TIDLIST({pasta,lemon}) = {T1, T2, T4}
TIDLIST({bread,orange}) = {T1}
DTIDLIST({pasta}) = {T1, T2, T3, T4}
DTIDLIST({pasta,lemon}) = {T1, T2, T3,T4}
DTIDLIST({bread,orange}) = {T1,T3, T4}<br>
slide23. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4} sup(…) = 4
TIDLIST({pasta,lemon}) = {T1, T2, T4} sup(…) = 3
TIDLIST({bread,orange}) = {T1} sup(…) = 1
DTIDLIST({pasta}) = {T1, T2, T3, T4}
DTIDLIST({pasta,lemon}) = {T1, T2, T3,T4}
DTIDLIST({bread,orange}) = {T1,T3, T4}<br>
slide24. TID-List and DTID-List TIDLIST({pasta}) = {T1, T2, T3, T4} sup(…) = 4
TIDLIST({pasta,lemon}) = {T1, T2, T4} sup(…) = 3
TIDLIST({bread,orange}) = {T1} sup(…) = 1
DTIDLIST({pasta}) = {T1, T2, T3, T4} dsup(…) = 4
DTIDLIST({pasta,lemon}) = {T1, T2, T3,T4} dsup(…) = 4
DTIDLIST({bread,orange}) = {T1,T3, T4} dsup(…) = 3 If we have these lists, we can calculate the bond!<br>
slide25. The CORI algorithm 25<br>
slide26. The All-CONFIDENCE mEASURE 26 Omiecinski, E.R.: Alternative interest measures for mining associations in databases. IEEE Transactions on Knowledge and Data Engineering 15(1), pp. 57-69 (2003)<br>
slide27. All-confidence 27<br>
slide28. All-confidence 28<br>
slide29. 29<br>
slide30. 30 (all-confidence is anti-monotonic)<br>
slide31. Algorithm It is easy to modify a frequent itemset mining algorithm such as Apriori and Eclat to calculate the all-confidence.
We just need to pre-calculate the support of each single item.
This is all we need to calculate the all-confidence. 31<br>
slide32. Two interesting observations 32<br>
slide33. Other correlation measures 33<br>
slide34. Other correlation measures 34<br>
slide35. Other correlation measures 35 Wu et al.: Re-examination of interestingness measures in pattern mining: a unified framework. DMKD 21:371-397 (2010) Note: the definitions are given the case of two items a and b.<br>
slide36. Statistically significant itemsets 36 Webb, G.I. and Vreeken, J., 2013. Efficient discovery of the most interesting associations. ACM Transactions on Knowledge Discovery from Data (TKDD), 8(3), pp.1-31.<br>
slide37. Introduction The various correlation measures help to ensure that items are correlated.
However, they do not ensure statistical significance.
We can raise the bar higher to select itemsets and use statistical tests.
This can be useful to analyze medical data and draw conclusions that are more likely to be valid… 37<br>
slide38. Non-redundant and productive itemsets Goal: Reduce the number of patterns found by identifying only those that are statistically significant.
Statistical significance is measured with Fisher Exact test. Productive itemsets Non-redundant Itemsets Webb, G. I., Vreeken, J.: Efficient discovery of the most interesting associations. ACM Transactions on Knowledge Discovery from Data. 8(3),15 (2014)<br>
slide39. Non-redundant itemsets An itemset is non-redundant (i.e., is a generator) if it has no proper subset having the same support (occurrence frequency)
Example:
sup({pregnant,heart disease,woman}) = sup({pregnant,heart disease})
Thus, {pregnant,heart disease,woman} is redundant.<br>
slide40. Productive itemsets Idea: Check if we can predict the support of an itemset by assuming that some of its partitions are independent.
An itemset is productive if all its partitions into two subsets are positively correlated with each other.
{a,b,c} two partitions: {a,b} and {c} another possibility: {a,c} and {b}
another possibility: {b,c} and {a}
Example:
{alchool, liver_cancer} is productive because alchool is positively correlated with liver_cancer<br>
slide41. Productive itemsets Example 2: {alchool, liver_cancer, black_hair}
{alchool, liver_cancer} is not positively correlated with {black_hair}.
Thus, {alchool, liver_cancer, black_hair} is not productive.<br>
slide42. Fisher Exact Test to evaluate statistical significance 0.001380 <0.05 {studying ,Men} is positively correlated? YES! =1 =11 =9 =3 =12 =12 =10 =14 Online calculator: www.statology.org/fishers-exact-test-calculator/<br>
slide43. How to find non-redundant productive itemsets? First algorithm: Opus-Miner (Webb, 2014)
It performs a depth-first search.
The user must set a parameter k.
The algorithm returns the top k non-redundant productive itemsets having the highest lift or leverage.
Other algorithms:
IDPI+: Allows to interactively search for productive itemsets (Fournier-Viger, et a., 2018, “Interactive Discovery of Statistically Significant Itemsets”) Query = X= {male, tobacco, Alzheimer} is productive?
…<br>
slide44. Conclusion We have discussed:
correlated patterns
bond
all-confidence
other correlation measures
the CORI algorithm
statistically significant patterns 44 Open source Java data mining software, 240 algorithms
http://www.phillippe-fournier-viger.com/spmf/<br>