1 Module: Association Analysis Last updated

slide1. 1 Module:
Association Analysis Last updated 11/2/20 Gary M. Weiss, CIS Dept, Fordham University Data Mining<br>
slide2. Overview of Association Analysis AA-1: Introduction to Association Analysis
AA-2: Mining Association Rules
Provide general background of process
AA-3: Apriori Algorithm
AA-4: Additional Issues Gary M. Weiss, CIS Dept, Fordham University 2<br>
slide3. AA-1: Intro to Assoc. Analysis What is association analysis?
What are association rules?
Goal of association rule mining
Representation of transactional data
Applications: Market Basket Analysis
Association rule format and evaluation
Support and confidence Gary M. Weiss, CIS Dept, Fordham University 3<br>
slide4. What Is Association Analysis? Finding associations in transactional data
Typically focuses on association rule mining
Used interchangeably with association analysis
Association rule mining finds frequent patterns (associations) among sets of items
Applications include market basket analysis 4 Gary M. Weiss, CIS Dept, Fordham University<br>
slide5. Association Rules Examples
Rule form: “Body ® Head [support, confidence]”.
buys(x, “diapers”) ® buys(x, “beer”) [0.5%, 60%]
major(x, "CS") /\ takes(x, "DB") ® grade(x, "A") [1%, 75%]
age=“30-45”, income=“50K-75K” ® car=“SUV”
Usually one relation so rule represented simply
Diapers ® Beer [0.5%, 60%]
We have seen rules before. In what context?
Rule Learning (e.g., Ripper)
What is the left side of a rule called? The right side?
LHS: antecedent RHS: consequent 5 Gary M. Weiss, CIS Dept, Fordham University<br>
slide6. The Goal of Association Mining Given:
Database of transactions
Each transaction is a list of items: an itemset
Example: a list of items purchased by a customer
Goal:
Find all rules that correlate one set of items with another set
Example: 98% of people who purchase tires & auto accessories also get automotive services 6 Gary M. Weiss, CIS Dept, Fordham University<br>
slide7. Transaction Data Representation A simplistic view of “shopping baskets”
Some important information not considered:
Quantity of each item purchased
Items are present or not (binary)
Unlike data with record format, if item not present there is no value– no missing or “0” value
Price paid, Profit, etc. Gary M. Weiss, CIS Dept, Fordham University 7<br>
slide8. Applications Many possible applications
Market basket analysis is good example
Looks for associations in items within a market basket
How can market basket analysis help store manager?
Assume you have a rule X  Y for some item X and Y
How can you apply this information to help your store?
If want to increase sales of Y then run sale on X
Locate X and Y near each other (e.g., bananas near cereal)
Locate X and Y far from apart to force shopper to walk thru store
On checkout print coupon for Y if shopper bought only X 8 Gary M. Weiss, CIS Dept, Fordham University<br>
slide9. Simple Supermarket Example Does “” mean causality or co-occurrence?
co-occurence 9 Market-Basket transactions Example of Association Rules {Diaper}  {Beer}{Milk, Bread}  {Eggs,Coke}{Beer, Bread}  {Milk} An itemset is simply a set of items Gary M. Weiss, CIS Dept, Fordham University<br>
slide10. Association Rule Format If {set of items}  Then {set of items} If {Diapers,
Baby Food} Condition {Beer, Chips} Results Then Customer
buys diaper Customer buys both Customer
buys beer Right side very often is a single item
Remember: Rules do not imply causality 10 Gary M. Weiss, CIS Dept, Fordham University<br>
slide11. Support of an Itemset / Rule Support Count:
Frequency count of occurrence of itemset
Example: ({Milk, Bread,Diaper}) = 2
Two transactions include Milk, Bread, and Diaper
Support:
Fraction of transactions containing the itemset
Example: s({Milk, Bread, Diaper}) = 2/5 (5 transactions)
40% of all transactions contain Milk, Bread, and Diaper
Meaning in terms of a rule
Frequency count or fraction of transactions containing all items in the entire rule (both sides) Gary M. Weiss, CIS Dept, Fordham University 11<br>
slide12. Rule Confidence Fraction of times items on the right side of rule are present in a transaction given that the items in the left side are present
Can be viewed as the precision or accuracy of rule
Corresponds to a conditional probability
As can be seen on the next slide, can be calculated from two support count calculations
I find this less intuitive than calculating it as a conditional probability Gary M. Weiss, CIS Dept, Fordham University 12<br>
slide13. Calculating Rule Confidence The confidence of a rule LHS => RHS can be computed as the support of the whole itemset divided by the support of LHS:

Confidence (LHS => RHS) = Support(LHS È RHS) / Support(LHS)

Note this represents the conditional probability: P((LHS È RHS)|LHS) Customer
buys diaper Customer buys both Customer
buys beer 13 Gary M. Weiss, CIS Dept, Fordham University<br>
slide14. Support and Confidence Calculations Given: {Milk, Diaper}  {Beer}
5 transactions in table
Compute: Rule Support (s)
Rule confidence (c) 14 Gary M. Weiss, CIS Dept, Fordham University |T| is # of transactions<br>
slide15. What is an Interesting Association? Requires domain-knowledge validation
Actionable, non-trivial, understandable
Algorithms provide statistics that help identify useful rules, but does not guarantee useful
Two statistics universally used:
Support and Confidence
In terms of support and confidence, what kind of rules are we most interested in?
High support and high confidence
These metrics may yield the obvious
For example, women give birth 15 Gary M. Weiss, CIS Dept, Fordham University<br>
slide16. AA-1 Review: Introduction to AA Described assoc. analysis at a high level and association rule format
The goal is to find interesting co-occurrences in transactional data
Transactional data excludes quantity, only lists what is present, and may omit other info
Market Basket Analysis
You should be able to list ways to use this info
You should be able to compute rule support and confidence Gary M. Weiss, CIS Dept, Fordham University 16<br>
slide17. AA-2: Mining Association Rules Describe the association rule mining task
Describe brute force approach
Evaluate computational complexity of this approach
Discuss relationship between itemsets and rules and develop a basic two-stage approach Gary M. Weiss, CIS Dept, Fordham University 17<br>
slide18. Association Rule Mining Task Given set of transactions, find all rules satisfying these conditions:
Rule Support Count ≥ minsup threshold
Rule Confidence ≥ minconf threshold
These thresholds are set by the user
Definitions:
Frequent Itemsets have support > minsup
Strong rules satisfy minsup and minconf 18 Gary M. Weiss, CIS Dept, Fordham University<br>
slide19. Association Rule Mining Approaches Brute-force approach:
List all possible association rules
Compute the support and confidence for each rule
Prune rules that fail minsup or minconf thresholds
Computationally prohibitive!
So what do we do? What have we seen done in such cases in this course?
Use heuristic methods and not achieve optimality
Not in this case! We will use some smarts and find optimal (i.e., correct) solution more efficiently than brute method. Gary M. Weiss, CIS Dept, Fordham University 19<br>
slide20. Number of Itemsets Rules are extracted from itemsets, so lets first discuss number of itemsets
Given d items, how many distinct itemsets?
Covered in any discrete math class that covers sets
Answer:
It is the number of all possible subsets
Therefore it is the cardinality of the Power Set of d items
Every item is either present or not present (2 choices)
So using basic counting theory, the answer is 2d 20 Gary M. Weiss, CIS Dept, Fordham University<br>
slide21. Given d unique items, how many association rules can we come up with?
Can someone explain the expression for R?
Do not worry about the expression below it
Pick k items for LHS and up to d-k items for RHS
C(d,k) is the number of ways to pick k LHS items and next expression is number ways to pick RHS items. Computational Complexity 21 Gary M. Weiss, CIS Dept, Fordham University<br>
slide22. Computational Complexity 22 Exponential Growth Gary M. Weiss, CIS Dept, Fordham University<br>
slide23. Mining Association Rules Example of Rules:
{Milk,Diaper}  {Beer} (s=0.4, c=0.67){Milk,Beer}  {Diaper} (s=0.4, c=1.0)
{Diaper,Beer}  {Milk} (s=0.4, c=0.67)
{Beer}  {Milk,Diaper} (s=0.4, c=0.67) {Diaper}  {Milk,Beer} (s=0.4, c=0.5)
{Milk}  {Diaper,Beer} (s=0.4, c=0.5) Observations:
Above rules are all partitions of the {Milk, Diaper, Beer} itemset
Rules originating from the same itemset have identical support (by definition) but may have different confidence values
This suggests decoupling support and confidence requirements 23 Gary M. Weiss, CIS Dept, Fordham University<br>
slide24. Mining Association Rules This suggests a two-step approach:
Frequent Itemset Generation
Generate all itemsets with support  minsup
Rule Generation
Generate all possible rules from each frequent itemset and only keep the ones where confidence > minconf
Step 1 is still computationally expensive
But we can often speed things up by being smart
Step 2 not bad if number of frequent itemsets is small and each has small number of items 24 Gary M. Weiss, CIS Dept, Fordham University<br>
slide25. Association Rule Mining Algorithms Many association rule algorithms
They use different strategies and data structures
The resulting sets of rules are the same
Given a transaction data set T and a minsup and minconf value, the set of association rules is uniquely determined
We study one famous algorithm: Apriori Algorithm
This algorithm, like Kmeans, was voted to be in the top-10 Data Mining algorithms 25 Gary M. Weiss, CIS Dept, Fordham University<br>
slide26. AA-2 Review: Mining Assoc. Rules Association rule mining task is to discover all rules with support and confidence above user-specified values
A brute force approach of generating all possible rules is intractable
Rules with support > minsup can only come from itemsets with support > minsup
Suggests approach of finding frequent itemsets then generating “confident” rules. Gary M. Weiss, CIS Dept, Fordham University 26<br>
slide27. AA-3: Apriori Algorithm Apriori Property
Two step process
Generating frequent itemsets
Overview of Join process with examples
Complete example of forming frequent itemsets
Forming rules from frequent itemsets
Discussion of efficiency of overall process Gary M. Weiss, CIS Dept, Fordham University 27<br>
slide28. The Apriori Algorithm Uses the two step approach
1) Frequent Itemset Generation
Generate all itemsets with support ≥ minsup
2) Rule Generation
Generate all rules with confidence ≥ minconf
User often provides template to restrict form of rules
Example: only one item on right side
Which step is more computationally expensive?
Frequent itemset generation (minsup critical) 28 Gary M. Weiss, CIS Dept, Fordham University<br>
slide29. Apriori property (downward closure property)
All subsets of a frequent itemset are also frequent
Does this make sense? 29 AB AC AD BC BD CD A B C D ABC ABD ACD BCD Gary M. Weiss, CIS Dept, Fordham University Apriori Property Will use algorithm in opposite direction:
if itemset not frequent then superset cannot be frequent<br>
slide30. Steps in Association Mining Find frequent itemsets
Itemsets with at least minimum support
Support is “downward closed” so a subset of a frequent itemset must be frequent
if {AB} is a frequent itemset, both {A} and {B} are frequent itemsets
If itemset does not satisfy minsup, none of its supersets will
This is key point that allows pruning of search space
Algorithm: iteratively find frequent itemsets with cardinality from 1 to k (k-itemsets)
Use the frequent itemsets to generate assoc. rules
Generate all binary partitions that fit template
Prune those with confidence < minconf 30 Gary M. Weiss, CIS Dept, Fordham University<br>
slide31. Frequent Itemset Generation 31 Given d items
2d possible itemsets

Since d=5, 32 here Gary M. Weiss, CIS Dept, Fordham University<br>
slide32. 32 Illustrating Apriori Principle Gary M. Weiss, CIS Dept, Fordham University Note AB not frequent even though A and B are frequent.<br>
slide33. Illustrating Apriori Principle Minimum Support = 3 Items (1-itemsets) If every subset is considered,
6C1 + 6C2 + 6C3
6 + 15 + 20 = 41
With support-based pruning,
6 + 6 + 4 = 16 33 Gary M. Weiss, CIS Dept, Fordham University<br>
slide34. Illustrating Apriori Principle Minimum Support = 3 If every subset is considered,
6C1 + 6C2 + 6C3
6 + 15 + 20 = 41
With support-based pruning,
6 + 6 + 4 = 16 Items (1-itemsets) 34 Gary M. Weiss, CIS Dept, Fordham University<br>
slide35. Illustrating Apriori Principle Items (1-itemsets) Pairs (2-itemsets)

(No need to generatecandidates involving Cokeor Eggs) Minimum Support = 3 If every subset is considered,
6C1 + 6C2 + 6C3
6 + 15 + 20 = 41
With support-based pruning,
6 + 6 + 4 = 16 35 Gary M. Weiss, CIS Dept, Fordham University<br>
slide36. Illustrating Apriori Principle Items (1-itemsets) Pairs (2-itemsets)

(No need to generatecandidates involving Cokeor Eggs) Minimum Support = 3 If every subset is considered,
6C1 + 6C2 + 6C3
6 + 15 + 20 = 41
With support-based pruning,
6 + 6 + 4 = 16 36 Gary M. Weiss, CIS Dept, Fordham University<br>
slide37. Illustrating Apriori Principle Items (1-itemsets) Pairs (2-itemsets)

(No need to generatecandidates involving Cokeor Eggs) Triplets (3-itemsets) Minimum Support = 3 If every subset is considered,
6C1 + 6C2 + 6C3
6 + 15 + 20 = 41
With support-based pruning,
6 + 6 + 4 = 16 37 Gary M. Weiss, CIS Dept, Fordham University<br>
slide38. Illustrating Apriori Principle Items (1-itemsets) Pairs (2-itemsets)

(No need to generatecandidates involving Cokeor Eggs) Triplets (3-itemsets) Minimum Support = 3 If every subset is considered,
6C1 + 6C2 + 6C3
6 + 15 + 20 = 41
With support-based pruning,
6 + 6 + 4 = 16 38 Gary M. Weiss, CIS Dept, Fordham University<br>
slide39. The Apriori Algorithm Terminology:
Ck is the set of candidate k-itemsets
Lk is the set of k-itemsets
Join step: Ck generated by joining 2 elements from Lk-1
Must be a lot of overlap for the join to only increase length by 1
k-2 items must overlap (each differs by 1)
Prune Step: For a k-itemset to be frequent all subsets must be frequent
If any subset not frequent, prune candidate k-itemset
To utilize this you simply start with k=1 (single-item itemsets) and work your way up from there! 39 Gary M. Weiss, CIS Dept, Fordham University<br>
slide40. The Apriori Algorithm An iterative algorithm that uses level-wise search
Find all 1-item frequent itemsets
Then all 2-item frequent itemsets, …
In each iteration k, only consider itemsets that contain some k-1 frequent itemset Gary M. Weiss, CIS Dept, Fordham University 40<br>
slide41. The Join Step Items in each itemset to be joined should be in a consistent order– any order
Such as lexicographic (alphabetical) order
Ck created by joining two itemsets from Lk-1 where the two itemsets have k-2 items in common
The two k-1 itemsets are joined only if they differ in the last position (so k-2 in common)
Then when you join them the size of the itemset goes up by one: (k-2) + 1 + 1 = k
Example: join pqr and pqs (you get pqrs) 41 Gary M. Weiss, CIS Dept, Fordham University<br>
slide42. Example of Generating Candidates (1) L3={abc, abd, acd, ace, bcd}
Self-joining: L3*L3. What do we merge first?
abc and abd yields abcd (add to C4)
Can abc merge with any other element in L3?
No, they do not differ in only last item!
What merges next?
acd and ace yields acde (add to C4)
abd and acd do not join since differ in 2nd pos.
Even though it would give abcd which is a candidate
Why: if the product were a candidate it would have already been generated (see next slide) 42 Gary M. Weiss, CIS Dept, Fordham University<br>
slide43. For abcd to be frequent by the Apriori property, abc, bcd, and abd must be frequent
abc and abd are alphabetically before bcd
So if we see abc and bcd we do not need to generate abcd because if abd were frequent then it would have already been generated 43 Gary M. Weiss, CIS Dept, Fordham University Example of Generating Candidates (2)<br>
slide44. So C4 tentatively equals {abcd, acde}
Recall L3={abc, abd, acd, ace, bcd}
Using Apriori property, do we remove abcd or acde from C4?
We remove acde but not abcd. Why?
We do not remove abcd since abc, abd, and acd are in L3
We remove acde since cde is not in L3
Join step does not guarantee Apriori not violated
If Apriori property not violated, then you still need to scan the database to ensure support > minsup before placing item into L4. 44 Gary M. Weiss, CIS Dept, Fordham University Example of Generating Candidates (3)<br>
slide45. Apriori Algorithm Example (minsup = 30%) 45 Database D Scan D C1 L1 L2 C2 C2 Scan D C3 L3 Scan D Gary M. Weiss, CIS Dept, Fordham University<br>
slide46. Do Not Forget Pruning Rules get pruned in two ways
Apriori property violated
If Apriori not violated, still must scan database and if minsup not exceeded then prune
Apriori property necessary but not sufficient to keep rule
If you forget to prune via Apriori property, will get same results since will catch on the scan
But I will take off points on a HW or exam. Make it clear when prune using Apriori property (do not fill in support count)
Apriori property cannot be violated until k=3
Gets trickier at k=4 since more subsets to check 46 Gary M. Weiss, CIS Dept, Fordham University<br>
slide47. More Complex Example Given the following database, list all frequent 3-itemsets and 4-itemsets with minsup of 40% 47 Gary M. Weiss, CIS Dept, Fordham University<br>
slide48. Solution The details are provided on the following webpage (example 2 not example 1):
http://www2.cs.uregina.ca/~dbd/cs831/notes/itemsets/itemset_apriori.html
Frequent 3-itemsets:
ABC, ABD, ACD, ACE,ADE, BCD,CDE
Frequent 4-itemsets
ABCD, ACDE 48 Gary M. Weiss, CIS Dept, Fordham University<br>
slide49. Step 2: Rules from Frequent Itemsets For each frequent itemset X,
Generate every possible association rule
Generate all binary partitions
That is, for each proper nonempty subset A of X
Let B = X - A
A  B is a potential association rule
Keep if confidence ≥ minconf 49 Gary M. Weiss, CIS Dept, Fordham University<br>
slide50. Generating Rules: an Example Suppose {2,3,4} is frequent, with sup=50%
Proper nonempty subsets:
{2,3}, {2,4}, {3,4}, {2}, {3}, {4}, with sup=50%, 50%, 75%, 75%, 75%, 75% respectively
These generate these association rules
Recall: Confidence(A  B) = support(A  B) / support(A), where support(A  B) =50%
2,3  4, confidence=100% (50%/50%)
2,4  3, confidence=100% (50%/50%)
3,4  2, confidence=67% (50%/75%)
2  3,4, confidence=67% (50%/75%)
3  2,4, confidence=67% (50%/75%)
4  2,3, confidence=67% (50%/75%)
All rules have support = 50%
Then apply confidence threshold to identify strong rules
Rules that meet the support and confidence requirements
If confidence threshold is 80% we are left with 2 strong rules 50 Gary M. Weiss, CIS Dept, Fordham University<br>
slide51. Generating Rules: Summary To recap, in order to obtain A  B, we need to have support(A  B) and support(A)
All the required information for confidence computation already recorded in itemset generation.
This step is not as time-consuming as frequent itemset generation
Hint: I almost always ask this on the exam 51 Gary M. Weiss, CIS Dept, Fordham University<br>
slide52. Efficiency of Apriori Algorithm Seems to be very expensive, but often runs quickly
If K = the size of the largest itemset
The algorithm makes at most K passes over data
In practice, K is often small (e.g., 10)
In this case the algorithm is very fast. Under some conditions, all rules can be found in linear time.
However if set minsup too small, then can have exponential blow up
Will take much time and will generate huge numbers of rules 52 Gary M. Weiss, CIS Dept, Fordham University<br>
slide53. AA-3 Review: Apriori Algorithm You should be able to do the following:
Explain the Apriori property and how it is used
Generate strong rules from a set of transactions given minsup and minconf
Explain the efficiency of the algorithm and what aspects may cause issues (e.g., small minsup)
Executing the Apriori algorithm by hand will certainly be testing on the final exam Gary M. Weiss, CIS Dept, Fordham University 53<br>
slide54. AA-4: Apriori Additional Issues Granularity and multi-level analysis
Rule lift as an improvement over confidence
Virtual Items
Pros and Cons of Association Rule Mining
Types of Association Rules Gary M. Weiss, CIS Dept, Fordham University 54<br>
slide55. Granularity of items The granularity of the items is an issue that impacts association rule mining
For example, which level of granularity should you use?
Diet coke?
Coke product?
Soft drink?
Beverage?
Should you include more than one level of granularity?
Some association finding techniques allow you to represent hierarchies explicitly 55 Gary M. Weiss, CIS Dept, Fordham University<br>
slide56. Multiple-Level Association Rules Items often form a hierarchy
Items at the lower level expected to have lower support
Rules regarding itemsets at appropriate levels can be quite useful
Transaction database can be coded based on dimensions & levels 56 Gary M. Weiss, CIS Dept, Fordham University<br>
slide57. Mining Multi-Level Associations A top-down, progressive deepening approach
First find high-level strong rules:
Milk  bread [20%, 60%]
Then find their lower-level “weaker” rules:
2% milk  wheat bread [6%, 50%]
Requires different thresholds at different levels
lower support at lower levels 57 Gary M. Weiss, CIS Dept, Fordham University<br>
slide58. Interestingness Measurements Objective measures
Two popular measurements:
Support
Confidence
Subjective measures
A rule (pattern) is interesting if
it is unexpected (surprising to the user); and/or
actionable (the user can do something with it) 58 Gary M. Weiss, CIS Dept, Fordham University<br>
slide59. Drawback of Confidence 59 Gary M. Weiss, CIS Dept, Fordham University Association Rule: Tea  Coffee
Confidence = P(Coffee|Tea) = 15/20 = 0.75
But P(Coffee) = 0.9! Although confidence is high, rule is misleading since:
P(Coffee|¬Tea) = 75/80 = 0.9375
Lesson: compare confidence to prior probability.
We address this by considering the lift of a rule, which should not be near 1.0 for interesting rules.
Lift = P(Y|X)/P(Y)
Lift(Tea  Coffee) = P(Coffee|Tea)/P(Coffee) = .75/.9 = .833
Note that if P(Coffee) was .25 then lift would have been 3.0<br>
slide60. Customer Number vs. Transaction ID You may encounter a problem where there is a customer id for each transaction
You can do association analysis based on customer id
If this is so, you need to aggregate the transactions to the customer level
If a customer has 3 transactions, then you create itemset containing all items in the union of the 3 transactions
Note we will ignore the frequency of purchase 60 Gary M. Weiss, CIS Dept, Fordham University<br>
slide61. Virtual Items If you’re interested in including other possible variables, can create “virtual items”

gift-wrap, used-coupon, new-store, winter-holidays, bought-nothing,… 61 Gary M. Weiss, CIS Dept, Fordham University<br>
slide62. Associations: Pros and Cons Pros
Can quickly mine patterns describing without major effort in problem formulation
Tool for hypothesis generation
Cons
Unfocused
not clear exactly how to apply mined “knowledge”
only hypothesis generation
Can produce many, many rules!
may only be a few nuggets among them (or none) 62 Gary M. Weiss, CIS Dept, Fordham University<br>
slide63. Association Rules Types Association rule types:
Actionable Rules – contain high-quality, actionable information
Trivial Rules – information already well-known by those familiar with the business
Inexplicable Rules – no explanation and do not suggest action
Trivial and inexplicable rules occur most often Gary M. Weiss, CIS Dept, Fordham University 63<br>
slide64. AA-4 Review: Additional Issues Can represent items and different levels of granularity and perform multi-level analysis
Rule confidence can be misleading and lift is a more useful metric
Virtual items can be used to represent factors that are not actual items
Pros and Cons
AA only for hypothesis generation, but good at it
Many association rules are not actionable Gary M. Weiss, CIS Dept, Fordham University 64<br>
slide65. Review of Assoc. Analysis Module AA-1: Introduction to Association Analysis
AA-2: Mining Association Rules
Provide general background of process
AA-3: Apriori Algorithm
AA-4: Additional Issues Gary M. Weiss, CIS Dept, Fordham University 65<br>