Accelerating Read Mapping with FastHASH Hongyi
AM
Published · 34 slides · 0 views
1 / 1
Description
Accelerating Read Mapping with FastHASH Hongyi Xin Donghyuk Lee Farhad Hormozdiari Samihan Yedkar Can Alkan Onur Mutlu Carnegie Mellon University University of Washington University of California Los Angeles Outline Read
Related Topics
Share
Embed code
Download this presentation From Below
"Accelerating Read Mapping with FastHASH Hongyi" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
01
Accelerating Read Mapping with FastHASH Hongyi Xin† Donghyuk Lee† Farhad Hormozdiari ‡ Samihan Yedkar† Can Alkan § Onur Mutlu†
† Carnegie Mellon University § University of Washington
‡ University of California Los Angeles<br>
† Carnegie Mellon University § University of Washington
‡ University of California Los Angeles<br>
02
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 2<br>
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 2<br>
03
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 3<br>
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 3<br>
04
Read Mapping A post-processing procedure after DNA sequencing
Map many short DNA fragments (reads) to a known reference genome with some minor differences allowed 4 Reference genome Reads DNA, logically DNA, physically Mapping short reads to reference genome is challenging (billions of 50-300 base pair reads)<br>
Map many short DNA fragments (reads) to a known reference genome with some minor differences allowed 4 Reference genome Reads DNA, logically DNA, physically Mapping short reads to reference genome is challenging (billions of 50-300 base pair reads)<br>
05
Challenges Need to find many mappings of each read
A short read may map to many locations, especially with Next Generation DNA Sequencing
How can we find all mappings efficiently?
Need to tolerate small variances/errors in each read
Each individual is different: Subject’s DNA may slightly differ from the reference (Mismatches, insertions, deletions)
How can we efficiently map each read with up to e errors present?
Need to map each read very fast (i.e., performance is important)
Human DNA is 3.2 billion base pairs long Millions to billions of reads (State-of-the-art mappers take weeks to map a human’s DNA)
How can we design a much higher performance read mapper? 5<br>
A short read may map to many locations, especially with Next Generation DNA Sequencing
How can we find all mappings efficiently?
Need to tolerate small variances/errors in each read
Each individual is different: Subject’s DNA may slightly differ from the reference (Mismatches, insertions, deletions)
How can we efficiently map each read with up to e errors present?
Need to map each read very fast (i.e., performance is important)
Human DNA is 3.2 billion base pairs long Millions to billions of reads (State-of-the-art mappers take weeks to map a human’s DNA)
How can we design a much higher performance read mapper? 5<br>
06
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Preprocess the reference into a Hash Table
Use Hash Table to map reads
Problem and Goal
Key Observations
Mechanisms
Results 6<br>
Hash Table-Based Mappers
Preprocess the reference into a Hash Table
Use Hash Table to map reads
Problem and Goal
Key Observations
Mechanisms
Results 6<br>
07
Hash Table-Based Mappers [Alkan+ NG’09] 7 AAAAAAAAAAAA AAAAAAAAAAAC AAAAAAAAAAAT ...... CCCCCCCCCCCC ...... ...... ...... TTTTTTTTTTTT NULL Reference genome k-mer or 12-mer Location list—where the k-mer
occurs in reference gnome Once for a reference<br>
occurs in reference gnome Once for a reference<br>
08
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Preprocess the reference into a Hash Table
Use Hash Table to map reads
Problem and Goal
Key Observations
Mechanisms
Results 8<br>
Hash Table-Based Mappers
Preprocess the reference into a Hash Table
Use Hash Table to map reads
Problem and Goal
Key Observations
Mechanisms
Results 8<br>
09
12 Hash Table-Based Mappers [Alkan+ NG’09] AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTT CCCCCCCCCCCC TTTTTTTTTTTT Reference Genome Hash Table (HT) read k-mers AAAAAAAAAAAA CCCCCCCCCCCC TTTTTTTTTTTT …AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT… AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT AAAAAAAAAAAA 324 .. AAAAAAAAAAAAAACGCTTCCACCTTAATCTGGTTG.. read *** ..****************************************.. Invalid mapping 9 Verification/Local Alignment<br>
10
Advantages of Hash Table Based Mappers + Guaranteed to find all mappings
+ Tolerate up to e errors 10<br>
+ Tolerate up to e errors 10<br>
11
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 11<br>
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 11<br>
12
Problem and Goal Poor performance of existing read mappers: Very slow
Verification/alignment takes too long to execute
Verification requires a memory access for reference genome + many base-pair wise comparisons between the reference and the read
Goal: Speed up the mapper by reducing the cost of verification 12 95%<br>
Verification/alignment takes too long to execute
Verification requires a memory access for reference genome + many base-pair wise comparisons between the reference and the read
Goal: Speed up the mapper by reducing the cost of verification 12 95%<br>
13
Reducing the Cost of Verification We observe that most verification calculations are unnecessary
1 out of 1000 potential locations passes the verification process
We also observe that we can get rid of unnecessary verification calculations by
Detecting and rejecting early invalid mappings
Reducing the number of potential mappings 13<br>
1 out of 1000 potential locations passes the verification process
We also observe that we can get rid of unnecessary verification calculations by
Detecting and rejecting early invalid mappings
Reducing the number of potential mappings 13<br>
14
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 14<br>
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 14<br>
15
Key Observations Observation 1
Adjacent k-mers in the read should also be adjacent in the reference genome
Hence, mapper can quickly reject mappings that do not satisfy this property
Observation 2
Some k-mers are cheaper to verify than others because they have shorter location lists (they occur less frequently in the reference genome)
Mapper needs to examine only e+1 k-mers’ locations to tolerate e errors
Hence, mapper can choose the cheapest e+1 k-mers and verify their locations 15<br>
Adjacent k-mers in the read should also be adjacent in the reference genome
Hence, mapper can quickly reject mappings that do not satisfy this property
Observation 2
Some k-mers are cheaper to verify than others because they have shorter location lists (they occur less frequently in the reference genome)
Mapper needs to examine only e+1 k-mers’ locations to tolerate e errors
Hence, mapper can choose the cheapest e+1 k-mers and verify their locations 15<br>
16
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 16<br>
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 16<br>
17
FastHASH Mechanisms Adjacency Filtering (AF): Rejects obviously invalid mapping locations at early stage to avoid unnecessary verifications
Cheap K-mer Selection (CKS): Reduces the absolute number of potential mapping locations 17<br>
Cheap K-mer Selection (CKS): Reduces the absolute number of potential mapping locations 17<br>
18
Adjacency Filtering (AF) Goal: detect invalid mappings at early stage
Key Insight: For a valid mapping, adjacent k-mers in the read are also adjacent in the reference genome
Key Idea: search for adjacent locations in the k-mers’ location lists
If more than e k-mers fail—there must be more than e errors—invalid mapping 18 AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTT read Reference genome Valid mapping Invalid mapping<br>
Key Insight: For a valid mapping, adjacent k-mers in the read are also adjacent in the reference genome
Key Idea: search for adjacent locations in the k-mers’ location lists
If more than e k-mers fail—there must be more than e errors—invalid mapping 18 AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTT read Reference genome Valid mapping Invalid mapping<br>
19
12 Adjacency Filtering (AF) AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTT CCCCCCCCCCCC TTTTTTTTTTTT Reference Genome Hash Table (HT) read k-mers AAAAAAAAAAAA CCCCCCCCCCCC TTTTTTTTTTTT …AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT… AAAAAAAAAAAACCCCCCCCCCCCTTTTTTTTTTTT AAAAAAAAAAAA 324 24? 36? 336? *** +12 +24 557 569? 940 952? ✗ 19<br>
20
FastHASH Mechanisms Adjacency Filtering (AF): Rejects obviously invalid mapping locations at early stage to avoid unnecessary verifications
Cheap K-mer Selection (CKS): Reduces the absolute number of potential mapping locations 20<br>
Cheap K-mer Selection (CKS): Reduces the absolute number of potential mapping locations 20<br>
21
Cheap K-mer Selection (CKS) Goal: Reduce the number of potential mappings
Key insight:
K-mers have different cost to examine: Some k-mers are cheaper as they have fewer locations than others (occur less frequently in reference genome)
Key idea:
Sort the k-mers based on their number of locations
Select the k-mers with fewest locations to verify 21<br>
Key insight:
K-mers have different cost to examine: Some k-mers are cheaper as they have fewer locations than others (occur less frequently in reference genome)
Key idea:
Sort the k-mers based on their number of locations
Select the k-mers with fewest locations to verify 21<br>
22
Cheap K-mer Selection e=2 (examine 3 k-mers) 22 AAGCTCAATTTC CCTCCTTAATTT TCCTCTTAAGAA GGGTATGGCTAG AAGGTTGAGAGC CTTAGGCTTACC read Previous work needs to verify:
3004 locations FastHASH verifies only:
8 locations Locations Number of Locations Cheapest 3 k-mers Expensive 3 k-mers<br>
3004 locations FastHASH verifies only:
8 locations Locations Number of Locations Cheapest 3 k-mers Expensive 3 k-mers<br>
23
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 23<br>
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 23<br>
24
Methodology Implemented FastHASH on top of state-of-the-art mapper: mrFAST
New version mrFAST-2.5.0.0 over mrFAST-2.1.0.6
Tested with real read sets generated from Illumina platform
1M reads of a human (160 base pairs)
500K reads of a chimpanzee (101 base pairs)
500K reads of a orangutan (70 base pairs)
Tested with simulated reads generated from reference genome
1M simulated reads of human (180 base pairs)
Evaluation system
Intel Core i7 Sandy Bridge machine
16 GB of main memory 24<br>
New version mrFAST-2.5.0.0 over mrFAST-2.1.0.6
Tested with real read sets generated from Illumina platform
1M reads of a human (160 base pairs)
500K reads of a chimpanzee (101 base pairs)
500K reads of a orangutan (70 base pairs)
Tested with simulated reads generated from reference genome
1M simulated reads of human (180 base pairs)
Evaluation system
Intel Core i7 Sandy Bridge machine
16 GB of main memory 24<br>
25
FastHASH Speedup 25 19x With FastHASH, new mrFAST obtains up to 19x speedup over previous version, without losing valid mappings<br>
26
Analysis Reduction of potential mappings with FastHASH 26 99% 99% 99% 99% 99% FastHASH filters out over 99% of the potential mappings without sacrificing any valid mappings<br>
27
Other Key Results (In the paper) FastHASH finds all possible valid mappings
Correctly mapped all simulated reads (with fewer than e artificially added errors) 27<br>
Correctly mapped all simulated reads (with fewer than e artificially added errors) 27<br>
28
Outline Read Mapping and its Challenges
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 28<br>
Hash Table-Based Mappers
Problem and Goal
Key Observations
Mechanisms
Results
Conclusion 28<br>
29
Conclusion Problem: Existing read mappers perform poorly in mapping billions of short reads to the reference genome, in the presence of errors
Observation: Most of the verification calculations are unnecessary
Key Idea: To reduce the cost of unnecessary verification
Reject invalid mappings early (Adjacency Filtering)
Reduce the number of possible mappings to examine (Cheap K-mer Selection)
Key Result: FastHASH obtains up to 19x speedup over the state-of-the-art mapper without losing valid mappings 29<br>
Observation: Most of the verification calculations are unnecessary
Key Idea: To reduce the cost of unnecessary verification
Reject invalid mappings early (Adjacency Filtering)
Reduce the number of possible mappings to examine (Cheap K-mer Selection)
Key Result: FastHASH obtains up to 19x speedup over the state-of-the-art mapper without losing valid mappings 29<br>
30
Acknowledgements Carnegie Mellon University (Hongyi Xin, Donghyuk Lee, Samihan Yedkar and Onur Mutlu, co-authors)
Bilkent University (Can Alkan, co-author)
University of Washington (Evan Eichler and Can Alkan)
UCLA (Farhad Hormozdiari, co-author)
NIH (National Institutes of Health) for financial support 30<br>
Bilkent University (Can Alkan, co-author)
University of Washington (Evan Eichler and Can Alkan)
UCLA (Farhad Hormozdiari, co-author)
NIH (National Institutes of Health) for financial support 30<br>
31
Thank you! Questions?
Download link to FastHASH
You can find the slides on SAFARI group website:
http://www.ece.cmu.edu/~safari 31<br>
Download link to FastHASH
You can find the slides on SAFARI group website:
http://www.ece.cmu.edu/~safari 31<br>
32
Accelerating Read Mapping with FastHASH Hongyi Xin† Donghyuk Lee† Farhad Hormozdiari ‡ Samihan Yedkar† Can Alkan § Onur Mutlu†
† Carnegie Mellon University § University of Washington
‡ University of California Los Angeles<br>
† Carnegie Mellon University § University of Washington
‡ University of California Los Angeles<br>
33
Mapper Comparison: Number of Valid Mappings 33 Bowtie does not support error threshold larger than 3 FastHASH is able to find many more valid mappings than Bowtie and BWA<br>
34
Mapper Comparison: Execution Time 34 Bowtie does not support error threshold larger than 3 FastHASH is slower for e <= 3, but is much more comprehensive (can find many more valid mappings)<br>