Presenters Muhammad Mohsin Buttg201103010 COE589 Paper Presentation Contents Introduction Background Traditional Recovery File Carving Smart Carver Conclusion Introduction This ID: 182270
Download Presentation The PPT/PDF document "The Evolution of File Carving" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
The Evolution of File Carving
Presenters: Muhammad Mohsin Butt(g201103010)
COE589
Paper
PresentationSlide2
Contents
IntroductionBackgroundTraditional RecoveryFile CarvingSmart Carver
ConclusionSlide3
Introduction
This Survey presents various File Carving techniques.File carving is a forensic technique to recover data based on file structure and content.No file system meta-data is required
Main Focus of this paper is on File carving techniques for Fragmented Data.Slide4
Background
File SystemPart of OS that manages the creation, deletion, allocation various other functions on files.FAT 32 and NTFS File Systems are most famous for Windows OS.Basic unit of data storage on disks is cluster.Clusters are usually multiples of 512 Bytes.Slide5
Background
Recovery In FAT -32(File Allocation Table)Files can be allocated in different ways.Contiguous Allocation.Linked Allocation.Indexed Allocation.Slide6
Background
Contiguous Allocation. Linked AllocationSlide7
Background
Indexed AllocationSlide8
Background
Indexed AllocationSlide9
Traditional Recovery Techniques
These recovery techniques use the met-data of file system to recover data.Data Storage in FAT32Slide10
Traditional Recovery Techniques
Deletion and Recovery in FAT32.Slide11
File Carving
What if we don’t have file system meta-data information ??File carving recovers data without using file system information.Knowledge of Structure of files to be recovered is used.File Carving can be divided into two categoriesFile Carving for non Fragmented data.File Carving for Fragmented data.Slide12
File Carving (First Generation)
Performed good for non fragmented data.In forensics user data (Images, documents etc) is important to recover.The search pool is reduced by removing operating system files which are detected using their MD5 Hash and keywords. Byte Sequences at prescribed offsets are used to identify files.Slide13
File Carving (First Generation)
Header and footer information of files to be recovered is used.JPEG image header cluster begin with sequence FFD8.JPEG image footer cluster contains the sequence FFD9.
Some files don’t have footer information.
BMP image has file size, number of clusters and other info present in header.
Number of unallocated clusters as indicated by the header of BMP image are merged for recovery.Slide14
File Carving (First Generation)
Foremost tool implemented both header to footer carving and also carving based on header and size of file information.Scalpel built on foremost engine improved the performance and memory usage of this file carving techniques.Both these suffer degradation in performance when data is fragmented.Slide15
Fragmentation
As files are edited, modified and deleted, most hard drives get fragmented.Also depends on allocation methodology of file system.Fragmentation in forensically important files like email, WORD document etc. is high. Why??Because of constant editing, deletion and addition PST files are most fragmented. Wear Leveling Algorithms in Next Gen Hard Drives (SSD) also cause fragmentation.Slide16
Fragmentation
Fragmented File RecoverySlide17
Graph Theoretic Carvers.
Provide Recovery of fragmented files.Recovery is formulated as a Hamiltonian Path Problem.Solved using alpha-beta heuristics.Slide18
Hamiltonian Path Problem.
Given a set of clusters.Find a permutation of these clusters that recovers the correct file.Identify pairs that are adjacent in original document.Assign weights between clusters which represent the likelihood one cluster following the other in original file.The best permutation is the on that maximizes the candidate weights of adjacent clusters.Slide19
Hamiltonian Path Problem.
Formulated as a graph.Vertices represent clusters.Edges represent weights between clusters.Problem Reduces to finding a maximum weight Hamiltonian path in this graph.Slide20
Assigning Weights
Weight assignment is the key in this type of carving.Prediction By Partial Matching (PPM) technique is used for assigning weights.PPM is good for Texts.Slide21
Assigning Weights
Weight Assignment in ImagesSlide22
K-Vertex Disjoint Path Problem.
Hamiltonian Path method assumed that all the clusters belong to same file.In actual systems multiple files are fragmented together.Headers of various files are identified from the pool of clusters. Graph is again formed using weights.Now K-disjoint paths are found in this graph using various algorithms where k represents number of headers found in previous step.Developed primarily for recovering images.Slide23
K-Vertex Disjoint Path Problem.
Various algorithms to find k disjoint paths.Unique Path (UP) Algorithms provides best performance.Each Cluster is assigned to only one file.Incorrect assignment may result in two files incorrectly recovered.Parallel Unique Path Algorithm.Shortest Path First Algorithm.Slide24
Parallel Unique Path (PUP).
Variation of dijkstra’s single source shortest path algorithm.Given k headers and a pool of clusters.Find the best cluster match for each of the headers.From the matches found in previous step take the best one and assign it to the header.
Remove the chosen cluster from the available clusters pool.
Find again the best match for found cluster and repeat the step3 until all files recovered.Slide25
Parallel Unique Path (PUP).Slide26
Shortest Path First
This algorithm presents the idea that best recoveries have lowest average path costs.The average path cost is simply the sum of the weights between the clusters of a recovered file divided by the
number
of clusters
.
Takes one image at a time.
Reconstruct the image.
After reconstruction the clusters used are not removed from the cluster pool.
This process is repeated for all the images.
Out of all the recovered images the one with lowest path cost is assumed as the best recovery.
Clusters associated with the best recovery are than removed.Slide27
Shortest Path First
This algorithm presents the idea that best recoveries have lowest average path costs.The average path cost is simply the sum of the weights between the clusters of a recovered file divided by the
number
of clusters
.
Takes one image at a time.
Reconstruct the image.
After reconstruction the clusters used are not removed from the cluster pool.
This process is repeated for all the images.
Out of all the recovered images the one with lowest path cost is assumed as the best recovery.
Clusters associated with the best recovery are than removed.Slide28
Results
Shortest Path First provides an accuracy of 88%PUP provides an accuracy of 83% but is faster.Both require edge weights to be pre computed.For large hard drives requirement of forming weights by checking the likelihood between clusters is a major drawback.Slide29
BiFragment Gap Carving
Most of the real world data is bi-fragmented. This technique works for files with known header and footer.Files should be decodable or be validated via their structure.Works by searching for combinations between identified header and footer.Slide30
BiFragment Gap CarvingSlide31
Smart Carver
Can work on fragmented and non fragmented data.Wide variety of file types supported.PreprocessingData clusters are decrypted or decompressed.CollatingClassification of cluster to various file types.ReassemblySlide32
Smart Carver (PreProcessing)
Compressed and encrypted drive are decrypted/decompressed in this stage.Removing known clusters from the disk based on file system met-data.Helps increase the speed and reduce the amount of data for next phases.Allocated files and Operating system specific data can be pruned since it doesn’t have any use in forensics.Slide33
Smart Carver (Collating)
Classifies the disk clusters as belonging to certain file types.Reduces the cluster pool in recovery of file of each type.Keyword/Pattern MatchingLooking for sequences to determine the type of cluster.E.g. <html> tags in a cluster collates to html file.ASCII characters frequencyHigh frequency of these indicate that data is non Video or Image.Slide34
Smart Carver (Collating)
File FingerprintsUses Byte Frequency Distribution (BFD) to determine the type of file.BFD is generated by creating a histogram for the file.A centroid model for each file type is created using the mean and standard deviation of each byte value.Still they face problem differentiating JPEG and ZIPStill a hot research topic.Slide35
Smart Carver (ReAssembly)
Reassembly can done byFinding the starting fragment of a file that contains the header.Merging clusters belonging to same fragment.Finding the fragmentation point i.e. the last cluster in current segment.Starting point of next fragment.Ending point of last fragment. Last cluster
contating
the footer.Slide36
Smart Carver (ReAssembly)
Merging of similar Clusters can be done in two ways.KeyWord/DictionaryThis occurs when a word is formed between the two cluster boundaries.E.g. One cluster ends at “he”, second starting at “llo World”. Both can be merged.
File Structure
File structure can help in merging. Length field in headers indicate the length of data. E.g. in PNG file if length value is k than after k clusters CRC of data associated is present. If the data in between has same CRC than we can merger all clusters in between. Otherwise fragmentation is present.Slide37
Smart Carver (ReAssembly)
Sequential Hypothesis Parallel Unique Path Algorithm( SHT-PUP) for reassembly.Modification of PUP algorithm.In PUP when best match is found for the available k headers and out of them the best one is selected.The clusters immediately following the newly found clusters are tested using sequential hypothesis testing until a fragmentation point is reached.Slide38
Smart Carver (ReAssembly)
Sequential Hypothesis Testing.This is done by using the weight vector. i.e. the weights of all clusters in the pool.Two Hypothesis are tested.One that says the clusters belong in sequence to fragmentOther says that they don’t.The ratio
is used to test the hypothesis. Slide39
Conclusion
Various File Carving methods for fragmented files are presented in the survey.Problem of finding best weight is still an open research issue.