Altered time for OH tomorrow 9001000 am Please complete midsemester feedback Semantic Segmentation The Task person grass trees motorbike road Evaluation metric Pixel classification Accuracy ID: 1044901
Download Presentation The PPT/PDF document "Announcements Project proposal due tomor..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. AnnouncementsProject proposal due tomorrowAltered time for OH tomorrow: 9:00-10:00 am.Please complete mid-semester feedback
2. Semantic Segmentation
3. The Taskpersongrasstreesmotorbikeroad
4. Evaluation metricPixel classification!Accuracy?Heavily unbalancedIntersection over UnionAverage across classes and imagesPer-class accuracyAverage across classes and images
5. Things vs StuffTHINGSPerson, cat, horse, etcConstrained shapeIndividual instances with separate identityMay need to look at objectsSTUFFRoad, grass, sky etcAmorphous, no shapeNo notion of instancesCan be done at pixel level“texture”
6. Challenges in data collectionPrecise localization is hard to annotateAnnotating every pixel leads to heavy tailsCommon solution: annotate few classes (often things), mark rest as “Other”Common datasets: PASCAL VOC 2012 (~1500 images, 20 categories), COCO (~100k images, 20 categories)
7. Pre-convnet semantic segmentationThingsDo object detection, then segment out detected objectsStuff”Texture classification”Compute histograms of filter responsesClassify local image patches
8. Semantic segmentation using convolutional networkshw3
9. Semantic segmentation using convolutional networksh/4w/4c
10. Semantic segmentation using convolutional networksch/4w/4
11. Semantic segmentation using convolutional networksh/4w/4c
12. Semantic segmentation using convolutional networkscConvolve with #classes 1x1 filters#classesh/4w/4
13. Semantic segmentation using convolutional networksPass image through convolution and subsampling layersFinal convolution with #classes outputsGet scores for subsampled imageUpsample back to original size
14. Semantic segmentation using convolutional networkspersonbicycle
15. The resolution issueProblem: Need fine details!Shallower network / earlier layers?Not very semantic!Remove subsampling?Looks at only a small window!
16. Solution 1: Image pyramidsLearning Hierarchical Features for Scene Labeling. Clement Farabet, Camille Couprie, Laurent Najman, Yann LeCun. In TPAMI, 2013.Higher resolutionLess context
17. Solution 2: Skip connectionsupsample
18. Solution 2: Skip connections
19. Skip connectionsFully convolutional networks for semantic segmentation. Evan Shelhamer, Jon Long, Trevor Darrell. In CVPR 2015
20. Skip connectionsProblem: early layers not semanticHorseVisualizations from : M. Zeiler and R. Fergus. Visualizing and Understanding Convolutional Networks. In ECCV 2014.
21. Solution 3: DilationNeed subsampling to allow convolutional layers to capture large regions with small filtersCan we do this without subsampling?
22. Solution 3: DilationNeed subsampling to allow convolutional layers to capture large regions with small filtersCan we do this without subsampling?
23. Solution 3: DilationNeed subsampling to allow convolutional layers to capture large regions with small filtersCan we do this without subsampling?
24. Solution 3: DilationInstead of subsampling by factor of 2: dilate by factor of 2Dilation can be seen as:Using a much larger filter, but with most entries set to 0Taking a small filter and “exploding”/ “dilating” itNot panacea: without subsampling, feature maps are much larger: memory issues
25. Solution 4: Conditional Random FieldsIdea: take convolutional network prediction and sharpen using classic techniquesConditional Random Field
26. Fully Connected CRFsTypically, only adjacent pixels connectedFewer connections => Easier to optimizeDense connectivity: every pixel connected to everything elseIntractable to optimize except if pairwise potential takes specific form Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials. Philipp Krahenbuhl, Vladlen Koltun. In NIPS, 2011.
27. Fully Connected CRFsGrid CRFFully connected CRFGround truth
28. Putting it all togetherBest Non-CNN approach: ~46.4%Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan Yuille. In ICLR, 2015.
29. Other additionsMethodmean IoU (%)VGG16 + Skip + Dilation65.8ResNet10168.7ResNet101 + Pyramid71.3ResNet101 + Pyramid + COCO74.9DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan Yuille. Arxiv 2016.
30. Image-to-image translation problems
31. Image-to-image translation problemsSegmentationOptical flow estimationDepth estimationNormal estimationBoundary detection…
32. Learning to segment images
33. The 3 core problemsReconstructionReorganization (Grouping)RecognitionMachine LearningConvolutional Networks
34. Revisiting contour detection
35. The BSDS Benchmark5 humans annotate boundaries, take unionAlgorithm assigns “probability of boundary” to each pixelThreshold gives a binary mapABC
36. The BSDS BenchmarkFor particular threshold, get predicted boundary mapMatch predicted boundaries to ground truthConstruct bipartite graphConstruct optimal matching
37. The BSDS BenchmarkFor particular threshold, get predicted boundary mapMatch predicted boundaries to ground truthConstruct bipartite graphConstruct optimal matchingMatched predicted boundaries = correctPrecision = #matched predictions / #predictionsRecall = #matched GT / #GT
38. The BSDS BenchmarkCompute precision recall curveF-measure = harmonic mean of precision and recallPick point with maximum F measure
39. Learning to detect boundariesTwo steps in boundary detection:Compute local gradients: brightness, color, textureCombine information over entire image using normalized cutsLocal gradients (e.g., texture gradients) involve complex pipelinesCan we replace local gradients by something better?
40. Learning to detect boundariesLearn to identify local boundary pattern
41. Decision Trees and Random ForestsNon-linear classifiersHas four legs?Has fur?Lives in water?catzebrasnakefish
42. Decision trees for classificationLeaf nodes store distribution of labelsFor each test data point, find correct leaf node
43. Decision trees for classificationGrow node by nodeAt each node, try candidate splits based on sampled featuresPick split that minimizes entropy of daughters
44. Shape?,+,+,-,-,-,+Shape?Color?Size?squarenot square
45. Decision trees for classificationEnd of training: entropy below threshold / maximum depthEach leaf node contains subset of training data pointscompute distribution of labels at leaves
46. Random forestsLooking for best decision at every node bad: overfittingSample a few candidatesConstruct an ensemble of trees: random forests
47. Random forests for boundary detectionNot classification : label is boundary mapTwo key questions:How do we split nodes during training?How do we make predictions during testing?
48. Random forests for boundary detectionIdea: cluster boundary maps into classes at each nodeFast Edge Detection Using Structured Forests. P. Dollar, C. Lawrence Zitnick. In TPAMI 2015
49. Random forests for boundary detectionNot classification : label is boundary mapTwo key questions:How do we split nodes during training?How do we make predictions during testing?
50. Random forests for boundary detectionEach leaf node has set of boundary patchesSimply output average of boundary patches at leaf
51. Convolutional network based edge detectionDeep supervision: Skip connections, but additional loss at each layerMethodMax F measureStructured Edges74.6HED without deep supervision77.1HED with deep supervision78.2Humans80Holistically-Nested Edge Detection. Saining Xie, Zhuowen Tu. In ICCV, 2015.
52. Convolutional network based edge detection
53. Why do convolutional networks work for edges?Structured forests look at patches, convnet receptive fields are much largerConvnets are pretrained on ImageNet : can implicitly recognize objects?