Offline Contd Recap Attributes What are attributes Slide Credit Devi Parikh Recap Attributes Rich Understanding Image Credit Ali Farhadi Recap Annotations Zeroshot learning Frogs are green have heads and legs What is ID: 1018969
Download Presentation The PPT/PDF document "Exploiting Big Data via Attributes" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Exploiting Big Data via Attributes(Offline Contd.)
2. Recap - AttributesWhat are attributes?Slide Credit: Devi Parikh
3. Recap - AttributesRich UnderstandingImage Credit: Ali Farhadi
4. Recap - AnnotationsZero-shot learningFrogs are green, have heads and legs. What is this?Image Credit: Olga Russakovsky
5. Recap - AnnotationsAttributes help in getting richer description from Annotators.Image Credit: Devi Parikh
6. Understanding Single Image Or Learning a Classifier (w/ Human Feedback) Single or Few images to Big DataSlide Credit: Abhinav Gupta
7. Big Data90% of web data is visual!142 Billion Images6 Billion added monthly6 Billion Images72 hours of videouploaded every minuteHow attributes can help in learning from big-data?Slide Credit: Abhinav Gupta
8. What this part is aboutSemi-supervised LearningSlide Credit: Abhinav Gupta
9. What this part is aboutBefore the start of the debate, Mr. Obama and Mrs. Clinton met with the moderators, Charles Gibson, left, and George Stephanopoulos, right, of ABC News. A officer on the left of car checks the speed of other cars on the road.Weakly-labeled LearningSlide Credit: Abhinav Gupta
10. Key-insightAttributes can help in coupling the learning and hence provide constraints for joint learningAmphitheatreAuditoriumGoal: Learn multiple classifiers simultaneously. BanquetBedroomSlide Credit: Abhinav Gupta
11. Semi-supervised LearningShrivastava et al., 2012Slide Credit: Abhinav Gupta
12. Semi-Supervised[Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009]Slide Credit: Abhinav Gupta
13. Labeled Seed ExamplesAmphitheatreUnlabeled DataSelect CandidatesTrainModelsAdd to Labeled SetRetrainModelsAmphitheatreBootstrappingSlide Credit: Abhinav Gupta
14. BootstrappingRetrainModelsLabeled Seed ExamplesAmphitheatreUnlabeled DataSelect CandidatesAdd to Labeled SetAmphitheatre25th Iteration[Curran et al., PACL 2007]Semantic DriftAmphitheatre + AuditoriumSlide Credit: Abhinav Gupta
15. Graph-based Methods[Ebert et al., ECCV 2010] [Fergus et al., NIPS 2009]Slide Credit: Abhinav Gupta
16. AmphitheatreAmphitheatreConstrained BootstrappingAmphitheatreAuditoriumAuditoriumAmphitheatreAuditoriumAuditoriumSlide Credit: Abhinav Gupta
17. AmphitheatreAuditoriumAmphitheatreAuditoriumJoint Learning[Carlson et al., NAACL HLT Workshop on SSL for NLP 2009]Share DataConstrained BootstrappingSlide Credit: Abhinav Gupta
18. AmphitheatreAmphitheatreAuditoriumAuditoriumBanquetHallBanquetHallConference RoomConference RoomBinary Attributes (BA)IndoorMan-madeTables and ChairsLarge Seating CapacityIndoorMan-madeTables and ChairsLarge Seating Capacity[Farhadi et al., CVPR 2009] [Lampert et al., CVPR 2009]Slide Credit: Abhinav Gupta
19. Tables and ChairsConference RoomBanquetHallAuditoriumAmphitheatreIndoorLarge Seating CapacityMan-made[Patterson and Hays, CVPR 2012]Tables and ChairsConference RoomBanquetHallAuditoriumAmphitheatreIndoorLarge Seating CapacityMan-madeBinary Attributes (BA)Slide Credit: Abhinav Gupta
20. AuditoriumIndoorHas Seat Rows✗Slide Credit: Abhinav Gupta
21. Sharing via DissimilarityAmphitheatreAuditoriumHas Larger Circular Structures[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]Slide Credit: Abhinav Gupta
22. AmphitheatreAuditoriumHas Larger Circular Structures?Slide Credit: Abhinav Gupta
23. ✗AmphitheatreAuditoriumHas Larger Circular StructuresSlide Credit: Abhinav Gupta
24. DissimilarityHas Larger Circular Structures[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]Comparative AttributesSlide Credit: Abhinav Gupta
25. Similar to Relative Attributes.Uses pair of images as data-points during learning.Instead of predicting a real number, it uses binary classifier.Comparative AttributesSlide Credit: Abhinav Gupta
26. DissimilarityComparative AttributesHas Larger Circular Structures[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]……………………FeaturesGISTRGB (Tiny Image)Line Histogram of:LengthOrientationLAB histogramSlide Credit: Abhinav Gupta
27. …………DissimilarityComparative Attributes[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]…………Has Larger Circular StructuresClassifierBoosted Decision Tree[Hoiem et al., IJCV 2007]✗orHas Larger Circular StructuresSlide Credit: Abhinav Gupta
28. Comparative Attributes[Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008]Amphitheatre >BarnAmphitheatre >Conference RoomDesert >BarnIs More OpenChurch (Outdoor)>CemeteryBarn >CemeteryHas Taller StructuresSlide Credit: Abhinav Gupta
29. AmphitheatreAuditoriumAmphitheatreAuditoriumLabeled Seed ExamplesBootstrappingSlide Credit: Abhinav Gupta
30. Labeled Seed ExamplesAmphitheatreAuditoriumAmphitheatreAuditoriumBootstrappingAmphitheatreAuditorium Constrained BootstrappingIndoorHas Seat RowsAttributesHas Larger Circular StructuresComparativeAttributesSlide Credit: Abhinav Gupta
31. BanquetBedroomLabeled DataUnlabeled Datahas more spacehas larger structuresTraining Pairwise DataPromoted InstancesConference RoomBanquet Hall[Gupta and Davis, ECCV 2008]Comparative Attribute Classifiersmore spacelarger structuresAttribute Classifiersindoorhas grassScene Classifiersbedroombanquet hallSlide Credit: Abhinav Gupta
32. BootstrappingBA ConstraintsAmphitheatreC-BootstrappingSeed Images
33. BA ConstraintsBridgeSeed ImagesBootstrappingC-BootstrappingSlide Credit: Abhinav Gupta
34. Attributes help improve RecallSlide Credit: Abhinav Gupta
35. 140Banquet Hall10IterationsSeed ImagesSlide Credit: Abhinav Gupta
36. Iteration-1Iteration-60BootstrappingC-BootstrappingIteration-1Iteration-60Seed ImagesBedroom
37. Scene ClassificationEigen Functions: [Fergus et al., NIPS 2009]Slide Credit: Abhinav Gupta
38. Co-training (large Scale)15 Scene Categories25 Seed images / categoryUnlabeled Set1Million (SUN Database + ImageNet)>95% distractorsSUN Database: [Xiao et al., CVPR 2010]ImageNet: [Deng et al., CVPR 2009]Improve 12 out of 15 scene classifiersSlide Credit: Abhinav Gupta
39. LimitationsC-bootstrapping uses semantic attributes and needs manually specified relationshipsAmphitheatre >BarnAmphitheatre >Conference RoomDesert >BarnIs More OpenCan we learn the relationships?Slide Credit: Abhinav Gupta
40. Choi et al., Adding Unlabeled Samples to Categories by Learned Attributes , CVPR 2013Framework for jointly learning visual classifiers and noun-attribute mapping.
41. FormulationA joint optimization forLearning classifier in visual feature space (wca)Learning classifier in attribute space (wcv)With finding the samples (I)Non-convexMixed integer program: NP-complete problemSolution: Block coordinate-descentLearning a classifier on visual feature spaceLearning a classifier on attribute spacewith a selection criterionMutual ExclusionNot convexdiscretecontinuousSlide Credit: Junghyun Choi
42. Overview DiagramInitial Labeled-SamplesBuild Attribute SpaceProjectFind Useful AttributesUnlabeled SamplesProjectChoose Confident Examples To AddAuxiliary dataSlide Credit: Jonghyun Choi
43. Example Qualitative ResultsCategorical: common traits of a categorySelected by Categorical AttributesInitial Labeled Training ExamplesDottedAnimal-like shape…Slide Credit: Jonghyun Choi
44. Weakly-Labeled LearningGupta et al., 2008Slide Credit: Abhinav Gupta
45. Captions - Bag of NounsLearning Classifiers involves establishing correspondence.road.Aofficeron the left ofcarchecks the speed of other cars on theofficercarroadofficercarroadSlide Credit: Abhinav Gupta
46. Correspondence - Co-occurrence RelationshipBearWaterBearFieldWaterBearFieldSlide Credit: Abhinav Gupta
47. Co-occurrence Relationship (Problems)RoadCarRoadCarRoadCarRoadCarRoadCarRoadCarCarRoadRoadCarHypothesis 1Hypothesis 2CarRoadSlide Credit: Abhinav Gupta
48. Beyond Nouns – Exploit RelationshipsUse annotated text to extract nouns and relationships between nouns.road.officeron the left ofcarchecks the speed of other cars on theAOn (car, road)Left (officer, car)car officer roadConstrain the correspondence problem using the relationships On (Car, Road)RoadCarRoadCarMore LikelyLess LikelyKey insight: Solve the correspondence problem jointly using constraints!Slide Credit: Abhinav Gupta
49. RelationshipsPrepositions – A preposition usually indicates the temporal, spatial or logical relationship of its object to the rest of the sentence The most common prepositions in English are "about," "above," "across," "after," "against," "along," "among," "around," "at," "before," "behind," "below," "beneath," "beside," "between," "beyond," "but," "by," "despite," "down," "during," "except," "for," "from," "in," "inside," "into," "like," "near," "of," "off," "on," "onto," "out," "outside," "over," "past," "since," "through," "throughout," "till," "to," "toward," "under," "underneath," "until," "up," "upon," "with," "within," and "without” where indicated in bold are the ones (the vast majority) that have clear utility for the analysis of images and video.Comparative attributes – relating to color, size, movement- “larger”, “smaller”, “taller”, “heavier”, “faster”………Goal: Learn models of nouns, prepositions, comparative attributes simultaneously from weakly-labeled data.Slide Credit: Abhinav Gupta
50. Learning the Model – Chicken Egg ProblemChicken-Egg Problem: We treat assignment as missing data and formulate an EM approach.RoadCarCarRoadAssignment ProblemLearning ProblemOn (car, road)Slide Credit: Abhinav Gupta
51. EM Approach- Learning the ModelE-Step: Compute the noun assignment for a given set of object and relationship models from previous iteration.M-Step: For the noun assignment computed in the E-step, we find the new ML parameters by learning both relationship and object classifiers.For initialization of the EM approach, we can use any image annotation approach with localization such as the translation based model described in [1].[1] Duygulu, P., Barnard, K., Freitas, N., Forsyth, D.: Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. ECCV (2002)
52. Relationships modeledMost relationships are learned “correctly”Above, behind, below, left, right, beside, bluer, greener, nearer, more-textured, smaller, larger, brighterBut some are associated with the wrong featuresIn (topological relationships not captured by color, shape and location)on-top-oftaller (most tall objects are thin and the segmentation algorithm tends to fragment them)Slide Credit: Abhinav Gupta
53. Resolution of Correspondence Ambiguities[2] Barnard, K., Fan, Q., Swaminathan, R., Hoogs, A., Collins, R., Rondot, P., Kaufold, J.: Evaluation of localized semantics: data, methodology and experiments. Univ. of Arizona, TR-2005 (2005)Duygulu et. al [1]Our Approach[1] Duygulu, P., Barnard, K., Freitas, N., Forsyth, D.: Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. ECCV (2002)below(birds,sun) above(sun, sea) brighter(sun,sea) below(waves,sun)above(statue,rocks);ontopof(rocks, water); larger(water,statue)below(flowers,horses); ontopof(horses,field); below(flowers,foals)Slide Credit: Abhinav Gupta
54. SummaryAttributes can help in exploiting big-data.Attributes represent how class A is similar to class B, and how class B is different from class A…These relationships can help in formulating joint-learning problem and improve learning from large unlabeled and weakly labeled data.Slide Credit: Abhinav Gupta
55. Slide Credit: Abhinav Gupta