The Unadorned Desk: Exploiting the Physical Space around a Display as - PDF document

The Unadorned Desk: Exploiting the Physical Space around a Display as an Input CanvasDoris Hausen, Sebastian Boring2,3, Saul GreenbergUniversity of Munich (LMU),HCI Group,Amalienstr. 17, 80333 Munich, GermUniversity of Copenhagen, Njalsgade 128, those that are in their peripheral vision. In contrast, working with computers requires almost everything to visually happen onscreen. Yet because space is limited, the socalled desktop metaphor usually separates object placement into one of several worspaces(see Figure 1a)the primary workspace, which covers most of the screen,holds the currently active document, which people normally work onthe secondary workspaceis the portion of onscreen space that contains a subset of artfacts related to the primary space’s activities, e.g., iconsand tool palettes; finally, the offscreen workspaceholds the remaining artifacts, where users through a series of operations make them explicitly visible in a temporary fashion (e.g., menus, dilog boxes).Yet, there is atension between these workspaces. The primary and secondary worspaces spatially tradeoff: the primary workspace dominates screen space, which leaves less space for its surrounding artifacts. This is espcially true for tablets and other devices with rather small diplays. The secondary and offscreen workspace also tradeoff: it is much easier to slect items in the secondary space, but only a few can be held there. In contrast, a huge number of items can be held in the offscreen workspace, but it is harder to select them (or to remember acelerator methods such as keyboard shortcuts) []. Instead of trying to fit everything on screen (directly or through menus), we investigate using the unadorned (i.e., uchanged except forsensing device) desk as a further space to contain artfacts. Our hypothesis is that people can then easily select commonly used functions (e.g., tools or other windows) located on the desk’s surface (see Figure 1This has several avantages. First, f we move artifacts from the secondary workspace to the desk, more display space can be allocated to the primary workspace. Second, if we move artifacts from the offscreen workspace to the desk, they will be easier to access. This also mimics the way we interact with everyday objects surrounding a document located on the desk (e.g., plaing paints and brushes nearby for rapid retrieval while draing). Fig. (a) The three workspaces present in the desktop metaphor: the primary workspace (1) holds the active document people work on; the secondary workspace (2) holds items related to the activities in the primary workspace and is permanently visible; and the off screen workspace (3) holds further items related to the document, yet people have to make them explicitly visible (e.g., menus). (b) The Unadorned Desk moves these workspaces onto a regular desk so that the primary workspace covers the entire display. When a person hovers over the interation area on the desk , feedback may be given onscreen. Toucing an item then select Previous work on digital deskrelies on a tight feedback loop, where visuals and interaction feedback were overlaid onto the regular desk surface i.e., by making the desk look and behave like a computer display. Examples include the use of projetors s &#x/MCI; 1 ;&#x/MCI; 1 ;20&#x/MCI; 2 ;&#x/MCI; 2 ;, &#x/MCI; 3 ;&#x/MCI; 3 ;31&#x/MCI; 4 ;&#x/MCI; 4 ;], a tabletop computer as desk replacement &#x/MCI; 5 ;&#x/MCI; 5 ;6&#x/MCI; 6 ;&#x/MCI; 6 ;], or by adding tablet compuers next to the display as an interactive region &#x/MCI; 7 ;&#x/MCI; 7 ;6&#x/MCI; 8 ;&#x/MCI; 8 ;]. These tend to be complex (or expesive) to set up. In contrast, the new generation of depthsensing technologies mean that detecting touches and hovering is lowcost, such as viaLeapMotionor Mcrosoft’s Knect camera. The problem is that thesetechnologies do not provide visual feedback. This begs the question: is visual feedback on the desk necessary?e are particularly interested in using the desk as iswith the smallest possible aterations. In this paper, wtake an extreme stancewhere we provide ether no feeback or feedback onscreen (rather than on the desk) solelyon dmand. Both aproaches keep desk instrumentation to a minimum, thus allowing for the use of desk such as at cafes to serve as a workspace. Using computer science terminolgy, this is a lower bounds investigation: we want to understand to what extent interation is possible using minimal or no augmentation (i.e., no visual targets or confirmtory feedback on the desk).To investigate how an unadorned deskcan be used as input space, we built a prottype using a Microsoft Kinect depth camera mounted atop a regular desk. Our dorned Detracks a person’s hand and allows for hoveringover and toucingcontent. As we were interested in how people can interact with offscreen content while keeping their attention on their main task, feedback is either not provied, or is given onscreen and on demand. We conducted two experimentshe first placementexperiment focused on placement strategies of participants. In the second acqusitionexperiment, varying numbers of virtual items were placed at predefined loctions and participants had to retrieve them to find out which number is still usable for offscreen interaction. Our work offers two contributions:(1)woring prototype that makeuseof an unadorned desk as input space by augmenting with a depth camera. And (2), xperimentalesults that inform the design of such interationswith respect to the amount of offscreen virtual items and the given onscreen feeback. Related WorkOur work builds on several areas of research that relate to how people organize douments on their desk, peripheral and bimanual interaction, interfaces without direct visual feedback, and augmented desks in general.Organizing the Desk.We routinely and fluidly arrange and manage documents on our physical desks without focusing much attention on it. We can do so because the document’s physical arrangement on the desk offers context information about the status and importance of certain tasks []. Malone studied desk organizations and found that files and (even more so) piles are the most commonly used arrangements on a desk []. Files are usually ordered systematically (e.g., in an alphabetic oder). Piles, however, are not organized delibeately, and people thus more likely use spatial organization for retrieval. Associated tools and materials are generally aranged so they are availble for reuse, such as by placing them nearby and readyhand during active use, or by oganizing them into known locations (such as desk drawers) []. Many systems try to bring this traditional way of organizing a desk into the digital world. In Data Mountainn&#x/MCI; 4 ;&#x/MCI; 4 ;28&#x/MCI; 5 ;&#x/MCI; 5 ;], people can organize browser bookmarks on a virtual table, which proved to be faster than bookmarking in Internet Explorer 4. BumpTopulates the desktop by allowing users to arrange documents in a virtual 3D space using physics []. Customization features in graphical user interfaces let people sptially arrange toolsaround the graphical desktop []. In contrast to these systems, we are iterested in using the desk as isinstead of mimicking it onscreen.Augmented and Interactive Desks. There is a history of work where digital content s brought onto the surface of the physical desk. This not only provides a workspace larger than the constraints of a computer display, but in some systems also allows both physical and digital artifacts to be used in tandem. Early work focused on (paially) digiizing the desk. The Digital Deskk&#x/MCI; 12;&#x 000;&#x/MCI; 12;&#x 000;31&#x/MCI; 13;&#x 000;&#x/MCI; 13;&#x 000;] uses a projected interface on the desk. A video camera senses interactions with fingers and/or a pen, and can capture content of paper materials (i.e. interacting with paper). Rekimoto et al.’s Augmented Surfacess&#x/MCI; 14;&#x 000;&#x/MCI; 14;&#x 000;26&#x/MCI; 15;&#x 000;&#x/MCI; 15;&#x 000;] are prjected extensions to a laptop’s display on a table or a wall. Users are able to drag cotent from their laptop onto the table where it is visible all times. Thus, the table servesas visual extension to the laptop’s display. Bonfiree&#x/MCI; 16;&#x 000;&#x/MCI; 16;&#x 000;20&#x/MCI; 17;&#x 000;&#x/MCI; 17;&#x 000;] projects additional content next to a laptop’s screen and allows touch input through caeras.More recent prototypes augment the computer screen with a horizontal digital dis-play (‘surface’) located underneath it. Surfaces typically allow for touch input, maing sensing of user interaction easy (e.g., Magic Deskk&#x/MCI; 19;&#x 000;&#x/MCI; 19;&#x 000;6&#x/MCI; 20;&#x 000;&#x/MCI; 20;&#x 000;]). Curvee&#x/MCI; 21;&#x 000;&#x/MCI; 21;&#x 000;33&#x/MCI; 22;&#x 000;&#x/MCI; 22;&#x 000;] and BendDeskk&#x/MCI; 23;&#x 000;&#x/MCI; 23;&#x 000;30&#x/MCI; 24;&#x 000;&#x/MCI; 24;&#x 000;] merge the horizontal desk area and the vertical display area into one gigantic highresolution touchsensitive display, where they are seamlessly connected through a curve. Variousstudies investigated how particular touch regions on both the horzontal and vertical displays are used e.g., to show that the regions next to keyboard and mouse are best suitable for coarse interaction []. Webuild on this in that we use the areas left and right of the keyboard/mouse in our two studies.Peripheral and Bimanual Interaction. Working with analogue documents on a desk often involves peripheral and bimanual interaction. Peripheral interactionoffers coarse input styles in the periphery of the user’s attention and thus quasiparallel to the current primary task. The fundamentalfor peripheral interaction are human capbilities such as divided attention (i.e., processing two tasks in parallel without switcing channels []), automatic and habitual processes (i.e., carried out with little metal effort and hardly any conscious control []), and propriception (i.e., being aware of one’s own body, its posture and orientation []). Today’s prototypes incorporating peripheral interaction mainly rely on TUIs(e.g., []) or freehand gestures []. Our work adds to this by investigatinghow people interact coarsely in their priphery. Bimanual (twohanded) interactionis the basis for peripheral interaction. While typically asymmetric, both hands influence each other leading to a kinematic chain &#x/MCI; 46;&#x 000;&#x/MCI; 46;&#x 000;13&#x/MCI; 47;&#x 000;&#x/MCI; 47;&#x 000;]. Studies show that bimanual interaction can improve performance []. At the same time, the body provides the kinesthetic reference frame, i.e., the user's sense of where one hand is relative to the body and the other hand []. Further, Balakrishnan et al. found that while separating visual feedback from the physical reference does affect performance, there is only a “remarkably small difference” when comparing interation with and without visual feedback as long as “bodyrelative kinesthetic cues are availablele&#x/MCI; 3 ;&#x/MCI; 3 ;5&#x/MCI; 4 ;&#x/MCI; 4 ;]. We build on this as we separate feedback from interaction.Interfaces without Direct Feedback.Spatial inteaction does not necessarily rely on direct feedback or feedback at all. Gustafson et al.’s Imaginary Iterfacesces&#x/MCI; 7 ;&#x/MCI; 7 ;14&#x/MCI; 8 ;&#x/MCI; 8 ;] make use of the visual shortterm and visuospatial memory. By forming an “L” with the nondominant hand a reference frame is createdSpin & Swingng&#x/MCI; 9 ;&#x/MCI; 9 ;2&#x/MCI; 10;&#x 000;&#x/MCI; 10;&#x 000;] depends on an aginary circle around theuser. By turning themselves, users navigate through the tentdisplayed on a handheld device. The concept of bodycentric interactionss&#x/MCI; 11;&#x 000;&#x/MCI; 11;&#x 000;10&#x/MCI; 12;&#x 000;&#x/MCI; 12;&#x 000;] employthe space around a person’s body to hold mobile phone functions. For exapleVirtual Shelvess&#x/MCI; 13;&#x 000;&#x/MCI; 13;&#x 000;22&#x/MCI; 14;&#x 000;&#x/MCI; 14;&#x 000;] positions items in a hemisphere in front of the usPoint upon Bodydy&#x/MCI; 15;&#x 000;&#x/MCI; 15;&#x 000;23&#x/MCI; 16;&#x 000;&#x/MCI; 16;&#x 000;] uses the forearm as interaction area, which can be divided at most into six distinct areas. GesturePadd&#x/MCI; 17;&#x 000;&#x/MCI; 17;&#x 000;27&#x/MCI; 18;&#x 000;&#x/MCI; 18;&#x 000;] and BodySpacee&#x/MCI; 19;&#x 000;&#x/MCI; 19;&#x 000;29&#x/MCI; 20;&#x 000;&#x/MCI; 20;&#x 000;] use different body loctions for commands. As with our system, no direct feedback is provided. These sys-tems rely primarily on spatial awareness and kinesthetic memory. Due to propriocetion, users have a good undestanding of where items are located and can easily even with closed eyes place and retrieve such objects []. These findings inspired us to miic regular desk use as means for interacting with digital content. Evaluating Offcreen InteractionsIn order to better understand how users can adapt to the novel input technology as well as how onscreen feedback for offscreen content would affect the interaction, we conducted two user studies. The first experiment aimed at understanding how people would spatially place various cotent items onto the desk that they would later retrieve. More precisely, we wanted to see whether people make use of special arangements of their content. In the second experiment (which was tuned to use the sults of the first study), we aimed to see how accuratelyparticipants couldlocate items placed in offscreen space as a function of the number of items in that space. The next section details the conditions and apparatus common to both experments.3.1Conditions Common to Both ExperimentsAlthough the tasks varied in both experiments, we had two conditions (addtiontheexperimentdependent ones) that were the same in both stuies: (1) the hand with which participants interacted in offscreen space, and (2) the type of feedback gien during the task. In the following, we describe these two condtions in more detail.Handedness:We chose to test our system with both hands. In the dominanthand condition, paticipants interacted with offscreen content using the hand they usually use to perform precise interactions (e.g., writing). In the nondominanthand condtion, they used the other hand. For each of the conditions, the interaction area was placed onthe desk so that it was closest to the hand with which they had to interact in offscreen space (i.e., not reachingleft of the keboard using theright hand). Fig.In the Singleeedback condition, the system showed the item closest to the particant’s hand (a: 1study, b: 2study). In the Fullfeedback condition, all items are shown with their correct spatial layout(c: 1study, b: 2study). Here the participant hovers over the word icon (and item #respectively). In aconditions, transparency encoed the distance to that item.Feedback:We had three conditions for onscreen feedback. In the No FeedbackNone) condition, participants did not receive any feedback on the computer’s dis-play, forcing them to rely solely on their spatial memory and propriception. In the Single Item FeedbackSingle) condition, participants only saw the item that was closest to their hand, with the distance being ecoded through transparency. That is, as participants moved closer to a respective item, the item’s icon became icreasingly opaque (see Figure ). In the Full Area FeedbackFull) condition, particpants saw all items in the interaction area with correct spatial layout. As in the Singlecondtion, the transparency of items again changed based on the distance btween them and the participants’ hands (see FigureThat is, the item directly below the hand was more opaque than the surounding items. The feedback area (400 400 pixels) was only shown onscreen while a partiipant’s hand was inside the interaction area and invisible otherwiseto not occpy valuable screen spaceIt was also located close to the interaction area (i.e., the bottom left or right corner of the display).We used a withinsubjects factorial design in both experiments: HandednessDominantNonDominantFeedbackNoneSingleFull). Feedbackwas couterbalanced across participants. To minimize changing the camera’s location for Handedness, we alternated participants so that the first participant had all three Feeacktypes with the Dominanthand and then again with the NonDominantone, while the scond one started with NonDominantetc.3.2ApparatusSetupand ParticipantsThe Unadorned Deskuses a Microsoft Kinect depth camera mounted on a tripod faing upside down(see Figure servinga subregion of the desk within which a person could interact using the hand. The prototype runs on an Intel i7 3.4 GHz coputer to allow for fast processing (i.e., 480 pixel frames at 30 frames per scond).We use the Kinect depth camera to gather hand information within the tracked region. The caera provides depth images where each pixel in a depth frame encodes that pixel’s ditance to the Fig. 3.TheUnadorned Desk: a Kinect track s the user’s hand. camera in millimeters.At startup, the system takes a series of depth images, averages them(to reduce noise), and uses them as ground truth. Once runing, it calculates the difference between the current depth frame and the calibrated depth image. The calclated diference image contains all points that are ‘new’ to the scene (e.g., a hand) with their distance to the desk. Using this point cloud, the system calclates the point of the hand closest to the corner of the interaction area that is the futhest away from the user (i.e., the tip of the middle finger). The vertical distance (depth) of that loction to the desk further determines the hand’s statetouching(depth thresholdhovering over(depth threshold), or absentif no hand is detected. Onscreen feeback is optionally provided once the user’s hand enters the interation area. When the hand touches an item, the system performs the action assciated with that item. In both experimentsparticipants were seated centrally in front of the computer’s display. The depth camera captured a rgion of 40cm cm (33.5cm on the top edge due to slight camera distortion) next to the keyboard aligned with the desk’s edges. For each Handedness condition, we moved the monitor, keyboard, mouse and chair to ensure that the participants are seated centrally in front of the display and close to the interation area. The tracked region on the desk was empty. The computer display’s background was set to a unorm color and had all desktop icons removed.Each study used 12 participants. Sexes were mixed (first: study 6 fmale; second: 4 female), and ages ranged from 19 to 30 (average was 24). Each person only particpated in one of the studies to minimize learning effects. Handedness varied, 9 werrighthanded in the first study, and all in the second. Each session lasted up to 1.5 hours, and all participants received $20 as compesation for their time.3.3HypothesesWe had similar hypotheses in both studies:H1.Itemetrieval time would increase as the number of offscreen items icreased. H2.Errorrate would increase as the number of offscreen items icreased. H3.Item retrieval time would increase when no feedback was prsent. H4.Offset and error rates would crease wfeedback prsent. Study 1Placing and Retrieving ContentThe purpose of our first study is to understand how participants would use of the adorned Deskto organizationally place and later retrieve an item, and the efect of having an increasing number of items placed within that space. In particular, we were interested in (1) how they arange a given number of items on their desk, and (the offset(and the item’s size respectively)when retrieving items to ensure sucessful pointing in the peripherOur HandednessFeedbackfactorial design was extended to include a third Setscondition, which is the number of items particpants had to place and retrieve in the offscreen space. We used wellknown, easily identfiable applications, which had meaning to our participants: Word, Excel, Power Point, Firfox, Thunderbird, Skype, QuickTime, and Internet Explorer. For each codition, the amount of items was ascending (to increase difficulty): 2, 4, 6, and 8 different items 4.1Tasks and ProcedureThe experiment consisted of two phases for each bination of Handednessand Feedback: placing items and later retrieing them. We instructed users to place items offscreen in a position of their own liing. However, items had to have a miimum distance of millimeters (and 50 pixels respectively) to avoid overlaps of them, which would make rtrieval more errorprone. Each set of items they had to place was shown on the monitor during the placement task (see Figure a), so that participants were aware of all tems and could group them if that would aid their memory. To place an item, participants first had to hit the spacebar to indicate they were staring the task, at which point timing began. Once the trial was active, they could move their hand into the interaction area and place the item by touching the desk’s surface. When feedback was given, already placed items were shown to give participants a feeling of the location of other items (see Figure ). Particpants repeated this step until they had placedall icons in the current set in the phyical offscreen space.Forretrievalthe system notified participantsscreen of which item to retrievebefore the trial began(Figure b). They then had to hit the spacebar to activate the trial (Figure c). Users would then retrieve that prevously placedoffscreen itemRetrieval worked exactly like the placement: hit spacbar for time measurement and touch a location to retrieve the itemterwards, the system prompted them with the next item until all itemswere retrieved.If the wrong item was retrieved, the particpant was not informed, the trial was not repeated and the experiment continuedbut theerror was recorded. For each Feedbackand Handednessbination, participants placed 4 Setsof items , and items) once and then retrieved each of them times. We collected placement sets (item trievals).For placement, we recorded all locations (as the center) of placed items. For the retrieval task, we measured the time from the beginning of a trial (i.e., hitting the spacebar) until they touched the desk’s surface. We further recorded the location they touched, the distance to the actual itemx,ylocation), and the amount of items that were closer than the correct one (i.e., errors). e manually countedthe particpants’ gazes, whether they looked at the interaction area, the feedback area, or both (the experimenter pressedkey for each gaze, which recorded).Finally, we asked ticipants to fill out a device assessment questionnaire: once after completing one Feebackand Handednesscondition, and again at the end of our study.4.2ResultsWe used heatmapsto uncover how people would freely place items on the desk. We then compared retrieval timeretrieval offset, and gazesusing epeated measures withinsubjects analyses of variance (ANOVA). For pairwise post hoc tests, we used Fig. 4. Commands: Placing an item (a), and retrieving it (b: before trial activation, c: after ). Bonferronicorrected confidence intervals to retain comparisons against = 0.05. When the assumption of sphericity was violated, we used GreenhouseGeisser to correct the degrees of fredom. All unstated values are � 0.05.We performed a HandednessFeedbackItems) withinsubjects ANOVA. As wedid not find any significant main effects or interactions for Handeness, we aggregated over Handednessfor all subjects in subsquent analyses. For heat map analysis, we mirrored interations performed in the area right to the keyboard to bring those into the coordinate system of the one left to the keboard.Strategic Placement of ItemsThrough aheat map analysis (see Figure 5) we found that many particpants tended to arrange items based on an imaginary grid(thusitem placement was not randomFurther, partiipants followed other semantic patterns: first, some placed items in a single row as in the dock in Mac X. During retrieval with feedback, participants then hovered over that line to find the correct item. Scond, some hierarchically grouped similar items together (e.g., all browser icons). They would later retrieve the temby first going to the general group area containing that item, and then selecting the particular itemFinaly, the more frequently they use an application based on their personal usage outside the study, the closer they would place it to the keboard.tems used lessoften are thus further away from the primary interation space. Participants did consider that areas further away would require more physical effort to access an item. However, all participants made use of the tirearea, as they felt more comfortable toaccess items placed further apart from each oter. We calculated three DistancesClosestAverageandHighest) between itemsthat they had placed offscreen. Participants placed items with an average distances for all conditions between 207.4 and 231.6millimeters (=219.2; =9.7). To understand whether Feedbackor the Setof items hadan influence on the distances between items, we performed separate 4 (FeedbackSetANOVAs for each DistanceFor the closest distance, we found a significant maineffect for Set1.953,21.487184.76, 0.00) and post hoc multiple means comparisons revealed that the dis-tance increases with a decreasing Set of items (all pairs except and itemsdiffer with 0.01regardless of FeedbackFeedbackhad an effect on the highest ditance between items, where we found significant main effects for both Feedback2,2215.49, 0.00and Set1280.001). Smaller Setslead to lower distances between itemsexcept for and items (all 0.001). More iportantly, in the Nonefeedback condition, participants placed items further away. The differences further increase with the Setsize. Particularly for items, None signifcantly differed from the other two (all .), and from Singlefor sizes and (all .). Thwhen relying on feedbackparticipants felt more comfortble placing items closer to each other. Interestingly, Singleand Full did not differ for any Setsize, and there was no significant difference between all three condtions for the Setwith 4 items, which we attribute to participants using the four coners of the area. Fig. 5. Heatmaps d: 2 to 8 items) show that users tend to arrange items in grids . Retrieval TimeWe compared retrieval times from the moment participants hit the spacebar until they retrieved an item. We only took into acountthe correct retrieval times (even so, we did not find significant differences between rtrieval times with and without errors). We performed a FeedbackSetwithin sujects ANOVA and found significant main efects for Feedback2,20= 31.098, 0.001and Set1.6,17.= 15.583, 011). Figure suggests that retrieval times slightly icrease with larger Setsowever, Feedbackinfluences retrieval times. Separate ANOVAs for each Setshowed that No Feedbackwas always faster (all 0.001). Furthermore, the two conditions with visual feeback were more strongly affected by the Setof itemsOverall, Nonewas the fastest (=1.40s, =0.36s), followed by Full=2.47s, =0.88s), and Single=2.68s, =1.06s). Fig. Results of theplacement study: (a) retrievaltime for one item for all feedback condtions and sets; (b) offsetfor correct retrievals measured as Euclidean distance between theitem’s centerand the touch’s location. Error barsrepresent 95% confidence intervalOffsWe compared the offset the distance between the touch point and the item’s center)We chose to only include successful retrievals to eliminate cases where paticipants did not remember an item’s location and thus radomly touched the desk.We performed a FeedbackSetwithin subjects ANOVA and only found significant main efect for Feedback= 4.201, ) but no effect for Setand no interactions. Figure summarizes the results for different Setsand FeebackFullhad the smallest offset between the touch and the item’s center millimeters), followed by Singlemillimeters) and Nonemillimters).Figure also reveals thatin order to have 95% successful selections regarless of Feedback and Setan item with a radius of at least millimetersis suffcientWe were also interested in the impact of Feebackon wrong retrievals (i.e., touch was closer to an incorrect item than to the correct one). We normalized the datavidingthe number of incorrect closer items by the maximum number of possible wrong items (i.e., 1 for a set of 2, 3 for a set of 4etc.). We performed a FeebackSetwithin subjects ANOVA and found a significant main effect for eedback= 4.914, 0.039). Post hoc tests revealed that only for a Setwithtems None was more errorprone than the other two conditions (). We blieve that particularly with no visual cues on the desk participants made use of space to more easily retrieve an item. That is, a larger offset still leads to correct rtrieval. In summary, the chance for an erroneous selection with Noneis 20% (=12%), and 15% (=8%) for SingleFullThis can be lowered, however, by increasing the required minmum distance between items. Gaze AnalysisWe told participants to minimize looking at the interaction area, and instead imagine that they were concentrating and looking at their primry onscreen task. We did not instruct them with respect to using the feedback widow, that is,they could freely make use of it. We report gazes averaged across both placement and retrieval phase. here were no gazes to the feedback area in the condtion.For Gazesto the Interaction Area, we performed a 3 × 4 (FeedbackSet) within subjects ANOVA and found significant main effects for Feedback1.126,12.3837.948, 0.01), Set3,33= 14.494, 0.001) andFeedbackSetinteraction 2.15,23.645= 8.618, 0.001). Post hoc tests revealed that for temsparticipants gazed at theinteraction area more often in the Nonecondition compared to gle(p 0.02). For tems, they gazed more often using Nonecompared to the other two conditions (all ). For Gazesto the Feedback Area, we did not test the Nonedition (as there was no feedback area) and performed a 2 × 4 (FeedbackSetwithin subjects ANOVA and found a significant main effect for Set2.121,23.3347.274, ). Pairwise comparisons showed that Gazesto the Feedback Area crease with larger Setsand differ from all and differs from 0.02). Overall, when No Feedbackwas presented, participants gazed at the interation area on the desk more often (0.24 times per trial), compared to gle(0.11) and Full(0.12). In conditions that had Feedback, participants gazed at the feedback area 0.72 (Full) and 0.66 (Single) times per trial.Thus, participants ‘left’ their fictive prmary task more often (i.e., looked away from it) when feedback was presented.4.3DiscussionDuring placement, we observed that participants used the whole interaction area, even though they stated that retrieval was easier if the item was placed closer to them. Placement was reasonably systematic, each followingsome kind of spatial organiztion. We noticed an increased time for placement and found significant diffeences for temdistances with No Feedbacke believe that participants put more effort into a good arrangement (with reasonably spaced items) to allow for easier retrieval aftewards, which was espially important when there was no visual feedback.During the retrieval stage, the condition caused two problems for particpants: (1) they had to remember where they put items, and (2) they were not iformed whether they actually had correctly acquired an item. Interestingly, participants stated aftewards that when feedback was provided they felt pressured to point more precisalthough this would not have been necessary(i.e., the selected item was always the one closest to the touched locatioresultingin longer retrieval times for conditions with feedbackne paticipant statedthathe started to search instead of think, which slowed him down.Our analysis of gazes supports this view: participants more often looked away from their fictivemary task when feedback was given. In fact, they looked more at the feedback area (when availablethan at the interaction area when no feedback was given.Feedbackdid help participants to remember loctions and creased their offset for larger Setsbut also slowed them down.Recall that these interaction techniques are to allow coarse interaction in the priphery (preferable with miimal attention). Our results suggest a suitable tradeoff between the item’s sizes and the overall number of items. We observed that particpants had problems recalling their spatial layout with or more items. Nevertheless, the results also indicate that participants were able to successfully retrieve or items even without feedback. While the number of maageable items in real life scenarios could be quite large (e.g., participants may want to place many items meaingful to their task on the interaction area), others have argued that a small number of such items could comprise a large number of the actions people actually do []. Eamples are frequently or recently used commands. Neverthless, this first experiment gests that having more items decreasesthe probability of a correct retrieval.Quite possibly, our results could be affected by less than optimal placements of items on the desk, e.g., due to a lack of visual cues on the experment’s desk. For this reason, we conducted a 2study that sptially separated items into a grid (a layout applied by manyin this first study), and that did not require to meorize locations, which is hard to achieveanyhowin a lab study setting, especialy for longterm memoryStudy 2Targeting ContentTo prevent memorizing(our lab study is only able to test shorttermmemorywhere items were placed and elimnate the potential influence of unfavorable placementpresented our participants with a predefined layoutBased on the 1study, where participants had arranged items in a grid, we created gridlike laouts with preplaced items, which was visible to them onscreen during each of the trials. We added a varable ItemSizewith three levels: Small10 cm wideMedium(13.3 cm), and Large(20 cm).To fill the entire interaction area, we decided to fill the grid accordinly. That is, we had 16 (4 4) Small, 9 (3 3) Med, and 4 (2 Large itemsIn this experiment, we were interested in getting more insights on item loctions and size with respect to retrieval time, offset and erors. 5.1Task and ProcedureBothtask and procedure weresimilar to the rtrieval task of the first study (though items are already preplaced on the desk). At the beginning of each trial, the system showed participants which item they had to retrieve (see ure ). As before, they activated the trial by hitting the spacebarand retrievethe respective item from offscreen space by touching the respective location. If they retrieved the correct item, the system promped them with the next item to retrieve. If they touched the wrong one, the system notified them that the trial was incorrect, icreased the item’s error count, and asked them to retrieve it again.However, to avoid frustration, the system moved on to the next item after three failed attempts.articipantshad to rtrieve each of the different emSizesthree times for all Handednessand Feedback combination, thus rquiring every participant to perform 522 retrievals. Ho Fig. 7. The target item (green), among all other items. a c: Largedium , and Smallitems. ever, we ecluded the first block as training block. We loggedtask timefrom the moment the spacebar was hituntil they eithersuccessfully retrieved the item or missed it; the Eucliean distance(i.e., offsetthe touch to the item’s center; and the number of errorswe allowed a maximumrors per item). As in the first study, we manually tracked whether the participantlooked at the interation area on the desk, on the feeback area on the screen or both of them. After each Feedbackand Handedness combination, participants illed out the same device assessment ques-tionnaire used in the first study as well as a closing questionaire.5.2Resultse performed a HandednessFeedbackItemSize) within subjects ANOVA. As in the first study, wedid not find any significant main effects or interations for Handedness. Thus, in subsequent analyses, we aggregated over ndednessacross all partiipants. We also excludeall erroneousunsuccessful retrievals from analyses of retrieval time and offset, as we ended a trial after three incorrect retrieals). Because of this, we ecluded 6.5% of all trials.Retrieval Timeegarding retrieval time for an item, we performed a FeebackItemSizewithin subjects ANOVA and found significant main effects for ItemSize1.272,13,997= 15.269, 0.001) and Feedback= 19.037, 0.001). We further found aItemSizeFeedback= 5.414, 0.001) interation. Post hoc multiple means comparisons showed that for all ItemSizesretrieval time difered significantly for the condition (usersneeded less time) copared to the other two (all 017). Furtherfor Singleand Fullthe retrieval time for the Small items differed significanly from the shorter retrieval time for the Mediumand Largeitems 0.001). Overall, Nonewas the fastest (=1.68s), followed by Single=2.25s), and Full=2.33s).Figuresummarizes these rsults. Fig. 8.Results of the targeting study: (a) retrieval time for one item for all feedback conditions and item sizes; (b) offset for correct retrievals measured as Euclidean ditance between the item’s center and the touch’s location. Error bars represent 95% confidence intervals.RetrievalOffsetFor the analysis of the offset of successful retrievals (meaured as Euclidean distance between the touch and the item’s center), we normalize the dis-tance as we had different ItemSizesTo do so, we dividthe measured offset by the maximum possible offset (i.e., the item’s actual size). With the normalized data, we performed a FeedbackItemSizewithin subjects ANOVA and found signifcant main effects for ItemSize39.318, 0.001), and Feedback 4.918, ), but no FeedbackItemSizeinteraction. Pairwise comparson of different ItemSizesacross all Feedback conditions furtherrevealed that participants were aways relatively closer to the item’s center (yet physically further away) forLargeitems 0.00). Overall, participants had the smallest ffset for Largeitems 46.9% of the item’s width, followed by Medium(52.1%), and Small(59.5%).ever, when looking at the nonnormalizedsetee Figure , the results are the exact opposite: participants had the least offset for Smallitems ), folowed by Medium) and Largeones.We normalized errors since we had a different amount of items depending on the ItemSizeWe divided the errors by the nuber of items in the grid for each trial. With these values, we performed a FeedbackItemSizewithin subjects ANOVA and found significant main effects for ItemSize= 88.909, 0.001), Feedback1.30914.4= 10.587, ), and a FeedbackItemSize2.12623.385= 4.036, ) interaction. Post hoc tests showed that the Nonecondition differed significanly from the other two for the Large(all 018) and from the Fullcondtion for the Smallitems). However, Feedbackconditions do not differ signifcantly for the Medium onesor all ItemSizesNonewas the most error prone (=0.41, =0.23), followed by Single=0.22, =0.16) and Full =0.18, =0.10). To understand the errorprone peformance, Figure visualizes the loctions where users had the most errorsas heat map. As trend, one can see that for larger items the corner furthest away from the user caused the most errors. However, the smaller items get, the more rors occur in the center, which can be explained by the desk’s edge (and the borders of the interaction area respely) being a reference frame. This made it easier to taget items close to the borders and harder in the center. In a second analysis, we excluded the items further away: for Largeitems, we exclued the top left item, for Mediumitems the three items furthest away, and for Small items the six items furthest away. We performed the same3 × 3FeedbackItemSizewithin subjects ANOVA using the reduced set and found sinificant main effects for ItemSize= 23.941, 0.001), Feedback1.33214.6489.973, 003), but no FeedbackItemSizeinteraction. Post hoc tests revealed that both, gle and Ful, differed significantly from Noneonly for Small items (all 0.04). This substantatesthat the corner furthest away was the most errorprone. erthelessNoneis still the most errorprone across all ItemSizeswith the least errors for Large itemswith 0.037 errors per trial (Singl: 0.012, Full: 0.019). Gaze AnalysisWe instructed participants in the same way as we did in the first eperiment. For Gazesto the Interaction Area, we peformed a within subjects ANOVA on Feedbackand found a significant main effect (1.136,12.4950.485, ). Multiple means comparisons revealed that usersgazed more often at the teractionAreain the None condition compared to the other (all We again excluded Fig. 9. Heatmaps showing errors (a gregated upon all feedback co n ditions, mirrored for the right interaction area) for a) large, b) medium and c) small items . Saturation indicates erors. the Nonecondition for Gazesto the Feedback Area, and peformed a within subjects ANOVA on the remaining two Feedback factorsand did not fina significant fect.Overall, None had the most gazes to the interaction area (0.23 times per trial), compared to Single(0.05) and Full(0.06). In Feedbackconditions, participantsgazed at the feedback area 0.69 (Full) and 0.65 (Single) times per trial.5.3DiscussionThe second study reenforces the findings from the first study. As before, No Feebackled to shortest retrieval times. Retrieval time also increased for Small items when feedback was present, yet it did not change when no feedback was given.NatrallySmallitems required participants to select more preciselyThe absolute from the center of an itemfor Large items (with m) would almost not suffice for Smalitems (as they only had a radius of 5 cm and a width of 10 cm respetivelyUsers seemed to make use of space for larger items (it did not matter how close to the ter they touched the item) and adjusted their offsets for smaller onesNo Feedbacksed significantly more errors with the corner further away from the user included in the analysis. Similar to Magic Desk [], where Bi et al. found that completion time was longer for areas further away from the keyboard, our sers had problems acquiring targets further away. When we excludeditems further awayfrom analysis (i.e., only considering that half of the interaction area closer to the particpant), the No Feedbackcondition only differed significantly from the otherfor the SmallitemsHowever, the error rate for Smallitemswas high regardless of feedback. Thus, items with a size of 10 cm or less are generally too small to be manageable in the periphery on an unadorned desk independent of the provision of feeback. General DiscussionIn both studies, more items in the interaction arearequire loweroffsetbetween a touch and the item’s centerin the first study to ensure that the correct one is still the closest item, and in the second study because items got smalleras their number icreased.As we hypothesized in H1, both studies showed that retrieval time increases as the number of itemsin the interaction area icreasesWhile H2 suggested that error rate increases with more offscreen items, our experiments only partially supportthisWe did not find evidence for more errors when increasing the item nuber (up to in the first study. Similarly, we did not find a significant effect in the second study for Mediumsized items, but did find a significant effectfor Smallones. ThusH2 (i.e., more errors with more items) is only supported for or more items. H3 suggested that participants’ time to retrieve items would increase when no feeback was present. Indeed, in both studies retrieval times were shorter whenparticipants did not have Feedbackwhich fully supports this hypothesisAnd finally, in we hypothesizedthat the participants’ offsets would increase and their error rate decrease when feeback was given. Yetour results at best show a tendency towards more errors and larger offsets without feedbackIn the first study, there was no significant effect for offset, and a significant efect on errors only for items, but not for . In the second studywe found an effect for Smalland Large itemsut not for Mediumones) tween No FeedbackandFull Feedback(yet not for Single. When only analying that half of the interaction area closer to the participant, No Feedbackonlydiffers signifcantly from the other two for Small itemsThus, our rsults therefore do not support H4 and only show a tendencytowards No Feedbackincreasing offset and errorsThe first study showed that participants made use of the whole interaction area, even with a small number of items. In the second studywe found that items located closer to keyboard and mouseare less error prone than those located further away. This suggests that a rectangular shape might not be the most suited interaction space. In isitu experiments, however, users would have better refeence frame (i.e., items on the table that convey meaning) instead of just the blank deskwhich ultimately would influence on the resultsur study showed that simple interaction on an unadorned desk is posible, albeit with a modest number of itemsand a reasonable item size (the first study revealed 85mm to achieve 95% successful retrievals, which would have sufficed for the second study).As the number of items increased, both retrieval times and error rate increased as well. However, previous studies on peripheral interaction showthat this interation style needs to be trained and learned to be effective [, which naturally is not possible in a shortterm laboratory experiment. Abandoning feedback leads to faster trieval times and functions (in terms of offset and errors) for a small numbersof items. Our finings suggest that the amount of items on the desk should be limited to less than ten. Similar to the shape of the interaction area, we epect this number to be higher if the desk contains more physical objects that serve as a visual cue or anchorand participants are used to the system and place meaningful items on the deskOverall, participants enjoyed interacting with the unadorned desk, and considered it to be fairly easy. All were able to carry out the interaction equally well with their dominant and nondominant hand, which strengthens our understanding that it is a ripheral interaction style. Interestingly, some of them were also irritated by this kind of interaction as they thought that the entire hand (and its palm respectively) acts as input, where in fact only a single point of the hand was tracked. Nevertheless, those participants adapted to the iteraction fairly quickly.Conclusion And Future Worke presented the Unadorned Desk, which supports peripheral coarse interaction and extends the inputand workspace beyond a computer’s display. The adorned Desk relies on hand tracking by a depth camera (Kinect). Our studies showedthat users are capable of interacing with virtual items on the desk, for small numbers of items even without screen feedback. It is a lowerbounds performance study, as we deliberatly did not place anything on the desk’s surface to indicate an item’svirtual loction.ur current experimental implementation suffers from threelimitations that restrict its deployment for everyday use. First, as with most optical tracing systems, the system is susceptible to false detections when sunlight hits the tracked areaThat problem also occurs with our depth camera, as the sun’s infrared light does interfere with the structured, infrared light of Microsoft’s Kinect.Second, the system requires mounting a depth camera atop a desk, which is unsuitable for situons where rapid setup and teardown is required (e.g., temporary desks). This limits our ability to study the Unadorned Desk during anticipated everday use. hird, the prototype does not yet address the fact that not every interaction on the desk is actualy meant as input to the computer(e.retrieving a bookWhile emerging technologies will likely adress the first two limitations, more research is needed to find an appropriate, distinct, yet not distracting gesture.Despite these limitations, our prototype allows us to evalate implications for interaction on unadorned desksand to envision example applictions such as those shown in the video figureThere are still many unanswered questions for future work. Our first experiments were carried outin an artificial lab setting, which brings with it usual concerns about external validity. The primary task was placement and retrieval, rather than one’s actual work. The items had no special significance. Interferences with other tasks carried out at the desk are not explored yet. Repeating the study in field cases could reveal nuances not seen in the lab. Our interaction area was rectagular, of a given size, and uncluttered; all these could both be varied to see how it affects performance. It was also in 2D (albeit with a hover plane). Yet a 3D iteraction space is possible, e.g., virtual piles where a user can naigate through it with the hovering hand. Finally, ours was a lower bounds study of an unadorned desk. There could be many possible ways of introducing modest adornments that indicate position. Although this would now introduce desk artifacts, it could improve performance significanly.AcknowledgmentsThis work is partially funded by the iCORE/NSERC/SMART Chair in Interactive Technologies, Alberta Innovates Technology Futures, NSERS, and SMART Technoogies Inc., and the German state of Bavaria. The authors thank Andreas Butz, Mathew Dunlap, Richard Fung, Ulrike Kister and the Interactions Lab at the Univesity of Calgary for their valuable input and their support throughout this project.We futher thank everyone paticipating in our two experiments.ReferencesAgarawala, A., Balakrishnan, R. Keepin’it real: pushing the desktop metaphor with phyics, piles and the pen. CHI200Altakrouri, B., Kawsar, F., Kortuem, G. Spin&Swing: Spatial interaction with orientation aware devices. Pervasive2010Bakker, S., Hoven, E. van den, Eggen, B. Design for the periphery. Eurohaptics, (2010Bakker, S., Hoven, E. van den, Eggen, B., Overbeeke, K. Exploring peripheral inteaction design for primary school teacers. TEI, (2012), 245252.Balakrishnan, R., Hinckley, K. The role of kinesthetic reference frames in twohanded iput performance. UIST, (1999)Bi, X., Grossman, T., Matejka, J., Fitzmaurice,G. Magic desk: bringing multitouch sufaces into desktop work. CHI, (2011).Boff, K.R., Kaufman, L., Thomas, J.P. Handbook of perception and human performance. (1986). Video Figure of the Unadorned Desk: http://youtu.be/ePQxR3EzJ_I Bondarenko, O., Ruud, J. Documents at hand : Learning from paper to improve digital technologies. CHI, (2005), 121130.Buxton, W., Myers, B.A. A study in twohanded input. CHI, (1986), 321326.Chen, X., Marquardt, N., Tang, A., Boring, S., Greenberg, S. Extending a mobile device's interaction space through bodycentric interaction. MobileHCI, (2012).Edge, D., Blackwell, A.F. Peripheral tangible interaction by analytic design. TEI, (2009)Greenberg, S. The computer user as toolsmith: The use, reuse, and organization of coputerbased tools. (1993).Guiard, Y. Asymmetric division on labor in human skilled bimanual action: The kiematic chain as a model. Journal of Motor Behavior 19, 4 (1987), 486517.Gustafson, S., Bierwirth, D., Baudisch, P. Imaginary interfaces: spatial interaction with empty hands and without visual feedback. UIST, (2010), 312.Harrison, S., Dourish, P. Replaceing space: The role of place and space in collaborative systems, CSCW, (1996), 6776.Hausen, D., Boring, S., Polleti, J., Butz, A. Exploring design and combination of ambient information and peripheral inteaction. DIS Work in Progress, (2012).Hausen, D., Boring, S., Lueling, C., Rodestock, S., Butz, A. StaTube: Facilitating state management in Instant Messaging Systems. TEI, (2012), 283290.Hausen, D., Wagner, C., Boring, S.,Butz, A. Comparing Modaltities and Feedback for Pripheral Interaction. CHI Extended Abstracts, (2013).Kabbash, P., Buxton, W., Sellen, A. Twohanded input in a compound task. CHI, (1994)Kane, S.K., Avrahami, D., Wobbrock, J.O., Harrison, B., Rea, A.D., Philipose, M., LMarca, A. Bonfire: a nomadic system for hybrid laptoptabletop interaction. UIST, (2009)Lane, D.M., Napier, H.A., Peres, S.C., Sandor, A. Hidden costs of graphical user interfaes: Failure to make the transition from menus and icon toolbars to keyboard shortcuts. ternational Journal of HCI 18, 2 (2005), 133144.Li, F.C.Y., Dearman, D., Truong, K. N. Virtual shelves: interactions with orientation aware devices. UIST, (2009), 125128.Lin, S., Su, C., Cheng, K., Liang, R., Kuo, T., Chen B.Y. PUBPoint upon body: Exploing eyesfree interaction and methods on an arm. UIST, (2011), 481487.Malone, T.W. How do people organize their desks?: Implications for the design of office information systems. ACM Transactions on Office Information Systems.1, 1 (1983).Mine, M.R., Brooks Jr., F.P., Sequin, C.H. Moving objects in space : Exploiting propriception in virtualenvironment inteaction. SIGGRAPH, (1997), 19Rekimoto, J., Saitoh, M. Augmented surfaces: a spatially cotinuous work space for hybrid computing environments. CHI, (1999), 378385.Rekimoto, J. GestureWrist and GesturePad: unobtrusive wearable interaction devices. Wearable Computers, (2001), 21Robertson, G., Czerwinski, M., Larson, K., Robbins, D.C., Thiel, D., Dantzich, M.V. Data Mountain : Using spatial memory for document management. UIST, (1998), 153162.Strachan, S., MurraySmith, R., O’Modhrain, S. BodySpace: inferring body pose for natral control of a music playerCHI EA, (2007), 20012006.Weiss, M., Voelker, S., Borchers, J. Benddesk: Seamless integration of horizontal and vetical multitouch surfaces in desk environments. Adjunct Proceeings ITS, (2009).Wellner, P. Interacting with paper on the DigitalDesk. ommof the ACM 36, 7 (1993)Wickens, C.D., McCarley J.S. Applied Attention Theory.Wimmer, R., Hennecke, F., Schulz, F., Boring, S., Butz, A., Hussmann, H. Curve: revisiing the digital desk. NordiCHI, (2010), 561570.