PDF-Optimizing Parallel Reduction in CUDA

Mark HarrisNVIDIA Developer Technology2Parallel ReductionCommon and important data parallel primitiveEasy to implement in CUDAHarder to get it rightServes as a great optimization exampleWell walk step. Basically a child CUDA Kernel can be called from within a parent CUDA kernel and then optionally synchronize on the completion of that child CUDA Kernel The parent CUDA kernel can consume the output produced from the child CUDA Kernel all withou t heterogeneous programming. Katia Oleinik. koleinik@bu.edu. Scientific Computing and Visualization. Boston . University. Architecture. NVIDIA Tesla M2070: . Core clock: 1.15GHz . Single instruction . 448 CUDA cores . . Acknowledgement: the lecture materials are based on the materials in NVIDIA teaching center CUDA course materials, including materials from Wisconsin (. Negrut. ), North Carolina Charlotte (. Wikinson. Shantanu. . Dutt. Univ. of Illinois at Chicago. An example of an SPMD message-passing parallel program. 2. SPMD message-passing parallel program (contd.). 3. node . xor. D,. 1. Reduction Computations & Their Parallelization. Martin Burtscher. Department of Computer Science. CUDA Optimization Tutorial. Martin Burtscher. burtscher@txstate.edu. http://www.cs.txstate.edu/~burtscher/. Tutorial slides. http://www.cs.txstate.edu/~burtscher/tutorials/COT5/slides.pptx. Håkon Kvale . Stensland. Simula Research Laboratory. PC Graphics Timeline. Challenges. :. Render infinitely complex scenes. And extremely high resolution. In 1/60. th. of one second (60 frames per second). on . Ubuntu. Cuda. download site. . https://developer.nvidia.com/cuda-downloads. $ . sudo. . dpkg. -. i. cuda-repo-ubuntu1404_7.5-18_amd64.deb . $ . sudo. apt-get update . $ . sudo. apt-get install . introduce the use of multiple CUDA streams to overlap memory transfers with kernel computations.. Also introduced is paged-locked memory. 2. Page-locked host memory. (also called pinned host memory). What is CUDA?. Data Parallelism. Host-Device model. Thread execution. Matrix-multiplication . GPU revised!. What is CUDA?. C. ompute . D. evice . U. nified . A. rchitecture. Programming interface to GPU. of. Split. Performance. . comparison. for NVIDIA CUDA . and. Intel . Xeon. . Phi. May, 2016. Contents. . Introduction. NVIDIA CUDA. Intel . Xeon. . Phi. . Conclusion. . tCSC. 2016. . t. oday’s. Goals for Rest of Course. Learn how to program massively parallel processors and achieve. high performance. functionality and maintainability. scalability across future generations. Acquire technical knowledge required to achieve the above goals. Se-Joon Chung. Background and Key Challenges. The trend in computing hardware is parallel systems.. It is challenging for programmers is to develop applications that transparently scales its parallelism to leverage the increasing number of processor cores.. Scientific Computing and Visualization. Boston . University. GPU Programming. GPU – graphics processing unit. Originally designed as a graphics processor. Nvidia's. GeForce 256 (1999) – first GPU. Agenda. Text book / resources. Eclipse . Nsight. , NVIDIA Visual Profiler. Available libraries. Questions. Certificate dispersal. (Optional) Multiple GPUs: Where’s Pixel-Waldo?. Text Book / Resources.