PDF-Optimizing Parallel Reduction in CUDA

Mark HarrisNVIDIA Developer Technology2Parallel ReductionCommon and important data parallel primitiveEasy to implement in CUDAHarder to get it rightServes as a great optimization exampleWell walk step. Basically a child CUDA Kernel can be called from within a parent CUDA kernel and then optionally synchronize on the completion of that child CUDA Kernel The parent CUDA kernel can consume the output produced from the child CUDA Kernel all withou t Shantanu. . Dutt. Univ. of Illinois at Chicago. An example of an SPMD message-passing parallel program. 2. SPMD message-passing parallel program (contd.). 3. node . xor. D,. 1. Reduction Computations & Their Parallelization. Goals for Rest of Course. Learn how to program massively parallel processors and achieve. high performance. functionality and maintainability. scalability across future generations. Acquire technical knowledge required to achieve the above goals. Martin Burtscher. Department of Computer Science. CUDA Optimization Tutorial. Martin Burtscher. burtscher@txstate.edu. http://www.cs.txstate.edu/~burtscher/. Tutorial slides. http://www.cs.txstate.edu/~burtscher/tutorials/COT5/slides.pptx. Karl Frinkle - Mike Morris. Getting HPC into Regional University Curricula with Few . Resources. Karl Frinkle - Mike Morris. (Doing big things with little $$$). Some of our goals . . .. Include . parallel. Karl Frinkle - Mike Morris. Getting HPC into Regional University Curricula with Few . Resources. Karl Frinkle - Mike Morris. (Doing big things with little $$$). Some of our goals . . .. Include . parallel. Challenges. . Rick Lindeman. . – . Rijkseaterstaat. Ecomm. May 31st 2017. Timeline. 2011. Optimizing . use. starts: . Largest. . mobility. . maangement. . programme. in (. dutch. ) . history. Vinay B Gavirangaswamy. Canny edge detection algorithm. Output. Original. Single Threaded. Output (contd.). Original . Multi-Threaded (. OpenMP. ). Output (contd.). Original. Multi-Threaded. (GPU-CUDA). Single machine, multi-core. P(OSIX) threads: bare metal multi-threading. OpenMP. :. compiler directives that implement various constructs like parallel-for. Single machine, GPU. CUDA/. OpenCL. :. bare metal GPU coding. of. Split. Performance. . comparison. for NVIDIA CUDA . and. Intel . Xeon. . Phi. May, 2016. Contents. . Introduction. NVIDIA CUDA. Intel . Xeon. . Phi. . Conclusion. . tCSC. 2016. . t. oday’s. Goals for Rest of Course. Learn how to program massively parallel processors and achieve. high performance. functionality and maintainability. scalability across future generations. Acquire technical knowledge required to achieve the above goals. Se-Joon Chung. Background and Key Challenges. The trend in computing hardware is parallel systems.. It is challenging for programmers is to develop applications that transparently scales its parallelism to leverage the increasing number of processor cores.. Scientific Computing and Visualization. Boston . University. GPU Programming. GPU – graphics processing unit. Originally designed as a graphics processor. Nvidia's. GeForce 256 (1999) – first GPU. Cliff Woolley NVIDIADeveloper Technology GroupGPUCPUGPGPU Revolutionizes ComputingLatency Processor Throughput processorLow Latency or High ThroughputCPUOptimized for low-latency access to cached dat