CUDA Programming

Preface

Chapter 1. A Short History of Supercomputing

Introduction

Von Neumann Architecture

Cray

Connection Machine

Cell Processor

Multinode Computing

The Early Days of Gpgpu Coding

The Death of the Single-Core Solution

Nvidia and Cuda

Gpu Hardware

Alternatives to Cuda

Conclusion

Chapter 2. Understanding Parallelism with GPUs

Introduction

Traditional Serial Code

Serial/Parallel Problems

Concurrency

Types of Parallelism

Flynn’s Taxonomy

Some Common Parallel Patterns

Conclusion

Chapter 3. CUDA Hardware Overview

PC Architecture

GPU Hardware

CPUs and GPUs

Compute Levels

Chapter 4. Setting Up CUDA

Introduction

Installing the Sdk Under Windows

Visual Studio

Linux

Mac

Installing a Debugger

Compilation Model

Error Handling

Conclusion

Chapter 5. Grids, Blocks, and Threads

What it all Means

Threads

Blocks

Grids

Warps

Block Scheduling

A Practical Example—Histograms

Conclusion

Chapter 6. Memory Handling with CUDA

Introduction

Caches

Register Usage

Shared Memory

Constant Memory

Global Memory

Texture Memory

Conclusion

Chapter 7. Using CUDA in Practice

Introduction

Serial and Parallel Code

Processing Datasets

Profiling

An Example Using AES

Conclusion

References

Chapter 8. Multi-CPU and Multi-GPU Solutions

Introduction

Locality

Multi-CPU Systems

Multi-GPU Systems

Algorithms on Multiple GPUS

Which GPU?

Single-Node Systems

Streams

Multiple-Node Systems

Conclusion

Chapter 9. Optimizing Your Application

Strategy 1: Parallel/Serial GPU/CPU Problem Breakdown

Strategy 2: Memory Considerations

Strategy 3: Transfers

Strategy 4: Thread Usage, Calculations, and Divergence

Strategy 5: Algorithms

Strategy 6: Resource Contentions

Strategy 7: Self-Tuning Applications

Conclusion

Chapter 10. Libraries and SDK

Introduction

Libraries

CUDA Computing SDK

Directive-Based Programming

Writing Your Own Kernels

Conclusion

Chapter 11. Designing GPU-Based Systems

Introduction

CPU Processor

GPU Device

PCI-E Bus

GeForce cards

CPU Memory

Air Cooling

Liquid Cooling

Desktop Cases and Motherboards

Mass Storage

Power Considerations

Operating Systems

Conclusion

Chapter 12. Common Problems, Causes, and Solutions

Introduction

Errors With CUDA Directives

Parallel Programming Issues

Algorithmic Issues

Finding and Avoiding Errors

Developing for Future GPUs

Further Resources

Conclusion

References

Index