Environment

Intel® ISPC for Xe is supported on Linux for quite a while (recommended and tested Linux distribution is Ubuntu 22.04) and it's got Windows support since v1.16.0.

You need to have a system with Intel(R) Processor Graphics Gen9 or later.

For the execution of ISPC programs on GPU, please install Intel(R) Graphics Compute Runtime on Linux or Intel(R) Graphics Driver for Windows on Windows. Additionally you need Level Zero Loader.

To use ISPC Run Time for CPU on Linux you need to have Intel(R) oneAPI Threading Basic Blocks installed on your system. Consult your Linux distribution documentation for the installation of TBB runtime instructions.

Basic Command-line Options

A bunch of new targets were introduced for GPU support: gen9-x8, gen9-x16, xelp-x8, xelp-x16, xehpg-x8, xehpg-x16, xehpc-x16 and xehpc-x32.

If the -o flag is given, ispc will generate a SPIR-V output file. Optionally you can use --emit-spirv flag:

ispc --target=gen9-x8 --emit-spirv foo.ispc -o foo.spv

To generate L0 binary, use --emit-zebin flag. When you use L0 binary you may want to pass some additional options to the vector backend. You can do this using --vc-options flag.

When targeting Xe targets, xe64 architecture must be used. It corresponds to 64-bit host and has 64-bit pointer size. We don't support 32-bit pointers for Xe targets.

To generate LLVM bitcode, use the --emit-llvm flag. To generate LLVM bitcode in textual form, use the --emit-llvm-text flag.

Optimizations are on by default; they can be turned off with -O0.

Generating a text assembly file using --emit-asm is not supported yet. See How to Get an Assembly File from SPIR-V? section about how to get the assembly from SPIR-V file.

There is a new link mode in ispc allowing to link several LLVM bitcode or SPIR-V files to selected output format: LLVM bitcode (default), LLVM bitcode text or SPIR-V.

Link two SPIR-V files to LLVM BC output:

ispc link test_a.spv test_b.spv --emit-llvm -o test.bc

Link LLVM bitcode files to SPIR-V output:

ispc link test_a.bc test_b.bc --emit-spirv -o test.spv

ISPCRT Objects

The ISPC Run Time uses the following abstractions to manage code execution:

• Device - represents a CPU or a GPU that can execute SPMD program and has some operational memory available. The user may select a particular type of device (CPU or GPU) or allow the runtime to decide which device will be used.
• Memory view - represents data that need to be accessed by different devices. For example, input data for code running on GPU must be firstly prepared by a CPU in its memory, then transferred to a GPU memory to perform computations on. Memory view can also represent memory allocated using a Unified Shared Memory mechanism provided by oneAPI Level Zero. Pointers to data allocated in the USM are valid both on the host and on the device. Also, there is no need to explicitly handle data movement between the CPU and the GPU. This is handled automatically by the oneAPI Level Zero runtime.
• Task queue - each device has a task (command) queue and executes commands from it. Commands may be executed simultaneously. To prevent that one should explicitly insert barriers in places where synchronization is required. Task queue sync method stops the host thread until GPU computation completed. For asynchronous computation, one should utilize CommandQueue and CommandList objects.
• CommandQueue - represents a logical input stream to the device and directly maps to L0 command queues.
• CommandList - represents commands to be executed on a command queue. It can be created by calling createCommandList method of CommandQueue object. Synchronization between all commands in list has to be done explicitly by putting barriers if needed. Fine-grained synchronization via Events are not supported yet.
• Fence - is a synchronization primitive to communicate to the host that command list execution has completed. Fence is created upon command list submission. It can be waited synchronously (sync) and asynchronously (periodically checking status). Fence has two states ISPCRT_FENCE_UNSIGNALED and ISPCRT_FENCE_SIGNALED returned by status method.
• Barrier - synchronization primitive that can be inserted into a task queue to make sure that all tasks previously inserted into this queue have completed execution. It is not needed to include barrier between memory copy and kernel execution. All memory scheduled to be copied before the kernel execution will complete before the kernel start. This is implemented by ISPC Runtime using finer grain mechanisms than a barrier and is more efficient.
• Module - represents a set of kernels that are compiled together and thus can share some common code. In this sense, SPIR-V file produced by ispc is a module for the ISPCRT. User can provide additional options for module compilation using ISPCRTModuleOptions. Currently ISPCRTModuleOptions structure allows to set stack size for VC backend which is used to compile SPIR-V. The set of supported options will be extended as needed.
• Kernel - is a function that is an entry point to a module and can be called by inserting kernel execution command into a task queue. A kernel has one parameter - a pointer to a structure of actual kernel parameters.
• Future - can be treated as a promise that at some point kernel execution connected to this object will be completed and the object will become valid. Futures are returned when a kernel invocation is inserted into a task queue. When the task queue is executed on a device, the future object becomes valid and can be used to retrieve information about the kernel execution.
• Array - Conveniently wraps up memory view objects and allows for easy allocation of memory on the device or in the Unified Shared Memory (USM). The ISPCRT also provides an example allocator that makes it even more simple to allocate data in the USM and a SharedVector class that serves the same purpose. See XPU examples and documentation for more details.

All ISPCRT objects support reference counting, which means that it is not necessary to perform detailed memory management. The objects will be released once they are not used.

Execution Model

The idea of ISPC tasks has been extended to support the execution of kernels on a GPU. Each kernel execution command inserted into a task queue is parametrized with the number of tasks (threads) that should be launched on a GPU. Each task must decide on which part of the problem it should work, exactly the same as it happens in the CPU case. Within tasks, the program executes in SPMD manner (again the regular ISPC execution model is copied). All built-in variables used for that purpose (such as taskIndex, taskCount, programIndex, programCount) are available for use on GPU.

Configuration

The behavior of ISPCRT can be configured using the following environment variables:

• ISPCRT_USE_ZEBIN - when defined as 1 forces to use experimental L0 native binary format. Unlike SPIR-V files, zebin files are not portable between different GPU types.
• ISPCRT_IGC_OPTIONS - ISPCRT is using an Intel® Graphics Compiler (IGC) to produce binary code that can be executed on the GPU. ISPCRT allows for passing certain options to the IGC via ISPCRT_IGC_OPTIONS variable. The content of this variable should be prefixed with + or = sign. + means that the content of the variable should be added to the default IGC options already passsed by the ISPCRT, while = tells the ISPCRT to replace the default options with the content of the environment variable.
• ISPCRT_GPU_DRIVER - if more than one supported GPU is present in the system, they may be managed by several GPU drivers. The user can select the GPU driver to be used by the ISPCRT using ISPCRT_GPU_DRIVER variable. It should be set to a corresponding driver number as enumerated by the Level Zero runtime. For example, in a system with two GPU drivers present, the variable can be set to 0 or 1.
• ISPCRT_GPU_DEVICE - if more than one supported GPU is present in the system, the user can select the GPU device to be used by the ISPCRT using ISPCRT_GPU_DEVICE variable. It should be set to a number of a device as enumerated by the Level Zero runtime. For example, in a system with two GPUs present, the variable can be set to 0 or 1.
• ISPCRT_MAX_KERNEL_LAUNCHES - there is a limit of the maximum number of enqueued kernel launches in a given task queue. If the limit is reached, sync() method needs to be called to submit the queue for execution. The limit is currently set to 100000, but can be lowered (for example for testing) using this environmental variable. Please note that the limit cannot be set to more than 100000. If a greater value is provided, the ISPCRT will set the limit to the default value and display a warning message.
• ISPCRT_VERBOSE - when defined as 1 enables verbose output.
• ISPCRT_MEM_POOL - when defined as 1 enables usage of memory pool for memory view allocations that created with appropriate shared memory allocation hints.
• ISCPRT_MEM_POOL_MIN_CHUNK_POW2 - provide the power of 2 for minimal chunk size that can be allocated without rounding up to the nearest power of 2.
• ISCPRT_MEM_POOL_MAX_CHUNK_POW2 - provide the power of 2 for maximal memory allocation that can fit into the memory pool.

Also you can use ISPCRTModuleOptions structure to pass specific options to GPU module. Currently we support only one setting - stackSize which determines the stack size in VC backend. The default value is 8192.

Performance Guide for GPU Programming

There are several rules for GPU programming which can bring you better performance.

Reduce register pressure

The first guidance is to reduce number of local variables. All variables are stored in GPU registers, and in the case when number of variables exceeds the number of registers, time-costly register spill occurs.

For example, Intel(R) Gen9 register file size is 128x8x32bit. Each 32-bit varying value takes 8x32bit in SIMD-8, and 16x32bit in SIMD-16.

To reduce number of local variables you can follow these simple rules:

• Use uniform instead of varyings wherever it is possible. This practice is good for both CPU and GPU but on GPU it is essential.

  // Good example
  for (uniform int j = 0; j < 3; j++) {
      do_something();
  }
  

  // Bad example
  for (int j = 0; j < 3; j++) {
      do_something();
  }
  

• Avoid nested code with a lot of local variables. It is more effective to split kernel into stages with separate variable scopes.

• Avoid returning complex structures from functions. Instead of operation that may need work on structure copy, consider to use reference or pointer. We're working to make such optimization automatically for future release:

  // Instead of this:
  struct ExampleStructure {
    //...
  }
  
  ExampleStructure createExampleStructure() {
    ExampleStructure retVal;
    //... initialize
    return retVal;
  }
  
  int test() {
    ExampleStructure s;
    s = createExampleStructure();
  }
  

  // Consider using pointer:
  struct ExampleStructure {
    //...
  }
  
  void initExampleStructure(ExampleStructure* init) {
    //... initialize
  }
  
  int test() {
    ExampleStructure s;
    initExampleStructure( &s );
  }
  

• Avoid recursion.

• Use SIMD-8 where it is impossible to fit in the available register number. If you see the warning message below during runtime, consider compiling your code for SIMD-8 target (--target=gen9-x8).

  Spill memory used = 32 bytes for kernel kernel_name___vyi
  

Code Branching

The second set of rules is related to code branching.

• Use select instead of branching:

  if (x > 0)
    a = x;
  else
    a = 7;
  

  // May be implemented without branch:
  a = (x > 0)? x : 7;
  

  When using select, try to simplify it as much as possible:

  // Not optimized version:
  varying int K;
  uniform bool Constant;
  ...
  return bConstant == true ? inParam[0] : InParam[K];
  

  // Optimized version
  return InParam[bConstant == true ? 0 : K];
  

• Keep branches as small as possible. Common operations should be moved outside the branch. In case when large code branches are necessary, consider changing your algorithm to group data processed by one task to follow the same path in the branch.

  // Both branches execute memory access to 'array'. In the case of split branch between
  // different lanes, two memory access instructions would be executed.
  if (x > 0)
    a = array[x];
  else
    a = array[0];
  

  // Instead move common part outside of the branch:
  int i;
  if (x > 0)
    i = x;
  else
    i = 0;
  a = array[i];
  

  Similar situation with loops:

  // Good example
  uniform int j;
  foreach (i = 0 ... WIDTH) {
    p->output[i + WIDTH * taskIndex] = 0;
    int temp = p->output[i + WIDTH * taskIndex];
    for (j = 0; j < DEPTH; j++) {
      temp += N;
      temp += M;
    }
    p->output[i + WIDTH * taskIndex] = temp;
  }
  

  // Bad example
  foreach (i = 0 ... WIDTH) {
    p->output[i + WIDTH * taskIndex] = 0;
    for (int j = 0; j < DEPTH; j++) {
      p->output[i + WIDTH * taskIndex] += N;
      p->output[i + WIDTH * taskIndex] += M;
    }
  }
  

Memory Operations

Remember that memory operations on GPU are expensive. We do not support dynamic memory allocations in kernel code for GPU so use fixed-size buffers preallocated by the host.

We have several memory optimizations for GPU like gather/scatter coalescing. However current implementation covers only limited number of cases and we expect to improve it for the next release.

Tools for Performance Analysis

To analyze performance of your program on Intel GPU we recommend the following tools:

• GTPin dynamic binary instrumentation command line framework for profiling a code running on Xe Execution Units.
• Profiling Tools Interfaces for GPU a bunch of useful tracing and instrumentation tools including ze_tracer that allows to analyze performance of Level Zero calls which is the base of ISPC Runtime.
• Intel(R) VTune Profiler a performance analysis tool for different hardware targets (CPU, GPU, FPGA) and OS platforms (Linux, Windows etc.).

Note, that most of these tools report SIMD width for ISPC kernels as 1. However, it actually means that ISPC kernel may have "any" SIMD width. VC backend can optimize some instructions to wider SIMD width than was requested by ISPC --target option.

How to Get an Assembly File from SPIR-V?

Use ocloc tool installed as part of intel-ocloc package:

// Create binary first
ocloc compile -file file.spv -spirv_input -options "-vc-codegen" -device <name>

// Then disassemble it
ocloc disasm -file file_Gen9core.bin -device <name> -dump <FOLDER_TO_DUMP>

You will get .asm files for each kernel in <FOLDER_TO_DUMP>.

To get more information from VC backend like vISA files, options used etc, try to set one of IGC configuration flags. For example to enable IGC shader dumps in shell:

export IGC_ShaderDumpEnable=1

How to Debug on GPU?

To debug your application, you can use oneAPI Debugger as described here: Get Started with GDB* for oneAPI on Linux* OS Host. Debugger support is quite limited at this time but you can set breakpoints in kernel code, do step-by-step execution and print variables.