qant_native_computing_toolkit_generic.h File Reference

#include <stddef.h>
#include <stdint.h>
#include <dlpack/dlpack.h>

Include dependency graph for qant_native_computing_toolkit_generic.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Typedefs
typedef struct QantMemoryHandle	QantMemoryHandle
	Opaque handle to a block of device memory sized to hold a matrix.
typedef enum QantDataRole	QantDataRole
	Role of a block of device memory.

Enumerations
enum	QantDataRole { Weights , Features }
	Role of a block of device memory. More...

Functions
int	init_npu (const uint32_t id)
	Initialises the Native Processing Unit.
int	release_npu (const uint32_t id)
	Releases an NPU and frees associated memory.
QantMemoryHandle *	npu_allocate (const uint32_t npu_id, const size_t rows, const size_t cols, const QantDataRole data_role)
	Allocates a block of device memory sized to hold a (rows, cols) matrix.
int	npu_free (QantMemoryHandle *handle)
	Frees a previously allocated block of device memory.
QantMemoryHandle *	npu_upload (const uint32_t npu_id, DLManagedTensorVersioned const *data, const QantDataRole data_role)
	Allocates a block of device memory and uploads a matrix to it.
int	npu_upload_into (QantMemoryHandle const *handle, DLManagedTensorVersioned const *data)
	Uploads a matrix to a previously allocated block of device memory.
DLManagedTensorVersioned *	npu_download (QantMemoryHandle const *handle)
	Downloads the matrix currently stored in a block of device memory.

Typedef Documentation

QantDataRole

typedef enum QantDataRole QantDataRole

Role of a block of device memory.

QantMemoryHandle

typedef struct QantMemoryHandle QantMemoryHandle

Opaque handle to a block of device memory sized to hold a matrix.

Instances are created by npu_allocate() and must be released exactly once with npu_free(). Using a handle after it has been freed is undefined behavior.

Enumeration Type Documentation

QantDataRole

enum QantDataRole

Role of a block of device memory.

Enumerator
Weights
Features

Function Documentation

init_npu()

int init_npu

(

const uint32_t

id

)

Initialises the Native Processing Unit.

Optional, not necessary for computation.

NPUs are automatically initialised on the first operation that uses the specific device. Within one process, concurrent operation calls on a single NPU are possible, this creates a first-come first-serve queue. A process has an exclusive lock on each NPU it initialised, NPUs can be released explicitly to free devices for use by a different process, see release_npu.

Parameters

id	The identifier of the NPU to be initialised.

Returns

0 if successful, or a negative error code.

npu_allocate()

QantMemoryHandle * npu_allocate	(	const uint32_t	npu_id,
		const size_t	rows,
		const size_t	cols,
		const QantDataRole	data_role )

Allocates a block of device memory sized to hold a (rows, cols) matrix.

Parameters

npu_id	The identifier of the NPU on which the memory is allocated.
rows	Number of rows of the matrix the memory is sized for.
cols	Number of columns of the matrix the memory is sized for.
data_role	Role of the memory block (Weights or Features).

Returns

An owning pointer to the allocated handle, or NULL on error. The caller must release it with npu_free().

npu_download()

DLManagedTensorVersioned * npu_download

(

QantMemoryHandle const *

handle

)

Downloads the matrix currently stored in a block of device memory.

Parameters

handle

The handle to download from, as returned by npu_allocate().

Returns

A 2D tensor with shape (rows, cols) and data type bfloat16, or NULL on error.

npu_free()

int npu_free

(

QantMemoryHandle *

handle

)

Frees a previously allocated block of device memory.

Parameters

handle

The handle to free, as returned by npu_allocate(). Must not be used again after this call.

Returns

0 if successful, or a negative error code.

npu_upload()

QantMemoryHandle * npu_upload	(	const uint32_t	npu_id,
		DLManagedTensorVersioned const *	data,
		const QantDataRole	data_role )

Allocates a block of device memory and uploads a matrix to it.

Parameters

npu_id	The identifier of the NPU on which the memory is allocated.
data	A 2D tensor with shape (rows, cols) and data type bfloat16.
data_role	Role of the memory block (Weights or Features).

Returns

An owning pointer to the allocated handle, or NULL on error. The caller must release it with npu_free().

npu_upload_into()

int npu_upload_into	(	QantMemoryHandle const *	handle,
		DLManagedTensorVersioned const *	data )

Uploads a matrix to a previously allocated block of device memory.

Parameters

handle	The handle to upload to, as returned by npu_allocate() or npu_upload().
data	A 2D tensor with shape (rows, cols) matching the handle. Must be data type bfloat16.

Returns

0 if successful, or a negative error code.

release_npu()

int release_npu

(

const uint32_t

id

)

Releases an NPU and frees associated memory.

Only relevant if multiple processes want to access the same device.

Parameters

id	The identifier of the NPU to be released.

Returns

0 if successful, or a negative error code.