Properties
Basic¤
ndim
property
¤
ndim: int
Returns the number of dimensions in the tensor.
t = Tensor([[1, 2], [3, 4]])
print(t.ndim)
2
numel
¤
numel() -> sint
Returns the total number of elements in the tensor.
t = Tensor([[[1, 2], [3, 4]], [[5, 6], [7, 8]]])
print(t.numel())
8
Source code in tinygrad/tensor.py
3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 |
|
element_size
¤
element_size() -> int
Returns the size in bytes of an individual element in the tensor.
t = Tensor([5], dtype=dtypes.int16)
print(t.element_size())
2
Source code in tinygrad/tensor.py
3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 |
|
nbytes
¤
nbytes() -> int
Returns the total number of bytes of all elements in the tensor.
t = Tensor([8, 9], dtype=dtypes.float)
print(t.nbytes())
8
Source code in tinygrad/tensor.py
3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 |
|
is_floating_point
¤
is_floating_point() -> bool
Returns True
if the tensor contains floating point types, i.e. is one of dtype.float64
, dtype.float32
,
dtype.float16
, dtype.bfloat16
.
t = Tensor([8, 9], dtype=dtypes.float32)
print(t.is_floating_point())
True
Source code in tinygrad/tensor.py
3937 3938 3939 3940 3941 3942 3943 3944 3945 3946 3947 |
|
size
¤
Return the size of the tensor. If dim
is specified, return the length along dimension dim
. Otherwise return the shape of the tensor.
t = Tensor([[4, 5, 6], [7, 8, 9]])
print(t.size())
(2, 3)
print(t.size(dim=1))
3
Source code in tinygrad/tensor.py
3949 3950 3951 3952 3953 3954 3955 3956 3957 3958 3959 3960 3961 |
|
Data Access¤
data
¤
data() -> memoryview
Returns the data of this tensor as a memoryview.
t = Tensor([1, 2, 3, 4])
print(np.frombuffer(t.data(), dtype=np.int32))
[1 2 3 4]
Source code in tinygrad/tensor.py
294 295 296 297 298 299 300 301 302 303 304 305 |
|
item
¤
item() -> ConstType
Returns the value of this tensor as a standard Python number.
t = Tensor(42)
print(t.item())
42
Source code in tinygrad/tensor.py
307 308 309 310 311 312 313 314 315 316 317 |
|
tolist
¤
Returns the value of this tensor as a nested list. Returns single value for const tensor.
t = Tensor([1, 2, 3, 4])
print(t.tolist())
[1, 2, 3, 4]
t = Tensor(5)
print(t.tolist())
5
Source code in tinygrad/tensor.py
321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 |
|
numpy
¤
numpy() -> 'np.ndarray'
Returns the value of this tensor as a numpy.ndarray
.
t = Tensor([1, 2, 3, 4])
print(repr(t.numpy()))
array([1, 2, 3, 4], dtype=int32)
Source code in tinygrad/tensor.py
337 338 339 340 341 342 343 344 345 346 347 348 349 350 |
|
tinygrad ops¤
schedule_with_vars
¤
Creates the schedule needed to realize these Tensor(s), with Variables.
Note
A Tensor can only be scheduled once.
Source code in tinygrad/tensor.py
227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 |
|
schedule
¤
schedule(*lst: Tensor) -> list[ScheduleItem]
Creates the schedule needed to realize these Tensor(s).
Source code in tinygrad/tensor.py
248 249 250 251 252 |
|
realize
¤
Triggers the computation needed to create these Tensor(s).
Source code in tinygrad/tensor.py
254 255 256 257 |
|
replace
¤
Replaces the data of this tensor with the data of another tensor. Only the shape of the tensors must match.
Source code in tinygrad/tensor.py
259 260 261 262 263 264 265 266 |
|
assign
¤
assign(x) -> Tensor
Source code in tinygrad/tensor.py
268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 |
|
detach
¤
detach() -> Tensor
Returns a new tensor with the same data as this tensor, but detached from the autograd graph.
Source code in tinygrad/tensor.py
285 286 287 288 289 |
|
clone
¤
clone() -> Tensor
Creates a clone of this tensor allocating a separate buffer for the data.
Source code in tinygrad/tensor.py
352 353 354 355 356 357 358 |
|
to
¤
Moves the tensor to the given device.
Source code in tinygrad/tensor.py
360 361 362 363 364 365 366 367 368 369 |
|
to_
¤
Moves the tensor to the given device in place.
Source code in tinygrad/tensor.py
371 372 373 374 375 376 377 |
|
shard
¤
Shards the tensor across the given devices. Optionally specify which axis to shard on.
t = Tensor.empty(2, 4)
print(t.shard((t.device, t.device), axis=1).lazydata)
UOp(Ops.MULTI, dtypes.float, arg=(1, (True, True)), src=(
UOp(Ops.CONTIGUOUS, dtypes.float, arg=None, src=(
UOp(Ops.COPY, dtypes.float, arg=False, src=(
x2:=UOp(Ops.DEVICE, dtypes.void, arg='CPU', src=()),
UOp(Ops.SHRINK, dtypes.float, arg=((0, 2), (0, 2)), src=(
x4:=UOp(Ops.RESHAPE, dtypes.float, arg=(2, 4), src=(
UOp(Ops.BUFFER, dtypes.float, arg=8, src=(
x2,
UOp(Ops.UNIQUE, dtypes.void, arg=1171, src=()),)),)),)),)),)),
UOp(Ops.CONTIGUOUS, dtypes.float, arg=None, src=(
UOp(Ops.COPY, dtypes.float, arg=False, src=(
x2,
UOp(Ops.SHRINK, dtypes.float, arg=((0, 2), (2, 4)), src=(
x4,)),)),)),))
Source code in tinygrad/tensor.py
379 380 381 382 383 384 385 386 387 388 389 390 391 |
|
shard_
¤
Shards the tensor across the given devices in place.
Source code in tinygrad/tensor.py
393 394 395 396 397 |
|
contiguous
¤
contiguous() -> Tensor
Returns a contiguous tensor.
Source code in tinygrad/tensor.py
2707 2708 2709 2710 2711 |
|
contiguous_backward
¤
contiguous_backward() -> Tensor
Inserts a contiguous operation in the backward pass.
Source code in tinygrad/tensor.py
2713 2714 2715 2716 2717 |
|
Gradient¤
gradient
¤
gradient(
*targets: Tensor,
gradient: Tensor | None = None,
materialize_grads=False
) -> list[Tensor]
Compute the gradient of the targets with respect to self.
x = Tensor.eye(3)
y = Tensor([[2.0,0,-2.0]])
z = y.matmul(x).sum()
dx, dy = z.gradient(x, y)
print(dx.tolist()) # dz/dx
print(dy.tolist()) # dz/dy
[[2.0, 2.0, 2.0], [0.0, 0.0, 0.0], [-2.0, -2.0, -2.0]]
[[1.0, 1.0, 1.0]]
Source code in tinygrad/tensor.py
870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 |
|
backward
¤
Propagates the gradient of a tensor backwards through the computation graph. If the 'gradient' argument is not provided, the tensor must be a scalar, and the gradient is implicitly set to 1.0.
t = Tensor([1.0, 2.0, 3.0, 4.0], requires_grad=True)
t.sum().backward()
print(t.grad.numpy())
[1. 1. 1. 1.]
Source code in tinygrad/tensor.py
899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 |
|