Skip to content

[FEA]: Support a MetaTensor that does not hold data for debug purposes. #56

Open
Open
[FEA]: Support a MetaTensor that does not hold data for debug purposes.#56
Labels
feature requeststatus: waiting-for-feedbackMaintainer is blocker and waiting for user feedback

Description

@Da1sypetals
Da1sypetals
opened on Jan 13, 2026

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request?

High

Please provide a clear description of problem this feature solves

Is it possible to support a MetaTensor that does not hold data for debug purposes?

Feature Description

We can utilize this to run the frontmost part of cuTile (python->hir->ir->TileIR) and either dump IR from each level for inspection or build tools like transpiler without having to create real data on GPU and even calling tileiras.

This comes from my personal experiments; (1) I have to develop on machines without Blackwell GPUs; (2) I have tried to implement a LSP server and cuTile -> numpy transpiler for debug purposes, but found it hard to catch up to the rapid development of cuTile mainstream.

Describe your ideal solution

If possible, I would like to discuss what the interface looks like and/or implement this.

Describe any alternatives you have considered

No response

Additional context

No response

Contributing Guidelines

  • I agree to follow cuTile Python's contributing guidelines
  • I have searched the open feature requests and have found no duplicates for this feature request

Activity

  1. Da1sypetals
    added
    status: needs-triageNew issue, not yet reviewed or categorized
    feature request
    on Jan 13, 2026

  2. gbonik commented on Jan 13, 2026

    @gbonik
    gbonik
    on Jan 13, 2026
    Collaborator

    Hi @Da1sypetals, thanks for you question. I believe it should be possible to implement a fake __cuda_array_interface__ to make this work (see https://numba.readthedocs.io/en/stable/cuda/cuda_array_interface.html ).

    Here is an example that Works On My Machine (TM):

    import cuda.tile as ct
from cuda.tile._cext import default_tile_context
from cuda.tile._compile import _get_final_ir

class FakeTensor:
    @property
    def __cuda_array_interface__(self):
        return dict(
            typestr="f4",  # float32
            shape=(100, 200),
            strides=None,  # C-contiguous format
            data=(0, False),
        )

@ct.kernel
def kernel(x):
    t = ct.gather(x, (0, 0))
    ct.scatter(x, (1, 1), t)

x = FakeTensor()
print(_get_final_ir(kernel._pyfunc, (x,), default_tile_context))

    This still requires building the C++ extension, unfortunately. Please let us know if there are any gaps that we could address.
    As for the rapid development, we hope the big disruptive changes will slow down soon, once we land certain features like lambdas etc.

  3. Da1sypetals commented on Jan 14, 2026

    @Da1sypetals
    Da1sypetals
    on Jan 14, 2026
    ContributorAuthor

    Hi @Da1sypetals, thanks for you question. I believe it should be possible to implement a fake __cuda_array_interface__ to make this work (see https://numba.readthedocs.io/en/stable/cuda/cuda_array_interface.html ).

    Here is an example that Works On My Machine (TM):

    import cuda.tile as ct
from cuda.tile._cext import default_tile_context
from cuda.tile._compile import _get_final_ir

class FakeTensor:
    @property
    def __cuda_array_interface__(self):
        return dict(
            typestr="f4",  # float32
            shape=(100, 200),
            strides=None,  # C-contiguous format
            data=(0, False),
        )

@ct.kernel
def kernel(x):
    t = ct.gather(x, (0, 0))
    ct.scatter(x, (1, 1), t)

x = FakeTensor()
print(_get_final_ir(kernel._pyfunc, (x,), default_tile_context))

    This still requires building the C++ extension, unfortunately. Please let us know if there are any gaps that we could address. As for the rapid development, we hope the big disruptive changes will slow down soon, once we land certain features like lambdas etc.

    1. I have already tried __cuda_array_interface__, but it comes with limitations (as of my knowledge), the primary one is it does not seem to support bfloat16 or lower precision
    2. Personally I would desire a solution without any C++ code... I have tried to workaround this in my personal repos, and would like to know if it is possible for the cuTile mainstream to include something like this. (line number may be wrong because this diff is based on a legacy commit)
    diff --git a/src/cuda/tile/_ir/typing_support.py b/src/cuda/tile/_ir/typing_support.py
index 305b2c5..0de71b6 100644
--- a/src/cuda/tile/_ir/typing_support.py
+++ b/src/cuda/tile/_ir/typing_support.py
@@ -194,6 +194,12 @@ def typeof_pyval(val, kernel_arg: bool = False) -> Type:
         return TypeTy(val)
     if isinstance(val, Enum):
         return EnumTy(type(val))
+    
+    if hasattr(val, '__mock_tensor_metadata__'):
+        try:
+            return _type_handler['__mock_tensor_metadata__'](val)
+        except KeyError:
+            pass
 
     if hasattr(val, '__cuda_array_interface__'):
         try:
@@ -375,3 +381,56 @@ def from_cuda_array_interface(x: Any) -> ArrayTy:
                    base_ptr_div_by=arr_special.base_ptr_div_by,
                    stride_div_by=arr_special.stride_div_by,
                    shape_div_by=arr_special.shape_div_by)
+
+
+
+from cuda.tile._datatype import _enum_to_dtype, DTypeEnum
+
+
+dtype_map = {
+    "bool": DTypeEnum.B1,
+    "uint8": DTypeEnum.U8,
+    "uint16": DTypeEnum.U16,
+    "uint32": DTypeEnum.U32,
+    "uint64": DTypeEnum.U64,
+    "int8": DTypeEnum.I8,
+    "int16": DTypeEnum.I16,
+    "int32": DTypeEnum.I32,
+    "int64": DTypeEnum.I64,
+    "float32": DTypeEnum.F32,
+    "float64": DTypeEnum.F64,
+    "float16": DTypeEnum.F16,
+    "bfloat16": DTypeEnum.BF,
+    "tfloat32": DTypeEnum.TF32,
+    "float8_e4m3fn": DTypeEnum.F8E4M3FN,
+    "float8_e5m2": DTypeEnum.F8E5M2,
+}
+
+@register_type_handler('__mock_tensor_metadata__')
+def from_mock_tensor_metadata(x: Any) -> ArrayTy:
+    desc = x.__mock_tensor_metadata__
+    # Assume dynamic shape
+    shape = TupleTy(SizeTy(None) for i in desc['shape'])
+    dtype = _enum_to_dtype[dtype_map[desc['dtype_str']]]
+    if dtype.bitwidth % BYTE_BITWIDTH != 0:
+        raise ValueError("Only byte-aligned types should be supported by __mock_tensor_metadata__")
+
+    dtype_bytewidth = dtype.bitwidth // BYTE_BITWIDTH
+    byte_strides = desc.get('strides', None)
+    elem_strides = _compute_elem_strides(desc['shape'], dtype_bytewidth, byte_strides)
+
+    assert len(elem_strides) == len(shape)
+
+    arr_special = ArraySpecialization(
+            desc['data'][0], dtype.bitwidth, tuple(desc['shape']), elem_strides)
+
+    # Assume dynamic strides except for the ones marked static
+    strides = TupleTy(SizeTy(i) if static else SizeTy(None)
+                      for i, static in
+                      zip(elem_strides, arr_special.stride_is_static))
+
+    return ArrayTy(dtype, shape=shape, strides=strides,
+                   elements_disjoint=arr_special.elements_disjoint,
+                   base_ptr_div_by=arr_special.base_ptr_div_by,
+                   stride_div_by=arr_special.stride_div_by,
+                   shape_div_by=arr_special.shape_div_by)

    (Added: _cext.py to mock the missing lib, this part is vibe coded)

    from cuda.tile._context import TileContextConfig


def launch(stream, grid, kernel, kernel_args, /):
    """Mock launch function - does not actually launch kernel"""
    raise RuntimeError(
        "Cannot launch kernel: CUDA driver not available. This mock module only supports IR generation."
    )


class TileDispatcher:
    """Mock TileDispatcher class"""

    def __new__(cls, *args, **kwargs):
        # 正确处理 __new__ 方法
        instance = object.__new__(cls)
        return instance

    def __init__(self, arg_constant_flags, compile_func):
        self._arg_constant_flags = arg_constant_flags
        self._compile_func = compile_func


class TileContext:
    """Mock TileContext class"""

    def __init__(self, config=None):
        if config is None:
            from cuda.tile._context import init_context_config_from_env

            config = init_context_config_from_env()
        self._config = config

    @property
    def config(self):
        return self._config


# Create default tile context
default_tile_context = TileContext()


def get_compute_capability():
    """Mock get_compute_capability - returns a default value"""
    # Return Hopper (9.0) as default
    # Users can modify this if needed
    return (9, 0)


class ArraySpecialization:
    """Mock ArraySpecialization class for array type analysis"""

    def __init__(self, base_ptr, dtype_bitwidth, shape, strides):
        self.base_ptr = base_ptr
        self.dtype_bitwidth = dtype_bitwidth
        self.shape = shape
        self.strides = strides

        # 简单的启发式规则：假设所有 stride 都是静态的
        self.stride_is_static = tuple(True for _ in strides)

        # 假设元素不重叠
        self.elements_disjoint = True

        # 对齐信息：假设基本对齐
        self.base_ptr_div_by = dtype_bitwidth // 8  # 按字节对齐
        self.stride_div_by = tuple(1 for _ in strides)
        self.shape_div_by = tuple(1 for _ in shape)
    1. Currently get_final_ir depend on TileContext (defined in C++ code?) which seems not referenced based on a simple code search. Is it possible to get rid of it, or does it serve some undocumented purposes? Could you please explain? Thanks a lot. If the _get_final_ir function can be independent on C++ code, can we split the _get_final_ir function into 2 halves (first pure python half, then python + C++ half) and expose a public interface for the first half?
    2. Sorry for my bad English, if there is anything not clear please ask for my clarification.

  4. Da1sypetals commented on Jan 14, 2026

    @Da1sypetals
    Da1sypetals
    on Jan 14, 2026
    ContributorAuthor

    Since we only need tensor metadata to compile the cuTile kernel (without tensor data), is it possible to first construct such metadata into a dedicated dataclass, compile the kernel, and launch the kernel with real data? This could make the FakeTensor workflow not an exception, but a "part of the main compiler workflow".

  5. gbonik commented on Jan 14, 2026

    @gbonik
    gbonik
    on Jan 14, 2026
    Collaborator

    Since we only need tensor metadata to compile the cuTile kernel (without tensor data), is it possible to first construct such metadata into a dedicated dataclass, compile the kernel, and launch the kernel with real data? This could make the FakeTensor workflow not an exception, but a "part of the main compiler workflow".

    Yes, something along those lines is on our TODO list. Right now there is duplicated logic between the C++ dispatcher code (classify_arg() and friends) and the Python side (typeof_pyval etc.), which is very fragile. It would be great if the C++ dispatcher decided what the types are and passed them to the compilation pipeline. This would also be very useful for supporting ahead-of-time compilation. However, I don't know when we can get to it. I'll try to prioritize this when I have some time free from scheduled feature work.

    I have already tried cuda_array_interface, but it comes with limitations (as of my knowledge), the primary one is it does not seem to support bfloat16

    I believe bfloat16 is supported (typestring V2, see parse_typestr() in tile_kernel.cpp), but yes, I agree it is a limited interface, e.g. it is not possible to express float8 types.

    Currently get_final_ir depend on TileContext (defined in C++ code?) which seems not referenced based on a simple code search. Is it possible to get rid of it, or does it serve some undocumented purposes? Could you please explain? Thanks a lot. If the _get_final_ir function can be independent on C++ code, can we split the _get_final_ir function into 2 halves (first pure python half, then python + C++ half) and expose a public interface for the first half?

    The goal of TileContext is to store all global state, like the flags set by environment variables etc. This is still work in progress, but it is already used in hir2ir in order to check the debug printing settings (via ir_ctx.tile_ctx). The idea is that if another library needs to depend on cutile, it can create an isolated context that is unaffected by environment variables etc. We need to access the TileContext object in the C++ dispatcher, so we can't move it to Python for performance reasons. Is there a specific reason you want to avoid building the C++ extension? It has very minimal dependencies and should be relatively simple to build. If you need to build it on a Mac, or don't want to install the CUDA toolkit, we could in theory add some conditional compilation to build a "dummy" extension with stubbed out functionality (this is also on my personal TODO list).

    As an alternative , I think maybe we could change _get_final_ir to accept a TileContextConfig instead of the full TileContext object, which is a pure python class.

    I have tried to workaround this in my personal repos, and would like to know if it is possible for the cuTile mainstream to include something like this.

    I'd be in favor of adding a more explicit compilation API, e.g. going the "part of the main compiler workflow" route. Also, once your LSP server reaches a usable state, would you be interested in upstreaming it? This could help us deal with the internal API breakages, as we could test it in the CI. It doesn't have to be part of the main package, as we can put it into the experimental package or similar.

  6. Da1sypetals commented on Jan 15, 2026

    @Da1sypetals
    Da1sypetals
    on Jan 15, 2026
    ContributorAuthor

    Also, once your LSP server reaches a usable state

    It is already usable in my sense, and it is here, but it is based on commit 7a8625 (before the introduction of hir), and it is probably buggy. Its type information comes from the ir after typecheck but before any optimization. There is also a to-numpy transpiler in transpile branch, which is also based on a legacy commit.

    going the "part of the main compiler workflow" route

    My naive idea can be illustrated in this flowchart
    Image

    Is there a specific reason you want to avoid building the C++ extension? 

    The main reason is to avoid building a C++ extension just for a debug tool where we actually don't need to use anything inside it, to Make it possible to do this level of debugging on computers without NVIDIA hardware or even Mac.

    As an alternative , I think maybe we could change _get_final_ir to accept a TileContextConfig instead of the full TileContext object, which is a pure python class.

    This would be great! As far as I know, syntax checking and type checking is done inside _get_final_ir, which means this approach makes it possible to raise any possible errors when converting a python program into a valid cuTile-python program without C++ libraries. I think this is pretty much everything needed to develop debug tools.

  7. gbonik commented on Jan 15, 2026

    @gbonik
    gbonik
    on Jan 15, 2026
    Collaborator

    The main reason is to avoid building a C++ extension just for a debug tool where we actually don't need to use anything inside it, to Make it possible to do this level of debugging on computers without NVIDIA hardware or even Mac.

    You shouldn't need NVIDIA hardware to compile the C++ extension, a normal C++ compiler should be sufficient (we support GCC and MSVC, hopefully clang should work as well). I agree that it would be nice to drive the compilation pipeline with pure Python, but I'm afraid this may not be viable in the long term. The C++ extension needs to have the logic similar to typeof_pyval() to be able to quickly classify the arguments, and thus we'd need to duplicate the logic in Python (which we currently do, but ideally we need to get rid of this duplication).

    In any case, I have staged a change to _get_final_ir that should hopefully address your use case:

    def _get_final_ir(pyfunc,
                  args: Sequence[ir.KernelArgument],
                  config: TileContextConfig) -> ir.Function:
    1. Now we take a TileContextConfig instead of TileContext. It can be obtained by calling init_context_config_from_env().
    2. Instead of args, we take a sequence of KernelArgument structs, which is a renamed ir.Argument:
    @dataclass
class KernelArgument:
    type: Type
    is_const: bool = False
    const_value: Any = None

    I hope this should allow you to drive the compilation pipeline without creating concrete array objects. The KernelArgument class should work similarly to a "meta tensor", with the downside of having to also call get_constant_annotations() etc. yourself.

    The change is stacked on top of a couple other in-progress PRs, so it should hopefully get into public main sometime next week.

  8. Da1sypetals commented on Jan 16, 2026

    @Da1sypetals
    Da1sypetals
    on Jan 16, 2026
    ContributorAuthor

    I have synced the cutile to numpy transpiler to newest main branch (it's here, and it's definitely buggy), as a proof of concept that everything it need is the output of _get_final_ir

  9. gbonik commented on Jan 30, 2026

    @gbonik
    gbonik
    on Jan 30, 2026
    Collaborator

    The _get_final_ir() change has been pushed to public main: 48ab3e3 . I hope this is sufficient to unblock you for now.

  10. haijieg
    added
    status: waiting-for-feedbackMaintainer is blocker and waiting for user feedback
    and removed
    status: needs-triageNew issue, not yet reviewed or categorized
    on Feb 2, 2026

  11. Da1sypetals commented on Feb 3, 2026

    @Da1sypetals
    Da1sypetals
    on Feb 3, 2026
    ContributorAuthor

    You shouldn't need NVIDIA hardware to compile the C++ extension, a normal C++ compiler should be sufficient (we support GCC and MSVC, hopefully clang should work as well).

    @gbonik May I ask if there are any instructions to compile the C++ extension without an NVIDIA GPU? When I run pip install -e . currently, it still requires CTK to be installed:

    error: subprocess-exited-with-error
  
  × Building editable for cuda-tile (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [40 lines of output]
      running editable_wheel
      creating /tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info
      writing /tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info/requires.txt
      writing top-level names to /tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info/top_level.txt
      writing manifest file '/tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info/SOURCES.txt'
      reading manifest file '/tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info/SOURCES.txt'
      adding license file 'LICENSES/Apache-2.0.txt'
      writing manifest file '/tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile.egg-info/SOURCES.txt'
      creating '/tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile-9.9.99.dist-info'
      creating /tmp/pip-wheel-tt5sdcxh/.tmp-g4l145e6/cuda_tile-9.9.99.dist-info/WHEEL
      running build_py
      running build_ext
      cmake -B /root/dev/cutile-dev/build /root/dev/cutile-dev -DDLPACK_PATH= -DCMAKE_BUILD_TYPE=Release -DCMAKE_POLICY_VERSION_MINIMUM=3.5
      -- The C compiler identification is GNU 12.3.1
      -- The CXX compiler identification is GNU 12.3.1
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found Python: /root/miniconda3/bin/python3.1 (found version "3.13.5") found components: Interpreter Development Development.Module Development.Embed
      -- Looking for CUDA Toolkit
      -- No cuda.h found in /usr/local/cuda/include
      CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find CUDAToolkit (missing: CUDAToolkit_INCLUDE_DIR)
      Call Stack (most recent call first):
        /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
        cmake/FindCUDAToolkit.cmake:32 (find_package_handle_standard_args)
        CMakeLists.txt:69 (find_package)
      
      
      -- Configuring incomplete, errors occurred!
      error: command '/usr/bin/cmake' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building editable for cuda-tile
Failed to build cuda-tile
ERROR: Failed to build installable wheels for some pyproject.toml based projects (cuda-tile)

  12. gbonik commented on Feb 4, 2026

    @gbonik
    gbonik
    on Feb 4, 2026
    Collaborator

    There is currently no way to build without CUDA SDK, but we can add an option to CMake and some #ifdefs to the code. I can try to sneak in the necessary change within the next couple of weeks.

  13. Da1sypetals commented on Feb 6, 2026

    @Da1sypetals
    Da1sypetals
    on Feb 6, 2026
    ContributorAuthor

    Also, once your LSP server reaches a usable state, would you be interested in upstreaming it? This could help us deal with the internal API breakages, as we could test it in the CI. It doesn't have to be part of the main package, as we can put it into the experimental package or similar.

    @gbonik The LSP server I implemented is here. It now works with cuda-tile v1.1.0 (or the current main branch) and I deem it usable though it is still buggy:

    Image Image
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requeststatus: waiting-for-feedbackMaintainer is blocker and waiting for user feedback

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions