You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For this opening statement to be true, it would need to include windows-based systems, especially old gaming rigs.
Forget expensive NVIDIA GPUs, unify your existing devices into one powerful GPU: iPhone, iPad, Android, Mac, Linux, pretty much any device!
At the very least, a thorough guide on setting up tinygrad via WSL/WSL2 would be appreciated, because this is your only documentation:
Example Usage on Multiple MacOS Devices
Reacted by Alex Cheema, CFM_Barcelona, Yazan Maarouf, Erick D. P., Antoine Viallon, taoli, 73649, Xeddius, Troy Gunawardene, Marcus Llewellyn and 1 more
I'd like to look into this. Adjacently, llamafiles might be worth looking into as they are binaries able to run on multiple desktop OSes without any configuration. Though I'm not sure about Android or IOS support.
I'd like to look into this. Adjacently, llamafiles might be worth looking into as they are binaries able to run on multiple desktop OSes without any configuration. Though I'm not sure about Android or IOS support.
@bayedieng I'd recommend looking at https://github.com/abetlen/llama-cpp-python -- it should hopefully be low level enough to do what we need to do. Also, I'd recommend looking at #139 for a minimal implementation of an inference engine that doesn't require explicitly defining every model -- it's a general solution.
It would seem that the LLAMA CPP API is too high level to perform sharded inference as it doesn't provide access to individual layers. In order to do so, one would have to use the GGML bindings directly to create a suitable inference engine compatible with Exo.
It was stated that it would be nice to have a similar API as the Pytorch engine where a base model and reused everywhere as provided with AutoModelForCausalLM however, GGML does not provide a simmilar API for inference, so each model would have to likely be implmented the same way that tinygrad and mlx does it.
@AlexCheema please let me know if you'd like to go forward and build the inference engine with ggml bindings.
Not familiar with them but they also likely just use the underlying GGML API as well, considering LLama CPP does inference end-to-end. LLama CPP also has a distributed inference example using GGML:
I've been caught up recently in a whole bunch of work recently and have had little time to work on the LLama support. I wouldn't would to hold the PR hostage if someone's capable of completing it go ahead!
Not familiar with them but they also likely just use the underlying GGML API as well, considering LLama CPP does inference end-to-end. LLama CPP also has a distributed inference example using GGML:
Thanks for the info. I looked through the RPC example and found out that it uses the GGML RPC backend. With other backends, it is scheduled by GGML, which requires all nodes to be GGML nodes.
However, in Exo, each node is independent, and some nodes may not use the GGML backend. So, I think we can just use the GGML backend API and then wrap it up for Exo.
Activity
danny-avila commentedon Aug 25, 2024
Thanks @AlexCheema
Myself and many others likely only have windows systems, and llama.cpp is practically the only option.
MLX is macOS and tinygrad:
For this opening statement to be true, it would need to include windows-based systems, especially old gaming rigs.
At the very least, a thorough guide on setting up tinygrad via WSL/WSL2 would be appreciated, because this is your only documentation:
bayedieng commentedon Aug 27, 2024
I'd like to look into this. Adjacently, llamafiles might be worth looking into as they are binaries able to run on multiple desktop OSes without any configuration. Though I'm not sure about Android or IOS support.
AlexCheema commentedon Aug 27, 2024
Go for it!
AlexCheema commentedon Aug 27, 2024
@bayedieng I'd recommend looking at https://github.com/abetlen/llama-cpp-python -- it should hopefully be low level enough to do what we need to do. Also, I'd recommend looking at #139 for a minimal implementation of an inference engine that doesn't require explicitly defining every model -- it's a general solution.
bayedieng commentedon Aug 27, 2024
Thanks for the suggestion. Yeah I had already seen the python bindings and went ahead and began a draft PR.
thegodone commentedon Sep 14, 2024
I wonder if WebGPU can be plugged on top of Llama.cpp via this https://github.com/AnswerDotAI/gpu.cpp wrapper ?
bayedieng commentedon Sep 21, 2024
It would seem that the LLAMA CPP API is too high level to perform sharded inference as it doesn't provide access to individual layers. In order to do so, one would have to use the GGML bindings directly to create a suitable inference engine compatible with Exo.
It was stated that it would be nice to have a similar API as the Pytorch engine where a base model and reused everywhere as provided with
AutoModelForCausalLMhowever, GGML does not provide a simmilar API for inference, so each model would have to likely be implmented the same way that tinygrad and mlx does it.@AlexCheema please let me know if you'd like to go forward and build the inference engine with ggml bindings.
danny-avila commentedon Sep 21, 2024
Heads up, LocalAI now has distributed inference:
https://localai.io/features/distribute/
bayedieng commentedon Sep 21, 2024
Not familiar with them but they also likely just use the underlying GGML API as well, considering LLama CPP does inference end-to-end. LLama CPP also has a distributed inference example using GGML:
https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc
bayedieng commentedon Oct 25, 2024
I've been caught up recently in a whole bunch of work recently and have had little time to work on the LLama support. I wouldn't would to hold the PR hostage if someone's capable of completing it go ahead!
SureD commentedon Dec 8, 2024
agree
Thanks for the info. I looked through the RPC example and found out that it uses the GGML RPC backend. With other backends, it is scheduled by GGML, which requires all nodes to be GGML nodes.
However, in Exo, each node is independent, and some nodes may not use the GGML backend. So, I think we can just use the GGML backend API and then wrap it up for Exo.
Add LlamaCppInferenceEngine
Xeddius commentedon Feb 18, 2025
For automatic detection you can look into Koboldcpp's implementation.
Leeaandrob commentedon Mar 22, 2025
Excited about it
Material-Scientist commentedon Apr 29, 2025
Looking forward
karsontsang23 commentedon Aug 8, 2025
@AlexCheem I'd recommend looking at https://github.com/ADT109119/llamacpp-distributed-inference -- it is a llamacpp-distributed-inference ,you need this to help you .
Evanev7 commentedon Dec 18, 2025
We're closing bounties, thank you to all who contributed, and apologies that we left this unmanaged for so long.