You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'll investigate if I have free time. A LOT OF CODE COPYING from the Qwen2.5-vl model seems to be necessary due to the complexity of the visual & textual encoder, maybe we shall copy these things or import them from the llama.cpp main repo and integrate better with them?
Yes this kind of things have been discussed before, it seems that including llama.cpp as a submodule in the project would be the way to do this, or maybe including this project into llama.cpp.
Yes this kind of things have been discussed before, it seems that including llama.cpp as a submodule in the project would be the way to do this, or maybe including this project into llama.cpp.
Yes this kind of things have been discussed before, it seems that including llama.cpp as a submodule in the project would be the way to do this, or maybe including this project into llama.cpp.
I didn't do the implementation though, I only have a minimal understanding of how it works, mostly using the existing clip api. ngxson is the one you want.
having said that, a submodule is often very messy as it becomes hard to track changes to files. I would recommend extracting out the files you need and modifying them instead.
a submodule is often very messy as it becomes hard to track changes to files. I would recommend extracting out the files you need and modifying them instead.
Doesn't that make tracking upstream changes even harder?
Doesn't that make tracking upstream changes even harder?
In some ways yes, but the idea is that only the necessary code for the vision/text encoders is ported out (in like one or two files) and handled independently - it essentially becomes part of sd.cpp's code base and frozen in it's current functional state for the purposes of qwen image.
I think that the overlap between the rest of llama.cpp's code and stable-diffusion.cpp's code is sufficiently small that you don't really want to grab all the other stuff just for the purposes of a single function or two. But that is just my 2c.
I've done similar things before. If the overlap is small enough and you sometimes want to modify the copied code and your modification is not general enough to make a PR for the upstream, then this approach is much better than submodule. Sometimes modification is unavoidable if you want to remove codes that you don't need, but is considered a major functionality in the upstream repo. If you don't need to update too frequently, it's actually quite manageable
When attemping to fix an issue with qwen2.5-vl in llama.cpp, I debugging through a lot of codes and becomes a little bit familiar with them, so I would like to have a try.
However, I have zero knowledge about stable-diffusion.cpp's codebase. Anyone can give me some guidance? Let's get things work first, then decide the best way (copy code? submodules? other ways?) later. Looking at the actual code may help us decide.
Activity
Seas0 commentedon Aug 9, 2025
I'll investigate if I have free time. A LOT OF CODE COPYING from the
Qwen2.5-vlmodel seems to be necessary due to the complexity of the visual & textual encoder, maybe we shall copy these things or import them from thellama.cppmain repo and integrate better with them?stduhpf commentedon Aug 9, 2025
Yes this kind of things have been discussed before, it seems that including llama.cpp as a submodule in the project would be the way to do this, or maybe including this project into llama.cpp.
Seas0 commentedon Aug 9, 2025
Any reference for discussion context?
stduhpf commentedon Aug 9, 2025
#653
henk717 commentedon Aug 13, 2025
@LostRuins considering you have experience in integrating them might be userful to chime in.
LostRuins commentedon Aug 13, 2025
I didn't do the implementation though, I only have a minimal understanding of how it works, mostly using the existing clip api. ngxson is the one you want.
having said that, a submodule is often very messy as it becomes hard to track changes to files. I would recommend extracting out the files you need and modifying them instead.
stduhpf commentedon Aug 13, 2025
Doesn't that make tracking upstream changes even harder?
LostRuins commentedon Aug 13, 2025
In some ways yes, but the idea is that only the necessary code for the vision/text encoders is ported out (in like one or two files) and handled independently - it essentially becomes part of sd.cpp's code base and frozen in it's current functional state for the purposes of qwen image.
I think that the overlap between the rest of llama.cpp's code and stable-diffusion.cpp's code is sufficiently small that you don't really want to grab all the other stuff just for the purposes of a single function or two. But that is just my 2c.
rujialiu commentedon Aug 13, 2025
I've done similar things before. If the overlap is small enough and you sometimes want to modify the copied code and your modification is not general enough to make a PR for the upstream, then this approach is much better than submodule. Sometimes modification is unavoidable if you want to remove codes that you don't need, but is considered a major functionality in the upstream repo. If you don't need to update too frequently, it's actually quite manageable
triple-mu commentedon Aug 21, 2025
same request!
rujialiu commentedon Aug 22, 2025
When attemping to fix an issue with qwen2.5-vl in llama.cpp, I debugging through a lot of codes and becomes a little bit familiar with them, so I would like to have a try.
However, I have zero knowledge about
stable-diffusion.cpp's codebase. Anyone can give me some guidance? Let's get things work first, then decide the best way (copy code? submodules? other ways?) later. Looking at the actual code may help us decide.leejet commentedon Aug 29, 2025
Once this PR #778 that adds Wan support is merged, I will try to add support for Qwen Image.
LostRuins commentedon Sep 13, 2025
Sounds exciting. Qwen image edit is basically SOTA for image editing.
leejet commentedon Sep 22, 2025
Support for Qwen image has been added #851.