Local Models integration

Hey Devs,
let me start by saying that this programme is great. Well done on your work, and thanks for sharing it.

My question is: is there any plan to allow for the integration of local models?
Even a just section in the documentation would be great.

Have a good day
theWW

+1 to this question

It makes no sense to shovel money into some closed source while we have a powerful GPU that can run 13b Llama with no problem with some of the other open source projects.

I'd also be very eager to use local models with ChatDev, Llama based models show great promise

Local model use and perhaps as a more advanced feature, assign different models to different agents in the company - so could use a local python-optimized model for an engineer, and a llama2 model for the CEO, etc.

@j-loquat

@j-loquat I love that idea. That is a thing i was considering more and more. Ai becoming more and more like greek gods, each with its characther and function that complete each other.. it was the original vision of Altman too, kind of, but they lost their way

No need to have 1 "god AGI" (which can not be ran locally as it demands crazy hardware) if we can have 20 agents with 20 different local narrow AI models that can be loaded one after another.

Oh god, sorry Devs but this conversation is too interesting. You may need to turn notifications off XD

I was trained as an artist, and the first thing to know is that limitations are the generator of creativity. A big Ai with all the knowledge of the world may just become the most boring thing to touch the planet. And this may be controversial, but I think that bad qualities are needed too...everything has its meaning and use in order to create balance. Just my opinion

This has been referenced in #33

Install LocalAI - OpenAI compatible server.
Create new model config file named gpt-3.5-turbo-16k.yaml and set the model name to gpt-3.5-turbo-16k-0613.
Start LocalAI server locally and run:

OPENAI_API_BASE=http://127.0.0.1:8000/v1 OPENAI_API_KEY="dummy" python run.py --task "Snake game in pure html" --name "WebSnake"

The command above did not work in Anaconda Prompt, but this version did:

(chatdev_conda_env) C:\chatdev>set OPENAI_API_BASE=http://127.0.0.1:5001/v1

(chatdev_conda_env) C:\chatdev>set OPENAI_API_KEY=123456

(chatdev_conda_env) C:\chatdev>python run.py --task "Hello world in python" --name "HelloWorld"
**[Preprocessing]**

**ChatDev Starts** (20230913191808)

**Timestamp**: 20230913191808

**config_path**: C:\chatdev\CompanyConfig\Default\ChatChainConfig.json

**config_phase_path**: C:\chatdev\CompanyConfig\Default\PhaseConfig.json

**config_role_path**: C:\chatdev\CompanyConfig\Default\RoleConfig.json

**task_prompt**: Hello world in python

**project_name**: HelloWorld

**Log File**: C:\chatdev\WareHouse\HelloWorld_DefaultOrganization_20230913191808.log

**ChatDevConfig**:
 ChatEnvConfig.clear_structure: True
ChatEnvConfig.brainstorming: False


**ChatGPTConfig**:
 ChatGPTConfig(temperature=0.2, top_p=1.0, n=1, stream=False, stop=None, max_tokens=None, presence_penalty=0.0, frequency_penalty=0.0, logit_bias={}, user='')

I am having a problem using it with local api:

It looks like all that the API returns is 1 token:

Text-generation-webui side:

llm_load_print_meta: model size     = 13.02 B
llm_load_print_meta: general.name   = openassistant_llama2-13b-orca-8k-3319
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token  = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.12 MB
llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required  =  128.35 MB (+ 1600.00 MB per state)
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloading v cache to GPU
llm_load_tensors: offloading k cache to GPU
llm_load_tensors: offloaded 43/43 layers to GPU
llm_load_tensors: VRAM used: 11656 MB
...................................................................................................
llama_new_context_with_model: kv self size  = 1600.00 MB
llama_new_context_with_model: compute buffer total size =  191.47 MB
llama_new_context_with_model: VRAM scratch buffer: 190.00 MB
AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 |
2023-09-13 19:11:27 INFO:Loaded the model in 7.52 seconds.

Warning: $This model maximum context length is 2048 tokens. However, your messages resulted in over 498 tokens and max_tokens is 15937.

llama_print_timings:        load time =   955.37 ms
llama_print_timings:      sample time =     0.23 ms /     1 runs   (    0.23 ms per token,  4424.78 tokens per second)
llama_print_timings: prompt eval time =   955.31 ms /   498 tokens (    1.92 ms per token,   521.30 tokens per second)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =   957.88 ms
Output generated in 1.41 seconds (0.00 tokens/s, 0 tokens, context 498, seed 1828391196)
127.0.0.1 - - [13/Sep/2023 19:11:42] "POST /v1/chat/completions HTTP/1.1" 200 -
Warning: $This model maximum context length is 2048 tokens. However, your messages resulted in over 551 tokens and max_tokens is 15885.
Llama.generate: prefix-match hit

llama_print_timings:        load time =   955.37 ms
llama_print_timings:      sample time =     0.16 ms /     1 runs   (    0.16 ms per token,  6410.26 tokens per second)
llama_print_timings: prompt eval time =   835.88 ms /   489 tokens (    1.71 ms per token,   585.01 tokens per second)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =   836.73 ms
Output generated in 1.24 seconds (0.00 tokens/s, 0 tokens, context 551, seed 192786861)
127.0.0.1 - - [13/Sep/2023 19:11:46] "POST /v1/chat/completions HTTP/1.1" 200 -
Warning: $This model maximum context length is 2048 tokens. However, your messages resulted in over 521 tokens and max_tokens is 15907.
Llama.generate: prefix-match hit

llama_print_timings:        load time =   955.37 ms
llama_print_timings:      sample time =     0.13 ms /     1 runs   (    0.13 ms per token,  7633.59 tokens per second)
llama_print_timings: prompt eval time =   884.39 ms /   459 tokens (    1.93 ms per token,   519.00 tokens per second)
llama_print_timings:        eval time =     0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_print_timings:       total time =   885.63 ms
Output generated in 1.26 seconds (0.00 tokens/s, 0 tokens, context 521, seed 1288396660)
127.0.0.1 - - [13/Sep/2023 19:11:53] "POST /v1/chat/completions HTTP/1.1" 200 -
Warning: $This model maximum context length is 2048 tokens. However, your messages resulted in over 574 tokens and max_tokens is 15854.
Llama.generate: prefix-match hit

ChatDev side



Chief Executive Officer: **Chief Product Officer<->Chief Executive Officer on : DemandAnalysis, turn 0**

[ChatDev is a software company powered by multiple intelligent agents, such as chief executive officer, chief human resources officer, chief product officer, chief technology officer, etc, with a multi-agent organizational structure and the mission of "changing the digital world through programming".
You are Chief Product Officer. we are both working at ChatDev. We share a common interest in collaborating to successfully complete a task assigned by a new customer.
You are responsible for all product-related matters in ChatDev. Usually includes product design, product strategy, product vision, product innovation, project management and product marketing.
Here is a new customer's task: Hello world in python.
To complete the task, you must write a response that appropriately solves the requested instruction based on your expertise and customer's needs.]



**[OpenAI_Usage_Info Receive]**
prompt_tokens: 521
completion_tokens: 1
total_tokens: 522


**[OpenAI_Usage_Info Receive]**
prompt_tokens: 574
completion_tokens: 1
total_tokens: 575


Chief Product Officer: **Chief Product Officer<->Chief Executive Officer on : DemandAnalysis, turn 1**

[ChatDev is a software company powered by multiple intelligent agents, such as chief executive officer, chief human resources officer, chief product officer, chief technology officer, etc, with a multi-agent organizational structure and the mission of "changing the digital world through programming".
You are Chief Executive Officer. Now, we are both working at ChatDev and we share a common interest in collaborating to successfully complete a task assigned by a new customer.
Your main responsibilities include being an active decision-maker on users' demands and other key policy issues, leader, manager, and executor. Your decision-making role involves high-level decisions about policy and strategy; and your communicator role can involve speaking to the organization's management and employees.
Here is a new customer's task: Hello world in python.
To complete the task, I will give you one or more instructions, and you must help me to write a specific solution that appropriately solves the requested instruction based on your expertise and my needs.]



Chief Executive Officer: **Chief Product Officer<->Chief Executive Officer on : DemandAnalysis, turn 1**

[ChatDev is a software company powered by multiple intelligent agents, such as chief executive officer, chief human resources officer, chief product officer, chief technology officer, etc, with a multi-agent organizational structure and the mission of "changing the digital world through programming".
You are Chief Product Officer. we are both working at ChatDev. We share a common interest in collaborating to successfully complete a task assigned by a new customer.
You are responsible for all product-related matters in ChatDev. Usually includes product design, product strategy, product vision, product innovation, project management and product marketing.
Here is a new customer's task: Hello world in python.
To complete the task, you must write a response that appropriately solves the requested instruction based on your expertise and customer's needs.]



**[OpenAI_Usage_Info Receive]**
prompt_tokens: 544
completion_tokens: 1
total_tokens: 545

@andraz

Yeah, the command above was for macOS, no troubles with conda environment here.

@andraz, why don't you increase the context to 4K or 8K tokens? Based on your model name it support context up to 8K tokens.

Warning: $This model maximum context length is 2048 tokens. However, your messages resulted in over 521 tokens and max_tokens is 15907.

As for one token response I guess it's streaming feature, so you don't need to wait for a full response.

Hello there,
I am trying to use llama-2-7B version as described above.

I created a new yaml file with name gpt-3.5-turbo-16k.yaml and set the model name to gpt-3.5-turbo-16k-0613. Then on model used I downloaded and use one of the hugginface model library `llama-2*.bin' models.

I can successfully run it and receive answers to my questions as part of the returning object via curl but also says that
"usage":"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

When then I try to run chatdev on a simple task ie python run.py --task "Hello world in python" --name "HelloWorld", chatdev prints the start up prompts and then receives no objects from the local llm with continuous empty usage logs

...
Note that we must ONLY discuss the product modality and do not discuss anything else! Once we all have expressed our opinion(s) and agree with the results of the discussion unanimously, any of us must actively terminate the discussion by replying with only one line, which starts with a single word <INFO>, followed by our final product modality without any other words, e.g., "<INFO> PowerPoint".

**[OpenAI_Usage_Info Receive]**
prompt_tokens: 0
completion_tokens: 0
total_tokens: 0


**[OpenAI_Usage_Info Receive]**
prompt_tokens: 0
completion_tokens: 0
total_tokens: 0


**[OpenAI_Usage_Info Receive]**
prompt_tokens: 0
completion_tokens: 0
total_tokens: 0

After 3 retries it crashes with the following KeyError.

Traceback (most recent call last):
  File "/home/**/miniconda3/envs/chatdev/lib/python3.9/site-packages/tenacity/__init__.py", line 382, in __call__
    result = fn(*args, **kwargs)
  File "/media/**/4TB_DATA/git/ChatDev/camel/utils.py", line 145, in wrapper
    return func(self, *args, **kwargs)
  File "/media/**/4TB_DATA/git/ChatDev/camel/agents/chat_agent.py", line 200, in step
    response["id"],
KeyError: 'id'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/media/**/4TB_DATA/git/ChatDev/run.py", line 111, in <module>
    chat_chain.execute_chain()
  File "/media/**/4TB_DATA/git/ChatDev/chatdev/chat_chain.py", line 160, in execute_chain
    self.execute_step(phase_item)
  File "/media/**/4TB_DATA/git/ChatDev/chatdev/chat_chain.py", line 130, in execute_step
    self.chat_env = self.phases[phase].execute(self.chat_env,
  File "/media/**/4TB_DATA/git/ChatDev/chatdev/phase.py", line 292, in execute
    self.chatting(chat_env=chat_env,
  File "/media/**/4TB_DATA/git/ChatDev/chatdev/utils.py", line 77, in wrapper
    return func(*args, **kwargs)
  File "/media/**/4TB_DATA/git/ChatDev/chatdev/phase.py", line 131, in chatting
    assistant_response, user_response = role_play_session.step(input_user_msg, chat_turn_limit == 1)
  File "/media/**/4TB_DATA/git/ChatDev/camel/agents/role_playing.py", line 242, in step
    assistant_response = self.assistant_agent.step(user_msg_rst)
  File "/home/**/miniconda3/envs/chatdev/lib/python3.9/site-packages/tenacity/__init__.py", line 289, in wrapped_f
    return self(f, *args, **kw)
  File "/home/**/miniconda3/envs/chatdev/lib/python3.9/site-packages/tenacity/__init__.py", line 379, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/**/miniconda3/envs/chatdev/lib/python3.9/site-packages/tenacity/__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7ff680181ac0 state=finished raised KeyError>]

I have already exported OPENAI_API_BASE and OPENAI_API_KEY to the localhost otherwise it crashed.

What can I do to successfully use the local LLM?

Thanks for any help and sorry if this is the wrong place to ask it!

@starkdmi

@starkdmi

Create new model config file named gpt-3.5-turbo-16k.yaml and set the model name to gpt-3.5-turbo-16k-0613.

What model are you using?

@starkdmi

Install LocalAI - OpenAI compatible server.

Create new model config file named gpt-3.5-turbo-16k.yaml and set the model name to gpt-3.5-turbo-16k-0613.

Hello @starkdmi , can you share the file gpt-3.5-turbo-16k.yaml?

使用Langchain-Chatchat这个项目，调用本地2000端口

我尝试了你的提议，端口上不出意外是2w,应该不是2k（可能是打错了），我用的也是chatglm3-6b-32k,知识库是BAAI/bge-large-zh,能跑，但奇怪的是响应很慢，不是没有响应，而是过好一会才响应，gpu的80g内存够的，最后花了93mins完成一款无法运行的“Snake game in pure html”

For me it wasn't OPENAI_API_BASE, but BASE_URL.
After setting this, everything works fine with LiteLLM + Ollama

FYI - its not OPEN_API_BASE. if using anaconda on windows you do SET BASE_URL="http://localhost:1234/v1" and then SET OPEN_API_KEY="not needed" . this is if you're using LMstudio. All working my end using Mistral instruct 7B.

If anyone has ollama integrated with this then please let me know. thanks a lot. happy coding.

how to go about using other services that like Together.ai offers an OpenAI compatible API, how to set host ?

If the API is OpenAI compatible you can point at the API endpoint using --api_base as with local models.

I'm just going to add a bit about how I got ChatDev running locally with LM Studio server for anyone searching. It was really easy if there would have been clear instructions but I had to read through all of the issues and tried to find stuff in the code to no avail.

Anyway. The basics:

Windows 10

Following the installation instructions from the readme for steps 1-3 (gitclone, conda, cd, install requirements)

On step 4 do this instead:

set OPENAI_API_BASE=http://localhost:1234/v1

And that's it (you'll need to start the LMS server and load a model), now you can just run ChatDev like you normally would but locally.

Trying to get this running on a Win10 machine.
I keep getting this error, like it needs a working API key of some sort:

.conda\envs\ChatDev_conda_env\lib\site-packages\openai_base_client.py", line 877, in _request
raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: abc123xyz. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

FYI - its not OPEN_API_BASE. if using anaconda on windows you do SET BASE_URL="http://localhost:1234/v1" and then SET OPEN_API_KEY="not needed" . this is if you're using LMstudio. All working my end using Mistral instruct 7B.

I tried with BASE_URL as well as OPEN_AI_BASE but getting an APIConnectionError

tenacity.RetryError: RetryError[<Future at 0x1cdcf0bce80 state=finished raised APIConnectionError>]

Can you help?

It's not BASE_URL, it's OPENAI_BASE_URL! But I also saw on begin that OPENAI_API_BASE is also used (maybe I'm wrong here but it works for me), so the correct command is OPENAI_API_BASE=http://localhost OPENAI_BASE_URL=http://localhost OPENAI_API_KEY="anything" python run.py --task "Snake game in pure html" --name "WebSnake"

hi all the contributors ,
I have found a github app that helps solving issues using AI and the approach to this is very interesting so i am sharing this , please anyone add this to this repo . : https://github.com/apps/sweep-ai
not sponsored or anything but i found it helpful so.

@acbp

@acbp has anyone wrote documentation specifically for things like Ollama (or LocalAI, Xorbit, or OpenLLM) to work with ChatDev? LiteLLM maybe?

If anyone has achieved or can suggest for this . In this we want to continuously index the updated code base in the HippoRAG index and query on the updated index and then make code changes and continuously do so. Here consider that we want to do this offline with ollama type models only and dont want to use OpenAI or Claude .

If anyone can suggest how can i do this ?

anyone had success with llama.cpp?

I got tenacity.RetryError: RetryError[<Future at 0x2595e8f1e10 state=finished raised NotFoundError>]

and

raise self._make_status_error_from_response(err.response) from None openai.NotFoundError: Error code: 404 - {'error': {'code': 404, 'message': 'File Not Found', 'type': 'not_found_error'}}

On Mac( m3) i used this with local install ( Anaconda)

BASE_URL=http://localhost:1234/v1 OPENAI_API_KEY=dummy python3 run.py --task "< task >" --name "<title>"

i've forked and made some change to the code. https://github.com/skypu/ChatDev
now it supports both openai-style-api and local ollam. all you need to do is set the env info

for openai-style-api:
$env:OPENAI_API_KEY="your_openai_api_key"
$env:BASE_URL="https://ark.cn-beijing.volces.com/api/v3"
$env:DEFAULT_AI_MODEL="deepseek-r1-250120"
$env:AI_PROVIDER="openai"

for local ollama:
$env:OPENAI_API_KEY="not needed, but keep it"
$env:BASE_URL="http://localhost:11434/api/generate"
$env:DEFAULT_AI_MODEL="gemma3:latest"
$env:AI_PROVIDER="ollama"

Navigation Menu

Repository navigation

Local Models integration #27

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Local Models integration #27

Description

Activity

andraz commented on Sep 8, 2023

thedualspace commented on Sep 8, 2023

j-loquat commented on Sep 8, 2023

TheWhiteWord commented on Sep 8, 2023

andraz commented on Sep 8, 2023

TheWhiteWord commented on Sep 8, 2023

hemangjoshi37a commented on Sep 9, 2023

starkdmi commented on Sep 11, 2023

andraz commented on Sep 13, 2023

starkdmi commented on Sep 13, 2023

xkaraman commented on Sep 20, 2023

GitSimply commented on Sep 20, 2023

jacktang commented on Sep 21, 2023

33 remaining items

BackMountainDevil commented on Dec 16, 2023

mroxso commented on Dec 26, 2023

evmond1 commented on Feb 13, 2024

hemangjoshi37a commented on Feb 20, 2024

akhil3417 commented on Feb 29, 2024

opencoca commented on Feb 29, 2024

resdig3 commented on Mar 1, 2024

vishusinghal commented on Apr 20, 2024

maramowicz commented on May 1, 2024

hemangjoshi37a commented on May 15, 2024

BradKML commented on Jul 24, 2024

hemangjoshi37a commented on Jul 24, 2024

hgftrdw45ud67is8o89 commented on Jul 26, 2024

syedshahzebhasnain commented on Jul 30, 2024

skypu commented on Mar 20, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions