Quality of generated code problem

[uzhao]

App Version

3.11.17

API Provider

VS Code LM API

Model Used

Claude 3.7 Sonnet

Actual vs. Expected Behavior

Actually I don't think this is an real issue or bug. Anyway, I watched a video (https://www.bilibili.com/video/BV15VR3YKErt?t=252.0) compared LLMs, and tried to reproduce its result (Mars Mission in https://github.com/KCORES/kcores-llm-arena). Later I found that compare to ask LLM directly, Roo-Code generated result is much worse. The result come from LLM chat work well just like the video one, while Roo-Code generated do not move at all and no spaceship launched. I've tried 3.7 and 3.5, VS Code API and my own token. Roo-Code result never use FuncAnimation or animation from matplotlib.animation which is a key point. This is just a single example but time to time I feel this problem in some other cases.

Detailed Steps to Reproduce

1.Create a new task
2.Ask "Generate code for an animated 3d plot of a launch from earth landing on mars and then back to earth at the next launch window"

Relevant API Request Output

Additional Context

No response

[dosubot]

Hello @uzhao! 👋 I'm Dosu, your friendly bot here to help with bugs, answer questions, and guide you in becoming a contributor. I'm here to assist you while we wait for a human maintainer. Let's tackle this together! 🚀

It seems that the issue you're experiencing with Roo-Code is related to its inability to generate code using matplotlib.animation.FuncAnimation or similar animation libraries, which is crucial for creating animated plots like the Mars mission example you mentioned. Currently, there is no indication in the Roo-Code repository or documentation that it supports generating animations using these libraries ^[1]. This might explain why the generated code does not meet your expectations compared to directly using LLMs like Claude 3.7 Sonnet.

If this functionality is critical for your tasks, you might consider manually integrating the necessary animation libraries into the generated code or exploring other tools that better support this feature. Additionally, you could check for any open issues or feature requests in the Roo-Code repository that might address this limitation.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[uzhao]

Hello @uzhao! 👋 I'm Dosu, your friendly bot here to help with bugs, answer questions, and guide you in becoming a contributor. I'm here to assist you while we wait for a human maintainer. Let's tackle this together! 🚀

It seems that the issue you're experiencing with Roo-Code is related to its inability to generate code using matplotlib.animation.FuncAnimation or similar animation libraries, which is crucial for creating animated plots like the Mars mission example you mentioned. Currently, there is no indication in the Roo-Code repository or documentation that it supports generating animations using these libraries [1]. This might explain why the generated code does not meet your expectations compared to directly using LLMs like Claude 3.7 Sonnet.

If this functionality is critical for your tasks, you might consider manually integrating the necessary animation libraries into the generated code or exploring other tools that better support this feature. Additionally, you could check for any open issues or feature requests in the Roo-Code repository that might address this limitation.

To continue talking to Dosu, mention @dosu.

Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other

I may not explain this clearly enough. I do not expect RooCode generate an animation directly, but RooCode generate a script which using matplotlib.animation to generate animations. If a native LLM can do this well, I don't understand why the same LLM can't do this through RooCode.

[Xytronix]

App Version

3.11.17

API Provider

VS Code LM API

Model Used

Claude 3.7 Sonnet

Actual vs. Expected Behavior

Actually I don't think this is an real issue or bug. Anyway, I watched a video (https://www.bilibili.com/video/BV15VR3YKErt?t=252.0) compared LLMs, and tried to reproduce its result (Mars Mission in https://github.com/KCORES/kcores-llm-arena). Later I found that compare to ask LLM directly, Roo-Code generated result is much worse. The result come from LLM chat work well just like the video one, while Roo-Code generated do not move at all and no spaceship launched. I've tried 3.7 and 3.5, VS Code API and my own token. Roo-Code result never use FuncAnimation or animation from matplotlib.animation which is a key point. This is just a single example but time to time I feel this problem in some other cases.

Detailed Steps to Reproduce

1.Create a new task 2.Ask "Generate code for an animated 3d plot of a launch from earth landing on mars and then back to earth at the next launch window"

Relevant API Request Output

Additional Context

No response

Do note that you have no option to tweak temperature with the VSCODE LLM API and that the maximum allowed context of the models is about 13.3k token size.

RooCode can't do anything else than provide different prompts. Please use LLMS outside of said API if you are not satisfied.

[hannesrudolph]

Roo code is only as strong as the model you select and the prompt which you provide that model.