Model selection UI broken

[aa956]

Version 0.1.8

Erroneously selected small during onboarding, looks there's no way to change back to large:

Models are present:

$ ollama list
NAME                       ID              SIZE      MODIFIED       
granite3.3:8b              fd429f23b909    4.9 GB    2 minutes ago     
nomic-embed-text:latest    0a109f422b47    274 MB    9 minutes ago     
granite3.2:2b              9d79a41f2f75    1.5 GB    9 minutes ago     
granite3.2:8b              9bcb3335083f    4.9 GB    13 minutes ago    
mistral-small3.1:latest    b9aaf0c2586a    15 GB     33 hours ago      
gemma3:27b-it-qat          29eb0b9aeda3    18 GB     2 days ago        
qwen3:30b                  2ee832bc15b5    18 GB     2 days ago        
qwen2.5-coder:1.5b-base    02e0f2817a89    986 MB    2 days ago        
llama3.1:8b                46e0c10c039e    4.9 GB    2 days ago        

[aa956]

Editing in config files does not work too.

As soon as vscode is started and granite-code opened it resets back to 3.2:2b.

Looks like this model is hardcoded somewhere.

$ killall code
$ grep -r granite3 ~/.granite-code/*
/home/user/.granite-code/index/globalContext.json:      "chat": "granite3.2:2b",
/home/user/.granite-code/index/globalContext.json:      "edit": "granite3.2:2b",
/home/user/.granite-code/index/globalContext.json:      "apply": "granite3.2:2b",
/home/user/.granite-code/index/globalContext.json:      "autocomplete": "granite3.2:2b",
$ sed -i -e 's/granite3\.2\:2b/granite3\.3\:8b/g' ~/.granite-code/index/globalContext.json
$ grep -r granite3 ~/.granite-code/*
/home/user/.granite-code/index/globalContext.json:      "chat": "granite3.3:8b",
/home/user/.granite-code/index/globalContext.json:      "edit": "granite3.3:8b",
/home/user/.granite-code/index/globalContext.json:      "apply": "granite3.3:8b",
/home/user/.granite-code/index/globalContext.json:      "autocomplete": "granite3.3:8b",
$ code
$ grep -r granite3 ~/.granite-code/*
/home/user/.granite-code/index/globalContext.json:      "chat": "granite3.2:2b",
/home/user/.granite-code/index/globalContext.json:      "edit": "granite3.2:2b",
/home/user/.granite-code/index/globalContext.json:      "apply": "granite3.2:2b",
/home/user/.granite-code/index/globalContext.json:      "autocomplete": "granite3.2:2b",
$ 

[aa956]

Installed continue.dev and now I see that settings are in config.yaml. But granite-code config.yaml is almost empty?

name: Local Assistant
version: 1.0.0
schema: v1
models: []
context:
  - provider: code
  - provider: docs
  - provider: diff
  - provider: terminal
  - provider: problems
  - provider: folder
  - provider: codebase

vs continue-dev's one:

name: Local Assistant
version: 1.0.0
schema: v1
models:
  - name: Qwen3 8B
    provider: ollama
    model: qwen3:8b
    roles:
      - chat
      - edit
      - apply
  - name: Qwen2.5 Coder 3B
    provider: ollama
    model: qwen2.5-coder:3b
    roles:
      - autocomplete
  - name: Nomic Embed
    provider: ollama
    model: nomic-embed-text:latest
    roles:
      - embed
  - name: Autodetect
    provider: ollama
    model: AUTODETECT
context:
  - provider: code
  - provider: docs
  - provider: diff
  - provider: terminal
  - provider: problems
  - provider: folder
  - provider: codebase

[fbricon]

So, you can change the selected model in the VS Code settings (granite.localModelSize) :

Alternatively, you can reopen the wizard via the "Granite.Code: Setup Granite as code assistant" command.

And yes, not being able to change the model in the wizard immediately after the 1st setup is pretty counterintuitive.

[aa956]

Thank you!

So just a user error, have not found the correct place in UI to set up the models.

[fbricon]

Reopening as we can certainly improve the UX here.

[aa956]

Ok, clear. Then it would be also nice to have:

• Ability to select older or newer model

For example granite3.2:8b is currently default model but user may want to select either granite3.3:8b or granite3-dense:8b for some reason, be it better results from older/newer model for their specific use case or trying to reproduce old results in the future.

[allanday]

We discussed this issue briefly yesterday and agreed that we need to decide on a plan for #117 first. If we decide to hide the small models in the UI, this issue will be obsolete.

[owtaylor]

We've decided for #117 to go ahead and remove the model selection in the wizard, and we'll remove the small/large choice from the settings as well, but we're not completely set here.

Use cases

Reasons that a user might want to customize the models:

• Using a different Granite autocomplete model. Right now, there are multiple reasonable options:

  granite-3.3-8b-base: this is the best Granite model for autocomplete. If someone has a fast GPU, they probably want to use this.
  granite-3.3-2b-base: worse, but faster. Best model for mid-end Macs
  granite-3.3-8b-instruct: Quality is similar to 2b-base, speed similar to 8b-base. If you have a GPU with limited memory and only have room for one Granite, this can make sense.

  In the future, it's also possible there will be multiple reasonable choices for chat models.

• Using Ollama on a different port

• Using hosted models. If you have access to an instance of VLLM running granite3.3:8b, use that instead of Ollama

• Using old models - I don't care about this one. The Granite models have a track record for improving over time, and I don't want users torturing themselves trying to figure out whether granite3.2:8b is better than granite3.3:2b - because model outputs are inherently random (even at temperature 0), you just can't tell based on a small number of prompts. If someone really wants to investigate, they can always configure the models themselves. (See below)

• Using third party models. Not our emphasis, but users (or at least Granite.Code developers...) will want to compare.

Proposed way it looks

I think a basic principle should be that selecting models and customizing models feels like an extension of the upstream UI rather than something alien to it.

To change your autocomplete model, you edit ~/.granite-code/models/autocomplete.yaml to change

  name: Granite.Code chat model
  version: 1.0.0
  schema: v1
  models:
-    - uses: granite.code/autocomplete@default
+    - uses: granite.code/granite-3.3:8b-base

To use a different Ollama port, you edit ~/.granite-code/models/{autocomplete,chat,embed}.yaml and add:

  name: Granite.Code chat model
  version: 1.0.0
  schema: v1
  models:
    - uses: granite.code/autocomplete@default
    - override:
       apiBase: ollama.local:11434

(OR we add a setting for this, OR we simply honor the OLLAMA_HOST environment variable - but this would be the general mechanism for overrides)

To use a hosted model, you replace ~/.granite-code/models/{autocomplete,chat,embed}.yaml with your own content.

To stop using a hosted model, you delete those files, and they will be recreated with the default content.

Notes:

• This requires a pretty simple code change to provide our own RegistryClient when unrolling the yaml file
• An alternative would be to uses: ./default-models/granite-autocomplete.yaml which avoids the code change and lets people actually open that file and see what is in there. Might be better.
• When you are using a hosted model, you want to completely replace not override: to repoint, otherwise changes to the default model will cause user's configurations to break.

[owtaylor]

Work is covered by #135, so closing this - thanks for reporting it, it's incredibly useful to have actual experience from users trying to find their way around!