local LLM -- option to select *.gguf models by myself

[weekendkoder]

Hello.

I am happy with this extension. It is the only one, that works without any dependencies and therefore it is instantly usable within VScode. Thank you for bringing us such an extension. I have some requests:

It would be nice, if we could have a option to select the *.gguf model file by our own. I can download it somewhere (f.e. huggingFace) and inside the options of FireCoder i can select the file. I recognized that the models are saved inside C:\ drive. inside the USERS subfolder: ".firecoder". If its possible it would also be very handy, if we could have an option where to store downloaded models.

In my case, the drive C: is reserved for the OS and stays clean. I put heavy stuff onto the D: partition. (Windows OS). I managed to use the MkLink command, to alias this folder to drive D:.

[gespispace]

Hello, thank you.
I will do it in next few days.

I am going to add a new option in configuration, something like "firecoder.homedir", where you will be able to overwrite default folder for saving models and server.

[weekendkoder]

Thats great news. :-)
Would it also mean that we can download different *.gguf models, than provided by FireCoder ? Should not be a big deal, or ?

[gespispace]

Yes, FireCoder had this possibility, but I've removed it.
I hadn't seen any reasons for it, because FireCoder uses the most powerful open-source model (DeepSeek-coder).
Of course in size between 1b-7b.

[weekendkoder]

Okay.

Did not know about the past implemented feature. I guess, the idea is, when users install "FireCoder", they dont need to do anything, because FireCoder download the models in background. What is quite nice. My suggestion to this Background action would be:

• prompt a message (maybe a popup) that FireCoder want to "start downloading" the model / models. The window can have buttons like: "start now" / "Later" / "Resume". With a small note that "FireCoder" cant work without any model. So the user know it need some time when the download gets finished. At least the user can decide if it should download now or later, due to his fast or low internet connection (mobile). I hope my concern is understandable.

• predefined models: I dont know if these models are downloaded from huggingface or from the FireCoder Servers. I have 2x
  concerns with predefined models: 1.) if they get downloaded from any specific link, it will work until the link is available (doesnt matter if its from hugginface or FireCoder). If the link is not available anymore or it was changed, it will not work. 2.) if you decide which model to download (the most powerful one), it can be that on another day there will be a updated model on hugginface. Which could mean, you need to update FireCoder and provide it on the marketplace again, in a short time. On the other hand, i don´t know what speaks against a 2nd option where the user just has an opportunity to decide by his own.

I tried to put manually a different *.gguf model inside the model-folder and i also rename it to "model-base-small.gguf" but FireCoder did not accept the file and start do download its own again.

Of course this suggestion makes mostly sense, if you as a dev are doing everything alone, which can cost time. Or if you barely have time to update some stuff. The suggestion doesnt make sense if you will update this extension every time, for min. 5 years long. I believe the predefined Solution & a given ability to the user to make his own decisions can coexist. I hope my concern is understandable.

[gespispace]

Hello. I've added a feature to define the homedir for storing models and server files. It will be available in v0.0.28 under firecoder.homedir.

if the link is not available anymore or it was changed, it will not work

Yes, Firecoder uses huggingface. You can see that I am checking SHA file before use a model, this is a reason why you couldn't put your model.

But about custom models, I am not sure that it is easy now.

1. Each model has a chat template or code insertion template, and I hardcoded these templates in the code. As the first step, I can move it to configuration.
   Anyway, I will do it for the chat template, because llama.cpp has started supporting the chat template a few days ago.
2. I hope users will be able to use chat models after I move to llama chat template, but about code insertion template, I think I will add it, later. But right now, it doesn't look like a critical feature for me.

I will keep this issue open until custom models are fully supported.

Thank you very much.

[weekendkoder]

Hi.

Yes, Firecoder uses huggingface. You can see that I am checking SHA file before use a model, this is a reason why you couldn't put your model.

Ah okay. Thats why. Good to know.

As for the rest, i cant estimate if its a lot of work to a dev or a quick task. I just have a little expirience with "lm-Studio" & "Jan Ai". Inside these Apps, it seems to be an easy task to change / download different models from hugginface or elsewhere. This is why i asked for it. Becasue it looked not too complicated, to me.

If its possible to do so with FireCoder in future, i will be satisfied. If its predefined and supported by FireCoder i am already happy, as long as there will be regular updates (with new models). Thank you, for your detailed information's & support.