server : fix handling of the ignore_eos flag #14710
Conversation
| for (llama_token i = 0; i < llama_vocab_n_tokens(vocab); i++) { | ||
| if (llama_vocab_is_eog(vocab, i)) { | ||
| //SRV_DBG("%s: added %s logit bias = %f\n", __func__, common_token_to_piece(ctx, i).c_str(), -INFINITY); | ||
| params.sampling.logit_bias.push_back({i, -INFINITY}); | ||
| } | ||
| } |
There was a problem hiding this comment.
If this is done for every token during generation, I suspect it is going to have a significant performance impact.
There was a problem hiding this comment.
It's done once per completion request, at the beginning, upon processing the input json parameters.
There was a problem hiding this comment.
If performance is a concern we could maybe provide a list of EoG tokens to iterate over instead of iterating over all tokens and checking whether each one is EoG. Although I think iterating over all tokens once per request is going to be negligible vs. iterating over all tokens once per generated token as is being done for sampling.
There was a problem hiding this comment.
In my system this takes about 0.3 ms for a 150k vocab model, so I suppose it is not that bad.
There was a problem hiding this comment.
I fixed this anyway: #14721
fix #14695 (comment)
slot_params.ignore_eosignore_eosis passed in the request, we ignore all EOG tokens by adding -INF logit bias to the sampler