the same way you'd change the temperature, top_k etc.. it is activated with a small value by default

Thank you. So in which order are sampling methods like min_p, top_k and top_p executed? Can I change that order?

take a look at these lines here..

Lines 1160 to 1168 in 4026166

    
           self._ctx.sample_top_k(candidates=self._candidates, k=top_k, min_keep=1) 
        
           self._ctx.sample_tail_free(candidates=self._candidates, z=tfs_z, min_keep=1) 
        
           self._ctx.sample_typical( 
        
               candidates=self._candidates, p=typical_p, min_keep=1 
        
           ) 
        
           self._ctx.sample_top_p(candidates=self._candidates, p=top_p, min_keep=1) 
        
           self._ctx.sample_min_p(candidates=self._candidates, p=min_p, min_keep=1) 
        
           self._ctx.sample_temp(candidates=self._candidates, temp=temp) 
        
           id = self._ctx.sample_token(candidates=self._candidates)

The order is hardcoded at the moment.. but in my opinion you don't really want or need mixing min_p with top_p and top_k samplers, it's probably better to just disable top_p (top_p=1) etc and just use min_p and temp.
But if you wanna change the order and play with it, just edit that file.. it's simple.

	self._ctx.sample_top_k(candidates=self._candidates, k=top_k, min_keep=1)
	self._ctx.sample_tail_free(candidates=self._candidates, z=tfs_z, min_keep=1)
	self._ctx.sample_typical(
	candidates=self._candidates, p=typical_p, min_keep=1
	)
	self._ctx.sample_top_p(candidates=self._candidates, p=top_p, min_keep=1)
	self._ctx.sample_min_p(candidates=self._candidates, p=min_p, min_keep=1)
	self._ctx.sample_temp(candidates=self._candidates, temp=temp)
	id = self._ctx.sample_token(candidates=self._candidates)

Add min p arg to server #911

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions