Skip to content

#1 Updating to latest repo version with LLM Monitoring metrics #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
May 9, 2024
Merged

#1 Updating to latest repo version with LLM Monitoring metrics #1

merged 16 commits into from
May 9, 2024

Conversation

juanroesel
Copy link

@juanroesel juanroesel commented May 7, 2024 •
edited
Loading

Closes ZenHubHQ/devops#2205

It also adds includes the following:

  • A new metric kv_cache_usage_ratio, which measures how much KV cache is being used.
  • Synced commits with the parent repo (not relevant for the PR review).
  • A Llama 3 8B model baked into the image.

New image us.gcr.io/zenhub-ops/llama_cpp_python-llama3_8b_f16:v0.3.1 was successfully deployed into staging.

abetlen and others added 15 commits May 2, 2024 11:32
@abetlen
feat: Add llama-3-vision-alpha chat format
31b1d95
@abetlen
feat: Update llama.cpp
d75dea1
@abetlen
chore: Bump version
2117122
@dthuerck
fix(server): Propagate flash_attn to model load. (abetlen#1424)
2138561
@abetlen
feat(server): Remove temperature bounds checks for server. Closes abe…
0a454be 
…tlen#1384
@abetlen
fix: Use memmove to copy str_value kv_override. Closes abetlen#1417
9f7a855
@abetlen
Merge branch 'main' of github.com:abetlen/llama_cpp_python into main
f9b7221
@jeffrey-fong @abetlen
feat: Implement streaming for Functionary v2 + Bug fixes (abetlen#1419)
1f56c64 
* set up streaming for v2

* assert v2 streaming, fix tool_call vs function_call

* fix streaming with tool_choice/function_call

* make functions return 1 function call only when 'auto'

* fix

---------

Co-authored-by: Andrei <abetlen@gmail.com>
@noamgat @abetlen
fix: detokenization case where first token does not start with a lead…
e0d7674 
…ing space (abetlen#1375)

* Fix tokenization edge case where llama output does not start with a space

See this notebook:
https://colab.research.google.com/drive/1Ooz11nFPk19zyJdMDx42CeesU8aWZMdI#scrollTo=oKpHw5PZ30uC

* Update _internals.py

Fixing to compare to b' ' instead of (str)' '

---------

Co-authored-by: Andrei <abetlen@gmail.com>
@abetlen
feat: Update llama.cpp
3e2597e
@Smartappli
feat(ci): Add docker checks and check deps more frequently (abetlen#1426
3666833 
)

* Update dependabot.yml

Add github-actions update

* Update dependabot.yml

* Update dependabot.yml
@abetlen
feat(server): Add support for setting root_path. Closes abetlen#1420
0318702
Ported over prometheus implementation from previous repo
1972445
Added kn_cache_usage_ratio metric
edd0ec6
@juanroesel
Merge branch 'abetlen:main' into llm-monitoring
e42c1d6
@juanroesel juanroesel requested review from m62534 and cwarje May 7, 2024 02:32
@juanroesel
Copy link
Author

NOTE: GH Actions need to be updated in this repo. I will create a ticket for this soon.

@juanroesel juanroesel requested a review from blacklander May 7, 2024 17:17
cwarje
cwarje approved these changes May 7, 2024
View reviewed changes
Copy link

@cwarje cwarje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@juanroesel
Copy link
Author

juanroesel commented May 9, 2024 •
edited
Loading

@m62534 @cwarje Just FYI, given today's events with Llama3, I built a new image us.gcr.io/zenhub-ops/llama_cpp_python_zh-mistral7b_f16:v0.2.1 containing these code changes plus the Mistral model and redeployed it in staging.

@juanroesel juanroesel merged commit 8cd638c into main May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@m62534 m62534 m62534 approved these changes

@cwarje cwarje cwarje approved these changes

@blacklander blacklander Awaiting requested review from blacklander

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@juanroesel @m62534 @cwarje @abetlen @dthuerck @jeffrey-fong @noamgat @Smartappli