Skip to content

Ianscrivener macos install md docs #367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 15, 2023
Merged

Ianscrivener macos install md docs #367

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
94f63a6
Create macos_install.md
ianscrivener Jun 12, 2023
7ca50a3
Update README.md
ianscrivener Jun 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ This package provides:

Documentation is available at [https://abetlen.github.io/llama-cpp-python](https://abetlen.github.io/llama-cpp-python).

Detailed MacOS Metal GPU install documentation is available at [docs/macos_install.md](docs/macos_install.md)


## Installation from PyPI (recommended)

Install from PyPI (requires a c compiler):
Expand Down
62 changes: 62 additions & 0 deletions docs/macos_install.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@

# llama-cpp-python - MacOS Install with Metal GPU


**(1) Make sure you have xcode installed... at least the command line parts**
```
# check the path of your xcode install
xcode-select -p

# xcode installed returns
# /Applications/Xcode-beta.app/Contents/Developer

# if xcode is missing then install it... it takes ages;
xcode-select --install
```

**(2) Install the conda version for MacOS that supports Metal GPU**
```
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.sh
```

**(3) Make a conda environment**
```
conda create -n llama python=3.9.16
conda activate llama
```

**(4) Install the LATEST llama-cpp-python.. which, as of just today, happily supports MacOS Metal GPU**
*(you needed xcode installed in order pip to build/compile the C++ code)*
```
pip uninstall llama-cpp-python -y
CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install -U llama-cpp-python --no-cache-dir
pip install 'llama-cpp-python[server]'

# you should now have llama-cpp-python v0.1.62 installed
llama-cpp-python         0.1.62     

```

**(4) Download a v3 ggml llama/vicuna/alpaca model**
- **ggmlv3**
- file name ends with **q4_0.bin** - indicating it is 4bit quantized, with quantisation method 0

https://huggingface.co/vicuna/ggml-vicuna-13b-1.1/blob/main/ggml-vic13b-q4_0.bin
https://huggingface.co/vicuna/ggml-vicuna-13b-1.1/blob/main/ggml-vic13b-uncensored-q4_0.bin
https://huggingface.co/TheBloke/LLaMa-7B-GGML/blob/main/llama-7b.ggmlv3.q4_0.bin
https://huggingface.co/TheBloke/LLaMa-13B-GGML/blob/main/llama-13b.ggmlv3.q4_0.bin


**(6) run the llama-cpp-python API server with MacOS Metal GPU support**
```
# config your ggml model path
# make sure it is ggml v3
# make sure it is q4_0
export MODEL=[path to your llama.cpp ggml models]]/[ggml-model-name]]q4_0.bin
python3 -m llama_cpp.server --model $MODEL --n_gpu_layers 1
```

***Note:** If you omit the `--n_gpu_layers 1` then CPU will be used*