Open Source Large Language Models on Apple Silicon

Apple’s M-series chips have made it possible to run advanced language models natively on macOS. Thanks to the chips’ unified memory and Metal-accelerated neural processing, many open-source and even some proprietary large language models (LLMs) can now run offline on Macs. This opens up use cases like AI chatbots, text generation, and code completion without relying on cloud services, improving both privacy and responsiveness. Below, we review popular models and tools for running LLMs on Apple Silicon, covering their capabilities, offline support, interfaces, setup, and performance on Macs.

Performance and User Experience on macOS

By using open source large language models on apple silicon, users have seen surprisingly strong performance on Apple Silicon Macs:

• Even with only 8GB of RAM, an M1 MacBook can run 7B–13B models at decent speeds, often outperforming older PC GPUs. The unified memory and optimization libraries make efficient use of the limited memory.
• High-end Macs (M1 Ultra, M2 Max/Ultra) with 32–128GB of RAM can handle 30B+ parameter models in quantized form, though the generation speeds may slow down to a few tokens per second. For many applications (like short-form completion or single-turn Q&A), this speed is still acceptable.
• Apple’s GPUs, combined with frameworks like MLX, are achieving GPU-level speeds. For example, a 4-bit quantized LLaMA on an M1 Max can generate text on par with a mid-range NVIDIA GPU from a couple of years ago. This helps close the gap between local and cloud model usage, especially as Apple continues improving its tools.

User Experience: The ecosystem is rapidly evolving. A year ago, running a local LLM on Mac required using Terminal with a C++ program. Now, polished apps like GPT4All, as well as easy-to-use CLI tools like Ollama, make the process as simple as one click or one command. Many of these tools even offer an API compatibility layer, allowing developers to replace an OpenAI API call with a local endpoint without changing their application logic.

Code Completion & Developer Tools: Integrating a local model into editors is becoming much easier. Tools like Tabby and Continue (for VS Code) let you use models like StarCoder behind the scenes. Users have reported that a 7B code model runs almost as smoothly as cloud AI assistants, enabling a completely offline coding assistant that works within Visual Studio Code.

Challenges & Outlook: Running LLMs locally may still require some patience and adjustments – for example, experimenting with different quantization levels to balance speed and accuracy, or accepting that very large models (70B+) can still be slow on a laptop. But the trend is clear: Apple Silicon has unlocked practical local AI, and the gap is closing quickly as model architectures become more efficient and Apple’s software support improves. What once required a datacenter GPU can now run on a MacBook, bringing truly personal AI within reach.

List of Local LLM Options

Model/Tool	Developer	License Type
Vicuna-13B	LMSYS (UC Berkeley)	Open (derivative of LLaMA)
Mistral 7B	Mistral AI	Apache 2.0
StarCoder 15B	BigCode (Hugging Face & ServiceNow)	OpenRAIL
Falcon 40B	Technology Innovation Institute (TII)	Apache 2.0
GPT4All	Nomic AI	MIT
Ollama	Ollama Inc.	Apache 2.0
Apple MLX Framework	Apple ML Research	MIT
llama.cpp	G. Gerganov & Community	MIT
Hugging Face Transformers	Hugging Face	Apache 2.0

Table: Overview of large language models and tools running natively on macOS Apple Silicon

Below, we provide more detail on each model and tool, including capabilities, installation, and Mac performance notes.

Open-Source Models Running Locally

Vicuna-13B

Description: Vicuna is an open, fine-tuned chatbot model based on the original LLaMA (1) 13B. Developed by the LMSYS group, Vicuna was trained on user-shared ChatGPT conversations and designed to mimic ChatGPT-like quality. It is often considered one of the best chatbot models available for local use.

Capabilities: Vicuna-13B excels at interactive dialogue. It can answer questions, follow instructions, and maintain context throughout a conversation. The creators report that Vicuna-13B achieves over 90% of the quality of OpenAI’s ChatGPT (GPT-3.5) and Google Bard in their evaluations. While this figure is subjective, it shows that Vicuna is highly advanced for a 13B-parameter model. It often produces fluent, detailed responses in a conversational style, making it ideal for chatbot applications.

Local Execution: Yes – since Vicuna uses LLaMA weights, it runs wherever LLaMA can. Running Vicuna-13B on Apple Silicon is possible through the same methods (e.g., llama.cpp). You will need the Vicuna weight delta (usually applied to LLaMA 13B to create the Vicuna model), which is available for non-commercial use. Once you have the merged weights (13B), you can quantize them. Many Mac users run Vicuna 13B in 4-bit mode on systems with 16GB or 32GB of RAM.

Interfaces: Vicuna can be used through any interface that supports LLaMA models. For example, the GPT4All app includes Vicuna-based models; Ollama allow users to download Vicuna or its variants. There are also web UIs, like oobabooga’s text-generation-webui, where you can load Vicuna and chat directly in a browser. Essentially, Vicuna is made up of model weights – you’ll use a tool (like those in the next section) to interact with it.

Performance: Similar to Vicuna-13B usually generates a few tokens per second on CPU-only. However, with 4-bit quantization and Apple’s Metal offload, it can reach around 10 tokens/second or more on an M1 Pro/Max. This speed is sufficient for a smooth chat experience (e.g., answering questions in a few seconds). The Vicuna-7B variant (if used) runs even faster on Mac, but with some quality trade-offs. Vicuna-13B is preferred for serious use due to its much better conversational quality.

Use Cases: Vicuna is primarily used as a chatbot/assistant. It’s well-suited for Q&A, conversations, or role-playing dialogues. Many users deploy Vicuna locally to have a ChatGPT-like agent that works offline for brainstorming or tech support. Since it’s fine-tuned on general user queries, it’s less suited for tasks like pure code generation or fact retrieval (it wasn’t specifically trained for coding or knowledge tasks), though it can still attempt them.

Other LLaMA Fine-Tunes: In addition to Vicuna, many community models build on LLaMA for specialized purposes. For example, Alpaca (7B) from Stanford was an early instruction-tuned LLaMA 1 model that performed well on M1 Macs. WizardLM, Orca, Guanaco, and others are fine-tuned models that improve instruction following. These models can also run on Apple Silicon with varying chat capabilities. Vicuna remains one of the best for quality among these options.

Mistral 7B

Description: Mistral 7B is a 7.3-billion-parameter model released in late 2023 by the startup Mistral AI. It is fully open source (Apache 2.0 license) and was trained from scratch on a large, diverse text corpus. Despite its relatively small size, Mistral 7B has been shown to outperform larger models like LLaMA 2 13B on many tasks.

Capabilities: Mistral is a general-purpose model with strong performance in English-language tasks. Evaluations show that it even competes with some 13B–34B models. An instruct fine-tuned version (Mistral 7B Instruct) is available for chatbot and instruction-following tasks, and this variant reportedly outperforms LLaMA 2 13B-chat in quality. Mistral excels at summarization, Q&A, and casual dialogue.

Local Execution: Yes – Mistral 7B is small and efficient, making it ideal for local deployment. It can easily run on MacBooks, even those with just 8–16 GB of RAM. For example, the 7B model in 4-bit integer format is about 4 GB, and even a higher-precision 6-bit quantization (Q6_K) uses around 8 GB of RAM, which fits in a MacBook Air. Users have successfully run Mistral 7B on an M1 Pro 16GB with excellent speed, thanks to llama.cpp optimizations and the compact model size.

Interfaces: Like other models, Mistral can be loaded via llama.cpp (it uses the same model format as LLaMA-family). Many community-quantized Mistral models (including instruct variants) are available on repositories like TheBloke on Hugging Face. These can be used directly with llama.cpp or through GUIs. Ollama also published a guide on deploying Mistral 7B with their tool on Macs.

Performance: Mistral 7B is known for its speed. On Apple Silicon, it generates text very quickly. Users have reported excellent performance on basic M1 setups. Given its small size, it can produce 20+ tokens per second even on an M1 Pro CPU, with even higher speeds when using Metal GPU offloading. This makes it one of the fastest options for local AI chatting or writing, without sacrificing too much quality.

Use Cases: Mistral 7B is ideal for those looking for fast, lightweight local models. Its instruct version is great for chatbot tasks, answering questions, and following commands. It’s also useful for general text generation (such as stories or emails) when fine-tuned properly. While it’s not specialized for code or specific domains out-of-the-box, its strong base performance means it can be fine-tuned or used with prompting techniques for various tasks.

StarCoder

Description: StarCoder is a 15.5B parameter language model specifically trained for source code by the BigCode research project, a collaboration between ServiceNow and Hugging Face. Released in May 2023 under an open license (OpenRAIL), it is one of the most powerful open code models. StarCoder was trained on over 1 trillion tokens of code, including 80+ programming languages, GitHub issues, and notebooks. There is also StarCoderBase (the base pre-fine-tuned model) and a fine-tuned version simply called “StarCoder,” optimized for following instructions in coding tasks.

Capabilities: StarCoder can generate code, complete partially written code, and even engage in dialogue about code when prompted. It has been shown to match or outperform OpenAI’s older Codex models (like code-cushman-001, which powered early Copilot) on coding benchmarks. With a context window of 8,000 tokens, it can handle large code files or multi-file snippets. Typical use cases include writing functions from descriptions, completing code in real-time, translating code between languages, or explaining code in plain English. StarCoder supports many languages, from Python and C++ to more niche languages, thanks to its diverse training data.

Local Execution: Yes – but because StarCoder is larger (15B), it requires more memory than smaller models. Running the full 15B model in 16-bit precision would need around 30GB of RAM, which exceeds the capacity of most MacBooks. However, quantization helps: by using 4-bit or 8-bit weights, users have been able to run StarCoder on 16GB and 32GB Apple Silicon Macs. For example, one report noted that StarCoder 15B (likely in 8-bit) can run on an M2 Pro with 16GB of RAM. The BigCode project later released StarCoder2 with smaller variants (3B, 7B) that are easier to run locally. On a 32GB M1, StarCoder2-3B ran without quantization. So, depending on which version you choose, a smaller StarCoder model may offer faster performance.

Interfaces: You can download StarCoder from Hugging Face and load it in Transformers (it follows the GPT-NeoX architecture). There are also community GGML conversions for llama.cpp, as StarCoder has been integrated into llama.cpp for Apple Metal support by projects like TabbyML. For coding tasks, the typical setup involves using an editor extension that calls the local model. Tabby is an open-source code assistant that runs a local server with StarCoder and provides a VS Code extension. In September 2023, they added Apple Silicon GPU support, achieving “comparable performance to NVIDIA GPUs” for the 1B–7B StarCoder models on M1/M2 Macs. Another tool, Continue (a VSCode extension), can be configured to use Ollama or local models for completion.

Performance: If using a quantized 15B model on an M2 Pro/Max, you might get around 5 tokens/sec for code generation tasks (depending on quantization). This is on the edge of real-time “tab completion,” but still fast enough – the model can fill in a line or two of code in under a second. The smaller StarCoder 7B or StarCoder2 7B models would roughly double this speed on the same hardware, though with some trade-off in accuracy. In practice, many developers find StarCoder 15B acceptable on higher-end Macs, and the StarCoder 7B family runs smoothly on any M1/M2. For training or fine-tuning, StarCoder 15B is heavy, but fine-tuning smaller StarCoder2 on Mac is possible with techniques like LoRA.

Use Cases: StarCoder is a great local alternative to GitHub Copilot or Amazon CodeWhisperer. It lets you use AI autocompletion in your IDE without sending code to an external server. It’s useful for writing boilerplate, generating unit tests, or learning from code explanations. Its 8K token context lets it process an entire file, suggest improvements, or add documentation. Since it’s open, companies can self-host StarCoder for internal coding assistance. On macOS, it allows individual developers to have AI coding help on the go, even offline on battery. For non-code tasks, StarCoder can function as a general-purpose model, but dedicated LLMs may handle plain text tasks more effectively.

Falcon 40B

Description: Falcon 40B is a 40-billion-parameter model released by TII (Abu Dhabi) in 2023 under the Apache 2.0 license. It was one of the top-performing open models before the release of LLaMA 2. Falcon was trained on a high-quality dataset (refined web data) and topped many leaderboards for open models upon its release. There is also a smaller Falcon 7B variant and an instruction-tuned version called Falcon 40B-Instruct.

Capabilities: Falcon 40B is a strong general-purpose model. It excels at natural language understanding and generation, providing fluent answers and engaging in dialogue. The instruct version is great at following prompts for summarization or Q&A. Falcon is also multilingual to some extent. While not specifically designed for code, it can generate code snippets reasonably well (though models like StarCoder are better for heavy coding tasks). On the Chatbot Arena rankings (an open model benchmark), Falcon 40B was competitive but later outperformed by LLaMA 2 70B and other models. Still, Falcon 40B remains notable as a fully open model with no usage restrictions.

Local Execution: Partially – Falcon 40B can run on Apple Silicon, but it requires significant memory and is slower than smaller models. The 16-bit version is about 80 GB, which exceeds the memory capacity of most Macs. However, quantized versions make it feasible on high-end configurations. For example, a 4-bit quantized Falcon 40B is around 20 GB, which could fit on a 24 GB or 32 GB RAM Mac (with unified memory, the Mac’s GPU can access it too). Some users have successfully loaded Falcon 40B 4-bit on an M2 Ultra (24-core GPU, 128GB RAM) and even on an M2 Pro 16GB using extremely low-bit modes. One report mentioned that an Apple M2 with 24GB ran Falcon 40B (quantized) at about 1 token/second. The Falcon 7B model is much easier to run – it fits in 8GB with 4-bit quantization and runs quickly (similar to LLaMA 7B).

Interfaces: You can download Falcon models from Hugging Face (TII released them openly). Running Falcon typically happens via Hugging Face Transformers (the model uses a GPT-style architecture). On Mac, you can use Transformers with the MPS device to load Falcon 7B or 40B (if it fits) onto the GPU. There’s also a fork of llama.cpp called ggllm.cpp that added support for Falcon in GGML format. Tools like text-generation-webui also support Falcon models. Since Falcon is larger than 13B, many GUI apps designed for smaller models might not list it by default. Advanced users with Mac Studios have used the command line to experiment with it.

Performance: Falcon 40B is slower compared to smaller models on Mac. You might get 1–2 tokens/second on a Mac Studio with the quantized version, according to user reports. This is fine for short answers but not ideal for longer generation. The smaller Falcon 7B, however, is much faster (similar to other 7B models). Given that LLaMA 2 13B or Mistral 7B often deliver similar or better quality than Falcon 40B on many tasks, most Mac users prefer those models for local use instead of Falcon 40B. Still, Falcon 40B shows the upper limits of what Apple Silicon can handle, and the upcoming M3/M4 chips may run such large models faster.

Use Cases: If you have the necessary hardware, Falcon 40B can be a powerful local assistant for chat and writing tasks, with the advantage of an open license (allowing free commercial use). Its instruct version can serve as an enterprise chatbot working with sensitive data offline. On a MacBook, Falcon is more of a proof-of-concept due to its size. However, Falcon 7B is a good lightweight model for simpler tasks if you prefer it over LLaMA (though Falcon 7B is somewhat weaker than LLaMA 2 7B). In summary, Falcon 40B is notable, but you’d typically choose a comparable LLaMA-family model for practical local use on a Mac.

Other Models of Note: GPT-J 6B and GPT-NeoX 20B by EleutherAI were early open models (2021–22) that can run on Apple Silicon. GPT-J (6B) can even run on 8GB devices (it was used in early GPT4All versions). Databricks’ Dolly 2.0 (12B) is an instruction-tuned model based on Eleuther’s Pythia family; it’s open for commercial use and can run in 8-bit on a 16GB Mac. While these older models are generally surpassed by LLaMA and Mistral models in output quality, they are still options, especially if licensing is a concern (Dolly 2.0’s data and model are fully commercial-friendly).

Finally, it’s worth noting Apple’s own foundation models: in 2024, Apple announced a ~3B on-device language model built into iOS/macOS for features like text generation in apps, along with a larger model for their cloud. These models are not directly user-accessible for custom use, but they show Apple’s commitment to running language models on Apple Silicon hardware. Apple also mentioned a dedicated code model for Xcode in development. As these models become available (possibly through Apple’s APIs), Mac developers could use them for built-in AI features.

Local LLM Tools and Frameworks on macOS

To run and interact with these models easily, several tools and frameworks have been developed. Some offer user-friendly chat interfaces, while others are low-level libraries or servers you can integrate into applications. Here are some key tools and frameworks enabling LLMs on Apple Silicon:

GPT4All

What it is: GPT4All is a popular open-source desktop chat application that lets you run various language models locally with an easy-to-use interface. It was one of the first “ChatGPT-like” apps for offline use and has a large community. Developed by Nomic AI, GPT4All bundles many open models under one platform.

Capabilities & Models: GPT4All supports over 1,000 models that users can download and run. This includes conversational models (such as Vicuna, WizardLM, Mistral-Instruct, etc.), as well as some code-capable models. Essentially, it acts as a launcher – you choose a model from their catalog (or add your own), and GPT4All handles loading it and providing a chat UI. The app supports multi-turn chat, message history, and even Retrieval-Augmented Generation (RAG) via a “LocalDocs” feature, allowing the model to reference your files. By design, all data and processing stay on your device (no API keys needed).

Apple Silicon Support: GPT4All is fully compatible with M-series Macs. It offers a native macOS app (downloadable as a .dmg) that uses Apple’s Metal backend for acceleration. Users have noted that GPT4All runs very efficiently on Macs – for instance, one user reported that an M1 MacBook Pro (8GB RAM) outperformed a high-end Intel i7 desktop in model inference, showing how well it uses Apple’s hardware. GPT4All uses a library (ggml/llama.cpp-based) that can automatically utilize the GPU (Metal) on M1/M2. You can choose in the settings whether to use the CPU or Metal GPU. Even without a GPU, it works on the CPU, but Metal usually improves speed on these chips.

Interface: The main interface is a GUI application that resembles a chat messenger. You type a prompt, and the model responds in a text window. You can select the model, adjust parameters like temperature, and more. It’s user-friendly and designed for non-developers. For developers, GPT4All offers a Python package (gpt4all) to load models programmatically, and it can be run headless via CLI if needed.

Installation: Simply download the macOS app from the official site (or via Homebrew cask). After launching, you select a model to download. GPT4All will handle downloading the model weights (many are hosted on Hugging Face or their repository). Once downloaded, you can chat offline. The documentation provides a FAQ and troubleshooting guide for Mac-specific issues (like conflicts with the Metal library).

Performance: The performance depends on the chosen model. Lighter models, like 7B, run quickly on Mac (often generating text in real-time), while larger models (13B+) will be slower. GPT4All’s default models are usually quantized to fit in memory and run smoothly on 8GB–16GB machines. For example, the fine-tuned “Nous Hermes 13B” offers a good balance between quality and speed on Macs. Overall, GPT4All simplifies the process, selecting optimal settings for you. Apple Silicon has proven to be especially capable – GPT4All’s team even highlights M-series support as a key feature, enabling truly portable AI chats.

Use Cases: GPT4All is perfect for a ChatGPT-like experience offline. You can ask general knowledge questions, have creative conversations, or even draft emails – all without an internet connection. It’s also used in education or workplaces where data can’t be sent to external servers. With the LocalDocs feature, it can act as a private assistant that knows your documents. Keep in mind that the models used aren’t as advanced as OpenAI GPT-4, so answers might be less accurate, but models are constantly being improved. For most casual and many professional uses, GPT4All provides a convenient and privacy-preserving solution on Mac.

Ollama

What it is: Ollama is an open-source toolchain for running LLMs locally through a command-line interface and API. Think of it as a package manager and server for language models on your Mac. It’s designed to simplify the process of downloading models and running them, especially for developers who want to integrate local LLMs into their workflows or applications.

Key Features: Ollama allows you to pull pre-built model images (similar to how you would pull a Docker image). For example, ollama pull llama2 will download a LLaMA 2 model in a ready-to-run format. Once downloaded, you can run inference with a simple command (ollama run model_name) or start a persistent server. By running ollama serve, you can launch a local HTTP API (on localhost:11434 by default) that you can send requests to, mimicking an OpenAI-like API. This makes it easy to use local models in applications by just pointing them to your Mac’s local endpoint.

Installation: On macOS, the easiest way is through Homebrew: brew install ollama. This installs the Ollama CLI. Alternatively, there is a direct macOS installer available. The only requirement is macOS 11 or later (Big Sur+) on Apple Silicon (it works on Intel Macs too, but slower).

Apple Silicon Support: Ollama was initially designed for Macs (leveraging M1/M2 optimizations), but it has since added support for Linux and Windows. However, Mac is still the primary platform. It uses the llama.cpp/ggml backend with Metal acceleration when possible, so performance on M1/M2 is excellent. The documentation advises against running it inside Docker on Mac (since Docker doesn’t use the GPU); instead, run Ollama natively to get full Apple Silicon support.

Models and Use Cases: Ollama’s model catalog includes a variety of models: Mistral, Vicuna, and specialized models like “DeepSeek” or “Gemini.” They curate these models, often applying quantization and packaging them for easy use. The typical use case is for developers or power users who want to script or programmatically use LLMs locally. For example, you could set up Ollama and use a small script to query the model for completions as part of an automation, without relying on an external API.

Interface: Ollama is primarily CLI-based. Here are a few usage examples:

• ollama pull vicuna – download the Vicuna model.
• ollama run vicuna --prompt "Hello, how are you?" – generate a completion from Vicuna.
• ollama serve – run the daemon. Once running, you can use curl http://localhost:11434/completions -d '{"model": "vicuna", "prompt": "Hello"}' to get a response (the API mimics OpenAI’s format for easy integration).

Additionally, Ollama can run as a background macOS service (brew services start ollama to start it on login). This means the local API is always available. Editors like VS Code (with certain extensions) or other apps can then connect to it. For instance, the “Continue” VSCode extension can be configured to use Ollama’s endpoint for code completions instead of OpenAI.

Performance: Since Ollama uses quantized models and optimizes for local hardware, its performance is similar to running the same model in llama.cpp. On an M1/M2 Mac, smaller models generate responses quickly. The advantage of Ollama is that it abstracts threading and hardware usage, trying to use the optimal settings for each model. In practice, users have found it easy and responsive for models up to 13B on Mac. One Reddit user shared their experience of running multiple models on an M1 Max (32GB) with Ollama without any issues.

Use Cases: Ollama is perfect if you want a local LLM server on your Mac. For example:

• You could build a small web app that sends user queries to your Mac’s Ollama server and gets AI-generated answers (all local).
• Integrate it with development tools for code suggestions by pointing them to Ollama (such as connecting with IntelliJ or VSCode).
• Experiment with different models quickly by pulling them and trying them out via the CLI, without needing manual conversions.

Because it’s designed for developers, Ollama doesn’t have a fancy GUI (though you could build a front-end that interacts with the API). It’s also useful for fine-tuning: you can pull base models and fine-tune them (though Ollama itself doesn’t handle training; you’d need other tools for that, and then possibly use Ollama for inference).

Apple MLX

What it is: Apple’s MLX is a new machine learning framework optimized for Apple Silicon, introduced in 2024. It’s Apple’s answer to frameworks like PyTorch and JAX, but designed specifically for their hardware. MLX is focused on training and deploying models on Macs, with a sub-framework called MLX-LM for working with large language models (LLMs).

Capabilities: MLX provides a Python API (and C++/Swift APIs) for writing machine learning code that runs efficiently on Apple chips. It supports features like automatic differentiation, JIT compilation, and multi-device (CPU/GPU/ANE) execution with unified memory. In simple terms, it lets you harness the full power of the M1/M2 GPU and Neural Engine for machine learning tasks without needing to deal with low-level Metal code. For LLMs, MLX-LM integrates with Hugging Face Transformers, allowing you to load models from Hugging Face and convert them into MLX format for fast execution. It also supports model quantization (including 4-bit) and optimized kernels for transformer operations.

Local Execution: Yes – MLX is designed specifically for local use on Apple Silicon (it doesn’t run on other hardware). For inference, MLX runs models on the GPU or Neural Engine. Apple demonstrated using MLX to deploy a LLaMA 3.1 8B model on an M1 Max, achieving over 16 tokens/sec in 4-bit mode, and up to 33 tokens/sec after further optimization. MLX can also be used for training or fine-tuning models on Mac GPUs. For example, Apple’s tools allow LoRA fine-tuning using MLX with the Neural Engine to accelerate the process.

Interfaces & Usage: MLX is a Python package (mlx or specifically mlx-lm). You can install it via pip (pip3 install mlx-lm). Here’s the typical workflow for using MLX with LLMs:

• Convert a Hugging Face model to MLX format:
  Example: python3 -m mlx_lm.convert --hf-path meta-llama/Meta-Llama-3-8B-Instruct -q converts Meta’s LLaMA 3 8B Instruct model into a quantized MLX model (4-bit by default).
• Generate text:
  Example: python3 -m mlx_lm.generate --model meta-llama/Meta-Llama-3-8B-Instruct --prompt "Hello" loads the model and produces output.
• Serve via API:
  Run: mlx_lm.server --model model_path to launch a local API on localhost:8080/v1/chat/completions (similar to an OpenAI endpoint).

You can also use MLX programmatically – import mlx in Python and use its tensors and modules to write custom training loops. However, for LLM deployment, the command-line interface (CLI) is sufficient for most needs.

Performance and Benchmarks: Apple’s research shows that MLX (with the Core ML backend) introduced features like stateful KV cache and fused attention, which significantly boost LLM inference speed on Apple GPUs. By reducing unnecessary data copying and using 4-bit quantization, MLX achieved a 13× speedup over naive implementations. For example, an 8B model that might run at ~2 tokens/sec on naive PyTorch could reach ~30 tokens/sec with MLX on the same Mac. This optimization specifically for Apple hardware is one of MLX’s strengths. It fully utilizes the M1/M2 GPU and the 16-core Neural Engine for certain operations, which frameworks like PyTorch only partially utilize.

One caveat: since MLX is new, community support is smaller, and it may require macOS 14+ to access all features (such as the stateful cache in macOS “Sequoia” 15.2 beta). But it’s clear that MLX is the direction Apple is heading for on-device machine learning.

Use Cases: MLX is ideal for developers who want to fine-tune or deploy custom models on Mac. For example, you can fine-tune a smaller LLaMA or Mistral model with LoRA on an M2 Pro laptop. A Medium post reported fine-tuning a model this way, though it was slow with 16GB of RAM until they quantized the model. MLX’s conversion tool can also compress models and integrate them efficiently into apps or services. Some users have combined Ollama and MLX, using MLX for fine-tuning and Ollama for serving. A tutorial even showed how to fine-tune using MLX and serve the model via Ollama.

For end-users, MLX also powers apps like Chat-with-MLX, an open-source chat interface specifically using MLX backends. Chat-with-MLX provides a Mac chat UI (similar to GPT4All) but leverages MLX for multilingual support and performance. You can install it via a tool like Pinokio or from source. This shows that MLX, while itself a framework, has led to easy-to-use applications built on top of it.

llama.cpp

What it is: llama.cpp is a lightweight, open-source C/C++ library designed to run LLMs (starting with Meta’s LLaMA) on nearly any device using efficient CPU inference and quantization. It’s not a standalone app but a backend used by many tools. On Mac, llama.cpp has been essential since early 2023, being one of the first to run LLaMA on M1 Macs, and it continues to be optimized for Apple hardware.

Key Features:

• Written in plain C/C++, with no external dependencies – works on Linux, Windows, macOS, iOS, and even WebAssembly.
• Supports various model architectures (LLaMA 1 & 2, GPT-J, Mistral, Falcon, etc.).
• Enables quantization to 8-bit, 4-bit, 3-bit, or 2-bit to significantly reduce memory usage.
• Includes an Apple Metal backend: setting LLAMA_METAL=1 offloads compute to the GPU, utilizing Apple’s Metal API directly.
• Offers a command-line interface and Python bindings (llama-cpp-python) for easy integration into scripts or apps.

Installation on Mac: Installation is simple. You can either compile from source or use a package manager:

• brew install llama-cpp (Homebrew CLI tool)
• pip install llama-cpp-python (Python bindings with Apple Silicon support)
• Community-maintained scripts offer one-liner installs with the Metal backend pre-configured.

Performance: llama.cpp is optimized for CPU cache and multithreading. On Apple Silicon, the performance is as follows:

• M1 Max with Metal offload: ~60 tokens/sec (7B model, 4-bit quant)
• M1 MacBook Air: ~20 tokens/sec (7B, 4-bit)
• M1 Ultra: ~75 tokens/sec (7B)

This throughput is for prompt ingestion; autoregressive generation is slightly slower but still real-time. It makes high-quality LLM inference possible even on fanless laptops.

Memory Efficiency:

• LLaMA 2 13B (FP16): ~26 GB
• Quantized 8-bit: ~13 GB
• Quantized 4-bit: ~7 GB

With this efficiency, a 13B model fits on a 16GB Mac, and even 30B models are usable on high-end Macs (e.g., 30B at 4-bit ≈ 16 GB).

Interface & Use:

• CLI: Run the model with ./main -m model.bin -p "Your prompt"
• Interactive mode: Keep a chat session going in the terminal with incremental input/output.
• Python: Import and use llama_cpp.Llama to run models from Python scripts or apps.
• Under the hood: Many apps (like GPT4All, and Tabby) use llama.cpp as their inference engine.

Use Cases: llama.cpp has made portable, offline LLMs practical. On Apple Silicon, it powers fast, private, no-cloud-needed AI apps. It’s widely used for:

• Embedding LLMs in macOS/iOS apps
• Developing local chat UIs, assistants, or coding tools
• Cross-platform inference (Linux, macOS, Windows, browser)

Limitations: llama.cpp only supports inference (not training). However, some forks, like alpaca.cpp, experiment with local fine-tuning and LoRA support.

In Summary: If you’re using a local LLM on Mac, it’s likely running on llama.cpp. This backend powers many AI apps with lightweight, cross-platform performance and strong Apple Silicon acceleration.

Hugging Face Transformers

What it is: The Hugging Face Transformers library is the go-to toolkit for working with pre-trained language models in Python. While it’s traditionally used with GPU (CUDA) or CPU, it can also run models on Apple Silicon using PyTorch’s MPS (Metal Performance Shaders) backend. This means you can load models from the Hugging Face Hub and run them on your Mac’s GPU or CPU.

Capabilities: Transformers supports all the popular architectures – GPT-2, GPT-NEO, T5, BERT, LLaMA, and more. So, you’re not limited to just the few models supported by llama.cpp. For example, you can run an 11B Flan-T5 (Google’s instruction-tuned model) on your Mac, a smaller conversational model like Microsoft’s DialoGPT, or any custom model you’ve fine-tuned. It’s very flexible for experimentation, pipeline tasks (text generation, fill-mask, translation, etc.), and integration with other libraries (e.g., using LangChain with local models).

Local Execution: To use the GPU, you need PyTorch 1.12+ which introduced the mps device for Apple Metal. Here’s an example of how to use it:

import torch from transformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("model-name", torch_dtype=torch.float16) tokenizer = AutoTokenizer.from_pretrained("model-name") model.to("mps") inputs = tokenizer("Hello, I'm a Mac and I ", return_tensors="pt").to("mps") outputs = model.generate(**inputs) print(tokenizer.decode(outputs[0])) 

This runs the model on the Mac’s GPU in float16 precision. Important: Not all operations are fully optimized for MPS yet. Some operations might fall back to the CPU, and as of 2024, PyTorch MPS doesn’t support half-precision accumulation or bfloat16. But for forward inference, it works well and is continuously improving.

Performance: Using Transformers on Mac is generally slower and more memory-intensive than using specialized libraries like llama.cpp or MLX for the same model. For example, running a 7B model in Transformers (without quantization) can use about 14GB of memory and might generate 2–5 tokens/sec on an M1 Pro GPU. However, for models that don’t have simple quantized alternatives (like specific DialoGPT variants or sequence-to-sequence models), this is a straightforward method to run them.

To improve performance and memory usage, you can:

• Use accelerate with device_map="auto" to split the model across CPU and GPU memory.
• Export smaller models to ONNX or Core ML and run them using Core ML Tools for better performance.

Use Cases: The Hugging Face Transformers + PyTorch combination is excellent for research and fine-tuning on Mac. You can fine-tune smaller models using the GPU (many in the machine learning community are fine-tuning 7B models with LoRA on a 32GB M1 Max using PyTorch+MPS). If you need to run a sequence-to-sequence model (like for translation or summarization with models like FLAN or PEGASUS), this is the best option since llama.cpp mainly handles decoder-only models. Additionally, Transformers provides advanced pre/post-processing tools like tokenization and generation strategies (beam search, temperature, top-p, etc.), which the llama.cpp CLI doesn’t offer.