Take the slides with you

Website

Blablador

• /ˈblæblæˌdɔɹ/
• Bla-bla-bla 🗣️ + Labrador 🐕‍🦺
• A stage for deploying and testing large language models
• Models change constantly (constantly improving rank, some good, some awful)
• Usually a small/fast model and fone of the top of the HF’s Open LLM Leaderboard
• It is a web server and an api server, and training code.

“I think the complexity of Python package management holds down AI application development more than is widely appreciated. AI faces multiple bottlenecks — we need more GPUs, better algorithms, cleaner data in large quantities. But when I look at the day-to-day work of application builders, there’s one additional bottleneck that I think is underappreciated: The time spent wrestling with version management is an inefficiency I hope we can reduce.”

Andrew Ng, 28.02.2024

“Building on top of open source can mean hours wrestling with package dependencies, or sometimes even juggling multiple virtual environments or using multiple versions of Python in one application. This is annoying but manageable for experienced developers, but creates a lot of friction for new AI developers entering our field without a background in computer science or software engineering.”

Andrew Ng, 28.02.2024

Why?

• AI is becoming basic infrastructure
• Which historically is Open Source
• We train a lot, deploy little: Here is your code/weights, tschüssi!
• Little experience with dealing with LLMs
• From the tools point of view, this is a FAST moving target 🎯💨
• Acquire local experience in issues like
  • data loading,
  • quantization,
  • distribution,
  • fine-tune LLMs for specific tasks,
  • inference speed,
  • deployment
• Projects like OpenGPT-X, TrustLLM and Laion need a place to run
• The usual: we want to be ready when the time comes
• TL;DR: BECAUSE WE CAN! 🤘

Some facts

• We have no models of our own (yet) deployed
• Models based on Llama2-70 🦙 take 7 gpus (or 8 with vLLM)
• VLLM: PagedAttention, batching etc. Speeds up inference at cost of gpus (no quantization)
• SGLang: RadixAttention - Even faster inference. Single-GPU for now
• Mixtral-8x7b: takes 7 gpus, but is faster and better than Llama2-70
• No data collection at all. I don’t keep ANY data whatsoever
  • You can use it AND keep your data private
  • We could, there’s code for ranking answers, or running models in parallel and voting
  • No records? Privacy! GDPR is happy

Deployment as a service

• Scientists from (currently just FZJ) can deploy their models on their hardware and point to blablador
• This solves a bunch of headaches for researchers:
  • Authentication
  • Web server
  • Firewall
  • Availability
  • Etc
• If you have a model and want to deploy it, contact me!

What is it?

• A classic MVC web application:
  • Model: large language model(s)
  • View: a web server and api server (openAI-compatible)
  • Controller: coordinates the models
• A collaboration with LM-Sys (From Vicuña 🦙 fame)
  • FastChat: https://chat.lmsys.org
• Python app, Runs on bare metal 🤘 with venvs
• Models run in different sc_venv_templates
  • Conflicting versions of libraries
• Website Authentication: Helmholtz AAI, no records, just to keep the bots out
• API Authentication: token with api scope from helmholtz codebase

OpenAI-compatible API

• Uses openai-python from OpenAI itself
• All services which can use OpenAI’s API can use Blablador’s API (VSCode’s Continue.dev, etc)
• The API is not yet rate-limited, logged, monitored, documented or well-tested.

Jülich Supercomputers

Jülich Supercomputers