Take the slides with you

Blablador

• /ˈblæblæˌdɔɹ/
• Bla-bla-bla 🗣️ + Labrador 🐕‍🦺
• A stage for deploying and testing large language models
• Models change constantly (constantly improving rank, some good, some awful)
• Usually a small/fast model and fone of the top of the HF’s Open LLM Leaderboard
• It is a web server and an api server, and training code.

“I think the complexity of Python package management holds down AI application development more than is widely appreciated. AI faces multiple bottlenecks — we need more GPUs, better algorithms, cleaner data in large quantities. But when I look at the day-to-day work of application builders, there’s one additional bottleneck that I think is underappreciated: The time spent wrestling with version management is an inefficiency I hope we can reduce.”

Andrew Ng, 28.02.2024

“Building on top of open source can mean hours wrestling with package dependencies, or sometimes even juggling multiple virtual environments or using multiple versions of Python in one application. This is annoying but manageable for experienced developers, but creates a lot of friction for new AI developers entering our field without a background in computer science or software engineering.”

Andrew Ng, 28.02.2024

Why?

• AI is becoming basic infrastructure
• Which historically is Open Source
• We train a lot, deploy little: Here is your code/weights, tschüssi!
• Little experience with dealing with LLMs
• From the tools point of view, this is a FAST moving target 🎯💨
• Acquire local experience in issues like
  • data loading,
  • quantization,
  • distribution,
  • fine-tune LLMs for specific tasks,
  • inference speed,
  • deployment
• Projects like OpenGPT-X, TrustLLM and Laion need a place to run
• The usual: we want to be ready when the time comes
• TL;DR: BECAUSE WE CAN! 🤘

Some facts

• I CAN HOST YOUR MODEL
• No data collection at all. I don’t keep ANY data whatsoever!
  • You can use it AND keep your data private
  • No records? Privacy (and GDPR is happy)

Deployment as a service

• Scientists from (currently just FZJ) can deploy their models on their own hardware and point to blablador
• This solves a bunch of headaches for researchers:
  • Authentication
  • Web server
  • Firewall
  • Availability
  • Etc
• If you have a model and want to deploy it, contact me!

OpenAI-compatible API

• Uses openai-python from OpenAI itself
• All services which can use OpenAI’s API can use Blablador’s API (VSCode’s Continue.dev, etc)
• The API is not yet rate-limited, logged, monitored, documented or well-tested.