Deploy an AI chatbot for Zulip in Kubernetes

Cylab has currently deployed a local LLM which can be accessed through Open WebUI. The next obvious step is to use it as the brain of a chatbot.

Zulip offers Generic bots that whose logic can be coded with python using the zulip library.

For info on how to deploy Zulip in Kubernetes read this. To populate the ollama_api_key, follow this and save it in an ansible-vault file.

The code

The following python code is a more than reasonable start for a chat bot.

# ollama_bot_with_context.py
import os
from collections import defaultdict
import requests
import zulip

# .zuliprc defines the connection settings for the client.
client = zulip.Client(config_file="/root/.zuliprc")

# Load env variables
API_KEY = os.environ.get("OLLAMA_API_KEY")
OLLAMA_URL = os.environ.get("OLLAMA_URL")
MODEL = os.environ.get("OLLAMA_MODEL")
DEBUG = os.environ.get("BOT_DEBUG", False)
system_prompt=os.environ.get("OLLAMA_SYSTEM_PROMPT", "You are a chat bot on a research institute chatting system. As a chat bot, your answers should be short, unless instructed otherwise.")

HEADERS = {
    "Authorization": f"Bearer {API_KEY}",
    "Content-Type": "application/json",
}

# Maximum number of past turns (user + assistant) to retain per thread
MAX_HISTORY_MESSAGES = int(os.environ.get("MAX_HISTORY_MESSAGES", 15))

# Stores conversation history: {conversation_key: [{"role": "...", "content": "..."}, ...]}
conversations = defaultdict(list)


def get_conversation_key(msg):
    """Generates a unique identifier for a channel topic or a DM thread."""
    if msg["type"] == "stream":
        stream_id = msg.get("stream_id", msg.get("display_recipient"))
        topic = msg.get("subject", "").lower()
        return f"stream:{stream_id}:{topic}"

    # Direct messages: display_recipient is a list of participant dicts [{'id': ...}, ...]
    if isinstance(msg.get("display_recipient"), list):
        user_ids = sorted(user["id"] for user in msg["display_recipient"])
        return f"dm:{'-'.join(map(str, user_ids))}"

    # Fallback to sender_id if display_recipient isn't a list
    return f"dm:{msg['sender_id']}"


def handle_message(msg):
    if DEBUG:
        print(msg)

    # Ignore messages sent by the bot itself
    if msg["sender_email"] == client.email:
        return

    user_prompt = f"{msg["sender_full_name"]}: {msg["content"].strip()}"
    thread_key = get_conversation_key(msg)

    # Optional: reset context on demand
    if user_prompt.lower() in ["/reset", "/clear"]:
        conversations[thread_key] = []
        reply_payload = {"type": msg["type"], "content": "Conversation history cleared."}
        if msg["type"] == "stream":
            reply_payload["to"] = msg["display_recipient"]
            reply_payload["topic"] = msg.get("subject", "")
        else:
            reply_payload["to"] = [msg["sender_id"]]
        client.send_message(reply_payload)
        return

    # Append user prompt to thread history. Even if the bot is not mentioned, it will still take the message into accout if mentioned in the future.
    conversations[thread_key].append({"role": "user", "content": user_prompt})

    # Keep only the last N messages to respect the model's context window
    if len(conversations[thread_key]) > MAX_HISTORY_MESSAGES:
        conversations[thread_key] = conversations[thread_key][-MAX_HISTORY_MESSAGES:]


    # reply to all private messages or to non-private messages where the bot is mentioned.
    if msg['type']=='private' or '@**ai-bot**' in user_prompt.lower():
        try:
            response = requests.post(
                OLLAMA_URL,
                headers=HEADERS,
                json={
                    "model": MODEL,
                    "messages": conversations[thread_key] + [{"role": "system", "content": system_prompt}],
                    "stream": False,
                },
                timeout=60,
            )
            response.raise_for_status()
            reply = response.json().get("message", {}).get("content", "No response received.")

            # Save model reply to context
            conversations[thread_key].append({"role": "assistant", "content": reply})
        except Exception as e:
            reply = f"Error reaching Ollama: {e}"
            # Pop user message if the request failed so it doesn't corrupt history
            if conversations[thread_key]:
                conversations[thread_key].pop()

        reply_payload = {
            "type": msg["type"],
            "content": reply,
        }

        if msg["type"] == "stream":
            reply_payload["to"] = msg["display_recipient"]
            reply_payload["topic"] = msg.get("subject", "")
        else:
            reply_payload["to"] = [msg["sender_id"]]

        client.send_message(reply_payload)


client.call_on_each_message(handle_message, all_public_streams=True)

Creating the bot in Zulip

The following Ansible tasks create the bot if it doesn’t exsits and register its API key. We will use it to later on to populate the .zuliprc template.

- name: Create AI bot
  ansible.builtin.uri:
    url: "https://chat.{{ k8s_top_level_domain }}/api/v1/bots"
    method: POST
    user: "{{ cluster_administrator_email }}"
    password: "{{ api_key.json.api_key }}"
    ca_path: "/etc/ssl/certs/cluster_CA.crt"
    body_format: form-urlencoded
    body:
      full_name: "AI-Bot"
      short_name: "ai"
      bot_type: "1"
  ignore_errors: true

- name: Get AI bot
  ansible.builtin.uri:
    url: "https://chat.{{ k8s_top_level_domain }}/api/v1/bots"
    method: GET
    user: "{{ cluster_administrator_email }}"
    password: "{{ api_key.json.api_key }}"
    ca_path: "/etc/ssl/certs/cluster_CA.crt"
  register: ai_bot_response
  ignore_errors: true

- name: Extract API key for specific bot
  ansible.builtin.set_fact:
      ai_bot_key: "{{ ai_bot_response.json.bots | selectattr('username', 'equalto', 'ai-bot@chat.' + k8s_top_level_domain ) | map(attribute='api_key') | first }}"

Ansible deployment

The app is stateless so it can be deployed as a Kubernetes Deployment:

# bot-deployment.yaml (Part 1)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ bot_name }}-bot"
  namespace: zulip
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "{{ bot_name }}-bot"
  template:
    metadata:
      labels:
        app: "{{ bot_name }}-bot"
    spec:
      restartPolicy: Always
      containers:
        - name: bot
          image: python:3.12-slim
          command:
            - /bin/sh
            - -c
            - pip install --no-cache-dir zulip && python3 -u /opt/bot/bot.py
          workingDir: /opt/bot
          volumeMounts:
            - name: zuliprc
              mountPath: /root/.zuliprc
              subPath: zuliprc
              readOnly: true
            - name: bot-code
              mountPath: /opt/bot
              readOnly: true
          env:
            - name: PYTHONUNBUFFERED
              value: "1"  # Ensure logs go to stdout immediately
            - name: OLLAMA_API_KEY
              valueFrom:
                secretKeyRef:
                  name: "{{ bot_name }}-bot-credentials"
                  key: ollama_api_key
            - name: OLLAMA_URL
              value: "{{ ai_bot_ollama_url }}"
            - name: OLLAMA_MODEL
              value: "{{ ai_bot_ollama_model }}"
            - name: MAX_HISTORY_MESSAGES
              value: "{{ ai_bot_max_history_messages | default('50')}}"
            - name: OLLAMA_SYSTEM_PROMPT
              value: "{{ ai_bot_system_prompt }}"
          resources:
            limits:
              memory: "256Mi"
              cpu: "200m"
      tolerations:
        - key: "node.kubernetes.io/unreachable"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 5
        - key: "node.kubernetes.io/not-ready"
          operator: "Exists"
          effect: "NoExecute"
          tolerationSeconds: 5
      volumes:
        - name: zuliprc
          secret:
            secretName: "{{ bot_name }}-bot-credentials"
        - name: bot-code
          configMap:
            name: "{{ bot_name }}-bot-code"

Where the secret and configMap are as follows:

# bot-deployment.yaml (Part 2)
---
apiVersion: v1
kind: Secret
metadata:
  name: "{{ bot_name }}-bot-credentials"
  namespace: "zulip"
type: Opaque
stringData:
  ollama_api_key: "{{ ai_bot_ollama_api_key }}"
data:
  zuliprc: "{{ lookup('template', 'zuliprc') | b64encode }}"
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "{{ bot_name }}-bot-code"
  namespace: zulip
binaryData:
  bot.py: "{{ lookup('file', bot_code) | b64encode }}"

and the zuliprc template is

[api]
email={{ bot_name }}-bot@chat.{{ k8s_top_level_domain }}
key={{ bot_key }}
site=https://chat.{{ k8s_top_level_domain }}
insecure=true

The following Ansible task will deploy the above

- name: Install ai-bot
  kubernetes.core.k8s:
    template: "bot-deployment.yaml"
    state: present
  vars:
    bot_name: ai
    bot_code: ollama_bot_with_context.py
    bot_key: "{{ ai_bot_key }}"

This blog post is licensed under CC BY-SA 4.0