An AI assistant inside WhatsApp: rebuild that connection step by step

What you're going to build

The whole system consists of two programs running side by side on one server, plus an AI model:

• The bridge — a small program (we wrote ours in Go) that attaches to your WhatsApp as a linked device, just like WhatsApp Web. It receives every message and can send messages.
• The middleware — a web server (Python in our case) that takes every incoming message, turns it into an instruction for the AI model and sends the answer back.
• The memory — a database holding every message per conversation, so the assistant knows what was said before.

The route of a single message: your phone → WhatsApp → the bridge → the middleware → the AI model → the middleware → the bridge → back in your chat. The bridge and the middleware talk to each other over two local ports on the same server — nothing in this chain needs to be exposed to the internet.

Requirements

• A Linux server or VPS (1 CPU and 1–2 GB of memory is enough to start)
• Go 1.21 or newer, for the bridge
• Python 3.11 or newer, for the middleware
• An open-source WhatsApp library; we use whatsmeow, which speaks WhatsApp's linked-device protocol
• Access to an AI model: an API key from a provider, or an AI tool you can call from the command line
• SQLite (bundled almost everywhere) for the session and message history
• Optional: ffmpeg and a speech-to-text model such as Whisper, for voice notes

You do not need a separate phone number: the bridge pairs as an extra device on an existing WhatsApp account. For business use, a second number is cleaner.

Phase 1 — the bridge: connecting to WhatsApp

The bridge only does two things: stay connected and pass messages along. Here's how to set it up:

1. Create a new Go project and add the WhatsApp library as a dependency.
2. Have the library store its session in a local SQLite file — that way the pairing survives a restart.
3. Start the program: on first run a QR code appears in the terminal.
4. Scan that QR code with your phone via WhatsApp → Linked devices, exactly like WhatsApp Web.
5. Register an event handler that fires on every incoming message.
6. In that handler, store every message in a second SQLite database with two tables: one for conversations (chat ID, name, last message) and one for messages (ID, chat ID, sender, text, timestamp, from-me-or-not).

Test this on its own first: send yourself a message and check it shows up in the database. Only move on once this works.

Phase 2 — the bridge as a relay point: webhook and send API

The bridge needs to pass messages on and send them. So it gets two faces:

1. Give the bridge a small HTTP server on a local port (we use 8080), reachable only from the server itself.
2. Add a send endpoint: a POST with recipient, text and optionally a file path, after which the bridge sends it as a WhatsApp message.
3. Add a download endpoint that fetches and decrypts media from an earlier message (WhatsApp media is encrypted; the key travels with the message and the library handles decryption).
4. Have the message handler from phase 1 also forward every incoming message: a POST with chat ID, sender, text, timestamp and media type to a fixed webhook address on the server — port 8769 in our case.

That webhook address is the middleware from phase 3. If it doesn't exist yet, a POST simply goes nowhere — the bridge keeps running.

Phase 3 — the middleware: from message to answer

This is the brain of the chain. A web server (we use FastAPI) with one job: message in, answer out.

1. Create a web server with a webhook endpoint that accepts the bridge's POST.
2. Build an allowlist first thing: only phone numbers you explicitly allow get processed, everything else is ignored. Never skip this step — otherwise anyone with your number can put your assistant to work.
3. For every message, fetch the latest messages of that conversation from the phase 1 database — that's your conversational memory.
4. Turn memory plus new message into a single instruction for the AI model, topped with a short system instruction: who the assistant is, what it may and may not do.
5. Call the AI model — through a provider's API, or by spawning an AI command-line tool as a separate process (that's what we do; the process receives the instruction and prints the answer).
6. POST the answer to the bridge's send endpoint from phase 2.

That closes the loop: anyone on the allowlist sends a message and gets an answer from the AI — with memory.

Phase 4 — photos, documents and voice

Messages are rarely just text. Three extensions that make the difference:

1. If the middleware sees a media type in the webhook payload, have it fetch the file through the bridge's download endpoint and pass the local path to the AI model — modern models can look at images directly.
2. If it's a voice note, fetch the audio file and run it through a speech-to-text model; the text then follows the same route as a normal message.
3. Build in a short wait (we use 2 seconds): if someone sends three photos in a row, you want to treat them as one instruction instead of calling the model three times.

If you send audio back: mind the format. iPhones won't play every audio file — AAC/M4A works, raw WAV or OGG often doesn't.

Phase 5 — keeping it running (the real work)

Everything above is a weekend of building. This is where the months go:

1. Run the bridge and middleware as systemd services with automatic restarts, so a crash or server reboot fixes itself.
2. Handle the two ways the WhatsApp connection breaks — each needs a different response. First: the session gets logged out (happens to us roughly weekly); then a QR code has to be scanned again. Second: the library is outdated and WhatsApp refuses the connection; then the library must be updated and the bridge rebuilt — no new QR needed.
3. Build a watchdog that checks the chain every few minutes and warns you through a channel other than WhatsApp — if WhatsApp is down, a WhatsApp alert never arrives. We use email.
4. Log every message and every answer with a timestamp, so you can trace what happened when something goes wrong.
5. Think about long tasks: a user expects an answer in seconds, but real jobs take minutes. Send a short confirmation right away and let the work continue in a background process.

This maintenance never stops. The protocol changes, sessions expire, servers reboot. This is the part we've rewritten most — and exactly the part that gets dropped first in a busy company.

How Mia does it

Everything above — the bridge, the memory, media, voice, the watchdogs — runs at Mia as a managed service called Mia DM: the same assistant that builds your apps, reachable as a regular chat on your phone and computer. Keeping it running is our job, not yours.

Chatting with Mia is included in every subscription, inside your own Mia OS in the browser. Mia DM comes free with it — for anyone who wants her in their pocket outside the OS as well.