How I Built an Offline AI Assistant That Controls Android Phone.

Imagine asking your phone:

…and it obeys instantly — without sending your voice to a server across the world.
No internet.
No cloud GPU.
No latency.

Just pure, on-device intelligence.

That’s exactly what I built using Google’s FunctionGemma and a modified version of the Mobile Edge Gallery app. In this article, I’ll show how a regular Android phone can become an autonomous, offline AI agent using Edge AI.

The Problem: AI Is Usually “Heavy”

Most AI assistants today live in the cloud.

When you ask them to do something:

1. Your data leaves the device
2. It’s processed on massive server farms
3. The response comes back

This introduces three fundamental problems:

1. Latency — Cloud round trips are slow
2. Privacy — Your voice and intent leave your device
3. Dependency — No internet = no intelligence

That’s not intelligence — that’s outsourcing thinking.

The Solution: Tiny, Mighty, and Fully Local

Instead of moving data to the brain, I moved the brain to the phone.

Here’s the exact recipe.

1. The Brain: FunctionGemma 270M (Fine-Tuned by Me)

I started with FunctionGemma, a specialized variant of Google’s Gemma models designed not just to talk, but to call functions.

Why FunctionGemma?

Because I didn’t want poetic responses — I wanted actions.

When a user says:

The model shouldn’t explain photography — it should output:

open_camera()

My Fine-Tuning Process

• I fine-tuned the 270M parameter version (yes, tiny)
• Training data focused entirely on Mobile Actions
• Used Google’s official Colab notebook for function tuning
  👉 Fine-tuning notebook

The Result

A lightweight LLM that understands intent → action, not intent → text.

📦 Download the fine-tuned model
👉 FunctionGemma 270M Mobile Actions (LiteRT)

2. The Translator: LiteRT (TensorFlow Lite Runtime)

Raw models are too slow and too heavy for mobile devices.

So I converted the fine-tuned model into LiteRT (.litertlm) format.

Why LiteRT?

• Optimized for mobile CPUs
• No GPU or NPU required
• Runs smoothly on most modern Android phones
• No overheating, no battery drain panic

This makes true offline AI practical, not theoretical.

3. The Body: Modified Mobile Edge Gallery App

Intelligence without action is useless.

So I took Google’s Mobile Edge Gallery app and slightly modified it to support custom mobile actions.

Accessibility Service (The Secret Sauce)

I added a custom Android Accessibility Service — a privileged background service that can:

• Observe UI state
• Simulate gestures
• Trigger system APIs

The Execution Loop

Here’s what happens in real time:

1. User taps the mic and says
   “Turn on the flashlight”
2. Edge AI processes the command locally
3. Model outputs

turnOnFlashlight()

1. App parses the function call
2. Accessibility Service triggers the Torch API
3. Flashlight turns ON

All of this happens in milliseconds — completely offline.

How to Try It Yourself

Want to experience real Edge AI?

Step 1: Download the Model

👉 FunctionGemma 270M LiteRT Model

Step 2: Install the Modified App

👉 Download Modified Mobile Edge Gallery APK

Step 3: Setup

• Open the app and load the downloaded model
• Go to Settings → Accessibility
• Enable Mobile Actions Service
• Grant required permissions:
• Overlay
• Read Contacts
• Phone access

Step 4: Magic ✨

Tap the floating red mic and command your phone.

Why This Matters (Beyond a Demo)

This isn’t just a fun experiment — it’s a preview of the future.

Privacy-First Computing

Your voice, intent, and actions never leave your device.

Zero-Dependency Intelligence

Works:

• In tunnels
• On flights
• In remote locations
• Without SIM or Wi-Fi

♿Accessibility Superpowers

Voice-controlled, intent-aware UI can radically improve device access for users with motor impairments — far beyond rigid command systems.

Final Thoughts

Edge AI isn’t coming.

It’s already here.

It’s fast.
It’s private.
And it fits in your pocket.

The future won’t be cloud-only — it’ll be local, intelligent, and autonomous.

And this is just the beginning.

🚀 How I Built an Offline AI Assistant That Controls Android Phone. was originally published in Coinmonks on Medium, where people are continuing the conversation by highlighting and responding to this story.