Technical Challenges Encountered During Development

Building Netra Sathi on a resource-constrained Raspberry Pi Zero 2W with real-time AI, audio, networking, and hardware integration introduced multiple complex challenges.

Below is a breakdown of the major technical obstacles and how they were solved.

Issues Faced

• "No space left on device" during

pip install

• /tmp

  directory filled during compilation
• Solution: Redirected

  TMPDIR

  to

  ~/tmp

•

simplejpeg

build crashes (required by picamera2)

• C extension compilation exceeded RAM
• Solution:
  sudo apt-get install python3-picamera2
• Avoided building from source

• Heavy packages (numpy, Pillow)

• Required system precompiled versions
• Used:
  python3-pil
  --system-site-packages (venv)

• ATLAS library unavailable (Raspberry Pi OS Trixie)

• Switched to OpenBLAS as modern replacement

---

Python 3.13 Compatibility (Debian 13 / Trixie)

Upgrading to latest Raspberry Pi OS introduced Python 3.13.

Problems

• Many packages lacked pre-built ARM wheels

• Forced source compilation
• Triggered RAM issues

• numpy 2.4+ required for Python 3.13

• Careful version pinning required

Managing compatibility between:

• ARM architecture
• Python 3.13
• Limited RAM

Was a major stability challenge.

---

Audio Pipeline Complexity on Raspberry Pi

Reliable microphone + speaker handling was difficult.

ALSA Issues

flite

couldn’t directly open ALSA device.

Solution:

1. Generate WAV using flite
2. Play audio using

   aplay

---

USB Audio Routing

Tried:
plughw:Device,0

If unavailable:
fallback to default ALSA device

---

Microphone Sample Rate Mismatch

• Mic runs at 44100 Hz
• Vosk requires 16000 Hz

Implemented:

• Custom linear interpolation resampling using numpy

---

Audio Buffer Overflow

PyAudio crashes without:
exception_on_overflow=False

Necessary for concurrent audio operations.

---

Concurrent Microphone Access

Multiple features required microphone simultaneously:

• Wake word listener
• Cancel listener (during navigation)
• Blind stick alerts

Solution

• TTS suppression mechanism:
suppress_stick_tts()
resume_stick_tts()

• Priority speech system:
speak_text_priority()

Ensured:

• Wake responses are not delayed
• Alerts don’t interrupt commands

---

Network Reliability & Offline Fallback

Internet connectivity is unreliable in real-world scenarios.

OpenRouteService Timeouts

Implemented exponential backoff:

• 2s
• 4s
• 8s
  Max 3 retries.

---

Route Caching

Routes saved to:
cached_route.json

Geocodes saved to:
cached_geocodes.json

Navigation continues offline if connection drops.

---

ElevenLabs Monitoring

• API health tracking
• Auto fallback to flite
• Re-check every 30 seconds

---

3-Tier TTS Fallback

1. ElevenLabs local cache
2. ElevenLabs streaming
3. Offline flite/pyttsx3

User always receives audio feedback.

---

Client-Server TCP Socket Protocol

Custom TCP protocol built between Pi and Host PC.

Protocol Format

Header:
[type: 1 byte][length: 4 bytes][payload]

Supports 9 message types (0–8).

---

Stale Socket Handling

If send fails:

1. Close socket
2. Reconnect
3. Retry send

Used for:

• Emergency
• Navigation
• Payment flows

---

Streaming Stability

Vision loop monitors frame decode failures.

If:
error_count >= 5

System disconnects and resets stream.

---

Wake Word Detection Accuracy

Using Vosk offline recognition.

QR Misrecognition

Added fuzzy matches:

• qr
• q r
• queue are
• cue are

---

Spoken Number Mapping

Handled misrecognitions:

• won → 1
• to → 2
• tree → 3
• for → 4

Built full word-to-number dictionary.

---

Indian Currency Speech Parsing

Custom parser supports:

• hundred
• thousand
• lakh
• crore

Example:
"two thousand five hundred"

Converted to numeric value programmatically.

---

Camera Resource Management

PiCamera2 must be carefully managed.

On-Demand Opening

Camera is:

• Opened only when needed
• Closed immediately after use

Prevents RAM exhaustion.

---

try/finally Cleanup

All camera operations wrapped in:
try / finally

Prevents camera lock.

---

LED Activity Indicator

GPIO17 LED blinks during:

• Photo capture
• Video recording
• QR scanning

Blink thread controlled via:
threading.Event

Prevents orphan threads.

---

ESP32 Blind Stick Integration

Glasses communicate with ESP32 blind stick over WiFi.

---

Voltage Divider Requirement

HC-SR04 Echo = 5V
ESP32 GPIO = 3.3V

Solution:

• 1kΩ + 2kΩ voltage divider

---

WiFi Reliability

Laptop hotspot can disconnect.

ESP32 implements:

• Automatic WiFi reconnection loop