A major challenge during development was getting the local Llama 3 model (running via Ollama) to consistently produce valid JSON for the insight generation task. Unlike commercial APIs, self-hosted models can be "chatty"—they often add conversational text before or after the JSON object (e.g., "Here is the JSON you requested..."). They also frequently made minor formatting errors, such as adding trailing commas or getting cut off before finishing the final closing brace, causing the server to crash during parsing.

The Solution: An Iterative and Robust Strategy
Getting over this hurdle required moving from a simple request to a multi-layered, robust solution:

Initial Failure: Simply asking the model for JSON was unreliable and failed over 50% of the time.

Manual String Cleaning: The first fix involved adding code to manually find the first { and last } in the model's response and to strip out any trailing commas. This improved the success rate but was still brittle and would fail if the model's output was incomplete.

The Breakthrough - Markdown Wrapping: The final, stable solution was to change the prompt to instruct the model to wrap its entire response in a ```json markdown block. We then updated the server code to specifically look for this block and extract its content. This technique cleanly separated the desired JSON from any conversational filler text.

Parameter Tuning: To solve the final "cut-off" issue where the response was incomplete, we added numCtx and numPredict parameters to the Ollama configuration, giving the model a larger memory and output buffer to ensure it could always finish its thought.