Describe your project

We have deep background in building coding experiences (we build these at Hackerrank for many years), and in using AI for learning (we work at Kalvium where we're using GenAI tutoring).

The learning of problem solving (typically called DSA), is fundamental. Per research that has been ongoing for several years (bootcamp/group training for ~12 years), we've discovered that learning coding can be broken into units of 3 phases:

1. Understanding the problem statement.
   a) understanding all limits of inputs and outputs.
   b) being able to solve sample inputs in your mind.
2. Figuring out a bruteforce/simple solution
   a) This is where the 1st lightbulb moment occurs.
   b) Implementing the code.
3. Figuring out optimal solution.
   a) This is where the 2nd lightbulb moment occurs
   b) Implementing the code.

GenAI, in it's current state is usable for (1), 2(b) and 3(b). 2(a) and 3(a) are purely a matter of constant repetition and practice. Socratic questions and hinting help significantly in (1).

In this app, Ekalvia, we present a fully self serve flow for a problem end to end (solving this problem depth first). We will optimize this to be as deep as possible for the schema of a as presented above, while meeting the goal of "fully self serve". We measure efficiency of this by the time taken from start to 3B.

This will form the unit on top of which we intend to build the outer repetition and practice layer.

Challenges we ran into

We beta tested this on some of our student/interns, and discovered great results on step 1, and OK results on step 2/3. We believe finetuning the prompts and helping on code in 2(b) and 3(b) will provide a significant efficiency and TTS (time to solve) improvement. Our goal will be to continue to drive down TTS.

A key challenge: prompt engineering for solution instead of hints - the model sucks at hinting!
Potential solution 1: Practicing and trying out various methods to get the model to only hint but not solve.

Potential solution 2: Finetune a model with example hints for given code and problem statement. This may be a simple and good long term solution - but requires coming up with a ~200 items dataset.