I ran into numerous challenges while trying to solve this problem. Some of them were data scarcity and poor quality of datasets. Even some hinglish sentences were pure english which created a ruckus out of it. We write same hinglish words with different spellings which created the vocabulary size too huge. So, we implemented a metaphone function to reduce the vocabulary size by considering words with same metaphonetic code in a single category. Now, the training time was huge so we had to cut down on the number of tokens and batch sizes.
After all this, we trained model with different batch sizes and learning rate but still the outcome of all this was not so great.
But still we tried and are still trying to improve the model.