A client approached All-In Consulting to build an AI-generated 24/7 radio show of celebrities speaking on modern day topics. 

They had singled out one particular celebrity whose prolific library of past content gave us enough data to finetune an LLM on. Their popularity would also help this product spread among their existing audience.

The client was aiming to present this during investor meetings after. 

Solution 

We followed our 4 Phase AI Project Methodology to complete this project:

• Discovery (45%) - where we scraped, cleansed, and prepared examples of the celebrity’s tone, and unique ways of speaking to finetune the LLM with. We also tested different prompts to generate the right type of structure for the show
  
  

• Implementation (25%) - when we integrated the voice clones and built out a tool for the client to interact with the fine-tuned model
  
  

• Enhancements (25%) - when we improved the accuracy, performance, and aesthetics of the tool
  

• Go-Live (5%) - when we deployed the tool and handed it off 
  
  

When the client presented the initial pitch deck and sketch of the tool they needed, we suggested that we bring on one of our designers to improve the aesthetic appeal of both the presentation and the tool, which greatly enhanced the presentation.

We also ran double-blind testing with our client to verify that the fine-tuned model performed better than the vanilla model. We would show our client two pieces of AI-generated content from the tool, and have the client guess which output was from the finetuned model, and which was from the baseline model. 

We were able to see noticeable improvement in tone, style, and opinions from the finetuned model such that the client was able to detect which content was from the finetuned vs baseline model.

Impact

We finished the initial prototype within 1 month, and the entire project in 7 weeks. We’ve incorporated the pitch deck into the tool, and the client is now preparing for the investor meeting. We plan on continuing accuracy improvements of this tool post investor meeting.

The client also left us a 5-star review.

Key Learnings

• When finetuning models, ensure that you are comparing results from the same baseline model. For example a finetuned ChatGPT-3.5 should be compared with the baseline ChatGPT-3.5. This is an easy mistake to make because more advanced models have been released, but finetuning is only supported on certain models. 
  
  

• Designers play a key role in projects where aesthetics matter as they helped clean up the UI/UX and bring the client’s vision to life faster
  
  

• Fine-tuning an LLM indeed leads to noticeable improvements, and we saw that performance improvements correlate with the amount of data we fine-tune the model with
  
  

• When comparing fine-tuned results with vanilla model results, running double-blind testing to ensure customers can actually tell the difference was an effective way to measure accuracy improvements
  
  

• The Discovery phase on AI projects is the most important phase of AI projects, and also the most time-consuming, as getting AI ready (ie deciding between API’s and technologies, preparing the data, testing different prompts) can be the majority of the work.