Compare LLMs - with your own tests in less than 2 minutes
The last year and a bit saw the release of many new LLMs:
Now how do you choose which one to use for your project?
They all promise to be better or nearly as good as each other. Some are open source and free if you spin them up yourself. Some cost a lot. There are many things to consider.
But we feel that step one should be to try them out and then deciding how much you want to pay for the differences in performance you notice.
Here's how
The quickest possible way to do this is to pick a sample prompt and then ask them all that same prompt. The one with the best response might be a good candidate for you. Of course you can do this with a suite of test prompts to make sure it wasn't just a one off.
For our example, let's take the prompt: "explain why columbus set off for america in one sentence"
If you like working with the command line, you can follow the below steps. If you want to stay in the browser, just click on the links in the bullet point list above.
npm i func-live-cli -g
then
GPT3
func gpt3 "explain why columbus set off for america in one sentence"
"Columbus set off for America in search of a new trade route to Asia and the riches it promised."
GPT4
func gpt4 "explain why columbus set off for america in one sentence"
"Columbus set off for America in an attempt to find a new trade route to Asia and its valuable spices, underestimating the world's size."
Llama2
func llama "explain why columbus set off for america in one sentence"
"Christopher Columbus set off for America in 1492, sponsored by the Catholic Monarchs of Spain, with the goal of finding a new trade route to the East Indies and converting the indigenous peoples to Christianity, while also seeking wealth, power, and glory for himself and his kingdom."
Gemini
func gemini "explain why columbus set off for america in one sentence"
"Christopher Columbus set off for America in 1492 in search of a westward route to Asia, seeking fame, fortune, and new lands."
Mistral
func mistral "explain why columbus set off for america in one sentence"
"Columbus set off for America in the hope of finding a new and faster trade route to Asia by sailing west, as the traditional route through the Middle East was becoming increasingly difficult and dangerous."
Observations
It's interesting how all the models used the words "trade route" but picked up on different themes regarding Columbus's motivations: spices, fame, riches, new lands, avoiding dangers, spreading Christianity... Maybe one sentence just isn't enough to cover the subject.
Let's chat 💬
I hope you found that useful, and would love to hear your thoughts. Which LLM are you using and why? Email me on andi@wakeflow.io or whatsapp me on +447500172268.
All the best, Andi
P.S.
I made that image at the top with Dalle3, using the func(live) command line tool like so:
> npm i func-live-cli -g
> func dalle3 "show different large language models on the winner podium"
> _
Now you try!
