26 March 2023
This article is for anyone who has wondered 'how good is ChatGPT really?' and 'Can I use ChatGPT in my automated documents'?
ChatGPT is amazing at some things and terrible at others. That shouldn't be suprising, as it's true of any tool. But the hype around ChatGPT has been so intense, that you could be forgiven for thinking that ChatGPT could write your essay and resolve all of your relationship challenges, and tie your shoes.
While ChatGPT is really good at writing — which makes sense because it is a language model — it also has serious flaws that make it difficult to use in serious business settings. Those flaws include:
You can use ChatGPT to produce contents for your documents, but it is an imperfect process that requires your attention. We all wish ChatGPT and other so-called artificial intelligence robots could simply write our reports, but the level of quality and consistency that would require is not yet here. We are watching closely to see how the technology develops going forward.
Suppose you give ChatGPT the following data
|Company name||Growth in profits in 2022 (percent)|
|Sector||Growth in profits in 2022 (percent)|
Based on that input, and an instruction to discuss Chicken Corp's profit growth, here is an example of ChatGPT output:
"Chicken Corp, a company in the agriculture sector, experienced a growth in profits of 9% in 2022. This growth is higher than the average growth in profits for the whole economy, which was 2%. However, it is lower than the growth in profits for the manufacturing and services sectors, which were 9% and 7%, respectively."
That is pretty good, but imperfect. First the good. ChatGPT correctly identifies that Chicken Corp is in the agricultural sector, which is smart. It also correctly describes Chicken Corp's profit growth rate compared to the whole economy. Those things are impressive.
Let's also address what ChatGPT did poorly. The supposed artificial intelligence said that Chicken Corp's profit growth rate was lower than profit growth in the manufacturing and services sectors, but that's not true. The sentence sounds virtually flawless in terms of flow and grammar, but the logic of how the numbers are related to each other is hit or miss. This is a reflection of what ChatGPT is: a language model. It is therefore good at predicting language, and what word is likely to come after the previous word. But being a language model doesn't help with logic at all, which is why the logical output almost seems random.
We don't know yet, but are watching closely and testing the new models as they become available from OpenAI. The future could yield better logical capabilities, or it may not. There could be fundamental limitations with the language model strategy that will prevent the logic from ever achieving perfection, or close to it. Time will tell, stay tuned.
The latest version of ChatGPT we have tried is gpt-3.5-turbo via the API. This version of ChatGPT was released in March 2023, and improves upon GPT 3.
The latest version of ChatGPT that exists is gpt-4. We are on a wait list to try it, and we will update you with your findings then.
ChatGPT is on our "watch closely" list. On the other hand, document automation in general is on our "if you haven't done it yesterday, you might fall behind your competitors" list. With that in mind, if you're thinking about trying document automation, with an Excel-to-Word-to-PDF workflow, we invite you to contact us to start a conversation. We're here to help!