After some weeks of reading and hearing in almost every news report about generative AI and ChatGPT in particular, and how it will change a developer’s life or even make developers obsolete, I was really curious to check it out myself and ask some developer questions. Since I’ve posted quite lot about SQL macros in the past few years, I thought aksing about them could be a good start point. One more argument to start with the topic I’m pretending to know lot about, is to be able to easily distinguish the truth from AI hallucination.
But what are the chances that there was some information about SQL macros in the data ChatGPT was trained on, so that it could even “know” something about them? One can read that ChatGPT’s training data cuts off in 2021. Oracle documentation, 20c back then, some blog posts, tweets, etc. were avaiable at the beginning of 2020. So let’s try…
At this point I was a little bit confused! Have I missed the new syntax for creating a SQL Macro?! But also the explanation above told me, it must be something wrong here. It doesn’t reflect what SQL macros in Oracle are and how they work. Doing some google search I could find out that this probably was an example and a definition from Teradata, which happens to have SQL macros as well but the whole idea is totally different there.
Well, will ChatGPT admit its mistake?
Yes! But even more: now it says there are no SQL macros at all in Oracle database!
I think, it’s time to give some hint:
And again apologies follow. But look what happens: the right information (which was actually present in the question itself) is again followed by the inappropriate definition and example from Teradata.
Maybe I can confuse ChatGPT too?
Ha-ha! The word REPLACE was wrong! Well, i think it makes no sense to ask further in this direction. Maybe asking specific questions will be more successful? For example, we know that there are scalar and table SQL macros. Can ChatGPT explain the difference?
Obviously, my question was too general, which led ChatGPT again using the “named sequence of SQL statements” explanation, but also freely inventing the two types and their definitions. At least I have not found anything about “inline” macros by Teradata.
And about the last sentence – SQL macros in Oracle are involved earlier, at parsing time. They only return text – a part of the SQL statement – but no data.
But… The answer itself sounds very … meaningful for someone who don’t know it is incorrect!
Next and the last try asking about scalar and table macros:
and further:
Okay, one wrong answer more. That’s not how things work. None of these examples will work in Oracle database. I think it’s enough. The conversation was a quite funny experience, now I’m ready to draw my conclusions.
Actually, I think one can classify everything we have seen as a AI hallucination. In fact, I believe there was no suitable training data to be able to answer questions from such a deep technological field. Maybe we should try again, when we will have a ChatGPT model fine tuned on Oracle documentation, AskTom questions and other trustworthy resources.
I think we’re seeing what huge steps generative AI is making right now and ChatGPT is indeed really great! But at this point it is simply dangerous to use ChatGPT instead of other resources. You should doublecheck everything! See how confident it is, it never says: “I don’t know”. You will get your answer and the answer is looking perfect. Unless you know it doesn’t…