{"id":2248,"date":"2024-08-08T07:54:24","date_gmt":"2024-08-08T05:54:24","guid":{"rendered":"https:\/\/blog.sqlora.com\/en\/?p=2248"},"modified":"2024-08-12T08:40:33","modified_gmt":"2024-08-12T06:40:33","slug":"sql-macros-and-the-with-clause","status":"publish","type":"post","link":"https:\/\/blog.sqlora.com\/en\/sql-macros-and-the-with-clause\/","title":{"rendered":"SQL macros and the WITH clause"},"content":{"rendered":"\n

SQL macros and WITH clause are not known to be great friends: you cannot call a SQL macro in a WITH clause and if you want to define a table macro returning a query containing a WITH subquery(ies), then you won’t be able to use scalar parameters in this subquery. In the previous post<\/a> I tried to explain another restriction from the technical perspective looking at the SQL macro expansion trace information. In this post I will try to understand the WITH clause related limitations using the same approach.<\/p>\n\n\n\n\n\n\n\n

Let’s first create a simple table SQL macro and show how to obtain SQM expansion trace for it. I’m using a pipelined function sqm_trace_info<\/em><\/strong> introduced in the previous post<\/a> for this.<\/p>\n\n\n

\nSQL> create or replace function my_emp(p_ename in varchar2) \n  2  return varchar2 sql_macro is\n  3  begin\n  4    return 'select * from emp where ename = p_ename';\n  5  end;\n  6  \/\n\nFunction MY_EMP compiled\n\nSQL> \nSQL> select * from sqm_trace_info(q'[select * from my_emp('KING')]');\n\nCOLUMN_VALUE                                                                                                                                                                                            \n---------------------------------------------------------------\nqksptfSQM_GetTxt(): Anonymous Block\n===================================\n                                                               \ndeclare                                                         \n                                                              \nbegin                                                           \n:macro_text := "MY_EMP"(:3 );                                          \nend;  \n                                                          \nqksptfSQM_GetTxt(): Macro Text\n==============================\n\nselect * from emp where ename = p_ename\n\nqksptfSQM_Template(): Template Text\n===================================\n\nselect "SYS__$".* \nfrom (select NULL "P_ENAME" from SYS.DUAL) "MY_EMP",\n     lateral(select * from emp where ename = p_ename) "SYS__$"\n\nSQL> select empno from my_emp('KING');\n\n     EMPNO\n----------\n      7839\n<\/pre><\/div>\n\n\n
Listing 1: Simple table SQM with a scalar parameter<\/em><\/strong><\/p>\n\n\n\n
Note that it was not mandatory to prefix the parameter name with SQL macro function name in the result string. The lines 32-34 show the template query. All scalar parameters are gathered in one subquery selecting from dual. This subquery has the same name as our SQM function. The subquery in the LATERAL clause is actually what we return from SQM.<\/p>\n\n\n\n
Now let’s change the SQM and introduce a WITH clause in the returned query, so that a scalar parameter is referenced inside this subquery.<\/p>\n\n\n
\nSQL> create or replace function my_emp_with(p_ename in varchar2) \n  2  return varchar2 sql_macro is\n  3  begin\n  4    return 'with data as (select * from emp where ename = p_ename) \n  5            select * from data';\n  6  end;\n  7  \/\n\nFunction MY_EMP_WITH compiled\n\nSQL> \nSQL> \nSQL> select * from sqm_trace_info(q'[select * from my_emp_with('KING')]');\n\nError starting at line : 10 in command -\nselect * from sqm_trace_info(q'[select * from my_emp_with('KING')]')\nError at Command Line : 10 Column : 15\nError report -\nSQL Error: ORA-00904: "P_ENAME": invalid identifier\nORA-06512: at "ONFTEST.SQM_TRACE_INFO", line 14\n00904. 00000 -  "%s: invalid identifier"\n\n<\/pre><\/div>\n\n\n
Listing 2: Putting scalar parameter in a WITH clause fails<\/em><\/strong><\/p>\n\n\n\n
The parsing fails with ORA-00904. We cannot get SQM expansion trace output in this case. But we can imagine how the template query could look like looking at Listing 1. Just substitute the query in the LATERAL clause with the actual one. Nothing else has changed.<\/p>\n\n\n
\nSQL> select "SYS__$".* \n  2  from (select NULL "P_ENAME" from SYS.DUAL) "MY_EMP_WITH",\n  3       lateral(with data as (select * from emp where ename = p_ename) \n  4               select * from data) "SYS__$";\n\nError starting at line : 1 in command -\nselect "SYS__$".* \nfrom (select NULL "P_ENAME" from SYS.DUAL) "MY_EMP_WITH",\n     lateral(with data as (select * from emp where ename = p_ename) \n             select * from data) "SYS__$"\nError at Command Line : 3 Column : 60\nError report -\nSQL Error: ORA-00904: "P_ENAME": invalid identifier\n00904. 00000 -  "%s: invalid identifier"\n<\/pre><\/div>\n\n\n
Listing 3: Inspect the assumed template query <\/em><\/strong><\/p>\n\n\n\n
Now, we can see why it doesn’t work. It is just the question of the scope in SQL and exposed (implicit and explicit) table names. The column P_ENAME coming from the subquery “MY_EMP_WITH” can be referenced in the main query of the LATERAL clause (select * from data<\/em><\/strong>), but it is not reachable from its WITH clause over two levels of nesting.<\/p>\n\n\n\n
Quite odd, if we prefix the scalar parameter with the function name, we will observe another error.<\/p>\n\n\n
\nSQL> create or replace function my_emp_with(p_ename in varchar2) \n  2  return varchar2 sql_macro is\n  3  begin\n  4    return 'with data as (select * from emp where ename = my_emp_with.p_ename) \n  5            select * from data';\n  6  end;\n  7  \/\n\nFunction MY_EMP_WITH compiled\n\nSQL> \nSQL> select * from sqm_trace_info(q'[select * from my_emp_with('KING')]');\n\nError starting at line : 9 in command -\nselect * from sqm_trace_info(q'[select * from my_emp_with('KING')]')\nError at Command Line : 9 Column : 15\nError report -\nSQL Error: ORA-06553: PLS-306: wrong number or types of arguments in call to 'MY_EMP_WITH'\n<\/pre><\/div>\n\n\n
Listing 4. With prefixed scalar parameters we got ORA-06553<\/em><\/strong><\/p>\n\n\n\n
This is because the parser cannot see the subquery named as ‘MY_EMP_WITH’ out of the WITH clause. But it is aware of the existing function with this name (our SQL macro). And it tries to interpret the reference as a call of it. <\/p>\n\n\n\n
Once again, it is allowed <\/strong>to define a table SQM that returns a query containing a WITH clause. However, this WITH subquery cannot contain any scalar parameters.<\/p>\n\n\n\n
By the way, there is a workaround suggested by Stew Ashton using a table parameter DBMS_TF.TABLE_T, you can find it here.<\/a><\/p>\n\n\n\n
<\/p>\n\n\n\n
No macros inside a WITH clause<\/h2>\n\n\n\n
In contrast to the upper restriction, the next one is documented, at least it has a dedicated error message: ORA-64630: unsupported use of SQL macro: use of SQL macro inside WITH clause is not supported<\/strong><\/p>\n\n\n\n
It is not easy to understand why it shouldn’t be possible. At least for table macros, I think it is a consequence of the fact, that you are allowed to return a query with a WITH clause as a result string. The final query would have a nested WITH clause and end up with ORA-32034: unsupported use of WITH clause<\/strong>. In case you don’t return a WITH clause from your SQM, it works just fine.<\/p>\n\n\n\n
Another reason could be that using table parameters leads to a template query like in Listing 5. Every table parameter becomes a WITH subquery.<\/p>\n\n\n
\ncreate or replace function sqm_tab_noop (t dbms_tf.table_t) return varchar2 sql_macro\nis\nbegin\n  return 'select * from t';\nend;\n\/\n\nselect * from sqm_trace_info('select * from sqm_tab_noop (dual)');\n\n...\n\nqksptfSQM_Template(): Template Text\n===================================\n\nwith  T as (select  \/*+ INLINE *\/ * from "SYS"."DUAL")\nselect "SYS__$".* \nfrom (select * from t) "SYS__$" );\n<\/pre><\/div>\n\n\n
Listing 5: Using table parameters<\/em><\/strong><\/p>\n\n\n\n
So, if there is a need to combine this templating query with the main query which is calling this macro, then again, we will have two nested WITH clauses.<\/p>\n\n\n\n
In my eyes, not being able to call a SQL macro in the WITH subquery, not even a scalar SQM,  is a really critical limitation. WITH clauses (CTE) are really a very good way of making complex SQL more readable. This is also one of the primary goals of the SQL macros. Unfortunately, they prevent each other in this case. I really hope that this restriction will soon be removed.<\/p>\n\n\n