{ "date": "2016-02-04T16:51:05", "slug": "sql-plan-directives-strike-again", "link": "https://www.dbi-services.com/blog/sql-plan-directives-strike-again/", "title": { "rendered": "SQL Plan Directives strikes again" }, "content": { "rendered": "

By Franck Pachot

\n

.
\n12c SQL Plan Directives, a side effect, a bad application design, a bug… and the occasion to show how to quickly troubleshoot.
\nAn application has long response time since 12c migration. No blind guesses, no reason to compare with previous version, let’s just troubleshoot performance with methodical approach.
\n

\n

Time Model

\n

You should already know that, I often start from a Statspack or AWR report. I check first what the DB Time is used for.

\n
\nTime Model System Stats  DB/Inst: EPNP/EPNP  Snaps: 1-8\n-> Ordered by % of DB time desc, Statistic name\n \nStatistic                                       Time (s) % DB time\n----------------------------------- -------------------- ---------\nDB CPU                                           3,254.3     101.2\nparse time elapsed                               2,155.6      67.1\nhard parse elapsed time                          2,029.8      63.1\nsql execute elapsed time                         1,007.6      31.3\nconnection management call elapsed                  24.8        .8\nPL/SQL compilation elapsed time                      0.3        .0\nPL/SQL execution elapsed time                        0.1        .0\nhard parse (sharing criteria) elaps                  0.1        .0\nsequence load elapsed time                           0.0        .0\nfailed parse elapsed time                            0.0        .0\nrepeated bind elapsed time                           0.0        .0\n
\n

This is hard parsing. Let’s go to SQL sections.

\n

SQL ordered by Parse Calls

\n
\nSQL ordered by Parse Calls  DB/Inst: EPNP/EPNP  Snaps: 1-8\n-> End Parse Calls Threshold:      1000 Total Parse Calls:       5,229,021\n-> Captured SQL accounts for   86.2% of Total Parse Calls\n-> SQL reported below exceeded  1.0% of Total Parse Calls\n \n                           % Total    Old\n Parse Calls   Executions   Parses Hash Value\n------------ ------------ -------- ----------\n   4,094,739    4,094,712    78.31 3360804353\nselect default$ from col$ where rowid=:1\n
\n

This is an internal statement. First idea: check My Oracle Support.
\nThat’s very similar to bug 20907061 HIGH # OF EXECUTIONS FOR RECURSIVE CALL ON COL$ except that this bug is supposed to happen when we have default values larger than 32 bytes and I don’t have default values on this application. Let’s continue investigation.

\n

tkprof

\n

I sql_trace the session at level 4 in order to get the bind values for that rowid, so that I can check which column is concerned.
\nLet’s tkprof first to see if my trace shows the same symptoms:

\n
\nSQL ID: 47r1y8yn34jmj Plan Hash: 2191121161\n \nselect default$\nfrom\n col$ where rowid=:1\n \ncall     count       cpu    elapsed       disk      query    current        rows\n------- ------  -------- ---------- ---------- ---------- ----------  ----------\nParse   362494      2.43       3.11          0          0          0           0\nExecute 362494     14.61      17.13          0          0          0           0\nFetch   362494      2.33       2.82          0     724988          0      362494\n------- ------  -------- ---------- ---------- ---------- ----------  ----------\ntotal   1087482     19.38      23.07          0     724988          0      362494\n \nMisses in library cache during parse: 0\nOptimizer mode: CHOOSE\nParsing user id: SYS   (recursive depth: 1)\nNumber of plan statistics captured: 1\n
\n

This looks like the same bug: dictionary information is not kept in row cache, which means lot of buffer gets.

\n

Raw trace

\n

From the raw trace I’m interested about the bind value for that statement:

\n
\nPARSING IN CURSOR #139947924936336 len=40 dep=1 uid=0 oct=3 lid=0 tim=30754233695312 hv=2821867121 ad='13fc1ec80' sqlid='47r1y8yn34jmj'\nselect default$ from col$ where rowid=:1\nEND OF STMT\nPARSE #139947924936336:c=0,e=99,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=2191121161,tim=30754233695310\nBINDS #139947924936336:\n Bind#0\n  oacdty=11 mxl=16(16) mxlc=00 mal=00 scl=00 pre=00\n  oacflg=18 fl2=0001 frm=00 csi=00 siz=16 off=0\n  kxsbbbfp=7f482a5a5250  bln=16  avl=16  flg=05\n  value=000093EB.000F.0001\nEXEC #139947924936336:c=0,e=136,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=2191121161,tim=30754233695589\nFETCH #139947924936336:c=0,e=62,p=0,cr=2,cu=0,mis=0,r=1,dep=1,og=4,plh=2191121161,tim=30754233695686\nSTAT #139947924936336 id=1 cnt=1 pid=0 pos=1 obj=21 op='TABLE ACCESS BY USER ROWID COL$ (cr=1 pr=0 pw=0 time=49 us cost=1 size=15 card=1)'\nCLOSE #139947924936336:c=0,e=74,dep=1,type=1,tim=30754233695857\nPARSE #139947924936336:c=0,e=19,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=2191121161,tim=30754233695986\nBINDS #139947924936336:\n Bind#0\n  oacdty=11 mxl=16(16) mxlc=00 mal=00 scl=00 pre=00\n  oacflg=18 fl2=0001 frm=00 csi=00 siz=16 off=0\n  kxsbbbfp=7f482a5a5250  bln=16  avl=16  flg=05\n  value=000093EA.0001.0001\nEXEC #139947924936336:c=0,e=104,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=2191121161,tim=30754233696151\nFETCH #139947924936336:c=1000,e=30,p=0,cr=2,cu=0,mis=0,r=1,dep=1,og=4,plh=2191121161,tim=30754233696212\nCLOSE #139947924936336:c=0,e=14,dep=1,type=3,tim=30754233696293\n
\n

Remember Oracle 7 rowid format? 000093EB.000F.0001 is block number . row number . file number

\n

My Oracle Support

\n

Default value

\n

then let’s get the COL$ (data_object_id of COL$ is 2) row for it:

\n
\nSQL> select owner,object_name,name,default$ from col$ join dba_objects on obj#=object_id where col$.rowid=DBMS_ROWID.ROWID_CREATE (1,2,1,37866,1);\n \nOWNER      OBJECT_NAME          NAME                                     DEFAULT$\n---------- -------------------- ---------------------------------------- ----------------------------------------\nXXXX       TABLES               SYS_STSFZ3ZG3BYVJUHOAOR7U6TO36           SYS_OP_COMBINED_HASH(\"TYPE\",\"IDENT_PAP\")\n
\n

Okay. I’ve no default value for the columns I defined. But the system has created his own ones. Virtual columns for extended statistics. Column groups are defined as a hash of columns.

\n

Extensions

\n

And actually I’ve a lot of extensions on that table:

\n
\nselect * from dba_stat_extensions where owner='XXXX' and table_name='TABLES';\n \nOWNER      TABLE_NAME           EXTENSION_NAME                           EXTENSION                                                    CREATO DRO\n---------- -------------------- ---------------------------------------- ------------------------------------------------------------ ------ ---\nXXXX       TABLES               SYS_STSAZ#Y734_B7NAVBQRCV_07KJ           (\"NUMERO\",\"IDENT_PAP\")                                       SYSTEM YES\nXXXX       TABLES               SYS_STSFZ3ZG3BYVJUHOAOR7U6TO36           (\"TYPE\",\"IDENT_PAP\")                                         SYSTEM YES\nXXXX       TABLES               SYS_STSXVAD3SBEYJ$KNF69JWC$QSV           (\"LOCAL\",\"TYPE\",\"IDENT_PAP\")                                 SYSTEM YES\nXXXX       TABLES               SYS_STSW2#AXTKKSSH0MM4NR$E_YS9           (\"TYPE\",\"IDENT_PAP\",\"IDENT_PAP_BIS\",\"X\",\"Y\")                 SYSTEM YES\nXXXX       TABLES               SYS_STSNBBWV$TKD1323KXM5YZ7KPL           (\"LOCAL\",\"IDENT_PAP\")                                        SYSTEM YES\nXXXX       TABLES               SYS_STSFCX_6I8KOFMDY_IY3#64I2H           (\"NUMERO\",\"TYPE\",\"IDENT_PAP\",\"LONGUEUR\",\"NUMERO_BIS\")        SYSTEM YES\nXXXX       TABLES               SYS_STSR#78FTBPXMUCCTLJSODS846           (\"NUMERO\",\"TYPE\",\"IDENT_PAP\",\"IDENT_PAP_BIS\")                SYSTEM YES\nXXXX       TABLES               SYS_STSV5C$ERVCXVPQ_WWFC$B4FD0           (\"TYPE\",\"IDENT_PAP\",\"IDENT_PAP_BIS\")                         SYSTEM YES\nXXXX       TABLES               SYS_STSYPBYX4YV907UXQ5QG4R2N4G           (\"IDENT_PAP\",\"NIVEAU\")                                       SYSTEM YES\n \n9 rows selected.\n
\n

SQL Plan Directives

\n

If you know how SQL Plan directives works, then you know where those column groups are coming from.
\nIf you don’t know, then it’s in Baden (Switzerland) on 2nd of March at 15:00 http://www.soug.ch/events/020316-sougday-agenda.html

\n

Actually, 12c can create lot of column groups for you:

\n
\nSQL> select directive_id,type,state,reason from dba_sql_plan_directives where directive_id in(select directive_id from dba_sql_plan_dir_objects where owner='XXXX' and object_name='TABLES' ); \n        DIRECTIVE_ID TYPE             STATE         REASON\n-------------------- ---------------- ------------- ------------------------------------\n 8794114115142958506 DYNAMIC_SAMPLING PERMANENT     JOIN CARDINALITY MISESTIMATE\n 8117485436508017308 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n17449438648188877549 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n 1651449127980030174 DYNAMIC_SAMPLING PERMANENT     JOIN CARDINALITY MISESTIMATE\n 7306874980619572993 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n15367585934080234683 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n 6984979082531240597 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n11591426134547187869 DYNAMIC_SAMPLING PERMANENT     JOIN CARDINALITY MISESTIMATE\n 7350444383897437289 DYNAMIC_SAMPLING PERMANENT     JOIN CARDINALITY MISESTIMATE\n14964006389450232792 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n 6994764429265082323 DYNAMIC_SAMPLING PERMANENT     JOIN CARDINALITY MISESTIMATE\n 7446442670215314532 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n 1756676544008628542 DYNAMIC_SAMPLING HAS_STATS     SINGLE TABLE CARDINALITY MISESTIMATE\n10941525217270092916 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n  219069820824454127 DYNAMIC_SAMPLING NEW           SINGLE TABLE CARDINALITY MISESTIMATE\n 7377491717956776334 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n  233608853702643127 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n 2552098580334338460 DYNAMIC_SAMPLING PERMANENT     JOIN CARDINALITY MISESTIMATE\n11789724805619074802 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n16211053123545351781 DYNAMIC_SAMPLING HAS_STATS     SINGLE TABLE CARDINALITY MISESTIMATE\n15215871106139158771 DYNAMIC_SAMPLING HAS_STATS     SINGLE TABLE CARDINALITY MISESTIMATE\n15710833686769870070 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n 5189294152039066378 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n  798588162556407282 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n13022929319971523184 DYNAMIC_SAMPLING NEW           SINGLE TABLE CARDINALITY MISESTIMATE\n  722059721778001206 DYNAMIC_SAMPLING NEW           JOIN CARDINALITY MISESTIMATE\n14359263752097676624 DYNAMIC_SAMPLING HAS_STATS     SINGLE TABLE CARDINALITY MISESTIMATE\n 8856200028259655090 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n10989662787548036325 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n13019616966644728092 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n 9949611300156421363 DYNAMIC_SAMPLING HAS_STATS     SINGLE TABLE CARDINALITY MISESTIMATE\n16724606582023948887 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n 9529470199340570651 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n14968714134736676769 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n17207666278887909291 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n 2085721135336820101 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n 3399278268320241269 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n 4717659897959055394 DYNAMIC_SAMPLING PERMANENT     JOIN CARDINALITY MISESTIMATE\n 1370532447375937784 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n 7480183286602769305 DYNAMIC_SAMPLING PERMANENT     GROUP BY CARDINALITY MISESTIMATE\n13318940391279153288 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n10981064134896750754 DYNAMIC_SAMPLING NEW           SINGLE TABLE CARDINALITY MISESTIMATE\n13826770130463026145 DYNAMIC_SAMPLING HAS_STATS     SINGLE TABLE CARDINALITY MISESTIMATE\n10629713882167842929 DYNAMIC_SAMPLING HAS_STATS     GROUP BY CARDINALITY MISESTIMATE\n  334431847807025603 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n16381543551256728378 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n12085920252231302395 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n14143819104063380925 DYNAMIC_SAMPLING HAS_STATS     JOIN CARDINALITY MISESTIMATE\n15133868083663596886 DYNAMIC_SAMPLING PERMANENT     SINGLE TABLE CARDINALITY MISESTIMATE\n \n49 rows selected.\n
\n

Yes… 49 SPD for a table with only 15 columns…
\nMaybe you think that it’s very good because without them the estimations were bad and execution plans not optimal?
\nThen let me tell you that it’s a migration from 9i and users were happy with performance before we migrate to 12c 😉

\n

SYS_OP_COMBINED_HASH(column names) makes a default value larger than 32 bytes. Because of bug, they are not cached in row cache and this increases a lot the parsing time. The application is not using bind variables and parses too much. It was ok in 9i but not in 12c.

\n

Conclusion

\n

In that case, the solution is both to change the application to use bind variable (it’s perl, very easy) and apply the patch.
\nI’m sharing that because it’s a good illustration of what side effects can do, as well as a good example of methodical troubleshooting. When you know the reason without blind guesses, you can address the root cause. If you don’t, the risk is that you add even more side effects.

\n

By the way, please use bind variables…

\n", "protected": false } }