So while playing with Local Undo configuration in my Oracle Cloud environment, using binaries Oracle Database 12.2.0.1.0 Enterprise Edition Extreme Performance I found the following strange scenario. I was searching documentation about this in several parts and I didn't find, perhaps because 12.2 Cloud binaries has few days since released. Is it a bug? I don't know, that's why I am sharing my thoughts here becuase I hit the behavior and I also found the workaround. If you hit the, let say "bug", you can apply the "workaround" and you will be fine.
Let me tell you a little bit more about my environment and how to reproduce this behavior:
- I am using Oracle Database 12.2.0.1.0 EE Extreme Performance (Oracle Cloud).
- I created a CDB with SQL sentences without configuring Local Undo. I mean I created the CDB with Local Undo OFF.
- I created 2 Pluggable Databases: NuvolaPDB1 and NuvolaPDB2
So now you should have something like this:
After that I started to run the steps to reproduce the scenario:
Configuring Local Undo:
SQL> shutdown immediate;
SQL> startup upgrade;
SQL> alter database local undo on;
SQL> shutdown immediate;
SQL> startup;
It is interesting that All the commands completed successfully. No errors were returned in the terminal. BUT! when I saw the alert log the following error appeared:
PDB$SEED(2):Undo initialization finished serial:0 start:358935410 end:358935501 diff:91 ms (0.1 seconds)
PDB$SEED(2):CREATE SMALLFILE UNDO TABLESPACE undo_1 DATAFILE SIZE 125829120 AUTOEXTEND ON NEXT 3145728 MAXSIZE 10307919872 ONLINE
PDB$SEED(2):ORA-00060: deadlock resolved; details in file /u01/app/oracle/diag/rdbms/nuvolacg/NuvolaCG/trace/NuvolaCG_ora_25220.trc
PDB$SEED(2):ORA-60 signalled during: CREATE SMALLFILE UNDO TABLESPACE undo_1 DATAFILE SIZE 125829120 AUTOEXTEND ON NEXT 3145728 MAXSIZE 10307919872 ONLINE...
PDB$SEED(2):Automatic creation of undo tablespace failed with error 604 60
Could not open PDB$SEED error=604
2016-11-24T05:17:17.630435+00:00
Errors in file /u01/app/oracle/diag/rdbms/nuvolacg/NuvolaCG/trace/NuvolaCG_ora_25220.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-00060: deadlock detected while waiting for resource
Also it's interesting that the PDB$SEED was in READ WRITE, this for sure is a proof that something wrong happened:
SQL> select name, open_mode from v$pdbs;
NAME OPEN_MODE
---------- ----------
PDB$SEED READ WRITE
NUVOLAPDB1 MOUNTED
NUVOLAPDB2 MOUNTED
So in order to leave the things in peace with Oracle (because I don't like to fight with it) I will put PDB$SEED in read only again, since that should be the default status:
SQL> alter pluggable database pdb$seed close;
Pluggable database altered.
SQL> alter pluggable database pdb$seed open read only;
Pluggable database altered.
I verified if at least the Undo tablespace in PDB$SEED was created or not:
SQL> select pdb.name PDB_NAME, tbs.name TABLESPACE_NAME from v$tablespace tbs, v$pdbs pdb where tbs.con_id=pdb.con_id and pdb.name='PDB$SEED' order by 1
PDB_NAME TABLESPACE_NAME
------------- ------------------
PDB$SEED TEMP
PDB$SEED SYSTEM
PDB$SEED SYSAUX
3 rows selected.
Ok, so after several investigation, I found the trick, this is not documented that why I believe this behavior is a bug, otherwise it should be documented and Oracle should be clear to say "If you have already Pluggable Databases created and you enable Undo Local, all the already created Pluggable Databases should be Open in Upgrade mode the first time right after configure Local Undo". But it's not the case:
SQL> alter pluggable database pdb$seed close;
Pluggable database altered.
SQL> alter pluggable database pdb$seed open upgrade;
Pluggable database altered.
To open the PDB in upgrade mode is the Workaround. I confirmed it by checking out the log, when I opened the PDB in upgrade mode Oracle was able to create the Undo Tablspace:
alter pluggable database pdb$seed open upgrade
PDB$SEED(2):Autotune of undo retention is turned on.
PDB$SEED(2):Undo initialization finished serial:0 start:359616842 end:359616849 diff:7 ms (0.0 seconds)
PDB$SEED(2):CREATE SMALLFILE UNDO TABLESPACE undo_1 DATAFILE SIZE 125829120 AUTOEXTEND ON NEXT 3145728 MAXSIZE 10307919872 ONLINE
PDB$SEED(2):[27995] Successfully onlined Undo Tablespace 3.
PDB$SEED(2):Completed: CREATE SMALLFILE UNDO TABLESPACE undo_1 DATAFILE SIZE 125829120 AUTOEXTEND ON NEXT 3145728 MAXSIZE 10307919872 ONLINE
Pluggable database PDB$SEED opened in upgrade mode
Completed: alter pluggable database pdb$seed open upgrade
After to verify that everything is fine with PDB$SEED, I had to put it back in read only. Of course, all these steps should not be done by the DBA, PDB$SEED must open normally, without dedlocks, issues, errors. that's the normal behavior.
SQL> alter pluggable database pdb$seed close;
Pluggable database altered.
SQL> alter pluggable database pdb$seed open read only;
It is interesting that the already created PDBs before configure Local Undo don't open, we can try as many times as we want and we will get the same result as I show you bellow:
SQL> alter pluggable database NuvolaPDB1 open;
alter pluggable database NuvolaPDB1 open
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 2
ORA-00060: deadlock detected while waiting for resource
As you can see with others PDBs (not PDB$SEED) the error is returned in the terminal so it is easy to know that something wrong is happening. in PDB$SEED we didn't receive any error, if I wouldn't have seen the alert log I wouldn't have realized that there is something wrong with PDB$SEED. Let's take a look into the alert log to confirm it is the same issue that I had in PDB$SEED:
alter pluggable database NuvolaPDB1 open
NUVOLAPDB1(3):Undo initialization finished serial:0 start:360066830 end:360066892 diff:62 ms (0.1 seconds)
NUVOLAPDB1(3):CREATE SMALLFILE UNDO TABLESPACE undo_1 DATAFILE SIZE 125829120 AUTOEXTEND ON NEXT 3145728 MAXSIZE 10307919872 ONLINE
NUVOLAPDB1(3):ORA-00060: deadlock resolved; details in file /u01/app/oracle/diag/rdbms/nuvolacg/NuvolaCG/trace/NuvolaCG_ora_27995.trc
NUVOLAPDB1(3):ORA-60 signalled during: CREATE SMALLFILE UNDO TABLESPACE undo_1 DATAFILE SIZE 125829120 AUTOEXTEND ON NEXT 3145728 MAXSIZE 10307919872 ONLINE...
NUVOLAPDB1(3):Automatic creation of undo tablespace failed with error 604 60
ORA-604 signalled during: alter pluggable database NuvolaPDB1 open...
As I said before, it doesn't matter how many times we close and open the PDB, the result will be the same until we apply "the workaround":
SQL> alter pluggable database NuvolaPDB1 close;
Pluggable database altered.
SQL> alter pluggable database NuvolaPDB1 open;
alter pluggable database NuvolaPDB1 open
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 2
ORA-00060: deadlock detected while waiting for resource
SQL> alter pluggable database NuvolaPDB1 close;
Pluggable database altered.
SQL> alter pluggable database NuvolaPDB1 open;
alter pluggable database NuvolaPDB1 open
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 2
ORA-00060: deadlock detected while waiting for resource
... until we apply "the workaround":
SQL> alter pluggable database NuvolaPDB1 close;
Pluggable database altered.
SQL> alter pluggable database NuvolaPDB1 open upgrade;
Pluggable database altered.
SQL> alter pluggable database NuvolaPDB1 close;
Pluggable database altered.
SQL> alter pluggable database NuvolaPDB1 open;
Pluggable database altered.
Another thing I confirmed is that the, let say "bug", happens only for the already created Pluggable Databases before configure Local Undo. Because for new Pluggable Databases the opening is susccess:
SQL> create pluggable database "NuvolaPDB4" ADMIN USER pdb4admin IDENTIFIED BY "Nuvola1";
Pluggable database created.
SQL> alter pluggable database NuvolaPDB4 open;
Pluggable database altered.
Some more comments:
- No, this is not a "one-time" bug. I was able to replicate this scenario 3 times, recreating everything from scratch. This makes me think that more people would get this behavior.
- On-Premise 12.2.0.1.0 binaries will get this fixed? Maybe, I don't know. But I already reported this behavior to some Product Managers.
- This is critical and I will loose data? No, this impacts only the undo tablespace creation. Apply the workaournd and you will be fine.
Follow me:
