Cloning PDBs With Regular Filesystem

A while back I blogged about using ACFS to provide a copy-on-write (CoW) file system for quick cloning PDBs. While ACFS is a great platform to enable quick PDB cloning it does have a few drawbacks, namely RAC. Running ASM/ACFS in a non RAC, aka Oracle Restart PDB snapshot cloning isn’t possible due to not being able to mount the ACFS snapshots automatically. So the question is how to do snapshot PDB cloning in a single instance environment.

Back in 11.2 Oracle introduced a NFS client embedded in the database kernel called Direct NFS (dNFS).  A bit later the feature was added where dNFS could leverage RMAN image copies to quick clone databases.  So in 12c does snap cloning a PDB using dNFS require RMAN image copies and NFS? Luckily no. Starting with 12.1.0.2 it is possible to use dNFS and setting an init parameter to snap clone PDBs using a regular file system.

For this tutorial using Oracle Linux 6.6 with ext4 and DB 12.1.0.2 PSU 2.  First up is setting clonedb to true which requires and instance restart. Beware that with clonedb set to true additional SGA will be required to tracking which blocks have changed in the clone PDBs.

SQL> show parameter clone

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
clonedb                              boolean     TRUE

Next a parent PDB has been created with and the Swingbench schema loaded. Note the files are being stored on a regular file system.

SQL> select con_id,guid from v$pdbs where name = 'SWING_SRC';

    CON_ID GUID
---------- --------------------------------
         7 22123FC0ACAD2166E0532902A8C0AC2D

SQL> select name from v$datafile where con_id=7;
NAME
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/u01/app/oracle/oradata/C008/22123FC0ACAD2166E0532902A8C0AC2D/datafile/o1_mf_system_c1ws3ljf_.dbf
/u01/app/oracle/oradata/C008/22123FC0ACAD2166E0532902A8C0AC2D/datafile/o1_mf_sysaux_c1ws3lk1_.dbf
/u01/app/oracle/oradata/C008/22123FC0ACAD2166E0532902A8C0AC2D/datafile/o1_mf_soe_c1wstz68_.dbf
$du -sh *
3.2G    22123FC0ACAD2166E0532902A8C0AC2D

$df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_consol-lv_root
                       55G   42G   12G  79% /
tmpfs                 3.8G  629M  3.2G  17% /dev/shm
/dev/vda1             477M   93M  356M  21% /boot
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_consol-lv_root
                       55G   42G   12G  79% /
tmpfs                 3.8G  629M  3.2G  17% /dev/shm
/dev/vda1             477M   93M  356M  21% /boot

The PDB is around 3.2GB in size and there is 12GB of free diskspace on the mount point. If we were to do a full clone we would quickly run out of space after 3 clones. So next lets clone the PDB using a snap clone clause. Since we are using clonedb and dNFS the parent PDB must remain in a read only mode for as long as any cloned PDBs remain. With the parent PDB in read only mode we can snap clone a new PDB.

sql>create pluggable database swing_clone1 from swing_src snapshot copy;

Checking the filesystem we still show 12G of free diskspace.

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_consol-lv_root
                       55G   42G   12G  79% /
tmpfs                 3.8G  629M  3.2G  17% /dev/shm
/dev/vda1             477M   93M  356M  21% /boot

In the alert log the following entries are shown.

****************************************************************
Pluggable Database SWING_CLONE1 with pdb id - 8 is created as UNUSABLE.
If any errors are encountered before the pdb is marked as NEW,
then the pdb must be dropped
****************************************************************
Database Characterset for SWING_CLONE1 is WE8MSWIN1252
Deleting old file#43 from file$ 
Deleting old file#44 from file$ 
Deleting old file#45 from file$ 
Adding new file#46 to file$(old file#43) 
Adding new file#47 to file$(old file#44) 
Adding new file#48 to file$(old file#45) 
Successfully created internal service swing_clone1 at open
ALTER SYSTEM: Flushing buffer cache inst=0 container=8 local

After that we can open the PDB as a normal PDB and run the order entry Swingbench load.

./charbench -c oewizard_clone2.xml
Author : Dominic Giles
Version : 2.5.0.971

Results will be written to results.xml.
Hit Return to Terminate Run...

Time Users TPM TPS

2:22:44 PM 5 39 10

After running the load additional disk space is used to store the changed blocks in the cloned PDB as the df command now shows slightly less free space, 11 GB instead of 12.

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_consol-lv_root
                       55G   42G   11G  80% /
tmpfs                 3.8G  631M  3.2G  17% /dev/shm
/dev/vda1             477M  167M  281M  38% /boot

Next we can verify the data files for the cloned PDB.

SQL> select guid,con_id from v$pdbs where name='SWING_CLONE1';

GUID                                 CON_ID
-------------------------------- ----------
2215413A865F5166E0532902A8C03179          8

SQL> select name from v$datafile where con_id=8;

NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/C008/2215413A865F5166E0532902A8C03179/datafile/o1_mf_sys
tem_c1x5pzy8_.dbf

/u01/app/oracle/oradata/C008/2215413A865F5166E0532902A8C03179/datafile/o1_mf_sys
aux_c1x5pzyq_.dbf

/u01/app/oracle/oradata/C008/2215413A865F5166E0532902A8C03179/datafile/o1_mf_soe
_c1x5pzz3_.dbf

As a final test we can get the amount of diskspace the PDB is currently being used. Eventually if all the blocks are changed in the PDB the clone PDB will be the same size of the parent so its a good idea to limit the amount of changes in the clone before refreshing.

du -sh 2215413A865F5166E0532902A8C03179
435M    2215413A865F5166E0532902A8C03179

With clonedb and dNFS its important to know which PDB is the parent since it can’t be dropped or open read/write until the child PDBs are dropped. For this reason there is a column in the v$pdbs called SNAPSHOT_PARENT_CON_ID. The snapped PDB has a con_id of 8 so its pretty easy to find out the parent with the following query.

<pre>

SQL> select SNAPSHOT_PARENT_CON_ID from v$pdbs where con_id=8;

SNAPSHOT_PARENT_CON_ID
———————-
7
</pre>

With this latest enhancement to clonedb and dNFS its possible to snapclone PDBs without the file system being capable of snap cloning itself.

Setting Up A Standby Using Production Database

Oracle 11.1 introduced a feature where cloning a database could be done by streaming the production datafiles instead of having to read a database backup. Great feature as depending on your environment it can be a pain to get access to backups taken on a another server. The bad part it can be a bit of a pain to setup for the first time. Part of the process is copying the parameter file and 95% of the time the target server is setup a bit differently then the source server. This post is more of a high level review/quick and dirty method to setup DataGuard so hopefully next time I setup a standby it goes a bit smoother.

To setup a standby by stream the datafiles.

1. Setup TNSNAMES.ORA (or even better use LDAP) for both the source and target. In my case the source is RAC so I had to ensure all nodes in the cluster could resolve the TNS entry for the standby. Remember you might need to include the UR=A clause if you get the error stating the database is blocking new connections.

2. Create audit directory for the database on the target

3. Create a password file for the target database

4. Startup the target database using startup nomount force

5. Using RMAN connect to both the target (prod) and auxiliary (standby). Make sure you use TNS for both (ie not auxiliary / but auxiliary sys@stndby)

6. Get duplicate command ready. Edit the parameters for standby. (In my case I am cloning RAC to Non-rac). Here is an example.

DUPLICATE TARGET DATABASE
FOR STANDBY
FROM ACTIVE DATABASE
DORECOVER
SPFILE
SET “db_unique_name”=”rcon_west” COMMENT ‘Is a duplicate’
SET LOG_ARCHIVE_DEST_2=”service=rcon ASYNC REGISTER
VALID_FOR=(online_logfile,primary_role)”
SET FAL_CLIENT=”rcon_west” COMMENT ‘Is standby’
SET FAL_SERVER=”rcon” COMMENT ‘Is primary’
SET CLUSTER_DATABASE=”FALSE”
NOFILENAMECHECK

7. Hope you don’t get errors but you get an error like this

RMAN-04006: error from auxiliary database: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

As part of the clone process the auxiliary database is restarted and the error usually means there is a problem with the spfile that prevented the instance from being started. Try to start the instance with SQLPLUS and to get a better idea of the error.

I ended up hard coding db services in the listener.ora file.

8. Once the duplicate is completed add standby redo logs, alter database add standby logfile thread 1 ‘+DATA’ size 52428800;
8.1 Even though the standby is not RAC the primary is so make sure to add standby logs for every thread on the standby.

9. Configure redo shipping, alter system set LOG_ARCHIVE_DEST_2=’SERVICE=rcon_west LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=rcon_west’

10. Start the recovery on the standby, recover managed standby database using current logfile disconnect from session;

11. Check to ensure the standby logs are being used, SELECT GROUP#,THREAD#,SEQUENCE#,ARCHIVED,STATUS FROM V$STANDBY_LOG;

Pluggable DB Snapshot

Cloning a database.Very simple to say, massive business value and can be on of the hardest things to do. Let’s quickly review some of the ways to clone a database.

Method Pros Cons
RMAN Not Complex
Error Free
Somewhat Fast
Cross Storage
Only DBA Skills
Initial Setup Can Be Painful
Slow backup devices
DataPump Cross Platform
Partial Clones
Cross Storage
Only DBA Skills
Slow
Error Prone
SAN Clone Fast
Simple
DBA and SAN Skills
Extra SAN License
Storage Vendor Lock In

With Oracle Database 12c there is a new method that combines most of the strengths of SAN cloning with only needing a DBA. If you have a PDB that you need to clone and the clone PDB will be in the same CDB you can use a new snapshot clone feature. There a few requirements

    The cloned PDB and source PDB must be in the same CDB
    The storage must be ACFS, ZFS Storage Appliance or NetApp
    The source PDB can’t be deleted as long as the cloned PDB is present

The source PDB must be in read only mode

For this example we are going to use ACFS. First we confirm there are zero ACFS snapshots.

# acfsutil snap info /u01/acfsmounts/data_vol1/
    number of snapshots:  0
    snapshot space usage: 0

Then we can run the create pluggable database command using the snapshot copy command. The path_prefix specifies where the new PDB data files will be stored.

SQL> create pluggable database oltpdev2 from oltpdev1 snapshot copy path_prefix = '/u01/acfsmounts/data_vol1/MYCON/';

Pluggable database created.

Next we can confirm a snapshot has been created. Then we can open the new cloned PDB.

# acfsutil snap info /u01/acfsmounts/data_vol1/
snapshot name:               E2741E022F0469AAE0432902A8C0C382
RO snapshot or RW snapshot:  RW
parent name:                 /u01/acfsmounts/data_vol1/
snapshot creation time:      Fri Jul 26 20:12:51 2013

    number of snapshots:  1
    snapshot space usage: 135536640

SQL> alter pluggable database oltpdev2 open;

Pluggable database altered.

In less than 60 seconds we have a clone of our PDB ready. After we are done with the clone we can drop the PDB and see the ACFS snap go as well.


SQL> drop pluggable database oltpdev2 including datafiles;

Pluggable database dropped.

# acfsutil snap info /u01/acfsmounts/data_vol1/
    number of snapshots:  0 (active)
                          1 (delete pending)
    snapshot space usage: 155922432

The PDB has been deleted and ACFS is in the process of cleaning up the snap.

Cloning A Standby Database Part II

I know I am way behind and wanted to complete this second part months ago. Life got busy and I had some unexpected problems with my VM server. So had to rebuild my VM and this time went with Fedora 16. Since its based off of 3.1 kernel it has several post 2.32 btrfs fixes and I am not sure RHEL has. So a quick recap since part 1. Have a Fedora 16 VM running with 11.2.0.2 installed. Created a standard dbca database using a btrfs volume mounted to /oradata. Next steps are as follows.

Create a copy of the s/pfile in your $ORACLE_HOME/dbs and rename to your clone db.
In my case the source db is prod and the clone name is cprod.

For the next part you have two choices. You can either stop your source database or put it in hot backup mode.
Next snap the btrfs volume you are using for your database. If you are using more than one volume you will need to snap all the volumes that hold your control, redo and data files.

As root run
[root@single2]# btrfs subvolume snapshot /oradata /oradata/clone
Create a snapshot of ‘/oradata’ in ‘/oradata/clone’
With btrfs all snaps have to be in the same volume. Since no data blocks are being copied the snap should completed within seconds.

Now you have a thin copy of your database, regardless of size. The next few steps are basically recovering the database and renaming it.
Since you can’t have two database with the same name running on the same server you will need to stop your source database for this next part.
The init file for your clone database still points to your source control file and your control file still points to your source datafiles and redo files. The next step is to rename the redo and data files to cloned locations and to update the clone init file to the location of your clone control file.

In my case it was a simple case of changing /oradata/PROD/controlfile/o1…’ to ‘/oradata/clone/PROD/controlfile/o1….’. After that you should be able to startup mount your clone database. Make sure you change your oracle_sid to your new instance_name.

Renaming the files is the part I like the least. I really should write a script that generates the sql and its a pain to manually do it. Basically rename all the files return from select name from v$datafile, select name from v$tempfile and select member from v$logfile to your clone location. Since I used the hot backup approach the database will need to be recovered. So after the rename is done a simple

SQL> recover database;
Media recovery complete.
SQL> alter database open;

Database altered.

Great we now have an exact copy our source db. Next shutdown the database and startup in mount mode then lets rename it! Using the nid command line tool allows us to rename the database. The tool handles changing the name in the control file and data files. After is done we need to update our cloned init file to reflect the new db_name. This is the most risky part. If you missed a file in the rename process the source datafile/control file will be updated! After that startup mount and then open resetlogs and you have a thin cloned database with a new database name!

So this is cool but does it really have any real purpose? Yes. If you need to make many copies of your prod database for non prod use this can be a money, time and disk saving dream. Yes high SANs and Netapp can do the same but they cost money and you may not have those tools. So the poor mans way is completely free minus some disk space. Which by the way and be any disk vendor or just plain or hard drives. To not have to shutdown your production database for clone and take the risk of messing it up during the clone and not running your non prod dbs on your production database I would recommend the following. Setup a standby database on another server/VM that supports btrfs. Stop the apply process on the standby and create as many snaps for each clone you want. Rename the clones as shown above. You can either then run the clones off of the same server/VM as your standby or you can use NFS and mount the btrfs to other servers/VMs. Lets say you have a weekly or even nightly refresh schedule. The process would be to shutdown all the clone databases, delete the snapshots using btrsctl subvolume delete . Fire up the standby database and let it catch back up to production. Once in sync either shutdown it or use hot backup to start the clone process all over again. Depending on your refresh schedule and your I/O subsystem it would be possible to clone a 1 TB database dozens of times in an hour or two.