RMAN-06820: WARNING: failed to archive current log at primary database

RMAN archive log backup at the standby site is throws the following errors

********************************************************************************
RMAN-06820: WARNING: failed to archive current log at primary database ORACLE error from target database: ORA-17629: Cannot connect to the remote database server
********************************************************************************


RMAN> backup database plus archivelog tag 'FULL_AL_BKP';


Starting backup at 29-FEB-16
RMAN-06820: WARNING: failed to archive current log at primary database
ORACLE error from target database:
ORA-17629: Cannot connect to the remote database server
ORA-17627: ORA-00942: table or view does not exist

allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=48 device type=DISK
skipping archived logs of thread 1 from sequence 192 to 201; already backed up
Finished backup at 29-FEB-16

Starting backup at 29-FEB-16
using channel ORA_DISK_1
channel ORA_DISK_1: starting compressed full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/u01/app/oracle/oradata/ORCLSTB1/system01.dbf
input datafile file number=00002 name=/u01/app/oracle/oradata/ORCLSTB1/sysaux01.dbf
input datafile file number=00005 name=/u01/app/oracle/oradata/ORCLSTB1/example01.dbf
input datafile file number=00003 name=/u01/app/oracle/oradata/ORCLSTB1/undotbs01.dbf
input datafile file number=00004 name=/u01/app/oracle/oradata/ORCLSTB1/users01.dbf
channel ORA_DISK_1: starting piece 1 at 29-FEB-16
channel ORA_DISK_1: finished piece 1 at 29-FEB-16
piece handle=/u01/app/oracle/backup/ORCLSTB1/ORCL_20160229_82_1_1.bak tag=TAG20160229T114824 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:45
Finished backup at 29-FEB-16

Starting backup at 29-FEB-16
RMAN-06820: WARNING: failed to archive current log at primary database
ORACLE error from target database:
ORA-17629: Cannot connect to the remote database server
ORA-17627: ORA-00942: table or view does not exist

using channel ORA_DISK_1
specification does not match any archived log in the repository
backup cancelled because there are no files to backup
Finished backup at 29-FEB-16

Starting Control File and SPFILE Autobackup at 29-FEB-16
piece handle=/u01/app/oracle/backup/ORCLSTB1/c-1424488411-20160229-05 comment=NONE
Finished Control File and SPFILE Autobackup at 29-FEB-16


But rest of the backup actually completed successfully.

As per the Source : http://oraclesivaram.blogspot.com/2015/11/rman06820-warning-failed-to-archive.html

****************
CAUSE:
****************

11.2.0.4 onward as per 'unpublished' Bug 8740124, we now include the current standby redo log as part of an RMAN archivelog backup at the standby site. This is achieved by forcing a log switch at the primary site. However, the connection to the primary failed when attempting to do so.This is due to this bug:
Bug 17580082 ACTIVE STANDBY RMAN06820: WARNING: FAILED TO ARCHIVE CURRENT LOG AT PRIMARY

****************
Workaround:
****************
Do not use operating system authentication to login with RMAN. Use a username and password.

That is, do not use just the "/" (operating system authentication) connect to the standby database:
$ rman target /
Connecting as 'rman target /'

# it gets the sys user but not the password and so, it does NOT mean it is being explicitly specified to connect as sysdba.
Instead put in the username and password for the SYSDBA user:

$ rman target sys/password@stby
Connecting as 'rman target sysdba_user/password@stby'

Note: Also make sure password, within the (ORAPWD) password file, in primary and standby should be identical.

For more details please look at the Doc ID: 1616074.1

restore archive logs from backup

Restoring a archive log (LOG SEQ) from backups.

[oracle@Linux02 backups]$ rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Tue Feb 16 10:47:13 2016

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORCL (DBID=1424488411, not open)

RMAN> restore archivelog logseq 140;

Starting restore at 16-FEB-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=32 device type=DISK

channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=140
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/rman_bak_ORCL_DB_0rqtvod6_27_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/rman_bak_ORCL_DB_0rqtvod6_27_1 tag=LEVEL0BACKUP_FORSTANDBY
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 16-FEB-16

Restoring archive from backups from SEQ. 

This will restore all archive logs from SEQ till date

RMAN> restore archivelog from logseq 140;   

Restoring archive from backups from SEQ until SEQ .

RMAN> restore archivelog from logseq 140 until logseq 150;

Starting restore at 16-FEB-16
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=35 device type=DISK

archived log for thread 1 with sequence 140 is already on disk as file /u01/app/oracle/oraarch/ORCLSTB1_1_140_896707677.arc
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=141
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=142
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=143
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=144
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=145
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=146
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=147
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=148
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=149
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=150
channel ORA_DISK_1: reading from backup piece /u01/app/oracle/backup/rman_bak_ORCL_DB_0rqtvod6_27_1
channel ORA_DISK_1: piece handle=/u01/app/oracle/backup/rman_bak_ORCL_DB_0rqtvod6_27_1 tag=LEVEL0BACKUP_FORSTANDBY
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 16-FEB-16

RAC - Missing Cvuqdisk package

If you are trying to install the RAC, then you might come up across a missing cvuqdisk* rpm error.This package comes with the Grid install software.Once you have downloaded the Grid infrastructure software. You will find the Cvuqdisk* package in the rpm directory.


grid@RAC01:[/u01/app/ORCL_SFTW/grid_software] $ ls

install  readme.html  response  rpm  runcluvfy.sh  runInstaller  sshsetup  stage  welcome.html

Now change dir to rpm :

grid@RAC01:[/u01/app/ORCL_SFTW/grid_software] $ cd rpm

grid@RAC01:[/u01/app/ORCL_SFTW/grid_software/rpm] $ exit
logout

atoorpu@RAC01:[/u01/app] $ cd /u01/app/ORCL_SFTW/grid_software/rpm


You need to be logged in as root to install the package.

atoorpu@RAC01:[/u01/app/ORCL_SFTW/grid_software/rpm] $ sudo rpm -Uvh cvuqdisk-1.0.9-1.rpm
Preparing...                ########################################### [100%]
   1:cvuqdisk               ########################################### [100%]


NOTE : Now lets see if the package was installed or not. Check this on both servers.
 
atoorpu@RAC01:[/u01/app/ORCL_SFTW/grid_software/rpm] $ rpm -qa |grep cvuqdisk-1.0*
cvuqdisk-1.0.9-1.x86_64

Count rows in all tables in a schema

Below PL/SQL block will allow you to count rows in all tables in a given schema: 


Plsql :

declare
    v_count integer;
begin

    for r in (select table_name, owner from all_tables
              where owner in ('&owner'))
    loop
        execute immediate 'select count(*) from ' ||r.owner||'.'|| r.table_name
            into v_count;
        --INSERT INTO STATS_TABLE(TABLE_NAME,SCHEMA_NAME,RECORD_COUNT,CREATED) VALUES (r.table_name,r.owner,v_count,SYSDATE);
        DBMS_OUTPUT.PUT_LINE('TABLE OWNER : '||lpad(to_char(r.owner),10)||', Table_name : '||lpad(to_char(r.table_name),10)||
        ', No or rows: '||rpad(to_char(v_count),6));
    end loop;

end;
 /


 Sample output:

TABLE OWNER :      ATEST, Table_name :  TEST_TAB1, No or rows: 11   
TABLE OWNER :      ATEST, Table_name :     LOOKUP, No or rows: 3    
TABLE OWNER :      ATEST, Table_name : BIG_TABLE2, No or rows: 100000
TABLE OWNER :      ATEST, Table_name :  BIG_TABLE, No or rows: 100000

Note :
when run this will prompt you for the schema you want to count the the table rows.You can alter this block to count for all tables in multiple schemas by changing below part to list of schemas like below.

or r in (select table_name, owner from all_tables
              where owner in ('&owner')

You can change the &owner  to 'SCHEMA1','SCHEMA2','SCHEMA3'...

ORACLE FAL_CLIENT and FAL_SERVER explained

FAL_CLIENT and FAL_SERVER are initialization parameters used to configure log gap detection and resolution at the standby database side of a physical database configuration. This functionality is provided by log apply services and is used by the physical standby database to manage the detection and resolution of archived redo logs.

FAL_CLIENT and FAL_SERVER only need to be defined in the initialization parameter file for the standby database(s). It is possible; however, to define these two parameters in the initialization parameter for the primary database server to ease the amount of work that would need to be performed if the primary database were required to transition its role.


Sample setup:

In Primary site:

FAL_SERVER=STANDBY
FAL_CLIENT=PRIMARY

In Standby site:

FAL_SERVER=PRIMARY
FAL_CLIENT=STANDBY

UNIX Permissions - table

Unix/Linux Permissions:

Here is the list of values and what they stand for in unix folder/directory permissions.Basically it is a simple you can count the value by using octa decimal binary numbers 421 :

See below image 

Value 0 directory listing is 000 = 0+0+0 

Value 1 directory listing is 001 = 0+0+1

Value 2 directory listing is 020 = 0+0+0  

This is where they gain their permission from :

so simple chmod 1 will give execute only permision on file/folder

so simple chmod 2 will give write only permision on file/folder

so simple chmod 3 will write & execute permision on file/folder

chmod 775 test.txt >> this mean 7 for 

UNIX Permissions

Value	Permission	Directory Listing
0	No read, no write, no execute	---
1	No read, no write, execute	--x
2	No read, write, no execute	-w-
3	No read, write, execute	-wx
4	Read, no write, no execute	r--
5	Read, no write, execute	r-x
6	Read, write, no execute	rw-
7	Read, write, execute	rwx

Configure second physical standby database - Oracle

I know there are various cases where in we have to setup a second physical standby database in our environments to have a redundant fail over strategy. I don't think there are enough documents out there that outline the process of adding the second standby database. I though I will share these steps today.

Parameters that should be considered while setting up the 2 node (single instance) standby Database.
In my case I will explain the parameters and their usage in setup. I am not going to explain the entire setup here. I you don't know how to setup physical standby database, please refer to this Physical_standby_setup.

At this point I am considering you know how to setup physical standby database and you are looking to add additional node to your setup.

Environments:

ORCL is PRIMARY INSTANCE and is on host        LINUX01   >> Primary Instance
ORCLSTB1 is STANDBY INSTANCE and is on host LINUX02  >> Standby already exists
ORCLSTB2 is STANDBY INSTANCE and is on host LINUX03  >>>  new instance to be added

PARAMETERS to be considered :

ARCHIVE_LOG_DEST_N   >>> these needs to be set to ship your logs to the new node,

This is a simple setup that can be used to makes sure all the three nodes will ship logs in case of switch over. In the below image I have configured such a way that each instance will send log files to other when they act as PRIMARY INSTANCE.

Note : I am using round robin process to ship logs between DB servers.

Example of Archive_Log_Dest setup

Adding Archive_log Dests:

ON DB SERVER Linux01 (ORCLPRIM):

alter system set log_archive_config='dg_config=(ORCL,ORCLSTB1,ORCLSTB2)';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=ORCLSTB1 NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ORCLSTB1';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=ORCLSTB2 NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ORCLSTB2';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE;

ON DB SERVER Linux02 (ORCLSTB1):

alter system set log_archive_config='dg_config=(ORCL,ORCLSTB1,ORCLSTB2)';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=ORCLPRIM NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ORCL';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=ORCLSTB2 NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ORCLSTB2';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE;

ON DB SERVER Linux03 (ORCLSTB2):

alter system set log_archive_config='dg_config=(ORCL,ORCLSTB1,ORCLSTB2)';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_2='SERVICE=ORCLPRIM NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ORCL';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2=ENABLE;

ALTER SYSTEM SET LOG_ARCHIVE_DEST_3='SERVICE=ORCLSTB1 NOAFFIRM ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=ORCLSTB1';

ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_3=ENABLE;

OTHER PARAMETER FILE SETTINGS: 

PFILE DB_* PARAMETERS on DB SERVER1 (Linux01)(ORCL): 

log_archive_config='dg_config=(ORCL,ORCLSTB1,ORCLSTB2)';  >> This is to tell oracle that what all instances are part of this config

*.db_file_name_convert='/ORCLSTB1/','/ORCL/','/ORCLSTB2/','/ORCL/'        -- > add both servers so oracle knows where to create file. 

*.db_name='ORCL'  ### same across all Databases 

*.db_unique_name='ORCL'  ### Unique across each Database 

FAL_SERVER=ORCLSTB1, ORCLSTB2-- > add both servers so oracle knows where to get the archive files from incase of switchover. 

FAL_CLIENT=ORCLThis is always the Current DB server typically standby DB. This is ignored when the DB is in Primary mode 

PFILE DB_* PARAMETERS on DB SERVER2 (Linux02)(ORCLSTB1): 

log_archive_config='dg_config=(ORCL,ORCLSTB1,ORCLSTB2)';
*.db_file_name_convert='/ORCL/','/ORCLSTB1/','/ORCLSTB2/','/ORCLSTB1/' 

*.db_name='ORCL'### same across all Databases 

*.db_unique_name='ORCLSTB1'                    ### Unique across each Database 

*.FAL_CLIENT=STANDBY_SERVER 

*.FAL_SERVER=PRIMARY_SERVERS 

PFILE DB_* PARAMETERS on DB SERVER3 (Linux03)(ORCLSTB2): 

log_archive_config='dg_config=(ORCL,ORCLSTB1,ORCLSTB2)';
*.db_file_name_convert='/ORCLSTB1/','/ORCLSTB2/','/ORCL/','/ORCLSTB2/' 

*.db_name='ORCL'### same across all Databases 

*.db_unique_name='ORCLSTB2'                 ### Unique across each Database 

*.FAL_CLIENT=STANDBY_SERVER 

*.FAL_SERVER=PRIMARY_SERVERS 

DATABASE NAME MAX SIZE in ORACLE

This is for 11gR2:

The length of DB_NAME is limited to 8 characters;

DB_UNIQUE_NAME is limited to 30 characters. DB_UNIQUE_NAME  can contain alphanumeric, underscore (_), dollar ($), and pound (#) characters, but must begin with an alphabetic character.

INSTANCE_NAME defaults to ORACLE_SID. Oracle document does not specify the limit for INSTANCE_NAME length. However if you describe V$INSTANCE view you can see that column INSTANCE_NAME  has VARCHAR2(16), indicating the  limit of 16 characters.

Because INSTANCE_NAME defaults to ORACLE_SID you need to be careful when deciding SID before creating a database. Oracle does not clearly define the limit for SID (system identifier). Following are what I found from two 11gR2 documents about ORACLE_SID.