ERROR: ORA-03135: connection lost contact

Was adding a target database to my new install of 12cR2 Cloud Control and after testing the connection string it would always time out. This error pop up did suggest to check for firewalls. In the past I have been burned by iptables being turned on by default and causing problems for RAC installs so I did check to ensure iptables where not only running on the db node but also the oms server as well but no such luck iptables were already off. To ensure it wasn’t some weird network problem I ssh’ed into the db node and was able to make a local connection and so I setup an entry in the tnsnames.ora file to check connecting through the listener. To my surprise I had the same problem with sqlplus connecting and would get the following error

sqlplus system@a250

SQL*Plus: Release 11.2.0.2.0 Production on Thu Oct 4 11:04:51 2012

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

ERROR:
ORA-03135: connection lost contact
Process ID: 0
Session ID: 0 Serial number: 0

I have seen type of problem before but its usually is a firewall or other network problem but connecting to the db from a console was new. I checked the listener log file and it had normal entries with no errors at all. Next I checked the alert log for the database and found something.

ORA-19815: WARNING: db_recovery_file_dest_size of 10737418240 bytes is 100.00% used, and has 0 remaining bytes available.
************************************************************************
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
   then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
   BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
   reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
   system command was used to delete files, then use RMAN CROSSCHECK and
   DELETE EXPIRED commands.
************************************************************************
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance a250_1 - Archival Error
ORA-16038: log 2 sequence# 360 cannot be archived
ORA-19809: limit exceeded for recovery files

This database is a test database and I didn’t need to backup the archivelogs. Since I am using the flash recovery area performing a simple delete is not enough to resolve the problem as the database dictionary will still think the archive logs are there. So after deleting the archivelogs I ran the following RMAN commands to inform the catalog the logs were removed. After which remote connections worked and I could finish registering the database it Cloud Control. The first RMAN command checks to make sure the archive logs do exist on disk and if not change the status to expired. Then the second command deletes the archive log entries out of the catalog.

RMAN> crosscheck archivelog all;

RMAN> delete expired archivelog all;