Connections could not be acquired from the underlying database! //不能连接当前所配置的数据库
TNS-12541: TNS:no listener //无监听程序
TNS-12560: TNS:protocol adapter error //协议适配器错误
TNS-00511:no listener //无监听程序

前言:
在开发环境，项目正常运行，突然登陆页面可以显示，登陆操作异常。
查询后台日志，错误描述：

Caused by: java.sql.SQLException: Connections could not be acquired from the underlying database!
    at com.mchange.v2.sql.SqlUtils.toSQLException(SqlUtils.java:106)
    at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:529)
    at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
    at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:56)
    at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:423)
    ... 148 more
Caused by: com.mchange.v2.resourcepool.CannotAcquireResourceException: A ResourcePool could not acquire a resource from its primary factory or source.
    at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1319)
    at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
    at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
    at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
    ... 151 more  

项目前一段时间运行正常，今天下午突然出现问题，说明程序配置是正常的，检查数据库连接数发现也是正常。这样问题只能出现在数据库上。

使用数据库连接工具连接，发现监听问题TNS
查看监听情况

lsnrctl status

问题如下：

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 07-11月-2018 18:22:47

Copyright (c) 1991, 2007, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
TNS-12541: TNS:no listener  
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=GCRM0CV1)(PORT=1521)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused

这个问题可能是监听配置文件有问题，但是这段时间没有调整过监听配置文件，检查了监听配置文件后，是没有问题的。继续排查中。。。。。

停止监听

lsnrctl stop

问题和上面描述一样。

启动监听

lsnrctl start  

问题如下：

LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 07-11月-2018 18:23:22

Copyright (c) 1991, 2007, Oracle.  All rights reserved.

Starting /home/db/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.4.0 - Production
NL-00280: error creating log stream /home/db/oracle/product/10.2.0/db_1/network/log/listener.log
 NL-00278: cannot open log file
  SNL-00016: snlfohd: error opening file
   Linux Error: 13: Permission denied

Listener failed to start. See the error message(s) above...

查看日志:

ls -lh /home/db/oracle/product/10.2.0/db_1/network/log/listener.log 

发现日志大小居然大于4G，找到问题所在，这个是oracle的bug,当listner日志增长到4GB,通过侦听器连接很慢。

解决方案:

1. lsnrctl进入交互模式  
2. 执行 set current_listener LISTENER  
3. set log_status off  
4. stop 停止监听器  
5. 手工删除指定的监听日志路径下的listener.log文件  
6. start 启动监听器  
7. status 查看状态