database

Menghapus Ratusan Juta Record dari Oracle Table

Baru kemarin ketemu case yang lumayan menarik, aplikasi yang dibuat tahun lalu kena timeout ketika delete seluruh isi table yang berisi paling tidak 200juta record. Berikut error lengkapnya, fyi framework yang dipakai adalah MyBatis dan Spring Framework dan database yang dipakai adalah Oracle,

<2016-10-31 02:15:01,723>,[http-/0.0.0.0:8080-16]>>[INFO]start deleting TABLE_NAME Data
<2016-10-31 02:18:02,426>,[http-/0.0.0.0:8080-16]>>[ERROR]
### Error updating database.  Cause: java.sql.SQLException: ORA-01013: user requested cancel of current operation

### The error may involve id.edwin.service.delete-Inline
### The error occurred while setting parameters
### SQL: delete from TABLE_NAME
### Cause: java.sql.SQLException: ORA-01013: user requested cancel of current operation

; uncategorized SQLException for SQL []; SQL state [72000]; error code [1013]; ORA-01013: user requested cancel of current operation
; nested exception is java.sql.SQLException: ORA-01013: user requested cancel of current operation

sepertinya sudah timeout duluan (3menit timeout), padahal belum semua data terhapus. 🙁

Setelah googling bentar, sepertinya ada dua solusi yaitu menggunakan TRUNCATE dan CTAS (CREATE TABLE AS SELECT). Setelah diskusi panjang lebar dengan kuncen (admin) Database, opsi terakhir (sepertinya) jauh lebih cepat, drop table tersebut kemudian di re-create ulang. Berikut adalah query-nya

CREATE TABLE TABLE_NAME_NEW AS
        SELECT * FROM TABLE_NAME WHERE ROWNUM = 1 ;
Rename TABLE_NAME to TABLE_NAME_OLD ;
Rename TABLE_NAME_NEW to TABLE_NAME;
drop table TABLE_NAME_OLD ;

Setelah itu baru didelete isi table TABLE_NAME, lebih cepat karena isi datanya hanya 1 row. Satu-satunya kekurangan adalah, tidak bisa replicate Primary Key dan Index dari table yang sebelumnya di drop, yang mana itu bukanlah masalah bagi saya 😀

Weird Exception, “Unsupported startup parameter: extra_float_digits” When Connecting to PostgreSQL

Today I’m planning on deploying my application (which runs well on my local laptop), to a production server. But there is a very weird error occur, it says

Caused by: org.postgresql.util.PSQLException: ERROR: Unsupported startup parameter: extra_float_digits
	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)

Somehow it seems that the error happen because of some wrong PostgreSql JDBC version. But the error stays even if I changed my Postgresql JDBC version several times. So my temporary conclusion is, improper Postgresql configuration.

After debating with several sysadmins and DBA, i found out that my sql connection actually goes to a Connection Pooling called PGBouncer, instead of connecting directly to Postgresql. Weird, because i also using Connection Pooling (Apache DBCP) for my application. So it means that my app is using a double connection pooling 😀

After spending sometime googling, i found out that all i need to do is adding this simple configuration on pgbouncer.ini

ignore_startup_parameters = extra_float_digits

hint :
If your are unable to found the location for pgbouncer.ini, just using a simple ps -ef command on console.

bash-4.1$ ps -ef | grep pgbouncer
500      18927     1  0 May30 ?        00:00:00 /opt/PostgresPlus/pgbouncer-1.6/bin/pgbouncer -d /opt/PostgresPlus/pgbouncer-1.6/share/pgbouncer.ini
500      19045 18913  0 00:11 pts/4    00:00:00 grep pgbouncer

As you can see, the pgbouncer.ini location is in /opt/PostgresPlus/pgbouncer-1.6/share/.

Well, one thing solve, and suddenly i found another error :sigh. So many errors today. Here is my complete stacktrace

 org.postgresql.util.PSQLException: ERROR: No such database: mydatabase
 	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:291)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:108)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
	at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
	at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
	at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
	at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30)

Somehow i cannot find mydatabase, despite i already created it.

Actually, the answer is simple. Again pgbouncer is the culprit. I cannot connect to a database if the database havent registered yet at pgbouncer.ini. Registering my database solve this issue.

;; database name = connect string
;;
;; connect string params:
;;   dbname= host= port= user= password=
;;   client_encoding= datestyle= timezone=
;;   pool_size= connect_query=
[databases]

; foodb over unix socket
edb = host=127.0.0.1 port=5444
mydatabase = host=127.0.0.1 port=5444

You can see my PostgreSQL’s port is 5444.

Hopefully, my tutorial helps other. Because i spent a ridiculously amount of time solving it 😀

Error No Buffer Space Available when Using Java, MyBatis and MySql

I’ve met a very weird error, that are not supposed to happen, when using MyBatis and MySql. Somehow my Windows OS is running out of ports, this is the complete stacktrace.

Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): connect
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.PlainSocketImpl.doConnect(Unknown Source)
    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at java.net.Socket.<init>(Unknown Source)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)]

I found out it happened because im using traditional unpooled connection, changing it into pooled connection in MyBatis solved it.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration
PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
"http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
    <environments default="development">
        <environment id="development">
            <transactionManager type="JDBC"/>
            <dataSource type="POOLED">
                <property name="driver" value="com.mysql.jdbc.Driver"/>
                <property name="url" value="jdbc:mysql://localhost/xxxx"/>
                <property name="username" value="xxxx"/>
                <property name="password" value=""/>
                <property name="poolMaximumActiveConnections" value="20"/>
            </dataSource>
        </environment>
    </environments>
    <mappers>

    </mappers>
</configuration>

[PostgreSQL] Got Error “FATAL: database databasename does not exist” Despite Database is Exists on PGAdmin

Today i spent a ridiculous amount of time trying to figure out how come suddenly my java application cannot connect to postgresql database. It gives error “FATAL: database databasename does not exist”, and the weird thing is that i can browse and query to my postgresql database from PGAdmin and other database browser.

Here is my application’s complete error stacktrace

Caused by: org.postgresql.util.PSQLException: FATAL: database "geoportal" does not exist
    at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:684)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:199)
    at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
    at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:127)
    at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29)
    at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21)
    at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:41)
    at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
    at org.postgresql.Driver.makeConnection(Driver.java:414)
    at org.postgresql.Driver.connect(Driver.java:282)

Weird because i can see my db both from PGAdmin and psql console,
pg geoportal

pg console

When suddenly i realize when im checking my postgresql port, and i can see 2 different postgresql with different PID occupied the same port, 5432. Somehow my PGAdmin and psql connect to the right instance while my application connect to the wrong instance.
pgsql

After killing my unwanted postgresql PID from taskmanager, suddenly my java application can run well again.
pg success connect