Skip to content
This repository was archived by the owner on Sep 17, 2019. It is now read-only.

Commit ae51d77

Browse files
theangryangeltheangryangel
committed
Move examples and split up connection code
Bump version
1 parent 529c98a commit ae51d77

File tree

11 files changed

+158
-130
lines changed

11 files changed

+158
-130
lines changed

‎README.md‎

Lines changed: 2 additions & 116 deletions
Original file line numberDiff line numberDiff line change
@@ -59,120 +59,6 @@ Tests are not yet 100% complete.
5959
| max_flush_exceptions | Number | Number of sequential flushes which cause an exception, before we stop logstash. Set to a value less than 1 if you never want it to stop. This should be carefully configured with relation to idle_flush_time if your SQL instance is not highly available. | No | 0 |
6060

6161
## Example configurations
62-
If you have a working sample configuration, for a DB thats not listed, pull requests are welcome.
63-
64-
### SQLite3
65-
* Tested using https://bitbucket.org/xerial/sqlite-jdbc
66-
* SQLite setup - `echo "CREATE table log (host text, timestamp datetime, message text);" | sqlite3 test.db`
67-
```
68-
input
69-
{
70-
stdin { }
71-
}
72-
output {
73-
stdout { }
74-
75-
jdbc {
76-
connection_string => 'jdbc:sqlite:test.db'
77-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
78-
}
79-
}
80-
```
81-
82-
### SQL Server
83-
* Tested using http://msdn.microsoft.com/en-gb/sqlserver/aa937724.aspx
84-
```
85-
input
86-
{
87-
stdin { }
88-
}
89-
output {
90-
jdbc {
91-
connection_string => "jdbc:sqlserver://server:1433;databaseName=databasename;user=username;password=password;autoReconnect=true;"
92-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
93-
}
94-
}
95-
```
96-
97-
### Postgres
98-
With thanks to [@roflmao](https://github.com/roflmao)
99-
```
100-
input
101-
{
102-
stdin { }
103-
}
104-
output {
105-
jdbc {
106-
connection_string => 'jdbc:postgresql://hostname:5432/database?user=username&password=password'
107-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
108-
}
109-
}
110-
```
111-
112-
### Oracle
113-
With thanks to [@josemazo](https://github.com/josemazo)
114-
* Tested with Express Edition 11g Release 2
115-
* Tested using http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html (ojdbc6.jar)
116-
```
117-
input
118-
{
119-
stdin { }
120-
}
121-
output {
122-
jdbc {
123-
connection_string => "jdbc:oracle:thin:USER/PASS@HOST:PORT:SID"
124-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
125-
}
126-
}
127-
```
62+
Example logstash configurations, can now be found in the examples directory. Where possible we try to link every configuration with a tested jar.
12863

129-
### Mysql
130-
With thanks to [@jMonsinjon](https://github.com/jMonsinjon)
131-
* Tested with Version 14.14 Distrib 5.5.43, for debian-linux-gnu (x86_64)
132-
* Tested using http://dev.mysql.com/downloads/file.php?id=457911 (mysql-connector-java-5.1.36-bin.jar)
133-
```
134-
input
135-
{
136-
stdin { }
137-
}
138-
output {
139-
jdbc {
140-
connection_string => "jdbc:mysql://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
141-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
142-
}
143-
}
144-
```
145-
146-
### MariaDB
147-
* Tested with Ubuntu 14.04.3 LTS, Server version: 10.1.9-MariaDB-1~trusty-log mariadb.org binary distribution
148-
* Tested using https://downloads.mariadb.com/enterprise/tqge-whfa/connectors/java/connector-java-1.3.2/mariadb-java-client-1.3.2.jar (mariadb-java-client-1.3.2.jar)
149-
```
150-
input
151-
{
152-
stdin { }
153-
}
154-
output {
155-
jdbc {
156-
connection_string => "jdbc:mariadb://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
157-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
158-
}
159-
160-
}
161-
```
162-
163-
### Apache Phoenix (HBase SQL)
164-
* Tested with Ubuntu 14.04.03 / Logstash 2.1 / Apache Phoenix 4.6
165-
* <!> HBase and Zookeeper must be both accessible from logstash machine <!>
166-
```
167-
input
168-
{
169-
stdin { }
170-
}
171-
output {
172-
jdbc {
173-
connection_string => "jdbc:phoenix:ZOOKEEPER_HOSTNAME"
174-
statement => [ "UPSERT INTO EVENTS log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
175-
}
176-
177-
}
178-
```
64+
If you have a working sample configuration, for a DB thats not listed, pull requests are welcome.

‎examples/apache-phoenix-hbase-sql.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Example: Apache Phoenix (HBase SQL)
2+
* Tested with Ubuntu 14.04.03 / Logstash 2.1 / Apache Phoenix 4.6
3+
* <!> HBase and Zookeeper must be both accessible from logstash machine <!>
4+
```
5+
input
6+
{
7+
stdin { }
8+
}
9+
output {
10+
jdbc {
11+
connection_string => "jdbc:phoenix:ZOOKEEPER_HOSTNAME"
12+
statement => [ "UPSERT INTO EVENTS log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
13+
}
14+
15+
}
16+
```

‎examples/mariadb.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Example: MariaDB
2+
* Tested with Ubuntu 14.04.3 LTS, Server version: 10.1.9-MariaDB-1~trusty-log mariadb.org binary distribution
3+
* Tested using https://downloads.mariadb.com/enterprise/tqge-whfa/connectors/java/connector-java-1.3.2/mariadb-java-client-1.3.2.jar (mariadb-java-client-1.3.2.jar)
4+
```
5+
input
6+
{
7+
stdin { }
8+
}
9+
output {
10+
jdbc {
11+
connection_string => "jdbc:mariadb://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
12+
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
13+
}
14+
15+
}
16+
```

‎examples/mysql.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Example: Mysql
2+
With thanks to [@jMonsinjon](https://github.com/jMonsinjon)
3+
* Tested with Version 14.14 Distrib 5.5.43, for debian-linux-gnu (x86_64)
4+
* Tested using http://dev.mysql.com/downloads/file.php?id=457911 (mysql-connector-java-5.1.36-bin.jar)
5+
```
6+
input
7+
{
8+
stdin { }
9+
}
10+
output {
11+
jdbc {
12+
connection_string => "jdbc:mysql://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
13+
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
14+
}
15+
}
16+
```

‎examples/odps.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Example: ODPS
2+
With thanks to [@onesuper](https://github.com/onesuper)
3+
```
4+
input
5+
{
6+
stdin { }
7+
}
8+
output {
9+
jdbc {
10+
driver_class => "com.aliyun.odps.jdbc.OdpsDriver"
11+
driver_auto_commit => false
12+
connection_string => "jdbc:odps:http://service.odps.aliyun.com/api?project=meta_dev&loglevel=DEBUG"
13+
username => "abcd"
14+
password => "1234"
15+
max_pool_size => 5
16+
flush_size => 10
17+
statement => [ "INSERT INTO test_logstash VALUES(?, ?, ?);", "host", "@timestamp", "message" ]
18+
}
19+
}
20+
```

‎examples/oracle.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
# Example: Oracle
2+
With thanks to [@josemazo](https://github.com/josemazo)
3+
* Tested with Express Edition 11g Release 2
4+
* Tested using http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html (ojdbc6.jar)
5+
```
6+
input
7+
{
8+
stdin { }
9+
}
10+
output {
11+
jdbc {
12+
connection_string => "jdbc:oracle:thin:USER/PASS@HOST:PORT:SID"
13+
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
14+
}
15+
}
16+
```

‎examples/postgres.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Example: Postgres
2+
With thanks to [@roflmao](https://github.com/roflmao)
3+
```
4+
input
5+
{
6+
stdin { }
7+
}
8+
output {
9+
jdbc {
10+
connection_string => 'jdbc:postgresql://hostname:5432/database?user=username&password=password'
11+
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
12+
}
13+
}
14+
```
15+

‎examples/sql-server.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
# Example: SQL Server
2+
* Tested using http://msdn.microsoft.com/en-gb/sqlserver/aa937724.aspx
3+
```
4+
input
5+
{
6+
stdin { }
7+
}
8+
output {
9+
jdbc {
10+
connection_string => "jdbc:sqlserver://server:1433;databaseName=databasename;user=username;password=password;autoReconnect=true;"
11+
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
12+
}
13+
}
14+
```

‎examples/sqlite.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Example: SQLite3
2+
* Tested using https://bitbucket.org/xerial/sqlite-jdbc
3+
* SQLite setup - `echo "CREATE table log (host text, timestamp datetime, message text);" | sqlite3 test.db`
4+
```
5+
input
6+
{
7+
stdin { }
8+
}
9+
output {
10+
stdout { }
11+
12+
jdbc {
13+
connection_string => 'jdbc:sqlite:test.db'
14+
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
15+
}
16+
}
17+
```

‎lib/logstash/outputs/jdbc.rb‎

Lines changed: 25 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -83,19 +83,6 @@ def register
8383

8484
load_jar_files!
8585

86-
@pool = Java::ComZaxxerHikari::HikariDataSource.new
87-
88-
@pool.setAutoCommit(@driver_auto_commit)
89-
@pool.setDriverClassName(@driver_class) if @driver_class
90-
91-
@pool.setJdbcUrl(@connection_string)
92-
93-
@pool.setUsername(@username) if @username
94-
@pool.setPassword(@password) if @password
95-
96-
@pool.setMaximumPoolSize(@max_pool_size)
97-
@pool.setConnectionTimeout(@connection_timeout)
98-
9986
@exceptions_tracker = RingBuffer.new(@max_flush_exceptions)
10087

10188
if (@flush_size > 1000)
@@ -109,6 +96,8 @@ def register
10996
if (!@unsafe_statement and @statement.length < 2)
11097
@logger.error("JDBC - Statement has no parameters. No events will be inserted into SQL as you're not passing any event data. Likely configuration error.")
11198
end
99+
100+
setup_and_test_pool!
112101

113102
buffer_initialize(
114103
:max_items => @flush_size,
@@ -152,6 +141,29 @@ def teardown
152141

153142
private
154143

144+
def setup_and_test_pool!
145+
# Setup pool
146+
@pool = Java::ComZaxxerHikari::HikariDataSource.new
147+
148+
@pool.setAutoCommit(@driver_auto_commit)
149+
@pool.setDriverClassName(@driver_class) if @driver_class
150+
151+
@pool.setJdbcUrl(@connection_string)
152+
153+
@pool.setUsername(@username) if @username
154+
@pool.setPassword(@password) if @password
155+
156+
@pool.setMaximumPoolSize(@max_pool_size)
157+
@pool.setConnectionTimeout(@connection_timeout)
158+
159+
# Test connection
160+
test_connection = @pool.getConnection()
161+
unless test_connection.isValid(10)
162+
@logger.error("JDBC - Connection is not valid. Please check connection string or that your JDBC endpoint is available.")
163+
end
164+
test_connection.close()
165+
end
166+
155167
def load_jar_files!
156168
# Load jar from driver path
157169
unless @driver_jar_path.nil?

0 commit comments

Comments
 (0)