Logstash实现mysql同步数据到elasticsearch

Table of Contents

注意添加ElasticSearch用户名和密码

Logstash安装：
1. 下载安装包：wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.2.tar.gz
2. 下载mysql的连接库jar包(版本库)：wget http://central.maven.org/maven2/mysql/mysql-connector-java/6.0.6/mysql-connector-java-6.0.6.jar
3. 解压安装包：tar -xvf logstash-6.6.2.tar.gz
4. 安装jdbc、elasticsearch插件:
  1
  2
  3
  4
  logstash插件安装命令
  cd logstash-6.6.2/bin
  ./logstash-plugin install logstash-input-jdbc
  ./logstash-plugin install logstash-output-elasticsearch

创建配置文件
在config目录下，创建配置文件（logstash-mysql-es.conf）：

input {
# stdin { }
 jdbc {
     jdbc_connection_string => "jdbc:mysql://localhost:30012/data_acquisition?useUnicode=true&characterEncoding=UTF-8&zeroDateTimeBehavior=convertToNull&allowMultiQueries=true"

     jdbc_user => "root"

     jdbc_password => "123456"

     jdbc_driver_library => "/path/mysql-connector-java-6.0.6.jar" #全路径

     jdbc_driver_class => "com.mysql.jdbc.Driver"

     jdbc_paging_enabled => "true"

     jdbc_page_size => "50000"

     statement => "select * from result_comment where create_time >= '2019-03-20'"
     #schedule => "* * * * *"
 }
}

output {
  # 这里输出调试，正式运行时可以注释掉
  stdout {
     codec => json_lines
 }
 elasticsearch {
     hosts => "127.0.0.1:9200"
     index => "wl_travel_analyze"
     document_type => "travel"
     document_id => "%{id}"
     #ssl => true
     #ssl_certificate_verification => true
     #truststore => "/opt/elk/search-guard-tlstool/search-guard-certificates/truststore.jks"
     #truststore_password => "286a7b7a8970af4e8467"
     user => "admin"
     password => "admin"
 }
}

注意添加ElasticSearch用户名和密码

启动改脚本
nohup ./bin/logstash -f config/logstash-mysql-es.conf > logs/logstash.out &
查看日志
tail -f logs/logstash.out

借鉴文章如下:
Elasticsearch - Logstash实现mysql同步数据到elasticsearch

Logstash实现mysql同步数据到elasticsearch

注意添加ElasticSearch用户名和密码

相关文章