-
Notifications
You must be signed in to change notification settings - Fork 2
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
2 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,10 +1,2 @@ | ||
| # dk-sqoop-plus | ||
| <div>本项目在原生sqoop 的基础上进行更多实用功能的扩展。</div><div><br></div><div>项目编译及sqoopo使用参见原生sqoop。</div> | ||
|
|
||
| Versions | ||
| IK version ES version | ||
| master 6.x -> master | ||
| 6.2.4 6.2.4 | ||
|
|
||
|
|
||
| <div><br></div><div>下面列举其扩展功能的使用：</div><div><br></div><div><b>1. RDBMS to Elasticsearch </b></div><div> 在其安装目录下执行 ： bin/sqoop import --connect jdbc:oracle:thin:@192.168.1.35:1521:orcl --username system --password dkss --table DK10 -m 1 --target-dir 192.168.1.99:9200@dkes@dksqoop2/oracle </div><div> 执行命令说明：</div><div> sqoop import #固定，默认</div><div> --connect jdbc:oracle:thin:@192.168.1.35:1521:orcl #为RDBMS数据库的连接信息 (如oracle , mysql , sqlserver)</div><div> --username system #RDBMS数据库用户名</div><div> --password dkss #RDBMS数据库密码</div><div> --table DK10 #RDBMS要导入到elasticsearch的数据库表</div><div> -m 1 #指定导入时产生的线程的数量。注意：如果RDBMS数据库表中没有主键，则-m的值只能是1</div><div> --target-dir 192.168.1.99:9200@dkes@dksqoop2/oracle #--target-dir的值包含三部分由“@”分隔，“@”前一部分192.168.1.99:9200为指定elasticsearch的主机地址和端口号，“@”中间一部分dkes为elasticsearch的集群名称,“@”后一部分dksqoop2/oracle包含两部分，由“/”分隔，“/”前一部分dksqoop2为elasticsearch的索引名称，“/”后一部分oracle为elasticsearch的类型名称。</div><div><br></div><div><b>2. HDFS to Elasticsearch</b></div><div> 在其安装目录下执行 ： bin/sqoop export --connect "es@192.168.1.126:9200@elasticsearch@hdfs/sqoop" --export-dir aaa --fields-terminated-by '@' -m 1</div><div> 执行命令说明：</div><div> sqoop export #固定，默认</div><div> --connect "es@192.168.1.126:9200@kes@hdfs/sqoop" #--connect的值包含四部分由“@”分隔,第一部分es为固定默认值，第二部分192.168.1.126:9200为 连接elasticsearch的主机地址和端口，第三部分dkes为elasticsearch的集群名称，第四部分hdfs/sqoop为elasticsearch的索引名/类型名。</div><div> --export-dir aaa #指的是要导出的hdfs文件所在的目录。</div><div> --fields-terminated-by '@' #--fields-terminated-by指的是hdfs文件内容各字段间的字段分隔符，如果不指定则默认为','分隔。</div><div> -m 1 #指定导入时产生的线程的数量。线程数不能小于文件内容的记录条数。</div><div><br></div><div><br></div><div><br></div><div><br></div> | ||
| <div><h1 style="box-sizing: border-box; margin-right: 0px; margin-bottom: 16px; margin-left: 0px; line-height: 1.25; padding-bottom: 0.3em; border-bottom: 1px solid rgb(234, 236, 239); color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; margin-top: 0px !important;">DK-Sqoop-plus</h1></div><div>本项目在原生sqoop <span style="color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">(</span><a href="http://sqoop.apache.org/" rel="nofollow" style="box-sizing: border-box; color: rgb(3, 102, 214); text-decoration-line: none; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">http://sqoop.apache.org/</a><span style="color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">) </span>的基础上进行更多实用功能的扩展。</div><div>暂时只支持 RDBMS（mysql 、oracle 、sqlserver） to Elasticsearch 和 HDFS to Elasticsearch 等功能，后续可能支持 Elasticsearch to RDBMS 等功能 。</div><div><br></div><div>项目编译及sqoopo使用参见原生sqoop。</div> | ||
| <div><br></div><div><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; line-height: 1.25; padding-bottom: 0.3em; border-bottom: 1px solid rgb(234, 236, 239); color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";">Versions</h2></div><div><table style="box-sizing: border-box; border-spacing: 0px; border-collapse: collapse; margin-top: 0px; margin-bottom: 16px; display: block; width: 888px; overflow: auto; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;"><thead style="box-sizing: border-box;"><tr style="box-sizing: border-box; border-top: 1px solid rgb(198, 203, 209);"><th style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">DK-Sqoop-plus version</th><th style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">ES version</th></tr></thead><tbody style="box-sizing: border-box;"><tr style="box-sizing: border-box; border-top: 1px solid rgb(198, 203, 209);"><td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">master</td><td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">5.x -> master</td></tr><tr style="box-sizing: border-box; background-color: rgb(246, 248, 250); border-top: 1px solid rgb(198, 203, 209);"><td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">1.5.4</td><td style="box-sizing: border-box; padding: 6px 13px; border: 1px solid rgb(223, 226, 229);">5.6.8</td></tr></tbody></table></div><div><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; line-height: 1.25; padding-bottom: 0.3em; border-bottom: 1px solid rgb(234, 236, 239); color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";">Install</h2><p style="box-sizing: border-box; margin-top: 0px; margin-bottom: 16px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">1.download </p><p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">optional 1 - download pre-build package from here: <a href="https://github.com/medcl/elasticsearch-analysis-ik/releases" style="box-sizing: border-box; background-color: transparent; color: rgb(3, 102, 214); text-decoration-line: none;">https://github.com/</a></p><p style="box-sizing: border-box; margin-top: 16px; margin-bottom: 16px; color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; font-size: 16px;">unzip plugin to folder <code style="box-sizing: border-box; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; padding: 0.2em 0.4em; margin: 0px; background-color: rgba(27, 31, 35, 0.05); border-radius: 3px;">your-es-root/plugins/</code></p></div><div><h2 style="box-sizing: border-box; margin-top: 24px; margin-bottom: 16px; line-height: 1.25; padding-bottom: 0.3em; border-bottom: 1px solid rgb(234, 236, 239); color: rgb(36, 41, 46); font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";">Quick Example</h2></div><div><b>1. RDBMS to Elasticsearch </b></div><div> 在其安装目录下执行 ： bin/sqoop import --connect jdbc:oracle:thin:@192.168.1.35:1521:orcl --username system --password dkss --table DK10 -m 1 --target-dir 192.168.1.99:9200@dkes@dksqoop2/oracle </div><div> 执行命令说明：</div><div> sqoop import #固定，默认</div><div> --connect jdbc:oracle:thin:@192.168.1.35:1521:orcl #为RDBMS数据库的连接信息 (如oracle , mysql , sqlserver)</div><div> --username system #RDBMS数据库用户名</div><div> --password dkss #RDBMS数据库密码</div><div> --table DK10 #RDBMS要导入到elasticsearch的数据库表</div><div> -m 1 #指定导入时产生的线程的数量。注意：如果RDBMS数据库表中没有主键，则-m的值只能是1</div><div> --target-dir 192.168.1.99:9200@dkes@dksqoop2/oracle #--target-dir的值包含三部分由“@”分隔，“@”前一部分192.168.1.99:9200为指定elasticsearch的主机地址和端口号，“@”中间一部分dkes为elasticsearch的集群名称,“@”后一部分dksqoop2/oracle包含两部分，由“/”分隔，“/”前一部分dksqoop2为elasticsearch的索引名称，“/”后一部分oracle为elasticsearch的类型名称。</div><div><br></div><div><b>2. HDFS to Elasticsearch</b></div><div> 在其安装目录下执行 ： bin/sqoop export --connect "es@192.168.1.126:9200@elasticsearch@hdfs/sqoop" --export-dir aaa --fields-terminated-by '@' -m 1</div><div> 执行命令说明：</div><div> sqoop export #固定，默认</div><div> --connect "es@192.168.1.126:9200@kes@hdfs/sqoop" #--connect的值包含四部分由“@”分隔,第一部分es为固定默认值，第二部分192.168.1.126:9200为 连接elasticsearch的主机地址和端口，第三部分dkes为elasticsearch的集群名称，第四部分hdfs/sqoop为elasticsearch的索引名/类型名。</div><div> --export-dir aaa #指的是要导出的hdfs文件所在的目录。</div><div> --fields-terminated-by '@' #--fields-terminated-by指的是hdfs文件内容各字段间的字段分隔符，如果不指定则默认为','分隔。</div><div> -m 1 #指定导入时产生的线程的数量。线程数不能小于文件内容的记录条数。</div><div><br></div><div><br></div><div><br></div><div><br></div> |