Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sync #2

Open
wants to merge 146 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
146 commits
Select commit Hold shift + click to select a range
37cea6f
add all code
Dec 25, 2015
5854f9d
modify README
Dec 25, 2015
c89bfd2
modify README
Dec 25, 2015
6ce4bae
delete iml
Dec 25, 2015
99d4a0a
delete README
Dec 25, 2015
035ef38
modify README
Dec 25, 2015
accd9c5
add logo
Dec 25, 2015
2054ed6
modify README
Dec 25, 2015
6481842
fix mysqlreader.md json error
Dec 30, 2015
312456c
Merge pull request #4 from binaryWorld/master
binaryWorld Dec 30, 2015
a5a2975
add email
Dec 31, 2015
a4507bb
Merge pull request #5 from binaryWorld/master
binaryWorld Dec 31, 2015
707d54c
mysql json template
Jan 5, 2016
2ff83e0
Merge pull request #6 from binaryWorld/master
binaryWorld Jan 5, 2016
fa72c02
hbase doc
Jan 15, 2016
baae267
Merge pull request #11 from binaryWorld/master
binaryWorld Jan 15, 2016
130d09e
update oracle/sqlserver package file
Jan 22, 2016
c44e9c1
update parent pom
Mar 8, 2016
0ec0ae9
修改datax支持windows
Mar 22, 2016
97a077f
param clerical error and add more help tips
Apr 5, 2016
91d0572
change spaces format
Apr 5, 2016
3c40fa4
Update README.md
binaryWorld Apr 8, 2016
5028375
Rename txtfielreader.md to txtfiereader.md
binaryWorld Apr 8, 2016
53cbda8
Update odpswriter.md
binaryWorld Apr 8, 2016
bb1dcae
Rename txtfiereader.md to txtfilereader.md
binaryWorld Apr 8, 2016
e3c9049
Update oraclereader.md
TrafalgarLuo Apr 22, 2016
19021fd
OTSWriter for public:
zhouzf05 May 10, 2016
8175b84
Merge pull request #41 from alibaba/otsreader_20160510
liupengjava May 12, 2016
b5783b8
Merge pull request #35 from gl2001wl/paramClericalError
liupengjava May 12, 2016
d52d995
Update oraclereader.md
TrafalgarLuo May 17, 2016
b53aaef
Update oraclewriter.md
TrafalgarLuo May 17, 2016
e05e132
Update hdfsreader.md
binaryWorld May 26, 2016
1572426
Update hdfswriter.md
binaryWorld May 26, 2016
72a0b1b
Update ftpreader.md
binaryWorld May 26, 2016
b91784e
Update README.md
binaryWorld Jun 11, 2016
d3ce09f
update wiki
TrafalgarLuo Jun 14, 2016
0bf9a84
update wiki
TrafalgarLuo Jun 16, 2016
efdde7a
update wiki
TrafalgarLuo Jun 16, 2016
f306fa7
update wiki
TrafalgarLuo Jun 16, 2016
023633c
Update README.md
binaryWorld Jun 24, 2016
a43a5f6
Update README.md
binaryWorld Jun 24, 2016
4dc5a5f
Update drdswriter.md
binaryWorld Jun 28, 2016
3325e6e
Update mysqlwriter.md
liupengjava Jun 30, 2016
ba62e43
Update odpswriter.md
tianshuang Jul 9, 2016
c4274d5
修正hdfswriter.md中的writeMode,应该为append/nonConflict,而不是truncate
projoy Jul 22, 2016
01700be
add dingding group
Jul 25, 2016
16e1827
Merge branch 'master' of github.com:alibaba/DataX
Jul 25, 2016
0e75e60
modify dingding group
Jul 25, 2016
68fa2cc
update datax code for opensource project
TrafalgarLuo Jul 25, 2016
ff718ac
update readme wiki
TrafalgarLuo Jul 25, 2016
177618a
add dingding group
TrafalgarLuo Jul 25, 2016
ee8ab30
Merge branch 'master' of https://github.com/alibaba/DataX
TrafalgarLuo Jul 25, 2016
f537ec2
add dingding group
TrafalgarLuo Jul 25, 2016
d8fc366
remote useless doc
TrafalgarLuo Jul 26, 2016
6f70d8e
Merge branch 'master' of https://github.com/alibaba/DataX
TrafalgarLuo Jul 26, 2016
21bfbc3
modify maven pom retates
TrafalgarLuo Jul 26, 2016
ae1b814
update oracle/sqlserver package file
TrafalgarLuo Jul 26, 2016
cdcc7f3
Create hbase094xreader.md
Aug 5, 2016
2d91a61
Create hbase11xreader.md
Aug 5, 2016
34c871f
Create hbase094xwriter.md
Aug 5, 2016
84fe5e0
Create hbase11xwriter.md
Aug 5, 2016
a75c18d
Update hbase094xwriter.md
Aug 5, 2016
fe39fd4
Create ftpwriter.md
Aug 5, 2016
900e274
Update datax.py
Aug 5, 2016
ff9ae34
Make restriction parameters configurable.
zhouzf05 Aug 10, 2016
8d065dd
Merge pull request #70 from alibaba/otswriter_20160810
liupengjava Aug 10, 2016
f2e17ca
Merge pull request #61 from tianshuang/patch-1
liupengjava Aug 10, 2016
7f56a55
modify readme
Aug 23, 2016
0c9c608
modify datax home page
Aug 23, 2016
bb583f6
Update README.md
binaryWorld Aug 24, 2016
e7ace9b
Merge pull request #63 from projoy/master
binaryWorld Aug 24, 2016
fa091fa
Update README.md
binaryWorld Aug 25, 2016
9c8e36f
补充缺少的plugin_job_template.json
Sep 1, 2016
c407fc2
Update odpswriter.md
officedog Sep 7, 2016
b15daee
Merge pull request #88 from officedog/patch-1
liupengjava Sep 7, 2016
f518daf
Update odpsreader.md
Sep 7, 2016
14f6098
Update odpswriter.md
Sep 7, 2016
5397eed
Update odpswriter.md
Sep 7, 2016
1c2d515
Update odpsreader.md
Sep 7, 2016
af1f856
Update oraclewriter.md
liupengjava Sep 20, 2016
4f0f1f2
Update hdfsreader.md
Sep 21, 2016
d89c778
Update hdfsreader.md
Sep 21, 2016
1c60b27
Update hdfsreader.md
Sep 21, 2016
5f98783
Update hdfsreader.md
Sep 21, 2016
522aced
Update hdfswriter.md
Sep 21, 2016
19b64de
hdfsreader support rcfile,sequencefile,lzo and snappy compress, hdfsr…
Sep 21, 2016
0ce576c
Update hdfswriter.md
Sep 21, 2016
d81db50
Update datax.py
Sep 27, 2016
9d16440
Update ftpreader.md
Oct 6, 2016
20829fd
修复当ftp server搭建在windows环境下时,ftpreader和ftpwriter无法正常工作的bug
kevinwangcs Oct 6, 2016
0d2a503
Update mysqlwriter.md
Oct 8, 2016
150fa52
修复ftpwriter不能递归创建目录的bug
kevinwangcs Oct 10, 2016
baf9888
修复errorLimit设置为空字符串时报错的问题
kevinwangcs Oct 10, 2016
28275dd
Update hdfsreader.md
Nov 16, 2016
1469a18
Update hdfsreader.md
Nov 16, 2016
41194ed
Update mysqlreader.md
Nov 29, 2016
24449a3
Update oraclereader.md
Nov 29, 2016
3e0c4ef
Update sqlserverreader.md
Nov 29, 2016
ce56786
Update postgresqlreader.md
Nov 29, 2016
fd22c5d
Update drdsreader.md
Nov 29, 2016
15d92ee
update split pk wiki
TrafalgarLuo Nov 30, 2016
ac7d0a5
update split pk wiki
TrafalgarLuo Nov 30, 2016
dcf505c
update split pk wiki
TrafalgarLuo Nov 30, 2016
942b25d
HdfsReader occur error when hdfs file is empty,add file length equals 0
changhongzi Dec 13, 2016
f1c37be
csv reader config
TrafalgarLuo Dec 23, 2016
c861efc
csv reader config
TrafalgarLuo Dec 23, 2016
3016214
csv reader config
TrafalgarLuo Dec 23, 2016
659db76
csv reader config
TrafalgarLuo Dec 23, 2016
6be2ea1
csv reader config
TrafalgarLuo Dec 23, 2016
35d5dad
csv reader config
TrafalgarLuo Dec 23, 2016
0e7ea17
csv reader config
TrafalgarLuo Dec 23, 2016
92c05c8
增加DataX Transform说明文档
TrafalgarLuo Dec 30, 2016
08428c2
update transformer wiki
TrafalgarLuo Dec 30, 2016
e39fac6
remove 官方旺旺群:1585662022
TrafalgarLuo Jan 13, 2017
9e80c10
Merge pull request #146 from changhongzi/master
Feb 16, 2017
1b4e73e
Update hdfsreader.md
Feb 21, 2017
02fff05
Update hdfsreader.md
Feb 21, 2017
7082330
Update README.md
binaryWorld Mar 2, 2017
ee17385
update mysql wiki
TrafalgarLuo Mar 16, 2017
9dc04d2
update sqlserver wiki
TrafalgarLuo Mar 16, 2017
2ed8475
update sqlserver wiki
TrafalgarLuo Mar 16, 2017
472906b
update pg wiki
TrafalgarLuo Mar 16, 2017
7b72562
update drdsreader wiki
TrafalgarLuo Mar 16, 2017
46497b6
add license file
TrafalgarLuo Mar 17, 2017
4a45d52
delete datax-opensource-dingding.png
TrafalgarLuo Apr 5, 2017
7e34941
add datax-opensource-dingding.png
TrafalgarLuo Apr 5, 2017
98cc11d
add datax-opensource-dingding.png
TrafalgarLuo Apr 5, 2017
0556c05
update readme wiki
TrafalgarLuo Apr 5, 2017
b2435c7
csvReader默认关闭安全模式, 放开10W字节的限制
kevinwangcs Apr 20, 2017
da21bfa
update drds plugin name
TrafalgarLuo May 10, 2017
26d83e6
ots stream reader.
zhouzf05 May 25, 2017
8be426e
增加rdbms reader writer wiki
TrafalgarLuo Jun 2, 2017
b84cef9
add rdbmsreader.md
TrafalgarLuo Jun 2, 2017
f98cc5d
add rdbmswriter.md
TrafalgarLuo Jun 2, 2017
fb697b8
update rdbmswriter
TrafalgarLuo Jun 2, 2017
7c04b8b
update rdbmswriter
TrafalgarLuo Jun 2, 2017
998e3e3
update rdbmswriter
TrafalgarLuo Jun 2, 2017
028fe89
update rdbmsreader
TrafalgarLuo Jun 2, 2017
d1b8a6e
Merge pull request #198 from alibaba/otsstreamreader_20170525
TrafalgarLuo Jun 14, 2017
3e9e7f3
Create README.md
binaryWorld Jun 14, 2017
44edd18
Create README.md
binaryWorld Jun 14, 2017
c5ff4b6
Create README.md
binaryWorld Jun 14, 2017
0f2b766
Create README.md
binaryWorld Jun 14, 2017
b3fbacd
修改odps reader writer 对bcprov-jdk15on的依赖
TrafalgarLuo Jun 14, 2017
d61684c
update Apache License wiki
TrafalgarLuo Jun 14, 2017
e8d9a46
暂时不讲otsstreamreader放到pom.xml里面
TrafalgarLuo Jul 12, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
.gitignore
.settings
.classpath
.project
target
.git
.idea
*iml
.DS_Store
106 changes: 105 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,107 @@
![Datax-logo](https://github.com/alibaba/DataX/blob/master/images/DataX-logo.jpg)



# DataX

DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台,实现包括 MySQL、Oracle、SqlServer、Postgre、HDFS、Hive、ADS、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。
代码近期会上传,敬请期待。



# Features

DataX本身作为数据同步框架,将不同数据源的同步抽象为从源头数据源读取数据的Reader插件,以及向目标端写入数据的Writer插件,理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统, 每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。



# DataX详细介绍

##### 请参考:[DataX-Introduction](https://github.com/alibaba/DataX/wiki/DataX-Introduction)



# Quick Start

##### Download [DataX下载地址](http://datax-opensource.oss-cn-hangzhou.aliyuncs.com/datax.tar.gz)

##### 请点击:[Quick Start](https://github.com/alibaba/DataX/wiki/Quick-Start)
* [配置示例:从MySQL读取数据 写入ODPS](https://github.com/alibaba/DataX/wiki/Quick-Start)
* [配置定时任务](https://github.com/alibaba/DataX/wiki/%E9%85%8D%E7%BD%AE%E5%AE%9A%E6%97%B6%E4%BB%BB%E5%8A%A1%EF%BC%88Linux%E7%8E%AF%E5%A2%83%EF%BC%89)
* [动态传入参数](https://github.com/alibaba/DataX/wiki/%E5%8A%A8%E6%80%81%E4%BC%A0%E5%85%A5%E5%8F%82%E6%95%B0)



# Support Data Channels

DataX目前已经有了比较全面的插件体系,主流的RDBMS数据库、NOSQL、大数据计算系统都已经接入,目前支持数据如下图,详情请点击:[DataX数据源参考指南](https://github.com/alibaba/DataX/wiki/DataX-all-data-channels)

| 类型 | 数据源 | Reader(读) | Writer(写) |
| ------------ | ---------- | :-------: | :-------: |
| RDBMS 关系型数据库 | Mysql | √ | √ |
| | Oracle | √ | √ |
| | SqlServer | √ | √ |
| | Postgresql | √ | √ |
| | 达梦 | √ | √ |
| 阿里云数仓数据存储 | ODPS | √ | √ |
| | ADS | | √ |
| | OSS | √ | √ |
| | OCS | √ | √ |
| NoSQL数据存储 | OTS | √ | √ |
| | Hbase0.94 | √ | √ |
| | Hbase1.1 | √ | √ |
| | MongoDB | √ | √ |
| 无结构化数据存储 | TxtFile | √ | √ |
| | FTP | √ | √ |
| | HDFS | √ | √ |


# 我要开发新的插件
请点击:[DataX插件开发宝典](https://github.com/alibaba/DataX/wiki/DataX%E6%8F%92%E4%BB%B6%E5%BC%80%E5%8F%91%E5%AE%9D%E5%85%B8)

# 项目成员

核心Contributions: 光戈、一斅、祁然、云时

感谢天烬、巴真对DataX做出的贡献。

# License

This software is free to use under the Apache License [Apache license](https://github.com/alibaba/DataX/blob/master/license.txt).

#
请及时提出issue给我们。请前往:[DataxIssue](https://github.com/alibaba/DataX/issues)

```
长期招聘 联系邮箱:hanfa.shf@alibaba-inc.com
【JAVA开发职位】
职位名称:JAVA资深开发工程师/专家/高级专家
工作年限 : 2年以上
学历要求 : 本科(如果能力靠谱,这些都不是条件)
期望层级 : P6/P7/P8

岗位描述:
1. 负责阿里云大数据平台(数加)的开发设计。
2. 负责面向政企客户的大数据相关产品开发;
3. 利用大规模机器学习算法挖掘数据之间的联系,探索数据挖掘技术在实际场景中的产品应用 ;
4. 一站式大数据开发平台
5. 大数据任务调度引擎
6. 任务执行引擎
7. 任务监控告警
8. 海量异构数据同步

岗位要求:
1. 拥有3年以上JAVA Web开发经验;
2. 熟悉Java的基础技术体系。包括JVM、类装载、线程、并发、IO资源管理、网络;
3. 熟练使用常用Java技术框架、对新技术框架有敏锐感知能力;深刻理解面向对象、设计原则、封装抽象;
4. 熟悉HTML/HTML5和JavaScript;熟悉SQL语言;
5. 执行力强,具有优秀的团队合作精神、敬业精神;
6. 深刻理解设计模式及应用场景者加分;
7. 具有较强的问题分析和处理能力、比较强的动手能力,对技术有强烈追求者优先考虑;
8. 对高并发、高稳定可用性、高性能、大数据处理有过实际项目及产品经验者优先考虑;
9. 有大数据产品、云产品、中间件技术解决方案者优先考虑。
````
钉钉用户请扫描以下二维码进行讨论:

![DataX-OpenSource-Dingding](https://raw.githubusercontent.com/alibaba/DataX/master/images/datax-opensource-dingding.png)


Loading