airflow by apache

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

created at April 13, 2015, 6:04 p.m.

754 +2

34,425 +83

13,523 +18

dash by plotly

Data Apps & Dashboards for Python. No JavaScript Required.

created at April 10, 2015, 1:53 a.m.

416 -1

20,471 +39

1,985 +2

luigi by spotify

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

created at Sept. 20, 2012, 3:06 p.m.

475 +0

17,300 +15

2,371 +1

dagster by dagster-io

An orchestration platform for the development, production, and observation of data assets.

created at April 30, 2018, 4:30 p.m.

112 +0

10,180 +58

1,259 +6

aws-sdk-pandas by aws

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

created at Feb. 26, 2019, 1:39 a.m.

61 +0

3,799 +8

666 +2

flocker by ClusterHQ

Container data volume manager for your Dockerized application

created at April 28, 2014, 6:02 p.m.

168 +0

3,376 +0

286 +0

smart_open by piskvorky

Utils for streaming large files (S3, HDFS, gzip, bz2...)

created at Jan. 2, 2015, 1:05 p.m.

49 +0

3,087 +3

378 +1

PyHive by dropbox

Python interface to Hive and Presto. 🐝

created at Feb. 1, 2014, 9:05 a.m.

62 +0

1,665 +2

552 +0

faust by faust-streaming

Python Stream Processing. A Faust fork

created at Oct. 22, 2020, 3:32 p.m.

28 +0

1,445 +6

170 -1

DataProfiler by capitalone

What's in your data? Extract schema, statistics and entities from datasets

created at Nov. 9, 2020, 3:20 p.m.

21 +0

1,359 +5

154 +0

ccm by riptano

A script to easily create and destroy an Apache Cassandra cluster on localhost

created at March 1, 2011, 9:42 a.m.

76 +0

1,212 +0

302 +0

mysql_utils by pinterest

Pinterest MySQL Management Tools

created at Oct. 24, 2015, 5:33 p.m.

72 +0

878 +0

146 +0

snakebite by spotify

A pure python HDFS client

created at May 7, 2013, 9:44 a.m.

130 +0

858 +0

216 +0

datacompy by capitalone

Pandas and Spark DataFrame comparison for humans and more!

created at March 23, 2018, 1:16 p.m.

25 +0

379 +2

118 +0