awesome-spark/awesome-spark

ipex-llm by intel-analytics

Accelerate local LLM inference and finetuning (LLaMA, Mistral, ChatGLM, Qwen, Baichuan, Mixtral, Gemma, Phi, etc.) on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max); seamlessly integrate with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, DeepSpeed, vLLM, FastChat, Axolotl, etc.

updated at June 9, 2024, 1:48 a.m.

Python

245 +1

6,135 +20

1,212 +4

GitHub

joblib by joblib

Computing with Python functions.

updated at June 9, 2024, 12:29 a.m.

Python

63 +0

3,706 +7

409 +0

GitHub

scikit-learn by scikit-learn

scikit-learn: machine learning in Python

updated at June 8, 2024, 8:56 p.m.

Python

2,140 -1

58,571 +82

25,067 +33

GitHub

quinn by MrPowers

pyspark methods to enhance developer productivity 📣 👯 🎉

updated at June 8, 2024, 3:53 p.m.

Python

19 +0

586 +2

92 +0

GitHub

hail by hail-is

Cloud-native genomic dataframes and batch computing

updated at June 7, 2024, 11:09 p.m.

Python

55 +0

946 +2

238 +0

GitHub

sparkmagic by jupyter-incubator

Jupyter magics and kernels for working with remote Spark clusters

updated at June 6, 2024, 8:59 a.m.

Python

49 +0

1,294 +3

439 +1

GitHub

joblib-spark by joblib

Joblib Apache Spark Backend

updated at June 5, 2024, 12:48 p.m.

Python

9 +0

239 +0

26 +0

GitHub

koalas by databricks

Koalas: pandas API on Apache Spark

updated at May 30, 2024, 5:09 p.m.

Python

319 +0

3,319 +0

354 +0

GitHub

spark-sklearn by databricks

(Deprecated) Scikit-learn integration package for Apache Spark

updated at May 27, 2024, 4:27 p.m.

Python

94 -1

1,076 +0

230 -1

GitHub

blaze by blaze

NumPy and Pandas interface to Big Data

updated at May 27, 2024, 8:30 a.m.

Python

195 +0

3,180 +0

389 +1

GitHub

flintrock by nchammas

A command-line tool for launching Apache Spark clusters.

updated at May 26, 2024, 5:09 p.m.

Python

32 +0

633 +0

115 +0

GitHub

dist-keras by cerndb

Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.

updated at May 10, 2024, 5:12 a.m.

Python

49 +0

623 +0

170 +0

GitHub

sparkly by Tubular

Helpers & syntactic sugar for PySpark.

updated at Dec. 22, 2023, 2:37 a.m.

Python

38 +0

60 +0

7 +0

GitHub

pyspark-stubs by zero323

Apache (Py)Spark type annotations (stub files).

updated at Sept. 16, 2023, 6:30 p.m.

Python

6 +0

114 +0

37 +0

GitHub