Skip to content

Commit

Permalink
Minor: consolidate doc roadmap pages (apache#11426)
Browse files Browse the repository at this point in the history
  • Loading branch information
alamb authored and findepi committed Jul 16, 2024
1 parent 99e34fa commit 8977cf5
Show file tree
Hide file tree
Showing 3 changed files with 81 additions and 97 deletions.
96 changes: 0 additions & 96 deletions docs/source/contributor-guide/quarterly_roadmap.md

This file was deleted.

81 changes: 81 additions & 0 deletions docs/source/contributor-guide/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,84 @@ start a conversation using a github issue or the
make review efficient and avoid surprises.

[The current list of `EPIC`s can be found here](https://github.com/apache/datafusion/issues?q=is%3Aissue+is%3Aopen+epic).

# Quarterly Roadmap

A quarterly roadmap will be published to give the DataFusion community
visibility into the priorities of the projects contributors. This roadmap is not
binding and we would welcome any/all contributions to help keep this list up to
date.

## 2023 Q4

- Improve data output (`COPY`, `INSERT` and DataFrame) output capability [#6569](https://github.com/apache/datafusion/issues/6569)
- Implementation of `ARRAY` types and related functions [#6980](https://github.com/apache/datafusion/issues/6980)
- Write an industrial paper about DataFusion for SIGMOD [#6782](https://github.com/apache/datafusion/issues/6782)

## 2022 Q2

### DataFusion Core

- IO Improvements
- Reading, registering, and writing more file formats from both DataFrame API and SQL
- Additional options for IO including partitioning and metadata support
- Work Scheduling
- Improve predictability, observability and performance of IO and CPU-bound work
- Develop a more explicit story for managing parallelism during plan execution
- Memory Management
- Add more operators for memory limited execution
- Performance
- Incorporate row-format into operators such as aggregate
- Add row-format benchmarks
- Explore JIT-compiling complex expressions
- Explore LLVM for JIT, with inline Rust functions as the primary goal
- Improve performance of Sort and Merge using Row Format / JIT expressions
- Documentation
- General improvements to DataFusion website
- Publish design documents
- Streaming
- Create `StreamProvider` trait

### Ballista

- Make production ready
- Shuffle file cleanup
- Fill functional gaps between DataFusion and Ballista
- Improve task scheduling and data exchange efficiency
- Better error handling
- Task failure
- Executor lost
- Schedule restart
- Improve monitoring and logging
- Auto scaling support
- Support for multi-scheduler deployments. Initially for resiliency and fault tolerance but ultimately to support sharding for scalability and more efficient caching.
- Executor deployment grouping based on resource allocation

### Extensions ([datafusion-contrib](https://github.com/datafusion-contrib))

### [DataFusion-Python](https://github.com/datafusion-contrib/datafusion-python)

- Add missing functionality to DataFrame and SessionContext
- Improve documentation

### [DataFusion-S3](https://github.com/datafusion-contrib/datafusion-objectstore-s3)

- Create Python bindings to use with datafusion-python

### [DataFusion-Tui](https://github.com/datafusion-contrib/datafusion-tui)

- Create multiple SQL editors
- Expose more Context and query metadata
- Support new data sources
- BigTable, HDFS, HTTP APIs

### [DataFusion-BigTable](https://github.com/datafusion-contrib/datafusion-bigtable)

- Python binding to use with datafusion-python
- Timestamp range predicate pushdown
- Multi-threaded partition aware execution
- Production ready Rust SDK

### [DataFusion-Streams](https://github.com/datafusion-contrib/datafusion-streams)

- Create experimental implementation of `StreamProvider` trait
1 change: 0 additions & 1 deletion docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,6 @@ To get started, see
contributor-guide/testing
contributor-guide/howtos
contributor-guide/roadmap
contributor-guide/quarterly_roadmap
contributor-guide/governance
contributor-guide/inviting
contributor-guide/specification/index
Expand Down

0 comments on commit 8977cf5

Please sign in to comment.