-
Notifications
You must be signed in to change notification settings - Fork 28.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-49378][DOCS][SS] Break apart the Structured Streaming Programm…
…ing Guide ### What changes were proposed in this pull request? These changes break the Structured Streaming Programming Guide into smaller sub-pages **without changing any content**. You can see a preview of it [here](https://nr-spark-site.vercel.app/). I broke up the pages by `h1` tag; within pages, the sub-sections on the left menu are broken up by `h2`. The SS programming guide now will resemble the SQL programming guide and the MLLib programming guide. Additionally, to avoid cluttering the top-level namespace (there are dozens of `sql-*` files for the SQL reference), we nest all streaming docs in by one directory, namely the `/streaming/`. This has the side-effect of breaking links from our `_layouts`, since we assume a flat top-level namespace. To fix this issue, URLs in global layout files now all use absolute paths. This move to `/streaming/` has the consequence that bookmarks of `https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html` will not refer to the actual programming guide content. In anticipation of this, I have kept all pages for existing URLs present with links to the pages in their new locations. This includes the new state data source and the Kafka integration guide. In the future, we'll be able to quite easily (and in-parallel) break the programming guide apart further. This PR does all of the plumbing to make it work.  It is future work to fix the oddly-sized left-navigation bar for our menus. ### Why are the changes needed? One of the major hurdles that users have with Structured Streaming is that our guide is exceptionally long—it feels insurmountable, especially compared to other engines like Flink, which has many sub-pages. Google also has a very tricky time indexing the single large page; if you Google "[structured streaming output mode](https://www.google.com/search?q=structured+streaming+output+mode)" and you click on the link to our programming guide... nothing happens. You aren't taken to the actual content, since Google has trouble with indexing to specific heading tags. ### Does this PR introduce _any_ user-facing change? The structure of the website, with respect to Structured Streaming-related pages, is now changed. See the earlier parts of the PR description for the specific changes. However, **no** content is changed. This should make reviewing the changes much easier. ### How was this patch tested? I have used automated tools (e.g. [Lychee](https://github.com/lycheeverse/lychee)) and manual verification (i.e. clicking on every link) to make sure that I didn't break any links. It isn't fool-proof, though. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47864 from neilramaswamy/nr/streaming-guide-breakapart. Lead-authored-by: Neil Ramaswamy <[email protected]> Co-authored-by: Kent Yao <[email protected]> Signed-off-by: Kent Yao <[email protected]>
- Loading branch information
1 parent
53c1f31
commit 493ca98
Showing
19 changed files
with
5,728 additions
and
5,506 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| - text: Overview | ||
| url: streaming/index.html | ||
| - text: Getting Started | ||
| url: streaming/getting-started.html | ||
| subitems: | ||
| - text: Quick Example | ||
| url: streaming/getting-started.html#quick-example | ||
| - text: Programming Model | ||
| url: streaming/getting-started.html#programming-model | ||
| - text: APIs on DataFrames and Datasets | ||
| url: streaming/apis-on-dataframes-and-datasets.html | ||
| subitems: | ||
| - text: Creating Streaming DataFrames and Streaming Datasets | ||
| url: streaming/apis-on-dataframes-and-datasets.html#creating-streaming-dataframes-and-streaming-datasets | ||
| - text: Operations on Streaming DataFrames/Datasets | ||
| url: streaming/apis-on-dataframes-and-datasets.html#operations-on-streaming-dataframesdatasets | ||
| - text: Starting Streaming Queries | ||
| url: streaming/apis-on-dataframes-and-datasets.html#starting-streaming-queries | ||
| - text: Managing Streaming Queries | ||
| url: streaming/apis-on-dataframes-and-datasets.html#managing-streaming-queries | ||
| - text: Monitoring Streaming Queries | ||
| url: streaming/apis-on-dataframes-and-datasets.html#monitoring-streaming-queries | ||
| - text: Recovering from Failures with Checkpointing | ||
| url: streaming/apis-on-dataframes-and-datasets.html#recovering-from-failures-with-checkpointing | ||
| - text: Recovery Semantics after Changes in a Streaming Query | ||
| url: streaming/apis-on-dataframes-and-datasets.html#recovery-semantics-after-changes-in-a-streaming-query | ||
| - text: Performance Tips | ||
| url: streaming/performance-tips.html | ||
| subitems: | ||
| - text: Asynchronous Progress Tracking | ||
| url: streaming/performance-tips.html#asynchronous-progress-tracking | ||
| - text: Continuous Processing | ||
| url: streaming/performance-tips.html#continuous-processing | ||
| - text: Additional Information | ||
| url: streaming/additional-information.html | ||
| subitems: | ||
| - text: Miscellaneous Notes | ||
| url: streaming/additional-information.html#miscellaneous-notes | ||
| - text: Related Resources | ||
| url: streaming/additional-information.html#related-resources | ||
| - text: Migration Guide | ||
| url: streaming/additional-information.html#migration-guide |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| {% comment %} | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| {% endcomment %} | ||
| <div class="left-menu-wrapper"> | ||
| <div class="left-menu"> | ||
| <h3><a href="{{ rel_path_to_root }}streaming/index.html">Structured Streaming Programming Guide</a></h3> | ||
| {% include nav-left.html nav=include.nav-streaming %} | ||
| </div> | ||
| </div> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.