[R] CRAN packaging checklist for version 18.0.0

[amoeba]

Describe the enhancement requested

Packaging checklist for CRAN release

For a high-level overview of the release process see the
Apache Arrow Release Management Guide.

cc @jonkeane @thisisnic @amoeba @nealrichardson @assignUser @paleolimbot for coordination on the various tasks to release

Before the release candidate is cut

• Create a GitHub issue entitled [R] CRAN packaging checklist for version X.X.X and copy this checklist to the issue.
• Review deprecated functions to advance their deprecation status, including removing preprocessor directives that no longer apply (search for ARROW_VERSION_MAJOR in r/src).
• Evaluate the status of any failing nightly tests and nightly packaging builds. These checks replicate most of the checks that CRAN runs, so we need them all to be passing or to understand that the failures may (though won't necessarily) result in a rejection from CRAN.
• Check current CRAN check results
• Ensure the contents of the README are accurate and up to date.
• Run urlchecker::url_check() on the R directory at the release candidate.
  commit. Ignore any errors with badges as they will be removed in the CRAN release branch.
• Polish NEWS but do not update version numbers (this is done automatically later). You can find commits by, for example, git log --oneline <sha of last release>..HEAD | grep "\[R\]"
• Run preliminary reverse dependency checks using archery docker run r-revdepcheck.
• For major releases, prepare tweet thread highlighting new features.

Wait for the release candidate to be cut:

After release candidate has been cut

• Create a CRAN-release branch from the release candidate commit, name the new branch e.g. maint-X.X.X-r and push to upstream
  • Created, see https://github.com/apache/arrow/tree/maint-18.0.0-r

Prepare and check the .tar.gz that will be released to CRAN.

• git fetch upstream && git checkout release-X.X.X-rcXX && git clean -f -d
• Run make build. This copies Arrow C++ into tools/cpp, prunes some
  unnecessary components, and runs R CMD build to generate the source tarball.
  Because this will install the package, you will need to ensure that the version
  of Arrow C++ available to the configure script is the same as the version
  that is vendored into the R package (e.g., you may need to unset ARROW_HOME).
• devtools::check_built("arrow_X.X.X.tar.gz") locally
• Run reverse dependency checks using archery docker run r-revdepcheck.

Release vote

• Release vote passed!

Generate R package to submit to CRAN

• If the release candidate commit updated, rebase the CRAN release branch
  on that commit.
• Pick any commits that were made to main since the release commit that
  were needed to fix CRAN-related submission issues identified in the above
  steps.
• Remove badges from README.md
• Run urlchecker::url_check() on the R directory
• Create a PR entitled WIP: [R] Verify CRAN release-10.0.1-rc0. Add
  a comment @github-actions crossbow submit --group r to run all R crossbow
  jobs against the CRAN-specific release branch.
• Run Rscript tools/update-checksums.R <libarrow version> to download the checksums for the pre-compiled binaries from the ASF artifactory into the tools directory.
• Regenerate arrow_X.X.X.tar.gz (i.e., make build)

Check binary Arrow C++ distributions specific to the R package

• Upload the .tar.gz to win-builder (r-devel only)
  and confirm (with Jon, who will automatically receive an email about the results) that the check is clean.
• Upload the .tar.gz to MacBuilder
  and confirm that the check is clean
• Check install.packages("arrow_X.X.X.tar.gz") on Ubuntu and ensure that the
  hosted binaries are used
• devtools::check_built("arrow_X.X.X.tar.gz") locally one more time (for luck)

CRAN submission

• Upload arrow_X.X.X.tar.gz to the
  CRAN submit page
• Confirm the submission email

Wait for CRAN...

• Accepted!
• Tag the tip of the CRAN-specific release branch with r-universe-release
• Add a new line to the matrix in the backwards compatability job
• (patch releases only) Update the package version in ci/scripts/PKGBUILD, dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb, r/DESCRIPTION, and r/NEWS.md
• (CRAN-only releases) Rebuild news page with pkgdown::build_news() and submit a PR to the asf-site branch of the docs site with the contents of arrow/r/docs/news/index.html replacing the current contents of arrow-site/docs/r/news/index.html
• (CRAN-only releases) Bump the version number in r/pkgdown/assets/versions.json, and update this on the the asf-site branch of the docs site too.
• Update the packaging checklist template to reflect any new realities of the
  packaging process.
• Wait for CRAN-hosted binaries on the
  CRAN package page to reflect the
  new version
• Tweet!
  • Use Bryce's script for contributor calculation.

Component(s)

R

[thisisnic]

Thanks for kicking this off @amoeba! I'll make a PR with the NEWS updates

[amoeba]

Thanks @thisisnic. I started on a NEWS.md update while working on other releases and so far have this if it helps:

18.0.0 NEWS.md content

## New features

* Dictionary columns can now be filtered with the `%in%` operator in dplyr
  pipelines. Previously you would have to cast the dictionary column to a string
  column prior to filtering. (#43440)
* The binding for stringr's `str_sub` function now properly handles negative end
  values instead of returning an empty string. (#43960)

## Minor improvements and fixes

* A performance regression in the binding for dplyr's `summarize` has been
  fixed. This should now be approximately as fast as it was in arrow 16.1.0.
  (#43627)
* The ParquetFileWriter class' WriteTable method can now take a RecordBatch as
  as an argument in addition to a Table. (#42240)
* Add link to ?acero from ?list_compute_functions. (#43509)

[thisisnic]

Ah, I just submitted https://github.com/apache/arrow/pull/44520/files - wanna add suggestions there?

[amoeba]

Will do! Working on revdep checks now and will look a bit later.

[amoeba]

I merged #44520 and cherry-picked it onto maint-18.0.0-r. Thanks @thisisnic.

This is waiting on RC vote to be finalized, release to be cut, binaries to be uploaded and then I think we can move forward once that's done. I can start up a social media post draft Monday.

[thisisnic]

Thanks very much @amoeba! Are you on BlueSky? I think we should consider posting there as it appears that a lot of the R community have gone there.

[amoeba]

I am , I see you too. I'll plan to post over there this release too. So that means we're covering Twitter, Mastodon, BlueSky (me), and LinkedIn (Jon).

[nealrichardson]

About the news, if it's not too late to cherry-pick, I made #44562. NBD if it's too late though.

[amoeba]

Not too late. Cherry-picked as 5a57166. I think libarrow binaries for R should be in place so I can kick that test off soon and I'll continue on this once I finish some Arrow release stuff today.

[amoeba]

I prepped the submission archive locally following the instructions, submitted to win-builder (r-devel only) and macbuilder (r-release and r-devel), and saved a copy of the archive to my personal Google Drive. We still have two CI failures in #44523 to investigate.

[thisisnic]

Looks like we may want to hold off a moment: https://lists.apache.org/thread/t8k7l2hsbgdt7cszj7hrpjdfpn91n5zb

[jonkeane]

I second @thisisnic on holding our release for the 18.0.1. On the positive side, aside from the mirror selection thing that Bryce found in #44523 CI looks good (the devdocs failure is a bit odd, but isn't failing on main, so might be an issue with that running on the release maintenance branch) so hopefully once 18.0.1 is here it'll be a pretty easy submission.

Thanks for all of the work here @amoeba (and others!)

[amoeba]

Holding off sounds good. I'm going to close this issue and note for future-us visiting this issue that we chose not to submit 18.0.0 to CRAN in favor of waiting for the upcoming 18.0.1/18.1.0 release to make our first submission in the 18.x.x series.

Thanks all for helping with this submission. I'll plan to kickstart the next submission checklist and begin on the items myself, pinging folks as needed.