+tree-sitter (380/548) #326

mfts · 2023-02-10T22:36:49Z

Formulae: https://formulae.brew.sh/formula/tree-sitter

king-11 · 2023-08-05T05:03:16Z

I am trying to work on this one

king-11 · 2023-08-05T05:06:45Z

@jhheider @mfts if you could help I ran the following operations and am seeing this error:

gh repo clone teaxyz/pantry
cd pantry
export TEA_PANTRY_PATH=$(pwd)
tea pkg init github.com/tree-sitter/tree-sitter
tea pkg build

Before build I edited package.yml as:

# this is the source tarball. Swap the version out for {{version}} or we won’t
# be able to upgrade the package when it updates!
distributable:
  url: https://github.com/tree-sitter/tree-sitter/archive/v{{version}}.tar.gz
  # this obscure option dictates how many components are removed from the
  # extracted tarball. It only works with tarballs. You almost always want
  # `1`. If after extraction the src-directory is funky try `0` or `2`.
  strip-components: 1

# if there’s a github then we can parse the versions
versions:
  # if the repo has a github then we can parse versions
  # this should be one of these (remove the rest or the YAML won’t parse)
  github: tree-sitter/tree-sitter  # reads github tags but only tags of releases (this is usually what you want)

dependencies:
  # to figure out dependencies you can read the project’s README
  # however it is often easier to read the [brew formula](https://formulae.brew.sh/formula/)
  #
  # see tea.xyz/+/ for the packages tea provides
  # if you need a package that isn’t there, you may need to build it!
  # we can help you: https://discord.tea.xyz
  #
  # see semver.org for version specifications
  node: '*'
  emscripten: '*'

build:
  dependencies:
    node: '*'
    emscripten: '*'
    rust: '*'
    # ^^ often packages need dependencies to build, it is very common for
    # packages to need a C compiler and make, the above provide those
  working-directory: build
  script: |
    cargo build -r
  # it’s extremely common for packages to require the above

  # script also supports a list of commands, which is useful for complex builds
  # or if you want to run a command in a different directory

provides:
  - {{prefix}}/target/release/tree-sitter
  # ^^ specify the binaries or tea magic won’t work

The error I am getting is:

error: Uncaught Error: Not in pantry: \ngithub.com/tree-sitter/tree-sitter

https://github.com/teaxyz/pantry#contributing
      throw new TeaError('not-found: pantry: package.yml', {project}, )
            ^
    at https://deno.land/x/[email protected]/src/hooks/usePantry.ts:49:13
    at project (https://deno.land/x/[email protected]/src/hooks/usePantry.ts:50:5)
    at Object.getDeps (file:///home/king-11/.tea/tea.xyz/brewkit/v0.44.1/lib/usePantry.ts:30:21)
    at file:///home/king-11/.tea/tea.xyz/brewkit/v0.44.1/libexec/sort.ts:24:29
    at ascend (https://deno.land/x/[email protected]/src/plumbing/hydrate.ts:62:31)
    at go (https://deno.land/x/[email protected]/src/plumbing/hydrate.ts:93:11)
    at hydrate (https://deno.land/x/[email protected]/src/plumbing/hydrate.ts:102:13)
    at file:///home/king-11/.tea/tea.xyz/brewkit/v0.44.1/libexec/sort.ts:23:19

jhheider · 2023-08-05T05:21:51Z

Error: Not in pantry: \ngithub.com/tree-sitter/tree-sitter looks strange. I wonder where that \n came from.

But other things that will need to be fixed:

node > nodejs.org
emscripten > not packaged; will need to be packaged if needed
rust > rust-lang.org
all runtime dependencies are also build dependencies, so node and emscripten are duplicates there
files must be created under {{prefix}} by the build script, or the build will be considered a failure; for crates, this is usually: cargo install --root "{{prefix}}" --path .
crates should generally be namespaced under crates.io, i.e. crates.io/tree-sitter
provides: should be an array of executables (usually bin/foo, and sbin/bar). here is the code that audits those lines. it'll likely be bin/tree-sitter.

All that said, I'm concerned that there's a newline in your package name somewhere (either the branch name, the project path, or being mentioned somewhere else. If I run the last two of your commands, I get the following (darwin+aarch64):

$ pkg init github.com/tree-sitter/tree-sitter; pkg build
branch 'new/github.com/tree-sitter/tree-sitter' set up to track 'origin/main'.
Switched to a new branch 'new/github.com/tree-sitter/tree-sitter'
  
  created  ./projects/github.com/tree-sitter/tree-sitter/package.yml 
  
  Now type  pkg edit  to open the yaml in your  $EDITOR 
  
  the package will need renaming before we can merge it.
  we typically name packages after their homepage.
  
  if you aren’t sure about the name you can submit and we’ll assist.
error: Uncaught Error: pantry entry invalid. please report this bug!

    https://github.com/teaxyz/pantry/issues/new

----------------------------------------------------->> attachment begin
github.com/tree-sitter/tree-sitter: duplicated mapping key in "/Users/jacob/tea/pantry/projects/github.com/tree-sitter/tree-sitter/package.yml" at line 23, column 3:
      url: git+https://github.com/user ... 
      ^
<<------------------------------------------------------- attachment end
          .catch(cause => { throw new TeaError('parser: pantry: package.yml', {cause, project, filename}) }))

which is expected, since that's an unedited package.yml.

Actually,

$ pwd | hexdump
0000000 552f 6573 7372 6a2f 6361 626f 742f 6165
0000010 702f 6e61 7274 0a79                    
0000018

it looks like pwd might include a new line. That might be the culprit. export TEA_PANTRY_PATH=$(pwd) does work for me, but that's the best guess I have at the moment. :/

jhheider · 2023-08-05T05:26:15Z

I updated my brewkit to be sure, but it all still works the same. So, my top possible culprits are:

does linux pwd work differently?
does your shell treat newlines in envvars differently than mine (OSX built-in zsh)
some other problem that's getting a newline in the middle of a string

Since most of brewkit is shell scripts, you can get more info out with bash -x "$(tea --prefix)"/tea.xyz/brewkit/v0/bin/pkg-build to have it show all the commands and values it's using.

king-11 · 2023-08-08T06:19:00Z

++ tr : '\n'
+ for x in $(echo "$TEA_PANTRY_PATH" | tr ':' '\n')
+ test -d /home/king-11/pantry/.git
++ GIT_WORK_TREE=/home/king-11/pantry
++ peek.sh
+ PKGS='\ncrates.io/tree-sitter '

Seems like the package name only has issues where can I correct that?

king-11 · 2023-08-08T09:38:49Z

So I tried exporting TEA_PANTRY_PATH manually without pwd still the error is same:

➜ bash -x "$(tea --prefix)"/tea.xyz/brewkit/v0/bin/pkg-build
+ set -e
+ pwd
+ grep -q ' '
+ test -n '' -a -n '' -o -n ''
+++ dirname /home/king-11/.tea/tea.xyz/brewkit/v0/bin/pkg-build
++ cd /home/king-11/.tea/tea.xyz/brewkit/v0/bin/..
++ pwd
+ d=/home/king-11/.tea/tea.xyz/brewkit/v0
+ PATH='/home/king-11/.tea/tea.xyz/brewkit/v0/libexec:/home/king-11/.local/share/zinit/polaris/bin:/home/king-11/.local/bin:/home/king-11/.local/share/pnpm:/run/user/1000/fnm_multishells/16926_1691475666114/bin:/home/king-11/.local/share/fnm:/home/king-11/.pyenv/shims:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/lib64/ccache:/home/king-11/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0/:/mnt/c/windows/System32/OpenSSH/:/mnt/c/Program Files/usbipd-win/:/Docker/host/bin:/mnt/c/Users/lakshyasingh/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/lakshyasingh/AppData/Local/Programs/Microsoft VS Code/bin:/var/lib/snapd/snap/bin'
+ test -z ''
+ test -z /home/king-11/pantry
++ echo /home/king-11/pantry
++ tr : '\n'
+ for x in $(echo "$TEA_PANTRY_PATH" | tr ':' '\n')
+ test -d /home/king-11/pantry/.git
++ GIT_WORK_TREE=/home/king-11/pantry
++ peek.sh
+ PKGS='\ncrates.io/tree-sitter '
+ test -n ''
+ available.ts
+ test -z ''
++ tea gh auth token
+ GITHUB_TOKEN=<GH_TOKEN>
+ test -z <GH_TOKEN>
+ export GITHUB_TOKEN
++ sort.ts '\ncrates.io/tree-sitter' --delimiter ' '
error: Uncaught Error: Not in pantry: \ncrates.io/tree-sitter

https://github.com/teaxyz/pantry#contributing
      throw new TeaError('not-found: pantry: package.yml', {project}, )
            ^
    at https://deno.land/x/[email protected]/src/hooks/usePantry.ts:49:13
    at project (https://deno.land/x/[email protected]/src/hooks/usePantry.ts:50:5)
    at Object.getDeps (file:///home/king-11/.tea/tea.xyz/brewkit/v0/lib/usePantry.ts:30:21)
    at file:///home/king-11/.tea/tea.xyz/brewkit/v0/libexec/sort.ts:24:29
    at ascend (https://deno.land/x/[email protected]/src/plumbing/hydrate.ts:62:31)
    at go (https://deno.land/x/[email protected]/src/plumbing/hydrate.ts:93:11)
    at hydrate (https://deno.land/x/[email protected]/src/plumbing/hydrate.ts:102:13)
    at file:///home/king-11/.tea/tea.xyz/brewkit/v0/libexec/sort.ts:23:19
+ PKGS=

jhheider · 2023-08-08T17:29:14Z

Ok, we'll we can definitely see the error showing up out of peek.sh. This is the call:

PKGS="$(GIT_WORK_TREE="$x" peek.sh) $PKGS"

The relevant code in peek.sh is:

OUTPUT=""

for CHANGED_FILE in $CHANGED_FILES; do
  PROJECT=$(echo "$CHANGED_FILE" | sed -n 's#projects/\(.*\)/package\.yml$#\1#p')
  if test -z "$PROJECT"
  then
    true # noop
  elif test "$1" = "--print-paths"; then
    OUTPUT="$OUTPUT\n$CHANGED_FILE"
  else
    OUTPUT="$OUTPUT\n$PROJECT"
  fi
done

# shellcheck disable=SC2046
echo $(echo $OUTPUT | sort | uniq)

So, your /bin/sh (dash I assume, under a ubuntu/debian derivative?) is taking:

echo $(echo "\ncrates.io/tree-sitter " | sort | uniq)

and returnting \ncrates.io/tree-sitter . That's not great. Quick test: if you replace the end of ~/.tea/tea.xyz/brewkit/v0/libexec/peek.sh to the following, does it work?

  elif test "$1" = "--print-paths"; then
    OUTPUT="$OUTPUT $CHANGED_FILE"
  else
    OUTPUT="$OUTPUT $PROJECT"
  fi
done

# shellcheck disable=SC2046
echo $(echo $OUTPUT | tr ' ' '\n' | sort | uniq | tr '\n' ' ')

king-11 · 2023-08-08T18:15:01Z

that does it

jhheider · 2023-08-08T18:26:46Z

Great. I think it might behoove us to just change all of brewkit to use #!/usr/bin/env -S tea bash for sanity's sake. This is at least the second dash idiosyncrasy that's shown up.

king-11 · 2023-08-08T18:29:25Z

Right when I execute pkg build it gets stuck:

pkg build                                                                    
  
  building crates.io/tree-sitter
  
  fetching /home/king-11/pantry/srcs/crates.io∕tree-sitter-0.20.8.tar.gz
/home/king-11/pantry/srcs/crates.io∕tree-sitter-0.20.8.tar.gz
  
  extracting /home/king-11/pantry/srcs/crates.io∕tree-sitter-0.20.8
  
  staging for build

Should it be taking this long? package.yml is as below now:

distributable:
  url: https://github.com/tree-sitter/tree-sitter/archive/refs/tags/v{{version}}.tar.gz
  strip-components: 1
 
provides:
  - bin/tree-sitter

platforms:
  - linux

versions:
  github: tree-sitter/tree-sitter
  strip: /v/

build:
  dependencies:
    rust-lang.org: ^1.65
    rust-lang.org/cargo: '*'
  script:
    cargo install --locked --path . --root {{prefix}}

king-11 · 2023-08-08T18:34:48Z

Didn't really get what you meant by dash there: the - redirection of pipe or the shell that am using? Very much not familiar with that 😅

jhheider · 2023-08-08T18:40:55Z

Sorry, dash is the shell debian/ubuntu and other derivatives use for /bin/sh. It's POSIX compliant, but it has a few dark corners that can catch you out if you develop under zsh/bash. We had a bug where dash doesn't support set -o pipefail. The real/final solution is pkgxdev/brewkit#163. Controlling environments is tea's raison d'être, so we should use it. ;)

jhheider · 2023-08-08T19:25:05Z

Ok, try tea -S +tea.xyz/brewkit=0.44.2 && cd . once this finishes and see if that fixes it. I've moved all the scripts to use tea's bash package.

king-11 · 2023-08-16T05:42:25Z

the build takes unusually long but seems to be working now. i will start getting the package.yml ready to send a PR.

mfts added the top-300 Top 300 packages label Feb 13, 2023

mfts changed the title ~~+tree-sitter~~ +tree-sitter (380/548) Mar 13, 2023

mfts mentioned this issue Mar 15, 2023

The Top 300 #99

Open

jhheider mentioned this issue Aug 8, 2023

use tea bash instead of sh to avoid odd dash issues pkgxdev/brewkit#162

Closed

jhheider mentioned this issue Aug 14, 2023

pkg command adds "\n" to the path on Ubuntu pkgxdev/brewkit#167

Closed

king-11 mentioned this issue Aug 16, 2023

feat: tree sitter pkg #2929

Merged

jhheider mentioned this issue Aug 16, 2023

fix peek.sh injecting leading newlines pkgxdev/brewkit#169

Merged

jhheider closed this as completed in #2929 Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

+tree-sitter (380/548) #326

+tree-sitter (380/548) #326

mfts commented Feb 10, 2023

king-11 commented Aug 5, 2023

king-11 commented Aug 5, 2023 •

edited

Loading

jhheider commented Aug 5, 2023

jhheider commented Aug 5, 2023

king-11 commented Aug 8, 2023 •

edited

Loading

king-11 commented Aug 8, 2023

jhheider commented Aug 8, 2023

king-11 commented Aug 8, 2023

jhheider commented Aug 8, 2023

king-11 commented Aug 8, 2023 •

edited

Loading

king-11 commented Aug 8, 2023

jhheider commented Aug 8, 2023 •

edited

Loading

jhheider commented Aug 8, 2023

king-11 commented Aug 16, 2023

+tree-sitter (380/548) #326

+tree-sitter (380/548) #326

Comments

mfts commented Feb 10, 2023

king-11 commented Aug 5, 2023

king-11 commented Aug 5, 2023 • edited Loading

jhheider commented Aug 5, 2023

jhheider commented Aug 5, 2023

king-11 commented Aug 8, 2023 • edited Loading

king-11 commented Aug 8, 2023

jhheider commented Aug 8, 2023

king-11 commented Aug 8, 2023

jhheider commented Aug 8, 2023

king-11 commented Aug 8, 2023 • edited Loading

king-11 commented Aug 8, 2023

jhheider commented Aug 8, 2023 • edited Loading

jhheider commented Aug 8, 2023

king-11 commented Aug 16, 2023

king-11 commented Aug 5, 2023 •

edited

Loading

king-11 commented Aug 8, 2023 •

edited

Loading

king-11 commented Aug 8, 2023 •

edited

Loading

jhheider commented Aug 8, 2023 •

edited

Loading