Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add folder mode #63

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

PauldeKoning
Copy link

@PauldeKoning PauldeKoning commented Jul 16, 2024

Thanks for the script everyone.

Changes

I added a 'folder mode' argument that takes the argument and reads the given directory, then parses the files in it as normal, regardless of file extention.

Why

I had it working on my work macbook (m3), however I was fooling around with it relentlessly on my personal PC (Windows + WSL2). I could not get it working for the life of me with the provided examples or while fooling around with the command.

Does it work?

I tested it on my local windows machine using WSL2 ubuntu 22 with 'moreutils' (for sponge) and 'postgres' installed. Using node v22 I ran the script beneath which resulted in the proper output as I got on my macbook without folder mode.

node cli.js --folder-mode --stops-location-index --require-dependencies -u -- gtfs | sponge | psql -h localhost -p 5432 -b -U postgres

(folder in example script is ./gtfs including all gtfs.txt files)

@derhuerst
Copy link
Member

As gtfs-via-postgres is designed currently, the shell takes care of globbing (expanding the glob into a set of paths).

This has some advantages:

  • Users can use their shell's familiar glob syntax.
  • Shells usually have a more powerful glob syntax (e.g. bash) than what most glob-supporting tools implement.
  • It keeps the tool lean, they don't need to bother with the complexity of glob syntaxes, plain files vs. directories vs. others, etc.

However:

  • Users need some level of familiarity with the shell to know how to make use of globs.
  • Some shells don't have (proper) globbing support](https://stackoverflow.com/a/72468013/1072129), forcing users to implement workarounds.

As-is, I'm inclined to reject this PR because it opens up a "pandoras box" of edge cases and new feature requests:

  1. support exlusion patterns
  2. support inclusion patterns overriding exclusions
  3. support recursing into directories
  4. etc.

However, I'd like to know how I can help future users figure out how to use gtfs-via-postgres. Make some docs on how to do the globbing on WSL2 would help?

@derhuerst derhuerst added the documentation Improvements or additions to documentation label Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Development

Successfully merging this pull request may close these issues.

2 participants