Move to Path::Tiny #1264

xsawyerx · 2016-10-06T20:36:35Z

Path::Tiny provides a clean, correct, and consistent interface to various path-related modules. We have our own module, which could be improved.

I want to at least replace the core of it with Path::Tiny and maybe remove it at some point in favor.

This is a work in progress. I have more work to commit once the tests pass.

Clean up FileUtils internals.
Clean up usages of File::Spec.
Clean up usages of other File::* modules.
File::Basename -> path($path)->basename.
(basedir is not available.)
Clean up usages of -d -> path($path)->is_dir.
Clean up usages of -f -> path($path)->is_file.
Clean up usages of -e -> path($path)->exists.

xsawyerx · 2017-02-28T20:33:14Z

I have rewritten and rebased this branch. It's basically waiting on me to finish merge_path from the File Handler, but I need to first figure out what it's doing. :/

xsawyerx · 2017-03-01T19:35:59Z

Last commit fixes that.
My next part is cleaning up the explicit calls to Path::Tiny::path() and then I'll start removing the stringify that exists everywhere.

wchristian · 2017-03-01T23:52:39Z

That merge_paths commit is no good in that form, i'm afraid.

All of these should throw errors instead of "succeeding", otherwise they present huge security problems and/or break tests on systems with more than one file system tree. @haarg probably has other ideas of how this could break as well.

C:\Users\Mithaldu>perl -e "use Path::Tiny; print Path::Tiny::path( 'c:/meep', 'c:/marp' )->stringify"
C:/meep/c:/marp
C:\Users\Mithaldu>perl -e "use Path::Tiny; print Path::Tiny::path( 'c:/meep', 'd:/marp' )->stringify"
C:/meep/d:/marp
C:\Users\Mithaldu>perl -e "use Path::Tiny; print Path::Tiny::path( 'c:/meep', '../marp' )->stringify"
C:/marp

xsawyerx · 2017-03-02T10:20:06Z

@wchristian, Thank you for testing them out. I'm not sure what these should have been doing to begin with. What does it mean to ask for "c:/meep", "d:/marp"? These are different volumes.

wchristian · 2017-03-02T12:58:16Z

The first two are both straight-up non-sense in the vein of merge_path( "~/.ssh", "ftp://some.server/whatever" ) so they should error out, nothing else.

The third is "merely" dangerous and should be prevented.

ambs · 2017-10-08T19:43:51Z

@xsawyerx, ,what is missing here? Probably update the todo list above?

xsawyerx · 2019-09-08T08:47:07Z

I just rebased this branch (had a few conflicts) and I wrote a solution to the issues raised by @Mithaldu.

The short answer is: 2 out of 3 cases get it wrong with both merge_paths (our code that I'm removing) and with Path::Tiny, but it's okay because our code will catch it.

The problem with the third case is a security issue, but not because of what Path::Tiny does. It does the right thing - but we assume the user is correct in doing this, which is a possible security issue.

I've resolved the third issue but I want to write some tests before I commit and push these too.

xsawyerx · 2019-09-08T20:18:17Z

I added the fix and the tests. Seems be be good. I originally also patched Dancer2::Handler::File but coudn't find any test for it. Then I couldn't find anything using it. I made a syntax error in i and ran the entire testing suite - successful. Are we even using this module anymore?

cromedome · 2019-12-14T19:20:08Z

You replaced the one place we were using Dancer2::Handler::File, so I think at this point we can effectively retire it.

This is good stuff. 👍 from me.

1. It's core. 2. We're already loading it elsewhere.

This both clears a call to read_glob_content and is faster and more memory efficient.

[First step in moving the core to Path::Tiny] This change helps us test the interaction between the core, FileUtils, and Path::Tiny. This assures we do not deviate.

[This is just a step and is not complete] Dancer2::FileUtils has moved to mostly use Path::Tiny, but this commit starts moving th core code itself to it, in the hopes of rendering most of FileUtils unnecessary. There are some cases that are more complicated, and they are addressed separately one on one.

This logic was trickier. The problem is that we both shift() and also not account for PATH_INFO being empty, which trips Path::Tiny. So, we're cautiously checking whether it is defined has length.

Dancer2::Template::Simple will render files *and* strings. This is why we cannot just use Path::Tiny.

We're using "realpath" because it converts to absolute.

There are two important changes here. The first defends against empty paths from the environment. We can see that there was a situation in which we would have returned an empty string (in case none of the three options worked) and that's bad. We should fix it in the future. The second is what might be a security issue. We use FileUtils' path() which returns path based on root (/) if it's empty. There is a situation in which the environments_location is empty (see paragraph above) and in that case, we will accidentally use the root directory. This does nothing if the file does not exist, but this is now officially fixed with Path::Tiny and checking whether it's empty or not.

The majority of this is rather simple, but annoying to type. We're not making full use of Path::Tiny here, which we could, once we clear it up.

@wchristian

This was a bit hard to figure out, but with the help of @wchristian, it seems as though this merges path from argument 1 onto argument 2. I have this replaced it with a single path() call. All tests pass, so I hope this is okay. We should run these tests under Windows to make sure it works there as well.

We're already using File::Basename. Why are we trying to lazily load it now?

Instead of moving from Path::Tiny back to string as early as possible, we're keeping it for a bit longer. The defined() and -f checks can be done with Path::Tiny too. Then we can just call openr_raw() instead of creating another Path::Tiny object. Once we're done with that, we can finally stringify and go back to string form. Eventually, we would want to make all of our internals assume on Path::Tiny.

If someone were to send a file to send_file() that includes '../', then we would allow them to reach outside the directory we choose. This is a possible security issue. (One can argue the user should sanitize their input, but I think we simply shouldn't allow it.) The problem is that Path::Tiny does the right thing and allows us to reach there. To prevent that, we're resolving paths using Path::Tiny's realpath() method and then subsumes() to see that the file is within the original directory. Otherwise, we send a 403 forbidden. There is also a test that verifies this is done correctly.

SysPete · 2020-04-11T16:31:44Z

@xsawyerx I just rebased this against current master as I'd really like to see it merged. I'm going to go back over all of the comments to check that all issues are addressed, and then will add a further comment.

SysPete · 2020-04-11T17:54:10Z

Some local test failures, but Travis is busy with maintenance atm. Lots of interesting appveyor failures too. I'll start looking tomorrow.

SysPete · 2020-04-12T14:39:19Z

lib/Dancer2/Core/Role/HasLocation.pm

+    $location->is_dir
+        or Carp::croak("Caller $script is not an existing file");


@xsawyerx this is causing the only current (non-Windows) test failure, and was the only failure before I rebased. I know it is a long time ago, but can you remember why this was added?

Hmm..... no memory at all.

I want to get back to this, but I can't promise when. Maybe Saturday?

@xsawyerx Saturday would be cool, but no sweat. With luck I'll have time to look at the Windows test failures this weekend.

SysPete · 2020-05-30T14:47:16Z

Removing from milestone since this still needs a huge amount of work.

"The journey of a thousand miles begins with a single step." We have discussed migrating to Path::Tiny (and started, see #1264 and #1544 for details), and this is the first concrete, discrete step in doing so. It takes a problem of small scope (app scaffolding and File::Find) to set the stage for the rest of the migration. There are a few TODO items here, and if there are better ways to handle them, I am all ears. If not, I will remove my comments when I merge. Feedback welcome/needed!

xsawyerx added the Enhancement label Oct 6, 2016

xsawyerx changed the title ~~Move to Path::Tin~~ Move to Path::Tiny Oct 6, 2016

xsawyerx force-pushed the cleanup/path-tiny branch from 3fb341a to ce201ca Compare February 20, 2017 17:53

xsawyerx force-pushed the cleanup/path-tiny branch from ce201ca to 9b29aca Compare February 28, 2017 20:32

xsawyerx force-pushed the cleanup/path-tiny branch from 27be625 to 3f293c6 Compare September 8, 2019 08:44

xsawyerx added 16 commits April 11, 2020 18:02

Normalize loading of Dancer2::FileUtils

73de648

Use UTF-8 instead of utf-8 encoding

d3a0d04

Suggest using Path::Tiny in documentation

e4ac234

Don't lazily load File::Copy:

39a926f

1. It's core. 2. We're already loading it elsewhere.

Replace set_file_mode with calling binmode directly

6bf2ad5

Remove unused modules loaded in test

9e6edd5

Explicitly use Carp

cd66734

Move to optimized slurping:

655c87e

This both clears a call to read_glob_content and is faster and more memory efficient.

Introduce Path::Tiny in FileUtils:

ac8e0b6

[First step in moving the core to Path::Tiny] This change helps us test the interaction between the core, FileUtils, and Path::Tiny. This assures we do not deviate.

Move ConditionalGET to Path::Tiny:

21d4025

This logic was trickier. The problem is that we both shift() and also not account for PATH_INFO being empty, which trips Path::Tiny. So, we're cautiously checking whether it is defined has length.

Move Dancer2::Template::Simple to Path::Tiny:

6a2ca49

Dancer2::Template::Simple will render files *and* strings. This is why we cannot just use Path::Tiny.

Move Dancer2::Corel::Role::HasLocation to Path::Tiny:

060f150

We're using "realpath" because it converts to absolute.

Move Dancer2::CLI::Command::gen to Path::Tiny:

46053f1

The majority of this is rather simple, but annoying to type. We're not making full use of Path::Tiny here, which we could, once we clear it up.

xsawyerx added 6 commits April 11, 2020 18:28

Remove unused dependency

55b30b9

Remove runtime module already loaded at compile time:

a43810f

We're already using File::Basename. Why are we trying to lazily load it now?

Explicitly use depdency

ef8e789

Remove unused dependency

c0fe438

SysPete force-pushed the cleanup/path-tiny branch from 77ef8a0 to c0fe438 Compare April 11, 2020 16:29

SysPete reviewed Apr 12, 2020

View reviewed changes

cromedome added this to the April 2020 Release milestone Apr 13, 2020

SysPete removed this from the April 2020 Release milestone May 30, 2020

cromedome mentioned this pull request Mar 23, 2021

Replace File::Find with Path::Tiny #1610

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move to Path::Tiny #1264

Move to Path::Tiny #1264

xsawyerx commented Oct 6, 2016 •

edited

Loading

xsawyerx commented Feb 28, 2017

xsawyerx commented Mar 1, 2017

wchristian commented Mar 1, 2017 •

edited

Loading

xsawyerx commented Mar 2, 2017

wchristian commented Mar 2, 2017

ambs commented Oct 8, 2017

xsawyerx commented Sep 8, 2019

xsawyerx commented Sep 8, 2019

cromedome commented Dec 14, 2019

SysPete commented Apr 11, 2020

SysPete commented Apr 11, 2020

SysPete Apr 12, 2020

xsawyerx Apr 15, 2020

SysPete Apr 15, 2020

SysPete commented May 30, 2020

		$location->is_dir
		or Carp::croak("Caller $script is not an existing file");

Move to Path::Tiny #1264

Are you sure you want to change the base?

Move to Path::Tiny #1264

Conversation

xsawyerx commented Oct 6, 2016 • edited Loading

xsawyerx commented Feb 28, 2017

xsawyerx commented Mar 1, 2017

wchristian commented Mar 1, 2017 • edited Loading

xsawyerx commented Mar 2, 2017

wchristian commented Mar 2, 2017

ambs commented Oct 8, 2017

xsawyerx commented Sep 8, 2019

xsawyerx commented Sep 8, 2019

cromedome commented Dec 14, 2019

SysPete commented Apr 11, 2020

SysPete commented Apr 11, 2020

SysPete Apr 12, 2020

Choose a reason for hiding this comment

xsawyerx Apr 15, 2020

Choose a reason for hiding this comment

SysPete Apr 15, 2020

Choose a reason for hiding this comment

SysPete commented May 30, 2020

xsawyerx commented Oct 6, 2016 •

edited

Loading

wchristian commented Mar 1, 2017 •

edited

Loading