Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A VersionedTap overrides mapreduce.input.fileinputformat.inputdir without taking in account the previous values #50

Open
bqm opened this issue May 21, 2015 · 0 comments

Comments

@bqm
Copy link

bqm commented May 21, 2015

The VersionedTap's sourceConfInit method overrides mapreduce.input.fileinputformat.inputdir in:

https://github.com/nathanmarz/dfs-datastores/blob/master/dfs-datastores-cascading/src/main/java/com/backtype/cascading/tap/VersionedTap.java#L96

As a result, we can't define out of the box a cascading.tap.MultiTap as a list of VersionedTap as each VersionedTap is going to override the previous input paths.

I am not familiar with the cascading code so I am not sure what would be the best solution to this issue. Maybe a VersionedTap could add its own path to the existing list of paths instead of overriding the whole list?
As a temporary solution in my code, I am using a custom MultiTap that redefines the input paths but that doesn't seem optimal.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant