Some performance related changes #14

khalidsalomao · 2013-09-11T17:08:51Z

I tried roll out some of the reasonable changes that I have already made to the FileHelpers build I use for my projects.

Performance related:

Caching of Stringbuilder on RecordOperations.cs for "RecordToString"
and "RecordValuesToString" methods to lessen a little the burden on the
GC (I actually didn't measure anything...)
Change to use "CompareOptions.OrdinalIgnoreCase" and
"CompareOptions.Ordinal" whenever possible (prety good performance gain
whenever we have some heavy string working to do...)
Change to use "ToLowerInvariant" instead of "ToLower" for a small
performance gain (10%)

Access Visibility

some access visibility level changes to useful Field properties on
"FieldBase", "FixedLengthField" and "DelimitedField" (similar to the
previous change on atributes!)

Other changes

FileHelperAsyncEngine.EOF, it is userful if using this engine using
ReadNextRecord () or ReadNexts () (usage in a while loop for example)
FileHelperAsyncEngine.Close, add a dispose to the TextWriter just as
a precaution...

Performance related: 1. Caching of Stringbuilder on RecordOperations.cs for "RecordToString" and "RecordValuesToString" methods to lessen a little the burden on the GC (I actually didn't measure anything...) 2. Change to use "CompareOptions.OrdinalIgnoreCase" and "CompareOptions.Ordinal" whenever possible (prety good performance gain whenever we have some heavy string working to do...) 3. Change to use "ToLowerInvariant" instead of "ToLower" for a small performance gain (10%) Access Visibility 1. some access visibility level changes to useful Field properties on "FieldBase", "FixedLengthField" and "DelimitedField" (similar to the previous change on atributes!) Other changes 1. FileHelperAsyncEngine.EOF, it is userful if using this engine using ReadNextRecord () or ReadNexts () (usage in a while loop for example) 2. FileHelperAsyncEngine.Close, add a dispose to the TextWriter just as a precaution...

MarcosMeli · 2013-09-11T18:07:22Z

FileHelpers/Converters/ConvertHelpers.cs

@@ -874,7 +874,7 @@ public BooleanConverter(string trueStr, string falseStr)
            public override object StringToField(string from)
            {
                object val;
-                string testTo = from.ToLower();
+                string testTo = from.ToLowerInvariant ();


I remember to get some problems with Invariant, maybe that don't convert Á É etc to lower, maybe we must use EqualsIgnoreCase instead of use ToLower

MarcosMeli · 2013-09-11T18:12:25Z

Excellent contribution, from a fast review more than of the 80% can be applied, the Invariant methods have some trubble so better to merge them in another wave, let me try to margue only some changes
Another little thing is some public methods that need more discussion, because would be other ways to do that, and may require more docs

Thanks !!!!

khalidsalomao · 2013-09-11T18:22:52Z

Hi!

Glad to hear that you accepted some of the suggestions... I should have
splitted them to keep it easier to merge!
Sorry about that!

About the invariant, I didn't encounter this kind of problem... Is it a
.Net 2.0 issue? it can be due to the fact that I usually don't use .net
2.0...
I put a small test on LinqPad (.net 4) and it worked as expected...

Anyway, the places where the ToLowerInvariant is used should not have this
kind of issue! It is only used in the "BooleanConverter" and the
"ClassBuilder". :)

About the public methods, yes it is more sensitive! ;)

I suggested it, because I kind of used the fields information generated by
this FileHelpers in some ETL applications and it helped a lot to
auto-discover some layout properties...

Regards,

On Wed, Sep 11, 2013 at 3:12 PM, Marcos Meli [email protected]:

Excellent contribution, from a fast review more than of the 80% can be
applied, the Invariant methods have some trubble so better to merge them in
another wave, let me try to margue only some changes
Another little thing is some public methods that need more discussion,
because would be other ways to do that, and may require more docs

Thanks !!!!

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24262990
.

MarcosMeli · 2013-09-11T18:28:42Z

Great !!

Is all fine to send it all, but I need to review later at home. But small batches are better for automerge them :)

In fact the ClassBuilder must be reworked to avoid compilation, it generate a ton of issues in web environment and requires full trust :( We must think in alternatives
I like the idea of public methods but via a proxy type or something like, not the actual internal class
I half do some work to allow for example

engine.Options.Fields.Last().Optional = true;

or some declarative changes for example

engine.Options.Fields[i].Width = 20;

etc

We must create some scenarios of run time options and go in that directions. I will like to discuss that !!

Thanks

khalidsalomao · 2013-09-11T18:45:10Z

Maybe to avoid using ClassBuilder in partial trust environment, we could:

Simple approach

Allow the manual creation of Fields, like:
engine.Options.AddField ( ... )
Read data without the object generation by using "engine.LastRecordValues"
(which is userful in a lot of scenarios!)

More complex approach (involves upgrading to .Net 3.5 or maintaining
versions to different frameworks...)

.Net 3.5 allows dynamic code generation in partial trust!
(.Net 4) dynamics...

On Wed, Sep 11, 2013 at 3:28 PM, Marcos Meli [email protected]:

Great !!

Is all fine to send it all, but I need to review later at home. But small
batches are better for automerge them :)

In fact the ClassBuilder must be reworked to avoid compilation, it
generate a ton of issues in web environment and requires full trust :( We
must think in alternatives

I like the idea of public methods but via a proxy type or something
like, not the actual internal class
I half do some work to allow for example

engine.Options.Fields.Last().Optional = true;

or some declarative changes for example

engine.Options.Fields[i].Width = 20;

etc

We must create some scenarios of run time options and go in that
directions. I will like to discuss that !!

Thanks

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24264279
.

khalidsalomao · 2013-09-11T18:51:40Z

Actually the run time creation of a layout without using the classbuilder
could be really interesting... like:
engine.Options.AddField (...)

We just have to suppress the object generation and use the "engine.
LastRecordValues" in this case. It could be an option that increases the
FileHelpers flexibility...

About the public access of the Field info: I didn't see the Fields internal
properties as a kind of sensitive information, since it provides a lot of
userful information about the layout without having to do reflection
outside of FielHelpers... Just an idea!

Regards

On Wed, Sep 11, 2013 at 3:45 PM, Khalid Salomão [email protected]:

Maybe to avoid using ClassBuilder in partial trust environment, we could:

Simple approach

Allow the manual creation of Fields, like:
engine.Options.AddField ( ... )

Read data without the object generation by using "engine.
LastRecordValues" (which is userful in a lot of scenarios!)

More complex approach (involves upgrading to .Net 3.5 or maintaining
versions to different frameworks...)

.Net 3.5 allows dynamic code generation in partial trust!

(.Net 4) dynamics...

On Wed, Sep 11, 2013 at 3:28 PM, Marcos Meli [email protected]:

Great !!

Is all fine to send it all, but I need to review later at home. But small
batches are better for automerge them :)

In fact the ClassBuilder must be reworked to avoid compilation, it
generate a ton of issues in web environment and requires full trust :( We
must think in alternatives

I like the idea of public methods but via a proxy type or something
like, not the actual internal class
I half do some work to allow for example

engine.Options.Fields.Last().Optional = true;

or some declarative changes for example

engine.Options.Fields[i].Width = 20;

etc

We must create some scenarios of run time options and go in that
directions. I will like to discuss that !!

Thanks

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24264279
.

MarcosMeli · 2013-09-11T19:07:37Z

Great feedback

About dinamic generation:

I like most this option, but to do that, the filehelpers core need some minor modifications that can break some code, but we can manage it to work
Yes but only of properties not fields, so we need to change filehelpers core too.
Yes dynamics can solve some problems but we get some others too because the fields names must be set, etc

I think that we must to try to reinvent FileHelpers, make it all backward compatible that we can, decoupling some things.

Structure Definition:
Must be type less, it must be a generic structure that instruct parsers and converters but has nothing to do with types of fields or properties, it must read to a list of objects
The current RecordInfo can create a StructureDefinition, but also a run time code like u are talking about
Something like ClassBuilder but in a typeless and codeless way

Values Assigment:
Now in the parsing core the values are assigned to record fields, but creating a ValueMapper for example or something like, we can use Fields, or can use Properties, or dynamics in .net 4.0 version
But again, the think that mustly stop me with properties is the attributes definition, the mix of properties and fields, is a ton of time to get all checked and error prone, but can be done

The best option to read generic CSV with FileHelpers (the current way simply dont work) would be to iterate through records and through fields in a enumerable way or with indexers (by index, by fields name) etc

foreach(var record in CommonEngine.ReadCsv(..))
{
Console.WriteLine(record[0] );
Console.WriteLine(record["FieldName"] );
}

What do u think ?

khalidsalomao · 2013-09-11T20:41:23Z

Yes I also like the option of run time creation of the file layout by
adding Fields and accessing its properties without the need to create an
actual class.
I think that having both options would be great for FileHelpers! So you can
use the method you prefer or feels better working with... :-)
About the usage of properties instead of fields is a breaking change,
but I think it is something to consider. Because it is a kind of default
for todays libs that uses reflection and data binding...
It make easier on WPF databing to Grids and similar stuff that will only
automatically works if your class has properties!
todays libs that use reflection to populate object like Dapper,
FastMember, etc... uses properties
About dynamics: I think the two above items have greater priority. This
could be a feature for a future release...

I like this kind of usage:
foreach(var record in CommonEngine.ReadCsv(..))
{
Console.WriteLine(record[0] );
Console.WriteLine(record["FieldName"] );
}

I actually like the 3.0 way of accessing the "engine.LastRecordValues" to
get the values...
If someone wants to avoid creating lots of object, another usage coud be
setting a flag to disble object creation and:

engine.ReadNext ();
while (!engine.Eof)
{
Console.WriteLine(engine[0] );
Console.WriteLine(engine["FieldName"] );
engine.ReadNext ();
}

If you are open to some degree of redesign in the 3.0 release, I think it
is great! After all it is a Major release and people should expect some
level of change, while for the older projects the 2.0 version continues to
work fine!

I would suggest only a small redesign and simplification to allow a fast
release of the 3.0 version. And later a 4.0 version could present some
other changes...

What do you think of doing a 3.0 release that is somewhat compatible with
the 2.0 and than releasing a 4.0 with a major redesign and breaking changes?

So I would suggest for the 3.0:

. The bunch of small changes and fixes that is already in the todays master
version
. Add some more adjustments to wrap up the current master version
. Replace the old Excel engine with the NPOI engine
. We could remove the BeginReadFile and make the engine into a disposable
that open the file to be read automatically in the first read request...
. What else do you feel, must go into the 3.0?

Obs.: I tried to access the link
http://filehelpers.myjetbrains.com/youtrack/rest/agile/FH/sprint and all I
get is a blank page... and in the login page I get the message "The
administration side of YouTrack is currently under maintenance. Please
contact your administrator for details."

For the 4.0, I would consider:

Merging several of the engines into one:
FixedFileEngine
FileHelperAsyncEngine
FileHelperEngine
DelimitedFileEngine
CsvEngine

I mainly use the FileHelperAsyncEngine, since I am usually dealing with
large files that doesn't fit in memory... And I feel that we could work out
a simpler model for merging these engines

The others engines are doing some transformation process like: merge,
sort, mapping, deduplication ...
It may be another category... like rename it to operations... I think that
making this conceptual separation could help to simplify the understanding
of the FileHelpers, no?
Some brainstorm: Maybe make it into something that can be plugged into a
FileHelperEngine instance.
upgrade the framework version to 4.0

On Wed, Sep 11, 2013 at 4:07 PM, Marcos Meli [email protected]:

Great feedback

About dinamic generation:

I like most this option, but to do that, the filehelpers core need some
minor modifications that can break some code, but we can manage it to work

Yes but only of properties not fields, so we need to change filehelpers
core too.

Yes dynamics can solve some problems but we get some others too because
the fields names must be set, etc

I think that we must to try to reinvent FileHelpers, make it all backward
compatible that we can, decoupling some things.

Structure Definition:
Must be type less, it must be a generic structure that instruct parsers
and converters but has nothing to do with types of fields or properties, it
must read to a list of objects
The current RecordInfo can create a StructureDefinition, but also a run
time code like u are talking about
Something like ClassBuilder but in a typeless and codeless way

Values Assigment:
Now in the parsing core the values are assigned to record fields, but
creating a ValueMapper for example or something like, we can use Fields, or
can use Properties, or dynamics in .net 4.0 version
But again, the think that mustly stop me with properties is the attributes
definition, the mix of properties and fields, is a ton of time to get all
checked and error prone, but can be done

The best option to read generic CSV with FileHelpers (the current way
simply dont work) would be to iterate through records and through fields in
a enumerable way or with indexers (by index, by fields name) etc

foreach(var record in CommonEngine.ReadCsv(..))
{
Console.WriteLine(record[0] );
Console.WriteLine(record["FieldName"] );
}

What do u think ?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24267413
.

khalidsalomao · 2013-09-11T21:06:57Z

Structure Definition

The FileHelpers already uses internally the FieldBase to store the field
information. It could also be used in the run time layout definition:
engine.RecodInfo.Add (new Field { ... })

OR
*
*
var def = new RecordDefinition ();
def.SetMode (Modes.Delimited, ";");

def.AddField ("f1").SetQuoted (""",
QuoteMode.OptionalForBoth).SetNullValue ("").SetTrim (TrimMode.Both);

engine.SetRecodInfo (def);

Values Assigment
The values parsing could be something like that:
foreach(var record in CommonEngine.ReadCsv(..))
{
// string
Console.WriteLine(record[0] );
// value convertion
Console.WriteLine(record.Get (0));
// OR
Console.WriteLine(record.GetAs (0));
}

On Wed, Sep 11, 2013 at 5:41 PM, Khalid Salomão [email protected]:

Yes I also like the option of run time creation of the file layout by
adding Fields and accessing its properties without the need to create an
actual class.
I think that having both options would be great for FileHelpers! So you
can use the method you prefer or feels better working with... :-)

About the usage of properties instead of fields is a breaking change,
but I think it is something to consider. Because it is a kind of default
for todays libs that uses reflection and data binding...

It make easier on WPF databing to Grids and similar stuff that will
only automatically works if your class has properties!

todays libs that use reflection to populate object like Dapper,
FastMember, etc... uses properties

About dynamics: I think the two above items have greater priority. This
could be a feature for a future release...

I like this kind of usage:
foreach(var record in CommonEngine.ReadCsv(..))
{
Console.WriteLine(record[0] );
Console.WriteLine(record["FieldName"] );
}

I actually like the 3.0 way of accessing the "engine.LastRecordValues" to
get the values...
If someone wants to avoid creating lots of object, another usage coud be
setting a flag to disble object creation and:

engine.ReadNext ();
while (!engine.Eof)
{
Console.WriteLine(engine[0] );
Console.WriteLine(engine["FieldName"] );
engine.ReadNext ();
}

If you are open to some degree of redesign in the 3.0 release, I think it
is great! After all it is a Major release and people should expect some
level of change, while for the older projects the 2.0 version continues to
work fine!

I would suggest only a small redesign and simplification to allow a fast
release of the 3.0 version. And later a 4.0 version could present some
other changes...

What do you think of doing a 3.0 release that is somewhat compatible with
the 2.0 and than releasing a 4.0 with a major redesign and breaking changes?

So I would suggest for the 3.0:

. The bunch of small changes and fixes that is already in the todays
master version
. Add some more adjustments to wrap up the current master version
. Replace the old Excel engine with the NPOI engine
. We could remove the BeginReadFile and make the engine into a disposable
that open the file to be read automatically in the first read request...
. What else do you feel, must go into the 3.0?

Obs.: I tried to access the link
http://filehelpers.myjetbrains.com/youtrack/rest/agile/FH/sprint and all
I get is a blank page... and in the login page I get the message "The
administration side of YouTrack is currently under maintenance. Please
contact your administrator for details."

For the 4.0, I would consider:

Merging several of the engines into one:
FixedFileEngine
FileHelperAsyncEngine
FileHelperEngine
DelimitedFileEngine
CsvEngine

I mainly use the FileHelperAsyncEngine, since I am usually dealing with
large files that doesn't fit in memory... And I feel that we could work out
a simpler model for merging these engines

The others engines are doing some transformation process like: merge,
sort, mapping, deduplication ...
It may be another category... like rename it to operations... I think that
making this conceptual separation could help to simplify the understanding
of the FileHelpers, no?
Some brainstorm: Maybe make it into something that can be plugged into a
FileHelperEngine instance.

upgrade the framework version to 4.0

On Wed, Sep 11, 2013 at 4:07 PM, Marcos Meli [email protected]:

Great feedback

About dinamic generation:

I like most this option, but to do that, the filehelpers core need
some minor modifications that can break some code, but we can manage it to
work

Yes but only of properties not fields, so we need to change
filehelpers core too.

Yes dynamics can solve some problems but we get some others too
because the fields names must be set, etc

I think that we must to try to reinvent FileHelpers, make it all backward
compatible that we can, decoupling some things.

Structure Definition:
Must be type less, it must be a generic structure that instruct parsers
and converters but has nothing to do with types of fields or properties, it
must read to a list of objects
The current RecordInfo can create a StructureDefinition, but also a run
time code like u are talking about
Something like ClassBuilder but in a typeless and codeless way

Values Assigment:
Now in the parsing core the values are assigned to record fields, but
creating a ValueMapper for example or something like, we can use Fields, or
can use Properties, or dynamics in .net 4.0 version
But again, the think that mustly stop me with properties is the
attributes definition, the mix of properties and fields, is a ton of time
to get all checked and error prone, but can be done

The best option to read generic CSV with FileHelpers (the current way
simply dont work) would be to iterate through records and through fields in
a enumerable way or with indexers (by index, by fields name) etc

foreach(var record in CommonEngine.ReadCsv(..))
{
Console.WriteLine(record[0] );
Console.WriteLine(record["FieldName"] );
}

What do u think ?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24267413
.

MarcosMeli · 2013-09-13T12:08:09Z

I'm back and the issue tracking too :)

http://filehelpers.myjetbrains.com/youtrack/rest/agile/FileHelpers-0/sprint/3.0

I used a wrapper of the records to be used with bulk insert, that must have the structure of a DataReader, and make BulkCopy operations really fast in our ETL Tasks

About Properties vs Fields, I want to support both of them, but define a policy that by default the library uses Fields, but you can specify just something like [DelimitedRecordUsingProperties(Chars.Tab)] or with a parameter or other name, but allowing the attributes to be applied to fields and properties, and only changing the get and set proxy methods all will work smothlly

I like the idea of release a 3.0 with little changes in core, but I'm trying to create a ton of examples of library usage, I get ton of mails about how to use the library in so common escenarios that we need to create a demo showcase. For that I created a examples framework that checks the examples in compilation mode, but show them later in a win forms app, maybe we can create a Silverlight app or something like to avoid exe running

Cheers

khalidsalomao · 2013-09-13T15:32:14Z

Hi Marcos,

I see what you mean! But sometimes an ETL process is from one text file to
another... (like a preprocessing before a databasse load, especially when
dealing with NOSQL and alternative data storages or BigData). That is why I
mentioned the option of avoiding creating object... But just as something
for some later thought!

About Properties vs Fields, I policy seems a fine idea. But I would suggest
that *by default *both properties and fields are used in
"[DelimitedRecord]".
Why? Because I am feel that it is more intuitive to the .net community
(looking into todays serializers and other tools) to have Properties
considered by default.
I suggested to consider both by default, because current users are already
used to work with fields...

I haven't looked into it yet, but I would suggest to keep it simpler than
creating new kinds of attributes. Something like a configuration option:
[DelimitedRecord(";", RecordMode.OnlyProperties)]

About the examples, great!
About the demos, do you plan to merge the old and new version and create a
new one?

I don't know if the silverlight app would add any value to the examples...
because silverlight has a more limited filesystem access...

About the emails, are you thinking in something like linqpad or scriptcs to
execute custom examples?

On Fri, Sep 13, 2013 at 9:08 AM, Marcos Meli [email protected]:

I'm back and the issue tracking too :)

http://filehelpers.myjetbrains.com/youtrack/rest/agile/FileHelpers-0/sprint/3.0

I used a wrapper of the records to be used with bulk insert, that must
have the structure of a DataReader, and make BulkCopy operations really
fast in our ETL Tasks

About Properties vs Fields, I want to support both of them, but define a
policy that by default the library uses Fields, but you can specify just
something like [DelimitedRecordUsingProperties(Chars.Tab)] or with a
parameter or other name, but allowing the attributes to be applied to
fields and properties, and only changing the get and set proxy methods all
will work smothlly

I like the idea of release a 3.0 with little changes in core, but I'm
trying to create a ton of examples of library usage, I get ton of mails
about how to use the library in so common escenarios that we need to create
a demo showcase. For that I created a examples framework that checks the
examples in compilation mode, but show them later in a win forms app, maybe
we can create a Silverlight app or something like to avoid exe running

Cheers

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24390110
.

Add to ErrorManager an recording limit to avoid possible out-of-memory errors in case of layout errors when dealing with large files. After the threshold is reached, successive errors are ignored.

MarcosMeli · 2013-09-13T17:31:59Z

About the examples, I have already created a reciclable framework for open source apps.

Open the FileHelpers solution and go to Examples - Demo folder in the solution and run FileHelpers.Examples

The projects uses a T4 template that go through files and convert compilable examples to html parts to be shown, also allow to run the examples and captures de console output

Let me know what u think :)

khalidsalomao · 2013-09-13T20:24:25Z

I will take a look in the weekend!

On Fri, Sep 13, 2013 at 2:32 PM, Marcos Meli [email protected]:

About the examples, I have already created a reciclable framework for open
source apps.

Open the FileHelpers solution and go to Examples - Demo folder in the
solution and run FileHelpers.Examples

The projects uses a T4 template that go through files and convert
compilable examples to html parts to be shown, also allow to run the
examples and captures de console output

Let me know what u think :)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24410965
.

khalidsalomao · 2013-09-15T02:03:12Z

Very nice!

Somethings I thougth about when reading:

html documentation (like "example_easy_write.html" ) could be rendered
as html:
WebBrowser or GeckoFx
it is creating a directory for every execution:maybe use
"Environment.SpecialFolder.LocalApplicationData"
Have you though about some playground with some basic feature?
. change the input data
. change the class (just to feel what is like to use fileHelpers) (an
editor like ScintillaNET to help with the color on this feature)

Regards!

On Fri, Sep 13, 2013 at 5:24 PM, Khalid Salomão [email protected]:

I will take a look in the weekend!

On Fri, Sep 13, 2013 at 2:32 PM, Marcos Meli [email protected]:

About the examples, I have already created a reciclable framework for
open source apps.

Open the FileHelpers solution and go to Examples - Demo folder in the
solution and run FileHelpers.Examples

The projects uses a T4 template that go through files and convert
compilable examples to html parts to be shown, also allow to run the
examples and captures de console output

Let me know what u think :)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24410965
.

MarcosMeli · 2013-09-17T00:24:36Z

Great !!

1, In fact html documentation was a secundary idea, the first idea was to create compilable examples that can be show in categories, and that the examples can be runnable, currently it uses WebBrowser, I use geckoFx at work but deployment is prohibitive too big, what do you think ?

I agree with that, we need to use that folder
Awesome idea too, and is very easy to runtime compilation of it.

The last and final idea is to provide a library called ExamplesFX (Examples Framework) that will allow any open source developer to write the examples in folder basics and only running a T4 template, it will get an executable with the examples and the html documentation to upload it, more yet, we can give a playground too :) but the problem with playground will be sandbox it.

We need to create some google docs to discuss all these items, because is hard in comments here in github lol :)

Here are the ExampleFx tickets: http://filehelpers.myjetbrains.com/youtrack/rest/agile/Examples%20Generator-1/sprint/1.0?q=%23Unresolved

FileHelpers 3.0 Tickets: http://filehelpers.myjetbrains.com/youtrack/rest/agile/FH-0/sprint/3.0?q=%23Unresolved

jamesmanning · 2013-09-17T18:26:46Z

If there are very common scenarios, would it make sense to have a wizard-like approach that stepped through the first few questions and pointed them in the right direction?

Easier said than done, certainly, but I know some of the 'optional' things (like attributes) might be less obvious/discoverable for some.

Bonus points for an interface that lets them just upload a sample file (csv, tsv, xls, xlsx, whatever) and generates a working console app to parse it into a collection of POCO objects. :)

On Sep 14, 2013, at 10:03 PM, khalidsalomao [email protected] wrote:

Very nice!

Somethings I thougth about when reading:

html documentation (like "example_easy_write.html" ) could be rendered
as html:
WebBrowser or GeckoFx

it is creating a directory for every execution:maybe use
"Environment.SpecialFolder.LocalApplicationData"

Have you though about some playground with some basic feature?
. change the input data
. change the class (just to feel what is like to use fileHelpers) (an
editor like ScintillaNET to help with the color on this feature)

Regards!

On Fri, Sep 13, 2013 at 5:24 PM, Khalid Salomão [email protected]:

I will take a look in the weekend!

On Fri, Sep 13, 2013 at 2:32 PM, Marcos Meli [email protected]:

About the examples, I have already created a reciclable framework for
open source apps.

Open the FileHelpers solution and go to Examples - Demo folder in the
solution and run FileHelpers.Examples

The projects uses a T4 template that go through files and convert
compilable examples to html parts to be shown, also allow to run the
examples and captures de console output

Let me know what u think :)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24410965
.

—
Reply to this email directly or view it on GitHub.

khalidsalomao · 2013-09-18T15:23:29Z

The current wizards is excelent as a tutorial! Maybe integrate it with the
demos?

I will try to continue this discussion on the GoogleDocs, maybe split into
several bag of ideas!

About the ExamplesFX , nice! I also though about cs-script for playground,
which has appdomain sandboxing...

Regards!

On Tue, Sep 17, 2013 at 3:26 PM, James Manning [email protected]:

If there are very common scenarios, would it make sense to have a
wizard-like approach that stepped through the first few questions and
pointed them in the right direction?

Easier said than done, certainly, but I know some of the 'optional' things
(like attributes) might be less obvious/discoverable for some.

Bonus points for an interface that lets them just upload a sample file
(csv, tsv, xls, xlsx, whatever) and generates a working console app to
parse it into a collection of POCO objects. :)

On Sep 14, 2013, at 10:03 PM, khalidsalomao [email protected]
wrote:

Very nice!

Somethings I thougth about when reading:

html documentation (like "example_easy_write.html" ) could be
rendered
as html:
WebBrowser or GeckoFx

it is creating a directory for every execution:maybe use
"Environment.SpecialFolder.LocalApplicationData"

Have you though about some playground with some basic feature?
. change the input data
. change the class (just to feel what is like to use fileHelpers) (an
editor like ScintillaNET to help with the color on this feature)

Regards!

On Fri, Sep 13, 2013 at 5:24 PM, Khalid Salomão [email protected]:

I will take a look in the weekend!

On Fri, Sep 13, 2013 at 2:32 PM, Marcos Meli [email protected]:

About the examples, I have already created a reciclable framework for
open source apps.

Open the FileHelpers solution and go to Examples - Demo folder in the
solution and run FileHelpers.Examples

The projects uses a T4 template that go through files and convert
compilable examples to html parts to be shown, also allow to run the
examples and captures de console output

Let me know what u think :)

—
Reply to this email directly or view it on GitHub<
https://github.com/MarcosMeli/FileHelpers/pull/14#issuecomment-24410965>
.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24611164
.

MarcosMeli · 2013-09-18T15:40:02Z

Great !! Lets create and share the docs these days :)

About the wizar, I like it, but we need to go to a FileHelpers IDE or something like, that allow to change properties in advace level, maybe like james point, the new project wizard will be the current one or something like, but later u can use advance things or press F5 to test the class :) Open varios projects, is a crazy idea, but will be awesome

There is already a sandboxes wow !! I really like that (and run it in the IDE)

Cheers and Thanks !!

khalidsalomao · 2013-09-18T15:42:01Z

Great! FileHelpers IDE!!!

On Wed, Sep 18, 2013 at 12:40 PM, Marcos Meli [email protected]:

Great !! Lets create and share the docs these days :)

About the wizar, I like it, but we need to go to a FileHelpers IDE or
something like, that allow to change properties in advace level, maybe like
james point, the new project wizard will be the current one or something
like, but later u can use advance things or press F5 to test the class :)
Open varios projects, is a crazy idea, but will be awesome

There is already a sandboxes wow !! I really like that (and run it in the
IDE)

Cheers

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/14#issuecomment-24674651
.

MarcosMeli · 2013-09-18T15:43:38Z

Is just and idea, hard to implement, but we can do a simple app with a toolbar, that opens some fhw files, or something like, and usefull for running some text processing tasks too
Can be an addin for VS too :P

MarcosMeli · 2015-07-26T01:29:45Z

I just merged the changes with the current master and push the resolved conflicts to the branch

https://github.com/MarcosMeli/FileHelpers/tree/khalidsalomao-performance

Lets review and merge them back to the master 👍

MarcosMeli reviewed Sep 11, 2013
View reviewed changes

Add to ErrorManager an recording limit

00483b5

Add to ErrorManager an recording limit to avoid possible out-of-memory errors in case of layout errors when dealing with large files. After the threshold is reached, successive errors are ignored.

MarcosMeli added enhancement performance labels Mar 26, 2015

MarcosMeli added the ready label Jul 26, 2015

MarcosMeli added this to the 4.0 milestone Jul 28, 2015

mcavigelli removed the ready label Feb 27, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some performance related changes #14

Some performance related changes #14

khalidsalomao commented Sep 11, 2013

MarcosMeli Sep 11, 2013

MarcosMeli commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

MarcosMeli commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

MarcosMeli commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

MarcosMeli commented Sep 13, 2013

khalidsalomao commented Sep 13, 2013

MarcosMeli commented Sep 13, 2013

khalidsalomao commented Sep 13, 2013

khalidsalomao commented Sep 15, 2013

MarcosMeli commented Sep 17, 2013

jamesmanning commented Sep 17, 2013

khalidsalomao commented Sep 18, 2013

MarcosMeli commented Sep 18, 2013

khalidsalomao commented Sep 18, 2013

MarcosMeli commented Sep 18, 2013

MarcosMeli commented Jul 26, 2015

Some performance related changes #14

Are you sure you want to change the base?

Some performance related changes #14

Conversation

khalidsalomao commented Sep 11, 2013

MarcosMeli Sep 11, 2013

Choose a reason for hiding this comment

MarcosMeli commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

MarcosMeli commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

MarcosMeli commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

khalidsalomao commented Sep 11, 2013

MarcosMeli commented Sep 13, 2013

khalidsalomao commented Sep 13, 2013

MarcosMeli commented Sep 13, 2013

khalidsalomao commented Sep 13, 2013

khalidsalomao commented Sep 15, 2013

MarcosMeli commented Sep 17, 2013

jamesmanning commented Sep 17, 2013

khalidsalomao commented Sep 18, 2013

MarcosMeli commented Sep 18, 2013

khalidsalomao commented Sep 18, 2013

MarcosMeli commented Sep 18, 2013

MarcosMeli commented Jul 26, 2015