This may be a matter of experience and cognitive style, just like the
question of which GUI design is more intuitive.

But the fact you stayed away from it for 8~10 years says a lot about its
usibility already. I like to preprocess a file also because it has the added
value of validating the data in the file, which makes it much easier to debug
data import problems.

Pre-processing a 1GB file is really rather trivial these days. Now, if it's
100GB, one may have to think a bit harder. Even there, going through that
file sequentially in one shot shouldn't be a huge issue as long as you don't
try to load the whole thing into memory as notepad likes to do. Considering
the benefit of precisely locate any data format problems during the
pre-process, I'd probably still go with pre-processing it.

Linchi

"Erland Sommarskog" wrote:

This may be a matter of experience and cognitive style, just like the
question of which GUI design is more intuitive.

But the fact you stayed away from it for 8~10 years says a lot about its
usibility already. I like to preprocess a file also because it has the added
value of validating the data in the file, which makes it much easier to debug
data import problems.

Pre-processing a 1GB file is really rather trivial these days. Now, if it's
100GB, one may have to think a bit harder. Even there, going through that
file sequentially in one shot shouldn't be a huge issue as long as you don't
try to load the whole thing into memory as notepad likes to do. Considering
the benefit of precisely locate any data format problems during the
pre-process, I'd probably still go with pre-processing it.

Linchi

"Erland Sommarskog" wrote:

This may be a matter of experience and cognitive style, just like the
question of which GUI design is more intuitive.

But the fact you stayed away from it for 8~10 years says a lot about its
usibility already. I like to preprocess a file also because it has the added
value of validating the data in the file, which makes it much easier to debug
data import problems.

Pre-processing a 1GB file is really rather trivial these days. Now, if it's
100GB, one may have to think a bit harder. Even there, going through that
file sequentially in one shot shouldn't be a huge issue as long as you don't
try to load the whole thing into memory as notepad likes to do. Considering
the benefit of precisely locate any data format problems during the
pre-process, I'd probably still go with pre-processing it.

Linchi

"Erland Sommarskog" wrote:

This may be a matter of experience and cognitive style, just like the
question of which GUI design is more intuitive.

But the fact you stayed away from it for 8~10 years says a lot about its
usibility already. I like to preprocess a file also because it has the added
value of validating the data in the file, which makes it much easier to debug
data import problems.

Pre-processing a 1GB file is really rather trivial these days. Now, if it's
100GB, one may have to think a bit harder. Even there, going through that
file sequentially in one shot shouldn't be a huge issue as long as you don't
try to load the whole thing into memory as notepad likes to do. Considering
the benefit of precisely locate any data format problems during the
pre-process, I'd probably still go with pre-processing it.

Linchi

"Erland Sommarskog" wrote:

This may be a matter of experience and cognitive style, just like the
question of which GUI design is more intuitive.

But the fact you stayed away from it for 8~10 years says a lot about its
usibility already. I like to preprocess a file also because it has the added
value of validating the data in the file, which makes it much easier to debug
data import problems.

Pre-processing a 1GB file is really rather trivial these days. Now, if it's
100GB, one may have to think a bit harder. Even there, going through that
file sequentially in one shot shouldn't be a huge issue as long as you don't
try to load the whole thing into memory as notepad likes to do. Considering
the benefit of precisely locate any data format problems during the
pre-process, I'd probably still go with pre-processing it.

Linchi

"Erland Sommarskog" wrote:

This may be a matter of experience and cognitive style, just like the
question of which GUI design is more intuitive.

But the fact you stayed away from it for 8~10 years says a lot about its
usibility already. I like to preprocess a file also because it has the added
value of validating the data in the file, which makes it much easier to debug
data import problems.

Pre-processing a 1GB file is really rather trivial these days. Now, if it's
100GB, one may have to think a bit harder. Even there, going through that
file sequentially in one shot shouldn't be a huge issue as long as you don't
try to load the whole thing into memory as notepad likes to do. Considering
the benefit of precisely locate any data format problems during the
pre-process, I'd probably still go with pre-processing it.

Linchi

"Erland Sommarskog" wrote:

This may be a matter of experience and cognitive style, just like the
question of which GUI design is more intuitive.

But the fact you stayed away from it for 8~10 years says a lot about its
usibility already. I like to preprocess a file also because it has the added
value of validating the data in the file, which makes it much easier to debug
data import problems.

Pre-processing a 1GB file is really rather trivial these days. Now, if it's
100GB, one may have to think a bit harder. Even there, going through that
file sequentially in one shot shouldn't be a huge issue as long as you don't
try to load the whole thing into memory as notepad likes to do. Considering
the benefit of precisely locate any data format problems during the
pre-process, I'd probably still go with pre-processing it.

Linchi

"Erland Sommarskog" wrote: