SSIS and ragged rows (variable number of columns) -
03-16-2006
, 12:14 PM
DTS used to work fine with comma delimited files like this
1,2,3,4,5,6,7
1,2,3
1
1,2,3,4,5,6
1,2,3
(my real data has up to 90 columns, and millions of rows...)
but in SSIS the columns have to be filled before the row delimiter is
looked at. The result is a scrambled import where a second row in this
case would cotain the 3th and the 4th row, and the imported 3rd row
would start at the 5th row...
If Microsoft only provided a simple option of whether the row delimiter
should be always checked before the column delimiter just like it
happens in the DTS world... Even though it seems to be logical to
always threat comma delimited files this way...
If anyone has any pointers, to fixes or a way to avoid this problem
using SSIS please let me know. I would like to avoid doing a scan and
adding commas to each line as I'll have gigabites of data per day...
And preferably would like to avoid making my own file connection
manager/data flow source
Thanks |