dbTalk Databases Forums  

How to count records in a file and split a file into multiple files

microsoft.public.sqlserver.dts microsoft.public.sqlserver.dts


Discuss How to count records in a file and split a file into multiple files in the microsoft.public.sqlserver.dts forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
karenmiddleol@yahoo.com
 
Posts: n/a

Default How to count records in a file and split a file into multiple files - 10-28-2005 , 06:53 AM






I have a large file to be processed. I would like to split the files
into say 80000 records in each file to manage the size of processing.

Please kindly share the code or procedure to split a file into
manageable sizes and handling the resulting split files from DTS.

Thanks
Karen


Reply With Quote
  #2  
Old   
ekkehard.horner
 
Posts: n/a

Default Re: How to count records in a file and split a file into multiplefiles - 10-28-2005 , 09:45 AM






karenmiddleol (AT) yahoo (DOT) com wrote:

Quote:
I have a large file to be processed. I would like to split the files
into say 80000 records in each file to manage the size of processing.

Processing large files (fast) was discussed here

http://www.visualbasicscript.com/m_23627/tm.htm

(but not in the context of DTS - sorry)
Quote:
Please kindly share the code or procedure to split a file into
manageable sizes and handling the resulting split files from DTS.

Thanks
Karen


Reply With Quote
  #3  
Old   
shiv_koirala@yahoo.com
 
Posts: n/a

Default Re: How to count records in a file and split a file into multiple files - 10-29-2005 , 12:41 AM



Because in the initial read you will have no clue about the records
best is to use the file size.
-------
Regards ,
C#, VB.NET , SQL SERVER , UML , DESIGN Patterns Interview question book
http://www.geocities.com/dotnetinterviews/
My Interview Blog
http://spaces.msn.com/members/dotnetinterviews/


Reply With Quote
  #4  
Old   
shiv_koirala@yahoo.com
 
Posts: n/a

Default Re: How to count records in a file and split a file into multiple files - 10-29-2005 , 12:42 AM



Because in the initial read you will have no clue about the records
best is to use the file size.ok again you have a logic when you should
know you have reached half and ensure that you have noit read half
record data.
-------
Regards ,
C#, VB.NET , SQL SERVER , UML , DESIGN Patterns Interview question book
http://www.geocities.com/dotnetinterviews/
My Interview Blog
http://spaces.msn.com/members/dotnetinterviews/


Reply With Quote
  #5  
Old   
Allan Mitchell
 
Posts: n/a

Default Re: How to count records in a file and split a file into multiple files - 10-29-2005 , 04:46 AM



Hello karenmiddleol (AT) yahoo (DOT) com,

Is this a large text file coming in to you?

Performing an intermediary step like this will do nothing for performance
of the package. BULK INSERT along with no indexing on the target table and
no constraints usually handles the job just fine. I BULK in files of more
than 17,000,000 rows without a problem. If you want to perform some AX stuff
on the rows as they come through then splitting this up will not help either.
They will still all need to be processed.


Allan



Quote:
I have a large file to be processed. I would like to split the files
into say 80000 records in each file to manage the size of processing.

Please kindly share the code or procedure to split a file into
manageable sizes and handling the resulting split files from DTS.

Thanks
Karen



Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.