Hi,

We have a function that selects and writes out 3000 records at a time
to multiple CSV files. The selection of the 3000 records is done in
one function and then another function is called to write out the
selected records (using WRITESEQ). The process slows dramatically
over time. As you can see the first File of 3000 records takes 6
minutes to write and the last 53 minutes.

It's almost like there is a memory leak. Any ideas as to what might
be going on here?

Thanks
Steve


CSVFile Created Time Duration
1 9:05:00 a.m. 0:06
2 9:11:00 a.m. 0:08
3 9:19:00 a.m. 0:14
4 9:33:00 a.m. 0:23
5 9:56:00 a.m. 0:36
6 10:32:00 a.m. 0:41
7 11:13:00 a.m. 0:48
8 12:01:00 p.m. 0:53
9 12:54:00 p.m. 0:59
10 1:53:00 p.m. 1:08
11 3:01:00 p.m. 1:45

Shot in the dark:

Depending on how much memory is being used by the records being written, it
seems possible that if these records are using different variable names, and
not being reused for the next batch, you could end up memory bound, with
these set of records taking up more and more space for their storage. The
first thing I would try is reuse the variable names when done with each
batch.

Ed

"Big kahuna" <steve.muller (AT) tauranga (DOT) govt.nz> wrote

Steve,

I would think there is something definitely wrong there; you are
saying that under the best of circumstances it takes 6 minutes to
write out 3000 records via writeseq?! Wow- how big are those records??
Unless those CSV records are being written to a USB drive (or a
floppy!) or via the internets to a remote site, I would expect a
process such as that to finish in seconds, not minutes. My guess is
that your UV basic program is doing something the hard way.

/Scott Ballinger
Pareto Corporation
Edmonds WA USA
206 713 6006

On Aug 15, 5:13*pm, Big kahuna <steve.mul... (AT) tauranga (DOT) govt.nz> wrote:

On Aug 15, 5:13*pm, Big kahuna <steve.mul... (AT) tauranga (DOT) govt.nz> wrote:

Let's see the code.

"Scott Ballinger" wrote
Yeah, if you're that hard up, you might try brute force and ignorance;
a) write one record to tom.txt
b) invoke a dos subshell {In D3 this would be like}
execute "!TYPE TOM.TXT >> TOM_N_JERRY.TXT" capturing bucket
c) rinse and repeat
Which in general is a dead slow way to do it, but you're already there,
man...

Hi, the function in question is a generic function for writing data to
a CSV file.

It get's called from a main function that selects:
- the records to be exported from the Universe file
- the dictionaries to include in the export
- the CSV export file name and path

These are passed through to the DSK.EXPORTCSV function as parameters.

The basic process is:

loop through Universe file selecting 3000 records at a time
call DSK.EXPORTCSV to export the current group of 3000 records
repeat

On Aug 17, 5:10*pm, Big kahuna <steve.mul... (AT) tauranga (DOT) govt.nz> wrote:
The exact specific details will be important.
Please post the actual code.

Steve,
By the sound of it something is wrong with the code. There is no code
to browse through, probably too many lines of code to publish. I
don't think that "select 3000 records" is as simple as it sounds nor
the call to DSK.EXPORTCSV is necessary for a mere WRITESEQ. Not
knowing anything about your code I would try to isolate the hog
first. Run the process without the call to DSK.EXPORTCSV and see how
the calls are spaced in time. If the selection time increases as you
go, most likely the select is done over and over from the beginning of
things, reparsing previously processed records. If the calls to
DSK.EXPORTCSV are equally spaced then look how your called routine
builds its data. It may use WRITESEQ to output but it may first read
in memory the existing data, update it, then write it etc. Check also
for possible COMMON variables that may get updated with tons of
information whenever you call your routine or other variables in your
main program that may hold growing lists of records etc.

Lucian

Thanks Lucian,

You are correct, there are too many lines of code to publish. The
DSK.RUNCSV function for example, converts Multivalue characters to
Linefeed characters for tabular output in the Word Merge etc.

I will give your testing approach a go and see if I can isolate the
problem.

Thanks again for your suggestions

Steve

On Aug 19, 1:32*pm, x <lucian_p... (AT) yahoo (DOT) com> wrote: