Hi Martijn,

Can I get PHP to remove the first row without reading the whole file in? If
there was a way where PHP would just chop the first row off that would be
ideal...

Thanks

adam


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Wouldn't it be easier to have PHP remove the first row?

And PHP has to be there because Apache is sending all the data to it.

Hope this helps,

On Fri, Nov 19, 2004 at 03:43:18PM +0000, Adam Witney wrote:
--
Martijn van Oosterhout <kleptog (AT) svana (DOT) org> http://svana.org/kleptog/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFBnhlmY5Twig3Ge+YRAnsWAKCBEN8tsJdalrkYYXlFCf duSYFbmQCgw+35
ID9bZBkVZ7+5pxR8zuRNGlg=
=xW5J
-----END PGP SIGNATURE-----

You could write a server side function to do the processing. You could
even use plPHP if that is your preferred language.

Sincerely,

Joshua D. Drake




--
Command Prompt, Inc., home of PostgreSQL Replication, and plPHP.
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd (AT) commandprompt (DOT) com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

Hi,

If you are using "COPY table FROM stdin" and pg_put_line() it's not much
difficult to skip the first line


Best regards
--
Matteo Beccati
http://phpadsnew.com
http://phppgads.com


---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)

PHP is slow, but not *that* slow, you have a problem somewhere !

If you read the whole file in memory, the server will kick your script (I
think the default limit is 8 megabytes or something)...

So, I'd advise reading the file line by line using fgets() (dunno how it
is spelled in php), and just skip the first line, and pg_put_line() the
rest. This way you just use memory for one line at a time. ALso you can
echo (and flush) messages like 'XX lines inserted...' to the user while it
crunches.

If you're really stuck, and have command execution privileges, why not
system() a command line like "awk -blah your file | psql copy to your
table", or even launch it as a background process ?

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo (AT) postgresql (DOT) org)

Since we can specify the order of columns in copy, how hard would it be
(he asked, naively) to specify a column name that points to /dev/null
(or the postgresql internals equivalent)? Sybase's copy utility is very
similar to our copy, and has a function that you can specify in your
column list instead of a real column name that ignores the data.

Something like

copy mytable (col1, col2, ignore(), col3) from '/tmp/dump';

Not a big deal, but kind of handy.

Ian Harding
Programmer/Analyst II
Tacoma-Pierce County Health Department
iharding (AT) tpchd (DOT) org
Phone: (253) 798-3549
Pager: (253) 754-0002

Nope. When you get into significant massaging of the input data,
usually the best bet is to COPY into a temp table that exactly matches
the format of the data file, and then do your rearrangements using an
INSERT/SELECT into the final target table.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if
your
joining column's datatypes do not match


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

MS SQL Server has a BULK INSERT command that can take a parameter "Start
Row" so that you can skip as many Rows as you want.
this is also available using the command line bcp utility. This would be
a nice feature for Postgresql to have , and I can't see it on any of the
to do lists, unless it is already implemented well enough in third party
utility or contrib. I shelled out a fair bit to get hold of EMS Hi Tech
Postgres Manager bundle, and whilst there is a lot to like about it
generally, at least part of the reason was because of the improved bulk
insert tools.





Adam Witney wrote:

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo (AT) postgresql (DOT) org so that your
message can get through to the mailing list cleanly

On 19/11/04 6:46 pm, "Pierre-Frédéric Caillaud"
<lists (AT) boutiquenumerique (DOT) com> wrote:

Aha yes, I was reading the file doing this

fgets($fh, 1048576)))

.... But changing it to

fgets($fh, 1024)))

Which is a large enough amount of data for my needs improved the speed
dramatically!

Thanks to all those that replied

adam


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org