So today we had the mother of all nasty scenarios.
The server crashed and when restarting said "Cannot roll forward from checkpoint, contact IBM support".
Logical logs were in a mess. Not good. So IBM dial in and truncate the logsand sort the system out so it starts - yeah for onlog
They recommend not using the database anymore, but unloading everything andmoving it to a new instance to ensure consistency.

I buy the argument completely and it is the right thing to do.

But... The issue I have is this - we can't really afford the downtime untilthe milk season winds down for winter in a month or two.
So... my question.
What is the consensus about running on a database that has been log truncated?

I want opinion on how urgent the full server rebuild is.
If we have to do it today, then we need to try to find a way to do it with as little downtime as possible because the plant will still be running. This will be VERY complicated and basically the definition of annoying.

On your marks, get set, discuss...

Thanks

Jarrod Teale

[cid:image001.jpg (AT) 01CC0B79 (DOT) 676B10C0]

Process Information Manager
Automation and Process Control

jarrod.teale (AT) fonterra (DOT) com...erra (DOT) com>
direct +64 7 850 7525 (ext 77525), mobile +64 21 968 364, fax +64 7 849 7855
Fonterra Co-operative Group Limited
PO Box 459, Hamilton, 3240, Automation and Process Control, Fonterra Te Rapa, SH1, Hamilton, New Zealand


[cid:image002.png (AT) 01CC0B79 (DOT) 676B10C0]


________________________________
DISCLAIMER
This email contains information that is confidential and which may be legally privileged. If you have received this email in error, please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way.

This is my personal opinion. Not my professional opinion nor my employer
view.
If the logs were "truncated" you should have the database in a perfectly
consistent state, but "in the past".
You will have problems if your system communicate with others, they may have
received information from it that currently it doesn't have. Or that the
other systems assume they have sent information to yours that it currently
does not recognize.
Also some users may find strange things, like having invoices printed that
are not stored on the system etc.

Note that an export/import of the system will not solve any of the above...

Ideally you should at least run complete onchecks... but these also take
time and may cause outage.

What you should also proceed is to find the root cause (if possible). If
it's caused by a bug, you should upgrade ASAP. If the root cause was the
server crash (and this caused corruption) than the recommendation would
certainly be to export/import.

Finally, there is a difference between "truncate the logs" and "tweak the
logs". Technical support can truncate the whole logical log, leaving your
system in a certain point in time, or eliminate some LL records that are
causing problems. What was done may influence the decision to export/import

Regards.



On Thu, May 5, 2011 at 12:08 PM, Jarrod Teale <Jarrod.Teale (AT) fonterra (DOT) com>wrote:


--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...

Oh - my bad - Redhat 5, Informix 11.50.UC5X2

From: informix-list-bounces (AT) iiug (DOT) org [mailto:informix-list-bounces (AT) iiug (DOT) org] On Behalf Of Jarrod Teale
Sent: Thursday, 5 May 2011 11:09 p.m.
To: 'informix-list (AT) iiug (DOT) org'
Subject: Discussion: Is a server with truncated logs safe?

So today we had the mother of all nasty scenarios.
The server crashed and when restarting said "Cannot roll forward from checkpoint, contact IBM support".
Logical logs were in a mess. Not good. So IBM dial in and truncate the logsand sort the system out so it starts - yeah for onlog
They recommend not using the database anymore, but unloading everything andmoving it to a new instance to ensure consistency.

I buy the argument completely and it is the right thing to do.

But... The issue I have is this - we can't really afford the downtime untilthe milk season winds down for winter in a month or two.
So... my question.
What is the consensus about running on a database that has been log truncated?

I want opinion on how urgent the full server rebuild is.
If we have to do it today, then we need to try to find a way to do it with as little downtime as possible because the plant will still be running. This will be VERY complicated and basically the definition of annoying.

On your marks, get set, discuss...

Thanks

Jarrod Teale

[cid:image001.jpg (AT) 01CC0B7C (DOT) 5FE25360]

Process Information Manager
Automation and Process Control

jarrod.teale (AT) fonterra (DOT) com...erra (DOT) com>
direct +64 7 850 7525 (ext 77525), mobile +64 21 968 364, fax +64 7 849 7855
Fonterra Co-operative Group Limited
PO Box 459, Hamilton, 3240, Automation and Process Control, Fonterra Te Rapa, SH1, Hamilton, New Zealand


[cid:image002.png (AT) 01CC0B7C (DOT) 5FE25360]


________________________________
DISCLAIMER
This email contains information that is confidential and which may be legally privileged. If you have received this email in error, please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way.

I've just looked at your email... ER of course is a situation where the
system exchanges information with others...
This may be an issue, or eventually a good way to recover some
information... You will know better the impact of it...

Regards.


On Thu, May 5, 2011 at 12:28 PM, Fernando Nunes <domusonline (AT) gmail (DOT) com>wrote:



--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...

Oh - and to satisfy the curiosities out there.
We crashed the server by trying to add a CPUVP online with the MULTIPROCESSOR onconfig parameter set to 0.
It got REALLY upset...
IBM are following up with the bug trace to stop the rest of you having the same issue.

As soon as the system was up and we had onchecked the main database tables and indexes, we took a level 0 archive and have it in a safe place - so wecan always go back.

Here's the history from the bashrc for what the Informix support team did (cat / vi / cd / tail all removed) for full detail:
315 onstat -m
316 onlog -n 72741 > /tmp/log41 &
317 export IFX_ASKEY=FMQEXPRC
322 onlog -n 72740 > /tmp/log40 &
323 onlog -n 72741 > /tmp/log41 &
326 oncheck -S n -l 72741 -p 0x16f1094
327 dbaccessv
329 oncheck -S n -l 72741 -p 0x16ef11c
332 oncheck -S d
333 oncheck -S d -c 1 -o 14 -n 1
335 oncheck -S n -l 72740 -p 0x270f440
336 oncheck -S d -c 8 -o 10000 -n 1

Like I said, not a good day.

Jarrod

From: informix-list-bounces (AT) iiug (DOT) org [mailto:informix-list-bounces (AT) iiug (DOT) org] On Behalf Of Jarrod Teale
Sent: Thursday, 5 May 2011 11:09 p.m.
To: 'informix-list (AT) iiug (DOT) org'
Subject: Discussion: Is a server with truncated logs safe?

So today we had the mother of all nasty scenarios.
The server crashed and when restarting said "Cannot roll forward from checkpoint, contact IBM support".
Logical logs were in a mess. Not good. So IBM dial in and truncate the logsand sort the system out so it starts - yeah for onlog
They recommend not using the database anymore, but unloading everything andmoving it to a new instance to ensure consistency.

I buy the argument completely and it is the right thing to do.

But... The issue I have is this - we can't really afford the downtime untilthe milk season winds down for winter in a month or two.
So... my question.
What is the consensus about running on a database that has been log truncated?

I want opinion on how urgent the full server rebuild is.
If we have to do it today, then we need to try to find a way to do it with as little downtime as possible because the plant will still be running. This will be VERY complicated and basically the definition of annoying.

On your marks, get set, discuss...

Thanks

Jarrod Teale

[cid:image001.jpg (AT) 01CC0B7C (DOT) FFEE0160]

Process Information Manager
Automation and Process Control

jarrod.teale (AT) fonterra (DOT) com...erra (DOT) com>
direct +64 7 850 7525 (ext 77525), mobile +64 21 968 364, fax +64 7 849 7855
Fonterra Co-operative Group Limited
PO Box 459, Hamilton, 3240, Automation and Process Control, Fonterra Te Rapa, SH1, Hamilton, New Zealand


[cid:image002.png (AT) 01CC0B7C (DOT) FFEE0160]


________________________________
DISCLAIMER
This email contains information that is confidential and which may be legally privileged. If you have received this email in error, please notify the sender immediately and delete the email. This email is intended solely for the use of the intended recipient and you may not use or disclose this email in any way.

Hello Jarod,

-> They recommend not using the database anymore, but unloading
everything and
-> moving it to a new instance to ensure consistency.
-> I buy the argument completely and it is the right thing to do.
-> But... The issue I have is this – we can’t really afford the
downtime until
-> the milk season winds down for winter in a month or two.

If a chunkfree list is a mess eq it does not count space allocated for
a table which has claimed space
you could end up having tables overlap which is one of the more
serious scenarios that could happen.
You will end up with major corruption and maybe with another crash
having TS zero the logs again.
It is time to bite the bullit and unload all recreate the instance and
reload.
Ask yourself the question if you can afford to run the risk. ( i would
not take the risk!!!!!!!)

dbexport may be used for it dependant on the db's size ; import may
fail due to a incorrect row count.-->> check it.
maybe Arts stuff or Jonathans stuff can help you here...

Superboer.

On 05/05/11 12:08, Jarrod Teale wrote:
The answer is 'how long is a piece of string'?
It very much depends on the extent of the corruption. Advanced support has
tools to selectively truncate recovery on specific tables or even records(I
know, because I have written them): in this particular circumstance you would
be able to get away with just fixing the tables involved.
But if the corruption was too widespread for surgical intervention, the amount
of potential corruption after recovery truncation could be enormous.
To top it off, the corruption could be apparent (corrupted indexes, data
pages, blobs) prompting you to rebuild the instance, on the grounds that it's
just unusable, or it might be not (as in, the data might be incorrect from a
business point of view, but correct as far as storage goes), and you'll find
yourself in the uncomfortable position of taking wrong business decisions
based on wrong data.
--
Ciao,
Marco
__________________________________________________ ____________________________
Marco Greco /UK /IBM Standard disclaimers apply!

Structured Query Scripting Language http://www.4glworks.com/sqsl.htm
4glworks http://www.4glworks.com
Informix on Linux http://www.4glworks.com/ifmxlinux.htm