dbTalk Databases Forums  

incremental save question

comp.databases.pick comp.databases.pick


Discuss incremental save question in the comp.databases.pick forum.



Reply
 
Thread Tools Display Modes
  #11  
Old   
Tony Gravagno
 
Posts: n/a

Default Re: incremental save question - 07-20-2006 , 05:51 PM






"bob" wrote:

Quote:
Yeah sorry, I did see the suggestion about using triggers.
I am using them (in the form of callx) as a logging facility to trace the
changes users make to key files.
Forgive my lack of knowlege regarding OSFI, is that something I would use in
a Linux system? I know about the VME and FSI in NT but haven't dealt with it
at all in Linux.
FSI is often confused with OSFI just because of the common letters,
but they are different things.

FSI = File System Interface which is D3 files stored as native OS
files rather than in the blob. Most MV environments support this
concept these days. D3 supports the Virtual Machine Environment
(blob) plus the FSI in D3NT but only the VME in *nix.

OSFI = Open Systems File Interface allows D3 to exchange data with the
host OS and with other D3 environments. All D3 platforms support
OSFI.


Quote:
But getting back to callx, I wanted something a little more "hands-off" if
that makes sense.
I'd rather take an approach where I wouldn't impact the working system and
possibly introduce a catalog issue or some other sort of bug that might
impact the client's daytime activity. Not trying to be lazy but this is just
one of those things instinct tells me to keep as simple as possible. I
figure if D3 is tracking updates using dirty bits anyhow, then ideally I
could take advantage of that and pull that information out whether that's
through looking at an incremental save, tx-log or even if I can write code
to check those dirty bits and do the synch. It's something that can be done
after hours and nothing new is introduced into the business code.
My suggestion was to use a trigger to do something like this:
execute "sselect triggers by timestamp"
execute "copy triggers" : @am : "(remote:acct,file,"
Then in the remote system you run a process to integrate the new data
into the existing dataset.

Only the triggers file is located on the production system and updates
are not done "live" (synchronously), they are done whenever you want,
like after hours (asynchronously). I don't care if you use triggers
on your files or if you have the application write updates to a
tickler/trigger file. The benefit of using triggers (which I rarely
do BTW) is that you can "set that dirty bit" no matter what process
does the update.


Quote:
I was looking for some documentation that would explain to me the tape
format since I can do readt's on the incremental pseudo floppy file but
haven't had any luck finding it yet.
Maybe you're not understanding the incremental save process. Here is
a more complete suggestion which can be used instead of the
trigger/OSFI concept. Since this is the sort of information I provide
to my clients for a living I'd prefer to not get into too much detail:

1) Do an account-save on system1.
2) Do an account-restore on system2.
3) On system2 in a different account, write a process that will
extract what you want from the restored account. Do not make changes
to the account in system2.
4) On system1 do an incremental save whenever you wish.
5) On system2 restore that incremental so that the account in system2
is now updated to the same state as system1.
6) Run your program(s) to extract updates. You need to have some
process that can figure out what updates have been made, handle this
any way you want, with a tickler file, checking timestamps, whatever.

Any subsequent incremental save on system1 might only contain a few
items. The data volume is trivial after the initial restore. If the
accounts get out of sync then just go back to step 1, get a new save,
delete the account in system2 and do the restore...

You can do this process in an automated fashion with pseudo floppies.
You can use named pipes to restore from a tape at the same time it's
being saved - this eliminates any disk consumption in the host OS.


Quote:
Also maybe there is a simpler way to work with triggers that I'm not aware
of. Something I could put in system wide might work.
Regarding the transaction log idea, tlog-restore, I was hoping that was
something that accessed those dirty bits but seems that's something also
that requires me to start and stop transaction logging and set up files that
I want to include. Is that true?
Funny but I haven't touched txlog in so long I've forgotten the
details but I believe your understanding is correct. You can use
transaction log directly from one system to another and completely
bypass any manual processes. Barring any communications issues, the
account in system2 will always be in sync with system1, the system
will track all dirty bits, no triggers required. But you still need
something on the receiving end to let you know when something of
significance has changed. You could overlay the ED, UP, or other
commands so that it checks to see if your developer copy and your
production copy are different, and if they are, do a substitution and
present you with the latest version.

Anything is possible here, lots of ways to skin this poor kitty.


Quote:
tlog-restore was asking me for a tape device but I'm not clear on how I
would create that tape.
This is an administration issue for which it's good to have a VAR. In
short there are two ways:
First, create a file in the host OS.
Now:
1) chg-device an existing device to point to that file.
or 2) create a tape entry in pick0 which is always available.

Quote:
Again thanks for all your time and suggestions on this and also to the
others who have made suggestions. I really appreciate it.
No prob really. It seems like you're asking good questions that
should be handled by a "Value-Add" Reseller.

Good luck.

T


Quote:
"Tony Gravagno" wrote
"bob" wrote:
It's not so much that I'm worried about developers clobbering each other.
As
far as code, I have a system in place to track changes and do version
control. This is more a matter of wanting to keep a fresh copy of live
data
on the development machine without having to download such a large psuedo
floppy file every night.

If you can ensure that people aren't going to be changing data in the
development environment then you can easily do an incremental save on
the production system and restore it to the development system.

I think an obvious problem is that people like to change data in a
development system, so how are you going to reconcile live data with
test data? If you can guarantee that the test data will remain in its
restored state then incrementals will work just fine. If not then the
target data will be out of sync with the updates - there's not a lot
that can be done here.

Also, I did suggest triggers and OSFI. I think that's really your
best bet because you can write your own code to sync the environments
rather than leaving it to a simple save/restore tool.

T



Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.