dbTalk Databases Forums  

Changed DNS Server and now database connection is very slow

comp.databases.informix comp.databases.informix


Discuss Changed DNS Server and now database connection is very slow in the comp.databases.informix forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Jackson Thompson
 
Posts: n/a

Default Changed DNS Server and now database connection is very slow - 03-26-2011 , 12:07 PM






I changed our DNS server and when I disconnect the old DNS server our
informix ids 11.5 instance takes up to 30 seconds to find the
databases. If you use dbaccess it starts and when is is going to list
the databases it stalls, the connect process and all query seem to run
fine. We I have the old DNS server off line I can get good forward and
reverse DNS lookups so the DNS for the os (debian linux) seems to be
fine. I have not restart the Informix instance, it is a production
system of a 24/7 contact center, so I don't want to restart without
knowing that there might be a fix in it.
Thanks
Jackson

Reply With Quote
  #2  
Old   
Fernando Nunes
 
Posts: n/a

Default Re: Changed DNS Server and now database connection is very slow - 03-26-2011 , 12:51 PM






Bad news... There is currently no workaround (except eventually some network
tweaking).

The problem lies in the reverse DNS system call - gethostbyaddr() - The
first time it runs it creates a cache of your DNS configuration.
If you later change it, it has no way of knowing the change happened. So it
keeps trying to talk to your previously configured DNS server.
The delay is the time it takes to give up....
This can be showed with a simple C program. The cache will only be forgotten
if the process calls another system function. Currently Informix doesn't do
that. A mechanism would have to be implemented to allow the DBA to "ask"
Informix to run that function. Note that other solutions like periodically
call it, would be bad for performance.

I've recently created a feature request for this. Note that this problem can
affect any kind of server software (I've seen it with sendmail a few years
ago).

If your configuration of the resolver system included a first try with the
files (/etc/hosts) you could try to workaround it by including all the
client host IP address/names in the file.
This is usually not advisable, but you may consider it a workaround until
you can restart Informix

Another possible option (and I'm getting completely out of my comfort zone
here) could be trying to full the network layer to re-write the packet that
are sent to the old DNS in a way that they're directed to the new one....

If you feel sufficiently annoyed by this you can also open a PMR (which will
not solve your problem) and make sure your need is mapped to the feature I'm
asked for. I could provide details.

Regards.

On Sat, Mar 26, 2011 at 6:07 PM, Jackson Thompson
<jackson (AT) valu-search (DOT) com>wrote:

Quote:
I changed our DNS server and when I disconnect the old DNS server our
informix ids 11.5 instance takes up to 30 seconds to find the
databases. If you use dbaccess it starts and when is is going to list
the databases it stalls, the connect process and all query seem to run
fine. We I have the old DNS server off line I can get good forward and
reverse DNS lookups so the DNS for the os (debian linux) seems to be
fine. I have not restart the Informix instance, it is a production
system of a 24/7 contact center, so I don't want to restart without
knowing that there might be a fix in it.
Thanks
Jackson
_______________________________________________
Informix-list mailing list
Informix-list (AT) iiug (DOT) org
http://www.iiug.org/mailman/listinfo/informix-list



--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...

Reply With Quote
  #3  
Old   
Ian Michael Gumby
 
Posts: n/a

Default RE: Changed DNS Server and now database connection is very slow - 03-26-2011 , 10:38 PM



Doesn't Informix age the cache?

That is to say that it will periodically flush the cache and re-fetch the address and insert it in to the cache?
You could then tune the TTL and don't have to worry about it.
(Yes, I agree that a DBA command would also be nice, but if you allow for a TTL setting, then periodically the cache renews itself. Then worst case, you take a performance hit until the cache is reset.)


Date: Sat, 26 Mar 2011 18:51:18 +0000
Subject: Re: Changed DNS Server and now database connection is very slow
From: domusonline (AT) gmail (DOT) com
To: jackson (AT) valu-search (DOT) com
CC: informix-list (AT) iiug (DOT) org

Bad news... There is currently no workaround (except eventually some network tweaking).

The problem lies in the reverse DNS system call - gethostbyaddr() - The first time it runs it creates a cache of your DNS configuration.

If you later change it, it has no way of knowing the change happened. So it keeps trying to talk to your previously configured DNS server.
The delay is the time it takes to give up....
This can be showed with a simple C program. The cache will only be forgotten if the process calls another system function. Currently Informix doesn't do that. A mechanism would have to be implemented to allow the DBA to "ask"Informix to run that function. Note that other solutions like periodicallycall it, would be bad for performance.


I've recently created a feature request for this. Note that this problem can affect any kind of server software (I've seen it with sendmail a few years ago).

If your configuration of the resolver system included a first try with the files (/etc/hosts) you could try to workaround it by including all the client host IP address/names in the file.

This is usually not advisable, but you may consider it a workaround untilyou can restart Informix

Another possible option (and I'm getting completely out of my comfort zone here) could be trying to full the network layer to re-write the packet thatare sent to the old DNS in a way that they're directed to the new one....


If you feel sufficiently annoyed by this you can also open a PMR (which will not solve your problem) and make sure your need is mapped to the feature I'm asked for. I could provide details.

Regards.


On Sat, Mar 26, 2011 at 6:07 PM, Jackson Thompson <jackson (AT) valu-search (DOT) com> wrote:

I changed our DNS server and when I disconnect the old DNS server our

informix ids 11.5 instance takes up to 30 seconds to find the

databases. If you use dbaccess it starts and when is is going to list

the databases it stalls, the connect process and all query seem to run

fine. We I have the old DNS server off line I can get good forward and

reverse DNS lookups so the DNS for the os (debian linux) seems to be

fine. I have not restart the Informix instance, it is a production

system of a 24/7 contact center, so I don't want to restart without

knowing that there might be a fix in it.

Thanks

Jackson

_______________________________________________

Informix-list mailing list

Informix-list (AT) iiug (DOT) org

http://www.iiug.org/mailman/listinfo/informix-list



--
Fernando Nunes
Portugal

http://informix-technology.blogspot.com
My email works... but I don't check it frequently...



_______________________________________________
Informix-list mailing list
Informix-list (AT) iiug (DOT) org
http://www.iiug.org/mailman/listinfo/informix-list

Reply With Quote
  #4  
Old   
Ian Goddard
 
Posts: n/a

Default Re: Changed DNS Server and now database connection is very slow - 03-27-2011 , 03:19 PM



Cesar Inacio Martins wrote:
Quote:
If the Jack problem is similar what occur with me a few months ago (with
RH 5), the cache on Informix will not solve the problem.

The problem is at OS level.
Some versions of Linux (RH and appear Debian too) cache this damn
configuration and just don't refresh it. The only way to solve is the
"Windows way"... kick the machine (if you are stress)... and reset it...
Not necessarily.

arp -d

Years ago - about the time of Informix 4.x - we had HP-UX boxes which
had to use DEC-Net to talk to a VAX. One of the characteristics of
DEC-Net was that it used the lower bits of the MAC as the network
address, the upper bits being assumed to be those used by DEC. So when
we brought the 3rd party S/W online it rewrote the MAC on the HP box.
All the clients, probably Win 3.0 back in those days or maybe just
terminal servers, lost connectivity. However, they timed out, re-arped
& were back in about a minute without, as far as I know, loosing the
actual telnet session. Odd how improvements don't always make things
better.

--
Ian

The Hotmail address is my spam-bin. Real mail address is iang
at austonley org uk

Reply With Quote
  #5  
Old   
Cesar Inacio Martins
 
Posts: n/a

Default Re: Changed DNS Server and now database connection is very slow - 03-27-2011 , 04:00 PM



Hum... I don't remember if we try this...

Fernando, can you test this solution?

--- Em dom, 27/3/11, Ian Goddard <goddai01 (AT) hotmail (DOT) co.uk> escreveu:

De: Ian Goddard <goddai01 (AT) hotmail (DOT) co.uk>
Assunto: Re: Changed DNS Server and now database connection is very slow
Para: informix-list (AT) iiug (DOT) org
Data: Domingo, 27 de Março de 2011, 17:19

Cesar Inacio Martins wrote:
Quote:
If the Jack problem is similar what occur with me a few months ago (with
RH 5), the cache on Informix will not solve the problem.

The problem is at OS level.
Some versions of Linux (RH and appear Debian too) cache this damn
configuration and just don't refresh it. The only way to solve is the
"Windows way"... kick the machine (if you are stress)... and* reset it....
Not necessarily.

arp -d

Years ago - about the time of Informix 4.x - we had HP-UX boxes which
had to use DEC-Net to talk to a VAX.* One of the characteristics of
DEC-Net was that it used the lower bits of the MAC as the network
address, the upper bits being assumed to be those used by DEC.* So when
we brought the 3rd party S/W online it rewrote the MAC on the HP box.
All the clients, probably Win 3.0 back in those days or maybe just
terminal servers, lost connectivity.* However, they timed out, re-arped
& were back in about a minute without, as far as I know, loosing the
actual telnet session.* Odd how improvements don't always make things
better.

--
Ian

The Hotmail address is my spam-bin.* Real mail address is iang
at austonley org uk
_______________________________________________
Informix-list mailing list
Informix-list (AT) iiug (DOT) org
http://www.iiug.org/mailman/listinfo/informix-list

Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.