dbTalk Databases Forums  

What is the limit for number of dimension members MSAS can handle?

microsoft.public.sqlserver.olap microsoft.public.sqlserver.olap


Discuss What is the limit for number of dimension members MSAS can handle? in the microsoft.public.sqlserver.olap forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Yubo Fan
 
Posts: n/a

Default What is the limit for number of dimension members MSAS can handle? - 07-11-2003 , 05:06 PM






One of my cubes uses 3 large dimensions. Each dimension has 1 million
members. After the dimensions and cube are processed, MSAS is taking 1.5 GB
memory. Does that mean MSAS would not be able to handle 6 million dimension
members, considering it has only 3GB to play with?

How does MSAS manage its memory when caching dimension members? It seems
MSAS just keeps allocating memory. Does it have some kind of LRU algorithm
to manage limited amount of cache based on memeory limitation?

A future cube of mine would involve a dimension with 9 million members. It
would be interesting to see how MSAS deals with it.

- Yubo Fan



Reply With Quote
  #2  
Old   
Yubo Fan
 
Posts: n/a

Default Re: What is the limit for number of dimension members MSAS can handle? - 07-11-2003 , 05:22 PM






Forgot to mention: all of my dimensions are parent-child dimensions with no
extra member properties.

"Yubo Fan" <yubo.fan (AT) sas (DOT) com> wrote

Quote:
One of my cubes uses 3 large dimensions. Each dimension has 1 million
members. After the dimensions and cube are processed, MSAS is taking 1.5
GB
memory. Does that mean MSAS would not be able to handle 6 million
dimension
members, considering it has only 3GB to play with?

How does MSAS manage its memory when caching dimension members? It seems
MSAS just keeps allocating memory. Does it have some kind of LRU algorithm
to manage limited amount of cache based on memeory limitation?

A future cube of mine would involve a dimension with 9 million members. It
would be interesting to see how MSAS deals with it.

- Yubo Fan





Reply With Quote
  #3  
Old   
Bill Cheng [MSFT]
 
Posts: n/a

Default RE: What is the limit for number of dimension members MSAS can handle? - 07-14-2003 , 08:14 AM



Hi Yubo,

Analysis Services uses the Very Large Dimension Manager (VLDM) to move
large dimensions out of the Analysis Services Server process space in an
attempt to improve the scalability of Analysis Services. The threshold for
large dimensions is ~64MB, this can be configured in the Registry by
setting the VLDMThreshold value. Any dimension that exceeds the
VLDMThreshold is automatically moved to the VLDM process.

You may check the following article.
http://msdn.microsoft.com/library/de...us/dnolap/html
/olapperf.asp

Memory
Memory consumed during processing includes both the memory used for
processing dimensions, and memory used for processing the facts on a
segment-by-segment basis.

Memory for processing dimensions + Memory for processing facts

When dimensions are processed, the intermediate structures are also kept in
memory. This means the memory requirement is potentially doubled. This can
make a difference when processing large dimensions. Currently, we estimate
taking about 100 bytes per member. This memory requirement can increase
tremendously depending on whether member properties are used and what their
sizes are.

When the fact table is processed, memory is consumed on a
segment-by-segment basis, therefore the maximum memory used would be for
storing a segment's worth of data in memory. As discussed above, changing
the ReadAheadBufferSize and ProcessReadSegmentSize will affect the memory
requirements.


Bill Cheng
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.
--------------------
Quote:
From: "Yubo Fan" <yubo.fan (AT) sas (DOT) com
Newsgroups: microsoft.public.sqlserver.olap
Subject: What is the limit for number of dimension members MSAS can
handle?
Date: Fri, 11 Jul 2003 15:06:41 -0700
Organization: SAS Institute, Inc.
Lines: 15
Message-ID: <bencdi$d5r$1 (AT) license1 (DOT) unx.sas.com
NNTP-Posting-Host: d7925.na.sas.com
X-Trace: license1.unx.sas.com 1057961202 13499 172.25.126.77 (11 Jul 2003
22:06:42 GMT)
X-Complaints-To: usenet (AT) unx (DOT) sas.com
NNTP-Posting-Date: 11 Jul 2003 22:06:42 GMT
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 6.00.2800.1106
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
Path:
cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!newsfeed 00.sul.t-online.de!t-onlin
e.de!news-lei1.dfn.de!news-fra1.dfn.de!npeer.de.kpn-eurorings.net!news-out.n
uthinbutnews.com!propagator2-sterling!In.nntp.be!vienna7.his.com!attws1!ip.a
tt.net!lamb.sas.com!newshost!not-for-mail
Quote:
Xref: cpmsftngxa06.phx.gbl microsoft.public.sqlserver.olap:40636
X-Tomcat-NG: microsoft.public.sqlserver.olap

One of my cubes uses 3 large dimensions. Each dimension has 1 million
members. After the dimensions and cube are processed, MSAS is taking 1.5
GB
memory. Does that mean MSAS would not be able to handle 6 million
dimension
members, considering it has only 3GB to play with?

How does MSAS manage its memory when caching dimension members? It seems
MSAS just keeps allocating memory. Does it have some kind of LRU algorithm
to manage limited amount of cache based on memeory limitation?

A future cube of mine would involve a dimension with 9 million members. It
would be interesting to see how MSAS deals with it.

- Yubo Fan





Reply With Quote
  #4  
Old   
Yubo Fan
 
Posts: n/a

Default Re: What is the limit for number of dimension members MSAS can handle? - 07-14-2003 , 01:12 PM



Bill,

Thanks for the reply. Here are some follow up questions:

1. Where can I find more information about VLDM?

2. Where's the registry entry you mentioned?

3. If the VLDM threshold is 64MB and each member takes 100 bytes, all of my
large dimensions (1 million member) should have been moved to the VLDM
process. However, why was I seeing the MSAS process (msmdsrv.exe) took 1.5
GB memory after the dimensions were processed? (It was taking 150 MB memory
before those dimensions were processed)

4. Even with the consideration of doubling of the memory usage during
dimension processing. The memory usage seems to indicate that each dimension
member would take 250 bytes.

5. What is the process name for this VLDM process?

Thanks again.

Best regards,

Yubo Fan

"Bill Cheng [MSFT]" <billchng (AT) online (DOT) microsoft.com> wrote

Quote:
Hi Yubo,

Analysis Services uses the Very Large Dimension Manager (VLDM) to move
large dimensions out of the Analysis Services Server process space in an
attempt to improve the scalability of Analysis Services. The threshold
for
large dimensions is ~64MB, this can be configured in the Registry by
setting the VLDMThreshold value. Any dimension that exceeds the
VLDMThreshold is automatically moved to the VLDM process.

You may check the following article.

http://msdn.microsoft.com/library/de...us/dnolap/html
/olapperf.asp

Memory
Memory consumed during processing includes both the memory used for
processing dimensions, and memory used for processing the facts on a
segment-by-segment basis.

Memory for processing dimensions + Memory for processing facts

When dimensions are processed, the intermediate structures are also kept
in
memory. This means the memory requirement is potentially doubled. This can
make a difference when processing large dimensions. Currently, we estimate
taking about 100 bytes per member. This memory requirement can increase
tremendously depending on whether member properties are used and what
their
sizes are.

When the fact table is processed, memory is consumed on a
segment-by-segment basis, therefore the maximum memory used would be for
storing a segment's worth of data in memory. As discussed above, changing
the ReadAheadBufferSize and ProcessReadSegmentSize will affect the memory
requirements.


Bill Cheng
Microsoft Online Partner Support

Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.
--------------------
| From: "Yubo Fan" <yubo.fan (AT) sas (DOT) com
| Newsgroups: microsoft.public.sqlserver.olap
| Subject: What is the limit for number of dimension members MSAS can
handle?
| Date: Fri, 11 Jul 2003 15:06:41 -0700
| Organization: SAS Institute, Inc.
| Lines: 15
| Message-ID: <bencdi$d5r$1 (AT) license1 (DOT) unx.sas.com
| NNTP-Posting-Host: d7925.na.sas.com
| X-Trace: license1.unx.sas.com 1057961202 13499 172.25.126.77 (11 Jul
2003
22:06:42 GMT)
| X-Complaints-To: usenet (AT) unx (DOT) sas.com
| NNTP-Posting-Date: 11 Jul 2003 22:06:42 GMT
| X-Priority: 3
| X-MSMail-Priority: Normal
| X-Newsreader: Microsoft Outlook Express 6.00.2800.1106
| X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
| Path:

cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!newsfeed 00.sul.t-online.de!t-onlin

e.de!news-lei1.dfn.de!news-fra1.dfn.de!npeer.de.kpn-eurorings.net!news-out.n

uthinbutnews.com!propagator2-sterling!In.nntp.be!vienna7.his.com!attws1!ip.a
tt.net!lamb.sas.com!newshost!not-for-mail
| Xref: cpmsftngxa06.phx.gbl microsoft.public.sqlserver.olap:40636
| X-Tomcat-NG: microsoft.public.sqlserver.olap
|
| One of my cubes uses 3 large dimensions. Each dimension has 1 million
| members. After the dimensions and cube are processed, MSAS is taking 1.5
GB
| memory. Does that mean MSAS would not be able to handle 6 million
dimension
| members, considering it has only 3GB to play with?
|
| How does MSAS manage its memory when caching dimension members? It seems
| MSAS just keeps allocating memory. Does it have some kind of LRU
algorithm
| to manage limited amount of cache based on memeory limitation?
|
| A future cube of mine would involve a dimension with 9 million members.
It
| would be interesting to see how MSAS deals with it.
|
| - Yubo Fan
|
|
|




Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.