dbTalk Databases Forums  

ETL / Dimension updating

microsoft.public.sqlserver.olap microsoft.public.sqlserver.olap


Discuss ETL / Dimension updating in the microsoft.public.sqlserver.olap forum.



Reply
 
Thread Tools Display Modes
  #1  
Old   
Michael Vardinghus
 
Posts: n/a

Default ETL / Dimension updating - 05-02-2004 , 11:49 AM






In Our environment we have chosen to drop the whole subject matter
environment and put in a new every time we update.

The reason for doing so is that we don't have that many records 3-4 million
(but we have sevearal dimensions) and we're pulling data out of one
ERP-system with a trustworthy datamodel.

So we wouldn't have any complexity in the transport that isn't necessary.

Now .. however...i'm getting some doubts - what does it mean dimension /
processing / aggretation wise when we drop the dimension table totally with
it's surrogate keys and rebuild the structure from scratch every time.

Is this just a "time used to process cube" problem ? If so I'm not worried -
this doesn't take long ... but do I need to make a total rebuild of
dimensions every time because of the approach above...? I could very well
have a dimension member which have a surrogate key before which now has a
new surrogate key because a member has been added since the last time ?

If it's necessary to make a total processing what impact does this have on
the aggregations ? Are they dropped and created again ? Even those based on
usage based optimization ?

Lots of questions - hope somebody can enlighten me.

\Michael Vardinghus



Reply With Quote
  #2  
Old   
Dave Wickert [MSFT]
 
Posts: n/a

Default Re: ETL / Dimension updating - 05-02-2004 , 09:21 PM






Full processing of a dimension always drops all aggregates in all cubes that
use that dimension regardless of the type or structure of the dimension.
--
Dave Wickert [MS]
dwickert (AT) online (DOT) microsoft.com
Program Manager
BI Practices Team
SQL BI Product Unit (Analysis Services)
--
This posting is provided "AS IS" with no warranties, and confers no rights.


"Michael Vardinghus" <michaelvardinghus (AT) hotmail (DOT) com> wrote

Quote:
In Our environment we have chosen to drop the whole subject matter
environment and put in a new every time we update.

The reason for doing so is that we don't have that many records 3-4
million
(but we have sevearal dimensions) and we're pulling data out of one
ERP-system with a trustworthy datamodel.

So we wouldn't have any complexity in the transport that isn't necessary.

Now .. however...i'm getting some doubts - what does it mean dimension /
processing / aggretation wise when we drop the dimension table totally
with
it's surrogate keys and rebuild the structure from scratch every time.

Is this just a "time used to process cube" problem ? If so I'm not
worried -
this doesn't take long ... but do I need to make a total rebuild of
dimensions every time because of the approach above...? I could very well
have a dimension member which have a surrogate key before which now has a
new surrogate key because a member has been added since the last time ?

If it's necessary to make a total processing what impact does this have on
the aggregations ? Are they dropped and created again ? Even those based
on
usage based optimization ?

Lots of questions - hope somebody can enlighten me.

\Michael Vardinghus





Reply With Quote
  #3  
Old   
Michael Vardinghus
 
Posts: n/a

Default Re: ETL / Dimension updating - 05-03-2004 , 04:09 AM



Aha....and translated does that mean that a dimension with for instance
surrogate key 10 must not never have this changed for incremental processing
to work ?

Incrementally processing a dimension table where the surrogate keys could
have changed on some of the existing dimensions is a dangerous action, yes ?

\Michael V.


"Dave Wickert [MSFT]" <dwickert (AT) online (DOT) microsoft.com> skrev i en meddelelse
news:ub9HMWLMEHA.1388 (AT) TK2MSFTNGP09 (DOT) phx.gbl...
Quote:
Full processing of a dimension always drops all aggregates in all cubes
that
use that dimension regardless of the type or structure of the dimension.
--
Dave Wickert [MS]
dwickert (AT) online (DOT) microsoft.com
Program Manager
BI Practices Team
SQL BI Product Unit (Analysis Services)
--
This posting is provided "AS IS" with no warranties, and confers no
rights.


"Michael Vardinghus" <michaelvardinghus (AT) hotmail (DOT) com> wrote in message
news:%23QsMiTGMEHA.3556 (AT) TK2MSFTNGP09 (DOT) phx.gbl...
In Our environment we have chosen to drop the whole subject matter
environment and put in a new every time we update.

The reason for doing so is that we don't have that many records 3-4
million
(but we have sevearal dimensions) and we're pulling data out of one
ERP-system with a trustworthy datamodel.

So we wouldn't have any complexity in the transport that isn't
necessary.

Now .. however...i'm getting some doubts - what does it mean dimension /
processing / aggretation wise when we drop the dimension table totally
with
it's surrogate keys and rebuild the structure from scratch every time.

Is this just a "time used to process cube" problem ? If so I'm not
worried -
this doesn't take long ... but do I need to make a total rebuild of
dimensions every time because of the approach above...? I could very
well
have a dimension member which have a surrogate key before which now has
a
new surrogate key because a member has been added since the last time ?

If it's necessary to make a total processing what impact does this have
on
the aggregations ? Are they dropped and created again ? Even those based
on
usage based optimization ?

Lots of questions - hope somebody can enlighten me.

\Michael Vardinghus







Reply With Quote
  #4  
Old   
Dave Wickert [MSFT]
 
Posts: n/a

Default Re: ETL / Dimension updating - 05-03-2004 , 09:39 AM



Changes to keys will result in a new dimension member being added -- not
your intended result. Analysis Services has no way to know that the member
is the same -- as the key has changed (which is the definition of a "key").
If you are changing keys, then you must do a full-process.
--
Dave Wickert [MS]
dwickert (AT) online (DOT) microsoft.com
Program Manager
BI Practices Team
SQL BI Product Unit (Analysis Services)
--
This posting is provided "AS IS" with no warranties, and confers no rights.


"Michael Vardinghus" <michaelvardinghus (AT) hotmail (DOT) com> wrote

Quote:
Aha....and translated does that mean that a dimension with for instance
surrogate key 10 must not never have this changed for incremental
processing
to work ?

Incrementally processing a dimension table where the surrogate keys could
have changed on some of the existing dimensions is a dangerous action, yes
?

\Michael V.


"Dave Wickert [MSFT]" <dwickert (AT) online (DOT) microsoft.com> skrev i en
meddelelse
news:ub9HMWLMEHA.1388 (AT) TK2MSFTNGP09 (DOT) phx.gbl...
Full processing of a dimension always drops all aggregates in all cubes
that
use that dimension regardless of the type or structure of the dimension.
--
Dave Wickert [MS]
dwickert (AT) online (DOT) microsoft.com
Program Manager
BI Practices Team
SQL BI Product Unit (Analysis Services)
--
This posting is provided "AS IS" with no warranties, and confers no
rights.


"Michael Vardinghus" <michaelvardinghus (AT) hotmail (DOT) com> wrote in message
news:%23QsMiTGMEHA.3556 (AT) TK2MSFTNGP09 (DOT) phx.gbl...
In Our environment we have chosen to drop the whole subject matter
environment and put in a new every time we update.

The reason for doing so is that we don't have that many records 3-4
million
(but we have sevearal dimensions) and we're pulling data out of one
ERP-system with a trustworthy datamodel.

So we wouldn't have any complexity in the transport that isn't
necessary.

Now .. however...i'm getting some doubts - what does it mean dimension
/
processing / aggretation wise when we drop the dimension table totally
with
it's surrogate keys and rebuild the structure from scratch every time.

Is this just a "time used to process cube" problem ? If so I'm not
worried -
this doesn't take long ... but do I need to make a total rebuild of
dimensions every time because of the approach above...? I could very
well
have a dimension member which have a surrogate key before which now
has
a
new surrogate key because a member has been added since the last time
?

If it's necessary to make a total processing what impact does this
have
on
the aggregations ? Are they dropped and created again ? Even those
based
on
usage based optimization ?

Lots of questions - hope somebody can enlighten me.

\Michael Vardinghus









Reply With Quote
  #5  
Old   
Michael Vardinghus
 
Posts: n/a

Default Re: ETL / Dimension updating - 05-03-2004 , 01:26 PM



Thanks Dave - thought so.

Just one more thing...

Is there anything wrong with dumping the fact table and regenerating it -
just as long as I don't change the surrogate keys of
the existing dimensions ?

I actually believe this will be the easiest approach instead of redesigning
and getting only new records from host.

If I dump the fact table every time I don't need to make a full proces...the
full process is only when changing dimension structure ?

\Michael V.


"Dave Wickert [MSFT]" <dwickert (AT) online (DOT) microsoft.com> skrev i en meddelelse
news:%238g8hyRMEHA.3012 (AT) tk2msftngp13 (DOT) phx.gbl...
Quote:
Changes to keys will result in a new dimension member being added -- not
your intended result. Analysis Services has no way to know that the member
is the same -- as the key has changed (which is the definition of a
"key").
If you are changing keys, then you must do a full-process.
--
Dave Wickert [MS]
dwickert (AT) online (DOT) microsoft.com
Program Manager
BI Practices Team
SQL BI Product Unit (Analysis Services)
--
This posting is provided "AS IS" with no warranties, and confers no
rights.


"Michael Vardinghus" <michaelvardinghus (AT) hotmail (DOT) com> wrote in message
news:Oxk4K3OMEHA.2704 (AT) TK2MSFTNGP10 (DOT) phx.gbl...
Aha....and translated does that mean that a dimension with for instance
surrogate key 10 must not never have this changed for incremental
processing
to work ?

Incrementally processing a dimension table where the surrogate keys
could
have changed on some of the existing dimensions is a dangerous action,
yes
?

\Michael V.


"Dave Wickert [MSFT]" <dwickert (AT) online (DOT) microsoft.com> skrev i en
meddelelse
news:ub9HMWLMEHA.1388 (AT) TK2MSFTNGP09 (DOT) phx.gbl...
Full processing of a dimension always drops all aggregates in all
cubes
that
use that dimension regardless of the type or structure of the
dimension.
--
Dave Wickert [MS]
dwickert (AT) online (DOT) microsoft.com
Program Manager
BI Practices Team
SQL BI Product Unit (Analysis Services)
--
This posting is provided "AS IS" with no warranties, and confers no
rights.


"Michael Vardinghus" <michaelvardinghus (AT) hotmail (DOT) com> wrote in message
news:%23QsMiTGMEHA.3556 (AT) TK2MSFTNGP09 (DOT) phx.gbl...
In Our environment we have chosen to drop the whole subject matter
environment and put in a new every time we update.

The reason for doing so is that we don't have that many records 3-4
million
(but we have sevearal dimensions) and we're pulling data out of one
ERP-system with a trustworthy datamodel.

So we wouldn't have any complexity in the transport that isn't
necessary.

Now .. however...i'm getting some doubts - what does it mean
dimension
/
processing / aggretation wise when we drop the dimension table
totally
with
it's surrogate keys and rebuild the structure from scratch every
time.

Is this just a "time used to process cube" problem ? If so I'm not
worried -
this doesn't take long ... but do I need to make a total rebuild of
dimensions every time because of the approach above...? I could very
well
have a dimension member which have a surrogate key before which now
has
a
new surrogate key because a member has been added since the last
time
?

If it's necessary to make a total processing what impact does this
have
on
the aggregations ? Are they dropped and created again ? Even those
based
on
usage based optimization ?

Lots of questions - hope somebody can enlighten me.

\Michael Vardinghus











Reply With Quote
Reply




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off



Powered by vBulletin Version 3.5.3
Copyright ©2000 - 2012, Jelsoft Enterprises Ltd.