On Jun 6, 2:13 pm, Erland Sommarskog <esq... (AT) sommarskog (DOT) se> wrote:

That can be, but the issue is the way counties update their records.

Some (VERY few) counties send a monthly "Update" file, but almost all
of them just send the whole dump of their DB.

This gives us two problems:

1) Normalizing the data in a common format (Naturally, no two counties
have the same record format)

2) Updating the db in an "Online" state for Orange County only (With
more than 3mil rows) checking for "Pre-Existing" records would just
kill the server, and there are 3077 counties...

Any better ideas?

Piero

Piero 'Giops' Giorgi (giorgi.piero (AT) gmail (DOT) com) writes:
But this is a problem you have no matter the solution to load the data,
right? Since you work with a partitioned table, you have the same schemas
for all counties. Once you have normalised the format, you can load it
into a staging table and work from there.

Did you actually try it? With proper indexing this does not have to be
that painful.

Then again, if you have code to normalise the data for 3077 counties, I
guess 50 tables with a total of 3077 partitions is a smaller headache.

--
Erland Sommarskog, SQL Server MVP, esquel (AT) sommarskog (DOT) se

Books Online for SQL Server 2005 at
http://www.microsoft.com/technet/pro...ads/books.mspx
Books Online for SQL Server 2000 at
http://www.microsoft.com/sql/prodinf...ons/books.mspx

Erland, thanks for providing Piero with he partitioned view example.


It's true that Piero's county import process will need to be state-table
aware. Rather than traditional dynamic SQL, an alternative is SQLCMD
scripts with variables executed via SQLCMD.EXE or SSMS in SQLCMD mode.
IMHO, the SQLCMD variable approach is a bit cleaner.

The example below assumes the state tables, partition functions and
partition schemes all have a state code suffix. The archive and staging
tables use the same partitioning scheme as the primary table to simply
things. This ensures those objects are on the same file groups and also
eliminates the need to create a check constraint on the staging table county
column.


I must admit I had not considered a partitioned view over partitioned tables
before this thread. I did some cursory testing with Piero's state/county
scenario and it seems to work as expected but there might be gochas. My
biggest concern here is with query complexity when joining the partitioned
view. This approach is probably rarely used so Piero should probably test
thoroughly before committing.


I'm fortunate because I had to develop a complex sliding-window partitioning
scheme for one of our applications that gave me the opportunity to learn the
finer points of SQL 2005 partitioning. There's just so many cool features
in the product nowadays that it's hard to find the time to thoroughly learn
most, let alone all. Now if I could only get fully up to speed on the new
features before SQL Server 2008 ;-)


--sample script for county import process

--define and initialize SQLCMD variables
:setvar StateCode WA
:setvar County Spokane

--create archive table for county
IF OBJECT_ID(N'dbo.CountyArchive', 'U') IS NOT NULL
DROP TABLE dbo.CountyArchive
GO
CREATE TABLE dbo.CountyArchive
(
StateCode char(2) NOT NULL,
County varchar(50) NOT NULL,
CountyData varchar(100),
CONSTRAINT PK_CountyArchive
PRIMARY KEY CLUSTERED (StateCode, County)
ON PS_State_$(StateCode)(county)
)
GO

--create staging table for county
IF OBJECT_ID(N'dbo.CountyStaging', 'U') IS NOT NULL
DROP TABLE dbo.CountyStaging
GO

CREATE TABLE dbo.CountyStaging
(
StateCode char(2) NOT NULL,
County varchar(50) NOT NULL,
CountyData varchar(100),
CONSTRAINT PK_CountyStaging
PRIMARY KEY CLUSTERED (StateCode, County)
ON PS_State_$(StateCode)(county)
)
GO

---------------------------------
--load dbo.CountyStaging table here
---------------------------------

--add constraint needed for parttitioned view
ALTER TABLE dbo.CountyStaging
ADD CONSTRAINT CK_CountyStaging_State CHECK (StateCode = '$(StateCode)')

--move old county data to archive table
ALTER TABLE dbo.State_$(StateCode)
SWITCH PARTITION $PARTITION.PF_State_$(StateCode)('$(County)') TO
dbo.CountyArchive PARTITION
$PARTITION.PF_State_$(StateCode)('$(County)')

--move new county data into state table
ALTER TABLE dbo.CountyStaging
SWITCH PARTITION $PARTITION.PF_State_$(StateCode)('$(County)') TO
dbo.State_$(StateCode) PARTITION
$PARTITION.PF_State_$(StateCode)('$(County)')


--
Hope this helps.

Dan Guzman
SQL Server MVP

"Erland Sommarskog" <esquel (AT) sommarskog (DOT) se> wrote

On Jun 7, 5:15 am, "Dan Guzman" <guzma... (AT) nospam-online (DOT) sbcglobal.net>
wrote:

That's exactly what I had in mind. Thank you!

One thing... is there any way to partition the states, too?
I mean Query the whole beast with only one SQL Query?

Piero

Piero 'Giops' Giorgi (giorgi.piero (AT) gmail (DOT) com) writes:
You would query the view.

One idea to occurred to me is that you could have a mix, so that big
counties like Orange County(*) have a single partition, where as
smaller counties and states would be gathered in the same partition.
This would mean that you would have different models to load the files.
For Orange County you switch table in and out, whereas for smaller
counties you delete and insert. Of course, this means more complex
code since there would be two code paths. But there would be far
fewer partitions to care about.


(*) When I picked Orange County as an example, I did not know that it
was one of the biggies. I just picked it as it was one of the county
names I knew; the name appears in a few Zappa tracks.


--
Erland Sommarskog, SQL Server MVP, esquel (AT) sommarskog (DOT) se

Books Online for SQL Server 2005 at
http://www.microsoft.com/technet/pro...ads/books.mspx
Books Online for SQL Server 2000 at
http://www.microsoft.com/sql/prodinf...ons/books.mspx

On Jun 7, 2:16 pm, Erland Sommarskog <esq... (AT) sommarskog (DOT) se> wrote:
Yes, that is what I'm doing. It's not the optimal, but it's working
enough.

That would give problems while updating the DB.
Ok, I drop the big ones and update the small ones, but IMHO it would
be better stay with one partition per County, grouped by state.
The BEST thing would be to be able to partition the states (Already
partitioned by County) in one big table.

Possible?

It's a BIG one!
But you should see New York...
I'm not saying that there are more criminals in NY, but the city is
bigger and there is a lot more people, so...

For now, I'm experimenting with 50 States, partitioned over 3077
Counties. Seems promising, but the UNION query is a resource hog. :-
(

Thank you all!

P

Piero