near duplicates in short text fields

#11

On Aug 15, 3:05*pm, merkury <david.oberm... (AT) idealo (DOT) de> wrote:

Quote:

Hi,

can anybody tell me how to find near duplicates in a large amount (20
million) short text labels?

Is there any database tool which does just this?

I give you some examples:

not near:
Rugby Polo - black/white - S; (Angebot von Kabelmeister)
Rugby Shirt Striped - aqua/white - S; (Angebot von Kabelmeister)

near:
Rugby Shirt Striped - aqua/white - S; (Angebot von Kabelmeister)
Shirt Striped - aqua/white - S; (Angebot von)

near:
301 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT BLAU in L (eBay Shop
jeanspoint74)
482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT SCHWARZ in L (eBay Shop
jeanspoint74)

near:
* *482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT SCHWARZ in L (eBay Shop
jeanspoint74)
* *482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT WEISS in M (eBay Shop
jeanspoint74)

Thanks

merkury

I think a fruitful direction is to avoid a generic comparison
algorithm on these strings, but rather to exploit your domain
knowledge. For example, removing the words that you know are likely to
refer to non-distinguishing attributes (all colour names in whatever
languages you are using) - *then* testing string equality; this
handles the 2nd example.

Your first example could be handled by 'regularising' synonymous terms
or removing redundant terms ('rugby shirt' == 'shirt' for your
purposes). Again, this should be based on domain knowledge.

Hope this helps.
--Toby

#12

On Aug 15, 3:05*pm, merkury <david.oberm... (AT) idealo (DOT) de> wrote:

Quote:

Hi,

can anybody tell me how to find near duplicates in a large amount (20
million) short text labels?

Is there any database tool which does just this?

I give you some examples:

not near:
Rugby Polo - black/white - S; (Angebot von Kabelmeister)
Rugby Shirt Striped - aqua/white - S; (Angebot von Kabelmeister)

near:
Rugby Shirt Striped - aqua/white - S; (Angebot von Kabelmeister)
Shirt Striped - aqua/white - S; (Angebot von)

near:
301 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT BLAU in L (eBay Shop
jeanspoint74)
482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT SCHWARZ in L (eBay Shop
jeanspoint74)

near:
* *482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT SCHWARZ in L (eBay Shop
jeanspoint74)
* *482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT WEISS in M (eBay Shop
jeanspoint74)

Thanks

merkury

I think a fruitful direction is to avoid a generic comparison
algorithm on these strings, but rather to exploit your domain
knowledge. For example, removing the words that you know are likely to
refer to non-distinguishing attributes (all colour names in whatever
languages you are using) - *then* testing string equality; this
handles the 2nd example.

Your first example could be handled by 'regularising' synonymous terms
or removing redundant terms ('rugby shirt' == 'shirt' for your
purposes). Again, this should be based on domain knowledge.

Hope this helps.
--Toby

#13

On Aug 15, 3:05*pm, merkury <david.oberm... (AT) idealo (DOT) de> wrote:

Quote:

Hi,

can anybody tell me how to find near duplicates in a large amount (20
million) short text labels?

Is there any database tool which does just this?

I give you some examples:

not near:
Rugby Polo - black/white - S; (Angebot von Kabelmeister)
Rugby Shirt Striped - aqua/white - S; (Angebot von Kabelmeister)

near:
Rugby Shirt Striped - aqua/white - S; (Angebot von Kabelmeister)
Shirt Striped - aqua/white - S; (Angebot von)

near:
301 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT BLAU in L (eBay Shop
jeanspoint74)
482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT SCHWARZ in L (eBay Shop
jeanspoint74)

near:
* *482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT SCHWARZ in L (eBay Shop
jeanspoint74)
* *482 LA RUGBY SEXY DISCO PARTY POLO T-SHIRT WEISS in M (eBay Shop
jeanspoint74)

Thanks

merkury

I think a fruitful direction is to avoid a generic comparison
algorithm on these strings, but rather to exploit your domain
knowledge. For example, removing the words that you know are likely to
refer to non-distinguishing attributes (all colour names in whatever
languages you are using) - *then* testing string equality; this
handles the 2nd example.

Your first example could be handled by 'regularising' synonymous terms
or removing redundant terms ('rugby shirt' == 'shirt' for your
purposes). Again, this should be based on domain knowledge.

Hope this helps.
--Toby

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

near duplicates in short text fields

comp.databases comp.databases

Re: near duplicates in short text fields - 08-19-2008 , 08:23 AM

Re: near duplicates in short text fields - 08-19-2008 , 08:23 AM

Re: near duplicates in short text fields - 08-19-2008 , 08:23 AM