tsearch2: List of lexemes and their count

#1

Hi,

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

Thomas

--
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de

#2

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

#3

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

#4

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

#5

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

#6

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

#7

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

#8

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

#9

Thomas Guettler <hv (AT) tbz-pariv (DOT) de> wrote:

Quote:

How can you get all lexemes of an indexed column and how
often they occur?

This would help me to find words which should
be added to the stop word list.

This is explained in
http://www.postgresql.org/docs/curre...RCH-STATISTICS

An example:

CREATE TABLE fts(id integer PRIMARY KEY, val text);
CREATE INDEX fts_index ON fts USING gin(to_tsvector('german', val));

INSERT INTO fts VALUES (1,
'dies ist ein deutscher Text mit mehreren Worten; '

Quote:

| 'etliche Worte doppelt.');
INSERT INTO fts VALUES (2,

'ein weiterer deutscher Text enthält andere Worte, '

Quote:

| 'aber natürlich auch die selben.');
INSERT INTO fts VALUES (3,

'Ein dritter deutscher Text rundet die Sache ab.');

Now, for the statistics, I use the same function as I used to define
the index:

SELECT * FROM ts_stat('SELECT to_tsvector(''german'', val) FROM fts');

word | ndoc | nentry
---------+------+--------
ab | 1 | 1
mehr | 1 | 1
sach | 1 | 1
selb | 1 | 1
text | 3 | 3
weit | 1 | 1
wort | 2 | 3
dritt | 1 | 1
natur | 1 | 1
etlich | 1 | 1
rundet | 1 | 1
deutsch | 3 | 3
doppelt | 1 | 1
enthalt | 1 | 1
(14 rows)

Yours,
Laurenz Albe

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

tsearch2: List of lexemes and their count

comp.databases.postgresql comp.databases.postgresql

tsearch2: List of lexemes and their count - 03-25-2008 , 09:36 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM

Re: tsearch2: List of lexemes and their count - 03-26-2008 , 03:26 AM