New addon: Fuzzy Search with PostgreSQL pg_trgm extension by christophlsa · Pull Request #462 · OCA/server-tools

christophlsa · 2016-06-22T07:34:23Z

This addon provides the ability to create GIN or GiST indexes of char and text fields and also to use the search operator % in search domains. This is a first basic implementation which should be extended later.

Requirements

First you need to have the pg_trgm extension available. In debian based distribution you have to install the postgresql-contrib module. Then you have to either install the pg_trgmextension to your database or you have to give your postgresql user the superadmin right (then the extension will be installed by this module).

Explanation

If you installed the odoo module you can define in Settings -> Database Structure -> Trigam Index the indexes. There the index name will automatically created.

For example you can create an index for the name field of res.partner. Then in a search you can use self.env['res.partner'].search([('name', '%', 'Jon Miller)]). In this Example it can find existing names like John Miller or John Mill.

Which strings can be found depends on the limit which is set (default: 0.3). Currently you have to set the limit by executing SQL: self.env.cr.execute("SELECT set_limit(0.2);").

Also you can use the similarity(column, 'text') function in the order parameter to order by the similarity. This is just a basic implementation which doesn't contains validations and has to start with this function. For example you can define this like "similarity(%s.name, 'John Mil') DESC" % self.env['res.partner']._table.

For further questions read the Documentation of the pg_trgm module.

Tasks/Goals

For this PR:

Later:

Modify the general search parts (e.g. in tree view or many2one fields)

lmignon · 2016-06-22T08:01:39Z

+def patch_generate_order_by(method):
+    def decorate_generate_order_by(self, order_spec, query):
+        if order_spec and order_spec.startswith('similarity('):
+            from pprint import pprint


remove pprint lines

Thanks. I think I committed to fast ^^

dreispt · 2016-06-22T08:37:04Z

The initial approach looks good.
Take care with the technique used to patch the core, it can be tricky, so unit tests will be your friend.

christophlsa · 2016-06-24T09:18:43Z

Renamed module base_trgm_search to base_search_fuzzy,
added Unit tests
added german translations
added access permissions
moved the monkey patching to method _register_hook of ir.model
fixed _auto_init (wrong parameter)
added README
cleaned up some parts

dreispt · 2016-06-24T12:32:42Z

Is it ready for review?

christophlsa · 2016-06-24T13:13:00Z

for me yes :)

elicoidal · 2016-06-24T14:09:38Z

+
+This addon provides the ability to create GIN or GiST indexes of char and text
+fields and also to use the search operator `%` in search domains. Currently
+this module doesn't change the backend search or something else. It provides


s/something/anything

elicoidal · 2016-06-24T14:19:55Z

Thanks for the contribution!

moylop260 · 2016-06-24T15:22:12Z

Please, Could you review our beta checks https://travis-ci.org/OCA/server-tools/jobs/139978373#L424-L430?

************* Module base_search_fuzzy.models.trgm_index
base_search_fuzzy/models/trgm_index.py:122: [C8107(translation-required), TrgmIndex.create_index] String parameter of raise "UserError" requires translation. Use _('The pg_trgm extension does not exists or cannot be installed.')

Ignore

************* Module base_search_fuzzy.models.ir_model
base_search_fuzzy/models/ir_model.py:24: [W0402(deprecated-module), ] Uses of a deprecated module 'openerp.osv'
************* Module base_search_fuzzy.tests.test_query_generation
base_search_fuzzy/tests/test_query_generation.py:3: [W0402(deprecated-module), ] Uses of a deprecated module 'openerp.osv'

StefanRijnhart · 2016-06-24T15:28:58Z

@moylop260 FYI two of those are the dreaded false positive for imports from openerp.osv again. OCA/maintainer-quality-tools#306. What is the recommended solution for that?

moylop260 · 2016-06-24T16:42:27Z

@StefanRijnhart
You are right I'll review this check

moylop260 · 2016-06-24T23:46:54Z

👍
(Just add _translate method in raises)
Good job! thanks

christophlsa · 2016-06-27T12:57:09Z

Thanks for the comments. I updated all. :)

…nd allowing the sql parameter %.

…t tests, added translations, added access permissions, moved the monkey patching to method _register_hook of ir.model and fixed _auto_init, added README, cleaned up some aprts

…on, Added one more test

…avoid decorating the function many times whcih leads to recursion errors

christophlsa · 2016-07-19T08:51:51Z

I added a fix to check if the functions which I overwrite are already decorated by the new decorator. We had problems that these functions got this decorator much times (in particular in the cron worker) and a maximum recursion depth error was thrown.

I also rebased the branch.

gurneyalex · 2016-08-09T13:46:14Z

travis red build should be fixed by #507

gurneyalex · 2016-08-09T13:47:36Z

+#. The PostgreSQL extension ``pg_trgm`` should be available. In debian based
+   distribution you have to install the `postgresql-contrib` module.
+#. Install the ``pg_trgm`` extension to your database or give your postgresql
+   user the ``SUEPRUSER`` right (this allows the odoo module to install the


s/SUEPRUSER/SUPERUSER/

gurneyalex · 2016-08-09T13:49:16Z

one small typo, but 👍

dreispt · 2016-08-10T09:03:22Z

PR https://github.com/blooparksystems/server-tools/pull/1

dreispt · 2016-08-10T09:04:32Z

TravisCI is red 🚧

Fix typo

christophlsa · 2016-08-10T11:03:55Z

After rerunning the checks (and merging the Readme fix) all are green.

Syncing from upstream OCA/server-tools (9.0)

lmignon reviewed Jun 22, 2016
View reviewed changes

dreispt added this to the 8.0 milestone Jun 22, 2016

dreispt added the work in progress label Jun 22, 2016

christophlsa changed the title ~~[WIP] New addon: PostgreSQL Trigram Search~~ New addon: Fuzzy Search with PostgreSQL pg_trgm extension Jun 24, 2016

elicoidal reviewed Jun 24, 2016
View reviewed changes

Christoph Giesel added 4 commits July 19, 2016 10:48

[ADD] First implementation of base_trgm_search for creating indexes a…

72a25a0

…nd allowing the sql parameter %.

[IMP] Allow to order by similarity.

ece8a4c

[IMP] Renamed module base_trgm_search to base_search_fuzzy, added Uni…

c4d5d0b

…t tests, added translations, added access permissions, moved the monkey patching to method _register_hook of ir.model and fixed _auto_init, added README, cleaned up some aprts

[IMP] base_search_fuzzy: Updated README, translated UserError Excepti…

94376b3

…on, Added one more test

Christoph Giesel added 3 commits July 19, 2016 10:48

[IMP] base_search_fuzzy: Updated license header

34cca79

[FIX] Fixed missing instance variable after cleanup

2a2c71d

base_search_fuzzy: [FIX] Check if functions are already decorated to …

e6c37fb

…avoid decorating the function many times whcih leads to recursion errors

gurneyalex added needs review and removed work in progress labels Aug 9, 2016

gurneyalex reviewed Aug 9, 2016
View reviewed changes

Fix typo

10cb5aa

Merge pull request #1 from dreispt/patch-1

1d2bf08

Fix typo

dreispt merged commit 9d1099e into OCA:8.0 Aug 10, 2016

SiesslPhillip pushed a commit to grueneerde/OCA-server-tools that referenced this pull request Nov 20, 2024

Merge pull request OCA#462 from OCA/9.0

8be3f1c

Syncing from upstream OCA/server-tools (9.0)

Uh oh!

Uh oh!

Conversation

christophlsa commented Jun 22, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Requirements

Explanation

Tasks/Goals

Uh oh!

lmignon Jun 22, 2016

Choose a reason for hiding this comment

Uh oh!

christophlsa Jun 24, 2016

Choose a reason for hiding this comment

Uh oh!

dreispt commented Jun 22, 2016

Uh oh!

christophlsa commented Jun 24, 2016

Uh oh!

dreispt commented Jun 24, 2016

Uh oh!

christophlsa commented Jun 24, 2016

Uh oh!

elicoidal Jun 24, 2016

Choose a reason for hiding this comment

Uh oh!

elicoidal commented Jun 24, 2016

Uh oh!

moylop260 commented Jun 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

StefanRijnhart commented Jun 24, 2016

Uh oh!

moylop260 commented Jun 24, 2016

Uh oh!

moylop260 commented Jun 24, 2016

Uh oh!

christophlsa commented Jun 27, 2016

Uh oh!

christophlsa commented Jul 19, 2016

Uh oh!

gurneyalex commented Aug 9, 2016

Uh oh!

gurneyalex Aug 9, 2016

Choose a reason for hiding this comment

Uh oh!

gurneyalex commented Aug 9, 2016

Uh oh!

dreispt commented Aug 10, 2016

Uh oh!

dreispt commented Aug 10, 2016

Uh oh!

christophlsa commented Aug 10, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

christophlsa commented Jun 22, 2016 •

edited

Loading

moylop260 commented Jun 24, 2016 •

edited

Loading