Updated tokenizer to better matching when search for code snippets (#32261)
This PR improves the accuracy of Gitea's code search.
Currently, Gitea does not consider statements such as
`onsole.log("hello")` as hits when the user searches for `log`. The
culprit is how both ES and Bleve are tokenizing the file contents (in
both cases, `console.log` is a whole token).
In ES' case, we changed the tokenizer to
[simple_pattern_split](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-simplepatternsplit-tokenizer.html#:~:text=The%20simple_pattern_split%20tokenizer%20uses%20a,the%20tokenization%20is%20generally%20faster.).
In such a case, tokens are words formed by digits and letters. In
Bleve's case, it employs a
[letter](https://blevesearch.com/docs/Tokenizers/) tokenizer.
Resolves #32220
---------
Signed-off-by: Bruno Sofiato <bruno.sofiato@gmail.com>
This commit is contained in:
@@ -4,5 +4,6 @@ This repository will be used to test code search. The snippet below shows its di
|
||||
├── avocado.md
|
||||
├── cucumber.md
|
||||
├── ham.md
|
||||
└── potato
|
||||
└── ham.md
|
||||
├── potato
|
||||
| └── ham.md
|
||||
└── example-file.js
|
||||
@@ -3,7 +3,7 @@
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/develop
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/feature/1
|
||||
78fb907e3a3309eae4fe8fef030874cebbf1cd5e refs/heads/home-md-img-check
|
||||
3731fe53b763859aaf83e703ee731f6b9447ff1e refs/heads/master
|
||||
9f894b61946fd2f7b8b9d8e370e4d62f915522f5 refs/heads/master
|
||||
62fb502a7172d4453f0322a2cc85bddffa57f07a refs/heads/pr-to-update
|
||||
4649299398e4d39a5c09eb4f534df6f1e1eb87cc refs/heads/sub-home-md-img-check
|
||||
3fa2f829675543ecfc16b2891aebe8bf0608a8f4 refs/notes/commits
|
||||
|
||||
Binary file not shown.
@@ -1,2 +1,2 @@
|
||||
P pack-393dc29256bc27cb2ec73898507df710be7a3cf5.pack
|
||||
P pack-a7bef76cf6e2b46bc816936ab69306fb10aea571.pack
|
||||
|
||||
|
||||
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -4,7 +4,7 @@
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/develop
|
||||
65f1bf27bc3bf70f64657658635e66094edbcb4d refs/heads/feature/1
|
||||
78fb907e3a3309eae4fe8fef030874cebbf1cd5e refs/heads/home-md-img-check
|
||||
3731fe53b763859aaf83e703ee731f6b9447ff1e refs/heads/master
|
||||
9f894b61946fd2f7b8b9d8e370e4d62f915522f5 refs/heads/master
|
||||
62fb502a7172d4453f0322a2cc85bddffa57f07a refs/heads/pr-to-update
|
||||
4649299398e4d39a5c09eb4f534df6f1e1eb87cc refs/heads/sub-home-md-img-check
|
||||
3fa2f829675543ecfc16b2891aebe8bf0608a8f4 refs/notes/commits
|
||||
|
||||
Reference in New Issue
Block a user