SWIRL SEARCH 1.6

[sidprobstein]

SWIRL SEARCH 1.6

Version 1.6 shifts focus from relevancy to query adaptation with the new AdaptiveQueryProcessor that rewrites NOT or -term queries depending on SearchProvider configuration:

Also new: AND, + and OR are passed along to each SearchProvider and ignored by SWIRL SEARCH for relevancy and highlighting purposes.

PLEASE STAR OUR REPO: http://swirl.today/

New Features

🔹 The new AdaptiveQueryProcessor rewrites simple NOT and -term queries to the format supported by each SearchProvider, as defined in the query_mappings.

        "query_mappings": "cx=7d473806dcdde5bc6,DATE_SORT=sort=date,PAGE=start=RESULT_INDEX,NOT_CHAR=-",

This indicates that the SearchProvider supports only the -term format. SWIRL rewrites a query like ...

elon musk NOT twitter

...to...

elon musk -twitter

This is noted in the Mixed results set under the SearchProvider block:

"Mergers & Acquisitions (web/Google PSE)": {
            "query_string_to_provider": "elon musk -twitter",
            "query_to_provider": "https://www.googleapis.com/customsearch/v1?cx=b384c4e79a5394479&key=AIzaSyDeB1y9l6OQW0dhVdZ9X_Xb2br_SK1K8YM&q=elon+musk+-twitter",
            "result_processor": "MappingResultProcessor",
        }
    ...etc...
},
        "search": {
            "query_string": "elon musk NOT twitter",
            "query_string_processed": "elon musk NOT twitter",
        },

🔹 The MappingResultsProcessor can map multiple result fields to a single SWIRL field using the | operator:

        "result_mappings": "body=content|description...",

This configures SWIRL to populate the body result field with the content and/or description if populated. If both are populated, the second description field is placed in the payload for clarity.

🔹 scripts/email_load.py has been included to make it easy to load the Enron email dataset into ElasticSearch

Changes

🔹 swirl_load.py has been moved to the SWIRL root directory

swirl-search% python swirl_load.py SearchProviders/google_pse.json -a admin -p some-admin-password
##S#W#I#R#L##1#.#6##############################################################

swirl_load.py: fed 3 into SWIRL, 0 errors

🔹 The former GenericResultProcessor has been renamed MappingResultProcessor

A new GenericResultProcessor now takes no option on results, allowing connectors that already produce the SWIRL format to save processing.

Issues Resolved

🔹 Need to highlight alternative word forms

"explain": {
                "stems": "agil oper",
                "body": {
                    "agile_operations_*": 0.7048520990053376,
                    "agile_24": 0.589995250602152,
                    "operations_10": 0.8256623948725578,
                    "operating_19": 0.6893337836077386
                }
            }

🔹 Highlighting of terms with 's

"body": "*Elon* *Musk’s* top lieutenant at Tesla ... is now working at SpaceX after leaving Tesla over a strange controversy. more… The post *Elon* *Musk* moves his top lieutenant at Tesla to SpaceX after a controversy appeared first on Electrek ."

🔹 Highlighting collisions

The updated CosineRelevancyProcessor should not allow this. Each term is highlighted once and only once. Please screen shot & report any examples to support.

🔹 swirl.py not working on some Ubuntu configs

swirl.py now checks to see if rabbitmq is running, and skips it if so:

sid@agentcooper swirl-search-master % python swirl.py start
##S#W#I#R#L##1#.#6##############################################################

Warning: rabbitmq appears to be running, skipping it:
  501 95899 95503   0  1:54PM ttys000    0:00.01 /bin/sh /usr/local/sbin/rabbitmq-server
Start: django -> daphne swirl_server.asgi:application ... Ok, pid: 95948
Start: celery-worker -> celery -A swirl_server worker --loglevel=info ... Ok, pid: 95963
Start: celery-beats -> celery -A swirl_server beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler ... Ok, pid: 95994
Updating .swirl... Ok

❤️ Thanks to all who reported this issue!!

Upgrading

For all platforms other than Docker, run the following from the command line, in the swirl installation folder:

./install.sh

Windows users should run install.bat instead!

🔑 If these scripts don't work for any reason, install manually:

pip install -r requirements.txt
python -m spacy download en_core_web_lg
python -m nltk.downloader stopwords
python -m nltk.downloader punkt

⚠️ Docker users need to restart their image to get the new version. Containers using sqlite3 for storage delete all content upon shut down! Read more: Docker Build for SWIRL

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch. Turn off Chrome prefetch. Turn off Safari prefetch

Please report any issues with this to support.

Documentation

• Quick Start
• User Guide

Support

🔹 Create an Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!

---

This discussion was created from the release SWIRL SEARCH 1.6.

[sidprobstein]

Please note: no new packages are required for 1.6; if you have 1.5 running, just:

git pull
python swirl.py setup
python swirl.py restart core

This will upgrade you to 1.6 and migrate the database.