Search Elastic
The elastic search module works alongside the Search Crawler to extract pages from the site into an elastic search instance to enable site wide search
The module registers itself to listen for the search crawler events and creates locale based indexes in the elastic repo for any pages it receives
When indexing the pages, if there is a primary data type then this will also be indexed against the page in primaryObject, this can then be used in query values as explained below
The indexing process is controlled by execution of the search crawler via the site automations
The module contains a Readme.txt file which gives help for the initial setup of elastic but the creation of the indexes/content are all automated.
Installation
The basic setup steps are
- Install the following socialstack modules
- Api/SearchCrawler
- Api/SearchElastic
- UI/SiteSearch
- Mark any pages not to be indexed
- Install Elastic Search (see Readme.txt in the repo for help and guidance)
- Configure the sizing/scale in the elasticsearch.yml file (mainly vm memory usage)
- Start Elastic Search for the first time and get the master password and certificate fingerprint
- Add the url, credentials and fingerprint values to the module configuration via the ss admin panel
- Run the automation to index the site, the indexing progress is shown via the console
The module also contains a postman collection which can be used to quick debug/verify the indexed content
Once indexed there is a UI/SiteSearch module which can be used as the basis of the site search
Social Stack Cloud Install Steps
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
sudo apt-get install apt-transport-https
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt-get update && sudo apt-get install elasticsearch
Remember to get the elastic user password from the output and update the ss configuration
"The generated password for the elastic built-in superuser is : xxxxxx"
Next get the ssl certificate fingerprint value for config
cd /etc/elasticsearch/certs
openssl x509 -fingerprint -sha256 -in http_ca.crt
The section you need looks like this (the base64encoded values) SHA256 Fingerprint=85:AA:9E:19:2F:4C:72:01:01:66:41:87:28:03:52:B7:88:61:9C:93:E8:18:D2:9C:C7:60:92:F9:E0:84:EE:AA
Now elasticsearch is installed, before starting look at the memory and diskspace section below
Finnaly you can now start the elasticsearch service by executing
sudo systemctl start elasticsearch.service
These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/ or you can run the command below
systemctl status elasticsearch.service
To configure Elasticsearch to start automatically when the system boots up, run the following commands:
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service
Configuring memory and diskspace
edit the config file to set diskspace limits
edit /etc/elasticsearch/elasticsearch.yml
cluster.routing.allocation.disk.threshold_enabled: false
cluster.routing.allocation.disk.watermark.flood_stage: 250mb
cluster.routing.allocation.disk.watermark.low: 2gb
cluster.routing.allocation.disk.watermark.high: 1gb
Set the memory usage (default 4gb)
edit /etc/elasticsearch/jvm.options
Uncomment and set these values
-Xms128m
-Xmx128m
Admin
linux
sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service
sudo systemctl status elasticsearch.service
To get the fingerprint
cd /etc/elasticsearch/certs
openssl x509 -fingerprint -sha256 -in http_ca.crt
Endpoints
GET /v1/sitesearch/reset
This removes any indexes from elastic to allow a clean start
POST /v1/sitesearch/query
This is the core search endpoint which currently accepts the following values
- queryString
- The query value entered by the user and/or the UI/SiteSearch and UI/ContentSearch components, elastic search uses the Lucene query language so you can build complex queries
- https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
- e.g.
Africa AND primaryObject.name:BotswanaprimaryObject.price:(>=1000 AND <5000)((quick AND fox) OR (brown AND fox) OR fox) AND NOT dog
- tags
- This expects a list of comma separated tags, if any are supplied the query will only return results which contain all the passed tag values
- contentTypes
- This expects a list of comma separated content types, if any are supplied the query will only return results which contain all the passed content types, these are normally the primary data object for the page
- pageSize
- This expects the page size and will default to 10 if not populated
- pageIndex
- This expects the starting page number and will default to 1
- aggregationFields
- This expects a list of comma separated field names, if any are supplied then aggregations/facets are returned for the fields, a good example of this can be be found in UI/ContentSearch component
Search taxonomy and filters/facets
You can filter or define facets(aggregations in ElasticSerach terminology) on any field exposed to Elastic from the social stack application, but they work best on metadata type fields such as tags, price etc
A basic example would be the tags field.
But a better solution is to setup contextual tags using the Api/Categories component and tags.
This allows you to give some context to a tag, so for example you could create an Animals category, then when adding a new tag for Cow or Horse you would set their category to be Animals
Any tags linked to a category can be referenced as taxonomy.xxxxx so in this example taxonomy.animals
These can then be referenced to show a contextual filter in the UI, the example below will pass back an aggregation of any animals tags in the result set, UI/ContentSearch then uses these render an auto generated filter. e.g.
{
"c": [
{
"t": "UI/ContentSearch",
"d": {
"aggregationFields": "taxonomy.animals",
"contentTypes": "ItineraryPageDataType"
}
}
]
}
When tags are assigned to category and indexed, you will expose the core tag values and also some auto generated taxonomy fields.
An example of the indexed data in Elastic looks something like the example below where Africa and Botswana are plain tags but Horse has been linked to the Animals category
The Horse value is still exposed in the tags element as that is used for free text search ranking.
{
"tags": [
"Africa",
"Botswana",
"Horse"
],
"taxonomy": {
"animals": [
"Horse"
]
}
}
UI Components
- UI/ContentSearch
This provides a good example with filtering and paging
The component will also allow you to override the default values/titles for any aggregation using the aggregationOverrides property, this can be useful to add grouping to values such as regions/countries etc
An example is shown below
{
"c": [
{
"t": "UI/ContentSearch",
"d": {
"aggregationFields": "taxonomy.price,taxonomy.destination,taxonomy.wildlife",
"contentTypes": "ItineraryPageDataType",
"aggregationOverrides": [
{
"name": "taxonomy.price",
"label": "Price",
"buckets": [
{
"label": "£3500",
"key": "3500"
},
{
"label": "£5000",
"key": "5000"
},
{
"label": "£7500",
"key": "7500"
}
]
},
{
"name": "taxonomy.destination",
"label": "Destination",
"columns": 3,
"groups": [
{
"name": "Africa and the Indian Ocean",
"buckets": [
{
"label": "Botswana",
"key": "Botswana"
},
{
"label": "Central African Republic",
"key": "Central African Republic"
},
{
"label": "Chad",
"key": "Chad"
},
{
"label": "Congo",
"key": "Congo"
},
{
"label": "Djibouti",
"key": "Djibouti"
},
{
"label": "Gabon",
"key": "Gabon"
},
{
"label": "Indian Ocean",
"key": "Indian Ocean"
},
{
"label": "Kenya",
"key": "Kenya"
},
{
"label": "Madagascar",
"key": "Madagascar"
},
{
"label": "Mozambique",
"key": "Mozambique"
},
{
"label": "Namibia",
"key": "Namibia"
},
{
"label": "Rwanda",
"key": "Rwanda"
},
{
"label": "South Africa",
"key": "South Africa"
},
{
"label": "Tanzania",
"key": "Tanzania"
},
{
"label": "Uganda",
"key": "Uganda"
},
{
"label": "Zambia",
"key": "Zambia"
},
{
"label": "Zimbabwe",
"key": "Zimbabwe"
}
]
},
{
"name": "Asia",
"buckets": [
{
"label": "Borneo",
"key": "Borneo"
},
{
"label": "India",
"key": "India"
},
{
"label": "Russia",
"key": "Russia"
},
{
"label": "Sri Lanka",
"key": "Sri Lanka"
}
]
},
{
"name": "Latin America",
"buckets": [
{
"label": "Argentina",
"key": "Argentina"
},
{
"label": "Belize",
"key": "Belize"
},
{
"label": "Brazil",
"key": "Brazil"
},
{
"label": "Chile",
"key": "Chile"
},
{
"label": "Colombia",
"key": "Colombia"
},
{
"label": "Costa Rica",
"key": "Costa Rica"
},
{
"label": "Ecuador",
"key": "Ecuador"
},
{
"label": "Galapagos",
"key": "Galapagos"
},
{
"label": "Peru",
"key": "Peru"
}
]
},
{
"name": "North America",
"buckets": [
{
"label": "Alaska",
"key": "Alaska"
},
{
"label": "Canada",
"key": "Canada"
},
{
"label": "Dominica",
"key": "Dominica"
}
]
},
{
"name": "Polar Regions",
"buckets": [
{
"label": "Antarctica",
"key": "Antarctica"
},
{
"label": "Arctic Canada",
"key": "Arctic Canada"
},
{
"label": "Greenland",
"key": "Greenland"
},
{
"label": "Norway",
"key": "Norway"
},
{
"label": "Svalbard",
"key": "Svalbard"
},
{
"label": "Arctic Russia",
"key": "Arctic Russia"
}
]
},
{
"name": "Australasia",
"buckets": [
{
"label": "Australia",
"key": "Australia"
}
]
},
{
"name": "Europe",
"buckets": [
{
"label": "Scotland",
"groupName": "Europe",
"key": "Scotland"
}
]
}
]
},
{
"name": "taxonomy.wildlife",
"label": "Wildlife",
"columns": 4,
"groups": [
{
"name": "Bears",
"buckets": [
{
"label": "Grizzly Bears",
"key": "Grizzly Bears"
},
{
"label": "Polar Bear",
"key": "Polar Bear"
},
{
"label": "Spirit Bear",
"key": "Spirit Bear"
},
{
"label": "Sun Bear",
"key": "Sun Bear"
}
]
},
{
"name": "Big Cats",
"buckets": [
{
"label": "African Leopard",
"key": "African Leopard"
},
{
"label": "African Lion",
"key": "African Lion"
},
{
"label": "Asiatic Leopard",
"key": "Asiatic Leopard"
},
{
"label": "Bengal Tiger",
"key": "Bengal Tiger"
},
{
"label": "Cheetah",
"key": "Cheetah"
},
{
"label": "Jaguar",
"key": "Jaguar"
},
{
"label": "Puma",
"key": "Puma"
},
{
"label": "Siberian Tiger",
"key": "Siberian Tiger"
},
{
"label": "Snow Leopard",
"key": "Snow Leopard"
}
]
},
{
"name": "Big Five",
"buckets": [
{
"label": "African Elephant",
"key": "African Elephant"
},
{
"label": "Black Rhinoceros",
"key": "Black Rhinoceros"
},
{
"label": "Cape Buffalo",
"key": "Cape Buffalo"
}
]
},
{
"name": "Primates",
"buckets": [
{
"label": "Chimpanzee",
"key": "Chimpanzee"
},
{
"label": "Lemurs",
"key": "Lemurs"
},
{
"label": "Mountain Gorilla",
"key": "Mountain Gorilla"
},
{
"label": "Orangutan",
"key": "Orangutan"
},
{
"label": "Western Lowland Gorilla",
"key": "Western Lowland Gorilla"
}
]
},
{
"name": "Whales",
"buckets": [
{
"label": "Beluga Whale",
"key": "Beluga Whale"
},
{
"label": "Blue Whale",
"key": "Blue Whale"
},
{
"label": "Killer Whale",
"key": "Killer Whale"
},
{
"label": "Narwhal",
"key": "Narwhal"
},
{
"label": "Sperm Whales",
"key": "Sperm Whales"
},
{
"label": "Whale Shark",
"key": "Whale Shark"
}
]
}
]
}
]
},
"i": 4
},
{
"t": "p",
"c": "",
"d": {},
"i": 2
}
],
"i": 3
}
