Search Elastic

The elastic search module works alongside the Search Crawler to extract pages from the site into an elastic search instance to enable site wide search

The module registers itself to listen for the search crawler events and creates locale based indexes in the elastic repo for any pages it receives

When indexing the pages, if there is a primary data type then this will also be indexed against the page in primaryObject, this can then be used in query values as explained below

The indexing process is controlled by execution of the search crawler via the site automations

The module contains a Readme.txt file which gives help for the initial setup of elastic but the creation of the indexes/content are all automated.

Installation

Social stack cloud can now deploy a standalone elastic node to Digital Ocean, but the manual steps for local deployment are detailed below

The basic setup steps are

1. Install the following socialstack modules
   • Api/SearchCrawler
   • Api/SearchElastic
   • UI/SiteSearch
2. Mark any pages not to be indexed
3. Install Elastic Search (see Readme.txt in the repo for help and guidance)
4. Configure the sizing/scale in the elasticsearch.yml file (mainly vm memory usage)
5. Start Elastic Search for the first time and get the master password and certificate fingerprint
6. Add the url, credentials and fingerprint values to the module configuration via the ss admin panel
7. Run the automation to index the site, the indexing progress is shown via the console

The module also contains a postman collection which can be used to quick debug/verify the indexed content

Once indexed there is a UI/SiteSearch module which can be used as the basis of the site search

Social Stack Cloud Install Steps

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

sudo apt-get install apt-transport-https

echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

sudo apt-get update && sudo apt-get install elasticsearch

Remember to get the elastic user password from the output and update the ss configuration

"The generated password for the elastic built-in superuser is : xxxxxx"

Next get the ssl certificate fingerprint value for config

cd /etc/elasticsearch/certs openssl x509 -fingerprint -sha256 -in http_ca.crt

The section you need looks like this (the base64encoded values) SHA256 Fingerprint=85:AA:9E:19:2F:4C:72:01:01:66:41:87:28:03:52:B7:88:61:9C:93:E8:18:D2:9C:C7:60:92:F9:E0:84:EE:AA

Now elasticsearch is installed, before starting look at the memory and diskspace section below

Finnaly you can now start the elasticsearch service by executing

sudo systemctl start elasticsearch.service

These commands provide no feedback as to whether Elasticsearch was started successfully or not. Instead, this information will be written in the log files located in /var/log/elasticsearch/ or you can run the command below

systemctl status elasticsearch.service

To configure Elasticsearch to start automatically when the system boots up, run the following commands:

sudo /bin/systemctl daemon-reload

sudo /bin/systemctl enable elasticsearch.service

Configuring memory and diskspace

It is recommended to setup the vm with at least 4gb ram, when running 2gb we have run into issues

edit the config file to set diskspace limits

edit /etc/elasticsearch/elasticsearch.yml

cluster.routing.allocation.disk.threshold_enabled: false

cluster.routing.allocation.disk.watermark.flood_stage: 250mb

cluster.routing.allocation.disk.watermark.low: 2gb

cluster.routing.allocation.disk.watermark.high: 1gb

To disable geo location, if the project is not using the geolocation extensionsit can be disabled as otherwise an additional index is created and the contents automatically downloaded

ingest.geoip.downloader.enabled: false

Set the memory usage (default 4gb) but for dev we have set it to the minimum as below

edit /etc/elasticsearch/jvm.options

Uncomment and set these values

-Xms128m

-Xmx128m

To check the heap size

sudo cat /var/log/elasticsearch/elasticsearch.log | grep "heap size"

Admin

linux

sudo systemctl start elasticsearch.service

sudo systemctl stop elasticsearch.service

To get the fingerprint

cd /etc/elasticsearch/certs

openssl x509 -fingerprint -sha256 -in http_ca.crt

To reset the user password

/usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

Endpoints

GET /v1/sitesearch/reset

This removes any indexes from elastic to allow a clean start

POST /v1/sitesearch/query

This is the core search endpoint which currently accepts the following values

queryString

The query value entered by the user and/or the UI/SiteSearch and UI/ContentSearch components, elastic search uses the Lucene query language so you can build complex queries

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

e.g.

Africa AND primaryObject.name:Botswana

primaryObject.price:(>=1000 AND <5000)

((quick AND fox) OR (brown AND fox) OR fox) AND NOT dog

tags

This expects a list of comma separated tags, if any are supplied the query will only return results which contain all the passed tag values

contentTypes

This expects a list of comma separated content types, if any are supplied the query will only return results which contain all the passed content types, these are normally the primary data object for the page

pageSize

This expects the page size and will default to 10 if not populated

pageIndex

This expects the starting page number and will default to 1

aggregationFields

This expects a list of comma separated field names, if any are supplied then aggregations/facets are returned for the fields, a good example of this can be be found in UI/ContentSearch component

allFields

If this is set to true then any indexed content will be searchable, this includes any 'include' files and metadata from the primary object, if you want to restrict the search to the managed text fields then set this to false (the default)

Search taxonomy and filters/facets

You can filter or define facets(aggregations in ElasticSerach terminology) on any field exposed to Elastic from the social stack application, but they work best on metadata type fields such as tags, price etc

A basic example would be the tags field.

But a better solution is to setup contextual tags using the Api/Categories component and tags.

This allows you to give some context to a tag, so for example you could create an Animals category, then when adding a new tag for Cow or Horse you would set their category to be Animals

Any tags linked to a category can be referenced as taxonomy.xxxxx so in this example taxonomy.animals

These can then be referenced to show a contextual filter in the UI, the example below will pass back an aggregation of any animals tags in the result set, UI/ContentSearch then uses these render an auto generated filter. e.g.

{
	"c": [
		 {
			"t": "UI/ContentSearch",
			"d": {
				"aggregationFields": "taxonomy.animals",
				"contentTypes": "ItineraryPageDataType"
            }   
         } 
         ]
}

When tags are assigned to category and indexed, you will expose the core tag values and also some auto generated taxonomy fields.

An example of the indexed data in Elastic looks something like the example below where Africa and Botswana are plain tags but Horse has been linked to the Animals category

The Horse value is still exposed in the tags element as that is used for free text search ranking.

{
   "tags": [
           "Africa",
           "Botswana",
           "Horse"
   ],
   "taxonomy": {
      "animals": [
         "Horse"
      ]
   }
}

UI Components

UI/ContentSearch

This provides a good example with filtering and paging see the amba repo

The component will also allow you to override the default values/titles for any aggregation using the aggregationOverrides property, this can be useful to add grouping to values such as regions/countries etc

The buckets property is optional and by default would normally be an empty array [] but can be used to override labels and the order of options

Some basic examples of how overrides can be used to control the search filters

2 Column example with set items for destination

{
	"name": "taxonomy.destination",
	"label": "Destination",
	"columns": 2,
	"buckets": [
		{
			"key": "Botswana",
			"label": null
		},
		{
			"key": "Madagascar",
			"label": null
		},
		{
			"key": "Scotland",
			"label": "Bonny Scotland"
		},
		{
			"key": "Svalbard",
			"label": null
		}
	]
}

Swapping the default select boxes for radio buttons

{
	"name": "taxonomy.audience",
	"label": "Audience",
	"prompt": "",
	"inputType": "radio",
	"buckets": []
}

Forcing a boolean value to show as a single checkbox

{
	"name": "primaryObject.isFeatured",
	"label": "Only show featured events",
	"prompt": "",
	"inputType": "checkbox",
	"isBoolean": true,
	"buckets": []
}

Forcing a boolean value to return both checked and unchecked values

{
	"name": "primaryObject.isCompleted",
	"label": "Show completed events",
	"prompt": "",
	"inputType": "checkbox",
	"isBoolean": true,
	"returnAllValues": true,
	"buckets": []
}

Forcing a simple select dropdown

{
	"name": "primaryObject.country",
	"label": "Location",
	"all": "Any location",
	"prompt": "",
	"inputType": "select",
	"allowAllOptions": true,
	"buckets": []
}

Allow multiple unrelated values to be chosen

Override the default functionality which disables any original filter options if the search results no longer contain results containing the values, so a drill down work flow dog -> spaniel -> english

With 'allowAllOptions' other unrelated documents can be brought into the results so for example when filtering events you could include multiple unrelated types such as "gala" OR "conference" when by default if you choose "gala", conferences would be disabled

{
	"name": "taxonomy.event type",
	"label": "Event Type",
	"prompt": "",
	"inputType": "checkbox",
	"allowAllOptions": true,
	"buckets": []
}

A complete example is shown below using separate components for the filters and results (2 column layout)

{
	"c": [
		{
			"t": "UI/ContentSearch/SearchContext",
			"d": {
				"comment": "SearchContext used to link search engine, results and filters"
			},
			"c": {
			    
			"t": "UI/SiteContainer",
			"d": {},
			"c": {
				"t": "UI/SiteRow",
				"d": {},
				"c": [
					{
						"t": "UI/SiteColumn",
						"d": {
							"width": "sidebar"
						},
						"c": {
							"t": "UI/ContentSearch",
							"d": {
								"title": "Search insights",
								"aggregationFields": "taxonomy.brand,taxonomy.audience,taxonomy.insight category,tags",
								"contentTypes": "Insight",
								"resultLayout": "list",
								"itemRenderer": "UI/InsightResultItem",
								"sortOptions": [
									{
										"name": "Ranking",
										"field": "_score",
										"order": "desc"
									},
									{
										"name": "Name (asc)",
										"field": "title.keyword",
										"order": "asc"
									},
									{
										"name": "Name (desc)",
										"field": "title.keyword",
										"order": "desc"
									},
									{
										"name": "Date (asc)",
										"field": "primaryObject.editedUtc",
										"order": "asc"
									},
									{
										"name": "Date (desc)",
										"field": "primaryObject.editedUtc",
										"order": "desc"
									}
								],
								"aggregationOverrides": [
									{
										"name": "taxonomy.brand",
										"label": "View insights by brand",
										"all": "Both",
										"prompt": "",
										"inputType": "radio",
										"allowAllOptions": true,
										"isExpanded": true,
										"buckets": []
									},
									{
										"name": "taxonomy.audience",
										"label": "Audience",
										"all": "Any audience type",
										"prompt": "",
										"inputType": "radio",
										"allowAllOptions": true,
										"isExpanded": true,
										"buckets": []
									},
									{
										"name": "taxonomy.insight category",
										"label": "View by insight category",
										"all": "Any category",
										"prompt": "",
										"inputType": "checkbox",
										"allowAllOptions": true,
										"isExpanded": true,
										"buckets": []
									},
									{
										"name": "tags",
										"label": "Topics",
										"all": "Any topics",
										"prompt": "",
										"inputType": "select",
										"allowAllOptions": true,
										"buckets": []
									}
								],
								"hideResults": true,
								"hideTitle": true,
								"pageSize": "4"
							},
							"i": 4
						},
						"i": 6
					},
					{
						"t": "UI/SiteColumn",
						"d": {
							"width": "content"
						},
						"c": [
							{
								"t": "UI/ContentSearch/ContentSearchResults",
								"d": {
									"hideTotal": true,
									"searchType": "insights",
									"pageSize": "4"
								},
								"i": 14
							}
						],
						"i": 11
					}
				],
				"i": 12
			},
			"i": 13
			}
		}
	],
	"i": 3
}

Multi domain sites

There are a couple of services which can be used to extend/manage how content is indexed for multi domain sites

Search Domain Service

This service interacts with events during the indexing cycle to determine which indexes are used for queries and storage based on the SiteDomains service

The basic implementation stores each sub-site into a separate index based on the domain code

Search Brand Service

This service interacts with events during the indexing cycle to expose additional meta data to the indexed documents based on domains and taxonomy

This can be used when you wish to keep a single search index for the instance but want to be able to filter based on the domain/taxonomy

The basic implementation looks for tags with a category as defined in config, in this example 'brand' alongside the page domain to add additional meta data of metaData.brand.xxx or metadata.brand.yyy

You can also define mappings which check the individual fields within the primaryObject and if they match expose a defined value

In the example below we check for a boolean for Bigcorp and/or the city being London.

Sample config

{
  "IsEnabled": true,
  "Category":"brand",
  "MetaDataKey":"brand",
  "Mappings": [
        {
            "Field":"IsBigCorp",
            "Value":"true",
            "Brand":"bigcorp"
        },
        {
            "Field":"City",
            "Value":"london",
            "Brand":"UK"
        }
    ]
}