HawkSearch WebsiteHawkSearch DashboardHawkSearch User Guide
Guides

Visual Search & Image Search Implementation Guide

Suggest Edits

Overview

Visual and Image Search is a sophisticated feature within HawkSearch that revolutionizes how users find what they're looking for by utilizing visual cues instead of traditional text-based queries. This powerful tool is divided into two distinct sections: "Visual Search - Search by Image" and "Image Search - Search with Description of Image," each designed to cater to different user needs and preferences for a more intuitive and effective search experience.

  1. Visual Search (Search by Image): Users can upload an image, and the system retrieves visually similar results.
  2. Image Search (Search with Description): Users can describe an image using text, and the system identifies and returns the most relevant images based on the description.

Both approaches utilize the same backend configuration but cater to different search behaviors, making search experience more intuitive and efficient.

Step-by-Step Guide to Configuring Visual and Image Search

Prerequisites

Before configuring Visual and Image Search, ensure the following:

  1. The Visual and Image Search feature must be activated in your account by our internal team.

  2. To request activation, contact our CSD team via email or submit an interest form within the Workbench.

  3. Once activated, you can proceed with configuration.

Configuring Visual and Image Search Fields in Workbench

After enabling Visual and Image Search, follow these steps to configure the fields for optimal search performance:

Step 1 - Access Visual and Image Search Admin

  1. Please go to the Workbench
  2. Please click on the Visual Search in the left navigation

Step 2 - Search for Fields

  1. The first search field allows you to search for the name of an existing Field. If you know the name as it appears in the data feed, or a keyword from the name, type it into the "Field Name"
  2. Search a field based on the type of Field by selecting an option in the drop down.
    1. Field values indexed "as is" AND are stemmed
    2. Field values are NOT stemmed
      Field values are ONLY stemmed (search only)
    3. Stored only, not used for search nor facets
  3. Search a field based on the Data Type by selecting an option in the drop down.
    1. Boolean
    2. Datetime
    3. GeoPoint
    4. Numeric
    5. Text
  4. You can also conduct a search based on the label of the Field. This name may differ from the corresponding field in the data field.
  5. To search for a Field by any tags associated with it, enter the tags here.
  6. Once you have added all search criteria, click the "Search" button to see your results.
  7. You can also clear the search criteria to see all the fields.

Step 3 - Choose the Right Fields for Visual and Image Search

Visual and Image Search relies on image URLs to generate embeddings. Selecting the correct field ensures accurate and efficient search functionality. Currently, only one field can be designated for storing image URLs.

📘

Tips

Ensure that the selected field contains valid, high-resolution image URLs to maximize search accuracy and performance.

Suggested Fields to Get Started

Image URL: This field should contain the URLs of the primary images for products or content. It serves as the foundation for both Visual Search (uploading an image) and Image Search (describing an image using text).

Unsure which fields contain the image URL?

No worries! You can check by following these steps:


  1. Navigate to Preview
  2. Select Default Search as Preview Type
  3. From any item in the search results, click on the Score tag
  4. In the popup, click on Explain
  5. Another popup for Item Information will appear, click on Info & Field Matches
  6. Review the fields and values listed to see which contain the image URL for Visual and Image Search
    1. For example, in the 2nd screenshot, the image URL is contain in the image_url field

Step 4 - Rebuild the Index

After configuring the fields, a crucial step is to rebuild the full index. This process allows the system to create the necessary embeddings for the configured field, ensuring that the Visual and Image Search can function as intended. Rebuilding the full index integrates all the setup changes and prepares the system for executing visual and image searches. Please note that this process involves creating a new index and updating all related components, as detailed in the provided documentation.

Step 5 - Validate the Results

Once the index rebuild is complete, let's validate that Visual and Image Search is working as expected.

  1. Navigate to Preview in the Workbench.

  2. Select Smart Search as the Preview Type

  3. Turn on Visual Search

  4. Upload an image or enter test queries related to your dataset.

  5. Review the search results to confirm that Visual and Image Search is returning relevant content.

📘

Tips

Guidelines for Testing Visual and Image Search

  1. For Visual Search (Search by Image): Upload multiple images of the same object from different angles to evaluate retrieval accuracy.
  2. For Image Search (Search with Description): Use detailed and precise textual descriptions to test how well the system matches images to search intent.
  3. Compare with Traditional Search: Run a standard keyword-based search for the same content to gauge the improvement in accuracy and relevance.
  4. Visual and Image Search are designed to interpret visual and descriptive patterns rather than exact keyword matches. If results are not as expected, consider ensuring high-quality images are used in the index. If you need more fine-tuning for custom adjustments, feel free to reach out to our CSD team—we’re happy to help you get the best results for your specific needs!

Step 6 - Making Search Request

Once Visual and Image Search is configured and validated, you can start making search requests using the Visual and Image Search request type through the Search API. This enables seamless integration with your search solution, leveraging Visual and Image Search for improved query understanding and result relevance. For more details please refer to our Search API documentation.

Example Request - Visual Search:

For this method, you will need to upload an image to serve as your search query. It's important to note that the image must be encoded to base64 format before sending the request through the "ImageData" field. The system then analyzes this image to identify and return items that are visually similar to the content depicted in the image.

{
    "ClientGuid": "xxxxx",
    "RequestType":"ImageSearch",
    "kValue":3,
    "ImageData": "data:image/png;base64,BASE64 DATA HERE"
}

Example Request - Image Search:

Alternatively, you can describe an image in text within the "Keyword" field when the "RequestType" is set to "ImageSearch". This option is useful when you do not have an image but can provide a detailed description of what you’re searching for. The system interprets your description to find matching or relevant items.

{
    "ClientGuid": "xxxxx",
    "RequestType":"ImageSearch",
    "kValue":3,
    "Keyword": "Show me the picture with a woman climbing a cliff"
}

Troubleshooting Common Issues

IssuePossible CauseSolution
Visual and Image Search is not returning relevant resultsThe selected field may not contain valid image URLsVerify that the selected field includes valid image URLs
Some expected results are missing or Visual/Image Search doesn't seem to be working after being enabledIndex rebuild may not have been completed successfully, or Visual/Image Search might not be correctly configuredPerform another full index rebuild, verify field selection, and ensure that Search is properly configured
Feature is not available in WorkbenchVisual/Image Search has not been enabledContact the CSD Team to activate the feature
API request is failing or returning errorsThe request may not be properly formattedEnsure that the JSON structure is correct and that required fields such as ClientGuid and correct RequestType are included
Visual Search request is not processing imagesImage data may not be properly encoded in base64Confirm that the ImageData field contains correctly formatted base64 image data

If issues persist after following these solutions, please reach out to our support team for further assistance.


Front-End Implementation

To use Visual and Image Search on your frontend, you can either integrate it using our Search API or take advantage of our prebuilt Rapid UI framework to streamline implementation.

Rapid UI

If you are using Rapid UI to build your frontend, you can add Visual Search and Image Search by simply including the following component:

  1. Visual Search 
    <div class="visualsearch-field">
    <div class="visualsearch-field__drop-area" hawksearch-visualsearch-droparea>
        <div class="visualsearch-field__drop-area__image-display" hawksearch-visualsearch-display></div>
        <p class="visualsearch-field__drop-area__prompt-msg">
            {{strings.dragImageMessage}}
            <label class="visualsearch-field__drop-area__upload-button">
                <input hawksearch-visualsearch-fileinput type="file" accept="image/*" capture="environment" />
                {{strings.uploadImageMessage}}
            </label>
        </p>
        <p class="visualsearch-field__drop-area__drop-msg">{{strings.dropImageMessage}}</p>
    </div>
</div>
  2. Image Search 
    <div class="search-field">
    <input type="text" hawksearch-input hawksearch-imagesearch-field value="{{query}}" placeholder="{{strings.placeholder}}" />
</div>

Steps to Implement in Rapid UI

  1. Ensure you are using Rapid UI in your frontend setup.

  2. Check your Rapid UI version: This feature is only available in Rapid UI 6.1.0 or newer. If you are on an older version, please update to 6.1.0.

  3. Verify your HawkSearch configuration by confirming that your clientId is correctly set and updated (to avoid uisng ourdated or incorrect values)

    1. ClientId can be located at: Admin > Account Info > Setup Info > Client Guid / Tracking Key

  4. Verify your API endpoint configuration

    1. Ensure that your endpoint is pointing to the correct environment (e.g., production instead of development)

  5. Add the Concept Search component in the appropriate section of your UI.

    1. Visual Search 
      <div class="visualsearch-field">
    <div class="visualsearch-field__drop-area" hawksearch-visualsearch-droparea>
        <div class="visualsearch-field__drop-area__image-display" hawksearch-visualsearch-display></div>
        <p class="visualsearch-field__drop-area__prompt-msg">
            {{strings.dragImageMessage}}
            <label class="visualsearch-field__drop-area__upload-button">
                <input hawksearch-visualsearch-fileinput type="file" accept="image/*" capture="environment" />
                {{strings.uploadImageMessage}}
            </label>
        </p>
        <p class="visualsearch-field__drop-area__drop-msg">{{strings.dropImageMessage}}</p>
    </div>
</div>
    2. Image Search 
      <div class="search-field">
    <input type="text" hawksearch-input hawksearch-imagesearch-field value="{{query}}" placeholder="{{strings.placeholder}}" />
</div>

    Code Example

  6. <html lang="en">
  <head>
    <title>Rapid UI Smart Search Example</title>
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <!-- CUSTOM STYLES - ADDED FOR DEMO PURPOSE -->
    <style type="text/css">
      body{font-family:Arial,Helvetica,sans-serif;font-size:16px;}
      .smartsearchdemo__container {margin: 0 auto;min-width: 340px;max-width: 1280px;}
      .smartsearchdemo__container .search-field__container,.smartsearchdemo__container .conceptsearch-field__container {display: flex;justify-content: space-around;gap: 12px;}
      .smartsearchdemo__container .search-field__bar{display: flex;justify-content: space-around;background-color: #f5f7fa;padding: 0px 16px;align-items: center;flex: 1 0 0;border-radius: 100px;border: 1px solid #e6e9ed;transition: all .25s ease-in-out;}
      .smartsearchdemo__container .search-field__bar:focus,.smartsearchdemo__container .search-field__bar:hover,.smartsearchdemo__container .search-field__bar:focus-within,.smartsearchdemo__container .search-field__bar:target,.smartsearchdemo__container .search-field__bar:focus-visible,
      .smartsearchdemo__container .search-field__actions .search-field__actions__button:focus,.smartsearchdemo__container .search-field__actions .search-field__actions__button:hover,.smartsearchdemo__container .search-field__actions .search-field__actions-button:focus-within,
      .smartsearchdemo__container .search-field__actions .search-field__actions__button:target,.smartsearchdemo__container .search-field__actions .search-field__actions__button:focus-visible {border: 1px solid #119FF7;background: #FFF;}
      .smartsearchdemo__container .search-field__bar .search-field__bar__icon {display: flex;align-items: center;justify-content: center;width: 40px;height: 40px;margin: 0;max-width: 7.5rem;vertical-align: top;}
      .smartsearchdemo__container .search-field__bar .search-field__bar__link {color: #656D78;padding: 16px 24px;cursor: pointer;}
      .smartsearchdemo__container .search-field__bar .search-field__bar__divider{display: flex;border-left: 1px solid #e6e9ed;height: 100%;}
      .smartsearchdemo__container .search-field__actions .search-field__actions__button {display: flex;padding: 16px;align-items: center;align-self: stretch;border-radius: 100px;border: 1px solid #e6e9ed;background: #f5f7fa;transition: all .25s ease-in-out;cursor: pointer;}
      .smartsearchdemo__container .search-field__actions .search-field__actions__icon {width: 20px;height: 20px;}
      .smartsearchdemo__container .visualsearch-field__dialog {display: flex;position: relative;box-sizing: border-box;max-width: 1280px;padding: 32px;flex-direction: column;justify-content: center;align-items: flex-start;gap: 32px;border-radius: 25px;border: 1px solid #119ff7;background: #fff;}
      .smartsearchdemo__container .visualsearch-field__dialog .visualsearch-field__dialog__title {display: flex;justify-content: space-between;width: 100%;}
      .smartsearchdemo__container .visualsearch-field__dialog .visualsearch-field__dialog__title > h2 {display: block;margin: 0;color: #353b48 !important;font-size: 18px;font-style: normal;font-weight: 700;line-height: 100%;letter-spacing: .2px;}
      .smartsearchdemo__container .visualsearch-field__dialog .visualsearch-field__dialog__body {display: flex;width: 100%;flex-direction: column;}
      .smartsearchdemo__container .visualsearch-field__close {display: flex;align-items: center;justify-content: center;width: 56px;height: 56px;margin: 0;position: absolute;top: 1px;right: 1px;box-sizing: border-box;vertical-align: top;border: none;background: #fff;border-radius: 50%;cursor: pointer;}
      .smartsearchdemo__container .visualsearch-field__close:hover {background-color: #F5F7FA }
    </style>
    <!-- CUSTOM JAVASCRIPT TO HANDLE TABS - ADDED FOR DEMO PURPOSE -->
    <script>function getRequestTypeIndex(e){const t=e.get("requestType");return t?"ConceptSearch"===t?1:"ImageSearch"===t?2:0:0}document.addEventListener("DOMContentLoaded",(function(){let e=getRequestTypeIndex(new URL(document.location).searchParams)||0;function t(t){if(t.preventDefault(),"keypress"===t.type&&13===t.which||"click"===t.type){let t=parseInt(this.dataset.tabIndex);t!==e&&document.querySelectorAll(".smartsearchdemo__container .tab-container").forEach((function(n){if(parseInt(n.dataset.tabIndex)===t)return document.querySelectorAll(".tab-container").forEach((function(e){e.style.display="none"})),n.style.display="block",void(e=parseInt(n.dataset.tabIndex))}))}}document.querySelectorAll(".tab-container").forEach((function(t){parseInt(t.dataset.tabIndex)===e?t.style.display="block":t.style.display="none"})),document.querySelectorAll(".tab-switcher").forEach((function(e){e.addEventListener("click",t),e.addEventListener("keypress",t)}))}));</script>
    <!-- Rapid UI LIBRARY -->
    <script type="module" src="https://cdn.jsdelivr.net/npm/@bridgeline-digital/[email protected]/dist/hawksearch-handlebars-ui.min.js"></script>
    <!-- HAWKSEARCH CONFIGURATION -->
    <script type="text/javascript">
      addEventListener('hawksearch:loaded', () => {
        HawkSearch.init({
            clientId: "59d7b9ce967d4fdf8262d1a49392b8be",
            components: {
              'search-field': {
                strings: {
                  placeholder: 'Search by Keyword'
                }
              },
              'conceptsearch-field': {
                strings: {
                  placeholder: 'Search by a Concept'
                }
              },
              'imagesearch-field': {
                strings: {
                  placeholder: 'Describe an image'
                }
              },
              'visualsearch-field': {
                template: 'visualsearch-field__template',
                strings: {
                  dragImageMessage: 'Search By Image',
                  dropImageMessage: 'Drop an image here'
                }
              },
              'search-results-item': {
                  template: 'search-results-item__template'
              }
            },
            search: {
              url: window.location.pathname,
              endpointUrl: "https://searchapi-dev.hawksearch.net"
            },
            urlPrefixes: {
              assets: "https://dev.hawksearch.net",
              content: "https://preview-dev.hawksearch.net/developerportalsandbox",
            },
            css: {
              customStyles: 'custom-styles__template'
            },
            debug: true
        });
      });
    </script>
  </head>
  <body>
    <div class="smartsearchdemo__container">
      <!-- CSS TEMPLATE USED TO CUSTOMIZE HANDLEBAR-UI NATIVE STYLES -->
      <template id="custom-styles__template">
        <style type="text/css">
          input[type="text"]:not(.facet__search) {border: none;background-color: transparent;}
          input[type="text"]:not(.facet__search):focus,
          input[type="text"]:not(.facet__search):hover,
          input[type="text"]:not(.facet__search):focus-within,
          input[type="text"]:not(.facet__search):target,
          input[type="text"]:not(.facet__search):focus-visible {outline: none;border: none;background-color: transparent;}
          .visualsearch-field__drop-area {display:flex;padding:40px 24px;flex-direction:column;align-items:center;gap: 24px;align-self:stretch;border-radius:8px;border:1px solid #e6e9ed;}
          .visualsearch-field__drop-area.dragging-state {border: 1px dashed #119ff7;background: #f2faff;}
          .visualsearch-field__drop-area.dragging-state > .visualsearch-field__drop-area__prompt-msg,
          .visualsearch-field__drop-area.dragging-state > .visualsearch-field__drop-area__upload-button,
          .visualsearch-field__drop-area.dragging-state > .visualsearch-field__drop-area__icon {pointer-events: none;visibility: hidden;}
          .visualsearch-field__drop-area__upload-button {display:flex;padding:12px 40px 12px 32px;align-items:center;gap:10px;color:#fff !important;background-color:#000;cursor:pointer;}
          .visualsearch-field__drop-area__prompt-msg {color:#000;text-align:center;font-size:18px;font-style:normal;font-weight:700;line-height:100%;letter-spacing:.2px;}
          .visualsearch-field__drop-area__drop-msg {pointer-events: none;color: #119ff7 !important;text-align:center;font-size:18px;font-style:normal;font-weight:700;line-height:100%;letter-spacing:.2px;}
        </style>
      </template>
    
      <!-- CUSTOM TEMPLATE TO MODIFY THE VISUALS OF <visualsearch-field /> COMPONENT -->
      <template id="visualsearch-field__template">
        <div class="visualsearch-field">
          <div class="visualsearch-field__drop-area" hawksearch-visualsearch-droparea>
            <hawksearch-icon name="drop-image" size="100"></hawksearch-icon>
            <p class="visualsearch-field__drop-area__prompt-msg">{{strings.dragImageMessage}}</p>
            <p class="visualsearch-field__drop-area__drop-msg">{{strings.dropImageMessage}}</p>
            <label class="visualsearch-field__drop-area__upload-button">
              <input hawksearch-visualsearch-fileinput type="file" accept="image/*" capture="environment">
              <hawksearch-icon name="upload" size="20"></hawksearch-icon>
              {{strings.uploadImageMessage}}
            </label>
            <div class="visualsearch-field__drop-area__image-display" hawksearch-visualsearch-display></div>
          </div>
        </div>
      </template>
      
      <!-- CUSTOM TEMPLATE TO MODIFY THE VISUALS OF <search-results-item__template-field /> COMPONENT -->
      <template id="search-results-item__template">
        <div class="search-results-list__item search-results-list__item--{{type}}">
            {{#if pinned}}
            <span class="search-results-list__item__pin">
                <hawksearch-icon name="star"></hawksearch-icon>
            </span>
            {{/if}}
            {{#if (lt salePrice price)}}
            <span class="search-results-list__item__sale-indicator">{{strings.sale}}</span>
            {{/if}}
            <a hawksearch-link href="{{url}}" class="search-results-list__item__image" aria-label="{{title}}">
                <img hawksearch-image src="{{imageUrl}}" alt="">
            </a>
            <div class="search-results-list__item__title">
                <a hawksearch-link href="{{url}}">{{title}}</a>
            </div>
            {{#if rating}}
            <hawksearch-rating rating="{{rating}}"></hawksearch-rating>
            {{/if}}
            {{#unless (eq price undefined)}}
            <div class="search-results-list__item__price" itemprop="offers" itemtype="http://schema.org/Offer" itemscope>
                {{#if (lt salePrice price)}}
                <span class="search-results-list__item__price__original">{{currency price}}</span>
                <span class="search-results-list__item__price__current" itemprop="price">{{currency salePrice}}</span>
                {{else}}
                <span class="search-results-list__item__price-__current" itemprop="price">{{currency price}}</span>
                {{/if}}
            </div>
            {{/unless}}
            <hawksearch-variant-selector></hawksearch-variant-selector>
        </div>
      </template>
      
      <!-- KEYWORD SEARCH TAB -->
      <div class="tab-container" data-tab-index="0">
        <div class="search-field__container">
          <div class="search-field__bar">
            <span class="search-field__bar__icon">
              <hawksearch-icon name="search" size="20"></hawksearch-icon>
            </span>
            <hawksearch-search-field style="flex: 1;"></hawksearch-search-field>
            <div class="search-field__bar__divider"></div>
            <a class="search-field__bar__link tab-switcher" data-tab-index="1">or Concepts</a>
          </div>
          <div class="search-field__actions">
            <a class="search-field__actions__button tab-switcher" data-tab-index="2">
              <hawksearch-icon name="image" size="20"></hawksearch-icon>
            </a>
          </div>
        </div>
      </div>

      <!-- CONCEPT SEARCH TAB -->
      <div class="tab-container" data-tab-index="1">
        <div class="search-field__container">
          <div class="search-field__bar">
            <span class="search-field__bar__icon">
              <hawksearch-icon name="search" size="20"></hawksearch-icon>
            </span>
            <hawksearch-conceptsearch-field style="flex: 1;"></hawksearch-conceptsearch-field>
            <div class="search-field__bar__divider"></div>
            <a class="search-field__bar__link tab-switcher" data-tab-index="0">or Keywords</a>
          </div>
          <div class="search-field__actions">
            <a class="search-field__actions__button tab-switcher" data-tab-index="2">
              <hawksearch-icon name="image" size="20"></hawksearch-icon>
            </a>
          </div>
        </div>
      </div>

      <!-- VISUAL SEARCH TAB -->
      <div class="tab-container" data-tab-index="2">
        <div class="visualsearch-field__dialog">
          <a class="visualsearch-field__close tab-switcher" data-tab-index="1" tabindex="0">
            <hawksearch-icon name="cross" size="20"></hawksearch-icon>
          </a>
          <div class="visualsearch-field__dialog__title">
            <h2>Visual Search</h2>
          </div>
          <div class="visualsearch-field__dialog__body">
            <hawksearch-visualsearch-field></hawksearch-visualsearch-field>
            <div class="search-field__bar">
              <span class="search-field__bar__icon">
                <hawksearch-icon name="search" size="20"></hawksearch-icon>
              </span>
              <hawksearch-imagesearch-field style="flex: 1;"></hawksearch-imagesearch-field>
            </div>
          </div>
        </div>
      </div>
      <div style="margin-top: 16px;">
        <hawksearch-search-results></hawksearch-search-results>
      </div>
    </div>
  </body>
</html>

Updated about 1 month ago