Recherche avancée

Sur d’autres sites (7198)

• What is Behavioural Segmentation and Why is it Important ?

  28 septembre 2023, par Erin — Analytics Tips

  Amidst the dynamic landscape of web analytics, understanding customers has grown increasingly vital for businesses to thrive. While traditional demographic-focused strategies possess merit, they need to uncover the nuanced intricacies of individual online behaviours and preferences. As customer expectations evolve in the digital realm, enterprises must recalibrate their approaches to remain relevant and cultivate enduring digital relationships.

  In this context, the surge of technology and advanced data analysis ushers in a marketing revolution : behavioural segmentation. Businesses can unearth invaluable insights by meticulously scrutinising user actions, preferences and online interactions. These insights lay the foundation for precisely honed, high-performing, personalised campaigns. The era dominated by blanket, catch-all marketing strategies is yielding to an era of surgical precision and tailored engagement. 

  While the insights from user behaviours empower businesses to optimise customer experiences, it’s essential to strike a delicate balance between personalisation and respecting user privacy. Ethical use of behavioural data ensures that the power of segmentation is wielded responsibly and in compliance, safeguarding user trust while enabling businesses to thrive in the digital age.

  What is behavioural segmentation ?

  Behavioural segmentation is a crucial concept in web analytics and marketing. It involves categorising individuals or groups of users based on their online behaviour, actions and interactions with a website. This segmentation method focuses on understanding how users engage with a website, their preferences and their responses to various stimuli. Behavioural segmentation classifies users into distinct segments based on their online activities, such as the pages they visit, the products they view, the actions they take and the time they spend on a site.

  Behavioural segmentation plays a pivotal role in web analytics for several reasons :

  1. Enhanced personalisation :

  Understanding user behaviour enables businesses to personalise online experiences. This aids with delivering tailored content and recommendations to boost conversion, customer loyalty and customer satisfaction.

  2. Improved user experience :

  Behavioural segmentation optimises user interfaces (UI) and navigation by identifying user paths and pain points, enhancing the level of engagement and retention.

  3. Targeted marketing :

  Behavioural segmentation enhances marketing efficiency by tailoring campaigns to user behaviour. This increases the likelihood of interest in specific products or services.

  4. Conversion rate optimisation :

  Analysing behavioural data reveals factors influencing user decisions, enabling website optimisation for a streamlined purchasing process and higher conversion rates.

  5. Data-driven decision-making :

  Behavioural segmentation empowers data-driven decisions. It identifies trends, behavioural patterns and emerging opportunities, facilitating adaptation to changing user preferences and market dynamics.

  6. Ethical considerations :

  Behavioural segmentation provides valuable insights but raises ethical concerns. User data collection and use must prioritise transparency, privacy and responsible handling to protect individuals’ rights.

  The significance of ethical behavioural segmentation will be explored more deeply in a later section, where we will delve into the ethical considerations and best practices for collecting, storing and utilising behavioural data in web analytics. It’s essential to strike a balance between harnessing the power of behavioural segmentation for business benefits and safeguarding user privacy and data rights in the digital age.

  

  Different types of behavioural segments with examples

  1. Visit-based segments : These segments hinge on users’ visit patterns. Analyse visit patterns, compare first-time visitors to returning ones, or compare users landing on specific pages to those landing on others.
     • Example : The real estate website Zillow can analyse how first-time visitors and returning users behave differently. By understanding these patterns, Zillow can customise its website for each group. For example, they can highlight featured listings and provide navigation tips for first-time visitors while offering personalised recommendations and saved search options for returning users. This could enhance user satisfaction and boost the chances of conversion.
  2. Interaction-based segments : Segments can be created based on user interactions like special events or goals completed on the site.
     • Example : Airbnb might use this to understand if users who successfully book accommodations exhibit different behaviours than those who don’t. This insight could guide refinements in the booking process for improved conversion rates.
  3. Campaign-based segments : Beyond tracking visit numbers, delve into usage differences of visitors from specific sources or ad campaigns for deeper insights.
     • Example : Nike might analyse user purchase behaviour from various traffic sources (referral websites, organic, direct, social media and ads). This informs marketing segmentation adjustments, focusing on high-performance channels. It also customises the website experience for different traffic sources, optimising content, promotions and navigation. This data-driven approach could boost user experiences and maximise marketing impact for improved brand engagement and sales conversions.
  4. Ecommerce segments : Separate users based on purchases, even examining the frequency of visits linked to specific products. Segment heavy users versus light users. This helps uncover diverse customer types and browsing behaviours.
     • Example : Amazon could create segments to differentiate between visitors who made purchases and those who didn’t. This segmentation could reveal distinct usage patterns and preferences, aiding Amazon in tailoring its recommendations and product offerings.
  5. Demographic segments : Build segments based on browser language or geographic location, for instance, to comprehend how user attributes influence site interactions.
     • Example : Netflix can create user segments based on demographic factors like geographic location to gain insight into how a visitor’s location can influence content preferences and viewing behaviour. This approach could allow for a more personalised experience.
  6. Technographic segments : Segment users by devices or browsers, revealing variations in site experience and potential platform-specific issues or user attitudes.
     • Example : Google could create segments based on users’ devices (e.g., mobile, desktop) to identify potential issues in rendering its search results. This information could be used to guide Google in providing consistent experiences regardless of device.

  

  The importance of ethical behavioural segmentation

  Respecting user privacy and data protection is crucial. Matomo offers features that align with ethical segmentation practices. These include :

  • Anonymization : Matomo allows for data anonymization, safeguarding individual identities while providing valuable insights.
  • GDPR compliance : Matomo is GDPR compliant, ensuring that user data is handled following European data protection regulations.
  • Data retention and deletion : Matomo enables businesses to set data retention policies and delete user data when it’s no longer needed, reducing the risk of data misuse.
  • Secured data handling : Matomo employs robust security measures to protect user data, reducing the risk of data breaches.

  Real-world examples of ethical behavioural segmentation :

  1. Content publishing : A leading news website could utilise data anonymization tools to ethically monitor user engagement. This approach allows them to optimise content delivery based on reader preferences while ensuring the anonymity and privacy of their target audience.
  2. Non-profit organisations : A charity organisation could embrace granular user control features. This could be used to empower its donors to manage their data preferences, building trust and loyalty among supporters by giving them control over their personal information.

  

  Examples of effective behavioural segmentation

  Companies are constantly using behavioural insights to engage their audiences effectively. In this section, we’ll delve into real-world examples showcasing how top companies use behavioural segmentation to enhance their marketing efforts.

  

  1. Coca-Cola’s behavioural insights for marketing strategy : Coca-Cola employs behavioural segmentation to evaluate its advertising campaigns. Through analysing user engagement across TV commercials, social media promotions and influencer partnerships, Coca-Cola’s marketing team can discover that video ads shared by influencers generate the highest ROI and web traffic.

     This insight guides the reallocation of resources, leading to increased sales and a more effective advertising strategy.

  2. eBay’s custom conversion approach : eBay excels in conversion optimisation through behavioural segmentation. When users abandon carts, eBay’s dynamic system sends personalised email reminders featuring abandoned items and related recommendations tailored to user interests and past purchase decisions.

     This strategy revives sales, elevates conversion rates and sparks engagement. eBay’s adeptness in leveraging behavioural insights transforms user experience, steering a customer journey toward conversion.

  3. Sephora’s data-driven conversion enhancement : Data analysts can use Sephora’s behavioural segmentation strategy to fuel revenue growth through meticulous data analysis. By identifying a dedicated subset of loyal customers who exhibit a consistent preference for premium skincare products, data analysts enable Sephora to customise loyalty programs.

     These personalised rewards programs provide exclusive discounts and early access to luxury skincare releases, resulting in heightened customer engagement and loyalty. The data-driven precision of this approach directly contributes to amplified revenue from this specific customer segment.

  Examples of the do’s and don’ts of behavioural segmentation 

  

  Behavioural segmentation is a powerful marketing and data analysis tool, but its success hinges on ethical and responsible practices. In this section, we will explore real-world examples of the do’s and don’ts of behavioural segmentation, highlighting companies that have excelled in their approach and those that have faced challenges due to lapses in ethical considerations.

  Do’s of behavioural segmentation :

  • Personalised messaging :
    • Example : Spotify
      • Spotify’s success lies in its ability to use behavioural data to curate personalised playlists and user recommendations, enhancing its music streaming experience.
  • Transparency :
    • Example : Basecamp
      • Basecamp’s transparency in sharing how user data is used fosters trust. They openly communicate data practices, ensuring users are informed and comfortable.
  • Anonymization
    • Example : Matomo’s anonymization features
      • Matomo employs anonymization features to protect user identities while providing valuable insights, setting a standard for responsible data handling.
  • Purpose limitation :
    • Example : Proton Mail
      • Proton Mail strictly limits the use of user data to email-related purposes, showcasing the importance of purpose-driven data practices.
  • Dynamic content delivery : 
    • Example : LinkedIn
      • LinkedIn uses behavioural segmentation to dynamically deliver job recommendations, showcasing the potential for relevant content delivery.
  • Data security :
    • Example : Apple
      • Apple’s stringent data security measures protect user information, setting a high bar for safeguarding sensitive data.
  • Adherence to regulatory compliance : 
    • Example : Matomo’s regulatory compliance features
      • Matomo’s regulatory compliance features ensure that businesses using the platform adhere to data protection regulations, further promoting responsible data usage.

  Don’ts of behavioural segmentation :

  • Ignoring changing regulations
    • Example : Equifax
      • Equifax faced major repercussions for neglecting evolving regulations, resulting in a data breach that exposed the sensitive information of millions.
  • Sensitive attributes
    • Example : Twitter
      • Twitter faced criticism for allowing advertisers to target users based on sensitive attributes, sparking concerns about user privacy and data ethics.
  • Data sharing without consent
    • Example : Meta & Cambridge Analytica
      • The Cambridge Analytica scandal involving Meta (formerly Facebook) revealed the consequences of sharing user data without clear consent, leading to a breach of trust.
  • Lack of control
    • Example : Uber
      • Uber faced backlash for its poor data security practices and a lack of control over user data, resulting in a data breach and compromised user information.
  • Don’t be creepy with invasive personalisation
    • Example : Offer Moment
      • Offer Moment’s overly invasive personalisation tactics crossed ethical boundaries, unsettling users and eroding trust.

  These examples are valuable lessons, emphasising the importance of ethical and responsible behavioural segmentation practices to maintain user trust and regulatory compliance in an increasingly data-driven world.

  Continue the conversation

  Diving into customer behaviours, preferences and interactions empowers businesses to forge meaningful connections with their target audience through targeted marketing segmentation strategies. This approach drives growth and fosters exceptional customer experiences, as evident from the various common examples spanning diverse industries.

  In the realm of ethical behavioural segmentation and regulatory compliance, Matomo is a trusted partner. Committed to safeguarding user privacy and data integrity, our advanced web analytics solution empowers your business to harness the power of behavioral segmentation, all while upholding the highest standards of compliance with stringent privacy regulations.

  To gain deeper insight into your visitors and execute impactful marketing campaigns, explore how Matomo can elevate your efforts. Try Matomo free for 21-days, no credit card required. 

• 8 Best Tools to Analyse Website Traffic

  12 septembre 2023, par Erin — Analytics Tips, Marketing

  Do you want to analyse your website traffic ?

  Maybe you want to know how well you’re converting your traffic. Or maybe you’re looking to track the performance and ROI of your marketing campaigns. Regardless, you won’t get far without relying on a dependable web traffic analysis platform.

  In this article, we’ve compiled a list of the top web analytics tools available (including the pricing for each one).

  Let’s dive in.

  What is website traffic analysis ?

  Curious about what it means to analyse website traffic ?

  

  Simply put, it involves collecting and examining data about your website visitors and the actions they take. Marketers, analysts and website owners can then take this data and use it to optimise their strategy to improve site traffic, conversion rates and ROI.

  A website analytics tool is software that tracks and measures various visitor activities and behaviours on your website. Common metrics include pageviews, traffic source, bounce rate and average time on page. Using a web analytics solution can give you insights into what’s working (and what’s not working) so you can optimise your website, campaigns or marketing strategy.

  Advantages of using a website traffic analysis tool

  1. Performance measurement and optimisation

  Tracking the success of your marketing efforts is a challenging task. The primary benefit of using a web analytics tool is implementing effective performance measurement. If you don’t know how to measure your efforts, you won’t know what’s working and what’s not with your campaigns and content. 

  A web analysis tool can give you the insights you need to understand whether your marketing initiatives have been successful or if they need to be improved.

  For instance, your new web design facelift may seem beautiful, but if visitors aren’t staying on your site as long and it is resulting in lower conversions, then it’s time to go back to the drawing board.

  2. Audience insights to improve the user experience

  Web traffic analysis platforms don’t just show you what your visitors are doing. It shows you who your audience is. A powerful website analytics tool will give you in-depth audience data, including demographics like geographical location (e.g., city, state or country), to help you better understand your audience.

  Additionally, you can learn more about your audience by seeing how they interact with different content on your site. You’ll start to see that certain content performs better than others, giving you a greater understanding of your audience’s needs and wants. This means you’ll be able to tailor your website content and marketing efforts to your audience to improve the overall user experience.

  3. Improve SEO

  In the first two advantages, we touched on how insights can help you craft better content for the visitors already coming to your site to improve the user experience and improve conversions. But did you know that using a website analytics tool can also help improve how much traffic you’re getting to your site ?

  Since a web analytics tool can help you craft better content, one side effect is an increase in traffic from organic search through SEO. Additionally, your platform will likely show you other traffic sources that your visitors are coming from (i.e., another website is referring traffic to you) so you can tap into those high-performing sources and optimise your incoming traffic over time.

  Top 8 Tools to Analyse Website Traffic

  Here’s a breakdown of the top eight web analytics platforms to help you analyse each tool’s unique features, price, advantages and disadvantages so you can make the best decision.

  1. Matomo

  Matomo is an open-source website analytics tool that’s focused on protecting user privacy and data while offering robust insights into your web traffic. It’s one of the most powerful tools to track the entire customer journey on your site.

  

  Why Matomo : As the leader in open-source, privacy-friendly and ethical web analytics, Matomo is trusted by more than 1 million websites, including NASA, the United Nations and the European Commission.

  Matomo plays well with Google Analytics to track your websites by filling in the gaps where Google Analytics has limitations (i.e., cookie consent banner requirement). Matomo combines traditional and behavioural web analytics for deeper insights while ensuring compliance with the strictest privacy regulations like GDPR, LGPD and HIPAA.

  Matomo Standout Features and Integrations :

  Standout features include comprehensive visitor tracking, multi-attribution, goal tracking, event tracking, custom dimensions, custom reports, automated email reports, session recordings, tag manager, roll-up reporting to pull data from multiple sites, Google Analytics importer, heatmaps and more.

  Integrations include WordPress, Google Ads, Wix, Drupal, Joomla, Cloudflare, Magento, Vue, SharePoint, WooCommerce and more.

  Pricing starts free for Matomo On-Premise (but requires technical skills and servers to set up) and $23/month for Matomo Cloud, which includes a 21-day free trial (no credit card required).

  Pros

  • Best for respecting visitor privacy
  • You own your data — ensuring that it’s not shared with third parties for purposes like advertising
  • Compliant with the strictest privacy laws
  • Greater flexibility with open-source advantages, as well as the option to either self-host or cloud host
  • Can run cookieless — providing 100% accurate data and a better user experience without the need for an annoying cookie consent banner 
  • Exceptional customisability — from white labeling, alerts and custom dimensions to dashboards and reports, tailor your insights for faster decisions, deeper insights and superior outcomes

  Cons

  • On-Premise is free, but there are additional costs for advanced features
  • On-Premise requires servers and technical expertise to manage

  2. Google Analytics

  Google Analytics is the most well-known and used web analytics platform in the world, with nearly 30 million active websites.

  

  Why Google Analytics : It’s one of the leading web traffic analysis tools backed by the Alphabet group of companies. For anyone getting started, it’s a great free option to understand your web traffic and your audience.

  Google Analytics Standout Features and Integrations :

  Standout features include in-depth visitor tracking, event tracking with Google Analytics 4 (GA4), easy integration with Google marketing tools (i.e., Google Search Console and Google Ads), custom reports and easy data importing from third-party sources.

  Integrations include Google Ads, Google Webmaster Tools, AdSense, WordPress, Wix, Shopify, Zendesk, Facebook, Marketo, WordPress, Hotjar, SEMrush, Salesforce, Hootsuite and more.

  Pricing is free.

  Pros

  • Detailed audience insights
  • Customisable reports
  • Seamless integration with other Google products
  • Easy to set up

  Cons

  • Not privacy-friendly — you don’t own your data (data is shared with third parties for advertising purposes)
  • Complex interface
  • Requires cookie consent banner for GDPR compliance, which negatively impacts data accuracy and user experience

  3. Fathom Analytics

  Founded in 2018, Fathom Analytics is a privacy-friendly and lightweight web analytics tool. The platform offers a simple, minimalistic dashboard.

  

  Why Fathom Analytics : Fathom Analytics is a minimalistic tool to help website owners gain insights into customer behaviour without compromising on privacy. It’s an easy-to-use tool that offers a simplified breakdown of the most popular data points. For newcomers to web analytics seeking essential metrics like visitor counts and traffic sources, Fathom Analytics provides a straightforward, cost-effective solution.

  Fathom Analytics Standout Features and Integrations :

  Standout features include easy, automated GA4 importing with lifetime data retention, a single-page dashboard for a quick overview of metrics, traffic summaries for chosen timeframes, visually striking graphs for better data digestion and privacy protection covering major compliance regulations.

  Integrations include Google Analytics, Squarespace, Drupal, WordPress, Discourse, Bloggi, ConvertKit, Webflow, Transistor, Remix, Gatsby and Carrd.

  Pricing starts at $14/month for up to 100k pageviews (with a 30-day free trial).

  Pros

  • Doesn’t use cookies
  • Out-of-the-box GDPR, ePrivacy, PECR and CCPA compliance
  • Great for visual data insights
  • Lightweight tracking script for fast loading

  Cons

  • Can’t easily see traffic trends on specific pages
  • Metrics may be too simple for those wanting advanced analytics

  4. Mixpanel

  Mixpanel is a web analytics platform that helps you track visitors as well as improve customer retention. The software has 8,000 customers worldwide, including Netflix, Yelp, BuzzFeed and CNN.

  

  Why Mixpanel : Mixpanel is great for websites with e-commerce functionality. The tool helps you understand both your site visitors and your customers so you can optimise your customer experience and improve conversions.

  Mixpanel Standout Features and Integrations :

  Standout features include deep insights into how your products are being used, including your most popular features, user cohorts that let you segment users based on specific actions, and visual analysis showing where users drop off.

  Integrations include Google Cloud, Figma, Mailchimp, Zoho CRM, Databox, Marketo, Hotjar, Slack, Zapier, Amazon Web Services, Google Ads and HubSpot.

  Pricing starts free for up to 20 million events per month and $20/month for Growth.

  Pros

  • Interface is easy for beginners
  • Exhaustive reporting options
  • Custom event tracking options
  • Predict user actions based on data science models
  • Send targeted messages to specific users to encourage action

  Cons

  • User-based pricing isn’t the most ideal for everyone
  • Alert management can be confusing

  5. Kissmetrics

  Kissmetrics is a marketing and product analytics tool that helps e-commerce and SaaS companies grow through qualitative data insights. The web analytics tool is trusted by 10,000 users, including Microsoft, Unbounce, AWeber, Dropbox DocSend and SendGrid.

  

  Why Kissmetrics : As an e-commerce-driven analytics platform, the platform is best suited for Enterprise businesses, but it also offers flexible pricing plans that make it easy for someone to get their feet wet with website analytics. 

  Kissmetrics Standout Features and Integrations :

  Standout features include a customisable dashboard to see key metrics at a glance, comprehensive visitor tracking, cohort analysis including power user tracking to understand your most active visitors and customers and insights into customer lifetime value and churn rate.

  Integrations include Chargify, HubSpot, Slack, Live Chat, Marketo, Optimizely, Mailchimp, Recurly, Wufoo Forms, Facebook Ads, WordPress, Shopify and WooCommerce.

  Pricing starts at $0.0025/event for the Pay As You Go Plan, $25.99/month for Build Your Plan and $199/month for Small Teams, which includes a 7-day free trial.

  Pros

  • Flexible pricing options
  • Easy to install
  • Several analytics viewing options
  • Visual checkout funnel insights
  • Track sessions by desktop or mobile

  Cons

  • Despite more pricing options, it’s still quite expensive overall
  • Difficult to use for beginners

  6. Adobe Analytics

  Adobe Analytics is a web and marketing analytics platform within the Adobe Experience Platform. Used by over 170,000 businesses, it’s one of the most popular analytics solutions available.

  

  Why Adobe Analytics : Adobe Analytics was created for large organisations. It’s essentially the enterprise version of Google Analytics. The tool does a great job of offering a customised analytics solution that’s capable of delivering personalised user experiences at scale.

  Adobe Analytics Standout Features and Integrations :

  Standout features include attribution, AI-driven predictive analytics, robust customer segmentation and automation based on customer behaviour.

  Integrations include all Adobe products, Salesforce, Hootsuite, Contentsquare, Sisense, Mouseflow, Google Ads, Google Search Console, HubSpot and Microsoft Teams.

  Pricing is custom and available upon request, but users can expect to pay at least $2,000 per month, and there is no free trial.

  Pros

  • Built for enterprise businesses
  • Seamless workflow integration for Adobe Experience Cloud users
  • Incredible customisation options
  • Integration process is flexible
  • Capable of accurately tracking large volumes of traffic

  Cons

  • Very expensive
  • Not suitable for small businesses
  • The setup is challenging for beginners

  7. SimilarWeb

  SimilarWeb is a robust analytics platform used to track your website data and compare it to other websites. Backed by a team of experienced data scientists and mathematicians for in-depth website traffic and search engine analysis. Founded in 2007, the platform is trusted by major brands like Adidas, DHL, PepsiCo and Walmart.

  

  Why SimilarWeb : The tool relies on multiple scientific methodologies and approaches to data analysis to help provide a better understanding of visitors and customers. The platform is great for crafting prediction models for customer acquisitions by using machine learning to offer SEO insights and competitive analysis.

  SimilarWeb Standout Features and Integrations :

  Standout features include competition traffic and engagement analysis, in-depth visitor tracking, keyword analysis to optimise your SEO and search ads, affiliate traffic analysis, search traffic analysis and funnel insights.

  Integrations include Salesforce, HubSpot, Google Analytics, Google Search Console, Shift, AT Internet, Adverity, SimilarTech, Biscience and more. 

  Pricing starts at $125/month for the Starter plan, which includes a 7-day free trial.

  Pros

  • Has a user-friendly dashboard for simple insights
  • Highly customisable platform to meet your specific needs
  • Easy competition analysis
  • Funnel insights to improve your conversion rates
  • Great customer support

  Cons

  • Expensive pricing
  • Doesn’t include a code snippet to pull data directly from websites
  • Doesn’t show sub-domains of your site

  8. Hotjar

  Hotjar is a behavioural website analytics tool with a focus on providing insights into individual user sessions with features like heatmaps and session recordings. Founded in 2014, Hotjar is used by 900,000 sites around the world.

  

  Why Hotjar : Unlike traditional web analytics tools like Google Analytics, Hotjar is a behavioural analytics tool that provides in-depth behaviour insights session by session. The tool offers a variety of features that give you a sneak peek into your users’ behaviours by watching how they interact with your site action by action.

  Hotjar Standout Features and Integrations :

  Standout features include comprehensive heat mapping, visitor session recordings to see what visitors did moment by moment, feedback polls to gain insights from site visitors and conversion funnels to help you analyse leaks in your funnel at each conversion stage.

  Integrations include HubSpot, Slack, Jira, WordPress, Shopify, Google Analytics, Mixpanel, Microsoft Teams, Zapier and ClickFunnels.

  Pricing starts at free for the Basic plan and $80/month for Business, which includes a 15-day free trial.

  Pros

  • You can see exactly where visitors click, move and scroll
  • Watch session replays to see what visitors did step-by-step
  • See what percentage of visitors take certain actions
  • Data segmentation features to help you understand KPIs in-depth
  • There are no user limits with the platform, making it easy to scale

  Cons

  • While it offers behavioural analytics, Hotjar doesn’t provide insights into traditional web analytics like Matomo does, including traffic sources and bounce rate
  • History data monitoring is complex

  Elevate your website performance today

  Understanding your visitors’ behaviour and needs is essential when you’re looking to improve your website performance.

  By leveraging a website analytics platform, you’ll be able to gain new insights into your visitors and use insights from your content and campaign performance to improve your user experience.

  If you’re looking to start using a web traffic analysis tool today, then Matomo is an excellent choice.

  Matomo is a powerful, privacy-friendly and compliant tool that gives in-depth insights into your audience, your content and your marketing efforts to help you improve your site’s performance.

  The platform also includes a variety of robust behavioural analytics features like heatmaps, session recording and more, which are included in your Cloud subscription. 

  Start your 21-day free trial of Matomo today (no credit card required).

• Overthinking My Search Engine Problem

  31 décembre 2013, par Multimedia Mike — General

  I wrote a search engine for my Game Music Appreciation website, because the site would have been significantly less valuable without it (and I would eventually realize that the search feature is probably the most valuable part of this endeavor). I came up with a search solution that was a bit sketchy, but worked… until it didn’t. I thought of a fix but still searched for more robust and modern solutions (where ‘modern’ is defined as something that doesn’t require compiling a C program into a static CGI script and hoping that it works on a server I can’t debug on).

  Finally, I realized that I was overthinking the problem– did you know that a bunch of relational database management systems (RDBMSs) support full text search (FTS) ? Okay, maybe you did, but I didn’t know this.

  Problem Statement
  My goal is to enable users to search the metadata (title, composer, copyright, other tags) attached to various games. To do this, I want to index a series of contrived documents that describe the metadata. 2 examples of these contrived documents, interesting because both of these games have very different titles depending on region, something the search engine needs to account for :

  system : Nintendo NES
  game : Snoopy’s Silly Sports Spectacular
  author : None ; copyright : 1988 Kemco ; dumped by : None
  additional tags : Donald Duck.nsf Donald Duck
  system : Super Nintendo
  
  game : Arcana
  
  author : Jun Ishikawa, Hirokazu Ando ; copyright : 1992 HAL Laboratory ; dumped by : Datschge
  
  additional tags : card.rsn.gamemusic Card Master Cardmaster
  

  The index needs to map these documents to various pieces of game music and the search solution needs to efficiently search these documents and find the various game music entries that match a user’s request.

  Now that I’ve been looking at it for long enough, I’m able to express the problem surprisingly succinctly. If I had understood that much originally, this probably would have been simpler.

  First Solution & Breakage
  My original solution was based on SWISH-E. The CGI script was a C program that statically linked the SWISH-E library into a binary that miraculously ran on my web provider. At least, it ran until it decided to stop working a month ago when I added a new feature unrelated to search. It was a very bizarre problem, the details of which would probably bore you to tears. But if you care, the details are all there in the Stack Overflow question I asked on the matter.

  While no one could think of a direct answer to the problem, I eventually thought of a roundabout fix. The problem seemed to pertain to the static linking. Since I couldn’t count on the relevant SWISH-E library to be on my host’s system, I uploaded the shared library to the same directory as the CGI script and used dlopen()/dlsym() to fetch the functions I needed. It worked again, but I didn’t know for how long.

  Searching For A Hosted Solution
  I know that anything is possible in this day and age ; while my web host is fairly limited, there are lots of solutions for things like this and you can deploy any technology you want, and for reasonable prices. I figured that there must be a hosted solution out there.

  I have long wanted a compelling reason to really dive into Amazon Web Services (AWS) and this sounded like a good opportunity. After all, my script works well enough ; if I could just find a simple Linux box out there where I could install the SWISH-E library and compile the CGI script, I should be good to go. AWS has a free tier and I started investigating this approach. But it seems like a rabbit hole with a lot of moving pieces necessary for such a simple task.

  I had heard that AWS had something in this area. Sure enough, it’s called CloudSearch. However, I’m somewhat discouraged by the fact that it would cost me around $75 per month to run the smallest type of search instance which is at the core of the service.

  Finally, I came to another platform called Heroku. It’s supposed to be super-scalable while having a free tier for hobbyists. I started investigating FTS on Heroku and found this article which recommends using the FTS capabilities of their standard hosted PostgreSQL solution. However, the free tier of Postgres hosting only allows for 10,000 rows of data. Right now, my database has about 5400 rows. I expect it to easily overflow the 10,000 limit as soon as I incorporate the C64 SID music corpus.

  However, this Postgres approach planted a seed.

  RDBMS Revelation
  I have 2 RDBMSs available on my hosting plan– MySQL and SQLite (the former is a separate service while SQLite is built into PHP). I quickly learned that both have FTS capabilities. Since I like using SQLite so much, I elected to leverage its FTS functionality. And it’s just this simple :

  CREATE VIRTUAL TABLE gamemusic_metadata_fts USING fts3
  ( content TEXT, game_id INT, title TEXT ) ;
  SELECT id, title FROM gamemusic_metadata_fts WHERE content MATCH "arcana" ;
  
  479|Arcana
  

  The ‘content’ column gets the metadata pseudo-documents. The SQL gets wrapped up in a little PHP so that it queries this small database and turns the result into JSON. The script is then ready as a drop-in replacement for the previous script.