Recherche avancée

Sur d’autres sites (12833)

• 8 Best Tools to Analyse Website Traffic

  12 septembre 2023, par Erin — Analytics Tips, Marketing

  Do you want to analyse your website traffic ?

  Maybe you want to know how well you’re converting your traffic. Or maybe you’re looking to track the performance and ROI of your marketing campaigns. Regardless, you won’t get far without relying on a dependable web traffic analysis platform.

  In this article, we’ve compiled a list of the top web analytics tools available (including the pricing for each one).

  Let’s dive in.

  What is website traffic analysis ?

  Curious about what it means to analyse website traffic ?

  

  Simply put, it involves collecting and examining data about your website visitors and the actions they take. Marketers, analysts and website owners can then take this data and use it to optimise their strategy to improve site traffic, conversion rates and ROI.

  A website analytics tool is software that tracks and measures various visitor activities and behaviours on your website. Common metrics include pageviews, traffic source, bounce rate and average time on page. Using a web analytics solution can give you insights into what’s working (and what’s not working) so you can optimise your website, campaigns or marketing strategy.

  Advantages of using a website traffic analysis tool

  1. Performance measurement and optimisation

  Tracking the success of your marketing efforts is a challenging task. The primary benefit of using a web analytics tool is implementing effective performance measurement. If you don’t know how to measure your efforts, you won’t know what’s working and what’s not with your campaigns and content. 

  A web analysis tool can give you the insights you need to understand whether your marketing initiatives have been successful or if they need to be improved.

  For instance, your new web design facelift may seem beautiful, but if visitors aren’t staying on your site as long and it is resulting in lower conversions, then it’s time to go back to the drawing board.

  2. Audience insights to improve the user experience

  Web traffic analysis platforms don’t just show you what your visitors are doing. It shows you who your audience is. A powerful website analytics tool will give you in-depth audience data, including demographics like geographical location (e.g., city, state or country), to help you better understand your audience.

  Additionally, you can learn more about your audience by seeing how they interact with different content on your site. You’ll start to see that certain content performs better than others, giving you a greater understanding of your audience’s needs and wants. This means you’ll be able to tailor your website content and marketing efforts to your audience to improve the overall user experience.

  3. Improve SEO

  In the first two advantages, we touched on how insights can help you craft better content for the visitors already coming to your site to improve the user experience and improve conversions. But did you know that using a website analytics tool can also help improve how much traffic you’re getting to your site ?

  Since a web analytics tool can help you craft better content, one side effect is an increase in traffic from organic search through SEO. Additionally, your platform will likely show you other traffic sources that your visitors are coming from (i.e., another website is referring traffic to you) so you can tap into those high-performing sources and optimise your incoming traffic over time.

  Top 8 Tools to Analyse Website Traffic

  Here’s a breakdown of the top eight web analytics platforms to help you analyse each tool’s unique features, price, advantages and disadvantages so you can make the best decision.

  1. Matomo

  Matomo is an open-source website analytics tool that’s focused on protecting user privacy and data while offering robust insights into your web traffic. It’s one of the most powerful tools to track the entire customer journey on your site.

  

  Why Matomo : As the leader in open-source, privacy-friendly and ethical web analytics, Matomo is trusted by more than 1 million websites, including NASA, the United Nations and the European Commission.

  Matomo plays well with Google Analytics to track your websites by filling in the gaps where Google Analytics has limitations (i.e., cookie consent banner requirement). Matomo combines traditional and behavioural web analytics for deeper insights while ensuring compliance with the strictest privacy regulations like GDPR, LGPD and HIPAA.

  Matomo Standout Features and Integrations :

  Standout features include comprehensive visitor tracking, multi-attribution, goal tracking, event tracking, custom dimensions, custom reports, automated email reports, session recordings, tag manager, roll-up reporting to pull data from multiple sites, Google Analytics importer, heatmaps and more.

  Integrations include WordPress, Google Ads, Wix, Drupal, Joomla, Cloudflare, Magento, Vue, SharePoint, WooCommerce and more.

  Pricing starts free for Matomo On-Premise (but requires technical skills and servers to set up) and $23/month for Matomo Cloud, which includes a 21-day free trial (no credit card required).

  Pros

  • Best for respecting visitor privacy
  • You own your data — ensuring that it’s not shared with third parties for purposes like advertising
  • Compliant with the strictest privacy laws
  • Greater flexibility with open-source advantages, as well as the option to either self-host or cloud host
  • Can run cookieless — providing 100% accurate data and a better user experience without the need for an annoying cookie consent banner 
  • Exceptional customisability — from white labeling, alerts and custom dimensions to dashboards and reports, tailor your insights for faster decisions, deeper insights and superior outcomes

  Cons

  • On-Premise is free, but there are additional costs for advanced features
  • On-Premise requires servers and technical expertise to manage

  2. Google Analytics

  Google Analytics is the most well-known and used web analytics platform in the world, with nearly 30 million active websites.

  

  Why Google Analytics : It’s one of the leading web traffic analysis tools backed by the Alphabet group of companies. For anyone getting started, it’s a great free option to understand your web traffic and your audience.

  Google Analytics Standout Features and Integrations :

  Standout features include in-depth visitor tracking, event tracking with Google Analytics 4 (GA4), easy integration with Google marketing tools (i.e., Google Search Console and Google Ads), custom reports and easy data importing from third-party sources.

  Integrations include Google Ads, Google Webmaster Tools, AdSense, WordPress, Wix, Shopify, Zendesk, Facebook, Marketo, WordPress, Hotjar, SEMrush, Salesforce, Hootsuite and more.

  Pricing is free.

  Pros

  • Detailed audience insights
  • Customisable reports
  • Seamless integration with other Google products
  • Easy to set up

  Cons

  • Not privacy-friendly — you don’t own your data (data is shared with third parties for advertising purposes)
  • Complex interface
  • Requires cookie consent banner for GDPR compliance, which negatively impacts data accuracy and user experience

  3. Fathom Analytics

  Founded in 2018, Fathom Analytics is a privacy-friendly and lightweight web analytics tool. The platform offers a simple, minimalistic dashboard.

  

  Why Fathom Analytics : Fathom Analytics is a minimalistic tool to help website owners gain insights into customer behaviour without compromising on privacy. It’s an easy-to-use tool that offers a simplified breakdown of the most popular data points. For newcomers to web analytics seeking essential metrics like visitor counts and traffic sources, Fathom Analytics provides a straightforward, cost-effective solution.

  Fathom Analytics Standout Features and Integrations :

  Standout features include easy, automated GA4 importing with lifetime data retention, a single-page dashboard for a quick overview of metrics, traffic summaries for chosen timeframes, visually striking graphs for better data digestion and privacy protection covering major compliance regulations.

  Integrations include Google Analytics, Squarespace, Drupal, WordPress, Discourse, Bloggi, ConvertKit, Webflow, Transistor, Remix, Gatsby and Carrd.

  Pricing starts at $14/month for up to 100k pageviews (with a 30-day free trial).

  Pros

  • Doesn’t use cookies
  • Out-of-the-box GDPR, ePrivacy, PECR and CCPA compliance
  • Great for visual data insights
  • Lightweight tracking script for fast loading

  Cons

  • Can’t easily see traffic trends on specific pages
  • Metrics may be too simple for those wanting advanced analytics

  4. Mixpanel

  Mixpanel is a web analytics platform that helps you track visitors as well as improve customer retention. The software has 8,000 customers worldwide, including Netflix, Yelp, BuzzFeed and CNN.

  

  Why Mixpanel : Mixpanel is great for websites with e-commerce functionality. The tool helps you understand both your site visitors and your customers so you can optimise your customer experience and improve conversions.

  Mixpanel Standout Features and Integrations :

  Standout features include deep insights into how your products are being used, including your most popular features, user cohorts that let you segment users based on specific actions, and visual analysis showing where users drop off.

  Integrations include Google Cloud, Figma, Mailchimp, Zoho CRM, Databox, Marketo, Hotjar, Slack, Zapier, Amazon Web Services, Google Ads and HubSpot.

  Pricing starts free for up to 20 million events per month and $20/month for Growth.

  Pros

  • Interface is easy for beginners
  • Exhaustive reporting options
  • Custom event tracking options
  • Predict user actions based on data science models
  • Send targeted messages to specific users to encourage action

  Cons

  • User-based pricing isn’t the most ideal for everyone
  • Alert management can be confusing

  5. Kissmetrics

  Kissmetrics is a marketing and product analytics tool that helps e-commerce and SaaS companies grow through qualitative data insights. The web analytics tool is trusted by 10,000 users, including Microsoft, Unbounce, AWeber, Dropbox DocSend and SendGrid.

  

  Why Kissmetrics : As an e-commerce-driven analytics platform, the platform is best suited for Enterprise businesses, but it also offers flexible pricing plans that make it easy for someone to get their feet wet with website analytics. 

  Kissmetrics Standout Features and Integrations :

  Standout features include a customisable dashboard to see key metrics at a glance, comprehensive visitor tracking, cohort analysis including power user tracking to understand your most active visitors and customers and insights into customer lifetime value and churn rate.

  Integrations include Chargify, HubSpot, Slack, Live Chat, Marketo, Optimizely, Mailchimp, Recurly, Wufoo Forms, Facebook Ads, WordPress, Shopify and WooCommerce.

  Pricing starts at $0.0025/event for the Pay As You Go Plan, $25.99/month for Build Your Plan and $199/month for Small Teams, which includes a 7-day free trial.

  Pros

  • Flexible pricing options
  • Easy to install
  • Several analytics viewing options
  • Visual checkout funnel insights
  • Track sessions by desktop or mobile

  Cons

  • Despite more pricing options, it’s still quite expensive overall
  • Difficult to use for beginners

  6. Adobe Analytics

  Adobe Analytics is a web and marketing analytics platform within the Adobe Experience Platform. Used by over 170,000 businesses, it’s one of the most popular analytics solutions available.

  

  Why Adobe Analytics : Adobe Analytics was created for large organisations. It’s essentially the enterprise version of Google Analytics. The tool does a great job of offering a customised analytics solution that’s capable of delivering personalised user experiences at scale.

  Adobe Analytics Standout Features and Integrations :

  Standout features include attribution, AI-driven predictive analytics, robust customer segmentation and automation based on customer behaviour.

  Integrations include all Adobe products, Salesforce, Hootsuite, Contentsquare, Sisense, Mouseflow, Google Ads, Google Search Console, HubSpot and Microsoft Teams.

  Pricing is custom and available upon request, but users can expect to pay at least $2,000 per month, and there is no free trial.

  Pros

  • Built for enterprise businesses
  • Seamless workflow integration for Adobe Experience Cloud users
  • Incredible customisation options
  • Integration process is flexible
  • Capable of accurately tracking large volumes of traffic

  Cons

  • Very expensive
  • Not suitable for small businesses
  • The setup is challenging for beginners

  7. SimilarWeb

  SimilarWeb is a robust analytics platform used to track your website data and compare it to other websites. Backed by a team of experienced data scientists and mathematicians for in-depth website traffic and search engine analysis. Founded in 2007, the platform is trusted by major brands like Adidas, DHL, PepsiCo and Walmart.

  

  Why SimilarWeb : The tool relies on multiple scientific methodologies and approaches to data analysis to help provide a better understanding of visitors and customers. The platform is great for crafting prediction models for customer acquisitions by using machine learning to offer SEO insights and competitive analysis.

  SimilarWeb Standout Features and Integrations :

  Standout features include competition traffic and engagement analysis, in-depth visitor tracking, keyword analysis to optimise your SEO and search ads, affiliate traffic analysis, search traffic analysis and funnel insights.

  Integrations include Salesforce, HubSpot, Google Analytics, Google Search Console, Shift, AT Internet, Adverity, SimilarTech, Biscience and more. 

  Pricing starts at $125/month for the Starter plan, which includes a 7-day free trial.

  Pros

  • Has a user-friendly dashboard for simple insights
  • Highly customisable platform to meet your specific needs
  • Easy competition analysis
  • Funnel insights to improve your conversion rates
  • Great customer support

  Cons

  • Expensive pricing
  • Doesn’t include a code snippet to pull data directly from websites
  • Doesn’t show sub-domains of your site

  8. Hotjar

  Hotjar is a behavioural website analytics tool with a focus on providing insights into individual user sessions with features like heatmaps and session recordings. Founded in 2014, Hotjar is used by 900,000 sites around the world.

  

  Why Hotjar : Unlike traditional web analytics tools like Google Analytics, Hotjar is a behavioural analytics tool that provides in-depth behaviour insights session by session. The tool offers a variety of features that give you a sneak peek into your users’ behaviours by watching how they interact with your site action by action.

  Hotjar Standout Features and Integrations :

  Standout features include comprehensive heat mapping, visitor session recordings to see what visitors did moment by moment, feedback polls to gain insights from site visitors and conversion funnels to help you analyse leaks in your funnel at each conversion stage.

  Integrations include HubSpot, Slack, Jira, WordPress, Shopify, Google Analytics, Mixpanel, Microsoft Teams, Zapier and ClickFunnels.

  Pricing starts at free for the Basic plan and $80/month for Business, which includes a 15-day free trial.

  Pros

  • You can see exactly where visitors click, move and scroll
  • Watch session replays to see what visitors did step-by-step
  • See what percentage of visitors take certain actions
  • Data segmentation features to help you understand KPIs in-depth
  • There are no user limits with the platform, making it easy to scale

  Cons

  • While it offers behavioural analytics, Hotjar doesn’t provide insights into traditional web analytics like Matomo does, including traffic sources and bounce rate
  • History data monitoring is complex

  Elevate your website performance today

  Understanding your visitors’ behaviour and needs is essential when you’re looking to improve your website performance.

  By leveraging a website analytics platform, you’ll be able to gain new insights into your visitors and use insights from your content and campaign performance to improve your user experience.

  If you’re looking to start using a web traffic analysis tool today, then Matomo is an excellent choice.

  Matomo is a powerful, privacy-friendly and compliant tool that gives in-depth insights into your audience, your content and your marketing efforts to help you improve your site’s performance.

  The platform also includes a variety of robust behavioural analytics features like heatmaps, session recording and more, which are included in your Cloud subscription. 

  Start your 21-day free trial of Matomo today (no credit card required).

• Data Privacy in Business : A Risk Leading to Major Opportunities

  9 août 2022, par Erin — Privacy

  Data privacy in business is a contentious issue. 

  Claims that “big data is the new oil of the digital economy” and strong links between “data-driven personalisation and customer experience” encourage leaders to set up massive data collection programmes.

  However, many of these conversations downplay the magnitude of security, compliance and ethical risks companies face when betting too much on customer data collection. 

  In this post, we discuss the double-edged nature of privacy issues in business — the risk-ridden and the opportunity-driven. ​​

  3 Major Risks of Ignoring Data Privacy in Business

  As the old adage goes : Just because everyone else is doing it doesn’t make it right.

  Easy data accessibility and ubiquity of analytics tools make data consumer collection and processing sound like a “given”. But the decision to do so opens your business to a spectrum of risks. 

  1. Compliance and Legal Risks 

  Data collection and customer privacy are protected by a host of international laws including GDPR, CCPA, and regional regulations. Only 15% of countries (mostly developing ones) don’t have dedicated laws for protecting consumer privacy. 

  

  Global legislature includes provisions on : 

  • Collectible data types
  • Allowed uses of obtained data 
  • Consent to data collection and online tracking 
  • Rights to request data removal 

  Personally identifiable information (PII) processing is prohibited or strictly regulated in most jurisdictions. Yet businesses repeatedly circumnavigate existing rules and break them on occasion.

  In Australia, for example, only 2% of brands use logos, icons or messages to transparently call out online tracking, data sharing or other specific uses of data at the sign-up stage. In Europe, around half of small businesses are still not fully GDPR-compliant — and Big Tech companies like Google, Amazon and Facebook can’t get a grip on their data collection practices even when pressed with horrendous fines. 

  Although the media mostly reports on compliance fines for “big names”, smaller businesses are increasingly receiving more scrutiny. 

  As Max Schrems, an Austrian privacy activist and founder of noyb NGO, explained in a Matomo webinar :

  “In Austria, my home country, there are a lot of €5,000 fines going out there as well [to smaller businesses]. Most of the time, they are just not reported. They just happen below the surface. [GDPR fines] are already a reality.”​

  In April 2022, the EU Court of Justice ruled that consumer groups can autonomously sue businesses for breaches of data protection — and nonprofit organisations like noyb enable more people to do so. 

  Finally, new data privacy legislation is underway across the globe. In the US, Colorado, Connecticut, Virginia and Utah have data protection acts at different stages of approval. South African authorities are working on the Protection of Personal Information Act (POPI) act and Brazil is working on a local General Data Protection Law (LGPD).

  Re-thinking your stance on user privacy and data protection now can significantly reduce the compliance burden in the future. 

  2. Security Risks 

  Data collection also mandates data protection for businesses. Yet, many organisations focus on the former and forget about the latter. 

  Lenient attitudes to consumer data protection resulted in a major spike in data breaches.

  Check Point research found that cyberattacks increased 50% year-over-year, with each organisation facing 925 cyberattacks per week globally.

  Many of these attacks end up being successful due to poor data security in place. As a result, billions of stolen consumer records become publicly available or get sold on dark web marketplaces.

  What’s even more troublesome is that stolen consumer records are often purchased by marketing firms or companies, specialising in spam campaigns. Buyers can also use stolen emails to distribute malware, stage phishing and other social engineering attacks – and harvest even more data for sale. 

  One business’s negligence creates a snowball effect of negative changes down the line with customers carrying the brunt of it all. 

  In 2020, hackers successfully targeted a Finnish psychotherapy practice. They managed to steal hundreds of patient records — and then demanded a ransom both from the firm and its patients for not exposing information about their mental health issues. Many patients refused to pay hackers and some 300 records ended up being posted online as Associated Press reported.

  Not only did the practice have to deal with the cyber-breach aftermath, but it also faced vocal regulatory and patient criticisms for failing to properly protect such sensitive information.

  Security negligence can carry both direct (heavy data breach fines) and indirect losses in the form of reputational damages. An overwhelming 90% of consumers say they wouldn’t buy from a business if it doesn’t adequately protect their data. This brings us to the last point. 

  3. Reputational Risks 

  Trust is the new currency. Data negligence and consumer privacy violations are the two fastest ways to lose it. 

  Globally, consumers are concerned about how businesses collect, use, and protect their data. 

  

  • According to Forrester, 47% of UK adults actively limit the amount of data they share with websites and apps. 49% of Italians express willingness to ask companies to delete their personal data. 36% of Germans use privacy and security tools to minimise online tracking of their activities. 
  • A GDMA survey also notes that globally, 82% of consumers want more control over their personal information, shared with companies. 77% also expect brands to be transparent about how their data is collected and used. 

  When businesses fail to hold their end of the bargain — collect just the right amount of data and use it with integrity — consumers are fast to cut ties. 

  Once the information about privacy violations becomes public, companies lose : 

  • Brand equity 
  • Market share 
  • Competitive positioning 

  An AON report estimates that post-data breach companies can lose as much as 25% of their initial value. In some cases, the losses can be even higher. 

  In 2015, British telecom TalkTalk suffered from a major data breach. Over 150,000 customer records were stolen by hackers. To contain the issue, TalkTalk had to throw between $60-$70 million into containment efforts. Still, they lost over 100,000 customers in a matter of months and one-third of their company value, equivalent to $1.4 billion, by the end of the year. 

  Fresher data from Infosys gives the following maximum cost estimates of brand damage, companies could experience after a data breach (accidental or malicious).

  

  3 Major Advantages of Privacy in Business 

  Despite all the industry mishaps, a reassuring 77% of CEOs now recognise that their companies must fundamentally change their approaches to customer engagement, in particular when it comes to ensuring data privacy. 

  Many organisations take proactive steps to cultivate a privacy-centred culture and implement transparent data collection policies. 

  Here’s why gaining the “privacy advantage” pays off.

  1. Market Competitiveness 

  There’s a reason why privacy-focused companies are booming. 

  Consumers’ mounting concerns and frustrations over the lack of online privacy, prompt many to look for alternative privacy-centred products and services. 

  The following B2C and B2B products are moving from the industry margins to the mainstream : 

  • Private search engines (Duckduckgo)
  • Password managers (1password, Dashlane)
  • Online identity networks (id.me)
  • Web analytics solutions (Matomo) 
  • And secure messaging apps (Signal) among others 

  Across the board, consumers express greater trust towards companies, protective of their privacy : 

  

  And as we well know : trust translates to higher engagement, loyalty, and – ultimately revenue. 

  By embedding privacy into the core of your product, you give users more reasons to select, stay and support your business. 

  2. Higher Operational Efficiency

  Customer data protection isn’t just a policy – it’s a culture of collecting “just enough” data, protecting it and using it responsibly. 

  Sadly, that’s the area where most organisations trail behind. At present, some 90% of businesses admit to having amassed massive data silos. 

  Siloed data is expensive to maintain and operationalise. Moreover, when left unattended, it can evolve into a pressing compliance issue. 

  A recently leaked document from Facebook says the company has no idea where all of its first-party, third-party and sensitive categories data goes or how it is processed. Because of this, Facebook struggles to achieve GDPR compliance and remains under regulatory pressure. 

  Similarly, Google Analytics is riddled with privacy issues. Other company products were found to be collecting and operationalising consumer data without users’ knowledge or consent. Again, this creates valid grounds for regulatory investigations. 

  Smaller companies have a better chance of making things right at the onset. 

  By curbing customer data collection, you can : 

  • Reduce data hosting and Cloud computation costs (aka trim your Cloud bill) 
  • Improve data security practices (since you would have fewer assets to protect) 
  • Make your staff more productive by consolidating essential data and making it easy and safe to access

  Privacy-mindful companies also have an easier time when it comes to compliance and can meet new data regulations faster. 

  3. Better Marketing Campaigns 

  The biggest counter-argument to reducing customer data collection is marketing. 

  How can we effectively sell our products if we know nothing about our customers ? – your team might be asking. 

  This might sound counterintuitive, but minimising data collection and usage can lead to better marketing outcomes. 

  Limiting the types of data that can be used encourages your people to become more creative and productive by focusing on fewer metrics that are more important.

  Think of it this way : Every other business uses the same targeting parameters on Facebook or Google for running paid ad campaigns on Facebook. As a result, we see ads everywhere — and people grow unresponsive to them or choose to limit exposure by using ad blocking software, private browsers and VPNs. Your ad budgets get wasted on chasing mirage metrics instead of actual prospects. 

  Case in point : In 2017 Marc Pritchard of Procter & Gamble decided to first cut the company’s digital advertising budget by 6% (or $200 million). Unilever made an even bolder move and reduced its ad budget by 30% in 2018. 

  Guess what happened ?

  P&G saw a 7.5% increase in organic sales and Unilever had a 3.8% gain as HBR reports. So how come both companies became more successful by spending less on advertising ? 

  They found that overexposure to online ads led to diminishing returns and annoyances among loyal customers. By minimising ad exposure and adopting alternative marketing strategies, the two companies managed to market better to new and existing customers. 

  The takeaway : There are more ways to engage consumers aside from pestering them with repetitive retargeting messages or creepy personalisation. 

  You can collect first-party data with consent to incrementally improve your product — and educate them on the benefits of your solution in transparent terms.

  Final Thoughts 

  The definitive advantage of privacy is consumers’ trust. 

  You can’t buy it, you can’t fake it, you can only cultivate it by aligning your external appearances with internal practices. 

  Because when you fail to address privacy internally, your mishaps will quickly become apparent either as social media call-outs or worse — as a security incident, a data breach or a legal investigation. 

  By choosing to treat consumer data with respect, you build an extra layer of protection around your business, plus draw in some banging benefits too. 

  Get one step closer to becoming a privacy-centred company by choosing Matomo as your web analytics solution. We offer robust privacy controls for ensuring ethical, compliant, privacy-friendly and secure website tracking. 

• Overthinking My Search Engine Problem

  31 décembre 2013, par Multimedia Mike — General

  I wrote a search engine for my Game Music Appreciation website, because the site would have been significantly less valuable without it (and I would eventually realize that the search feature is probably the most valuable part of this endeavor). I came up with a search solution that was a bit sketchy, but worked… until it didn’t. I thought of a fix but still searched for more robust and modern solutions (where ‘modern’ is defined as something that doesn’t require compiling a C program into a static CGI script and hoping that it works on a server I can’t debug on).

  Finally, I realized that I was overthinking the problem– did you know that a bunch of relational database management systems (RDBMSs) support full text search (FTS) ? Okay, maybe you did, but I didn’t know this.

  Problem Statement
  My goal is to enable users to search the metadata (title, composer, copyright, other tags) attached to various games. To do this, I want to index a series of contrived documents that describe the metadata. 2 examples of these contrived documents, interesting because both of these games have very different titles depending on region, something the search engine needs to account for :

  system : Nintendo NES
  game : Snoopy’s Silly Sports Spectacular
  author : None ; copyright : 1988 Kemco ; dumped by : None
  additional tags : Donald Duck.nsf Donald Duck
  system : Super Nintendo
  
  game : Arcana
  
  author : Jun Ishikawa, Hirokazu Ando ; copyright : 1992 HAL Laboratory ; dumped by : Datschge
  
  additional tags : card.rsn.gamemusic Card Master Cardmaster
  

  The index needs to map these documents to various pieces of game music and the search solution needs to efficiently search these documents and find the various game music entries that match a user’s request.

  Now that I’ve been looking at it for long enough, I’m able to express the problem surprisingly succinctly. If I had understood that much originally, this probably would have been simpler.

  First Solution & Breakage
  My original solution was based on SWISH-E. The CGI script was a C program that statically linked the SWISH-E library into a binary that miraculously ran on my web provider. At least, it ran until it decided to stop working a month ago when I added a new feature unrelated to search. It was a very bizarre problem, the details of which would probably bore you to tears. But if you care, the details are all there in the Stack Overflow question I asked on the matter.

  While no one could think of a direct answer to the problem, I eventually thought of a roundabout fix. The problem seemed to pertain to the static linking. Since I couldn’t count on the relevant SWISH-E library to be on my host’s system, I uploaded the shared library to the same directory as the CGI script and used dlopen()/dlsym() to fetch the functions I needed. It worked again, but I didn’t know for how long.

  Searching For A Hosted Solution
  I know that anything is possible in this day and age ; while my web host is fairly limited, there are lots of solutions for things like this and you can deploy any technology you want, and for reasonable prices. I figured that there must be a hosted solution out there.

  I have long wanted a compelling reason to really dive into Amazon Web Services (AWS) and this sounded like a good opportunity. After all, my script works well enough ; if I could just find a simple Linux box out there where I could install the SWISH-E library and compile the CGI script, I should be good to go. AWS has a free tier and I started investigating this approach. But it seems like a rabbit hole with a lot of moving pieces necessary for such a simple task.

  I had heard that AWS had something in this area. Sure enough, it’s called CloudSearch. However, I’m somewhat discouraged by the fact that it would cost me around $75 per month to run the smallest type of search instance which is at the core of the service.

  Finally, I came to another platform called Heroku. It’s supposed to be super-scalable while having a free tier for hobbyists. I started investigating FTS on Heroku and found this article which recommends using the FTS capabilities of their standard hosted PostgreSQL solution. However, the free tier of Postgres hosting only allows for 10,000 rows of data. Right now, my database has about 5400 rows. I expect it to easily overflow the 10,000 limit as soon as I incorporate the C64 SID music corpus.

  However, this Postgres approach planted a seed.

  RDBMS Revelation
  I have 2 RDBMSs available on my hosting plan– MySQL and SQLite (the former is a separate service while SQLite is built into PHP). I quickly learned that both have FTS capabilities. Since I like using SQLite so much, I elected to leverage its FTS functionality. And it’s just this simple :

  CREATE VIRTUAL TABLE gamemusic_metadata_fts USING fts3
  ( content TEXT, game_id INT, title TEXT ) ;
  SELECT id, title FROM gamemusic_metadata_fts WHERE content MATCH "arcana" ;
  
  479|Arcana
  

  The ‘content’ column gets the metadata pseudo-documents. The SQL gets wrapped up in a little PHP so that it queries this small database and turns the result into JSON. The script is then ready as a drop-in replacement for the previous script.