Recherche avancée

Sur d’autres sites (4238)

• Developing MobyCAIRO

  26 mai 2021, par Multimedia Mike — General

  I recently published a tool called MobyCAIRO. The ‘CAIRO’ part stands for Computer-Assisted Image ROtation, while the ‘Moby’ prefix refers to its role in helping process artifact image scans to submit to the MobyGames database. The tool is meant to provide an accelerated workflow for rotating and cropping image scans. It works on both Windows and Linux. Hopefully, it can solve similar workflow problems for other people.

  As of this writing, MobyCAIRO has not been tested on Mac OS X yet– I expect some issues there that should be easily solvable if someone cares to test it.

  The rest of this post describes my motivations and how I arrived at the solution.

  Background
  I have scanned well in excess of 2100 images for MobyGames and other purposes in the past 16 years or so. The workflow looks like this :

  
  

  

  Image workflow

  
  

  It should be noted that my original workflow featured me manually rotating the artifact on the scanner bed in order to ensure straightness, because I guess I thought that rotate functions in image editing programs constituted dark, unholy magic or something. So my workflow used to be even more arduous :

  
  

  

  I can’t believe I had the patience to do this for hundreds of scans

  
  

  Sometime last year, I was sitting down to perform some more scanning and found myself dreading the oncoming tedium of straightening and cropping the images. This prompted a pivotal question :

  
  Why can’t a computer do this for me ?
  

  After all, I have always been a huge proponent of making computers handle the most tedious, repetitive, mind-numbing, and error-prone tasks. So I did some web searching to find if there were any solutions that dealt with this. I also consulted with some like-minded folks who have to cope with the same tedious workflow.

  I came up empty-handed. So I endeavored to develop my own solution.

  Problem Statement and Prior Work
  
  I want to develop a workflow that can automatically rotate an image so that it is straight, and also find the most likely crop rectangle, uniformly whitening the area outside of the crop area (in the case of circles).

  As mentioned, I checked to see if any other programs can handle this, starting with my usual workhorse, Photoshop Elements. But I can’t expect the trimmed down version to do everything. I tried to find out if its big brother could handle the task, but couldn’t find a definitive answer on that. Nor could I find any other tools that seem to take an interest in optimizing this particular workflow.

  When I brought this up to some peers, I received some suggestions, including an idea that the venerable GIMP had a feature like this, but I could not find any evidence. Further, I would get responses of “Program XYZ can do image rotation and cropping.” I had to tamp down on the snark to avoid saying “Wow ! An image editor that can perform rotation AND cropping ? What a game-changer !” Rotation and cropping features are table stakes for any halfway competent image editor for the last 25 or so years at least. I am hoping to find or create a program which can lend a bit of programmatic assistance to the task.

  Why can’t other programs handle this ? The answer seems fairly obvious : Image editing tools are general tools and I want a highly customized workflow. It’s not reasonable to expect a turnkey solution to do this.

  Brainstorming An Approach
  I started with the happiest of happy cases— A disc that needed archiving (a marketing/press assets CD-ROM from a video game company, contents described here) which appeared to have some pretty clear straight lines :

  
  
  

  My idea was to try to find straight lines in the image and then rotate the image so that the image is parallel to the horizontal based on the longest single straight line detected.

  I just needed to figure out how to find a straight line inside of an image. Fortunately, I quickly learned that this is very much a solved problem thanks to something called the Hough transform. As a bonus, I read that this is also the tool I would want to use for finding circles, when I got to that part. The nice thing about knowing the formal algorithm to use is being able to find efficient, optimized libraries which already implement it.

  Early Prototype
  A little searching for how to perform a Hough transform in Python led me first to scikit. I was able to rapidly produce a prototype that did some basic image processing. However, running the Hough transform directly on the image and rotating according to the longest line segment discovered turned out not to yield expected results.

  
  
  

  It also took a very long time to chew on the 3300×3300 raw image– certainly longer than I care to wait for an accelerated workflow concept. The key, however, is that you are apparently not supposed to run the Hough transform on a raw image– you need to compute the edges first, and then attempt to determine which edges are ‘straight’. The recommended algorithm for this step is the Canny edge detector. After applying this, I get the expected rotation :

  
  
  

  The algorithm also completes in a few seconds. So this is a good early result and I was feeling pretty confident. But, again– happiest of happy cases. I should also mention at this point that I had originally envisioned a tool that I would simply run against a scanned image and it would automatically/magically make the image straight, followed by a perfect crop.

  Along came my MobyGames comrade Foxhack to disabuse me of the hope of ever developing a fully automated tool. Just try and find a usefully long straight line in this :

  
  

  Darn it, Foxhack…
  

  There are straight edges, to be sure. But my initial brainstorm of rotating according to the longest straight edge looks infeasible. Further, it’s at this point that we start brainstorming that perhaps we could match on ratings badges such as the standard ESRB badges omnipresent on U.S. video games. This gets into feature detection and complicates things.

  This Needs To Be Interactive
  At this point in the effort, I came to terms with the fact that the solution will need to have some element of interactivity. I will also need to get out of my safe Linux haven and figure out how to develop this on a Windows desktop, something I am not experienced with.

  I initially dreamed up an impressive beast of a program written in C++ that leverages Windows desktop GUI frameworks, OpenGL for display and real-time rotation, GPU acceleration for image analysis and processing tricks, and some novel input concepts. I thought GPU acceleration would be crucial since I have a fairly good GPU on my main Windows desktop and I hear that these things are pretty good at image processing.

  I created a list of prototyping tasks on a Trello board and made a decent amount of headway on prototyping all the various pieces that I would need to tie together in order to make this a reality. But it was ultimately slowgoing when you can only grab an hour or 2 here and there to try to get anything done.

  Settling On A Solution
  Recently, I was determined to get a set of old shareware discs archived. I ripped the data a year ago but I was blocked on the scanning task because I knew that would also involve tedious straightening and cropping. So I finally got all the scans done, which was reasonably quick. But I was determined to not manually post-process them.

  This was fairly recent, but I can’t quite recall how I managed to come across the OpenCV library and its Python bindings. OpenCV is an amazing library that provides a significant toolbox for performing image processing tasks. Not only that, it provides “just enough” UI primitives to be able to quickly create a basic GUI for your program, including image display via multiple windows, buttons, and keyboard/mouse input. Furthermore, OpenCV seems to be plenty fast enough to do everything I need in real time, just with (accelerated where appropriate) CPU processing.

  So I went to work porting the ideas from the simple standalone Python/scikit tool. I thought of a refinement to the straight line detector– instead of just finding the longest straight edge, it creates a histogram of 360 rotation angles, and builds a list of lines corresponding to each angle. Then it sorts the angles by cumulative line length and allows the user to iterate through this list, which will hopefully provide the most likely straightened angle up front. Further, the tool allows making fine adjustments by 1/10 of an angle via the keyboard, not the mouse. It does all this while highlighting in red the straight line segments that are parallel to the horizontal axis, per the current candidate angle.

  
  
  

  The tool draws a light-colored grid over the frame to aid the user in visually verifying the straightness of the image. Further, the program has a mode that allows the user to see the algorithm’s detected edges :

  
  
  

  For the cropping phase, the program uses the Hough circle transform in a similar manner, finding the most likely circles (if the image to be processed is supposed to be a circle) and allowing the user to cycle among them while making precise adjustments via the keyboard, again, rather than the mouse.

  
  
  

  Running the Hough circle transform is a significantly more intensive operation than the line transform. When I ran it on a full 3300×3300 image, it ran for a long time. I didn’t let it run longer than a minute before forcibly ending the program. Is this approach unworkable ? Not quite– It turns out that the transform is just as effective when shrinking the image to 400×400, and completes in under 2 seconds on my Core i5 CPU.

  For rectangular cropping, I just settled on using OpenCV’s built-in region-of-interest (ROI) facility. I tried to intelligently find the best candidate rectangle and allow fine adjustments via the keyboard, but I wasn’t having much success, so I took a path of lesser resistance.

  Packaging and Residual Weirdness
  I realized that this tool would be more useful to a broader Windows-using base of digital preservationists if they didn’t have to install Python, establish a virtual environment, and install the prerequisite dependencies. Thus, I made the effort to figure out how to wrap the entire thing up into a monolithic Windows EXE binary. It is available from the project’s Github release page (another thing I figured out for the sake of this project !).

  The binary is pretty heavy, weighing in at a bit over 50 megabytes. You might advise using compression– it IS compressed ! Before I figured out the --onefile command for pyinstaller.exe, the generated dist/ subdirectory was 150 MB. Among other things, there’s a 30 MB FORTRAN BLAS library packaged in !

  Conclusion and Future Directions
  Once I got it all working with a simple tkinter UI up front in order to select between circle and rectangle crop modes, I unleashed the tool on 60 or so scans in bulk, using the Windows forfiles command (another learning experience). I didn’t put a clock on the effort, but it felt faster. Of course, I was livid with proudness the whole time because I was using my own tool. I just wish I had thought of it sooner. But, really, with 2100+ scans under my belt, I’m just getting started– I literally have thousands more artifacts to scan for preservation.

  The tool isn’t perfect, of course. Just tonight, I threw another scan at MobyCAIRO. Just go ahead and try to find straight lines in this specimen :

  
  
  

  I eventually had to use the text left and right of center to line up against the grid with the manual keyboard adjustments. Still, I’m impressed by how these computer vision algorithms can see patterns I can’t, highlighting lines I never would have guessed at.

  I’m eager to play with OpenCV some more, particularly the video processing functions, perhaps even some GPU-accelerated versions.

  The post Developing MobyCAIRO first appeared on Breaking Eggs And Making Omelettes.

• 9 Ways to Customise Your Matomo Like a Pro

  5 octobre 2022, par Erin

  Matomo is a feature-rich web analytics platform. As such, it has many layers of depth — core features, extra plug-ins, custom dimensions, reports, extensions and integrations. 

  Most of the product elements you see can be personalised and customised to your needs with minimal restrictions. However, this breadth of choice can be overlooked by new users. 

  In this post, we explain how to get the most out of Matomo with custom reports, dashboards, dimensions and even app design. 

  How to customise your Matomo web analytics

  To make major changes to Matomo (e.g., create custom dashboards or install new plugins), you’ll have to be a Matomo Super User (a.k.a. The Admin). Super Users can also grant administrator permissions to others so that more people could customise your Matomo deployment. 

  Most feature-related customisations (e.g. configuring a custom report, adding custom goal tracking, etc.) can be done by all users. 

  With the above in mind, here’s how you can tweak Matomo to better serve your website analytics needs : 

  1. Custom dashboards

  

  Dashboards provide a panorama view of all the collected website statistics. We display different categories of stats and KPIs as separate widgets — a standalone module you can also customise. 

  On your dashboard, you can change the type, position and number of widgets on display. This is an easy way to create separate dashboard views for different projects, clients or team members. Rather than a one-size-fits-all dashboard, a custom dashboard designed for a specific role or business unit will increase data-driven decision-making and efficiency across the business.

  You can create a new dashboard view in a few clicks. Then select a preferred layout — a split-page view or multi columns. Next, populate the new dashboard area with preferred widgets showing :

  • SEO website rankings 
  • Visits, conversions and revenue 
  • Custom reports 

  Or code a custom widget area to pull specific website stats or other reporting data you need. Once you are done, arrange everything with our drag-and-drop functionality. 

  

  Popular feature use cases

  • Personalised website statistics layout for convenient viewing 
  • Simplified analytics dashboards for the line of business leaders/stakeholders 
  • Project- or client-specific dashboards for easy report sharing 

  Read more about customising Matomo dashboards and widget areas. 

  2. Custom reports

  

  As the name implies, Custom Reports widget allows you to mesh any of the dimensions and metrics collected by Matomo into a custom website traffic analysis. Custom reports save users time by providing specific data needed in one view so there is no need to jump back and forth between multiple reports or toggle through a report to find data.

  For each custom report, you can select up to three dimensions and then apply additional quantitative measurements (metrics) to drill down into the data.

  For example, if you want to closely observe mobile conversion rates in one market, you can create the following custom report :

  • Dimensions : User Type (registered), Device type (mobile), Location (France)
  • Metrics : Visits, Conversion Rate, Revenue, Avg. Generation Time.

  Custom Report widget is available within Matomo Cloud and as a plugin for Matomo On-Premise.

  &lt;script type=&quot;text/javascript&quot;&gt;<br />
       if ('function' === typeof window.playMatomoVideo){<br />
       window.playMatomoVideo(&quot;custom_reports&quot;, &quot;#custom_reports&quot;)<br />
       } else {<br />
       document.addEventListener(&quot;DOMContentLoaded&quot;, function() { window.playMatomoVideo(&quot;custom_reports&quot;, &quot;#custom_reports&quot;); });<br />
       }<br />
  &lt;/script&gt;

  Popular feature use cases

  • Campaign-specific reporting to better understand the impact of different promo strategies 
  • Advanced event tracking for conversion optimization 
  • Market segmentation reports to analyse different audience cohorts 

  Read more about creating and analysing Custom Reports.

  3. Custom widgets

  

  We realise that our users have different degrees of analytics knowledge. Some love in-depth reporting dimensions and multi-row reporting tables. Others just want to see essential stats. 

  To delight both the pros and the novice users, we’ve created widgets — reporting sub-modules you can add, delete or rearrange in a few clicks. Essentially, a widget is a slice of a dashboard area you can populate with extra information. 

  You can add pre-made custom widgets to Matomo or develop your own widget to display custom reports or even external data (e.g., offline sales volume). At the same time, you can also embed Matomo widgets into other applications (e.g., a website CMS or corporate portal).

  Popular feature use cases

  • Display main goals (e.g., new trial sign-ups) on the main dashboard for greater visibility 
  • Highlight cost-per-conversion reporting by combining goals and conversion data to keep your budgets in check 
  • Run omnichannel eCommerce analytics (with embedded offline sales data) to get a 360-degree view into your operations 

  Read more about creating widgets in Matomo (beginner’s guide). 

  4. Custom dimensions 

  

  Dimensions describe the characteristics of reported data. Think of them as “filters” — a means to organise website analytics data by a certain parameter such as “Browser”, “Country”, “Device Type”, “User Type” and many more. 

  Custom Dimensions come in handy for all sorts of segmentation reports. For example, comparing conversion rates between registered and guest users. Or tracking revenue by device type and location. 

  For convenience, we’ve grouped Custom Dimensions in two categories :

  Visit dimensions. These associate metadata about a user with Visitor profiles — a summary of different knowledge you have about your audience. Reports for Visit scoped custom dimensions are available in the Visitors section of your dashboard. 

  Action dimensions. These segment users by specific actions tracked by Matomo such as pageviews, events completion, downloads, form clicks, etc. When configuring Custom Dimensions, you can select among pre-defined action types or code extra action dimensions. Action scoped custom dimensions are available in the Behaviours section of Matomo. 

  Depending on your Matomo version, you can apply 5 – 15 custom dimensions to reports. 

  Important : Since you can’t delete dimensions (only deactivate them), think about your use case first. Custom Dimensions each have their own dedicated reports page on your Matomo dashboard. 

  Popular custom dimension use cases among users :

  • Segmenting reports by users’ screen resolution size to understand how your website performs on different devices
  • Monitor conversion rates for different page types to determine your best-performing assets 

  Read more about creating, tracking and managing Custom Dimensions. 

  5. Custom scheduled reports

  Manually sending reports can be time consuming, especially if you have multiple clients or provide reports to numerous stakeholders. Custom scheduled reports remove this manual process to improve efficiency and ensure timely distribution of data to relevant users.

  Any report in Matomo (default or custom) can be shared with others by email as a PDF file, HTML content or as an attached CSV document. 

  You can customise which data you want to send to different people — your colleagues, upper management, clients or other company divisions. Then set up the frequency of email dispatches and Matomo will do the rest. 

  Auto-scheduling an email report is easy. Name your report, select a Segment (aka custom or standard report), pick time, file format and sender. 

  

  You can also share links to Matomo reports as text messages, if you are using ASPSMS or Clockwork SMS. 

  Popular feature use cases

  • Convenient stakeholder reporting on key website KPIs 
  • Automated client updates to keep clients informed and reduce workload 
  • Easy data downloads for doing custom analysis with business intelligence tools 

  Read more about email reporting features in Matomo. 

  6. Custom alerts

  

  Custom Alerts is a Matomo plugin for keeping you updated on the most important analytics events. Unlike Custom Reports, which provide a complete or segmented analytics snapshot, alerts are better suited for tracking individual events. For example, significant traffic increases from a specific channel, new 404 pages or major goal achievement (e.g., hitting 1,000 sales in a week). 

  Custom Alerts are a convenient way to keep your finger on the pulse of your site so you can quickly remedy an issue or get updated on reaching a crucial KPI promptly. You can receive custom alerts via email or text message in a matter of minutes.

  To avoid flooding your inbox with alerts, we recommend reserving Custom Alerts for a select few use cases (3 to 5) and schedule custom Email Reports to receive general web page analytics. 

  Popular custom alerts use cases among users :

  • Monitor sudden drops in revenue to investigate the cause behind them and solve any issues promptly 
  • Get notified of traffic spikes or sudden dips to better manage your website’s technical performance 

  Read more about creating and managing Custom Alerts. 

  7. Goals

  

  Goals feature helps you better understand how your website performs on certain business objectives such as lead generation, online sales or content discovery. A goal is a quantifiable action you want to measure (e.g., a specific page visit, form submission or a file download). 

  When combined together, Goals make up your sales funnel — a series of specific actions you expect users to complete in order to convert. 

  Goals-setting and Funnel Analytics are a powerful, customisable combo for understanding how people navigate your website ; what makes them take action or, on the contrary, lose interest and bounce off. 

  On Matomo, you can simultaneously track multiple goals, monitor multiple conversions per one visit (e.g., when one user requests two content downloads) and assign revenue targets to specific goals.

  &lt;script type=&quot;text/javascript&quot;&gt;<br />
       if ('function' === typeof window.playMatomoVideo){<br />
       window.playMatomoVideo(&quot;goals&quot;, &quot;#goals&quot;)<br />
       } else {<br />
       document.addEventListener(&quot;DOMContentLoaded&quot;, function() { window.playMatomoVideo(&quot;goals&quot;, &quot;#goals&quot;); });<br />
       }<br />
  &lt;/script&gt;

  Separately, Matomo Cloud users also get access to a premium Funnels feature and Multi Channel Conversion Attribution. On-Premises Matomo users can get both as paid plugins via our Marketplace.

  Popular goal tracking use cases among users :

  • Tracking newsletter subscription to maximise subscriber growth 
  • Conversion tracking for gated content (e.g., eBooks) to understand how each asset performs 
  • Analysing the volume of job applications per post to better interpret your HR marketing performance 

  Read more about creating and managing Goals in Matomo.

  8. Themes

  

  Want to give your Matomo app a distinctive visual flair ? Pick a new free theme for your On-Premises installation. Minimalistic, dark or classic — our community created six different looks that other Matomo users can download and install in a few clicks. 

  If you have some HTML/CSS/JS knowledge, you can also design your own Matomo theme. Since Matomo is an open-source project, we don’t restrict interface customisation and always welcome creativity from our users.

  Read more about designing your own Matomo theme (developer documentation).

  9. White labelling

  

  Matomo is one of the few website analytics tools to support white labelling. White labelling means that you can distribute our product to others under your brand. 

  For example, as a web design agency, you can delight customers with pre-installed GDPR-friendly website analytics. Marketing services providers, in turn, can present their clients with embedded reporting widgets, robust funnel analytics and 100% unsampled data. 

  Apart from selecting a custom theme, you can also align Matomo with your brand by :

  • Customising product name
  • Using custom header/font colours 
  • Change your tracking endpoint
  • Remove links to Matomo.org

  To streamline Matomo customisation and set-up, we developed a White Label plug-in. It provides a convenient set of controls for changing your Matomo deployment and distributing access rights to other users or sharing embedded Matomo widgets). 

  Read more about white labelling Matomo. 

  Learning more about Matomo 

  Matomo has an ever-growing list of features, ranging from standard website tracking controls to unique conversion rate optimisation tools (heatmaps, media analytics, user cohorts and more).

  To learn more about Matomo features you can check our free video web analytics training series where we cover the basics. For feature-specific tips, tricks and configurations, browse our video content or written guides. 

  21 day free trial. No credit card required.