Recherche avancée

Sur d’autres sites (4147)

• Developing MobyCAIRO

  26 mai 2021, par Multimedia Mike — General

  I recently published a tool called MobyCAIRO. The ‘CAIRO’ part stands for Computer-Assisted Image ROtation, while the ‘Moby’ prefix refers to its role in helping process artifact image scans to submit to the MobyGames database. The tool is meant to provide an accelerated workflow for rotating and cropping image scans. It works on both Windows and Linux. Hopefully, it can solve similar workflow problems for other people.

  As of this writing, MobyCAIRO has not been tested on Mac OS X yet– I expect some issues there that should be easily solvable if someone cares to test it.

  The rest of this post describes my motivations and how I arrived at the solution.

  Background
  I have scanned well in excess of 2100 images for MobyGames and other purposes in the past 16 years or so. The workflow looks like this :

  
  

  

  Image workflow

  
  

  It should be noted that my original workflow featured me manually rotating the artifact on the scanner bed in order to ensure straightness, because I guess I thought that rotate functions in image editing programs constituted dark, unholy magic or something. So my workflow used to be even more arduous :

  
  

  

  I can’t believe I had the patience to do this for hundreds of scans

  
  

  Sometime last year, I was sitting down to perform some more scanning and found myself dreading the oncoming tedium of straightening and cropping the images. This prompted a pivotal question :

  
  Why can’t a computer do this for me ?
  

  After all, I have always been a huge proponent of making computers handle the most tedious, repetitive, mind-numbing, and error-prone tasks. So I did some web searching to find if there were any solutions that dealt with this. I also consulted with some like-minded folks who have to cope with the same tedious workflow.

  I came up empty-handed. So I endeavored to develop my own solution.

  Problem Statement and Prior Work
  
  I want to develop a workflow that can automatically rotate an image so that it is straight, and also find the most likely crop rectangle, uniformly whitening the area outside of the crop area (in the case of circles).

  As mentioned, I checked to see if any other programs can handle this, starting with my usual workhorse, Photoshop Elements. But I can’t expect the trimmed down version to do everything. I tried to find out if its big brother could handle the task, but couldn’t find a definitive answer on that. Nor could I find any other tools that seem to take an interest in optimizing this particular workflow.

  When I brought this up to some peers, I received some suggestions, including an idea that the venerable GIMP had a feature like this, but I could not find any evidence. Further, I would get responses of “Program XYZ can do image rotation and cropping.” I had to tamp down on the snark to avoid saying “Wow ! An image editor that can perform rotation AND cropping ? What a game-changer !” Rotation and cropping features are table stakes for any halfway competent image editor for the last 25 or so years at least. I am hoping to find or create a program which can lend a bit of programmatic assistance to the task.

  Why can’t other programs handle this ? The answer seems fairly obvious : Image editing tools are general tools and I want a highly customized workflow. It’s not reasonable to expect a turnkey solution to do this.

  Brainstorming An Approach
  I started with the happiest of happy cases— A disc that needed archiving (a marketing/press assets CD-ROM from a video game company, contents described here) which appeared to have some pretty clear straight lines :

  
  
  

  My idea was to try to find straight lines in the image and then rotate the image so that the image is parallel to the horizontal based on the longest single straight line detected.

  I just needed to figure out how to find a straight line inside of an image. Fortunately, I quickly learned that this is very much a solved problem thanks to something called the Hough transform. As a bonus, I read that this is also the tool I would want to use for finding circles, when I got to that part. The nice thing about knowing the formal algorithm to use is being able to find efficient, optimized libraries which already implement it.

  Early Prototype
  A little searching for how to perform a Hough transform in Python led me first to scikit. I was able to rapidly produce a prototype that did some basic image processing. However, running the Hough transform directly on the image and rotating according to the longest line segment discovered turned out not to yield expected results.

  
  
  

  It also took a very long time to chew on the 3300×3300 raw image– certainly longer than I care to wait for an accelerated workflow concept. The key, however, is that you are apparently not supposed to run the Hough transform on a raw image– you need to compute the edges first, and then attempt to determine which edges are ‘straight’. The recommended algorithm for this step is the Canny edge detector. After applying this, I get the expected rotation :

  
  
  

  The algorithm also completes in a few seconds. So this is a good early result and I was feeling pretty confident. But, again– happiest of happy cases. I should also mention at this point that I had originally envisioned a tool that I would simply run against a scanned image and it would automatically/magically make the image straight, followed by a perfect crop.

  Along came my MobyGames comrade Foxhack to disabuse me of the hope of ever developing a fully automated tool. Just try and find a usefully long straight line in this :

  
  

  Darn it, Foxhack…
  

  There are straight edges, to be sure. But my initial brainstorm of rotating according to the longest straight edge looks infeasible. Further, it’s at this point that we start brainstorming that perhaps we could match on ratings badges such as the standard ESRB badges omnipresent on U.S. video games. This gets into feature detection and complicates things.

  This Needs To Be Interactive
  At this point in the effort, I came to terms with the fact that the solution will need to have some element of interactivity. I will also need to get out of my safe Linux haven and figure out how to develop this on a Windows desktop, something I am not experienced with.

  I initially dreamed up an impressive beast of a program written in C++ that leverages Windows desktop GUI frameworks, OpenGL for display and real-time rotation, GPU acceleration for image analysis and processing tricks, and some novel input concepts. I thought GPU acceleration would be crucial since I have a fairly good GPU on my main Windows desktop and I hear that these things are pretty good at image processing.

  I created a list of prototyping tasks on a Trello board and made a decent amount of headway on prototyping all the various pieces that I would need to tie together in order to make this a reality. But it was ultimately slowgoing when you can only grab an hour or 2 here and there to try to get anything done.

  Settling On A Solution
  Recently, I was determined to get a set of old shareware discs archived. I ripped the data a year ago but I was blocked on the scanning task because I knew that would also involve tedious straightening and cropping. So I finally got all the scans done, which was reasonably quick. But I was determined to not manually post-process them.

  This was fairly recent, but I can’t quite recall how I managed to come across the OpenCV library and its Python bindings. OpenCV is an amazing library that provides a significant toolbox for performing image processing tasks. Not only that, it provides “just enough” UI primitives to be able to quickly create a basic GUI for your program, including image display via multiple windows, buttons, and keyboard/mouse input. Furthermore, OpenCV seems to be plenty fast enough to do everything I need in real time, just with (accelerated where appropriate) CPU processing.

  So I went to work porting the ideas from the simple standalone Python/scikit tool. I thought of a refinement to the straight line detector– instead of just finding the longest straight edge, it creates a histogram of 360 rotation angles, and builds a list of lines corresponding to each angle. Then it sorts the angles by cumulative line length and allows the user to iterate through this list, which will hopefully provide the most likely straightened angle up front. Further, the tool allows making fine adjustments by 1/10 of an angle via the keyboard, not the mouse. It does all this while highlighting in red the straight line segments that are parallel to the horizontal axis, per the current candidate angle.

  
  
  

  The tool draws a light-colored grid over the frame to aid the user in visually verifying the straightness of the image. Further, the program has a mode that allows the user to see the algorithm’s detected edges :

  
  
  

  For the cropping phase, the program uses the Hough circle transform in a similar manner, finding the most likely circles (if the image to be processed is supposed to be a circle) and allowing the user to cycle among them while making precise adjustments via the keyboard, again, rather than the mouse.

  
  
  

  Running the Hough circle transform is a significantly more intensive operation than the line transform. When I ran it on a full 3300×3300 image, it ran for a long time. I didn’t let it run longer than a minute before forcibly ending the program. Is this approach unworkable ? Not quite– It turns out that the transform is just as effective when shrinking the image to 400×400, and completes in under 2 seconds on my Core i5 CPU.

  For rectangular cropping, I just settled on using OpenCV’s built-in region-of-interest (ROI) facility. I tried to intelligently find the best candidate rectangle and allow fine adjustments via the keyboard, but I wasn’t having much success, so I took a path of lesser resistance.

  Packaging and Residual Weirdness
  I realized that this tool would be more useful to a broader Windows-using base of digital preservationists if they didn’t have to install Python, establish a virtual environment, and install the prerequisite dependencies. Thus, I made the effort to figure out how to wrap the entire thing up into a monolithic Windows EXE binary. It is available from the project’s Github release page (another thing I figured out for the sake of this project !).

  The binary is pretty heavy, weighing in at a bit over 50 megabytes. You might advise using compression– it IS compressed ! Before I figured out the --onefile command for pyinstaller.exe, the generated dist/ subdirectory was 150 MB. Among other things, there’s a 30 MB FORTRAN BLAS library packaged in !

  Conclusion and Future Directions
  Once I got it all working with a simple tkinter UI up front in order to select between circle and rectangle crop modes, I unleashed the tool on 60 or so scans in bulk, using the Windows forfiles command (another learning experience). I didn’t put a clock on the effort, but it felt faster. Of course, I was livid with proudness the whole time because I was using my own tool. I just wish I had thought of it sooner. But, really, with 2100+ scans under my belt, I’m just getting started– I literally have thousands more artifacts to scan for preservation.

  The tool isn’t perfect, of course. Just tonight, I threw another scan at MobyCAIRO. Just go ahead and try to find straight lines in this specimen :

  
  
  

  I eventually had to use the text left and right of center to line up against the grid with the manual keyboard adjustments. Still, I’m impressed by how these computer vision algorithms can see patterns I can’t, highlighting lines I never would have guessed at.

  I’m eager to play with OpenCV some more, particularly the video processing functions, perhaps even some GPU-accelerated versions.

  The post Developing MobyCAIRO first appeared on Breaking Eggs And Making Omelettes.

• Google Analytics Now Illegal in Austria ; Other EU Member States Expected to Follow

  18 janvier 2022, par Erin — Privacy

  Breaking news : The Austrian Data Protection Authority (“Datenschutzbehörde” or “DSB” or “DPA”) has ruled that Austrian website providers using Google Analytics are in violation of the GDPR. 

  This ruling stems from a decision made in 2020 by the Court of Justice of the European Union (CJEU) that stated that cloud services hosted in the US are incapable of complying with the GDPR and EU privacy laws. The decision was made because of the US surveillance laws requiring US providers (like Google or Facebook) to provide personal data to US authorities. 

  The 2020 ruling, known as “Schrems II”, marked the ending of the Privacy Shield, a framework that allowed for EU data to be transferred to US companies that became certified. 

  The tech industry was sent into a frenzy following this decision, but many US and EU companies decided to ignore the case. The choice to ignore is what landed one Austrian business in the DPA’s line of fire, damaging the brand’s reputation and possibly resulting in a hefty fine of up to €20 million or 4% of the organisation’s global turnover. 

  About the Austrian DPA’s Model Case 

  In this specific case, noyb (the European Center for Digital Rights) found that IP addresses (which are classified as personal data by the GDPR) and other identifiers were sent to the US in cookie data as a result of the organisation using Google Analytics. 

  This model case led to the DPA’s decision to rule that Austrian website providers using Google Analytics are in violation of GDPR. It is believed that other EU Member States will soon follow in this decision as well.

  "We expect similar decisions to now drop gradually in most EU member states. We have filed 101 complaints in almost all Member States and the authorities coordinated the response. A similar decision was also issued by the European Data Protection Supervisor last week."

  What does this mean if you are using Google Analytics ?

  If there is one thing to learn from this case, it is that ignoring these court rulings and continuing to use Google Analytics is not a viable option. 

  If you are operating a website in Austria, or your website services Austrian citizens, you should remove Google Analytics from your website immediately. 

  For businesses in other EU Member States, it is also highly recommended that you take action before noyb and local data protection authorities start targeting more businesses. 

  "Instead of actually adapting services to be GDPR compliant, US companies have tried to simply add some text to their privacy policies and ignore the Court of Justice. Many EU companies have followed the lead instead of switching to legal options."

  Removing Google Analytics from your site doesn’t mean that you need to give up website analytics altogether though. There are a variety of Google Analytics alternatives available today. Matomo in particular is a powerful open-source web analytics platform that gives you 100% data ownership and GDPR compliance. 

  

  Matomo is one of the best Google Analytics alternatives offering privacy by design on our Cloud, On-Premise and Matomo for WordPress. So you can get the insights you need while remaining compliant. As the GDPR continues to evolve, you can rest assured that Matomo will be at the forefront of these changes. 

  In addition, all Google Analytics data can be imported into Matomo so no historical data is lost. To make your migration as seamless as possible, we’ve put together a guide to migrating from Google Analytics to Matomo. 

  Ready to begin your journey to GDPR compliance ? Check out our live demo and start your 21-day free trial now – no credit card required.

  If you are interested in learning more about GDPR compliance and Matomo, check out our GDPR resources below :    

  • GDPR Manager user guide
  • How to fill in the information asset registration when using Matomo
  • How to make Matomo GDPR compliant in 12 easy steps
  • How to not process any personal data with Matomo and what it means for you
  • PII, personal data and GDPR. Discover what this means for your privacy
  • How should I write my privacy notice for Matomo Analytics under GDPR
  • Lawful basis for processing personal data under GDPR with Matomo
  • How to complete your privacy policy with Matomo Analytics under GDPR
  • GDPR compliance for Premium Features
  • How to get your Matomo plugin ready for GDPR
  • Subprocessors on Matomo Cloud (subprocessors not applicable to On-Premise)
  • Privacy in Matomo

  What does this mean if you are using Matomo ? 

  Our users can rest assured that Matomo remains in compliance with GDPR as all data is stored in the EU (Matomo Cloud) or in any country of your choice (Matomo On-Premise). With Matomo you’re able to continue analysing your website and not worry about GDPR.

  Final thoughts

  For EU businesses operating websites, now is the time to act. While Google pushes out false narratives to try and convince users that it is safe to continue using Google Analytics, it’s clear from these court rulings that the data protection authorities across the EU disagree with Google’s narrative.

  The fines, reputational damage and stresses mounting from using Google Analytics are imminent. Find an alternative to Google Analytics as this problem is not going away. 

  Getting started with Matomo is easy. Make the switch today and start your free 21-day trial – no credit card required. 

• Google Analytics 4 (GA4) vs Matomo

  7 avril 2022, par Erin

  Google announced that Universal Analytics’ days are numbered. Universal Analytics will be replaced by Google Analytics 4 (or GA4) on the 1st of July 2023. 

  If Google Analytics users want to compare year-on-year data, they have until July 2022 to get set up and start collecting data before the sun sets on Universal Analytics (or UA).

  But is upgrading to Google Analytics 4 the right move ? There’s a lot to consider, and many organisations are looking for an alternative to Google Analytics. So in this blog, we’ll compare GA4 to Matomo – the leading Google Analytics alternative. 

  In this blog, we’ll look at :

  What is Matomo ?

  Matomo is a powerful privacy-first web analytics platform that gives you 100% data ownership. First launched in 2007, Matomo is now the world’s leading open-source web analytics platform and is used by more than 1 million websites. 

  Matomo’s core values are based on ethical data collection and processing. Consistently more businesses and organisations from around the globe are adopting data-privacy-compliant web analytics solutions like Matomo. 

  Matomo offers both Cloud and On-Premise solutions (and a five-star rated WordPress plugin), making for an adaptable and flexible solution. 

  What is Google Analytics 4 ?

  Google Analytics 4 is the latest version of Google Analytics and represents a completely new approach to data-modelling than its predecessor, Universal Analytics. For an in-depth look at how GA4 and UA compare, check out this Google Analytics 4 vs Universal Analytics comparison. 

  Google Analytics 4 will soon be the only available version of analytics software from Google. So what’s the issue ? Surely, in 2022, Google makes it easy to migrate to their newest (and only) analytics platform ? Not quite.

  Google Analytics 4 vs Matomo

  Whilst the core purpose of GA4 and Matomo is similar (providing web analytics that help to optimise your website and grow your business), there are several key differences that organisations should consider before making the switch.

  Importing Historical Data from Universal Analytics

  Google Analytics 4

  Users assuming that historical data from Universal Analytics could be imported into Google Analytics 4 were faced with swift disappointment. Unfortunately, Google Analytics 4 does not have an option to import data from its predecessor, Universal Analytics. This means that businesses won’t be able to import and compare data from previous years.

  Matomo

  If you don’t want to start from scratch with your web analytics data, then Matomo is an ideal solution for data continuity. Matomo offers users the ability to import their historical Universal Analytics data. So you can keep all that valuable historical data you’ve collected over the years.

  

  User Interface

  Google Analytics 4

  GA4’s new user interface has been met with mixed reviews. Many claim that it’s overly complex and difficult to navigate. Some have even suggested that the tool has been designed specifically for enterprises with specialised analytics teams. 

  

  Matomo

  Matomo, on the other hand, is recognised for an easy to use interface, with a rating of 4.5 out of 5 stars for ease of use on Capterra. Matomo perfectly balances powerful features with a user-friendly interface so valuable insights are only a click away. There’s a reason why over 1 million websites are using Matomo. 

  

  Advanced Behavioural Analytics Features 

  Google Analytics 4

  While Google Analytics is undoubtedly robust in some areas (machine learning, for instance), what it really lacks is advanced behavioural analytics. Heatmaps, session recordings and other advanced tools can give you valuable insights into how users are engaging with your site. Well beyond pageviews and other metrics.

  Unfortunately, with this new generation of GA, Google still hasn’t introduced these features. So users have to manage subscriptions and tracking in third-party behavioural analytics tools like Hotjar or Lucky Orange, for example. This is inefficient, costly and time-consuming to manage. 

  

  Matomo 

  Meanwhile, Matomo is a one-stop shop for all of your web analytics needs. Not only do you get access to the metrics you’ve grown accustomed to with Universal Analytics, but you also get built-in behavioural analytics features like Heatmaps, Scroll Depth, Session Recordings and more. 

  Want to know if visitors are reaching your call to action at the bottom of the page ? Scroll Depth will answer that.

  Want to know why visitors aren’t clicking through to the next page ? Heatmaps will give you the insights you need.

  You get the picture – the full picture, that is. 

  

  Data Accuracy

  Google Analytics 4

  GA4 aims to make web and app analytics more privacy-centric by reducing the reliance on cookies to record certain events across platforms and devices. 

  However, when site and application visitors opt-out of cookie tracking, GA4 instead relies on machine learning to fill in the gaps. Data sampling could mean that your business is making business decisions based on inaccurate reports. 

  Matomo

  Data is the backbone of web analytics, so why make critical business decisions on sampled data ? With Matomo, you’re guaranteed 100% unsampled accurate data. So you can rest assured that any decisions you make are based on actual facts. 

  Compliance with Privacy Laws (GDPR, CCPA, etc.) 

  Google Analytics 4

  Google is making changes in an attempt to become compliant with privacy laws. However, even with GA4, users are still transferring data to the US. For this reason, both Austrian and French governments have ruled Google Analytics illegal under GDPR.

  The only possible workaround is “Privacy Shield 2.0”, but GDPR experts are still sceptical of this one. 

  Matomo

  If compliance with global privacy laws is a concern (and it should be), then Matomo is the clear winner here. 

  As an EU hosted web analytics tool, your data is stored in Europe, and no data is transferred to the US. On the other hand, if you choose to self-host, the data is stored in your country of choice.

  In addition, with cookieless tracking enabled, you can say goodbye to those pesky cookie consent screens. 

  Also, remember that under GDPR, and many other data privacy laws like CCPA and LGPD, end users have a legal right to access, amend and/or erase the personal data collected about them. 

  With Matomo you get 100% ownership of your web analytics data. This means that we don’t on-sell to third parties ; can’t claim ownership of the data ; and you can export your data at any time.

  

  Wrap up

  At the end of the day, the worst thing an organisation can do is nothing. Waiting until July 2023 to migrate to GA4 or another web analytics platform would be very disruptive and costly. Organisations need to consider their options now and start migrating in the next few months. 

  With all that said, moving to Google Analytics 4 could prove to be a costly and time-consuming operation. The global trend towards increased data privacy is a threat to platforms like Google Analytics which uses data for advertising and transfers data across borders.

  With Matomo, you get an easy to use all-in-one web analytics platform and keep your historical Universal Analytics data. Plus, you can future-proof your business by being compliant with global privacy laws and get access to advanced behavioural analytics features. 

  There’s a lot to weigh up here but fortunately, getting started with Matomo is easy. Try it free for 21-days (no credit card required) and see for yourself why over 1 million websites choose Matomo. 

  While this is the end of the road for Universal Analytics, it’s also an opportune time for organisations to find a better fit web analytics tool. 

  21 day free trial. No credit card required.

  Your email address
  

  Your website address
  

  Your Analytics subdomain will be .matomo.cloud (change)

  Choose your analytics subdomain
  

  .matomo.cloud

  I hereby accept the terms and conditions and the data processing agreeement (DPA).

  
  Your information will be used to create an account on our cloud service. For more information please consult our privacy policy.
  
  

  » Start improving your websites now