Recherche avancée

Sur d’autres sites (5404)

• Subtitling Sierra VMD Files

  1er juin 2016, par Multimedia Mike — Game Hacking

  I was contacted by a game translation hobbyist from Spain (henceforth known as The Translator). He had set his sights on Sierra’s 7-CD Phantasmagoria. This mammoth game was driven by a lot of FMV files and animations that have speech. These require language translation in the form of video subtitling. He’s lucky that he found possibly the one person on the whole internet who has just the right combination of skill, time, and interest to pull this off. And why would I care about helping ? I guess I share a certain camaraderie with game hackers. Don’t act so surprised. You know what kind of stuff I like to work on.

  The FMV format used in this game is VMD, which makes an appearance in numerous Sierra titles. FFmpeg already supports decoding this format. FFmpeg also supports subtitling video. So, ideally, all that’s necessary to support this goal is to add a muxer for the VMD format which can encode raw video and audio, which the format supports. Implement video compression as extra credit.

  The pipeline that I envisioned looks like this :

  
  

  

  VMD Subtitling Process

  
  

  “Trivial !” I surmised. I just never learn, do I ?

  The Plan
  So here’s my initial pitch, outlining the work I estimated that I would need to do towards the stated goal :

  1. Create a new file muxer that produces a syntactically valid VMD file with bogus video and audio data. Make sure it works with both FFmpeg’s playback system as well as the proper Phantasmagoria engine.
  2. Create a new video encoder that essentially operates in pass-through mode while correctly building a palette.
  3. Create a new basic encoder for the video frames.

  A big unknown for me was exactly how subtitle handling operates in FFmpeg. Thanks to this project, I now know. I was concerned because I was pretty sure that font rendering entails anti-aliasing which bodes poorly for keeping the palette count under 256 unique colors.

  Computer Science Puzzle
  When pondering how to process the palette, I was excited for the opportunity to exercise actual computer science. FFmpeg converts frames from paletted frames to full RGB frames. Then it needs to convert them back to paletted frames. I had a vague recollection of solving this problem once before when I was experimenting with a new paletted video codec. I seem to recall that I did the palette conversion in a very naive manner. I just used a static 256-element array and processed each RGB pixel of the frame, seeing if the value already occurred in the table (O(n) lookup) and adding it otherwise.
  

  There are more efficient algorithms, however, such as hash tables and trees. Somewhere along the line, FFmpeg helpfully acquired a rarely-used tree data structure, which was perfect for this project.

  So I was pretty pleased with this optimization. Too bad this wouldn’t survive to the end of the effort.

  Another palette-related challenge was the fact that a group of pictures would be accumulating a new palette but that palette needed to be recorded before the group. Thus, the muxer needed to have extra logic to rewind the file when the video encoder transmitted a palette change.

  Video Compression
  VMD has a few methods in its compression toolbox. It can use interframe differencing, it has some RLE, or it can code a frame raw. It can also use a custom LZ-like format on top of these. For early prototypes, I elected to leave each frame coded raw. After the concept was proved, I implemented the frame differencing.

  
  

  

  
  Top frame compared with the middle frame yields the bottom frame : red pixels indicate changes
  

  Encoding only those red dots in between vast runs of unchanged pixels yielded a vast measurable improvement. The next step was to try wiring up FFmpeg’s existing LZ compression facilities to the encoder. This turned out to be implausible since VMD’s LZ variant has nothing to do with anything FFmpeg already provides. Fortunately, the LZ piece is not absolutely required and the frame differencing + RLE provides plenty of compression.

  Subtitling
  I’ve never done anything, multimedia programming-wise, concerning subtitles. I guess all the entertainment I care about has always been in my native tongue. What a good excuse to program outside of my comfort zone !

  First, I needed to know how to access FFmpeg’s subtitling facilities. Fortunately, The Translator did the legwork on this matter so I didn’t have to figure it out.

  However, I intuitively had misgivings about this phase. I had heard that the subtitling process performs anti-aliasing. That means that the image would need to be promoted to a higher colorspace for this phase and that the anti-aliasing process would likely push the color count way past 256. Some quick tests revealed this to be the case, as the running color count would leap by several hundred colors as soon as the palette accounting algorithm encountered a subtitle.

  So I dug into the subtitle subsystem. I discovered that the subtitle library operates by creating a linked list of subtitle bitmaps that the client app must render. The bitmaps are comprised of 8-bit alpha transparency values that must be composited onto the target frame (i.e., 0 = transparent, 255 = 100% opaque). For example, the letter ‘H’ :

                                    (with 00s removed)
  13 F8 41 00 00 00 00 68 E4  |  13 F8 41             68 E4    
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF DC D0 D0 D0 D0 E4 EC  |  14 FF DC D0 D0 D0 D0 E4 EC
  14 FF 7E 50 50 50 50 9A EC  |  14 FF 7E 50 50 50 50 9A EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  11 E0 3B 00 00 00 00 5E CE  |  11 E0 3B             5E CE
  

  To get around the color explosion problem, I chose a threshold value and quantized values above and below to 255 and 0, respectively. Further, the process chooses an appropriate color from the existing palette rather than introducing any new colors.

  Muxing Matters
  In order to force VMD into a general purpose media framework, a lot of special information needs to be passed around. Like many paletted codecs, the palette needs to be transmitted from the file demuxer to the video decoder via some side channel. For re-encoding, this also implies that the palette needs to make the trip from the video encoder to the file muxer. As if this wasn’t enough, individual VMD frames have even more data that needs to be ferried between the muxer and codec levels, including frame change boundaries. FFmpeg provides methods to do these things, but I could not always rely on the systems to relay the data in all cases. I was probably doing something wrong ; I accept that. Instead, I just packed all the information at the front of an encoded frame and split it apart in the muxer.

  I could not quite figure out how to get the audio and video muxed correctly. As a result, neither FFmpeg nor the Phantasmagoria engine could replay the files correctly.

  Plan B
  Since I was having so much trouble creating an entirely new VMD file, likely due to numerous unknown bits of the file format, I thought of another angle : re-use the existing VMD file. For this approach, I kept the video encoder and file muxer that I created in the initial phase, but modified the file muxer to emit a special intermediate file. Then, I created a Python tool to repackage the original VMD file using compressed video data in the intermediate file.

  For this phase, I also implemented a command line switch for FFmpeg to disable subtitle blending, to make the feature feel like less of an unofficial hack, as though this nonsense would ever have a chance of being incorporated upstream.

  At this point, I was seeing some success with the complete, albeit roundabout, subtitling process. I constructed a subtitle file using “Spanish I Learned From Mexican Telenovelas” and the frames turned out fairly readable :

  
  

  

  “she cheated on him”

  
  

  

  “he’s a scumbag” … these random subtitles could fit surprisingly well !

  
  

  The few files that I tested appeared to work fine. But then I handed off my work to The Translator and he immediately found a bunch of problems. According to my notes, the problems mostly took the form of flashing, solid color frames. Further, I found tiny, mostly imperceptible flaws in my RLE compressor, usually only detectable by running strict comparison tools ; but I wasn’t satisfied.

  At this point, I think I attempted to just encode the entire palette at the front of each frame, as allowed by the format, but that did not seem to fix any problems. My notes are not completely clear on this matter (likely because I was still trying to figure out the exact problem), but I think it had to do with FFmpeg inserting extra video frames in order to even out gaps in the video framerate.

  Sigh, Plan C
  At this point, I was getting tired of trying to force FFmpeg to do this. So I decided to minimize its involvement using lessons learned up to this point.

  The next pitch :

  1. Create a new C program that can open an existing VMD file and output an identical VMD file. I know this sounds easy, but the specific method of copying entails interpreting individual parts of the file and writing those individual parts to the new file. This is in preparation for…
  2. Import the VMD video decoder functions directly into the program to decode the individual video frames and re-encode them, replacing the video frames as the file is rewritten.
  3. Wire up the subtitle system. During the adventure to disable subtitle blending, I accidentally learned enough about interfacing to the subtitle library to just invoke it directly.
  4. Rewrite the RLE method so that it is 100% correct.

  Off to work I went. That part about lifting the existing VMD decoder functions out of their libavcodec nest turned out to not be that straightforward. As an alternative, I modified the decoder to dump the raw frames to an intermediate file. In doing so, I think I was able to avoid the issue of the duplicated frames that plagued the previous efforts.

  Also, remember how I was really pleased with the palette conversion technique in which I was able to leverage computer science big-O theory ? By this stage, I had no reason to convert the paletted video to RGB in the first place ; all of the decoding, subtitling and re-encoding operates in the paletted colorspace.

  This approach seemed to work pretty well. The final program is subtitle-vmd.c. The process is still a little weird. The modifications in my own FFmpeg fork are necessary to create an intermediate file that the new C tool can operate with.

  Next Steps
  The Translator has found some assorted bugs and corner cases that still need to be ironed out. Further, for extra credit, I need find the change windows for each frame to improve compression just a little more. I don’t think I will be trying for LZ compression, though.

  However, almost as soon as I had this whole system working, The Translator informed me that there is another, different movie format in play in the Phantasmagoria engine called ROBOT, with an extension of RBT. Fortunately, enough of the algorithms have been reverse engineered and re-implemented in ScummVM that I was able to sort out enough details for another subtitling project. That will be the subject of a future post.

  See Also :

  • Subtitling Sierra RBT Files : The followup in which I discuss how to scribble text on the other animation format

  The post Subtitling Sierra VMD Files first appeared on Breaking Eggs And Making Omelettes.

• Piwik 1.12, New Features, API Improvements, Stability — The Last Piwik 1.X Release

  30 mai 2013, par Piwik team — Development

  We are very excited to announce the immediate availability of Piwik v1.12 !

  • Download Link
  • How to update Piwik ?
  • List of all tickets closed : Changelog

  Piwik v1.12 is a major new release with four big new features, seven smaller new features, several API improvements and all together 82 tickets fixed. This is also the last major 1.X release, which means after this release we will be working on releasing Piwik 2.0. This also means that you should upgrade to PHP 5.3 or higher if you haven’t already, since Piwik 2.0 will only support PHP 5.3 and above.

  Finally, this release contains two breaking changes to the API. If you use the Piwik API click here or scroll down to see if you’re affected.

  Table of Contents :

  1. New Big Feature – Beta Release Channel
  2. New Big Feature – Segment Editor
  3. New Big Feature – Page Speed Reports
  4. New Big Feature – Device Detection Reports
  5. Other Improvements
  6. API Changes and Improvements
  7. What we’re working on
  8. Participate in Piwik
  9. Summary

  New Big Feature – Beta Release Channel

  

  For those of you who want to help test Piwik 2.0-beta releases as soon as they come up, we’ve made it easier to use our beta releases. Navigate to the Settings > General Settings page and click the The latest beta release radio button. You will then be able to upgrade to beta releases.

  This isn’t truly a major feature, but we think it’s just as important because it will allow us to create more beta releases and thus catch more bugs before we make a final release. This means more releases and more stability for you.

  New Big Feature – Segment Editor

  

  The Segment Editor is a long-awaited new feature that allows you to view, save and edit your segments.

  Piwik has supported segmentation (filtering visits and reports by arbitrary criteria, like browser family) for quite some time now, but it has never been possible to visually create and modify them. Nor could they be saved for later recall.

  Thanks to the eighty individuals and company who funded this feature, it is now possible to :

  • visually segment your visitors, instead of creating URLs.
  • save segments and easily switch between them, instead of remembering URLs.
  • get suggestions for segments that might be helpful to view.
  • learn more in the Segmentating Analytics reports user documentation..

  New Big Feature – Page Speed Reports

  You can now see how long it took your webserver to generate and send pages over HTTP through the new Avg. Generation Time metric.

  This metric can be viewed on both the Pages and Page Titles reports :

  

  And the average page generation time for all the pages in your website/webapp is displayed on the visitors overview :

  

  You can use this new information to benchmark your webapp and web server.

  New Big Feature – Device Detection Reports

  Piwik 1.12 also includes a new plugin that provides reports on the device types (tablet, desktop, smartphone, etc.), device brands (Apple, Google, Samsung, etc.) and device models (iPad, Nexus 7, etc.) your visitors use to access your website :

  

  The new plugin also enhances Operating system detections (detecting sub versions of Linux, Windows, and more).

  Note : This plugin is not enabled by default, but will be in Piwik 2.0. If you want to view these reports now, you can activate the plugin in the Installed Plugins admin page. Navigate to Visitors > Devices to see the new reports. You may also use the new (beta) ‘Device type’.

  The new plugin was developed with the support of Clearcode.cc our technology partner

  

  Other improvements

  Majestic SEO Metrics

  

  We’ve added two new SEO metrics to the SEO widget, both of which are calculated by MajesticSEO.com. These metrics will tell you the number of external backlinks (the number of links to your site from other sites) and the number of referrer domains (the number of domains that link to your site).

  We thank the team at Majestic for their support and hard work in bringing you these metrics to your Piwik dashboards !

  Real-time Visitor Count Dashboard Widget

  

  There is now a simple new widget you can use to see the number of visitors, visits and actions that occurred in the last couple minutes. We call it the Real Time Visitor Counter !

  New segment parameter : siteSearchKeyword.

  There is now a new segment parameter you can use to segment your visits : siteSearchKeyword. This parameter will let you select visits that had site searches with a specific keyword.

  Ignore URL letter case when importing log files.

  We’ve added a new option to the log import script, –force-lowercase-path. When used, the importer will change URL paths to lowercase before tracking them. This way http://domain.com/MY/BLOG will be treated the same as http://domain.com/my/blog.

  Updated ISP Names

  

  We’ve also modified the Providers report so prettier and more up-to-date names of ISPs are displayed.

  Customize the background/text/axis color of graphs.

  

  It is now possible to change the background color, text color and/or axis color of the graph images generated by the ImageGraph plugin. To access this functionality, use the following URL query parameters when generating an image :

  • backgroundColor
  • textColor
  • axisColor

  For example :

  http://demo.piwik.org/index.php?module=API&method=ImageGraph.get&idSite=7&apiModule=UserSettings&apiAction=getBrowser&token_auth=anonymous&period=day&date=2013-03-21,2013-04-19&language=en&width=779&height=150&fontSize=9&showMetricTitle=0&aliasedGraph=1&legendAppendMetric=0&backgroundColor=efefef&gridColor=dcdcdc&colors=cb2026

  Send your users to a custom URL after they logout.

  If you manage a Piwik installation with many users and you want to send them to a custom page or website after they log out of Piwik, you can now specify the URL to redirect users after they log out.

  API Changes and Improvements

  BREAKING CHANGE – renamed segment parameters.

  The following segment parameters have been renamed :

  • continent renamed to : continentCode
  • browserName renamed to : browserCode
  • operatingSystem renamed to : operatingSystemCode
  • lat renamed to : latitude
  • long renamed to : longitude
  • region renamed to : regionCode
  • country renamed to : countryCode
  • continent renamed to : continentCode

  If you use one of the old segment parameter names, Piwik will throw an exception, so you should notice when you’re using an old name.

  BREAKING CHANGE – changes to the input & output of the Live.getLastVisitsDetails method.

  The following changes were made to the Live.getLastVisitsDetails API method :

  • The method no longer uses the maxIdVisit query parameter. It has been replaced by the filter_offset parameter.
  • Site search keywords are now displayed in a <siteSearchKeyword> element. They were formerly in <pageTitle> elements.
  • Custom variables with page scope now have ‘Page’ in their element names when displayed. For example, <customVariablePageName1>, <customVariablePageName2>, etc.

  Filter results of MultiSites.getAll by website name.

  It is now possible to filter the results of MultiSites.getAll by website name. To do this, set the pattern query parameter to the desired regex pattern.

  Get suggested values to use for a segment parameter.

  The new API method API.getSuggestedValuesForSegment can now be used to get suggested values for a segment parameter. This method will return a list of the most seen values (in the last 60 days) for a certain segment parameter. So for browserCode, this would return the codes for the browsers most visitors used in the last 60 days.

  Use extra tracking query parameters with the JS tracker (such as ‘lat’ & ‘long’).

  We’ve added a new method to the JavaScript tracker named appendToTrackingUrl. You can use this method to add extra query parameters to a tracking request, like so :

  _paq.push(['appendToTrackingUrl', 'lat=X&long=Y']);

  What we’re working on

  As we said above, Piwik v1.12 is the last in the 1.X series of releases. This means we are now officially working on Piwik 2.0.

  Piwik 2.0 will be a big release, to be sure, but it’s going to bring you more than just a couple new features and a bag of bug fixes. For Piwik 2.0 we will be revisiting the user needs and the ideals that originally prompted us to create Piwik in order to build our vision of the future of web analytics.

  Piwik 2.0 won’t just be a bigger, better web app, but a new platform for observing and analyzing the things that matter to you.

  Participate in Piwik

  Are you a talented developer or an experienced User Interface designer ? Or maybe you like to write documentation or are a marketing guru ?

  If you have some free time and if you want to contribute to one of the most awesome open source projects around, please get in touch with the Piwik team, or read this page to learn more…

  Summary

  For the full list of changes in Piwik 1.12 check out the Changelog.

  Thank you to the core developers, all the beta testers and users, our official supporters, the translators & everyone who reported bugs or feature requests. Also thank you to softwares we use, and the libraries we use.

  If you are a company and would like to help an important project like Piwik grow, please get in touch, it means a lot to us. You can also participate in the project —

  –> if you like what you read, please tell your friends and colleagues or write on your website, blog, forums, stackoverflow, etc. <–
  

  Peace. Enjoy !

• Custom Segmentation Guide : How it Works & Segments to Test

  13 novembre 2023, par Erin — Analytics Tips, Uncategorized

  Struggling to get the insights you’re looking for with premade reports and audience segments in your analytics ?

  Custom segmentation can help you better understand your customers, app users or website visitors, but only if you know what you’re doing.

  You can derive false insights with the wrong segments, leading your marketing campaigns or product development in the wrong direction.

  In this article, we’ll break down what custom segmentation is, useful custom segments to consider, how new privacy laws affect segmentation options and how to create these segments in an analytics platform.

  What is custom segmentation ?

  Custom segmentation is when you divide your audience (customers, users, website visitors) into bespoke segments of your own design, not premade segments designed by the analytics or marketing platform provider.

  To do this, you single out “custom segment input” — data points you will use to pinpoint certain users. For example, it could be everyone who has visited a certain page on your site.

  

  Segmentation isn’t just useful for targeting marketing campaigns and also for analysing your customer data. Creating segments is a great way to dive deeper into your data beyond surface-level insights.

  You can explore how various factors impact engagement, conversion rates, and customer lifetime value. These insights can help guide your higher-level strategy, not just campaigns.

  How custom segments can help your business

  As the global business world clamours to become more “data-driven,” even smaller companies collect all sorts of data on visitors, users, and customers.

  However, inexperienced organisations often become “data hoarders” without meaningful insights. They have in-house servers full of data or gigabytes stored by Google Analytics and other third-party providers.

  

  One way to leverage this data is with standard customer segmentation models. This can help you get insights into your most valuable customer groups and other standard segments.

  Custom segments, in turn, can help you dive deeper. They help you unlock insights into the “why” of certain behaviours. They can help you segment customers and your audience to figure out :

  • Why and how someone became a loyal customer
  • How high-order-value customers interact with your site before purchases
  • Which behaviours indicate audience members are likely to convert
  • Which traffic sources drive the most valuable customers

  This specific insight’s power led Gartner to predict that 70% of companies will shift focus from “big data” to “small and wide” by 2025. The lateral detail is what helps inform your marketing strategy. 

  You don’t need the same volume of data if you’re analysing and segmenting it effectively.

  Custom segment inputs : 6 data points you can use to create valuable custom segments 

  To help you get started, here are six useful data points you can use as a basis to create segments — AKA customer segment inputs :

  

  Visits to certain pages

  A basic data point that’s great for custom segments is visits to certain pages. Create segments for popular middle-of-funnel pages and compare their engagement and conversion rates. 

  For example, if a user visits a case study page, you can compare their likelihood to convert vs. other visitors.

  This is a type of behavioural segmentation, but it is the easiest custom segment to set up in terms of analysis and marketing efforts.

  Visitors who perform certain actions

  The other important type of behavioural segment is visitors or users who take certain actions. Think of things like downloading a file, clicking a link, playing a video or scrolling a certain amount.

  For instance, you can create a segment of all visitors who have downloaded a white paper. This can help you explore, for example, what drives someone to download a white paper. You can look at the typical user journey and make it easier for them to access the white paper — especially if your sales reps indicate many inbound leads mention it as a key driver of their interest.

  User devices

  Device-based segmentation lets you compare engagement and conversion rates on mobile, desktop and tablets. You can also get insights into their usage patterns and potential issues with certain mobile elements.

  

  This is one aspect of technographic segmentation, where you segment based on users’ hardware or software. You can also create segments based on browser software or even specific versions.

  Loyal or high-value customers

  The best way to get more loyal or high-value customers is to explore their journey in more detail. These types of segments can help you better understand your ideal customers and how they act on your site.

  You can then use this insight to alter your campaigns or how you communicate with your target audience.

  For example, you might notice that high-value customers tend to come from a certain source. You can then focus your marketing efforts on this source to reach more of your ideal customers.

  Visitor or customer source

  You need to track the results if you’re investing in marketing (like an influencer campaign or a sponsored post) outside platforms with their own analytics.

  

  Before you can create a reliable segment, you need to make sure that you use campaign tracking parameters to reliably track the source. You can use our free campaign tracking URL builder for that.

  Demographic segments — location (country, state) and more

  Web analytics tools, such as Matomo, use visitors’ IP addresses to pinpoint their location more accurately by cross-referencing with a database of known and estimated IP locations. In addition, these tools can detect a visitor’s location through the language settings in their browser. 

  This can help create segments based on location or language. By exploring these trends, you can identify patterns in behaviour, tailor your content to specific audiences, and adapt your overall strategy to better meet the preferences and needs of your diverse visitor base.

  How new privacy laws affect segmentation options

  Over the past few years, new legislation regarding privacy and customer data has been passed globally. The most notable privacy laws are the GDPR in the EU, the CCPA in California and the VCDPA in Virginia.

  

  For most companies, it can save a lot of work and future headaches to choose a GDPR-compliant web analytics solution not only streamlines operations, saving considerable effort and preventing future headaches, but also ensures peace of mind by guaranteeing the collection of compliant and accurate data. This approach allows companies to maintain compliance with privacy regulations while remaining firmly committed to a data-driven strategy.

  Create your very own custom segments in Matomo (while ensuring compliance and data accuracy)

  Crafting precise marketing messages and optimising ROI is crucial, but it becomes challenging without the right tools, especially when it comes to maintaining accurate data.

  That’s where Matomo comes in. Our privacy-friendly web analytics platform is GDPR-compliant and ensures accurate data, empowering you to effortlessly create and analyse precise custom segments.

  If you want to improve your marketing campaigns while remaining GDPR-compliant, start your 21-day free trial of Matomo. No credit card required.