Recherche avancée

Sur d’autres sites (2716)

• A Beginner’s Guide to Omnichannel Analytics

  14 avril 2024, par Erin

  Linear customer journeys are as obsolete as dial-up internet and floppy disks. As a marketing manager, you know better than anyone that customers interact with your brand hundreds of times across dozens of channels before purchasing. That can make tracking them a nightmare unless you build an omnichannel analytics solution. 

  Alas, if only it were that simple. 

  Unfortunately, it’s not enough to collect data on your customers’ complex journeys just by buying an omnichannel platform. You need to generate actionable insights by using marketing attribution to tie channels to conversions. 

  This article will explain how to build a useful omnichannel analytics solution that lets you understand and improve the customer journey.

  What is omnichannel analytics ?

  Omnichannel analytics collects and analyses customer data from every touchpoint and device. The goal is to collect all this omnichannel data in one place, creating a single, real-time, unified view of your customer’s journey.

  

  Unfortunately, most businesses haven’t achieved this yet. As Karen Lellouche Tordjman and Marco Bertini say :

  “Despite all the buzz around the concept of omnichannel, most companies still view customer journeys as a linear sequence of standardised touchpoints within a given channel. But the future of customer engagement transforms touchpoints from nodes along a predefined distribution path to full-blown portals that can serve as points of sale or pathways to many other digital and virtual interactions. They link to chatbots, kiosks, robo-advisors, and other tools that customers — especially younger ones — want to engage with.”

  However, doing so is more important than ever — especially when consumers have over 300 digital touchpoints, and the average number of touchpoints in the B2B buyer journey is 27.

  Not only that, but customers expect personalised experiences across every platform — that’s the kind you can only create when you have access to omnichannel data.

  

  What might omnichannel analytics look like in practice for an e-commerce store ?

  An online store would integrate data from channels like its website, mobile app, social media accounts, Google Ads and customer service records. This would show how customers find its brand, how they use each channel to interact with it and which channels convert the most customers. 

  This would allow the e-commerce store to tailor marketing channels to customers’ needs. For instance, they could focus social media use on product discovery and customer support. Google Ads campaigns could target the best-converting products. While all this is happening, the store could also ensure every channel looks the same and delivers the same experience. 

  What are the benefits of omnichannel analytics ?

  Why go to all the trouble of creating a comprehensive view of the customer’s experience ? Because you stand to gain some pretty significant benefits when implementing omnichannel analytics.

  

  Understand the customer journey

  You want to understand how your customers behave, right ? No other method will allow you to fully understand your customer journey the way omnichannel analytics does. 

  It doesn’t matter how customers engage with your brand — whether that’s your website, app, social media profiles or physical stores — omnichannel analytics capture every interaction.

  With this 360-degree view of your customers, it’s easy to understand how they move between channels, where they encounter issues and what bottlenecks prevent them from converting. 

  Deliver better personalisation

  We don’t have to tell you that personalisation matters. But do you know just how important it is ? Since 56% of customers will become repeat buyers after a personalised experience, delivering them as often as possible is critical. 

  Omnichannel analytics helps in your quest for personalisation by highlighting the individual preferences of customer segments. For example, e-commerce stores can use omnichannel analytics to understand how shoppers behave across different devices and tailor their offers accordingly. 

  Upgrade the customer experience

  Omnichannel analytics gives you the insights to improve every aspect of the customer experience. 

  For starters, you can ensure a consistent brand experience across all your top channels by making sure they look and behave the same.

  Then, you can use omnichannel insights to tailor each channel to your customers’ requirements. For example, most people interacting with your brand on social media may seek support. Knowing that you can create dedicated support accounts to assist users. 

  Improve marketing campaigns

  Which marketing campaigns or traffic sources convert the most customers ? How can you improve these campaigns ? Omnichannel analytics has the answers. 

  When you implement omnichannel analytics you automatically track the performance of every marketing channel by attributing each conversion to one or more traffic sources. This lets you see whether Google Ads bring in more customers than your SEO efforts. Or whether social media ads are the most profitable acquisition channel. 

  Armed with this information, you can improve your marketing efforts — either by focusing on your profitable channels or rectifying problems that stop less profitable channels from converting.

  What are the challenges of omnichannel analytics ?

  There are three challenges when implementing an omnichannel analytics solution :

  

  • Complex customer journeys : Customer journeys aren’t linear and can be incredibly difficult to track. 
  • Regulatory and privacy issues : When you start gathering customer data, you quickly come up against consumer privacy laws. 
  • No underlying goal : There has to be a reason to go to all this effort, but brands don’t always have goals in mind before they start. 

  You can’t do anything about the first challenge. 

  After all, your customer journey will almost never be linear. And isn’t the point of implementing an omnichannel solution to understand these complex journeys in the first place ? Once you set up omnichannel analytics, these journeys will be much easier to decipher. 

  As for the other two :

  Using the right software that respects user privacy and complies with all major privacy laws will avoid regulatory issues. Take Matomo, for instance. Our software was designed with privacy in mind and is configured to follow the strictest privacy laws, such as GDPR. 

  Tying omnichannel analytics to marketing attribution will solve the final challenge by giving your omnichannel efforts a goal. When you tie omnichannel analytics to your marketing efforts, you aren’t just getting a 360-degree view of your customer journey for the sake of it. You are getting that view to improve your marketing efforts and increase sales.

  Try Matomo for Free

  Get the web insights you need, without compromising data accuracy.

  Start for free

  No credit card required

  How to set up an omnichannel analytics solution

  Want to set up a seamless analytical environment that incorporates data from every possible source ? Follow these five steps :

  Choose one or more analytics providers

  You can use several tools to build an omnichannel analytics solution. These include web and app analytics tools, customer data platforms that centralise first-party data and business intelligence tools (typically used for visualisation). 

  Which tools you use will depend on your goals and your budget — the loftier your ambitions and the higher your budget, the more tools you can use. 

  Ideally, you should use as few tools as possible to capture your data. Most teams won’t need business intelligence platforms, for example. However, you may or may not need both an analytics platform and a customer data platform. Your decision will depend on how many channels your customers use and how well your analytics tool tracks everything.

  If it can capture web and app usage while integrating with third-party platforms like your back-end e-commerce platform, then it’s probably enough.

  Collect accurate data at every touchpoint 

  Your omnichannel analytics efforts hinge on the quantity and quality of data you can collect. You want to gather data from every touchpoint possible and store that data in as few places as possible. That’s why choosing as few tools as possible in the step above is so important. 

  So, where should you start ? Common data sources include :

  • Your website
  • Apps (iOS and Android)
  • Social media profiles
  • ERPs
  • PoS systems

  At the same time, make sure you’re tracking all relevant metrics. Revenue, customer engagement and conversion-focused metrics like conversion rate, dwell time, cart abandonment rate and churn rate are particularly important. 

  Set up marketing attribution

  Setting up marketing attribution (also known as multi-touch attribution) is essential to tie omnichannel data to business goals. It’s the only way to know exactly how valuable each marketing channel is and where each customer comes from. 

  You’ll want to use multi-touch attribution, given you have data from across the customer journey.

  

  Multi-touch attribution models can include (but are not limited to) :

  • Linear : where each touchpoint is given equal weighting
  • Time decay : where touchpoints are more valuable the nearer they are to conversion
  • Position-based : where the first and last touch points are more valuable than all the others. 

  You don’t have to use just one of the models above, however. One of the benefits of using a web analytics tool like Matomo is that you can choose between different attribution models and compare them.

  Try Matomo for Free

  Get the web insights you need, without compromising data accuracy.

  Start for free

  No credit card required

  Create reports that help you visualise data

  Dashboards are your friend here. They’ll let you see KPIs at a glance, allowing you to keep track of day-to-day changes in your customer journey. Ideally, you’ll want a platform that lets you customise dashboard widgets so only relevant KPIs are shown. 

  

  Setting up standard and custom reports is also important. Custom reports allow you to choose metrics and dimensions that align with your goals. They will also allow you to present your data most meaningfully to your team, increasing the likelihood they act upon insights. 

  Analyse data and take action

  Now that you have customer journey data at your fingertips, it’s time to analyse it. After all, there’s no point in implementing an omnichannel analytics solution if you aren’t going to take action. 

  If you’re unsure where to start, re-read the benefits we listed at the start of this article. You could use your omnichannel insights to improve your marketing campaigns by doubling down on the channels that bring in the best customers.

  Or you could identify (and fix) bottlenecks in the customer journey so customers are less likely to fall out of your funnel between certain channels. 

  Just make sure you take action based on your data alone.

  Make the most of omnichannel analytics with Matomo

  A comprehensive web and app analytics platform is vital to any omnichannel analytics strategy. 

  But not just any solution will do. When privacy regulations impede an omnichannel analytics solution, you need a platform to capture accurate data without breaking privacy laws or your users’ trust. 

  That’s where Matomo comes in. Our privacy-friendly web analytics platform ensures accurate tracking of web traffic while keeping you compliant with even the strictest regulations. Moreover, our range of APIs and SDKs makes it easy to track interactions from all your digital products (website, apps, e-commerce back-ends, etc.) in one place. 

  Try Matomo for free for 21 days. No credit card required.

  Try Matomo for Free

  21 day free trial. No credit card required.

  
  

  Your email address
  

  Your website address
  

  Your Analytics subdomain will be .matomo.cloud (change)

  Choose your analytics subdomain
  

  .matomo.cloud

  I accept the terms and conditions and data processing agreement (DPA)

  
  Your information will be used to create an account on our cloud service. For more information please consult our privacy policy.
  
  
  

  Start improving your websites now

  

• Subtitling Sierra VMD Files

  1er juin 2016, par Multimedia Mike — Game Hacking

  I was contacted by a game translation hobbyist from Spain (henceforth known as The Translator). He had set his sights on Sierra’s 7-CD Phantasmagoria. This mammoth game was driven by a lot of FMV files and animations that have speech. These require language translation in the form of video subtitling. He’s lucky that he found possibly the one person on the whole internet who has just the right combination of skill, time, and interest to pull this off. And why would I care about helping ? I guess I share a certain camaraderie with game hackers. Don’t act so surprised. You know what kind of stuff I like to work on.

  The FMV format used in this game is VMD, which makes an appearance in numerous Sierra titles. FFmpeg already supports decoding this format. FFmpeg also supports subtitling video. So, ideally, all that’s necessary to support this goal is to add a muxer for the VMD format which can encode raw video and audio, which the format supports. Implement video compression as extra credit.

  The pipeline that I envisioned looks like this :

  
  

  

  VMD Subtitling Process

  
  

  “Trivial !” I surmised. I just never learn, do I ?

  The Plan
  So here’s my initial pitch, outlining the work I estimated that I would need to do towards the stated goal :

  1. Create a new file muxer that produces a syntactically valid VMD file with bogus video and audio data. Make sure it works with both FFmpeg’s playback system as well as the proper Phantasmagoria engine.
  2. Create a new video encoder that essentially operates in pass-through mode while correctly building a palette.
  3. Create a new basic encoder for the video frames.

  A big unknown for me was exactly how subtitle handling operates in FFmpeg. Thanks to this project, I now know. I was concerned because I was pretty sure that font rendering entails anti-aliasing which bodes poorly for keeping the palette count under 256 unique colors.

  Computer Science Puzzle
  When pondering how to process the palette, I was excited for the opportunity to exercise actual computer science. FFmpeg converts frames from paletted frames to full RGB frames. Then it needs to convert them back to paletted frames. I had a vague recollection of solving this problem once before when I was experimenting with a new paletted video codec. I seem to recall that I did the palette conversion in a very naive manner. I just used a static 256-element array and processed each RGB pixel of the frame, seeing if the value already occurred in the table (O(n) lookup) and adding it otherwise.
  

  There are more efficient algorithms, however, such as hash tables and trees. Somewhere along the line, FFmpeg helpfully acquired a rarely-used tree data structure, which was perfect for this project.

  So I was pretty pleased with this optimization. Too bad this wouldn’t survive to the end of the effort.

  Another palette-related challenge was the fact that a group of pictures would be accumulating a new palette but that palette needed to be recorded before the group. Thus, the muxer needed to have extra logic to rewind the file when the video encoder transmitted a palette change.

  Video Compression
  VMD has a few methods in its compression toolbox. It can use interframe differencing, it has some RLE, or it can code a frame raw. It can also use a custom LZ-like format on top of these. For early prototypes, I elected to leave each frame coded raw. After the concept was proved, I implemented the frame differencing.

  
  

  

  
  Top frame compared with the middle frame yields the bottom frame : red pixels indicate changes
  

  Encoding only those red dots in between vast runs of unchanged pixels yielded a vast measurable improvement. The next step was to try wiring up FFmpeg’s existing LZ compression facilities to the encoder. This turned out to be implausible since VMD’s LZ variant has nothing to do with anything FFmpeg already provides. Fortunately, the LZ piece is not absolutely required and the frame differencing + RLE provides plenty of compression.

  Subtitling
  I’ve never done anything, multimedia programming-wise, concerning subtitles. I guess all the entertainment I care about has always been in my native tongue. What a good excuse to program outside of my comfort zone !

  First, I needed to know how to access FFmpeg’s subtitling facilities. Fortunately, The Translator did the legwork on this matter so I didn’t have to figure it out.

  However, I intuitively had misgivings about this phase. I had heard that the subtitling process performs anti-aliasing. That means that the image would need to be promoted to a higher colorspace for this phase and that the anti-aliasing process would likely push the color count way past 256. Some quick tests revealed this to be the case, as the running color count would leap by several hundred colors as soon as the palette accounting algorithm encountered a subtitle.

  So I dug into the subtitle subsystem. I discovered that the subtitle library operates by creating a linked list of subtitle bitmaps that the client app must render. The bitmaps are comprised of 8-bit alpha transparency values that must be composited onto the target frame (i.e., 0 = transparent, 255 = 100% opaque). For example, the letter ‘H’ :

                                    (with 00s removed)
  13 F8 41 00 00 00 00 68 E4  |  13 F8 41             68 E4    
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF DC D0 D0 D0 D0 E4 EC  |  14 FF DC D0 D0 D0 D0 E4 EC
  14 FF 7E 50 50 50 50 9A EC  |  14 FF 7E 50 50 50 50 9A EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  11 E0 3B 00 00 00 00 5E CE  |  11 E0 3B             5E CE
  

  To get around the color explosion problem, I chose a threshold value and quantized values above and below to 255 and 0, respectively. Further, the process chooses an appropriate color from the existing palette rather than introducing any new colors.

  Muxing Matters
  In order to force VMD into a general purpose media framework, a lot of special information needs to be passed around. Like many paletted codecs, the palette needs to be transmitted from the file demuxer to the video decoder via some side channel. For re-encoding, this also implies that the palette needs to make the trip from the video encoder to the file muxer. As if this wasn’t enough, individual VMD frames have even more data that needs to be ferried between the muxer and codec levels, including frame change boundaries. FFmpeg provides methods to do these things, but I could not always rely on the systems to relay the data in all cases. I was probably doing something wrong ; I accept that. Instead, I just packed all the information at the front of an encoded frame and split it apart in the muxer.

  I could not quite figure out how to get the audio and video muxed correctly. As a result, neither FFmpeg nor the Phantasmagoria engine could replay the files correctly.

  Plan B
  Since I was having so much trouble creating an entirely new VMD file, likely due to numerous unknown bits of the file format, I thought of another angle : re-use the existing VMD file. For this approach, I kept the video encoder and file muxer that I created in the initial phase, but modified the file muxer to emit a special intermediate file. Then, I created a Python tool to repackage the original VMD file using compressed video data in the intermediate file.

  For this phase, I also implemented a command line switch for FFmpeg to disable subtitle blending, to make the feature feel like less of an unofficial hack, as though this nonsense would ever have a chance of being incorporated upstream.

  At this point, I was seeing some success with the complete, albeit roundabout, subtitling process. I constructed a subtitle file using “Spanish I Learned From Mexican Telenovelas” and the frames turned out fairly readable :

  
  

  

  “she cheated on him”

  
  

  

  “he’s a scumbag” … these random subtitles could fit surprisingly well !

  
  

  The few files that I tested appeared to work fine. But then I handed off my work to The Translator and he immediately found a bunch of problems. According to my notes, the problems mostly took the form of flashing, solid color frames. Further, I found tiny, mostly imperceptible flaws in my RLE compressor, usually only detectable by running strict comparison tools ; but I wasn’t satisfied.

  At this point, I think I attempted to just encode the entire palette at the front of each frame, as allowed by the format, but that did not seem to fix any problems. My notes are not completely clear on this matter (likely because I was still trying to figure out the exact problem), but I think it had to do with FFmpeg inserting extra video frames in order to even out gaps in the video framerate.

  Sigh, Plan C
  At this point, I was getting tired of trying to force FFmpeg to do this. So I decided to minimize its involvement using lessons learned up to this point.

  The next pitch :

  1. Create a new C program that can open an existing VMD file and output an identical VMD file. I know this sounds easy, but the specific method of copying entails interpreting individual parts of the file and writing those individual parts to the new file. This is in preparation for…
  2. Import the VMD video decoder functions directly into the program to decode the individual video frames and re-encode them, replacing the video frames as the file is rewritten.
  3. Wire up the subtitle system. During the adventure to disable subtitle blending, I accidentally learned enough about interfacing to the subtitle library to just invoke it directly.
  4. Rewrite the RLE method so that it is 100% correct.

  Off to work I went. That part about lifting the existing VMD decoder functions out of their libavcodec nest turned out to not be that straightforward. As an alternative, I modified the decoder to dump the raw frames to an intermediate file. In doing so, I think I was able to avoid the issue of the duplicated frames that plagued the previous efforts.

  Also, remember how I was really pleased with the palette conversion technique in which I was able to leverage computer science big-O theory ? By this stage, I had no reason to convert the paletted video to RGB in the first place ; all of the decoding, subtitling and re-encoding operates in the paletted colorspace.

  This approach seemed to work pretty well. The final program is subtitle-vmd.c. The process is still a little weird. The modifications in my own FFmpeg fork are necessary to create an intermediate file that the new C tool can operate with.

  Next Steps
  The Translator has found some assorted bugs and corner cases that still need to be ironed out. Further, for extra credit, I need find the change windows for each frame to improve compression just a little more. I don’t think I will be trying for LZ compression, though.

  However, almost as soon as I had this whole system working, The Translator informed me that there is another, different movie format in play in the Phantasmagoria engine called ROBOT, with an extension of RBT. Fortunately, enough of the algorithms have been reverse engineered and re-implemented in ScummVM that I was able to sort out enough details for another subtitling project. That will be the subject of a future post.

  See Also :

  • Subtitling Sierra RBT Files : The followup in which I discuss how to scribble text on the other animation format

• Subtitling Sierra VMD Files

  1er juin 2016, par Multimedia Mike — Game Hacking

  I was contacted by a game translation hobbyist from Spain (henceforth known as The Translator). He had set his sights on Sierra’s 7-CD Phantasmagoria. This mammoth game was driven by a lot of FMV files and animations that have speech. These require language translation in the form of video subtitling. He’s lucky that he found possibly the one person on the whole internet who has just the right combination of skill, time, and interest to pull this off. And why would I care about helping ? I guess I share a certain camaraderie with game hackers. Don’t act so surprised. You know what kind of stuff I like to work on.

  The FMV format used in this game is VMD, which makes an appearance in numerous Sierra titles. FFmpeg already supports decoding this format. FFmpeg also supports subtitling video. So, ideally, all that’s necessary to support this goal is to add a muxer for the VMD format which can encode raw video and audio, which the format supports. Implement video compression as extra credit.

  The pipeline that I envisioned looks like this :

  
  

  

  VMD Subtitling Process

  
  

  “Trivial !” I surmised. I just never learn, do I ?

  The Plan
  So here’s my initial pitch, outlining the work I estimated that I would need to do towards the stated goal :

  1. Create a new file muxer that produces a syntactically valid VMD file with bogus video and audio data. Make sure it works with both FFmpeg’s playback system as well as the proper Phantasmagoria engine.
  2. Create a new video encoder that essentially operates in pass-through mode while correctly building a palette.
  3. Create a new basic encoder for the video frames.

  A big unknown for me was exactly how subtitle handling operates in FFmpeg. Thanks to this project, I now know. I was concerned because I was pretty sure that font rendering entails anti-aliasing which bodes poorly for keeping the palette count under 256 unique colors.

  Computer Science Puzzle
  When pondering how to process the palette, I was excited for the opportunity to exercise actual computer science. FFmpeg converts frames from paletted frames to full RGB frames. Then it needs to convert them back to paletted frames. I had a vague recollection of solving this problem once before when I was experimenting with a new paletted video codec. I seem to recall that I did the palette conversion in a very naive manner. I just used a static 256-element array and processed each RGB pixel of the frame, seeing if the value already occurred in the table (O(n) lookup) and adding it otherwise.
  

  There are more efficient algorithms, however, such as hash tables and trees. Somewhere along the line, FFmpeg helpfully acquired a rarely-used tree data structure, which was perfect for this project.

  So I was pretty pleased with this optimization. Too bad this wouldn’t survive to the end of the effort.

  Another palette-related challenge was the fact that a group of pictures would be accumulating a new palette but that palette needed to be recorded before the group. Thus, the muxer needed to have extra logic to rewind the file when the video encoder transmitted a palette change.

  Video Compression
  VMD has a few methods in its compression toolbox. It can use interframe differencing, it has some RLE, or it can code a frame raw. It can also use a custom LZ-like format on top of these. For early prototypes, I elected to leave each frame coded raw. After the concept was proved, I implemented the frame differencing.

  
  

  

  
  Top frame compared with the middle frame yields the bottom frame : red pixels indicate changes
  

  Encoding only those red dots in between vast runs of unchanged pixels yielded a vast measurable improvement. The next step was to try wiring up FFmpeg’s existing LZ compression facilities to the encoder. This turned out to be implausible since VMD’s LZ variant has nothing to do with anything FFmpeg already provides. Fortunately, the LZ piece is not absolutely required and the frame differencing + RLE provides plenty of compression.

  Subtitling
  I’ve never done anything, multimedia programming-wise, concerning subtitles. I guess all the entertainment I care about has always been in my native tongue. What a good excuse to program outside of my comfort zone !

  First, I needed to know how to access FFmpeg’s subtitling facilities. Fortunately, The Translator did the legwork on this matter so I didn’t have to figure it out.

  However, I intuitively had misgivings about this phase. I had heard that the subtitling process performs anti-aliasing. That means that the image would need to be promoted to a higher colorspace for this phase and that the anti-aliasing process would likely push the color count way past 256. Some quick tests revealed this to be the case, as the running color count would leap by several hundred colors as soon as the palette accounting algorithm encountered a subtitle.

  So I dug into the subtitle subsystem. I discovered that the subtitle library operates by creating a linked list of subtitle bitmaps that the client app must render. The bitmaps are comprised of 8-bit alpha transparency values that must be composited onto the target frame (i.e., 0 = transparent, 255 = 100% opaque). For example, the letter ‘H’ :

                                    (with 00s removed)
  13 F8 41 00 00 00 00 68 E4  |  13 F8 41             68 E4    
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF DC D0 D0 D0 D0 E4 EC  |  14 FF DC D0 D0 D0 D0 E4 EC
  14 FF 7E 50 50 50 50 9A EC  |  14 FF 7E 50 50 50 50 9A EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  14 FF 44 00 00 00 00 6C EC  |  14 FF 44             6C EC
  11 E0 3B 00 00 00 00 5E CE  |  11 E0 3B             5E CE
  

  To get around the color explosion problem, I chose a threshold value and quantized values above and below to 255 and 0, respectively. Further, the process chooses an appropriate color from the existing palette rather than introducing any new colors.

  Muxing Matters
  In order to force VMD into a general purpose media framework, a lot of special information needs to be passed around. Like many paletted codecs, the palette needs to be transmitted from the file demuxer to the video decoder via some side channel. For re-encoding, this also implies that the palette needs to make the trip from the video encoder to the file muxer. As if this wasn’t enough, individual VMD frames have even more data that needs to be ferried between the muxer and codec levels, including frame change boundaries. FFmpeg provides methods to do these things, but I could not always rely on the systems to relay the data in all cases. I was probably doing something wrong ; I accept that. Instead, I just packed all the information at the front of an encoded frame and split it apart in the muxer.

  I could not quite figure out how to get the audio and video muxed correctly. As a result, neither FFmpeg nor the Phantasmagoria engine could replay the files correctly.

  Plan B
  Since I was having so much trouble creating an entirely new VMD file, likely due to numerous unknown bits of the file format, I thought of another angle : re-use the existing VMD file. For this approach, I kept the video encoder and file muxer that I created in the initial phase, but modified the file muxer to emit a special intermediate file. Then, I created a Python tool to repackage the original VMD file using compressed video data in the intermediate file.

  For this phase, I also implemented a command line switch for FFmpeg to disable subtitle blending, to make the feature feel like less of an unofficial hack, as though this nonsense would ever have a chance of being incorporated upstream.

  At this point, I was seeing some success with the complete, albeit roundabout, subtitling process. I constructed a subtitle file using “Spanish I Learned From Mexican Telenovelas” and the frames turned out fairly readable :

  
  

  

  “she cheated on him”

  
  

  

  “he’s a scumbag” … these random subtitles could fit surprisingly well !

  
  

  The few files that I tested appeared to work fine. But then I handed off my work to The Translator and he immediately found a bunch of problems. According to my notes, the problems mostly took the form of flashing, solid color frames. Further, I found tiny, mostly imperceptible flaws in my RLE compressor, usually only detectable by running strict comparison tools ; but I wasn’t satisfied.

  At this point, I think I attempted to just encode the entire palette at the front of each frame, as allowed by the format, but that did not seem to fix any problems. My notes are not completely clear on this matter (likely because I was still trying to figure out the exact problem), but I think it had to do with FFmpeg inserting extra video frames in order to even out gaps in the video framerate.

  Sigh, Plan C
  At this point, I was getting tired of trying to force FFmpeg to do this. So I decided to minimize its involvement using lessons learned up to this point.

  The next pitch :

  1. Create a new C program that can open an existing VMD file and output an identical VMD file. I know this sounds easy, but the specific method of copying entails interpreting individual parts of the file and writing those individual parts to the new file. This is in preparation for…
  2. Import the VMD video decoder functions directly into the program to decode the individual video frames and re-encode them, replacing the video frames as the file is rewritten.
  3. Wire up the subtitle system. During the adventure to disable subtitle blending, I accidentally learned enough about interfacing to the subtitle library to just invoke it directly.
  4. Rewrite the RLE method so that it is 100% correct.

  Off to work I went. That part about lifting the existing VMD decoder functions out of their libavcodec nest turned out to not be that straightforward. As an alternative, I modified the decoder to dump the raw frames to an intermediate file. In doing so, I think I was able to avoid the issue of the duplicated frames that plagued the previous efforts.

  Also, remember how I was really pleased with the palette conversion technique in which I was able to leverage computer science big-O theory ? By this stage, I had no reason to convert the paletted video to RGB in the first place ; all of the decoding, subtitling and re-encoding operates in the paletted colorspace.

  This approach seemed to work pretty well. The final program is subtitle-vmd.c. The process is still a little weird. The modifications in my own FFmpeg fork are necessary to create an intermediate file that the new C tool can operate with.

  Next Steps
  The Translator has found some assorted bugs and corner cases that still need to be ironed out. Further, for extra credit, I need find the change windows for each frame to improve compression just a little more. I don’t think I will be trying for LZ compression, though.

  However, almost as soon as I had this whole system working, The Translator informed me that there is another, different movie format in play in the Phantasmagoria engine called ROBOT, with an extension of RBT. Fortunately, enough of the algorithms have been reverse engineered and re-implemented in ScummVM that I was able to sort out enough details for another subtitling project. That will be the subject of a future post.

  See Also :

  • Subtitling Sierra RBT Files : The followup in which I discuss how to scribble text on the other animation format

  The post Subtitling Sierra VMD Files first appeared on Breaking Eggs And Making Omelettes.