Recherche avancée

Sur d’autres sites (5295)

• Stream ffmpeg transcoding result to S3

  7 juin 2019, par mabead

  I want to transcode a large file using FFMPEG and store the result directly on AWS S3. This will be done inside of an AWS Lambda that has limited tmp space so I can’t store the transcoding result locally and then upload it to S3 in a second step. I won’t have enough tmp space. I therefore want to store the FFMPEG output directly on S3.

  I therefore created a S3 pre-signed url that allows ’PUT’ :

  var outputPath = s3Client.GetPreSignedURL(new Amazon.S3.Model.GetPreSignedUrlRequest
  
  {
  
      BucketName = "my-bucket",
  
      Expires = DateTime.UtcNow.AddMinutes(5),
  
      Key = "output.mp3",
  
      Verb = HttpVerb.PUT,
  
  });

  I then called ffmpeg with the resulting pre-signed url :

  ffmpeg -i C:\input.wav -y -vn -ar 44100 -ac 2 -ab 192k -f mp3 https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550427237&Signature=%2BE8Wc%2F%2FQYrvGxzc%2FgXnsvauKnac%3D

  FFMPEG returns an exit code of 1 with the following output :

  ffmpeg version N-93120-ga84af760b8 Copyright (c) 2000-2019 the FFmpeg developers
  
    built with gcc 8.2.1 (GCC) 20190212
  
    configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  
    libavutil      56. 26.100 / 56. 26.100
  
    libavcodec     58. 47.100 / 58. 47.100
  
    libavformat    58. 26.101 / 58. 26.101
  
    libavdevice    58.  6.101 / 58.  6.101
  
    libavfilter     7. 48.100 /  7. 48.100
  
    libswscale      5.  4.100 /  5.  4.100
  
    libswresample   3.  4.100 /  3.  4.100
  
    libpostproc    55.  4.100 / 55.  4.100
  
  Guessed Channel Layout for Input Stream #0.0 : stereo
  
  Input #0, wav, from 'C:\input.wav':
  
    Duration: 00:04:16.72, bitrate: 3072 kb/s
  
      Stream #0:0: Audio: pcm_s32le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s32, 3072 kb/s
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (pcm_s32le (native) -> mp3 (libmp3lame))
  
  Press [q] to stop, [?] for help
  
  Output #0, mp3, to 'https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550427237&Signature=%2BE8Wc%2F%2FQYrvGxzc%2FgXnsvauKnac%3D':
  
    Metadata:
  
      TSSE            : Lavf58.26.101
  
      Stream #0:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s32p, 192 kb/s
  
      Metadata:
  
        encoder         : Lavc58.47.100 libmp3lame
  
  size=     577kB time=00:00:24.58 bitrate= 192.2kbits/s speed=49.1x    
  
  size=    1109kB time=00:00:47.28 bitrate= 192.1kbits/s speed=47.2x    
  
  [tls @ 000001d73d786b00] Error in the push function.
  
  av_interleaved_write_frame(): I/O error
  
  Error writing trailer of https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550427237&Signature=%2BE8Wc%2F%2FQYrvGxzc%2FgXnsvauKnac%3D: I/O error
  
  size=    1143kB time=00:00:48.77 bitrate= 192.0kbits/s speed=  47x    
  
  video:0kB audio:1144kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
  
  [tls @ 000001d73d786b00] The specified session has been invalidated for some reason.
  
  [tls @ 000001d73d786b00] Error in the pull function.
  
  [https @ 000001d73d784fc0] URL read error:  -5
  
  Conversion failed!

  As you can see, I have a URL read error. This is a little surprising to me since I want to output to this url and not read it.

  Anybody know how I can store directly my FFMPEG output directly to S3 without having to store it locally first ?

  Edit 1
  I then tried to use the -method PUT parameter and use http instead of https to remove TLS from the equation. Here’s the output that I got when running ffmpeg with the -v trace option.

  ffmpeg version N-93120-ga84af760b8 Copyright (c) 2000-2019 the FFmpeg developers
  
    built with gcc 8.2.1 (GCC) 20190212
  
    configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
  
    libavutil      56. 26.100 / 56. 26.100
  
    libavcodec     58. 47.100 / 58. 47.100
  
    libavformat    58. 26.101 / 58. 26.101
  
    libavdevice    58.  6.101 / 58.  6.101
  
    libavfilter     7. 48.100 /  7. 48.100
  
    libswscale      5.  4.100 /  5.  4.100
  
    libswresample   3.  4.100 /  3.  4.100
  
    libpostproc    55.  4.100 / 55.  4.100
  
  Splitting the commandline.
  
  Reading option '-i' ... matched as input url with argument 'C:\input.wav'.
  
  Reading option '-y' ... matched as option 'y' (overwrite output files) with argument '1'.
  
  Reading option '-vn' ... matched as option 'vn' (disable video) with argument '1'.
  
  Reading option '-ar' ... matched as option 'ar' (set audio sampling rate (in Hz)) with argument '44100'.
  
  Reading option '-ac' ... matched as option 'ac' (set number of audio channels) with argument '2'.
  
  Reading option '-ab' ... matched as option 'ab' (audio bitrate (please use -b:a)) with argument '192k'.
  
  Reading option '-f' ... matched as option 'f' (force format) with argument 'mp3'.
  
  Reading option '-method' ... matched as AVOption 'method' with argument 'PUT'.
  
  Reading option '-v' ... matched as option 'v' (set logging level) with argument 'trace'.
  
  Reading option 'https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D' ... matched as output url.
  
  Finished splitting the commandline.
  
  Parsing a group of options: global .
  
  Applying option y (overwrite output files) with argument 1.
  
  Applying option v (set logging level) with argument trace.
  
  Successfully parsed a group of options.
  
  Parsing a group of options: input url C:\input.wav.
  
  Successfully parsed a group of options.
  
  Opening an input file: C:\input.wav.
  
  [NULL @ 000001fb37abb180] Opening 'C:\input.wav' for reading
  
  [file @ 000001fb37abc180] Setting default whitelist 'file,crypto'
  
  Probing wav score:99 size:2048
  
  [wav @ 000001fb37abb180] Format wav probed with size=2048 and score=99
  
  [wav @ 000001fb37abb180] Before avformat_find_stream_info() pos: 54 bytes read:65590 seeks:1 nb_streams:1
  
  [wav @ 000001fb37abb180] parser not found for codec pcm_s32le, packets or times may be invalid.
  
      Last message repeated 1 times
  
  [wav @ 000001fb37abb180] All info found
  
  [wav @ 000001fb37abb180] stream 0: start_time: -192153584101141.156 duration: 256.716
  
  [wav @ 000001fb37abb180] format: start_time: -9223372036854.775 duration: 256.716 bitrate=3072 kb/s
  
  [wav @ 000001fb37abb180] After avformat_find_stream_info() pos: 204854 bytes read:294966 seeks:1 frames:50
  
  Guessed Channel Layout for Input Stream #0.0 : stereo
  
  Input #0, wav, from 'C:\input.wav':
  
    Duration: 00:04:16.72, bitrate: 3072 kb/s
  
      Stream #0:0, 50, 1/48000: Audio: pcm_s32le ([1][0][0][0] / 0x0001), 48000 Hz, stereo, s32, 3072 kb/s
  
  Successfully opened the file.
  
  Parsing a group of options: output url https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D.
  
  Applying option vn (disable video) with argument 1.
  
  Applying option ar (set audio sampling rate (in Hz)) with argument 44100.
  
  Applying option ac (set number of audio channels) with argument 2.
  
  Applying option ab (audio bitrate (please use -b:a)) with argument 192k.
  
  Applying option f (force format) with argument mp3.
  
  Successfully parsed a group of options.
  
  Opening an output file: https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D.
  
  [http @ 000001fb37b15140] Setting default whitelist 'http,https,tls,rtp,tcp,udp,crypto,httpproxy'
  
  [tcp @ 000001fb37b16c80] Original list of addresses:
  
  [tcp @ 000001fb37b16c80] Address 52.216.8.203 port 80
  
  [tcp @ 000001fb37b16c80] Interleaved list of addresses:
  
  [tcp @ 000001fb37b16c80] Address 52.216.8.203 port 80
  
  [tcp @ 000001fb37b16c80] Starting connection attempt to 52.216.8.203 port 80
  
  [tcp @ 000001fb37b16c80] Successfully connected to 52.216.8.203 port 80
  
  [http @ 000001fb37b15140] request: PUT /output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D HTTP/1.1
  
  Transfer-Encoding: chunked
  
  User-Agent: Lavf/58.26.101
  
  Accept: */*
  
  Connection: close
  
  Host: landr-distribution-reportsdev-mb.s3.amazonaws.com
  
  Icy-MetaData: 1
  
  Successfully opened the file.
  
  Stream mapping:
  
    Stream #0:0 -> #0:0 (pcm_s32le (native) -> mp3 (libmp3lame))
  
  Press [q] to stop, [?] for help
  
  cur_dts is invalid (this is harmless if it occurs once at the start per stream)
  
  detected 8 logical cores
  
  [graph_0_in_0_0 @ 000001fb37b21080] Setting 'time_base' to value '1/48000'
  
  [graph_0_in_0_0 @ 000001fb37b21080] Setting 'sample_rate' to value '48000'
  
  [graph_0_in_0_0 @ 000001fb37b21080] Setting 'sample_fmt' to value 's32'
  
  [graph_0_in_0_0 @ 000001fb37b21080] Setting 'channel_layout' to value '0x3'
  
  [graph_0_in_0_0 @ 000001fb37b21080] tb:1/48000 samplefmt:s32 samplerate:48000 chlayout:0x3
  
  [format_out_0_0 @ 000001fb37b22cc0] Setting 'sample_fmts' to value 's32p|fltp|s16p'
  
  [format_out_0_0 @ 000001fb37b22cc0] Setting 'sample_rates' to value '44100'
  
  [format_out_0_0 @ 000001fb37b22cc0] Setting 'channel_layouts' to value '0x3'
  
  [format_out_0_0 @ 000001fb37b22cc0] auto-inserting filter 'auto_resampler_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_0'
  
  [AVFilterGraph @ 000001fb37b0d940] query_formats: 4 queried, 6 merged, 3 already done, 0 delayed
  
  [auto_resampler_0 @ 000001fb37b251c0] picking s32p out of 3 ref:s32
  
  [auto_resampler_0 @ 000001fb37b251c0] [SWR @ 000001fb37b252c0] Using fltp internally between filters
  
  [auto_resampler_0 @ 000001fb37b251c0] ch:2 chl:stereo fmt:s32 r:48000Hz -> ch:2 chl:stereo fmt:s32p r:44100Hz
  
  Output #0, mp3, to 'https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D':
  
    Metadata:
  
      TSSE            : Lavf58.26.101
  
      Stream #0:0, 0, 1/44100: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s32p, delay 1105, 192 kb/s
  
      Metadata:
  
        encoder         : Lavc58.47.100 libmp3lame
  
  cur_dts is invalid (this is harmless if it occurs once at the start per stream)
  
      Last message repeated 6 times
  
  size=     649kB time=00:00:27.66 bitrate= 192.2kbits/s speed=55.3x    
  
  size=    1207kB time=00:00:51.48 bitrate= 192.1kbits/s speed=51.5x    
  
  av_interleaved_write_frame(): Unknown error
  
  No more output streams to write to, finishing.
  
  [libmp3lame @ 000001fb37b147c0] Trying to remove 47 more samples than there are in the queue
  
  Error writing trailer of https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D: Error number -10054 occurred
  
  size=    1251kB time=00:00:53.39 bitrate= 192.0kbits/s speed=51.5x    
  
  video:0kB audio:1252kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
  
  Input file #0 (C:\input.wav):
  
    Input stream #0:0 (audio): 5014 packets read (20537344 bytes); 5014 frames decoded (2567168 samples); 
  
    Total: 5014 packets (20537344 bytes) demuxed
  
  Output file #0 (https://my-bucket.s3.amazonaws.com/output.mp3?AWSAccessKeyId=AKIAJDSGJWM63VQEXHIQ&Expires=1550695990&Signature=dy3RVqDlX%2BlJ0INlDkl0Lm1Rqb4%3D):
  
    Output stream #0:0 (audio): 2047 frames encoded (2358144 samples); 2045 packets muxed (1282089 bytes); 
  
    Total: 2045 packets (1282089 bytes) muxed
  
  5014 frames successfully decoded, 0 decoding errors
  
  [AVIOContext @ 000001fb37b1f440] Statistics: 0 seeks, 2046 writeouts
  
  [http @ 000001fb37b15140] URL read error:  -10054
  
  [AVIOContext @ 000001fb37ac4400] Statistics: 20611126 bytes read, 1 seeks
  
  Conversion failed!

  So it looks like it is able to connect to my S3 pre-signed url but I still have the Error writing trailer error coupled with a URL read error.

• A Guide to GDPR Sensitive Personal Data

  13 mai 2024, par Erin

  The General Data Protection Regulation (GDPR) is one of the world’s most stringent data protection laws. It provides a legal framework for collection and processing of the personal data of EU individuals.

  The GDPR distinguishes between “special categories of personal data” (also referred to as “sensitive”) and other personal data and imposes stricter requirements on collection and processing of sensitive data. Understanding these differences will help your company comply with the requirements and avoid heavy penalties.

  In this article, we’ll explain what personal data is considered “sensitive” according to the GDPR. We’ll also examine how a web analytics solution like Matomo can help you maintain compliance.

  What is sensitive personal data ?

  The following categories of data are treated as sensitive :

  1. 1. Personal data revealing :
        • Racial or ethnic origin ;
        • Political opinions ;
        • Religious or philosophical beliefs ;
        • Trade union membership ;
     2. Genetic and biometric data ;
     3. Data concerning a person’s :
        • Health ; or
        • Sex life or sexual orientation.

  

  Sensitive vs. non-sensitive personal data : What’s the difference ?

  While both categories include information about an individual, sensitive data is seen as more private, or requiring a greater protection. 

  Sensitive data often carries a higher degree of risk and harm to the data subject, if the data is exposed. For example, a data breach exposing health records could lead to discrimination for the individuals involved. An insurance company could use the information to increase premiums or deny coverage. 

  In contrast, personal data like name or gender is considered less sensitive because it doesn’t carry the same degree of harm as sensitive data. 

  Unauthorised access to someone’s name alone is less likely to harm them or infringe on their fundamental rights and freedoms than an unauthorised access to their health records or biometric data. Note that financial information (e.g. credit card details) does not fall into the special categories of data.

  

  Legality of processing

  Under the GDPR, both sensitive and nonsensitive personal data are protected. However, the rules and conditions for processing sensitive data are more stringent.

  Article 6 deals with processing of non-sensitive data and it states that processing is lawful if one of the six lawful bases for processing applies. 

  In contrast, Art. 9 of the GDPR states that processing of sensitive data is prohibited as a rule, but provides ten exceptions. 

  It is important to note that the lawful bases in Art. 6 are not the same as exceptions in Art. 9. For example, while performance of a contract or legitimate interest of the controller are a lawful basis for processing non-sensitive personal data, they are not included as an exception in Art. 9. What follows is that controllers are not permitted to process sensitive data on the basis of contract or legitimate interest. 

  The exceptions where processing of sensitive personal data is permitted (subject to additional requirements) are : 

  • Explicit consent : The individual has given explicit consent to processing their sensitive personal data for specified purpose(s), except where an EU member state prohibits such consent. See below for more information about explicit consent. 
  • Employment, social security or social protection : Processing sensitive data is necessary to perform tasks under employment, social security or social protection law.
  • Vital interests : Processing sensitive data is necessary to protect the interests of a data subject or if the individual is physically or legally incapable of consenting. 
  • Non-for-profit bodies : Foundations, associations or nonprofits with a political, philosophical, religious or trade union aim may process the sensitive data of their members or those they are in regular contact with, in connection with their purposes (and no disclosure of the data is permitted outside the organisation, without the data subject’s consent).
  • Made public : In some cases, it may be permissible to process the sensitive data of a data subject if the individual has already made it public and accessible. 
  • Legal claims : Processing sensitive data is necessary to establish, exercise or defend legal claims, including legal or in court proceedings.
  • Public interest : Processing is necessary for reasons of substantial public interest, like preventing unlawful acts or protecting the public.
  • Health or social care : Processing special category data is necessary for : preventative or occupational medicine, providing health and social care, medical diagnosis or managing healthcare systems.
  • Public health : It is permissible to process sensitive data for public health reasons, like protecting against cross-border threats to health or ensuring the safety of medicinal products or medical devices. 
  • Archiving, research and statistics : You may process sensitive data if it’s done for archiving purposes in the public interest, scientific or historical research purposes or statistical purposes.

  In addition, you must adhere to all data handling requirements set by the GDPR.

  Important : Note that for any data sent that you are processing, you always need to identify a lawful basis under Art. 6. In addition, if the data sent contains sensitive data, you must comply with Art. 9.

  Explicit consent

  While consent is a valid lawful basis for processing non-sensitive personal data, controllers are permitted to process sensitive data only with an “explicit consent” of the data subject.

  The GDPR does not define “explicit” consent, but it is accepted that it must meet all Art. 7 conditions for consent, at a higher threshold. To be “explicit” a consent requires a clear statement (oral or written) of the data subject. Consent inferred from the data subject’s actions does not meet the threshold. 

  The controller must retain records of the explicit consent and provide appropriate consent withdrawal method to allow the data subject to exercise their rights.

  Examples of compliant and non-compliant sensitive data processing

  Here are examples of when you can and can’t process sensitive data :

  • When you can process sensitive data : A doctor logs sensitive data about a patient, including their name, symptoms and medicine prescribed. The hospital can process this data to provide appropriate medical care to their patients. An IoT device and software manufacturer processes their customers’ health data based on explicit consent of each customer. 
  • When you can’t process sensitive data : One example is when you don’t have explicit consent from a data subject. Another is when there’s no lawful basis for processing it or you are collecting personal data you simply do not need. For example, you don’t need your customer’s ethnic origin to fulfil an online order.

  Other implications of processing sensitive data

  If you process sensitive data, especially on a large scale, GDPR imposes additional requirements, such as having Data Privacy Impact Assessments, appointing Data Protection Officers and EU Representatives, if you are a controller based outside the EU.

  Penalties for GDPR non-compliance

  Mishandling sensitive data (or processing it when you’re not allowed to) can result in huge penalties. There are two tiers of GDPR fines :

  • €10 million or 2% of a company’s annual revenue for less severe infringements
  • €20 million or 4% of a company’s annual revenue for more severe infringements

  In the first half of 2023 alone, fines imposed in the EU due to GDPR violations exceeded €1.6 billion, up from €73 million in 2019.

  Examples of high-profile violations in the last few years include :

  • Amazon : The Luxembourg National Commission fined the retail giant with a massive $887 million fine in 2021 for not processing personal data per the GDPR. 
  • Google : The National Data Protection Commission (CNIL) fined Google €50 million for not getting proper consent to display personalised ads.
  • H&M : The Hamburg Commissioner for Data Protection and Freedom of Information hit the multinational clothing company with a €35.3 million fine in 2020 for unlawfully gathering and storing employees’ data in its service centre.

  One of the criteria that affects the severity of a fine is “data category” — the type of personal data being processed. Companies need to take extra precautions with sensitive data, or they risk receiving more severe penalties.

  What’s more, GDPR violations can negatively affect your brand’s reputation and cause you to lose business opportunities from consumers concerned about your data practices. 76% of consumers indicated they wouldn’t buy from companies they don’t trust with their personal data.

  Organisations should lay out their data practices in simple terms and make this information easily accessible so customers know how their data is being handled.

  Get started with GDPR-compliant web analytics

  The GDPR offers a framework for securing and protecting personal data. But it also distinguishes between sensitive and non-sensitive data. Understanding these differences and applying the lawful basis for processing this data type will help ensure compliance.

  Looking for a GDPR-compliant web analytics solution ?

  At Matomo, we take data privacy seriously. 

  Our platform ensures 100% data ownership, putting you in complete control of your data. Unlike other web analytics solutions, your data remains solely yours and isn’t sold or auctioned off to advertisers. 

  Additionally, with Matomo, you can be confident in the accuracy of the insights you receive, as we provide reliable, unsampled data.

  Matomo also fully complies with GDPR and other data privacy laws like CCPA, LGPD and more.

  Start your 21-day free trial today ; no credit card required. 

  Disclaimer

  We are not lawyers and don’t claim to be. The information provided here is to help give an introduction to GDPR. We encourage every business and website to take data privacy seriously and discuss these issues with your lawyer if you have any concerns.

  Try Matomo for Free

  21 day free trial. No credit card required.

  
  

  Your email address
  

  Your website address
  

  Your Analytics subdomain will be .matomo.cloud (change)

  Choose your analytics subdomain
  

  .matomo.cloud

  I accept the terms and conditions and data processing agreement (DPA)

  
  Your information will be used to create an account on our cloud service. For more information please consult our privacy policy.
  
  
  

  Start improving your websites now

  

• trying to make OpenCV 3.2.0 work with virtualenv

  24 juillet 2017, par lollercoaster

  I’m on Ubuntu 16.04 with Python 2.7 and virtualenv & virtualenvwrapper.

  By following this guide I managed to get the following script working with my system Python2.7 which has cv2 globally installed.

  I used this script to install it :

  ######################################
  
  # INSTALL OPENCV ON UBUNTU OR DEBIAN #
  
  ######################################
  
  
  
  # |         THIS SCRIPT IS TESTED CORRECTLY ON         |
  
  # |----------------------------------------------------|
  
  # | OS             | OpenCV       | Test | Last test   |
  
  # |----------------|--------------|------|-------------|
  
  # | Ubuntu 16.04.2 | OpenCV 3.2.0 | OK   | 20 May 2017 |
  
  # | Debian 8.8     | OpenCV 3.2.0 | OK   | 20 May 2017 |
  
  # | Debian 9.0     | OpenCV 3.2.0 | OK   | 25 Jun 2017 |
  
  
  
  # 1. KEEP UBUNTU OR DEBIAN UP TO DATE
  
  
  
  sudo apt-get -y update
  
  sudo apt-get -y upgrade
  
  sudo apt-get -y dist-upgrade
  
  sudo apt-get -y autoremove
  
  
  
  
  
  # 2. INSTALL THE DEPENDENCIES
  
  
  
  # Build tools:
  
  sudo apt-get install -y build-essential cmake
  
  
  
  # GUI (if you want to use GTK instead of Qt, replace 'qt5-default' with 'libgtkglext1-dev' and remove '-DWITH_QT=ON' option in CMake):
  
  sudo apt-get install -y qt5-default libvtk6-dev
  
  
  
  # Media I/O:
  
  sudo apt-get install -y zlib1g-dev libjpeg-dev libwebp-dev libpng-dev libtiff5-dev libjasper-dev libopenexr-dev libgdal-dev
  
  
  
  # Video I/O:
  
  sudo apt-get install -y libdc1394-22-dev libavcodec-dev libavformat-dev libswscale-dev libtheora-dev libvorbis-dev libxvidcore-dev libx264-dev yasm libopencore-amrnb-dev libopencore-amrwb-dev libv4l-dev libxine2-dev
  
  
  
  # Parallelism and linear algebra libraries:
  
  sudo apt-get install -y libtbb-dev libeigen3-dev
  
  
  
  # Python:
  
  sudo apt-get install -y python-dev python-tk python-numpy python3-dev python3-tk python3-numpy
  
  
  
  # Documentation:
  
  sudo apt-get install -y doxygen
  
  
  
  # UI stuff
  
  sudo apt-get install libgtk-3-dev libatlas-base-dev gfortran
  
  
  
  
  
  # 3. INSTALL THE LIBRARY (YOU CAN CHANGE '3.2.0' FOR THE LAST STABLE VERSION)
  
  sudo apt-get install -y unzip wget
  
  
  
  # opencv contrib
  
  wget https://github.com/opencv/opencv_contrib/archive/3.2.0.zip -O opencv_contrib-3.2.0.zip
  
  unzip opencv_contrib-3.2.0.zip
  
  rm opencv_contrib-3.2.0.zip
  
  
  
  # opencv
  
  wget https://github.com/opencv/opencv/archive/3.2.0.zip
  
  unzip 3.2.0.zip
  
  rm 3.2.0.zip
  
  mv opencv-3.2.0 OpenCV-3.2.0
  
  cd OpenCV-3.2.0
  
  
  
  mkdir build
  
  cd build
  
  cmake -D WITH_QT=ON \
  
      -D WITH_OPENGL=ON \
  
      -D FORCE_VTK=ON \
  
      -D WITH_TBB=ON \
  
      -D WITH_GDAL=ON \
  
      -D WITH_XINE=ON \
  
      -D BUILD_EXAMPLES=ON \
  
      -D INSTALL_PYTHON_EXAMPLES=ON \
  
      -D ENABLE_PRECOMPILED_HEADERS=OFF \
  
      -D BUILD_NEW_PYTHON_SUPPORT=ON \
  
      ..
  
  
  
  make -j4
  
  sudo make install
  
  sudo ldconfig
  
  
  
  
  
  # 4. EXECUTE SOME OPENCV EXAMPLES AND COMPILE A DEMONSTRATION
  
  
  
  # To complete this step, please visit 'http://milq.github.io/install-opencv-ubuntu-debian'.

  The following script below works great with that system-wide installation :

  import cv2
  
  
  
  img = cv2.imread('some_img.jpg')

  Though this one doesn’t - even the system Python can’t read videos for some reason...

  import cv2
  
  
  
  video_capture = cv2.VideoCapture(0)
  
  ret, frame = video_capture.read()
  
  print ret  # always False

  but I want it to work with my virtualenv. So I recompiled OpenCV with :

  cmake -D WITH_QT=ON \
  
      -D WITH_OPENGL=ON \
  
      -D FORCE_VTK=ON \
  
      -D WITH_TBB=ON \
  
      -D WITH_GDAL=ON \
  
      -D WITH_XINE=ON \
  
      -D BUILD_EXAMPLES=ON \
  
      -D INSTALL_PYTHON_EXAMPLES=ON \
  
      -D ENABLE_PRECOMPILED_HEADERS=OFF \
  
      -D BUILD_NEW_PYTHON_SUPPORT=ON \
  
      -D OPENCV_EXTRA_MODULES_PATH=/home/me/code/myproject/opencv_contrib-3.2.0/modules \
  
      -D PYTHON_EXECUTABLE=~/.envs/myenv/bin/python \
  
      ..
  
  
  
  make -j4
  
  sudo make install
  
  sudo ldconfig

  Here’s the CMake log :

  -- Found VTK ver. 6.2.0 (usefile: /usr/lib/cmake/vtk-6.2/UseVTK.cmake)
  
  -- Caffe:   NO
  
  -- Protobuf:   YES
  
  -- Glog:   NO
  
  -- freetype2:   YES
  
  -- harfbuzz:    YES
  
  -- Module opencv_sfm disabled because the following dependencies are not found: Glog/Gflags
  
  -- freetype2:   YES
  
  -- harfbuzz:    YES
  
  -- Checking for modules 'tesseract;lept'
  
  --   No package 'tesseract' found
  
  --   No package 'lept' found
  
  -- Tesseract:   NO
  
  -- Check contents of vgg_generated_48.i ...
  
  -- Check contents of vgg_generated_64.i ...
  
  -- Check contents of vgg_generated_80.i ...
  
  -- Check contents of vgg_generated_120.i ...
  
  -- Check contents of boostdesc_bgm.i ...
  
  -- Check contents of boostdesc_bgm_bi.i ...
  
  -- Check contents of boostdesc_bgm_hd.i ...
  
  -- Check contents of boostdesc_binboost_064.i ...
  
  -- Check contents of boostdesc_binboost_128.i ...
  
  -- Check contents of boostdesc_binboost_256.i ...
  
  -- Check contents of boostdesc_lbgm.i ...
  
  -- 
  
  -- General configuration for OpenCV 3.2.0 =====================================
  
  --   Version control:               817bd7b-dirty
  
  -- 
  
  --   Extra modules:
  
  --     Location (extra):            /home/me/code/myproject/opencv_contrib-3.2.0/modules
  
  --     Version control (extra):     817bd7b-dirty
  
  -- 
  
  --   Platform:
  
  --     Timestamp:                   2017-07-20T18:25:26Z
  
  --     Host:                        Linux 4.8.0-58-generic x86_64
  
  --     CMake:                       3.5.1
  
  --     CMake generator:             Unix Makefiles
  
  --     CMake build tool:            /usr/bin/make
  
  --     Configuration:               Release
  
  -- 
  
  --   C/C++:
  
  --     Built as dynamic libs?:      YES
  
  --     C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
  
  --     C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
  
  --     C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
  
  --     C Compiler:                  /usr/bin/cc
  
  --     C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
  
  --     C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -Wno-comment -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
  
  --     Linker flags (Release):
  
  --     Linker flags (Debug):
  
  --     ccache:                      NO
  
  --     Precompiled headers:         NO
  
  --     Extra dependencies:          Qt5::Test Qt5::Concurrent Qt5::OpenGL /usr/lib/x86_64-linux-gnu/libwebp.so /usr/lib/x86_64-linux-gnu/libjasper.so /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so /usr/lib/libgdal.so dc1394 xine avcodec-ffmpeg avformat-ffmpeg avutil-ffmpeg swscale-ffmpeg Qt5::Core Qt5::Gui Qt5::Widgets /usr/lib/x86_64-linux-gnu/hdf5/serial/lib/libhdf5.so /usr/lib/x86_64-linux-gnu/libpthread.so /usr/lib/x86_64-linux-gnu/libsz.so /usr/lib/x86_64-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libm.so vtkRenderingOpenGL vtkImagingHybrid vtkIOImage vtkCommonDataModel vtkCommonMath vtkCommonCore vtksys vtkCommonMisc vtkCommonSystem vtkCommonTransforms vtkCommonExecutionModel vtkDICOMParser vtkIOCore /usr/lib/x86_64-linux-gnu/libz.so vtkmetaio /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/x86_64-linux-gnu/libpng.so /usr/lib/x86_64-linux-gnu/libtiff.so vtkImagingCore vtkRenderingCore vtkCommonColor vtkFiltersExtraction vtkFiltersCore vtkFiltersGeneral vtkCommonComputationalGeometry vtkFiltersStatistics vtkImagingFourier vtkalglib vtkFiltersGeometry vtkFiltersSources vtkInteractionStyle vtkRenderingLOD vtkFiltersModeling vtkIOPLY vtkIOGeometry /usr/lib/x86_64-linux-gnu/libjsoncpp.so vtkFiltersTexture vtkRenderingFreeType /usr/lib/x86_64-linux-gnu/libfreetype.so vtkftgl vtkIOExport vtkRenderingAnnotation vtkImagingColor vtkRenderingContext2D vtkRenderingGL2PS vtkRenderingContextOpenGL /usr/lib/libgl2ps.so vtkRenderingLabel dl m pthread rt /usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so tbb
  
  --     3rdparty dependencies:       libprotobuf
  
  -- 
  
  --   OpenCV modules:
  
  --     To be built:                 core flann hdf imgproc ml photo reg surface_matching video viz dnn freetype fuzzy imgcodecs shape videoio highgui objdetect plot superres ts xobjdetect xphoto bgsegm bioinspired dpm face features2d line_descriptor saliency text calib3d ccalib cvv datasets rgbd stereo tracking videostab xfeatures2d ximgproc aruco optflow phase_unwrapping stitching structured_light java python2 python3
  
  --     Disabled:                    world contrib_world
  
  --     Disabled by dependency:      -
  
  --     Unavailable:                 cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev cnn_3dobj matlab sfm
  
  -- 
  
  --   GUI: 
  
  --     QT 5.x:                      YES (ver 5.5.1)
  
  --     QT OpenGL support:           YES (Qt5::OpenGL 5.5.1)
  
  --     OpenGL support:              YES (/usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so)
  
  --     VTK support:                 YES (ver 6.2.0)
  
  -- 
  
  --   Media I/O: 
  
  --     ZLib:                        /usr/lib/x86_64-linux-gnu/libz.so (ver 1.2.8)
  
  --     JPEG:                        /usr/lib/x86_64-linux-gnu/libjpeg.so (ver )
  
  --     WEBP:                        /usr/lib/x86_64-linux-gnu/libwebp.so (ver encoder: 0x0202)
  
  --     PNG:                         /usr/lib/x86_64-linux-gnu/libpng.so (ver 1.2.54)
  
  --     TIFF:                        /usr/lib/x86_64-linux-gnu/libtiff.so (ver 42 - 4.0.6)
  
  --     JPEG 2000:                   /usr/lib/x86_64-linux-gnu/libjasper.so (ver 1.900.1)
  
  --     OpenEXR:                     /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so (ver 2.2.0)
  
  --     GDAL:                        /usr/lib/libgdal.so
  
  --     GDCM:                        NO
  
  -- 
  
  --   Video I/O:
  
  --     DC1394 1.x:                  NO
  
  --     DC1394 2.x:                  YES (ver 2.2.4)
  
  --     FFMPEG:                      YES
  
  --       avcodec:                   YES (ver 56.60.100)
  
  --       avformat:                  YES (ver 56.40.101)
  
  --       avutil:                    YES (ver 54.31.100)
  
  --       swscale:                   YES (ver 3.1.101)
  
  --       avresample:                NO
  
  --     GStreamer:                   NO
  
  --     OpenNI:                      NO
  
  --     OpenNI PrimeSensor Modules:  NO
  
  --     OpenNI2:                     NO
  
  --     PvAPI:                       NO
  
  --     GigEVisionSDK:               NO
  
  --     Aravis SDK:                  NO
  
  --     UniCap:                      NO
  
  --     UniCap ucil:                 NO
  
  --     V4L/V4L2:                    NO/YES
  
  --     XIMEA:                       NO
  
  --     Xine:                        YES (ver 1.2.6)
  
  --     gPhoto2:                     NO
  
  -- 
  
  --   Parallel framework:            TBB (ver 4.4 interface 9002)
  
  -- 
  
  --   Other third-party libraries:
  
  --     Use IPP:                     9.0.1 [9.0.1]
  
  --          at:                     /home/me/code/myproject/OpenCV-3.2.0/build/3rdparty/ippicv/ippicv_lnx
  
  --     Use IPP Async:               NO
  
  --     Use VA:                      NO
  
  --     Use Intel VA-API/OpenCL:     NO
  
  --     Use Lapack:                  NO
  
  --     Use Eigen:                   YES (ver 3.2.92)
  
  --     Use Cuda:                    NO
  
  --     Use OpenCL:                  YES
  
  --     Use OpenVX:                  NO
  
  --     Use custom HAL:              NO
  
  -- 
  
  --   OpenCL:                        <dynamic loading="loading" of="of" opencl="opencl" library="library">
  
  --     Include path:                /home/me/code/myproject/OpenCV-3.2.0/3rdparty/include/opencl/1.2
  
  --     Use AMDFFT:                  NO
  
  --     Use AMDBLAS:                 NO
  
  -- 
  
  --   Python 2:
  
  --     Interpreter:                 /home/me/.envs/myenv/bin/python (ver 2.7.12)
  
  --     Libraries:                   /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12)
  
  --     numpy:                       /home/me/.envs/myenv/local/lib/python2.7/site-packages/numpy/core/include (ver 1.13.1)
  
  --     packages path:               lib/python2.7/site-packages
  
  -- 
  
  --   Python 3:
  
  --     Interpreter:                 /usr/bin/python3 (ver 3.5.2)
  
  --     Libraries:                   /usr/lib/x86_64-linux-gnu/libpython3.5m.so (ver 3.5.2)
  
  --     numpy:                       /usr/lib/python3/dist-packages/numpy/core/include (ver 1.11.0)
  
  --     packages path:               lib/python3.5/dist-packages
  
  -- 
  
  --   Python (for build):            /home/me/.envs/myenv/bin/python
  
  -- 
  
  --   Java:
  
  --     ant:                         /usr/bin/ant (ver 1.9.6)
  
  --     JNI:                         /usr/lib/jvm/default-java/include /usr/lib/jvm/default-java/include/linux /usr/lib/jvm/default-java/include
  
  --     Java wrappers:               YES
  
  --     Java tests:                  YES
  
  -- 
  
  --   Matlab:                        Matlab not found or implicitly disabled
  
  -- 
  
  --   Documentation:
  
  --     Doxygen:                     /usr/bin/doxygen (ver 1.8.11)
  
  -- 
  
  --   Tests and samples:
  
  --     Tests:                       YES
  
  --     Performance tests:           YES
  
  --     C/C++ Examples:              YES
  
  -- 
  
  --   Install path:                  /usr/local
  
  -- 
  
  --   cvconfig.h is in:              /home/me/code/myproject/OpenCV-3.2.0/build
  
  -- -----------------------------------------------------------------
  
  -- 
  
  </dynamic>

  Unfortunately, while this works and I can import cv2 in the shell, it cannot read video using the above script, probably due to incorrect compilation or linking of ffmpeg ? The confusing part is the system-wide installation of OpenCV works fine, even without ffmpeg installed !

  What am I doing wrong ? How can I get OpenCV working with a virtualenv ?

  ====

  EDIT : Running the C++ video writing example has this result :

  $ cd /home/me/code/myproject/OpenCV-3.2.0/build/bin
  
  $ ./cpp-tutorial-video-write ../../samples/data/vtest.avi R Y
  
  ------------------------------------------------------------------------------
  
  This program shows how to write video files.
  
  You can extract the R or G or B color channel of the input video.
  
  Usage:
  
  ./video-write  [ R | G | B] [Y | N]
  
  ------------------------------------------------------------------------------
  
  
  
  OpenCV: FFMPEG: tag 0xffffffff/'����' is not found (format 'avi / AVI (Audio Video Interleaved)')'
  
  
  
  (cpp-tutorial-video-write:19523): GStreamer-CRITICAL **: gst_element_make_from_uri: assertion 'gst_uri_is_valid (uri)' failed
  
  OpenCV Error: Unsupported format or combination of formats (Gstreamer Opencv backend does not support this codec.) in CvVideoWriter_GStreamer::open, file /home/me/code/myproject/OpenCV-3.2.0/modules/videoio/src/cap_gstreamer.cpp, line 1388
  
  VIDEOIO(cvCreateVideoWriter_GStreamer(filename, fourcc, fps, frameSize, is_color)): raised OpenCV exception:
  
  
  
  /home/me/code/myproject/OpenCV-3.2.0/modules/videoio/src/cap_gstreamer.cpp:1388: error: (-210) Gstreamer Opencv backend does not support this codec. in function CvVideoWriter_GStreamer::open
  
  
  
  Could not open the output video for write: ../../samples/data/vtest.avi

  And the opencv_test_videoio unit test reports the following : https://pastebin.com/q4mf224Q

  However, running the c++ video starter example DOES work, with the following command and output, I can see the webcam working and streaming video in the highgui interface :

  $ ./cpp-example-videocapture_starter 0
  
  VIDEOIO ERROR: V4L: device 0: Unable to query number of channels
  
  (ERROR)icvOpenAVI_XINE(): Unable to initialize video driver.
  
  GStreamer: Error opening bin: no element "0"
  
  press space to save a picture. q or esc to quit
  
  init done
  
  opengl support available