Recherche avancée

Sur d’autres sites (10295)

• ffmpeg azure function consumption plan low CPU availability for high volume requests

  27 novembre 2017, par The Lemon

  I am running an azure queue function on a consumption plan ; my function starts an FFMpeg process and accordingly is very CPU intensive. When I run the function with less than 100 items in the queue at once it works perfectly, azure scales up and gives me plenty of servers and all of the tasks complete very quickly. My problem is once I start doing more than 300 or 400 items at once, it starts fine but after a while the CPU slowly goes from 80% utilisation to only around 10% utilisation - my functions cant finish in time with only 10% CPU. This can be seen in the image shown below.
  Does anyone know why the CPU useage is going lower the more instances my function creates ? Thanks in advance Cuan

  edit : the function is set to only run one at a time per instance, but the problem exists when set to 2 or 3 concurrent processes per instance in the host.json

  edit : the CPU drops get noticeable at 15-20 servers, and start causing failures at around 60. After that the CPU bottoms out at an average of 8-10% with individuals reaching 0-3%, and the server count seems to increase without limit (which would be more helpful if I got some CPU with the servers)

  Thanks again, Cuan.

  I’ve also added the function code to the bottom of this post in case it helps.

  

  

  using System.Net;
  
  using System;
  
  using System.Diagnostics;
  
  using System.ComponentModel;
  
  
  
  public static void Run(string myQueueItem, TraceWriter log)
  
  {
  
      log.Info($"C# Queue trigger function processed a request: {myQueueItem}");
  
      //Basic Parameters
  
          string ffmpegFile = @"D:\home\site\wwwroot\CommonResources\ffmpeg.exe";
  
          string outputpath = @"D:\home\site\wwwroot\queue-ffmpeg-test\output\";
  
          string reloutputpath = "output/";
  
          string relinputpath = "input/";
  
          string outputfile = "video2.mp4";
  
          string dir =  @"D:\home\site\wwwroot\queue-ffmpeg-test\";
  
  
  
      //Special Parameters
  
  
  
          string videoFile = "1 minute basic.mp4";
  
          string sub = "1 minute sub.ass";
  
      //guid tmp files
  
  
  
          // Guid g1=Guid.NewGuid();
  
          // Guid g2=Guid.NewGuid();
  
          // string f1 = g1 + ".mp4";
  
          // string f2 = g2 + ".ass";
  
          string f1 = videoFile;
  
          string f2 = sub;
  
      //guid output - we will now do this at the caller level
  
          string g3 = myQueueItem;
  
          string outputGuid = g3+".mp4";
  
      //get input files
  
      //argument
  
          string tmp = subArg(f1, f2, outputGuid );
  
      //String.Format("-i \"" + @"input/tmp.mp4" + "\" -vf \"ass = '" + sub + "'\" \"" + reloutputpath +outputfile + "\" -y");
  
      log.Info("ffmpeg argument is: "+tmp);
  
  
  
  
  
      //startprocess parameters
  
      Process process = new Process();
  
      process.StartInfo.FileName = ffmpegFile;
  
      process.StartInfo.Arguments =  tmp;
  
      process.StartInfo.UseShellExecute = false;
  
      process.StartInfo.RedirectStandardOutput = true;
  
      process.StartInfo.RedirectStandardError = true;
  
      process.StartInfo.WorkingDirectory = dir;
  
      //output handler
  
  
  
      process.OutputDataReceived += new DataReceivedEventHandler(
  
          (s, e) => 
  
          { 
  
              log.Info("O: "+e.Data);
  
          }
  
      );
  
      process.ErrorDataReceived += new DataReceivedEventHandler(
  
          (s, e) => 
  
          { 
  
              log.Info("E: "+e.Data);
  
          }
  
      );
  
      //start process
  
      process.Start();
  
      log.Info("process started");
  
      process.BeginOutputReadLine();
  
      process.BeginErrorReadLine();
  
      process.WaitForExit();
  
  }
  
  public static void getFile(string link, string fileName, string dir, string relInputPath){
  
      using (var client = new WebClient()){
  
          client.DownloadFile(link, dir + relInputPath+ fileName);
  
          }
  
  
  
  }
  
  public static string subArg(string input1, string input2, string output1){
  
      return String.Format("-i \"" + @"input/" +input1+ "\" -vf \"ass = '" + @"input/"+input2 + "'\" \"" + @"output/" +output1 + "\" -y");
  
  
  
  }

• Evolution #4462 (Nouveau) : Rendu du plan

  20 mars 2020, par jluc -

  Sur la page ?exec=plan on peut déployer l’arborescence des rubriques. Il y a toutefois un déficit d’indications visuelles sur cette possibilité car
  - lorsqu’on hover le nom d’une rubrique, le style change et on sait qu’on peut cliquer, mais lorsqu’on clique sur le nom d’une rubriques on est projeté vers la page de cette rubrique, dans le même onglet
  - il y a bien un minitruc pâlot triangulaire à gauche, mais il ne change pas de style au survol (à part le pointeur de la souris) et sa minitaille et son faible contraste n’invitent pas au survol, et le simple changement de curseur invite mal au clic. C’est pourtant lui qu’il faut cliquer pour déployer la rubrique.

  Il serait utile de bénéficier d’indications plus claires facilitant l’usage de l’arbre du plan.
  - peut être un ’+’ plus gros et plus contrasté que ce triangle
  - et en tout cas, un changement d’apparence au survol (+ un title ?)
  - une indication textuelle sur ce que permet cette page (le compagnon le fait il déjà peut être ?). Par exemple "Vous pouvez déployer les rubriques en cliquant le + ou le triangle à gauche, et vous pouvez déplacer les rubriques et leur contenu dans l’arborescence par glisser-déposer"

• CJEU rules US cloud servers don’t comply with GDPR and what this means for web analytics

  17 juillet 2020, par Jake Thornton
  Breaking news : On July 16, 2020, the Court of Justice of the European Union (CJEU) has ruled that any cloud services hosted in the US are incapable of complying with the GDPR and EU privacy laws.

  In August 2016, the EU-US Privacy Shield framework came into effect, which “protects the fundamental rights of anyone in the EU whose personal data is transferred to the United States for commercial purposes. It allows the free transfer of data to companies that are certified in the US under the Privacy Shield.” – European Commission website

  However after today’s CJEU ruling, this Privacy Shield framework became invalidated due to significant differences between EU and US privacy laws.

  European privacy law activist Max Schrems summarises with “The Court clarified for a second time now that there is a clash between EU privacy law and US surveillance law. As the EU will not change its fundamental rights to please the NSA, the only way to overcome this clash is for the US to introduce solid privacy rights for all people – including foreigners. Surveillance reform thereby becomes crucial for the business interests of Silicon Valley.” – noyb website

  Today’s ruling also continues to spark concern into the legitimacy of US privacy laws which doesn’t fully protect people’s personal data when hosted on cloud servers based in the US.

  

  How will this affect you ?

  For any business operating a website in the EU or if you have traffic coming to your website from EU visitors, you need to know what data you’re capturing and where this data is being stored.

  Here’s what Maja Smoltczyk (Berlin’s Commissioner for Data Protection and Freedom of Information) says :

  Controllers who transfer personal data to the USA, especially when using cloud-based services, are now required to switch immediately to service providers based in the European Union or a country that can
  ensure an adequate level of data protection. The CJEU has made it refreshingly clear that data exports are not just financial decisions, as people’s fundamental rights must also be considered as a matter of priority. This ruling will put
  an end to the transfer of personal data to the USA for the sake of convenience or to cut costs.

  The controller is you (not Google) and by transferring data to the US you are at risk of being fined up to €20 million or 4% of your annual worldwide turnover for not being GDPR compliant. 

  It’s you who has to take action, not Google or other US companies. The court’s decision has immediate effect. While we assume there will be a grace period, companies should act now as finding and implementing alternatives solution can take a while. 

  Can no data be exported outside the EU anymore ?

  Data can still be exported outside the EU if an adequate level of data protection is guaranteed. This is the case for some trading partners of the EU such as New Zealand, Japan, Switzerland, and Canada. They have been certified by the EU as having a comparable level of privacy protection and therefore demonstrate adequacy at a country level.

  Necessary data can still flow to countries like the US too. This is for example the case when someone books a hotel in the US or when sending an email to someone in the US. Backups for disaster recovery and most other reasons don’t qualify as necessary.

  In all other cases you can still send data to countries like the US if you get explicit and informed consent from a user. Meaning the user has been informed about all possible risks of sending the data to the US and who can access the data (for example the US government).

  How this affects Google Analytics and Google Tag Manager users

  If your website is using Google Analytics, the safest bet is to deactivate it immediately. Otherwise, you must ask for consent from everyone who visits your website and inform them that the data will be processed in the United States under less strict privacy laws and all associated risks. If you don’t, you could be liable to privacy law infringements and face being fined for not complying with the GDPR. This also applies to Google Tag Manager as it transfers the IP address to the US which is considered personal data under the GDPR.

  Consent needs to be :

  • Freely given (the user must have a choice to not give consent and be able to opt out at any time) 
  • Informed (you need to disclose who is processing the data, what data is processed, where the data will be stored and how to opt out) 
  • Specific (consent is only valid for the specific informed purpose) 
  • Unambiguous (for example pre-ticked boxes or similar aren’t allowed)

  

  If users don’t give you consent, you are not allowed to track them using Google Analytics or any other US based cloud solution.

  Update August 19, 2020

  A month after this ruling, over 100 complaints have been filed against websites for continuing to send data to the US via Google Analytics or Facebook, by the European privacy campaign group noyb. It’s clear Google and Facebook fall under US surveillance laws such as FISA 702 and the court clearly ruled these companies cannot rely on SCCs to transfer data to the US. Anyone still using Google Analytics is now at risk of facing fines and compensation damages. 

  How this affects Matomo users

  Our cloud servers are based in Germany.

  Matomo On-Premise users choose the location of their data themselves. If the servers are located in the EU nothing changes. If the servers are located outside the EU and the website targets EU users and tracks personal data, then you need to assess whether you are required to ask for tracking consent.

  If the data is stored inside the EU you can use Matomo without asking for any consent and you can continue tracking users even if they reject a consent screen which greatly increases the quality of your data.

  Want to avoid informing users about transferring their data to the US and all associated risks ?

  Try Matomo now for free ! No credit card required.

  Download Matomo