Recherche avancée

Sur d’autres sites (4996)

• Is there problem with 'read' command in Bash, or in Bash itself when using multiprocessing, or may be I make some mistake ? [duplicate]

  27 janvier 2023, par myQs

  First to mention that I do not have lot of experiences with Bash scripting.
  
Here is the problem that I observe :

  
When I execute read command and inside the cycle I run background processes, the read command misses some of the arguments in some very rare cases.

  
For example : if I read the output of ls -la for big number of video files and on each of them I execute ffmpeg command in a different sub-process, then in some very rare cases there are missing some of the first parameters of read command.
  
In that case the rest of parameters of the ls are wrong (having partial of their real values or wrongly assigned).

  
I most of the cases I have an output like this (which is correct) :
  
p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_453.mp4'
  

  


  but for very few lines I have not correct output and it is like this :
  
p1: 'an'; p2: '1080519'; p3: 'Jan'; p4: '27'; p5: '05:49'; p6: 'origVideo_454.mp4'; p7: ''; p8: ''; p9: ''
  

  


  Here p1 and p2 are missing and p3 should be "uman" but is just "an". And p3 becomes p1, p4 becomes p2, etc, in this way p7, p8 and p9 remain without values.

  


  Here is my bash script :

  


  #!/bin/bash

#src_dir=/tmp/text_files
src_dir=/tmp/video_files

dest_dir=/tmp/video_files_dest

mkdir -p $dest_dir

handle_video() {
  echo "handling file: '$1'"
  ffmpeg -loglevel error -i $src_dir/$1  -acodec copy -vcodec copy $dest_dir/$1
}

generate_text() {
  str=''
  for k in {1..512}
  do
    random=$(openssl rand -hex 20)
    str="${str}${random} "

    if [ $(( $k % 4 )) -eq 0 ]; then
      str="${str} ${new_line}"
    fi
  done

  echo "${str}" > $src_dir/$1
}

while read p1 p2 p3 p4 p5 p6 p7 p8 p9; do
echo "p1: '$p1'; p2: '$p2'; p3: '$p3'; p4: '$p4'; p5: '$p5'; p6: '$p6'; p7: '$p7'; p8: '$p8'; p9: '$p9'"
  if test -f $src_dir/$p9; then
    handle_video $p9 &
#    generate_text $p9 &
  fi
done << EOF
$(ls -la $src_dir)
EOF


  

  
**When I run the `handle_video` not in background but in same thread** I do not have such problem (remove `&` from line 33).

  
First I thought the issue might be in the output of the command `la -ls` and I tried with other commands, but I saw the same kind of results - in most of the executions `read` has correct parameters but in very few cases they are wrong.

  


  I also tried the script instead with handle_video (which uses ffmpeg invocation) to run different function that is executed inside read cycle : the generate_text.
  
To do this I comment lines 3 and 33 and uncomment lines 4 and 34.
  
And the interesting thing is that when executing it with handle_video problem exists but when executing with generate_text there is no such problem at all. At least I have never observed it in all my tests.
  
When executing it with handle_video I put 1200 video .mp4 files (1.1 MB each) in directory /tmp/video_files and run bash script.

  
When executing it with generate_text I generate 1200 empty files in directory /tmp/text_files and run the bash script.

  


  I also tried to execute the read command with piping like this, but the result is the same :
  

  


  ls -la $src_dir | while read p1 p2 p3 p4 p5 p6 p7 p8 p9; do
  echo "p1: '$p1'; p2: '$p2'; p3: '$p3'; p4: '$p4'; p5: '$p5'; p6: '$p6'; p7: '$p7'; p8: '$p8'; p9: '$p9'"
  if test -f $src_dir/$p9; then
    handle_video $p9 &
#   generate_text $p9 &
  fi
done


  

  


  Bash version is : 5.2.15(1)-release
  
ffmpeg version 5.0.2
  
Guest OS : Fedora version : "36 (Workstation Edition)"
  
VirtualBox 7.0.2
  
Host OS : is Windows 10 version : 21H2
  


  


  Once again when I do not run the function handle_video in background (at line 33 remove the ampersand &) there are no problems.
  
And when I use instead of handle_video the function generate_text again there are no problems.

  
So I wonder is there problem in the read method and how it gets the arguments, or is there problem with bash how it is being executing multiple processes, or there is something that I do not understand.
  
Any help and tips are appreciated.

  


  Here is a snippet of real output :
  

  


  p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_453.mp4'
handling file: 'origVideo_448.mp4'
handling file: 'origVideo_449.mp4'
handling file: 'origVideo_44.mp4'
handling file: 'origVideo_450.mp4'
handling file: 'origVideo_451.mp4'
handling file: 'origVideo_452.mp4'
p1: 'an'; p2: '1080519'; p3: 'Jan'; p4: '27'; p5: '05:49'; p6: 'origVideo_454.mp4'; p7: ''; p8: ''; p9: ''
p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_455.mp4'
p1: '-rwxr-x---.'; p2: '1'; p3: 'uman'; p4: 'uman'; p5: '1080519'; p6: 'Jan'; p7: '27'; p8: '05:49'; p9: 'origVideo_456.mp4'


  


• avcodec/h2645_parse : Don't treat 0x000002 as a start code and truncate

  13 février 2024, par Mattias Wadman

  avcodec/h2645_parse : Don't treat 0x000002 as a start code and truncate
  According to ITU-T H.265 7.4.2.1 this byte sequence should not appear in a
  
  NAL unit but in practice in rare cases it seems it does, possibly due to buggy
  
  encoders. Other players like VLC and Quicktime seem to be fine with it.
  Currently when this sequence is found it is treated as if the next start code
  
  has been found and the NAL unit gets truncated.
  This change limits the code to only look for first start code 0x0000001 or
  
  first escape 0x000003.
  Sadly i can't share the original source file with the issue but the first
  
  80 bytes of the NAL unit looks like this :
         │00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f│0123456789abcdef│
  
  0x00000│00 00 00 01 02 01 d0 bc 57 a1 b8 44 70 01 00 0b│........W..Dp...│
  
  0x00010│80 2e 00 c2 6c ec 3e b9 e3 03 fb 91 2e d2 43 cb│....l.>.......C.│
  
  0x00020│1d 2c 00 00 02 00 02 00 5c 93 72 6f 31 76 18 00│.,......\.ro1v..│
  
  0x00030│08 38 aa b1 4c 33 3f fd 08 cb 77 9b d4 3c db 02│.8..L3 ?...w..<..│
  
  0x00040│a2 04 73 15 75 de 3b c4 67 c0 8f ca ad 31 f1 99│..s.u. ;.g....1..│
  Signed-off-by : Anton Khirnov <anton@khirnov.net>
  

  • [DH] libavcodec/h2645_parse.c

• Failed to decode HLS by FFMpeg command. Invalid NAL unit 0

  9 mars 2024, par Fyodor Khruschov

  On front-end I create stream with chrome.tabCapture.capture or navigator.mediaDevices.getDisplayMedia methods. Then send chunks generated by MediaRecorder to server. On the server I have FFMpeg command which decodes chunks into .mp4 file. This is the command :

  


  ffmpeg -y -i - -preset veryfast -tune zerolatency -filter_complex [0:v]split=3[v1][v2][v3];[v1]scale=w=-2:h=1080,fps=30[v1out];[v2]scale=w=-2:h=720,fps=30[v2out];[v3]scale=w=-2:h=480,fps=30[v3out] -map [v1out] -maxrate:0 6M -bufsize:0 12M -keyint_min 100 -g 100 -map [v2out] -maxrate:1 3M -bufsize:1 6M -keyint_min 100 -g 100 -map [v3out] -maxrate:2 1.5M -bufsize:2 3M -keyint_min 100 -g 100 -c:v libx264 -map a:0 -c:a:0 aac -b:a:0 128k -ac 2 -map a:0 -c:a:1 aac -b:a:1 96k -map a:0 -c:a:2 aac -b:a:2 96k -f hls -hls_time 2 -hls_playlist_type vod -hls_flags independent_segments+temp_file -hls_segment_type fmp4 -hls_segment_filename ./output/ready/output_%v_%03d.m4s -var_stream_map v:0,a:0 v:1,a:1 v:2,a:2 -master_pl_name master.m3u8 ./output/ready/stream_%v.m3u8 -map 0:v:0 -map 0:a:0 -c:v copy -c:a aac ./output/download/video.mp4 -map 0:a:0 -ar 16000 -ac 1 -c:a pcm_s16le ./output/captions/audio.wav -loglevel info


  


  During the process of decoding I have these errors in logs :

  


  [extract_extradata @ 0x60000264b250] Invalid NAL unit 0, skipping.
[h264 @ 0x13ff04e60] Invalid NAL unit 0, skipping.
[h264 @ 0x13ff04e60] co located POCs unavailable
[h264 @ 0x13ff04e60] negative number of zero coeffs at 17 0
[h264 @ 0x13ff04e60] error while decoding MB 17 0
[h264 @ 0x13ff04e60] concealing 3388 DC, 3388 AC, 3388 MV errors in B frame
[h264 @ 0x13ff04e60] missing picture in access unit with size 24158
[h264 @ 0x13ff04e60] Invalid NAL unit 0, skipping.
[h264 @ 0x13ff04e60] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x13ff04e60] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[h264 @ 0x13ff04e60] no frame!
[h264 @ 0x13ff04e60] Unknown SAR index: 18.
[h264 @ 0x13ff04e60] Invalid NAL unit 0, skipping.
[h264 @ 0x13ff04e60] Unknown SAR index: 18.
[h264 @ 0x13ff04e60] number of reference frames (2+4) exceeds max (5; probably corrupt input), discarding one
[h264 @ 0x13ff04e60] number of reference frames (3+3) exceeds max (5; probably corrupt input), discarding one
[h264 @ 0x13ff04e60] number of reference frames (4+2) exceeds max (5; probably corrupt input), discarding one
[h264 @ 0x13ff04e60] FMO is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[h264 @ 0x13ff04e60] sps_id 4 out of range


  


  This issue is very inconsistent and happen in rare cases (I can't understand the logic). Most of the time chunks decoded successfully, but sometimes not.

  


  How to understand where the issue is coming from ? Is it possible for FFMpeg to skip wrong data and generate mp4 file anyway even with glitches, but don't crush ?