Recherche avancée

Médias (91)

Autres articles (96)

  • MediaSPIP 0.1 Beta version

    25 avril 2011, par

    MediaSPIP 0.1 beta is the first version of MediaSPIP proclaimed as "usable".
    The zip file provided here only contains the sources of MediaSPIP in its standalone version.
    To get a working installation, you must manually install all-software dependencies on the server.
    If you want to use this archive for an installation in "farm mode", you will also need to proceed to other manual (...)

  • Multilang : améliorer l’interface pour les blocs multilingues

    18 février 2011, par

    Multilang est un plugin supplémentaire qui n’est pas activé par défaut lors de l’initialisation de MediaSPIP.
    Après son activation, une préconfiguration est mise en place automatiquement par MediaSPIP init permettant à la nouvelle fonctionnalité d’être automatiquement opérationnelle. Il n’est donc pas obligatoire de passer par une étape de configuration pour cela.

  • Formulaire personnalisable

    21 juin 2013, par

    Cette page présente les champs disponibles dans le formulaire de publication d’un média et il indique les différents champs qu’on peut ajouter. Formulaire de création d’un Media
    Dans le cas d’un document de type média, les champs proposés par défaut sont : Texte Activer/Désactiver le forum ( on peut désactiver l’invite au commentaire pour chaque article ) Licence Ajout/suppression d’auteurs Tags
    On peut modifier ce formulaire dans la partie :
    Administration > Configuration des masques de formulaire. (...)

Sur d’autres sites (8150)

  • MediaCodec hardware decoder much slower with different server configuration ?

    19 octobre 2013, par mathieujofis

    I've been using the Android MediaCodec in order to (hardware) decode H.264 frames on my Galaxy S4 coming from a Live 555 RTSP live (real-time) stream. After changing my Live 555 server configuration from using ffmpeg (with x264) to encode frames, to using strictly x264 to encode frames, the time to decode frames with MediaCodec takes much longer. Basically, MediaCodec can't keep up with the stream, and displays the video in slow motion, getting slower and slower as time goes on. Going back to ffmpeg isn't a solution for me, because I need the ability to encode into discrete NAL units, rather than a whole frame like ffmpeg does.

    I was wondering if this was either : A) An issue with the way my server is encoding NAL units, or B) An issue with my Android client, specifically the way it is receiving and decoding NAL units.

    My encoding configuration with x264 is :

    x264_param_default_preset(&param,"ultrafast", "zerolatency:fastdecode");
    
param.i_threads = 1;
    
param.i_bframe = 1;
    
param.i_width = image_width;
    
param.i_height = image_height;
    
param.i_fps_num = 60;
    
param.i_fps_den = 1;
    
param.i_keyint_max = 10;
    

    
param.rc.i_rc_method = X264_RC_ABR;
    
param.rc.i.bitrate = 6000;
    
param.i_sps_id = 7;
    
param.b_repeat_headers = 1;
    
param.b_annexb = 0;

    My Android MediaCodec client is set up as follows :

    I receive each individual NAL unit on a separate Live 555 RTSP client thread. Each NAL is put into a queue along with its size and presentation time. A separate decoder thread grabs NALs from this queue, and if there are none available, waits until there are.

    Some notes :

    What I generally see happen is the queue starts filling up with NALs, instead of staying close to empty. So, I know that the decoder thread is not working fast enough. I don't think this is an inherent problem with decoding on an Android phone (for example, processing limitations) because it does the same thing for very low bitrates— also again, it DID work when I was using ffmpeg to encode. If I omit certain NAL units, the decoder can start to keep up. Since I'm using Cyanogenmod 10.1, bumping up the minimum CPU frequency helps, too.

    Edit :

    Here is a log of the Android client, as well as a log highlighting the garbage collector specifically—

    Entire Logcat :

    10-15 16:40:03.955: D/DecodeActivity(18859): INFO_OUTPUT_BUFFERS_CHANGED
    
10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Sync frame received
    
10-15 16:40:03.995: E/OMX-VDEC-1080P(288):  No color conversion required
    
10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Get_parameter: OMX_IndexParamPortDefinition: nPortIndex (1), nFrameWidth (1280), nFrameHeight (720), nStride (1280), nSliceHeight (736), nBitrate (-1073741824), xFramerate (0x1e), nBufferSize (1433600), nBufferCountMin (4), nBufferCountActual (8), bBuffersContiguous (1918394328), nBufferAlignment (1075643347), bEnabled (1), bPopulated (1), eCompressionFormat (0x0), eColorFormat (0x7fa30c03)
    
10-15 16:40:03.995: D/DecodeActivity(18859): New format {height=720, what=1869968451, color-format=2141391875, slice-height=736, crop-left=0, width=1280, crop-bottom=719, crop-top=0, mime=video/raw, stride=1280, crop-right=1279}
    
10-15 16:40:04.005: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4966) fps(201.369308)
    
10-15 16:40:04.015: W/IInputConnectionWrapper(1069): showStatusIcon on inactive InputConnection
    
10-15 16:40:04.025: I/ActivityManager(698): Displayed com.mathieu.alloclient.javadecoder/.MainActivity: +617ms
    
10-15 16:40:04.025: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4766) fps(209.819550)
    
10-15 16:40:04.125: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4733) fps(211.282486)
    
10-15 16:40:04.445: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4711) fps(212.269150)
    
10-15 16:40:04.495: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4700) fps(212.765961)
    
10-15 16:40:05.676: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4688) fps(213.310577)
    
10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
    
10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K, paused 7ms+9ms, total 117ms
    
10-15 16:40:06.537: D/ALSADevice(288): standby: handle 0x40024450 h 0x0
    
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _verb value Inactive
    
10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for HiFi Lowlatency enable 0
    
10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: SLIMBUS_0_RX Audio Mixer MultiMedia5, value: 0
    
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Line
    
10-15 16:40:06.577: D/alsa_ucm(288): disdev: device Line not enabled, no need to disable
    
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Speaker
    
10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for Speaker enable 0
    
10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP1, value: ZERO
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP2, value: ZERO
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT2 Volume, value: 0
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT4 Volume, value: 0
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 Digital Volume, value: 0
    
10-15 16:40:06.587: D/AudioUsbALSA(288): exitPlaybackThread, mproxypfdPlayback: -1
    
10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
    
10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
    
10-15 16:40:17.638: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4677) fps(213.812271)
    
10-15 16:40:17.698: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4644) fps(215.331604)
    
10-15 16:40:20.681: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4633) fps(215.842865)
    
10-15 16:40:21.111: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4611) fps(216.872696)
    
10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
    
10-15 16:40:28.448: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 74
    
10-15 16:40:28.448: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
    
10-15 16:40:30.841: W/SystemClock(698): time going backwards: prev 16555345563411(ioctl) vs now 16555345441341(ioctl), tid=764
    
10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
    
10-15 16:40:57.467: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4577) fps(218.483719)
    
10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
    
10-15 16:41:14.403: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.500000 hotplug_avg_load_dw: 89
    
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:922.000000
    
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 48
    
10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
    
10-15 16:41:27.466: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
    
10-15 16:41:27.466: E/kickstart(862): EVENT: sahara_mode                         = 2
    
10-15 16:41:27.466: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
    
10-15 16:41:27.466: E/kickstart(862): EVENT: helloRx.mode                        = 2
    
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 64
    
10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
    
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 116
    
10-15 16:41:27.466: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs1", ""
    
10-15 16:41:27.466: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
    
10-15 16:41:27.466: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs1"
    
10-15 16:41:27.526: E/kickstart(862): Total bytes received so far: 786548
    
10-15 16:41:27.526: E/kickstart(862): EVENT: Received: 786432 bytes
    
10-15 16:41:27.526: E/kickstart(862): EVENT: Writing to disk
    
10-15 16:41:27.526: E/kickstart(862): EVENT: Successfully wrote to disk
    
10-15 16:41:27.526: E/kickstart(862): Received file "m9kefs1"
    
10-15 16:41:27.576: E/kickstart(862): Sync finish Received file "m9kefs1"
    
10-15 16:41:27.576: E/kickstart(862): 786432 bytes transferred in 0.106s (7.10 MBps)
    
10-15 16:41:27.576: E/kickstart(862): EVENT: num_debug_entries not >=0
    
10-15 16:41:27.576: E/kickstart(862): Successfully downloaded files from target 
    
10-15 16:41:27.576: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
    
10-15 16:41:27.576: E/kickstart(862): Total bytes received so far: 786556
    
10-15 16:41:27.576: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
    
10-15 16:41:27.576: E/kickstart(862): Sahara protocol completed
    
10-15 16:41:27.576: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
    
10-15 16:41:27.746: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.100000 hotplug_avg_load_dw: 79
    
10-15 16:41:27.746: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
    
10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
    
10-15 16:42:17.225: I/ActivityManager(698): Start proc com.cyanogenmod.lockclock for service com.cyanogenmod.lockclock/.weather.WeatherUpdateService: pid=18954 uid=10028 gids={50028, 3003, 1028}
    
10-15 16:42:17.865: D/WeatherXmlParser(18954): Weather updated: WeatherInfo for Santa Barbara@ Tue Oct 15 16:42:17 PDT 2013: Fair(34), temperature 29°C, low 11°, high 27°, humidity 14%, wind 11km/h at W
    
10-15 16:42:17.945: I/ActivityManager(698): No longer want com.google.android.apps.uploader (pid 13565): empty #17
    
10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
    
10-15 16:42:35.983: W/ThrottleService(698): unable to find stats for iface rmnet0
    
10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
    
10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 48
    
10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
    
10-15 16:43:02.098: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
    
10-15 16:43:02.098: E/kickstart(862): EVENT: sahara_mode                         = 2
    
10-15 16:43:02.098: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
    
10-15 16:43:02.098: E/kickstart(862): EVENT: helloRx.mode                        = 2
    
10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 64
    
10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
    
10-15 16:43:02.108: E/kickstart(862): Total bytes received so far: 116
    
10-15 16:43:02.108: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs2", ""
    
10-15 16:43:02.108: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
    
10-15 16:43:02.108: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs2"
    
10-15 16:43:02.158: E/kickstart(862): Total bytes received so far: 786548
    
10-15 16:43:02.158: E/kickstart(862): EVENT: Received: 786432 bytes
    
10-15 16:43:02.158: E/kickstart(862): EVENT: Writing to disk
    
10-15 16:43:02.168: E/kickstart(862): EVENT: Successfully wrote to disk
    
10-15 16:43:02.168: E/kickstart(862): Received file "m9kefs2"
    
10-15 16:43:02.218: E/kickstart(862): Sync finish Received file "m9kefs2"
    
10-15 16:43:02.218: E/kickstart(862): 786432 bytes transferred in 0.113s (6.65 MBps)
    
10-15 16:43:02.218: E/kickstart(862): EVENT: num_debug_entries not >=0
    
10-15 16:43:02.218: E/kickstart(862): Successfully downloaded files from target 
    
10-15 16:43:02.218: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
    
10-15 16:43:02.218: E/kickstart(862): Total bytes received so far: 786556
    
10-15 16:43:02.218: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
    
10-15 16:43:02.218: E/kickstart(862): Sahara protocol completed
    
10-15 16:43:02.218: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
    
10-15 16:43:17.563: W/SystemClock(698): time going backwards: prev 16722067121029(ioctl) vs now 16722066968441(ioctl), tid=764
    
10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
    
10-15 16:43:36.452: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 100
    
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:194.000000
    
10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
    
10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
    
10-15 16:44:10.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 133
    
10-15 16:44:10.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
    
10-15 16:44:12.707: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.100000 hotplug_avg_load_dw: 116
    
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
    
10-15 16:44:14.008: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 140
    
10-15 16:44:14.008: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:240.000000
    
10-15 16:44:14.759: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.900000 hotplug_avg_load_dw: 132
    
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
    
10-15 16:44:15.360: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.300000 hotplug_avg_load_dw: 139
    
10-15 16:44:15.360: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
    
10-15 16:44:15.560: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 133
    
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:95.000000
    
10-15 16:44:16.361: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.500000 hotplug_avg_load_dw: 132
    
10-15 16:44:16.361: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:242.000000
    
10-15 16:44:16.661: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.700000 hotplug_avg_load_dw: 127
    
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
    
10-15 16:44:20.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 122
    
10-15 16:44:20.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
    
10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19038
    
10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19041

    Garbage collector entries in Logcat :

    10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
    
10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K,   paused 7ms+9ms, total 117ms
    
10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
    
10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
    
10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
    
10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
    
10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
    
10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
    
10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
    
10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
    
10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
    
10-15 16:44:24.389: D/dalvikvm(698): GC_EXPLICIT freed 3618K, 41% free 23768K/39664K, paused 12ms+11ms, total 123ms

  • MediaCodec hardware decoder much slower with different server configuration ?

    19 octobre 2013, par mathieujofis

    I've been using the Android MediaCodec in order to (hardware) decode H.264 frames on my Galaxy S4 coming from a Live 555 RTSP live (real-time) stream. After changing my Live 555 server configuration from using ffmpeg (with x264) to encode frames, to using strictly x264 to encode frames, the time to decode frames with MediaCodec takes much longer. Basically, MediaCodec can't keep up with the stream, and displays the video in slow motion, getting slower and slower as time goes on. Going back to ffmpeg isn't a solution for me, because I need the ability to encode into discrete NAL units, rather than a whole frame like ffmpeg does.

    I was wondering if this was either : A) An issue with the way my server is encoding NAL units, or B) An issue with my Android client, specifically the way it is receiving and decoding NAL units.

    My encoding configuration with x264 is :

    x264_param_default_preset(&param,"ultrafast", "zerolatency:fastdecode");
    
param.i_threads = 1;
    
param.i_bframe = 1;
    
param.i_width = image_width;
    
param.i_height = image_height;
    
param.i_fps_num = 60;
    
param.i_fps_den = 1;
    
param.i_keyint_max = 10;
    

    
param.rc.i_rc_method = X264_RC_ABR;
    
param.rc.i.bitrate = 6000;
    
param.i_sps_id = 7;
    
param.b_repeat_headers = 1;
    
param.b_annexb = 0;

    My Android MediaCodec client is set up as follows :

    I receive each individual NAL unit on a separate Live 555 RTSP client thread. Each NAL is put into a queue along with its size and presentation time. A separate decoder thread grabs NALs from this queue, and if there are none available, waits until there are.

    Some notes :

    What I generally see happen is the queue starts filling up with NALs, instead of staying close to empty. So, I know that the decoder thread is not working fast enough. I don't think this is an inherent problem with decoding on an Android phone (for example, processing limitations) because it does the same thing for very low bitrates— also again, it DID work when I was using ffmpeg to encode. If I omit certain NAL units, the decoder can start to keep up. Since I'm using Cyanogenmod 10.1, bumping up the minimum CPU frequency helps, too.

    Edit :

    Here is a log of the Android client, as well as a log highlighting the garbage collector specifically—

    Entire Logcat :

    10-15 16:40:03.955: D/DecodeActivity(18859): INFO_OUTPUT_BUFFERS_CHANGED
    
10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Sync frame received
    
10-15 16:40:03.995: E/OMX-VDEC-1080P(288):  No color conversion required
    
10-15 16:40:03.995: E/OMX-VDEC-1080P(288): Get_parameter: OMX_IndexParamPortDefinition: nPortIndex (1), nFrameWidth (1280), nFrameHeight (720), nStride (1280), nSliceHeight (736), nBitrate (-1073741824), xFramerate (0x1e), nBufferSize (1433600), nBufferCountMin (4), nBufferCountActual (8), bBuffersContiguous (1918394328), nBufferAlignment (1075643347), bEnabled (1), bPopulated (1), eCompressionFormat (0x0), eColorFormat (0x7fa30c03)
    
10-15 16:40:03.995: D/DecodeActivity(18859): New format {height=720, what=1869968451, color-format=2141391875, slice-height=736, crop-left=0, width=1280, crop-bottom=719, crop-top=0, mime=video/raw, stride=1280, crop-right=1279}
    
10-15 16:40:04.005: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4966) fps(201.369308)
    
10-15 16:40:04.015: W/IInputConnectionWrapper(1069): showStatusIcon on inactive InputConnection
    
10-15 16:40:04.025: I/ActivityManager(698): Displayed com.mathieu.alloclient.javadecoder/.MainActivity: +617ms
    
10-15 16:40:04.025: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4766) fps(209.819550)
    
10-15 16:40:04.125: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4733) fps(211.282486)
    
10-15 16:40:04.445: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4711) fps(212.269150)
    
10-15 16:40:04.495: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4700) fps(212.765961)
    
10-15 16:40:05.676: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4688) fps(213.310577)
    
10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
    
10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K, paused 7ms+9ms, total 117ms
    
10-15 16:40:06.537: D/ALSADevice(288): standby: handle 0x40024450 h 0x0
    
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _verb value Inactive
    
10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for HiFi Lowlatency enable 0
    
10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: SLIMBUS_0_RX Audio Mixer MultiMedia5, value: 0
    
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Line
    
10-15 16:40:06.577: D/alsa_ucm(288): disdev: device Line not enabled, no need to disable
    
10-15 16:40:06.577: D/alsa_ucm(288): snd_use_case_set(): uc_mgr 0x400fbfb0 identifier _disdev value Speaker
    
10-15 16:40:06.577: D/alsa_ucm(288): Set mixer controls for Speaker enable 0
    
10-15 16:40:06.577: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP1, value: ZERO
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 MIX1 INP2, value: ZERO
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT2 Volume, value: 0
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: LINEOUT4 Volume, value: 0
    
10-15 16:40:06.587: D/alsa_ucm(288): Setting mixer control: RX5 Digital Volume, value: 0
    
10-15 16:40:06.587: D/AudioUsbALSA(288): exitPlaybackThread, mproxypfdPlayback: -1
    
10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
    
10-15 16:40:06.587: D/AudioUsbALSA(288): closeDevice handle 0x0
    
10-15 16:40:17.638: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4677) fps(213.812271)
    
10-15 16:40:17.698: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4644) fps(215.331604)
    
10-15 16:40:20.681: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4633) fps(215.842865)
    
10-15 16:40:21.111: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4611) fps(216.872696)
    
10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
    
10-15 16:40:28.448: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 74
    
10-15 16:40:28.448: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
    
10-15 16:40:30.841: W/SystemClock(698): time going backwards: prev 16555345563411(ioctl) vs now 16555345441341(ioctl), tid=764
    
10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
    
10-15 16:40:57.467: E/OMX-VDEC-1080P(288): set_frame_rate: frm_int(4577) fps(218.483719)
    
10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
    
10-15 16:41:14.403: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.500000 hotplug_avg_load_dw: 89
    
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:41:14.403: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:922.000000
    
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 48
    
10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
    
10-15 16:41:27.466: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
    
10-15 16:41:27.466: E/kickstart(862): EVENT: sahara_mode                         = 2
    
10-15 16:41:27.466: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
    
10-15 16:41:27.466: E/kickstart(862): EVENT: helloRx.mode                        = 2
    
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 64
    
10-15 16:41:27.466: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
    
10-15 16:41:27.466: E/kickstart(862): Total bytes received so far: 116
    
10-15 16:41:27.466: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs1", ""
    
10-15 16:41:27.466: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
    
10-15 16:41:27.466: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs1"
    
10-15 16:41:27.526: E/kickstart(862): Total bytes received so far: 786548
    
10-15 16:41:27.526: E/kickstart(862): EVENT: Received: 786432 bytes
    
10-15 16:41:27.526: E/kickstart(862): EVENT: Writing to disk
    
10-15 16:41:27.526: E/kickstart(862): EVENT: Successfully wrote to disk
    
10-15 16:41:27.526: E/kickstart(862): Received file "m9kefs1"
    
10-15 16:41:27.576: E/kickstart(862): Sync finish Received file "m9kefs1"
    
10-15 16:41:27.576: E/kickstart(862): 786432 bytes transferred in 0.106s (7.10 MBps)
    
10-15 16:41:27.576: E/kickstart(862): EVENT: num_debug_entries not >=0
    
10-15 16:41:27.576: E/kickstart(862): Successfully downloaded files from target 
    
10-15 16:41:27.576: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
    
10-15 16:41:27.576: E/kickstart(862): Total bytes received so far: 786556
    
10-15 16:41:27.576: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
    
10-15 16:41:27.576: E/kickstart(862): Sahara protocol completed
    
10-15 16:41:27.576: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
    
10-15 16:41:27.746: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.100000 hotplug_avg_load_dw: 79
    
10-15 16:41:27.746: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
    
10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
    
10-15 16:42:17.225: I/ActivityManager(698): Start proc com.cyanogenmod.lockclock for service com.cyanogenmod.lockclock/.weather.WeatherUpdateService: pid=18954 uid=10028 gids={50028, 3003, 1028}
    
10-15 16:42:17.865: D/WeatherXmlParser(18954): Weather updated: WeatherInfo for Santa Barbara@ Tue Oct 15 16:42:17 PDT 2013: Fair(34), temperature 29°C, low 11°, high 27°, humidity 14%, wind 11km/h at W
    
10-15 16:42:17.945: I/ActivityManager(698): No longer want com.google.android.apps.uploader (pid 13565): empty #17
    
10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
    
10-15 16:42:35.983: W/ThrottleService(698): unable to find stats for iface rmnet0
    
10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
    
10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 48
    
10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_HELLO
    
10-15 16:43:02.098: E/kickstart(862): EVENT: SENDING --> SAHARA_HELLO_RESPONSE
    
10-15 16:43:02.098: E/kickstart(862): EVENT: sahara_mode                         = 2
    
10-15 16:43:02.098: E/kickstart(862): EVENT: m_comm->sahara_hello_packet_rx.mode = 2
    
10-15 16:43:02.098: E/kickstart(862): EVENT: helloRx.mode                        = 2
    
10-15 16:43:02.098: E/kickstart(862): Total bytes received so far: 64
    
10-15 16:43:02.098: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_MEMORY_DEBUG
    
10-15 16:43:02.108: E/kickstart(862): Total bytes received so far: 116
    
10-15 16:43:02.108: E/kickstart(862): EVENT: 0x46980000, len=000C0000, "m9kefs2", ""
    
10-15 16:43:02.108: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_MEMORY_REGION
    
10-15 16:43:02.108: E/kickstart(862): EVENT: Saving "/dev/block/platform/msm_sdcc.1/by-name/m9kefs2"
    
10-15 16:43:02.158: E/kickstart(862): Total bytes received so far: 786548
    
10-15 16:43:02.158: E/kickstart(862): EVENT: Received: 786432 bytes
    
10-15 16:43:02.158: E/kickstart(862): EVENT: Writing to disk
    
10-15 16:43:02.168: E/kickstart(862): EVENT: Successfully wrote to disk
    
10-15 16:43:02.168: E/kickstart(862): Received file "m9kefs2"
    
10-15 16:43:02.218: E/kickstart(862): Sync finish Received file "m9kefs2"
    
10-15 16:43:02.218: E/kickstart(862): 786432 bytes transferred in 0.113s (6.65 MBps)
    
10-15 16:43:02.218: E/kickstart(862): EVENT: num_debug_entries not >=0
    
10-15 16:43:02.218: E/kickstart(862): Successfully downloaded files from target 
    
10-15 16:43:02.218: E/kickstart(862): EVENT: SENDING --> SAHARA_RESET
    
10-15 16:43:02.218: E/kickstart(862): Total bytes received so far: 786556
    
10-15 16:43:02.218: E/kickstart(862): EVENT: RECEIVED <-- SAHARA_RESET_RESP
    
10-15 16:43:02.218: E/kickstart(862): Sahara protocol completed
    
10-15 16:43:02.218: E/kickstart(862): EVENT: STATE <-- SAHARA_WAIT_HELLO
    
10-15 16:43:17.563: W/SystemClock(698): time going backwards: prev 16722067121029(ioctl) vs now 16722066968441(ioctl), tid=764
    
10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
    
10-15 16:43:36.452: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 100
    
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:43:36.452: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:194.000000
    
10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
    
10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
    
10-15 16:44:10.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.800000 hotplug_avg_load_dw: 133
    
10-15 16:44:10.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
    
10-15 16:44:12.707: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.100000 hotplug_avg_load_dw: 116
    
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:12.707: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
    
10-15 16:44:14.008: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 140
    
10-15 16:44:14.008: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:240.000000
    
10-15 16:44:14.759: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:4.900000 hotplug_avg_load_dw: 132
    
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:14.759: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
    
10-15 16:44:15.360: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.300000 hotplug_avg_load_dw: 139
    
10-15 16:44:15.360: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:243.000000
    
10-15 16:44:15.560: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.900000 hotplug_avg_load_dw: 133
    
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:15.560: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:95.000000
    
10-15 16:44:16.361: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:2.500000 hotplug_avg_load_dw: 132
    
10-15 16:44:16.361: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:242.000000
    
10-15 16:44:16.661: E/MP-Decision(1385): num online cores: 3 reqd : 4 available : 4 rq_depth:3.700000 hotplug_avg_load_dw: 127
    
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:1 core_idx:1 Nw:1.900000 Tw:140 total_time_up:0.000000
    
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:2 core_idx:2 Nw:2.700000 Tw:90 total_time_up:0.000000
    
10-15 16:44:16.661: E/MP-Decision(1385): UP cpu:3 core_idx:3 Nw:3.500000 Tw:90 total_time_up:97.000000
    
10-15 16:44:20.605: E/MP-Decision(1385): num online cores: 4 reqd : 3 available : 4 rq_depth:1.700000 hotplug_avg_load_dw: 122
    
10-15 16:44:20.605: E/MP-Decision(1385): DOWN cpu:3 core_idx:3 Ns:3.100000 Ts:240 total_time_down:244.000000
    
10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19038
    
10-15 16:44:20.615: W/ProcessStats(698): Skipping unknown process pid 19041

    Garbage collector entries in Logcat :

    10-15 16:40:06.087: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 1ms
    
10-15 16:40:06.207: D/dalvikvm(698): GC_EXPLICIT freed 4120K, 39% free 24216K/39664K,   paused 7ms+9ms, total 117ms
    
10-15 16:40:25.746: D/dalvikvm(698): GC_CONCURRENT freed 5829K, 41% free 23778K/39664K, paused 10ms+24ms, total 165ms
    
10-15 16:40:44.684: D/dalvikvm(698): GC_CONCURRENT freed 5302K, 41% free 23774K/39664K, paused 6ms+9ms, total 107ms
    
10-15 16:41:14.383: D/dalvikvm(698): GC_CONCURRENT freed 5371K, 41% free 23768K/39664K, paused 7ms+8ms, total 146ms
    
10-15 16:41:49.598: D/dalvikvm(698): GC_CONCURRENT freed 5339K, 41% free 23775K/39664K, paused 10ms+7ms, total 134ms
    
10-15 16:42:24.021: D/dalvikvm(698): GC_CONCURRENT freed 5277K, 41% free 23770K/39664K, paused 8ms+11ms, total 106ms
    
10-15 16:42:59.476: D/dalvikvm(698): GC_CONCURRENT freed 5345K, 41% free 23770K/39664K, paused 6ms+13ms, total 168ms
    
10-15 16:43:34.610: D/dalvikvm(698): GC_CONCURRENT freed 5359K, 41% free 23772K/39664K, paused 9ms+9ms, total 113ms
    
10-15 16:44:09.965: D/dalvikvm(698): GC_CONCURRENT freed 5369K, 41% free 23779K/39664K, paused 8ms+8ms, total 103ms
    
10-15 16:44:09.965: D/dalvikvm(698): WAIT_FOR_CONCURRENT_GC blocked 30ms
    
10-15 16:44:24.389: D/dalvikvm(698): GC_EXPLICIT freed 3618K, 41% free 23768K/39664K, paused 12ms+11ms, total 123ms

  • openGL ES 2.0 on android , YUV to RGB and Rendering with ffMpeg

    14 octobre 2013, par 101110101100111111101101

    My renderer dies 1 2 frames later when video shows after.

    FATAL ERROR 11 : blabla...(Exactly occurs in glDrawElements (Y part))

    I think problem is 'glPixelStorei' or 'GL_RGB', 'GL_LUMINANCE' but.. I don't get it.

    My rendering way :

    1. Decode data that got from network, (SDK Getting-> NDK Decoding), Enqueueing.

    2. Dequeueing another threads (of course synchronized) get ready to setup OpenGL ES 2.0.(SDK)

    3. When onDrawFrame, onSurfaceCreated, onSurfaceChanged methods are called, it shrink down to NDK. (My Renderer source in NDK will attach below.)

    4. Rendering.

    As you know, Fragment shader is using for conversion.
    My Data is YUV 420p (pix_fmt_YUV420p) (12bit per pixel)

    Here is my entire source.

    I haven't any knowledge about OpenGL ES before, this is first time.

    Please let me know what am I do improving performance.

    and What am I use parameters in 'glTexImage2D', 'glTexSubImage2D', 'glRenderbufferStorage' ????
    GL_LUMINANCE ? GL_RGBA ? GL_RGB ? (GL_LUMINANCE is using now)

    void Renderer::set_draw_frame(JNIEnv* jenv, jbyteArray yData, jbyteArray uData, jbyteArray vData)
    
{
    
    for (int i = 0; i < 3; i++) {
    
        if (yuv_data_[i] != NULL) {
    
            free(yuv_data_[i]);
    
        }
    
    }
    

    
  int YSIZE = -1;
    
  int USIZE = -1;
    
  int VSIZE = -1;
    

    
  if (yData != NULL) {
    
        YSIZE = (int)jenv->GetArrayLength(yData);
    
    LOG_DEBUG("YSIZE : %d", YSIZE);
    
        yuv_data_[0] = (unsigned char*)malloc(sizeof(unsigned char) * YSIZE);
    
    memset(yuv_data_[0], 0, YSIZE);
    
        jenv->GetByteArrayRegion(yData, 0, YSIZE, (jbyte*)yuv_data_[0]);
    
    yuv_data_[0] = reinterpret_cast<unsigned>(yuv_data_[0]);
    
    } else {
    
        YSIZE = (int)jenv->GetArrayLength(yData);
    
        yuv_data_[0] = (unsigned char*)malloc(sizeof(unsigned char) * YSIZE);
    
    memset(yuv_data_[0], 1, YSIZE);
    
  }
    

    
    if (uData != NULL) {
    
        USIZE = (int)jenv->GetArrayLength(uData);
    
    LOG_DEBUG("USIZE : %d", USIZE);
    
        yuv_data_[1] = (unsigned char*)malloc(sizeof(unsigned char) * USIZE);
    
    memset(yuv_data_[1], 0, USIZE);
    
        jenv->GetByteArrayRegion(uData, 0, USIZE, (jbyte*)yuv_data_[1]);
    
    yuv_data_[1] = reinterpret_cast<unsigned>(yuv_data_[1]);
    
    } else {
    
        USIZE = YSIZE/4;
    
        yuv_data_[1] = (unsigned char*)malloc(sizeof(unsigned char) * USIZE);
    
    memset(yuv_data_[1], 1, USIZE);
    
  }
    

    
    if (vData != NULL) {
    
        VSIZE = (int)jenv->GetArrayLength(vData);
    
    LOG_DEBUG("VSIZE : %d", VSIZE);
    
        yuv_data_[2] = (unsigned char*)malloc(sizeof(unsigned char) * VSIZE);
    
    memset(yuv_data_[2], 0, VSIZE);
    
        jenv->GetByteArrayRegion(vData, 0, VSIZE, (jbyte*)yuv_data_[2]);
    
    yuv_data_[2] = reinterpret_cast<unsigned>(yuv_data_[2]);
    
    } else {
    
        VSIZE = YSIZE/4;
    
        yuv_data_[2] = (unsigned char*)malloc(sizeof(unsigned char) * VSIZE);
    
    memset(yuv_data_[2], 1, VSIZE);
    
  }
    

    
    glClearColor(1.0F, 1.0F, 1.0F, 1.0F);
    
    check_gl_error("glClearColor");
    
    glClear(GL_COLOR_BUFFER_BIT);
    
    check_gl_error("glClear");
    
}
    

    
void Renderer::draw_frame()
    
{
    
  // Binding created FBO
    
  glBindFramebuffer(GL_FRAMEBUFFER, frame_buffer_object_);
    
  check_gl_error("glBindFramebuffer");
    
    // Add program to OpenGL environment
    
    glUseProgram(program_object_);
    
    check_gl_error("glUseProgram");
    

    
  for (int i = 0; i &lt; 3; i++) {
    
    LOG_DEBUG("Success");
    
      //Bind texture
    
      glActiveTexture(GL_TEXTURE0 + i);
    
      check_gl_error("glActiveTexture");
    
      glBindTexture(GL_TEXTURE_2D, yuv_texture_id_[i]);
    
      check_gl_error("glBindTexture");
    
      glUniform1i(yuv_texture_object_[i], i);
    
      check_gl_error("glBindTexture");
    
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, stream_yuv_width_[i], stream_yuv_height_[i], GL_RGBA, GL_UNSIGNED_BYTE, yuv_data_[i]);
    
      check_gl_error("glTexSubImage2D");
    
  }
    

    
  LOG_DEBUG("Success");
    
    // Load vertex information
    
    glVertexAttribPointer(position_object_, 2, GL_FLOAT, GL_FALSE, kStride, kVertexInformation);
    
    check_gl_error("glVertexAttribPointer");
    
    // Load texture information
    
    glVertexAttribPointer(texture_position_object_, 2, GL_SHORT, GL_FALSE, kStride, kTextureCoordinateInformation);
    
    check_gl_error("glVertexAttribPointer");
    

    
LOG_DEBUG("9");
    
    glEnableVertexAttribArray(position_object_);
    
    check_gl_error("glEnableVertexAttribArray");
    
    glEnableVertexAttribArray(texture_position_object_);
    
    check_gl_error("glEnableVertexAttribArray");
    

    
  // Back to window buffer
    
  glBindFramebuffer(GL_FRAMEBUFFER, 0);
    
  check_gl_error("glBindFramebuffer");
    
  LOG_DEBUG("Success");
    
    // Draw the Square
    
    glDrawElements(GL_TRIANGLE_STRIP, 6, GL_UNSIGNED_SHORT, kIndicesInformation);
    
    check_gl_error("glDrawElements");
    
}
    

    
void Renderer::setup_render_to_texture()
    
{
    
    glGenFramebuffers(1, &amp;frame_buffer_object_);
    
    check_gl_error("glGenFramebuffers");
    
    glBindFramebuffer(GL_FRAMEBUFFER, frame_buffer_object_);
    
    check_gl_error("glBindFramebuffer");
    
    glGenRenderbuffers(1, &amp;render_buffer_object_);
    
    check_gl_error("glGenRenderbuffers");
    
    glBindRenderbuffer(GL_RENDERBUFFER, render_buffer_object_);
    
    check_gl_error("glBindRenderbuffer");
    
    glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA4, stream_yuv_width_[0], stream_yuv_height_[0]);
    
    check_gl_error("glRenderbufferStorage");
    
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, render_buffer_object_);
    
    check_gl_error("glFramebufferRenderbuffer");
    
  glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, yuv_texture_id_[0], 0);
    
    check_gl_error("glFramebufferTexture2D");  
    
  glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, yuv_texture_id_[1], 0);
    
    check_gl_error("glFramebufferTexture2D");  
    
  glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, yuv_texture_id_[2], 0);
    
    check_gl_error("glFramebufferTexture2D");  
    

    
  glBindFramebuffer(GL_FRAMEBUFFER, 0);
    
    check_gl_error("glBindFramebuffer");
    

    
    GLint status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
    
    if (status != GL_FRAMEBUFFER_COMPLETE) {
    
        print_log("renderer.cpp", "setup_graphics", "FBO setting fault.", LOGERROR);
    
        LOG_ERROR("%d\n", status);
    
        return;
    
    }
    
}
    

    
void Renderer::setup_yuv_texture() 
    
{
    
    // Use tightly packed data
    
    glPixelStorei(GL_UNPACK_ALIGNMENT, 1);
    
    check_gl_error("glPixelStorei");
    

    
  for (int i = 0; i &lt; 3; i++) {
    
    if (yuv_texture_id_[i]) {
    
      glDeleteTextures(1, &amp;yuv_texture_id_[i]);
    
      check_gl_error("glDeleteTextures");
    
    }
    
      glActiveTexture(GL_TEXTURE0+i);
    
      check_gl_error("glActiveTexture"); 
    
      // Generate texture object
    
      glGenTextures(1, &amp;yuv_texture_id_[i]);
    
      check_gl_error("glGenTextures");
    
      glBindTexture(GL_TEXTURE_2D, yuv_texture_id_[i]);
    
      check_gl_error("glBindTexture");
    
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
    
      check_gl_error("glTexParameteri");
    
      glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
    
      check_gl_error("glTexParameteri");
    
      glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    
      check_gl_error("glTexParameterf");
    
      glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
    
      check_gl_error("glTexParameterf"); 
    
    glEnable(GL_TEXTURE_2D);
    
    check_gl_error("glEnable");
    
    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, maximum_yuv_width_[i], maximum_yuv_height_[i], 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL);
    
      check_gl_error("glTexImage2D");
    
  }
    
}
    

    
void Renderer::setup_graphics()
    
{
    
    print_gl_string("Version", GL_VERSION);
    
    print_gl_string("Vendor", GL_VENDOR);
    
    print_gl_string("Renderer", GL_RENDERER);
    
    print_gl_string("Extensions", GL_EXTENSIONS);
    

    
    program_object_ = create_program(kVertexShader, kFragmentShader);
    
    if (!program_object_) {
    
        print_log("renderer.cpp", "setup_graphics", "Could not create program.", LOGERROR);
    
        return;
    
    }
    

    
    position_object_ = glGetAttribLocation(program_object_, "vPosition");
    
    check_gl_error("glGetAttribLocation");
    
    texture_position_object_ = glGetAttribLocation(program_object_, "vTexCoord");
    
    check_gl_error("glGetAttribLocation");
    

    
    yuv_texture_object_[0] = glGetUniformLocation(program_object_, "yTexture");
    
    check_gl_error("glGetUniformLocation");
    
  yuv_texture_object_[1] = glGetUniformLocation(program_object_, "uTexture");
    
    check_gl_error("glGetUniformLocation");
    
    yuv_texture_object_[2] = glGetUniformLocation(program_object_, "vTexture");
    
    check_gl_error("glGetUniformLocation");
    

    
  setup_yuv_texture();
    
    setup_render_to_texture();
    

    
  glViewport(0, 0, stream_yuv_width_[0], stream_yuv_height_[0]);//736, 480);//1920, 1080);//maximum_yuv_width_[0], maximum_yuv_height_[0]);
    
  check_gl_error("glViewport");
    
}
    

    
GLuint Renderer::create_program(const char* vertex_source, const char* fragment_source)
    
{
    
    GLuint vertexShader = load_shader(GL_VERTEX_SHADER, vertex_source);
    
    if (!vertexShader) {
    
        return 0;
    
    }
    

    
    GLuint pixelShader = load_shader(GL_FRAGMENT_SHADER, fragment_source);
    
    if (!pixelShader) {
    
        return 0;
    
    }
    

    
    GLuint program = glCreateProgram();
    
    if (program) {
    
        glAttachShader(program, vertexShader);
    
        check_gl_error("glAttachShader");
    
        glAttachShader(program, pixelShader);
    
        check_gl_error("glAttachShader");
    
        glLinkProgram(program);
    
        /* Get a Status */
    
        GLint linkStatus = GL_FALSE;
    
        glGetProgramiv(program, GL_LINK_STATUS, &amp;linkStatus);
    
        if (linkStatus != GL_TRUE) {
    
            GLint bufLength = 0;
    
            glGetProgramiv(program, GL_INFO_LOG_LENGTH, &amp;bufLength);
    
            if (bufLength) {
    
                char* buf = (char*) malloc(bufLength);
    
                if (buf) {
    
                    glGetProgramInfoLog(program, bufLength, NULL, buf);
    
                    print_log("renderer.cpp", "create_program", "Could not link program.", LOGERROR);
    
                    LOG_ERROR("%s\n", buf);
    
                    free(buf);
    
                }
    
            }
    
            glDeleteProgram(program);
    
            program = 0;
    
        }
    
    }
    
    return program;
    
}
    

    
GLuint Renderer::load_shader(GLenum shaderType, const char* pSource)
    
{
    
    GLuint shader = glCreateShader(shaderType);
    
        if (shader) {
    
            glShaderSource(shader, 1, &amp;pSource, NULL);
    
            glCompileShader(shader);
    
            /* Get a Status */
    
            GLint compiled = 0;
    
            glGetShaderiv(shader, GL_COMPILE_STATUS, &amp;compiled);
    
            if (!compiled) {
    
                GLint infoLen = 0;
    
                glGetShaderiv(shader, GL_INFO_LOG_LENGTH, &amp;infoLen);
    
                if (infoLen) {
    
                    char* buf = (char*) malloc(infoLen);
    
                    if (buf) {
    
                        glGetShaderInfoLog(shader, infoLen, NULL, buf);
    
                        print_log("renderer.cpp", "load_shader", "Could not link program.", LOGERROR);
    
                                  LOG_ERROR("%d :: %s\n", shaderType, buf);
    
                        free(buf);
    
                    }
    
                    glDeleteShader(shader);
    
                    shader = 0;
    
                }
    
            }
    
        }
    
    return shader;
    
}
    

    

    
void Renderer::onDrawFrame(JNIEnv* jenv, jbyteArray yData, jbyteArray uData, jbyteArray vData)
    
{
    
    set_draw_frame(jenv, yData, uData, vData);
    
    draw_frame();
    
    return;
    
}
    

    
void Renderer::setSize(int stream_width, int stream_height) {
    
  stream_yuv_width_[0] = stream_width;
    
  stream_yuv_width_[1] = stream_width/2;
    
  stream_yuv_width_[2] = stream_width/2;
    
  stream_yuv_height_[0] = stream_height;
    
  stream_yuv_height_[1] = stream_height/2;
    
  stream_yuv_height_[2] = stream_height/2;
    
}
    

    
void Renderer::onSurfaceChanged(int width, int height)
    
{
    
  mobile_yuv_width_[0] = width;
    
  mobile_yuv_width_[1] = width/2;
    
  mobile_yuv_width_[2] = width/2; 
    
  mobile_yuv_height_[0] = height;
    
  mobile_yuv_height_[1] = height/2;
    
  mobile_yuv_height_[2] = height/2;
    

    
  maximum_yuv_width_[0] = 1920;
    
  maximum_yuv_width_[1] = 1920/2;
    
  maximum_yuv_width_[2] = 1920/2;
    
  maximum_yuv_height_[0] = 1080;
    
  maximum_yuv_height_[1] = 1080/2;
    
  maximum_yuv_height_[2] = 1080/2;
    

    
  // If stream size not setting, default size D1
    
  //if (stream_yuv_width_[0] == 0) {
    
    stream_yuv_width_[0] = 736;
    
    stream_yuv_width_[1] = 736/2;
    
    stream_yuv_width_[2] = 736/2;
    
    stream_yuv_height_[0] = 480;
    
    stream_yuv_height_[1] = 480/2;
    
    stream_yuv_height_[2] = 480/2;
    
  //}
    

    
    setup_graphics();
    
    return;
    
}
    
</unsigned></unsigned></unsigned>

    Here is my Fragment, Vertex source and coordinates :

    static const char kVertexShader[] =
    
    "attribute vec4 vPosition;      \n"
    
      "attribute vec2 vTexCoord;        \n"
    
      "varying vec2 v_vTexCoord;        \n"
    
    "void main() {                        \n"
    
        "gl_Position = vPosition;       \n"
    
        "v_vTexCoord = vTexCoord;       \n"
    
    "}                                          \n";
    

    
static const char kFragmentShader[] =
    
        "precision mediump float;               \n"
    
        "varying vec2 v_vTexCoord;          \n"
    
        "uniform sampler2D yTexture;        \n"
    
        "uniform sampler2D uTexture;        \n"
    
        "uniform sampler2D vTexture;        \n"
    
        "void main() {                      \n"
    
            "float y=texture2D(yTexture, v_vTexCoord).r;\n"
    
            "float u=texture2D(uTexture, v_vTexCoord).r - 0.5;\n"
    
            "float v=texture2D(vTexture, v_vTexCoord).r - 0.5;\n"
    
            "float r=y + 1.13983 * v;\n"
    
            "float g=y - 0.39465 * u - 0.58060 * v;\n"
    
            "float b=y + 2.03211 * u;\n"
    
            "gl_FragColor = vec4(r, g, b, 1.0);\n"
    
        "}\n";
    

    
static const GLfloat kVertexInformation[] =
    
{
    
         -1.0f, 1.0f,           // TexCoord 0 top left
    
         -1.0f,-1.0f,           // TexCoord 1 bottom left
    
          1.0f,-1.0f,           // TexCoord 2 bottom right
    
          1.0f, 1.0f            // TexCoord 3 top right
    
};
    
static const GLshort kTextureCoordinateInformation[] =
    
{
    
          0, 0,         // TexCoord 0 top left
    
          0, 1,         // TexCoord 1 bottom left
    
          1, 1,         // TexCoord 2 bottom right
    
          1, 0          // TexCoord 3 top right
    
};
    
static const GLuint kStride = 0;//COORDS_PER_VERTEX * 4;
    
static const GLshort kIndicesInformation[] =
    
{
    
    0, 1, 2, 
    
    0, 2, 3
    
};

Navigation