Recherche avancée

Sur d’autres sites (5176)

• Generating test data – Introducing the Piwik Platform

  9 octobre 2014, par Thomas Steur — Development

  This is the next post of our blog series where we introduce the capabilities of the Piwik platform (our previous post was How to create a command). This time you’ll learn how to generate test data.

  Developers are developing on their local Piwik instance which usually does not contain useful data compared to a real Piwik installation in production (only a few test visits and a few tests users and websites). The ‘VisitorGenerator’ plugin lets you generate any number of visits, websites, users, goals and more. The generator makes sure there will be data for each report so you can easily test anything.

  Getting started

  In this series of posts, we assume that you have already installed Piwik. If not, visit the Piwik Developer Zone where you’ll find the Installation guide for developers.

  Installing the VisitorGenerator plugin

  The easiest way to install the plugin is by using the Marketplace in Piwik itself. It is accessible via Settings => Marketplace => Get new functionality. There you’ll find the plugin “VisitorGenerator” which you can install and activate in one click.

  If your Piwik instance is not connected to the internet you can download the plugin from the VisitorGenerator page on the Marketplace. Afterwards you can install the plugin by going to Settings => Marketplace => Uploading a plugin and uploading the previously downloaded ZIP file.

  If you have already installed the plugin make sure it is activated by going to Settings => Plugins.

  Generating websites

  After you have installed the plugin you can add as many websites as you need. This is useful for instance when you want to test something that affects many websites such as the ‘All Websites’ dashboard or the Websites manager. To generate any number of websites use the following command :

  ./console visitorgenerator:generate-website --limit=10

  This will generate 10 websites. If you need more websites simply specify a higher limit. In case you are wondering the names and URLs of the websites are randomly generated by the Faker PHP library.

  Generating goals

  In case you want to test anything related to Goals you should execute the following command :

  ./console visitorgenerator:generate-goals --idsite=1

  This will generate a few goals for the specified site. The generated goals are defined in a way to make sure there will be conversions when generating the visits in the next step.

  Generating visits

  To generate visits there are two possibilities. Either via the Piwik UI by going to Settings => Visitor Generator or by using the command line. The UI is a bit limited in generating visits so we recommend to use the command line. There you can generate visits as follows :

  ./console visitorgenerator:generate-visits --idsite=1

  This will generate many different visits for the current day. Don’t worry if it takes a while, it will insert quite a few visits by default.

  In case you want to generate visits for multiple days in the past as well you can specify the --days option.

  ./console visitorgenerator:generate-visits --idsite=1 --days=5

  Providing your own logs

  Half of the generated visits are randomly generated and half of the visits are based on real logs to make sure there is data for each report. If you want to generate visits based on your own logs for a more realistic testing just place your log files in the plugins/VisitorGenerator/data folder and make sure the file name ends with .log. You can find a few examples in the VisitorGenerator data folder.

  To generate visits based only on real log files then use the --no-fake option.

  ./console visitorgenerator:generate-visits --idsite=1 --no-fake

  All generated visits will come from the logs and no random visits nor random fake data will be used.

  Advanced features

  We are regularly adding new commands, tools and runtime checks to make your life as a developer easier. For instance you can also generate users and annotations. In the future we want to extend the plugin to create visits in the background to make sure there will be constantly new actions in the real time report.

  Are you missing any kind of generator or any other feature to make your life as a developer easier ? Let us know by email, we are listening !

  Would you like to know more about the Piwik platform ? Go to our Piwik Developer Zone where you’ll find guides and references on how to develop plugin and themes.

• dockerfile apt-get install ffmpeg unmet dependencies

  8 juillet 2020, par Martin

  I have a Dockerfile which looks like this :

  


  FROM node:carbon
VOLUME ["/root"]
ADD setup-ffmpeg.sh /root
RUN /root/setup-ffmpeg.sh
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./
RUN npm install
# If you are building your code for production
# RUN npm install --only=production
# Bundle app source
COPY . .
EXPOSE 8080
CMD [ "npm", "start" ]


  


  Which tries to run the file setup-ffmpeg.sh :

  


  #!/usr/bin/env bash
echo 'deb http://www.deb-multimedia.org jessie main non-free' >> /etc/apt/sources.list
echo 'deb-src http://www.deb-multimedia.org jessie main non-free' >> /etc/apt/sources.list
apt-get update
apt-get install -y --force-yes deb-multimedia-keyring
apt-get remove -y --force-yes ffmpeg
apt-get install -y --force-yes build-essential libmp3lame-dev libvorbis-dev libtheora-dev     libspeex-dev yasm pkg-config libfaac-dev libx264-dev libav-tools 
echo "ffmpeg time"
apt-get install ffmpeg 


  


  I am on Ubuntu 20.04 LTS, and try to run this the Dockerfile, from terminal, with the command :

  


  docker build -t alien35/node-web-app .


  


  This causes an error when reaching the ffmpeg part though :

  


  Sending build context to Docker daemon  9.793MB
Step 1/10 : FROM node:carbon
 ---> 8eeadf3757f4
Step 2/10 : VOLUME ["/root"]
 ---> Using cache
 ---> 3658672462de
Step 3/10 : ADD setup-ffmpeg.sh /root
 ---> c9c192aa97c5
Step 4/10 : RUN /root/setup-ffmpeg.sh
 ---> Running in b9240ac8d351
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://security.debian.org/debian-security stretch/updates InRelease [53.0 kB]
Get:3 http://deb.debian.org/debian stretch-updates InRelease [93.6 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [2410 B]
Get:6 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [529 kB]
Get:7 http://deb.debian.org/debian stretch-updates/main amd64 Packages [28.2 kB]
Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7083 kB]
Get:9 http://www.deb-multimedia.org jessie InRelease [24.6 kB]
Ign:9 http://www.deb-multimedia.org jessie InRelease
Get:10 http://www.deb-multimedia.org jessie/main Sources [60.8 kB]
Get:11 http://www.deb-multimedia.org jessie/non-free Sources [2480 B]
Get:12 http://www.deb-multimedia.org jessie/main amd64 Packages [82.8 kB]
Get:13 http://www.deb-multimedia.org jessie/non-free amd64 Packages [1776 B]
Fetched 8080 kB in 2s (3402 kB/s)
Reading package lists...
W: GPG error: http://www.deb-multimedia.org jessie InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 5C808C2B65558117
W: The repository 'http://www.deb-multimedia.org jessie InRelease' is not signed.
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  deb-multimedia-keyring
0 upgraded, 1 newly installed, 0 to remove and 56 not upgraded.
Need to get 10.7 kB of archives.
After this operation, 25.6 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  deb-multimedia-keyring
Get:1 http://www.deb-multimedia.org jessie/main amd64 deb-multimedia-keyring all 2016.8.1 [10.7 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 10.7 kB in 0s (23.3 kB/s)
Selecting previously unselected package deb-multimedia-keyring.
(Reading database ... 29962 files and directories currently installed.)
Preparing to unpack .../deb-multimedia-keyring_2016.8.1_all.deb ...
Unpacking deb-multimedia-keyring (2016.8.1) ...
Setting up deb-multimedia-keyring (2016.8.1) ...
W: --force-yes is deprecated, use one of the options starting with --allow instead.
W: --force-yes is deprecated, use one of the options starting with --allow instead.
Reading package lists...
Building dependency tree...
Reading state information...
Package 'ffmpeg' is not installed, so not removed
0 upgraded, 0 newly installed, 0 to remove and 56 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
pkg-config is already the newest version (0.29-4+b1).
pkg-config set to manually installed.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 libav-tools : Depends: ffmpeg (= 10:2.6.9-dmo1+deb8u2) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
ffmpeg time
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 ffmpeg : Depends: libavcodec56 (>= 10:2.6.9) but it is not going to be installed
          Depends: libavdevice56 (>= 10:2.6.9) but it is not going to be installed
          Depends: libavfilter5 (>= 10:2.6.9) but it is not going to be installed
          Depends: libavformat56 (>= 10:2.6.9) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c /root/setup-ffmpeg.sh' returned a non-zero code: 100


  


  I have tried adding the dependencies to apt-get install in my Dockerfile, and adding -f to the ffmpeg command as well, but I am still getting this error

  


• How can I build a custom version of opencv while enabling CUDA and opengl ?

  1er avril, par Josh

  I have a hard requirement of python3.7 for certain libraries (aeneas & afaligner). I've been using the regular opencv-python and ffmpeg libraries in my program and they've been working find.

  


  Recently I wanted to adjust my program to use h264 instead of mpeg4 and ran down a licensing rabbit hole of how opencv-python uses a build of ffmpeg with opengl codecs off to avoid licensing issues. x264 is apparently opengl, and is disabled in the opencv-python library.

  


  In order to solve this issue, I built a custom build of opencv using another custom build of ffmpeg both with opengl enabled. This allowed me to use the x264 encoder with the VideoWriter in my python program.

  


  Here's the dockerfile of how I've been running it :

  



  FROM python:3.7-slim

# Set optimization flags and number of cores globally
ENV CFLAGS="-O3 -march=native -ffast-math -flto -fno-fat-lto-objects -ffunction-sections -fdata-sections" \
    CXXFLAGS="-O3 -march=native -ffast-math -flto -fno-fat-lto-objects -ffunction-sections -fdata-sections" \
    LDFLAGS="-flto -fno-fat-lto-objects -Wl,--gc-sections" \
    MAKEFLAGS="-j\$(nproc)"

# Combine all system dependencies in a single layer
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    cmake \
    git \
    wget \
    unzip \
    yasm \
    pkg-config \
    libsm6 \
    libxext6 \
    libxrender-dev \
    libglib2.0-0 \
    libavcodec-dev \
    libavformat-dev \
    libswscale-dev \
    libavutil-dev \
    libswresample-dev \
    nasm \
    mercurial \
    libnuma-dev \
    espeak \
    libespeak-dev \
    libtiff5-dev \
    libjpeg62-turbo-dev \
    libopenjp2-7-dev \
    zlib1g-dev \
    libfreetype6-dev \
    liblcms2-dev \
    libwebp-dev \
    tcl8.6-dev \
    tk8.6-dev \
    python3-tk \
    libharfbuzz-dev \
    libfribidi-dev \
    libxcb1-dev \
    python3-dev \
    python3-setuptools \
    libsndfile1 \
    libavdevice-dev \
    libavfilter-dev \
    libpostproc-dev \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Build x264 with optimizations
RUN cd /tmp && \
    wget https://code.videolan.org/videolan/x264/-/archive/master/x264-master.tar.bz2 && \
    tar xjf x264-master.tar.bz2 && \
    cd x264-master && \
    ./configure \
        --enable-shared \
        --enable-pic \
        --enable-asm \
        --enable-lto \
        --enable-strip \
        --enable-optimizations \
        --bit-depth=8 \
        --disable-avs \
        --disable-swscale \
        --disable-lavf \
        --disable-ffms \
        --disable-gpac \
        --disable-lsmash \
        --extra-cflags="-O3 -march=native -ffast-math -fomit-frame-pointer -flto -fno-fat-lto-objects" \
        --extra-ldflags="-O3 -flto -fno-fat-lto-objects" && \
    make && \
    make install && \
    cd /tmp && \
    # Build FFmpeg with optimizations
    wget https://ffmpeg.org/releases/ffmpeg-7.1.tar.bz2 && \
    tar xjf ffmpeg-7.1.tar.bz2 && \
    cd ffmpeg-7.1 && \
    ./configure \
        --enable-gpl \
        --enable-libx264 \
        --enable-shared \
        --enable-nonfree \
        --enable-pic \
        --enable-asm \
        --enable-optimizations \
        --enable-lto \
        --enable-pthreads \
        --disable-debug \
        --disable-static \
        --disable-doc \
        --disable-ffplay \
        --disable-ffprobe \
        --disable-filters \
        --disable-programs \
        --disable-postproc \
        --extra-cflags="-O3 -march=native -ffast-math -fomit-frame-pointer -flto -fno-fat-lto-objects -ffunction-sections -fdata-sections" \
        --extra-ldflags="-O3 -flto -fno-fat-lto-objects -Wl,--gc-sections" \
        --prefix=/usr/local && \
    make && \
    make install && \
    ldconfig && \
    rm -rf /tmp/*

# Install Python dependencies first
RUN pip install --no-cache-dir --upgrade pip setuptools wheel && \
    pip install --no-cache-dir numpy py-spy

# Build OpenCV with optimized configuration
RUN cd /tmp && \
    # Download specific OpenCV version archives
    wget -O opencv.zip https://github.com/opencv/opencv/archive/4.8.0.zip && \
    wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.8.0.zip && \
    unzip opencv.zip && \
    unzip opencv_contrib.zip && \
    mv opencv-4.8.0 opencv && \
    mv opencv_contrib-4.8.0 opencv_contrib && \
    rm opencv.zip opencv_contrib.zip && \
    cd opencv && \
    mkdir build && cd build && \
    cmake \
        -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_C_FLAGS="-O3 -march=native -ffast-math -flto -fno-fat-lto-objects -ffunction-sections -fdata-sections" \
        -D CMAKE_CXX_FLAGS="-O3 -march=native -ffast-math -flto -fno-fat-lto-objects -ffunction-sections -fdata-sections -Wno-deprecated" \
        -D CMAKE_EXE_LINKER_FLAGS="-flto -fno-fat-lto-objects -Wl,--gc-sections" \
        -D CMAKE_SHARED_LINKER_FLAGS="-flto -fno-fat-lto-objects -Wl,--gc-sections" \
        -D CMAKE_INSTALL_PREFIX=/usr/local \
        -D ENABLE_FAST_MATH=ON \
        -D CPU_BASELINE_DETECT=ON \
        -D CPU_BASELINE=SSE3 \
        -D CPU_DISPATCH=SSE4_1,SSE4_2,AVX,AVX2,AVX512_SKX,FP16 \
        -D WITH_OPENMP=ON \
        -D OPENCV_ENABLE_NONFREE=ON \
        -D WITH_FFMPEG=ON \
        -D FFMPEG_ROOT=/usr/local \
        -D OPENCV_EXTRA_MODULES_PATH=/tmp/opencv_contrib/modules \
        -D PYTHON_EXECUTABLE=/usr/local/bin/python3.7 \
        -D PYTHON3_EXECUTABLE=/usr/local/bin/python3.7 \
        -D PYTHON3_INCLUDE_DIR=/usr/local/include/python3.7m \
        -D PYTHON3_LIBRARY=/usr/local/lib/libpython3.7m.so \
        -D PYTHON3_PACKAGES_PATH=/usr/local/lib/python3.7/site-packages \
        -D PYTHON3_NUMPY_INCLUDE_DIRS=/usr/local/lib/python3.7/site-packages/numpy/core/include \
        -D BUILD_opencv_python3=ON \
        -D INSTALL_PYTHON_EXAMPLES=OFF \
        -D BUILD_TESTS=OFF \
        -D BUILD_PERF_TESTS=OFF \
        -D BUILD_EXAMPLES=OFF \
        -D BUILD_DOCS=OFF \
        -D BUILD_opencv_apps=OFF \
        -D WITH_OPENCL=OFF \
        -D WITH_CUDA=OFF \
        -D WITH_IPP=OFF \
        -D WITH_TBB=OFF \
        -D WITH_V4L=OFF \
        -D WITH_QT=OFF \
        -D WITH_GTK=OFF \
        -D BUILD_LIST=core,imgproc,imgcodecs,videoio,python3 \
        .. && \
    make && \
    make install && \
    ldconfig && \
    rm -rf /tmp/*

# Set working directory and copy application code
WORKDIR /app

COPY requirements.txt .

RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg

RUN pip install --no-cache-dir aeneas afaligner && \
    pip install --no-cache-dir -r requirements.txt

COPY . .

# Make entrypoint executable
RUN chmod +x entrypoint.sh
ENTRYPOINT ["./entrypoint.sh"]


  


  My trouble now, is I've been considering running parts of my program on my GPU, it's creating graphics for a video after all. I have no idea how to edit my Dockerfile to make the opencv build run with CUDA enabled, every combination I try leads to issues.

  


  How can I tell which version of CUDA, opencv and ffmpeg are compatible with python 3.7 ?