Recherche avancée
Autres articles (100)
-
MediaSPIP 0.1 Beta version
25 avril 2011, parMediaSPIP 0.1 beta is the first version of MediaSPIP proclaimed as "usable".
The zip file provided here only contains the sources of MediaSPIP in its standalone version.
To get a working installation, you must manually install all-software dependencies on the server.
If you want to use this archive for an installation in "farm mode", you will also need to proceed to other manual (...) -
Multilang : améliorer l’interface pour les blocs multilingues
18 février 2011, parMultilang est un plugin supplémentaire qui n’est pas activé par défaut lors de l’initialisation de MediaSPIP.
Après son activation, une préconfiguration est mise en place automatiquement par MediaSPIP init permettant à la nouvelle fonctionnalité d’être automatiquement opérationnelle. Il n’est donc pas obligatoire de passer par une étape de configuration pour cela. -
Personnaliser en ajoutant son logo, sa bannière ou son image de fond
5 septembre 2013, parCertains thèmes prennent en compte trois éléments de personnalisation : l’ajout d’un logo ; l’ajout d’une bannière l’ajout d’une image de fond ;
Sur d’autres sites (12592)
-
ffmpeg : libavformat/libswresample to transcode and resample at same time
21 février 2024, par whatdoidoI want to transcode and down/re-sample the audio for output using ffmpeg's libav*/libswresample - I am using ffmpeg's (4.x) transcode_aac.c and resample_audio.c as reference - but the code produces audio with glitches that is clearly not what ffmpeg itself would produce (ie ffmpeg -i foo.wav -ar 22050 foo.m4a)


Based on the ffmpeg examples, to resample audio it appears that I need to set the output AVAudioContext and SwrContext sample_rate to what I desire and ensure the swr_convert() is provided with the correct number of output samples based av_rescale_rnd( swr_delay(), ...) once I have an decoded input audio. I've taken care to ensure all the relevant calculations of samples for output are taken into account in the merged code (below) :


- 

- open_output_file() - AVCodecContext.sample_rate (avctx variable) set to our target (down sampled) sample_rate
- read_decode_convert_and_store() is where the work happens : input audio is decoded to an AVFrame and this input frame is converted before being encoded.

- 

- init_converted_samples() and av_samples_alloc() uses the input frame's nb_samples
- ADDED : calc the number of output samples via av_rescale_rnd() and swr_delay()
- UPDATED : convert_samples() and swr_convert() uses the input frame's samples and our calculated output samples as parameters














However the resulting audio file is produced with audio glitches. Does the community know of any references for how transcode AND resample should be done or what is missing in this example ?


/* compile and run:
 gcc -I/usr/include/ffmpeg transcode-swr-aac.c -lavformat -lavutil -lavcodec -lswresample -lm
 ./a.out foo.wav foo.m4a
 */

/*
 * Copyright (c) 2013-2018 Andreas Unterweger
 * 
 * This file is part of FFmpeg. 
 ... ...
 * 
 * @example transcode_aac.c 
 * Convert an input audio file to AAC in an MP4 container using FFmpeg. 
 * Formats other than MP4 are supported based on the output file extension. 
 * @author Andreas Unterweger (xxxx@xxxxx.com)
 */ 
 #include 
 

 #include "libavformat/avformat.h"
 #include "libavformat/avio.h"
 
 #include "libavcodec/avcodec.h"
 
 #include "libavutil/audio_fifo.h"
 #include "libavutil/avassert.h"
 #include "libavutil/avstring.h"
 #include "libavutil/channel_layout.h"
 #include "libavutil/frame.h"
 #include "libavutil/opt.h"
 
 #include "libswresample/swresample.h"
 
 #define OUTPUT_BIT_RATE 128000
 #define OUTPUT_CHANNELS 2
 
 static int open_input_file(const char *filename,
 AVFormatContext **input_format_context,
 AVCodecContext **input_codec_context)
 {
 AVCodecContext *avctx;
 const AVCodec *input_codec;
 const AVStream *stream;
 int error;
 
 if ((error = avformat_open_input(input_format_context, filename, NULL,
 NULL)) < 0) {
 fprintf(stderr, "Could not open input file '%s' (error '%s')\n",
 filename, av_err2str(error));
 *input_format_context = NULL;
 return error;
 }
 

 if ((error = avformat_find_stream_info(*input_format_context, NULL)) < 0) {
 fprintf(stderr, "Could not open find stream info (error '%s')\n",
 av_err2str(error));
 avformat_close_input(input_format_context);
 return error;
 }
 
 if ((*input_format_context)->nb_streams != 1) {
 fprintf(stderr, "Expected one audio input stream, but found %d\n",
 (*input_format_context)->nb_streams);
 avformat_close_input(input_format_context);
 return AVERROR_EXIT;
 }
 
 stream = (*input_format_context)->streams[0];
 
 if (!(input_codec = avcodec_find_decoder(stream->codecpar->codec_id))) {
 fprintf(stderr, "Could not find input codec\n");
 avformat_close_input(input_format_context);
 return AVERROR_EXIT;
 }
 
 avctx = avcodec_alloc_context3(input_codec);
 if (!avctx) {
 fprintf(stderr, "Could not allocate a decoding context\n");
 avformat_close_input(input_format_context);
 return AVERROR(ENOMEM);
 }
 
 /* Initialize the stream parameters with demuxer information. */
 error = avcodec_parameters_to_context(avctx, stream->codecpar);
 if (error < 0) {
 avformat_close_input(input_format_context);
 avcodec_free_context(&avctx);
 return error;
 }
 
 /* Open the decoder for the audio stream to use it later. */
 if ((error = avcodec_open2(avctx, input_codec, NULL)) < 0) {
 fprintf(stderr, "Could not open input codec (error '%s')\n",
 av_err2str(error));
 avcodec_free_context(&avctx);
 avformat_close_input(input_format_context);
 return error;
 }
 
 /* Set the packet timebase for the decoder. */
 avctx->pkt_timebase = stream->time_base;
 
 /* Save the decoder context for easier access later. */
 *input_codec_context = avctx;
 
 return 0;
 }
 
 static int open_output_file(const char *filename,
 AVCodecContext *input_codec_context,
 AVFormatContext **output_format_context,
 AVCodecContext **output_codec_context)
 {
 AVCodecContext *avctx = NULL;
 AVIOContext *output_io_context = NULL;
 AVStream *stream = NULL;
 const AVCodec *output_codec = NULL;
 int error;
 

 if ((error = avio_open(&output_io_context, filename,
 AVIO_FLAG_WRITE)) < 0) {
 fprintf(stderr, "Could not open output file '%s' (error '%s')\n",
 filename, av_err2str(error));
 return error;
 }
 

 if (!(*output_format_context = avformat_alloc_context())) {
 fprintf(stderr, "Could not allocate output format context\n");
 return AVERROR(ENOMEM);
 }
 

 (*output_format_context)->pb = output_io_context;
 

 if (!((*output_format_context)->oformat = av_guess_format(NULL, filename,
 NULL))) {
 fprintf(stderr, "Could not find output file format\n");
 goto cleanup;
 }
 
 if (!((*output_format_context)->url = av_strdup(filename))) {
 fprintf(stderr, "Could not allocate url.\n");
 error = AVERROR(ENOMEM);
 goto cleanup;
 }
 

 if (!(output_codec = avcodec_find_encoder(AV_CODEC_ID_AAC))) {
 fprintf(stderr, "Could not find an AAC encoder.\n");
 goto cleanup;
 }
 
 /* Create a new audio stream in the output file container. */
 if (!(stream = avformat_new_stream(*output_format_context, NULL))) {
 fprintf(stderr, "Could not create new stream\n");
 error = AVERROR(ENOMEM);
 goto cleanup;
 }
 
 avctx = avcodec_alloc_context3(output_codec);
 if (!avctx) {
 fprintf(stderr, "Could not allocate an encoding context\n");
 error = AVERROR(ENOMEM);
 goto cleanup;
 }
 
 /* Set the basic encoder parameters.
 * SET OUR DESIRED output sample_rate here
 */
 avctx->channels = OUTPUT_CHANNELS;
 avctx->channel_layout = av_get_default_channel_layout(OUTPUT_CHANNELS);
 // avctx->sample_rate = input_codec_context->sample_rate;
 avctx->sample_rate = 22050;
 avctx->sample_fmt = output_codec->sample_fmts[0];
 avctx->bit_rate = OUTPUT_BIT_RATE;
 
 avctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;
 
 /* Set the sample rate for the container. */
 stream->time_base.den = avctx->sample_rate;
 stream->time_base.num = 1;
 
 if ((*output_format_context)->oformat->flags & AVFMT_GLOBALHEADER)
 avctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
 
 if ((error = avcodec_open2(avctx, output_codec, NULL)) < 0) {
 fprintf(stderr, "Could not open output codec (error '%s')\n",
 av_err2str(error));
 goto cleanup;
 }
 
 error = avcodec_parameters_from_context(stream->codecpar, avctx);
 if (error < 0) {
 fprintf(stderr, "Could not initialize stream parameters\n");
 goto cleanup;
 }
 
 /* Save the encoder context for easier access later. */
 *output_codec_context = avctx;
 
 return 0;
 
 cleanup:
 avcodec_free_context(&avctx);
 avio_closep(&(*output_format_context)->pb);
 avformat_free_context(*output_format_context);
 *output_format_context = NULL;
 return error < 0 ? error : AVERROR_EXIT;
 }
 
 /**
 * Initialize one data packet for reading or writing.
 */
 static int init_packet(AVPacket **packet)
 {
 if (!(*packet = av_packet_alloc())) {
 fprintf(stderr, "Could not allocate packet\n");
 return AVERROR(ENOMEM);
 }
 return 0;
 }
 
 static int init_input_frame(AVFrame **frame)
 {
 if (!(*frame = av_frame_alloc())) {
 fprintf(stderr, "Could not allocate input frame\n");
 return AVERROR(ENOMEM);
 }
 return 0;
 }
 
 static int init_resampler(AVCodecContext *input_codec_context,
 AVCodecContext *output_codec_context,
 SwrContext **resample_context)
 {
 int error;

 /**
 * create the resample, including ref to the desired output sample rate
 */
 *resample_context = swr_alloc_set_opts(NULL,
 av_get_default_channel_layout(output_codec_context->channels),
 output_codec_context->sample_fmt,
 output_codec_context->sample_rate,
 av_get_default_channel_layout(input_codec_context->channels),
 input_codec_context->sample_fmt,
 input_codec_context->sample_rate,
 0, NULL);
 if (!*resample_context < 0) {
 fprintf(stderr, "Could not allocate resample context\n");
 return AVERROR(ENOMEM);
 }
 
 if ((error = swr_init(*resample_context)) < 0) {
 fprintf(stderr, "Could not open resample context\n");
 swr_free(resample_context);
 return error;
 }
 return 0;
 }
 
 static int init_fifo(AVAudioFifo **fifo, AVCodecContext *output_codec_context)
 {
 if (!(*fifo = av_audio_fifo_alloc(output_codec_context->sample_fmt,
 output_codec_context->channels, 1))) {
 fprintf(stderr, "Could not allocate FIFO\n");
 return AVERROR(ENOMEM);
 }
 return 0;
 }
 
 static int write_output_file_header(AVFormatContext *output_format_context)
 {
 int error;
 if ((error = avformat_write_header(output_format_context, NULL)) < 0) {
 fprintf(stderr, "Could not write output file header (error '%s')\n",
 av_err2str(error));
 return error;
 }
 return 0;
 }
 
 static int decode_audio_frame(AVFrame *frame,
 AVFormatContext *input_format_context,
 AVCodecContext *input_codec_context,
 int *data_present, int *finished)
 {
 AVPacket *input_packet;
 int error;
 
 error = init_packet(&input_packet);
 if (error < 0)
 return error;
 
 *data_present = 0;
 *finished = 0;

 if ((error = av_read_frame(input_format_context, input_packet)) < 0) {
 if (error == AVERROR_EOF)
 *finished = 1;
 else {
 fprintf(stderr, "Could not read frame (error '%s')\n",
 av_err2str(error));
 goto cleanup;
 }
 }
 
 if ((error = avcodec_send_packet(input_codec_context, input_packet)) < 0) {
 fprintf(stderr, "Could not send packet for decoding (error '%s')\n",
 av_err2str(error));
 goto cleanup;
 }
 
 error = avcodec_receive_frame(input_codec_context, frame);
 if (error == AVERROR(EAGAIN)) {
 error = 0;
 goto cleanup;
 } else if (error == AVERROR_EOF) {
 *finished = 1;
 error = 0;
 goto cleanup;
 } else if (error < 0) {
 fprintf(stderr, "Could not decode frame (error '%s')\n",
 av_err2str(error));
 goto cleanup;
 } else {
 *data_present = 1;
 goto cleanup;
 }
 
 cleanup:
 av_packet_free(&input_packet);
 return error;
 }
 
 static int init_converted_samples(uint8_t ***converted_input_samples,
 AVCodecContext *output_codec_context,
 int frame_size)
 {
 int error;
 
 if (!(*converted_input_samples = calloc(output_codec_context->channels,
 sizeof(**converted_input_samples)))) {
 fprintf(stderr, "Could not allocate converted input sample pointers\n");
 return AVERROR(ENOMEM);
 }
 

 if ((error = av_samples_alloc(*converted_input_samples, NULL,
 output_codec_context->channels,
 frame_size,
 output_codec_context->sample_fmt, 0)) < 0) {
 fprintf(stderr,
 "Could not allocate converted input samples (error '%s')\n",
 av_err2str(error));
 av_freep(&(*converted_input_samples)[0]);
 free(*converted_input_samples);
 return error;
 }
 return 0;
 }
 
 static int convert_samples(const uint8_t **input_data, const int input_nb_samples,
 uint8_t **converted_data, const int output_nb_samples,
 SwrContext *resample_context)
 {
 int error;
 
 if ((error = swr_convert(resample_context,
 converted_data, output_nb_samples,
 input_data , input_nb_samples)) < 0) {
 fprintf(stderr, "Could not convert input samples (error '%s')\n",
 av_err2str(error));
 return error;
 }
 
 return 0;
 }
 
 static int add_samples_to_fifo(AVAudioFifo *fifo,
 uint8_t **converted_input_samples,
 const int frame_size)
 {
 int error;
 
 if ((error = av_audio_fifo_realloc(fifo, av_audio_fifo_size(fifo) + frame_size)) < 0) {
 fprintf(stderr, "Could not reallocate FIFO\n");
 return error;
 }
 
 if (av_audio_fifo_write(fifo, (void **)converted_input_samples,
 frame_size) < frame_size) {
 fprintf(stderr, "Could not write data to FIFO\n");
 return AVERROR_EXIT;
 }
 return 0;
 }
 
 static int read_decode_convert_and_store(AVAudioFifo *fifo,
 AVFormatContext *input_format_context,
 AVCodecContext *input_codec_context,
 AVCodecContext *output_codec_context,
 SwrContext *resampler_context,
 int *finished)
 {
 AVFrame *input_frame = NULL;
 uint8_t **converted_input_samples = NULL;
 int data_present;
 int ret = AVERROR_EXIT;
 

 if (init_input_frame(&input_frame))
 goto cleanup;

 if (decode_audio_frame(input_frame, input_format_context,
 input_codec_context, &data_present, finished))
 goto cleanup;

 if (*finished) {
 ret = 0;
 goto cleanup;
 }

 if (data_present) {
 /* Initialize the temporary storage for the converted input samples. */
 if (init_converted_samples(&converted_input_samples, output_codec_context,
 input_frame->nb_samples))
 goto cleanup;
 
 /* figure out how many samples are required for target sample_rate incl
 * any items left in the swr buffer
 */ 
 int output_nb_samples = av_rescale_rnd(
 swr_get_delay(resampler_context, input_codec_context->sample_rate) + input_frame->nb_samples,
 output_codec_context->sample_rate, 
 input_codec_context->sample_rate,
 AV_ROUND_UP);
 
 /* ignore, just to ensure we've got enough buffer alloc'd for conversion buffer */
 av_assert1(input_frame->nb_samples > output_nb_samples);
 
 /* Convert the input samples to the desired output sample format, via swr_convert().
 */
 if (convert_samples((const uint8_t**)input_frame->extended_data, input_frame->nb_samples,
 converted_input_samples, output_nb_samples,
 resampler_context))
 goto cleanup;
 
 /* Add the converted input samples to the FIFO buffer for later processing. */
 if (add_samples_to_fifo(fifo, converted_input_samples,
 output_nb_samples))
 goto cleanup;
 ret = 0;
 }
 ret = 0;
 
 cleanup:
 if (converted_input_samples) {
 av_freep(&converted_input_samples[0]);
 free(converted_input_samples);
 }
 av_frame_free(&input_frame);
 
 return ret;
 }
 
 static int init_output_frame(AVFrame **frame,
 AVCodecContext *output_codec_context,
 int frame_size)
 {
 int error;
 
 if (!(*frame = av_frame_alloc())) {
 fprintf(stderr, "Could not allocate output frame\n");
 return AVERROR_EXIT;
 }
 
 /* Set the frame's parameters, especially its size and format.
 * av_frame_get_buffer needs this to allocate memory for the
 * audio samples of the frame.
 * Default channel layouts based on the number of channels
 * are assumed for simplicity. */
 (*frame)->nb_samples = frame_size;
 (*frame)->channel_layout = output_codec_context->channel_layout;
 (*frame)->format = output_codec_context->sample_fmt;
 (*frame)->sample_rate = output_codec_context->sample_rate;
 
 /* Allocate the samples of the created frame. This call will make
 * sure that the audio frame can hold as many samples as specified. */
 if ((error = av_frame_get_buffer(*frame, 0)) < 0) {
 fprintf(stderr, "Could not allocate output frame samples (error '%s')\n",
 av_err2str(error));
 av_frame_free(frame);
 return error;
 }
 
 return 0;
 }
 
 /* Global timestamp for the audio frames. */
 static int64_t pts = 0;
 
 /**
 * Encode one frame worth of audio to the output file.
 */
 static int encode_audio_frame(AVFrame *frame,
 AVFormatContext *output_format_context,
 AVCodecContext *output_codec_context,
 int *data_present)
 {
 AVPacket *output_packet;
 int error;
 
 error = init_packet(&output_packet);
 if (error < 0)
 return error;
 
 /* Set a timestamp based on the sample rate for the container. */
 if (frame) {
 frame->pts = pts;
 pts += frame->nb_samples;
 }
 
 *data_present = 0;
 error = avcodec_send_frame(output_codec_context, frame);
 if (error < 0 && error != AVERROR_EOF) {
 fprintf(stderr, "Could not send packet for encoding (error '%s')\n",
 av_err2str(error));
 goto cleanup;
 }
 

 error = avcodec_receive_packet(output_codec_context, output_packet);
 if (error == AVERROR(EAGAIN)) {
 error = 0;
 goto cleanup;
 } else if (error == AVERROR_EOF) {
 error = 0;
 goto cleanup;
 } else if (error < 0) {
 fprintf(stderr, "Could not encode frame (error '%s')\n",
 av_err2str(error));
 goto cleanup;
 } else {
 *data_present = 1;
 }
 
 /* Write one audio frame from the temporary packet to the output file. */
 if (*data_present &&
 (error = av_write_frame(output_format_context, output_packet)) < 0) {
 fprintf(stderr, "Could not write frame (error '%s')\n",
 av_err2str(error));
 goto cleanup;
 }
 
 cleanup:
 av_packet_free(&output_packet);
 return error;
 }
 
 /**
 * Load one audio frame from the FIFO buffer, encode and write it to the
 * output file.
 */
 static int load_encode_and_write(AVAudioFifo *fifo,
 AVFormatContext *output_format_context,
 AVCodecContext *output_codec_context)
 {
 AVFrame *output_frame;
 /* Use the maximum number of possible samples per frame.
 * If there is less than the maximum possible frame size in the FIFO
 * buffer use this number. Otherwise, use the maximum possible frame size. */
 const int frame_size = FFMIN(av_audio_fifo_size(fifo),
 output_codec_context->frame_size);
 int data_written;
 
 if (init_output_frame(&output_frame, output_codec_context, frame_size))
 return AVERROR_EXIT;
 
 /* Read as many samples from the FIFO buffer as required to fill the frame.
 * The samples are stored in the frame temporarily. */
 if (av_audio_fifo_read(fifo, (void **)output_frame->data, frame_size) < frame_size) {
 fprintf(stderr, "Could not read data from FIFO\n");
 av_frame_free(&output_frame);
 return AVERROR_EXIT;
 }
 
 /* Encode one frame worth of audio samples. */
 if (encode_audio_frame(output_frame, output_format_context,
 output_codec_context, &data_written)) {
 av_frame_free(&output_frame);
 return AVERROR_EXIT;
 }
 av_frame_free(&output_frame);
 return 0;
 }
 
 /**
 * Write the trailer of the output file container.
 */
 static int write_output_file_trailer(AVFormatContext *output_format_context)
 {
 int error;
 if ((error = av_write_trailer(output_format_context)) < 0) {
 fprintf(stderr, "Could not write output file trailer (error '%s')\n",
 av_err2str(error));
 return error;
 }
 return 0;
 }
 
 int main(int argc, char **argv)
 {
 AVFormatContext *input_format_context = NULL, *output_format_context = NULL;
 AVCodecContext *input_codec_context = NULL, *output_codec_context = NULL;
 SwrContext *resample_context = NULL;
 AVAudioFifo *fifo = NULL;
 int ret = AVERROR_EXIT;
 
 if (argc != 3) {
 fprintf(stderr, "Usage: %s <input file="file" /> <output file="file">\n", argv[0]);
 exit(1);
 }
 

 if (open_input_file(argv[1], &input_format_context,
 &input_codec_context))
 goto cleanup;

 if (open_output_file(argv[2], input_codec_context,
 &output_format_context, &output_codec_context))
 goto cleanup;

 if (init_resampler(input_codec_context, output_codec_context,
 &resample_context))
 goto cleanup;

 if (init_fifo(&fifo, output_codec_context))
 goto cleanup;

 if (write_output_file_header(output_format_context))
 goto cleanup;
 
 while (1) {
 /* Use the encoder's desired frame size for processing. */
 const int output_frame_size = output_codec_context->frame_size;
 int finished = 0;
 
 while (av_audio_fifo_size(fifo) < output_frame_size) {
 /* Decode one frame worth of audio samples, convert it to the
 * output sample format and put it into the FIFO buffer. */
 if (read_decode_convert_and_store(fifo, input_format_context,
 input_codec_context,
 output_codec_context,
 resample_context, &finished))
 goto cleanup;
 
 if (finished)
 break;
 }
 
 while (av_audio_fifo_size(fifo) >= output_frame_size ||
 (finished && av_audio_fifo_size(fifo) > 0))
 if (load_encode_and_write(fifo, output_format_context,
 output_codec_context))
 goto cleanup;
 
 if (finished) {
 int data_written;
 do {
 if (encode_audio_frame(NULL, output_format_context,
 output_codec_context, &data_written))
 goto cleanup;
 } while (data_written);
 break;
 }
 }
 
 if (write_output_file_trailer(output_format_context))
 goto cleanup;
 ret = 0;
 
 cleanup:
 if (fifo)
 av_audio_fifo_free(fifo);
 swr_free(&resample_context);
 if (output_codec_context)
 avcodec_free_context(&output_codec_context);
 if (output_format_context) {
 avio_closep(&output_format_context->pb);
 avformat_free_context(output_format_context);
 }
 if (input_codec_context)
 avcodec_free_context(&input_codec_context);
 if (input_format_context)
 avformat_close_input(&input_format_context);
 
 return ret;
 }
</output>

-
Lawful basis for processing personal data under GDPR with Matomo
30 avril 2018, par InnoCraftDisclaimer : this blog post has been written by digital analysts, not lawyers. The purpose of this article is to explain what is a lawful basis and which one you can use with Matomo in order to be GDPR compliant. This work comes from our interpretation of the following web page from the UK privacy commission : ICO. It cannot be considered as professional legal advice. So as GDPR, this information is subject to change. GDPR may be also known as DSGVO in German, BDAR in Lithuanian, RGPD in Spanish, French, Italian, Portuguese. This blog post contains public sector information licensed under the Open Government Licence v3.0.
The golden rule under GDPR is that you need to have a lawful basis in order to process personal data. Note that it is possible to not process personal data with Matomo. When you do not collect any personal data, then you do not need to determine a lawful basis and this article wouldn’t apply to you.
“If no lawful basis applies to your processing, your processing will be unlawful and in breach of the first principle.“
Source : ICO, based on article 6 of GDPR.
As you may process personal data in Matomo, you have to :
Even if you think you don’t process personal data, we recommend reading this post about personal data in Matomo (personal data may be hidden in many ways).
Note that if you are processing special category data (ethnic origin, politics, religion, trade union membership…) or criminal offence data ; extra responsibilities are applied, and we will not detail them in this blog post.
1 – Define a lawful basis
There are 6 different lawful bases all defined within article 6 of the GDPR official text :
- Consent : the data subject has given consent to the processing of his or her personal data for one or more specific purposes.
- Contract : processing is necessary for the performance of a contract to which the data subject is party or in order to take steps at the request of the data subject prior to entering into a contract.
- Legal obligation : processing is necessary for compliance with a legal obligation to which the controller is subject.
- Vital interests : processing is necessary in order to protect the vital interests of the data subject or of another natural person.
- Public task : processing is necessary for the performance of a task carried out in the public interest or in the exercise of an official authority vested in the controller.
- Legitimate interests : processing is necessary for the purposes of the legitimate interests pursued by the controller or by a third party ; except where such interests are overridden by the interests or fundamental rights and freedoms of the data subject which require protection of personal data, in particular where the data subject is a child.
As you can see, most of them are not applicable to Matomo. As ICO is mentioning it within their documentation :
“In many cases you are likely to have a choice between using legitimate interests or consent.”
“Consent” or “Legitimate interests” : which lawful basis is the best when using Matomo ?
Well, there is no right or wrong answer here.
In order to make this choice, ICO listed on their website different questions you should keep in mind :
- Who does the processing benefit ?
- Would individuals expect this processing to take place ?
- What is your relationship with the individual ?
- Are you in a position of power over them ?
- What is the impact of the processing on the individual ?
- Are they vulnerable ?
- Are some of the individuals concerns likely to object ?
- Are you able to stop the processing at any time on request ?
From our perspective, “Legitimate interests” should be used in most of the cases as :
- The processing benefits to the owner of the website and not to a third party company.
- A user expects to have their data kept by the website itself.
- Matomo provides many features in order to show how personal data is processed and how users can exercise their rights.
- As the data is not used for profiling, the impact of processing personal data is very low.
But once more, it really depends ; if you are processing personal data which may represent a risk to the final user, then getting consent is for us the right lawful basis.
If you are not sure, at the time of writing ICO is providing a tool in order to help you make this decision :
Note that once you choose a lawful basis, it is highly recommended not to switch to another unless you have a good reason.
What are the rights that a data subject can exercise ?
According to the lawful basis you choose for processing personal data with Matomo, your users will be able to exercise different rights :
Right to be informed Right of access Right to erasure Right to portability Right to object Right to withdraw consent Legitimate interests X X X X Consent X X X X X - Right to be informed : whatever the lawful basis you choose, you need to inform your visitor about it within your privacy notice.
- Right of access : as described in article 15 of GDPR. Your visitor has the right to access the personal data you are processing about them. You can exercise their right directly within the page “GDPR Tools” in your Matomo.
- Right to erasure : it means that a visitor will be able to ask you to erase all their data. You can exercise the right to erasure directly within the page “GDPR Tools” in your Matomo.
- Right to portability : it means that you need to export the data which concern the individual in a machine-readable format and provide them with their personal data. You can exercise their right directly within the page “GDPR Tools” in your Matomo.
- Right to object : it means that your visitor has the right to say no to the processing of their personal data. In order to exercise this right, you need to implement the opt-out feature on your website.
- Right to withdraw consent : it means that your visitor can remove their consent at any time. We developed a feature in order to do just that. You can learn more by opening the page “Privacy > Asking for consent” in your Matomo.
2 – Document your choice
Once you choose “Legitimate interests” or “Consent” lawful basis, you will have some obligations to fulfill. From our interpretation, “Legitimate interests” means writing more documentation, “Consent” means a more technical approach.
What should I do if I am processing personal data with Matomo based on “Legitimate interests ?
ICO is providing a checklist for “Legitimate interests”, below is our interpretation :
- Check that legitimate interests is the most appropriate lawful basis.
Our interpretation : document and justify why you choose this lawful basis in particular. This tool from ICO can help you.
- Understand your responsibility to protect the individual’s interests.
Our interpretation : you need to take all the measures in order to protect your users privacy and data security. Please refer to our guide in order to secure your Matomo installation.
- Conduct a legitimate interests assessment (LIA) and keep a record of it to ensure that you can justify your decision. This document is composed of a set of questions on those 3 key concerns : 1) purpose, 2) necessity, 3) balancing.
1) Purpose :
- Why do you want to process the data – what are you trying to achieve ?
- Who benefits from the processing ? In what way ?
- Are there any wider public benefits to the processing ?
- How important are those benefits ?
- What would the impact be if you couldn’t go ahead ?
- Would your use of the data be unethical or unlawful in any way ?
2) Necessity :
- Does this processing actually help to further that interest ?
- Is it a reasonable way to go about it ?
- Is there another less intrusive way to achieve the same result ?
3) Balancing :
- What is the nature of your relationship with the individual ?
- Is any of the data particularly sensitive or private ?
- Would people expect you to use their data in this way ?
- Are you happy to explain it to them ?
- Are some people likely to object or find it intrusive ?
- What is the possible impact on the individual ?
- How big an impact might it have on them ?
- Are you processing children’s data ?
- Are any of the individuals vulnerable in any other way ?
- Can you adopt any safeguards to minimise the impact ?
- Can you offer an opt-out ?
- Identify the relevant legitimate interests.
- Check that the processing is necessary and there is no less intrusive way to achieve the same result.
- Perform a balancing test, and be confident that the individual’s interests do not override those legitimate interests.
- Use individuals’ data in ways they would reasonably expect, unless you have a very good reason.
Our interpretation : use those data to improve user experience for example.
- Do not use people’s data in ways they would find intrusive or which could cause them harm, unless you have a very good reason.
Our interpretation : ask yourself if this data is representing a risk for the individuals.
- If you process children’s data, take extra care to make sure you protect their interests.
- Consider safeguards to reduce the impact where possible.
Our interpretation : Check if your web hosting provider is providing appropriate safeguards.
- Consider whether you can offer an opt out.
Our interpretation : Matomo is providing you the opt-out feature.
- If your LIA identifies a significant privacy impact, consider whether you also need to conduct a DPIA.
Our interpretation : A DPIA can easily be conducted by using this software from the French privacy commission.
- Regularly review your LIA and update it when circumstances change.
- Include information about your legitimate interests in your privacy information.
As you see, going for “Legitimate interests” requires a lot of written documentation. Let’s see how “Consent” differ.
What should I do if I am processing personal data with Matomo based on “Consent” ?
As previously mentioned, using “Consent” rather than “Legitimate interests” is more technical but less intense in terms of documentation. Like for “Legitimate interests”, ICO is providing a checklist for “Consent” which is divided into 3 key categories : 1) asking for consent, 2) recording consent, and 3) managing consent.
- Asking for consent :
- Check that consent is the most appropriate lawful basis for processing.
- Make the request for consent prominent and separate from your terms and conditions.
- Ask people to positively opt in. Don’t use pre-ticked boxes or any other type of default consent.
- Use clear, plain language that is easy to understand.
- Specify why you want the data and what you are going to do with it.
- Give individual (‘granular’) options to consent separately to different purposes and types of processing.
- Name your organisation and any third party controllers who will be relying on the consent.
- Tell individuals they can withdraw their consent.
- Ensure that individuals can refuse to consent without detriment.
- Avoid making consent a precondition of a service.
- If you offer online services directly to children, only seek consent if you have age-verification measures (and parental-consent measures for younger children) in place.
- Recording consent :
- Keep a record of when and how you got consent from the individual.
- Keep a record of exactly what you told them at the time.
- Managing consent :
- Regularly review consents to check that the relationship, the processing and the purposes have not changed.
- Have processes in place to refresh consent at appropriate intervals, including any parental consent.
- Consider using privacy dashboards or other preference-management tools as a matter of good practice.
- Make it easy for individuals to withdraw their consent at any time, and publicise how to do so.
- Act on withdrawals of consent as soon as you can.
- Don’t penalise individuals who wish to withdraw consent.
3 – Inform your visitor about it in a privacy notice
Privacy notices are an important part within the GDPR process. Read our blog post dedicated to privacy notices to learn more.
We really hope you enjoyed reading this blog post. Please have a look at our Matomo GDPR guide for more information.
The post Lawful basis for processing personal data under GDPR with Matomo appeared first on Analytics Platform - Matomo.
-
Reverse Engineering Italian Literature
1er juillet 2014, par Multimedia Mike — Reverse EngineeringSome time ago, Diego “Flameeyes” Pettenò tried his hand at reverse engineering a set of really old CD-ROMs containing even older Italian literature. The goal of this RE endeavor would be to extract the useful literature along with any structural metadata (chapters, etc.) and convert it to a more open format suitable for publication at, e.g., Project Gutenberg or Archive.org.
Unfortunately, the structure of the data thwarted the more simplistic analysis attempts (like inspecting for blocks of textual data). This will require deeper RE techniques. Further frustrating the effort, however, is the fact that the binaries that implement the reading program are written for the now-archaic Windows 3.1 operating system.
In pursuit of this RE goal, I recently thought of a way to glean more intelligence using DOSBox.
Prior Work
There are 6 discs in the full set (distributed along with 6 sequential issues of a print magazine named L’Espresso). Analysis of the contents of the various discs reveals that many of the files are the same on each disc. It was straightforward to identify the set of files which are unique on each disc. This set of files all end with the extension “LZn”, where n = 1..6 depending on the disc number. Further, the root directory of each disc has a file indicating the sequence number (1..6) of the CD. Obviously, these are the interesting targets.The LZ file extensions stand out to an individual skilled in the art of compression– could it be a variation of the venerable LZ compression ? That’s actually unlikely because LZ — also seen as LIZ — stands for Letteratura Italiana Zanichelli (Zanichelli’s Italian Literature).
The Unix ‘file’ command was of limited utility, unable to plausibly identify any of the files.
Progress was stalled.
Saying Hello To An Old Frenemy
I have been showing this screenshot to younger coworkers to see if any of them recognize it :
Not a single one has seen it before. Senior computer citizen status : Confirmed.
I recently watched an Ancient DOS Games video about Windows 3.1 games. This episode showed Windows 3.1 running under DOSBox. I had heard this was possible but that it took a little work to get running. I had a hunch that someone else had probably already done the hard stuff so I took to the BitTorrent networks and quickly found a download that had the goods ready to go– a directory of Windows 3.1 files that just had to be dropped into a DOSBox directory and they would be ready to run.
Aside : Running OS software procured from a BitTorrent network ? Isn’t that an insane security nightmare ? I’m not too worried since it effectively runs under a sandboxed virtual machine, courtesy of DOSBox. I suppose there’s the risk of trojan’d OS software infecting binaries that eventually leave the sandbox.
Using DOSBox Like ‘strace’
strace is a tool available on some Unix systems, including Linux, which is able to monitor the system calls that a program makes. In reverse engineering contexts, it can be useful to monitor an opaque, binary program to see the names of the files it opens and how many bytes it reads, and from which locations. I have written examples of this before (wow, almost 10 years ago to the day ; now I feel old for the second time in this post).Here’s the pitch : Make DOSBox perform as strace in order to serve as a platform for reverse engineering Windows 3.1 applications. I formed a mental model about how DOSBox operates — abstracted file system classes with methods for opening and reading files — and then jumped into the source code. Sure enough, the code was exactly as I suspected and a few strategic print statements gave me the data I was looking for.
Eventually, I even took to running DOSBox under the GNU Debugger (GDB). This hasn’t proven especially useful yet, but it has led to an absurd level of nesting :
The target application runs under Windows 3.1, which is running under DOSBox, which is running under GDB. This led to a crazy situation in which DOSBox had the mouse focus when a GDB breakpoint was triggered. At this point, DOSBox had all desktop input focus and couldn’t surrender it because it wasn’t running. I had no way to interact with the Linux desktop and had to reboot the computer. The next time, I took care to only use the keyboard to navigate the application and trigger the breakpoint and not allow DOSBox to consume the mouse focus.
New Intelligence
By instrumenting the local file class (virtual HD files) and the ISO file class (CD-ROM files), I was able to watch which programs and dynamic libraries are loaded and which data files the code cares about. I was able to narrow down the fact that the most interesting programs are called LEGGENDO.EXE (‘reading’) and LEGGENDA.EXE (‘legend’ ; this has been a great Italian lesson as well as RE puzzle). The first calls the latter, which displays this view of the data we are trying to get at :
When first run, the program takes an interest in a file called DBBIBLIO (‘database library’, I suspect) :
=== Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x0 === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x151 === Read(’LIZ98\DBBIBLIO.LZ1’) : req 337 bytes ; read 337 bytes from pos 0x2A2 [...]
While we were unable to sort out all of the data files in our cursory investigation, a few things were obvious. The structure of this file looked to contain 336-byte records. Turns out I was off by 1– the records are actually 337 bytes each. The count of records read from disc is equal to the number of items shown in the UI.
Next, the program is interested in a few more files :
*** isoFile() : ’DEPOSITO\BLOKCTC.LZ1’, offset 0x27D6000, 2911488 bytes large === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 96 bytes ; read 96 bytes from pos 0x0 *** isoFile() : ’DEPOSITO\BLOKCTX0.LZ1’, offset 0x2A9D000, 17152 bytes large === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 128 bytes ; read 128 bytes from pos 0x0 === Seek(’DEPOSITO\BLOKCTX0.LZ1’) : seek 384 (0x180) bytes, type 0 === Read(’DEPOSITO\BLOKCTX0.LZ1’) : req 256 bytes ; read 256 bytes from pos 0x180 === Seek(’DEPOSITO\BLOKCTC.LZ1’) : seek 1152 (0x480) bytes, type 0 === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 32 bytes ; read 32 bytes from pos 0x480 === Read(’DEPOSITO\BLOKCTC.LZ1’) : req 1504 bytes ; read 1504 bytes from pos 0x4A0 [...]
Eventually, it becomes obvious that BLOKCTC has the juicy meat. There are 32-byte records followed by variable-length encoded text sections. Since there is no text to be found in these files, the text is either compressed, encrypted, or both. Some rough counting (the program seems to disable copy/paste, which thwarts more precise counting), indicates that the text size is larger than the data chunks being read from disc, so compression seems likely. Encryption isn’t out of the question (especially since the program deems it necessary to disable copy and pasting of this public domain literary data), and if it’s in use, that means the key is being read from one of these files.
Blocked On Disassembly
So I’m a bit blocked right now. I know exactly where the data lives, but it’s clear that I need to reverse engineer some binary code. The big problem is that I have no idea how to disassemble Windows 3.1 binaries. These are NE-type executable files. Disassemblers abound for MZ files (MS-DOS executables) and PE files (executables for Windows 95 and beyond). NE files get no respect. It’s difficult (but not impossible) to even find data about the format anymore, and details are incomplete. It should be noted, however, the DOSBox-as-strace method described here lends insight into how Windows 3.1 processes NE-type EXEs. You can’t get any more authoritative than that.So far, I have tried the freeware version of IDA Pro. Unfortunately, I haven’t been able to get the program to work on my Windows machine for a long time. Even if I could, I can’t find any evidence that it actually supports NE files (the free version specifically mentions MZ and PE, but does not mention NE or LE).
I found an old copy of Borland’s beloved Turbo Assembler and Debugger package. It has Turbo Debugger for Windows, both regular and 32-bit versions. Unfortunately, the normal version just hangs Windows 3.1 in DOSBox. The 32-bit Turbo Debugger loads just fine but can’t load the NE file.
I’ve also wondered if DOSBox contains any advanced features for trapping program execution and disassembling. I haven’t looked too deeply into this yet.
Future Work
NE files seem to be the executable format that time forgot. I have a crazy brainstorm about repacking NE files as MZ executables so that they could be taken apart with an MZ disassembler. But this will take some experimenting.If anyone else has any ideas about ripping open these binaries, I would appreciate hearing them.
And I guess I shouldn’t be too surprised to learn that all the literature in this corpus is already freely available and easily downloadable anyway. But you shouldn’t be too surprised if that doesn’t discourage me from trying to crack the format that’s keeping this particular copy of the data locked up.
Syndiquer tout le site
