Skip to content

Subtitle Filtering 2022 #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
softworkz wants to merge 25 commits into
base: master
Choose a base branch
from
Open

Subtitle Filtering 2022 #18

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
bc849a2
avcodec,avutil: Move enum AVSubtitleType to avutil, add new and depre…
softworkz Dec 10, 2021
95e9b5f
avutil/frame: Prepare AVFrame for subtitle handling
softworkz Jun 11, 2022
3598ec3
avcodec/subtitles: Introduce new frame-based subtitle decoding API
softworkz May 28, 2022
76ed737
avcodec/libzvbi: set subtitle type
softworkz Mar 10, 2022
2043071
avfilter/subtitles: Update vf_subtitles to use new decoding api
softworkz May 28, 2022
abfe4c9
avcodec,avutil: Move ass helper functions to avutil as avpriv_ and ex…
softworkz Jun 24, 2022
4acdb3a
avcodec/subtitles: Replace deprecated enum values
softworkz Jan 9, 2022
e03ce50
fftools/play,probe: Adjust for subtitle changes
softworkz Jan 13, 2022
2ea0e71
avfilter/subtitles: Add subtitles.c for subtitle frame allocation
softworkz Jan 13, 2022
e7055c7
avfilter/avfilter: Handle subtitle frames
softworkz Jun 26, 2022
0b31d69
avfilter/avfilter: Fix hardcoded input index
softworkz Nov 27, 2021
066b754
avfilter/sbuffer: Add sbuffersrc and sbuffersink filters
softworkz May 26, 2022
2dce3f8
avfilter/overlaygraphicsubs: Add overlaygraphicsubs and graphicsub2vi…
softworkz May 28, 2022
14d291d
avfilter/overlaytextsubs: Add overlaytextsubs and textsubs2video filters
softworkz May 28, 2022
15a8984
avfilter/textmod: Add textmod, censor and show_speaker filters
softworkz Jun 23, 2022
5433d27
avfilter/stripstyles: Add stripstyles filter
softworkz Jun 23, 2022
39bb78e
avfilter/splitcc: Add splitcc filter for closed caption handling
softworkz Dec 3, 2021
80f4c93
avfilter/graphicsub2text: Add new graphicsub2text filter (OCR)
softworkz Jun 23, 2022
4e6888c
avfilter/subscale: Add filter for scaling and/or re-arranging graphic…
softworkz May 28, 2022
5874f5c
avfilter/subfeed: add subtitle feed filter
softworkz Jun 21, 2022
938d0ef
avfilter/text2graphicsub: Added text2graphicsub subtitle filter
softworkz Jun 26, 2022
99e5c11
avfilter/snull,strim: Add snull and strim filters
softworkz Jun 26, 2022
300c14f
avcodec/subtitles: Migrate subtitle encoders to frame-based API
softworkz Oct 2, 2022
6046dde
fftools/ffmpeg: Introduce subtitle filtering and new frame-based subt…
softworkz Oct 24, 2022
bb11798
avcodec/dvbsubdec: Fix conditions for fallback to default resolution
softworkz Dec 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 1 addition & 18 deletions libavcodec/avcodec.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@
#include "libavutil/frame.h"
#include "libavutil/log.h"
#include "libavutil/pixfmt.h"
#include "libavutil/subfmt.h"
#include "libavutil/rational.h"

#include "codec.h"
Expand Down Expand Up @@ -2257,24 +2258,6 @@ typedef struct AVHWAccel {
* @}
*/

enum AVSubtitleType {
SUBTITLE_NONE,

SUBTITLE_BITMAP, ///< A bitmap, pict will be set

/**
* Plain text, the text field must be set by the decoder and is
* authoritative. ass and pict fields may contain approximations.
*/
SUBTITLE_TEXT,

/**
* Formatted text, the ass field must be set by the decoder and is
* authoritative. pict and text fields may contain approximations.
*/
SUBTITLE_ASS,
};

#define AV_SUBTITLE_FLAG_FORCED 0x00000001

typedef struct AVSubtitleRect {
Expand Down
2 changes: 2 additions & 0 deletions libavutil/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ HEADERS = adler32.h \
sha512.h \
Copy link

@ffmpeg-codebot ffmpeg-codebot bot Oct 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the FFmpeg mailing list, Hendrik Leppkes wrote (reply to this):

On Tue, Oct 25, 2022 at 11:14 AM softworkz <ffmpegagent@gmail.com> wrote:
>
> @@ -712,6 +712,53 @@ typedef struct AVFrame {
>       * Duration of the frame, in the same units as pts. 0 if unknown.
>       */
>      int64_t duration;
> +
> +    /**
> +     * Media type of the frame (audio, video, subtitles..)
> +     *
> +     * See AVMEDIA_TYPE_xxx
> +     */
> +    enum AVMediaType type;
> +
> +    /**
> +     * Number of items in the @ref subtitle_areas array.
> +     */
> +    unsigned num_subtitle_areas;
> +
> +    /**
> +     * Array of subtitle areas, may be empty.
> +     */
> +    AVSubtitleArea **subtitle_areas;
> +
> +    /**
> +     * Header containing style information for text subtitles.
> +     */
> +    AVBufferRef *subtitle_header;
> +
> +    /**
> +     * Indicates that a subtitle frame is a repeated frame for arbitrating flow
> +     * in a filter graph.
> +     * The field subtitle_timing.start_pts always indicates the original presentation
> +     * time, while the frame's pts field may be different.
> +     */
> +    int repeat_sub;
> +
> +    struct SubtitleTiming
> +    {
> +        /**
> +         * The display start time, in AV_TIME_BASE.
> +         *
> +         * For subtitle frames, AVFrame.pts is populated from the packet pts value,
> +         * which is not always the same as this value.
> +         */
> +        int64_t start_pts;

There is still no explanation here why they are not the same, why they
could not just be the same, and which field a user should look at.
The cover letter talks about clarity why this is needed is important,
but then provides none of that.

"Its required" is not an argument. So please enlighten us once again
why we absolutely need two timestamps for subtitles, instead of just
one. As far as I can see, subtitle frames only have one relevant time
- when its supposed to be shown on the screen. What would the other
time ever be good for to a user?
Similarly for the duration, of course. I can even see the
AVFrame.duration field in this patch snippet just above the additions
that would seem to fully replace this one.

- Hendrik



> +
> +        /**
> +         * Display duration, in AV_TIME_BASE.
> +         */
> +        int64_t duration;
> +
> +    } subtitle_timing;
>  } AVFrame;
>
>
> @@ -788,6 +835,8 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src);
>  /**
>   * Allocate new buffer(s) for audio or video data.
>   *
> + * Note: For subtitle data, use av_frame_get_buffer2
> + *
>   * The following fields must be set on frame before calling this function:
>   * - format (pixel format for video, sample format for audio)
>   * - width and height for video
> @@ -807,9 +856,39 @@ void av_frame_move_ref(AVFrame *dst, AVFrame *src);
>   *              recommended to pass 0 here unless you know what you are doing.
>   *
>   * @return 0 on success, a negative AVERROR on error.
> + *
> + * @deprecated Use @ref av_frame_get_buffer2 instead and set @ref AVFrame.type
> + * before calling.
>   */
> +attribute_deprecated
>  int av_frame_get_buffer(AVFrame *frame, int align);
>
> +/**
> + * Allocate new buffer(s) for audio, video or subtitle data.
> + *
> + * The following fields must be set on frame before calling this function:
> + * - format (pixel format for video, sample format for audio)
> + * - width and height for video
> + * - nb_samples and channel_layout for audio
> + * - type (AVMediaType)
> + *
> + * This function will fill AVFrame.data and AVFrame.buf arrays and, if
> + * necessary, allocate and fill AVFrame.extended_data and AVFrame.extended_buf.
> + * For planar formats, one buffer will be allocated for each plane.
> + *
> + * @warning: if frame already has been allocated, calling this function will
> + *           leak memory. In addition, undefined behavior can occur in certain
> + *           cases.
> + *
> + * @param frame frame in which to store the new buffers.
> + * @param align Required buffer size alignment. If equal to 0, alignment will be
> + *              chosen automatically for the current CPU. It is highly
> + *              recommended to pass 0 here unless you know what you are doing.
> + *
> + * @return 0 on success, a negative AVERROR on error.
> + */
> +int av_frame_get_buffer2(AVFrame *frame, int align);
> +
>  /**
>   * Check if the frame data is writable.
>   *
> diff --git a/libavutil/subfmt.c b/libavutil/subfmt.c
> new file mode 100644
> index 0000000000..c72ebe2a43
> --- /dev/null
> +++ b/libavutil/subfmt.c
> @@ -0,0 +1,45 @@
> +/*
> + * Copyright (c) 2021 softworkz
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <string.h>
> +#include "common.h"
> +#include "subfmt.h"
> +
> +static const char sub_fmt_info[AV_SUBTITLE_FMT_NB][24] = {
> +    [AV_SUBTITLE_FMT_UNKNOWN] = "Unknown subtitle format",
> +    [AV_SUBTITLE_FMT_BITMAP]  = "Graphical subtitles",
> +    [AV_SUBTITLE_FMT_TEXT]    = "Text subtitles (plain)",
> +    [AV_SUBTITLE_FMT_ASS]     = "Text subtitles (ass)",
> +};
> +
> +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt)
> +{
> +    if (sub_fmt < 0 || sub_fmt >= AV_SUBTITLE_FMT_NB)
> +        return NULL;
> +    return sub_fmt_info[sub_fmt];
> +}
> +
> +enum AVSubtitleType av_get_subtitle_fmt(const char *name)
> +{
> +    for (int i = 0; i < AV_SUBTITLE_FMT_NB; i++)
> +        if (!strcmp(sub_fmt_info[i], name))
> +            return i;
> +    return AV_SUBTITLE_FMT_NONE;
> +}
> diff --git a/libavutil/subfmt.h b/libavutil/subfmt.h
> index 791b45519f..3b8a0613b2 100644
> --- a/libavutil/subfmt.h
> +++ b/libavutil/subfmt.h
> @@ -21,6 +21,9 @@
>  #ifndef AVUTIL_SUBFMT_H
>  #define AVUTIL_SUBFMT_H
>
> +#include <stdint.h>
> +
> +#include "buffer.h"
>  #include "version.h"
>
>  enum AVSubtitleType {
> @@ -65,4 +68,48 @@ enum AVSubtitleType {
>      AV_SUBTITLE_FMT_NB,         ///< number of subtitle formats, DO NOT USE THIS if you want to link with shared libav* because the number of formats might differ between versions.
>  };
>
> +typedef struct AVSubtitleArea {
> +#define AV_NUM_BUFFER_POINTERS 1
> +
> +    enum AVSubtitleType type;
> +    int flags;
> +
> +    int x;         ///< top left corner  of area.
> +    int y;         ///< top left corner  of area.
> +    int w;         ///< width            of area.
> +    int h;         ///< height           of area.
> +    int nb_colors; ///< number of colors in bitmap palette (@ref pal).
> +
> +    /**
> +     * Buffers and line sizes for the bitmap of this subtitle.
> +     *
> +     * @{
> +     */
> +    AVBufferRef *buf[AV_NUM_BUFFER_POINTERS];
> +    int linesize[AV_NUM_BUFFER_POINTERS];
> +    /**
> +     * @}
> +     */
> +
> +    uint32_t pal[256]; ///< RGBA palette for the bitmap.
> +
> +    char *text;        ///< 0-terminated plain UTF-8 text
> +    char *ass;         ///< 0-terminated ASS/SSA compatible event line.
> +
> +} AVSubtitleArea;
> +
> +/**
> + * Return the name of sub_fmt, or NULL if sub_fmt is not
> + * recognized.
> + */
> +const char *av_get_subtitle_fmt_name(enum AVSubtitleType sub_fmt);
> +
> +/**
> + * Return a subtitle format corresponding to name, or AV_SUBTITLE_FMT_NONE
> + * on error.
> + *
> + * @param name Subtitle format name.
> + */
> +enum AVSubtitleType av_get_subtitle_fmt(const char *name);
> +
>  #endif /* AVUTIL_SUBFMT_H */
> --
> ffmpeg-codebot
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Copy link

@ffmpeg-codebot ffmpeg-codebot bot Oct 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the FFmpeg mailing list, Soft Works wrote (reply to this):



> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Hendrik Leppkes
> Sent: Tuesday, October 25, 2022 11:38 AM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH v9 02/25] avutil/frame: Prepare
> AVFrame for subtitle handling
> 
> On Tue, Oct 25, 2022 at 11:14 AM softworkz <ffmpegagent@gmail.com>
> wrote:
> >
> > @@ -712,6 +712,53 @@ typedef struct AVFrame {
> >       * Duration of the frame, in the same units as pts. 0 if
> unknown.
> >       */
> >      int64_t duration;
> > +
> > +    /**
> > +     * Media type of the frame (audio, video, subtitles..)
> > +     *
> > +     * See AVMEDIA_TYPE_xxx
> > +     */
> > +    enum AVMediaType type;
> > +
> > +    /**
> > +     * Number of items in the @ref subtitle_areas array.
> > +     */
> > +    unsigned num_subtitle_areas;
> > +
> > +    /**
> > +     * Array of subtitle areas, may be empty.
> > +     */
> > +    AVSubtitleArea **subtitle_areas;
> > +
> > +    /**
> > +     * Header containing style information for text subtitles.
> > +     */
> > +    AVBufferRef *subtitle_header;
> > +
> > +    /**
> > +     * Indicates that a subtitle frame is a repeated frame for
> arbitrating flow
> > +     * in a filter graph.
> > +     * The field subtitle_timing.start_pts always indicates the
> original presentation
> > +     * time, while the frame's pts field may be different.
> > +     */
> > +    int repeat_sub;
> > +
> > +    struct SubtitleTiming
> > +    {
> > +        /**
> > +         * The display start time, in AV_TIME_BASE.
> > +         *
> > +         * For subtitle frames, AVFrame.pts is populated from the
> packet pts value,
> > +         * which is not always the same as this value.
> > +         */
> > +        int64_t start_pts;
> 
> There is still no explanation here why they are not the same, why
> they
> could not just be the same, and which field a user should look at.
> The cover letter talks about clarity why this is needed is important,
> but then provides none of that.
> 
> "Its required" is not an argument. So please enlighten us once again
> why we absolutely need two timestamps for subtitles, instead of just
> one. As far as I can see, subtitle frames only have one relevant time
> - when its supposed to be shown on the screen. What would the other
> time ever be good for to a user?
> Similarly for the duration, of course. I can even see the
> AVFrame.duration field in this patch snippet just above the additions
> that would seem to fully replace this one.
> 
> - Hendrik

Hi Hendrik,

thanks a lot for your reply.

Probably I should have better advertised the article I had written 
specifically to explain the background of this:

https://github.com/softworkz/SubtitleFilteringDemos/issues/1

I hope it's understandable - please let me know when you have 
questions.


Thanks again,
softworkz
_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

Copy link

@ffmpeg-codebot ffmpeg-codebot bot Oct 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the FFmpeg mailing list, Soft Works wrote (reply to this):



> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> Soft Works
> Sent: Tuesday, October 25, 2022 11:59 AM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel@ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH v9 02/25] avutil/frame: Prepare
> AVFrame for subtitle handling
> 
> 
> 
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces@ffmpeg.org> On Behalf Of
> > Hendrik Leppkes
> > Sent: Tuesday, October 25, 2022 11:38 AM
> > To: FFmpeg development discussions and patches <ffmpeg-
> > devel@ffmpeg.org>
> > Subject: Re: [FFmpeg-devel] [PATCH v9 02/25] avutil/frame: Prepare
> > AVFrame for subtitle handling
> >
> > On Tue, Oct 25, 2022 at 11:14 AM softworkz <ffmpegagent@gmail.com>
> > wrote:
> > >
> > > @@ -712,6 +712,53 @@ typedef struct AVFrame {
> > >       * Duration of the frame, in the same units as pts. 0 if
> > unknown.
> > >       */
> > >      int64_t duration;
> > > +
> > > +    /**
> > > +     * Media type of the frame (audio, video, subtitles..)
> > > +     *
> > > +     * See AVMEDIA_TYPE_xxx
> > > +     */
> > > +    enum AVMediaType type;
> > > +
> > > +    /**
> > > +     * Number of items in the @ref subtitle_areas array.
> > > +     */
> > > +    unsigned num_subtitle_areas;
> > > +
> > > +    /**
> > > +     * Array of subtitle areas, may be empty.
> > > +     */
> > > +    AVSubtitleArea **subtitle_areas;
> > > +
> > > +    /**
> > > +     * Header containing style information for text subtitles.
> > > +     */
> > > +    AVBufferRef *subtitle_header;
> > > +
> > > +    /**
> > > +     * Indicates that a subtitle frame is a repeated frame for
> > arbitrating flow
> > > +     * in a filter graph.
> > > +     * The field subtitle_timing.start_pts always indicates the
> > original presentation
> > > +     * time, while the frame's pts field may be different.
> > > +     */
> > > +    int repeat_sub;
> > > +
> > > +    struct SubtitleTiming
> > > +    {
> > > +        /**
> > > +         * The display start time, in AV_TIME_BASE.
> > > +         *
> > > +         * For subtitle frames, AVFrame.pts is populated from
> the
> > packet pts value,
> > > +         * which is not always the same as this value.
> > > +         */
> > > +        int64_t start_pts;
> >
> > There is still no explanation here why they are not the same, why
> > they
> > could not just be the same, and which field a user should look at.
> > The cover letter talks about clarity why this is needed is
> important,
> > but then provides none of that.
> >
> > "Its required" is not an argument. So please enlighten us once
> again
> > why we absolutely need two timestamps for subtitles, instead of
> just
> > one. As far as I can see, subtitle frames only have one relevant
> time
> > - when its supposed to be shown on the screen. What would the other
> > time ever be good for to a user?
> > Similarly for the duration, of course. I can even see the
> > AVFrame.duration field in this patch snippet just above the
> additions
> > that would seem to fully replace this one.
> >
> > - Hendrik
> 
> Hi Hendrik,
> 
> thanks a lot for your reply.
> 
> Probably I should have better advertised the article I had written
> specifically to explain the background of this:
> 
> https://github.com/softworkz/SubtitleFilteringDemos/issues/1

@Hendrik - did that answer your question?

You are also welcome to contact me directly for discussing details
or going through specific examples; or just to ping me for logging
on to IRC for chat.

softworkz

_______________________________________________
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel

To unsubscribe, visit link above, or email
ffmpeg-devel-request@ffmpeg.org with subject "unsubscribe".

spherical.h \
stereo3d.h \
subfmt.h \
threadmessage.h \
time.h \
timecode.h \
Expand Down Expand Up @@ -164,6 +165,7 @@ OBJS = adler32.o \
slicethread.o \
spherical.o \
stereo3d.o \
subfmt.o \
threadmessage.o \
time.o \
timecode.o \
Expand Down
Loading