IPTV/HLS/M3U: Failing to parse EXT-X-VERSION 7

[angelaschmid]

Running latest master.

Streams from Zattoo, Teleboy, famous IPTV providers in Germany/Switzerland, are failing to play in MythTV. They play fine in VLC.
MythTV does not have version 7 M3U implemented.

The stream contains the following M3U data:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA:TYPE=AUDIO,GROUP-ID="audio",NAME="deutsch",DEFAULT=YES,AUTOSELECT=YES,LANGUAGE="deu",URI="https://zba6-0-hls7-live.zahs.tv/telebielingue/t_track_audio_bw_128000000_num_0_tid_2_p_10_l_de_nd_1600_mbr_2999.m3u8?z32=MF2......"
#EXT-X-STREAM-INF:BANDWIDTH=2999000,CODECS="avc1.4d4020,mp4a.40.2",RESOLUTION=1024x576,FRAME-RATE=50,AUDIO="audio",CLOSED-CAPTIONS=NONE
https://zba6-0-hls7-live.zahs.tv/telebielingue/t_track_video_bw_2799000_num_0_tid_1_nd_1600_mbr_2999.m3u8?z32=MF2......

• MythTV version:
  Latest master

• Component:
  mythbackend

What do you see instead?

Logfile mythbackend:
StreamHandler m3u.cpp:106 (ParseVersion) - #EXT-X-VERSION is 7, but we only understand 0 through 3

Changing the source to also allow version 7, gives the following failures, and the stream does not play:

2024-08-05 10:41:19.523190 I [344463/352425] StreamHandler m3u.cpp:120 (ParseStreamInformation) - Parsing stream from https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L
2024-08-05 10:41:19.523196 I [344463/352425] StreamHandler m3u.cpp:132 (ParseStreamInformation) - #EXT-X-STREAM-INF: expected PROGRAM-ID=, using -1
2024-08-05 10:41:19.523200 I [344463/352425] StreamHandler m3u.cpp:157 (ParseStreamInformation) - bandwidth adaptation detected (program-id=-1, bandwidth=8000000
2024-08-05 10:41:20.590946 D [344463/352428] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L tag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:39:56.800+00:00
2024-08-05 10:41:21.603634 D [344463/352428] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L tag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:39:56.800+00:00
2024-08-05 10:41:22.615515 D [344463/352428] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zba6-1-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2WI2L tag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:39:58.400+00:00
2024-08-05 10:44:24.025961 I [344463/358923] StreamHandler m3u.cpp:120 (ParseStreamInformation) - Parsing stream from https://zh4-11-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2W
2024-08-05 10:44:24.025965 I [344463/358923] StreamHandler m3u.cpp:132 (ParseStreamInformation) - #EXT-X-STREAM-INF: expected PROGRAM-ID=, using -1
2024-08-05 10:44:24.025969 I [344463/358923] StreamHandler m3u.cpp:157 (ParseStreamInformation) - bandwidth adaptation detected (program-id=-1, bandwidth=8000000
2024-08-05 10:44:25.103642 D [344463/358926] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zh4-11-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2Wtag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:43:00.800+00:00
2024-08-05 10:44:26.117132 D [344463/358926] HLSPlaylist m3u.cpp:367 (ParseProgramDateTime) - https://zh4-11-hls7-live.zahs.tv/HD_s1/t_track_video_bw_7800000_num_0_tid_1_nd_1600_mbr_8000.m3u8?z32=MF2Wtag not supported: #EXT-X-PROGRAM-DATE-TIME #EXT-X-PROGRAM-DATE-TIME:2024-08-05T08:43:02.400+00:00

[kmdewaal]

Can you please provide a complete version 7 m3u file?
Actually playing something from Zattoo might be difficult in The Netherlands because of geofencing but a fix for the parser can be tested.

[kmdewaal]

HLS version 7 is described here https://datatracker.ietf.org/doc/html/rfc8216 and it looks to be non-trivial undertaking...

[angelaschmid]

Hi Klaas

Please find attached mythtv_936_m3u.zip

• hls_init7_list.m3u, program list of my local internet provider, only available on their network
• hls_init7_ext6_stream.m3u, EXT6 stream
• hls_zattoo_list.m3u, program list of Zattoo Switzerland, GEO-blocked
• hls_zattoo_ext7_stream.m3u, EXT7 stream

Zattoo is offered only in Germany, Switzerland and Austria. Each having their own account and GEO-blocking.
To use it, I use "telerising", a daemon who acts in the middle to login to Zattoo (cookie handling) and produces a program.m3u and stream.m3u:

• Use a VPN to one of the countries
• Create a free account, allows 30 hours of streaming/month.
• download, https://github.com/sunsettrack4/telerising-api: telerising-v0.13.4_x86-64_linux.zip
• unpack
• exec ./api, shows the available web server ports
• browser, e.g. http://192.168.1.11:5000
• configure: choose admin password, choose e.g. Zattoo CH (Smart TV), press SELECT, add account details, press "Add TV Setup", use M3U playlist URL

https://www.kodinerds.net/thread/72127-telerising-api-zattoo-blue-tv-sky-ch-f%C3%BCr-tvheadend-und-vlc-web-app/
https://github.com/applesh00ter/telerising-api

Currently I am stuck on master of ~August, due to #955, but will be available to test.

Cheers, Angela

[kmdewaal]

Testing with a local IPTV stream for channel EWTN Europe did show up one bug.

The tag EXT-X-DISCONTINUITY-SEQUENCE is parsed and executed as EXT-X-DISCONTINUITY which is another tag, and this leads to a total reset every time the tag is encountered. This then causes downloading at maximum bandwidth and in my case this then downloads 60Mbyte per second. Playback however is OK.

I vaguely recall that this problem has been reported by somebody long ago.

Adding parsing of EXT-X-DISCONTINUITY-SEQUENCE does fix this even when not yet doing anything with the value.
I intend to add a fix for this in the near future.

To be continued.

[kmdewaal]

@angelaschmid I have committed a fix to master. With this fix many channels that previously could not be received are now playing good. I have not yet tested with your channels as I could produce already many problems with the readily available IPTV channels. It is possible that your issue is now resolved. Testing is appreciated.

[kmdewaal]

The above fix does errors in playing the RTVE channel https://rtvelivestream.akamaized.net/rtvesec/24h/24h_main_dvr_720.m3u8.
Reason for the overflow is that the segment list for this channel contains all segments for the next two hours, in segments of 5 seconds. The size of a segment times the number of segments causes the integer overflow. This makes sense; the total size of 2 hours recording can easily be far more than 2Gbyte, the limit of a 32-bit integer.

However, the remaining issue is that IPTV/HLS does read all the segments in the segment list. This means it does a readahead of the next two hours. This gives really big recording files even if you do only a few minutes of Live TV watching on that channel. I think that this is not the behavior that we want, especially not with Live TV but maybe also not when making recordings.

One possible approach is to limit the read-ahead time to about 1 minute or so (12 segments of 5 seconds each) but how to do that exactly needs to be investigated.

[angelaschmid]

A snapshot of https://rtvelivestream.akamaized.net/rtvesec/24h/24h_main_dvr_720.m3u8
produced the last line: https://rtvelivestream.akamaized.net/rtvesec/24h/GL0/2_v_5_2024_11_24_23_02_02_12256.ts
the scene showed a time of 23:02:02, so the last line is the livetv scene, the previous chunks allow to skip back.

vlc shows the total time as 2 hours, forwards to the end (starting with the last chunk), and allows the user to skip back.

HLSPlaylist HLSReader.cpp:457 (ParseM3U8), is executed every second, it should keep track what already was read. LiveTV would mean, only reading the last, but that might be fragile, so reading some chunks back would make sense. Somebody who want to have near realtime LiveTV, can skip forward. I think the same is expected with recordings, as it should start near time, forgetting too many available previous chunks.

Besides your fix, as the channel list contain URLs without an extension, I had to change:

diff --git a/mythtv/libs/libmythtv/iptvtuningdata.h b/mythtv/libs/libmythtv/iptvtuningdata.h
index a1cf9b951b..9a287703fc 100644
--- a/mythtv/libs/libmythtv/iptvtuningdata.h
+++ b/mythtv/libs/libmythtv/iptvtuningdata.h
@@ -241,7 +241,7 @@ class MTV_PUBLIC IPTVTuningData
         auto path = m_dataUrl.path();

         // check url is valid for a playlist before downloading (see trac ticket #12856)
-        if(path.endsWith(".m3u8", Qt::CaseInsensitive) ||
+        if(true || path.endsWith(".m3u8", Qt::CaseInsensitive) ||
            path.endsWith(".m3u", Qt::CaseInsensitive))
         {
             LOG(VB_RECORD, LOG_INFO, QString("IsHLSPlaylist url ends with either .m3u8 or .m3u %1").arg(url));

I have seen URLs like 'http://.....m3u8/?....." (from https://github.com/iptv-org/iptv?tab=readme-ov-file#playlists).

Within GuessProtocol(void)::IsHLSPlaylist(void), the if/else can/should be removed, as downloader.DownloadURL(url, &buffer, 5s, 0, 10000); reads the content, which is verified in HLSReader::IsValidPlaylist(text). It checks that the first line starts with: #EXTM3U. I doubt if it makes sense to read next lines to be checked for other tags, but will for sure be available on the second line. Maybe even reducing it to some 1000 characters, so that IPTVTuningData::http_ts channels are not badly influenced.

[kmdewaal]

About the IsHLSPlaylist function, I do agree with your analysis. The presence of the substring "m3u" either at the end or halfway the URL should not be used to decide on the content of the file.
I did check trac ticket 12856 mentioned in the comment and this has to do with channel up and down from a Kodi plugin with channels received from a VBox. As the check on the presence of m3u does not change the logic but only the timing it has likely been a race condition or so back then. To be ignored for now and if it is reported again it should be properly fixed and not by a time delay.
I also agree that reading 1000 bytes instead of 10000 should be enough and this then also reduces the amount of time spent in this function.
A fix is forthcoming. Thanks.

[kmdewaal]

Reducing the number of bytes read from 10000 to 1000 does cause failure on playing that RTVE channel so this will remain at 10000 for now. Size of the RTVE file is 7.7 kbyte so to get the last chunk we need to read the whole file.

[kmdewaal]

Status update:

EXT-X-VERSION:7 files can be decoded now. Most of the channels present in https://iptv-org.github.io/iptv/countries/nl.m3u can be played; the ones that cannot do not exist anymore, are geo-blocked or are encrypted. All the fixes to get this far have been fairly trivial.

The biggest remaining issue is that some channels do have video playback but no sound. This is because the EXT-X-MEDIA tag is not yet implemented.
Usually the HTTP stream does contain a transport stream with both audio and video. However, it can also happen that the HTTP stream does contain only video and that the tag EXT-X-MEDIA contains the URL for the audio.
This means that two streams must be downloaded in parallel and that they must be remuxed into a single stream. It is clear what needs to be done but it is not trivial.

@angelaschmid I expect that the streams that you mention in this issue can al be played by MythTV but al the streams that use the EXT-X-MEDIA tag to specify the audio URL will play without sound. Can you please check?

To be continued.

[angelaschmid]

Hi @kmdewaal

I tested with latest master (includes your latest channel scanning updates) and found the following issues:

• the m3u playlist should be read UTF8 encoded, see screenshot from webapp: channel scan: playlist history not cleared #1008

• looking at the resulting channel.channum, you might have interpreted EXTINF as the channel number, which it is not, the value is a duration https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.2.1. The following attributes might exist:
  #EXTINF:0001 tvg-id="ORF1.at" tvg-chno="66" group-title="German" tvg-logo="https://images.zattic.com/.../210x120.png", ORF1 HD
  https://github.com/HamzaBhf00/m3u-tags-iptv, as a side note, they might also be uppercase.
  Not interpreting EXTINF as a channum would also fix the issue that with the same EXTINF, only the last channel is saved.

• In my opinion, MythTV wrongly uses in the themes and in the webapp guide, channel.callsign (technical identifier), instead of channel.name (a more descriptive and human-readable name for the channel).
  callsign could be set to tvg-id (or when not available use the descriptive name of the channel information) and which also should/could be used to compare re-scans.

• EXT-X-VERSION:7

  • EXT-X-MEDIA tag can appear multiple times, to accommodate multiple audio tracks, subtitles, closed captions, alternate video renditions with descriptive audio (hearing-impaired, clear audio) or commentary.
  • the question arises, to merge them all, or only the user preferred ones (setting)
  • playing Zattoo which uses newer tags was not possible, as their might also be more "magic" needed.
    Added an example of the downloaded files to play a stream extinf7_zattoo_example.zip
    cat i_track_video_bw_4800000_nd_1600_num_0_tid_1_mbr_5000.mp4 f_track_video_ts_1735579876800_bw_4800000_nd_1600_d_1600_num_0_tid_1_mbr_5000.m4s >combined_mp4_m4s.mp4

[kmdewaal]

Hi @angelaschmid thanks for testing. I will have a look at all issues raised, starting with the channel scan.

About channel callsign and name.
As far as I know this has USA origins where channels can have a callsign, e.g. WXYZ, and also a name.
There is no such thing as a callsign in DVB and in a DVB channelscan the name is stored in both channel.name and channel.callsign. For IPTV we should do the same.
N.B. In MythTV the callsign is used to identifiy channels for scheduling and recording.

About the xmltvid
There is also the channel.xmltvid and this field is used to uniquely identify the channel for the purpose of adding guide data from an XML file with mythfilldatabase. It looks like the tvg-id field is the one to be used as xmltvid. If you have an XML file with guide data maybe you can verify this.

[angelaschmid]

Exactly, the callsign is the "technical" name and can be "cryptic". But the callsign is used in the guides for the channel name, which I think is wrong. Better would be IMO to use the channel.name in the guide. For USA and DVB these can be (initially) the same.
This opens the possibility to use tvg-id (which is in the above example "ORF1.at") to be saved into callsign.

xmltvid; IPTV does not come with guide data, so has to be sourced in from a third-party. These have different xmltvid's, which needs to be configured in xmltvid by the user. Saving tvg-id into it, would overwrite the settings with a channel re-scan.

Init7 provider
#EXTINF:0 tvg-logo="https://api.tv.init7.net/media/logos/1102_SRF1.ch.png" tvg-name="SRF1.ch" group-title="chde", SRF 1
https://api.tv.init7.net/api/live/?channel=b87abb69-d5ed-44c5-8cab-0f7be4ef51b1

Zattoo
#EXTINF:0001 tvg-id="ITV1London.uk" tvg-chno="1" group-title="General Interest" tvg-logo="https://images.zattic.com/logos/ee3c9d2ac083eb2154b5/black/210x120.png", ITV 1 HD
http://127.0.1.1:5000/api/zch/live/itv-1-london

Just noticed a funny side-effect, I removed all IPTV channels, but loaded guide data in a non-existing source (/usr/bin/mythfilldatabase --sourceid 5 --file --xmlfile $XMLFILE), which generated the channels.
I noticed that callsign was generated as 5 characters, when having two words, it took the first three from the first word and the first two from the second word. When having one word, it took the first 5 characters. Converting to uppercase.
After assigning an existing source-id to the imported channels, the guide data shows this generated name.
This behavior underlines my thought, that callsign is a technical name and should not be used to be displayed in the guide.

callsign    name
DISCH	DISNEY CHANNEL
PHOEN	PHOENIX
PLANE	PLANET
PULS8	Puls 8
PRO7	Pro 7
RBB          rbb
RTL2	RTL 2
RTL	        RTL
RTLNI	RTL Nitro
ZDFIN	ZDF Info
ZDFNE	ZDFneo

XMLVT file:
  <channel id="rtlnitro.search.ch">
    <display-name lang="de">RTL Nitro</display-name>
  </channel>

[kmdewaal]

the m3u playlist should be read UTF8 encoded, see screenshot from #1008

Fixed.

looking at the resulting channel.channum, you might have interpreted EXTINF as the channel number, which it is not, the value is a duration https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.2.1. The following attributes might exist:
#EXTINF:0001 tvg-id="ORF1.at" tvg-chno="66" group-title="German" tvg-logo="https://images.zattic.com/.../210x120.png", ORF1 HD
https://github.com/HamzaBhf00/m3u-tags-iptv, as a side note, they might also be uppercase.
Not interpreting EXTINF as a channum would also fix the issue that with the same EXTINF, only the last channel is saved.

Fixed.

[angelaschmid]

User friendly channel names

The program guide channel list in the Arclight theme previously used a template displaying the callsign for channel names. This has been updated to utilize the ChannelFormat setting, which I have set to display channels as <num> <name>.
MythTV-Themes/Arclight@1088916
The rest of the guide and schedule UI used this 'ChannelFormat' setting already.

The following areas still use the callsign (hardcoded):

• mythbackend --printsched (would also be nice to have a wider version)
• webapp: programguide/recordings/upcoming

Your latest changes

I will have a look into your latest changes by end of next week.