Python3 th0ma7 updates #37

th0ma7 · 2021-06-01T01:39:04Z

Now working on the proper branch...

In order for the title & description to be fully compliant in both TVH & Kodi all escape characters needs to be converted to legacy style HTML such as the following:

        data = data.replace('&','&amp;')
        data = data.replace('"','&quot;')
        data = data.replace("'",'&apos;')
        data = data.replace('<','&lt;')
        data = data.replace('>','&gt;')

Other improvements

When CC tag is found then <subtitles type="teletext" /> can be added
Add minimal categories when xdetails=false
Add <length units="minutes">60</length> tag
Manage premiere and last-chance flags
Add full channel name
Only delete irrelevant ShowCache files
Converted Cast, Premiere, New, Finale to using xmltv tags instead of extending the description.
Remove non-essential english wording in description to be more generic and "bilingual"
Follow XMLTV Capabilities for baseline

Notes

Patch related to "full channel name": the proposed variable exception = "..." may be better fitted with an xml config file.
Added more detailed log information such as:

2021/06/12 17:19:50 Running zap2epg-3.0
2021/06/12 17:19:50 Kodi settings version: 2
2021/06/12 17:19:50 	Country: Canada [CAN]
2021/06/12 17:19:50 	Postal code: J3B1M4
2021/06/12 17:19:50 	TV Guide duration: 13 days
2021/06/12 17:19:50 	TV Guide Start: 1623862800 [offset: 4 days]
2021/06/12 17:19:50 	Lineup: Local Over the Air Broadcast
2021/06/12 17:19:50 	Configuration file: /home/spksrc/script.module.zap2epg/epggrab/conf/zap2epg.xml
2021/06/12 17:19:50 	Caching directory path: /home/spksrc/script.module.zap2epg/epggrab/cache

When setting xdetails to false it currently does not process the default minimal non-extended categories that can be found in edict['epfilter'] JSON tag such as: <category>family</category> <category>movie</category> <category>news</category> <category>sports</category> <category>talk</category>

When switching xdetails between true or false it ends-up removing all "unecessary" ShowCache files when set to false. This lead to a full re-downloading of the ShowCache when resetting to true. The following patch aims to solve this by generating a showList using a dedicated function genShowList() so that only irrelevant files gets cleaned-up instead of everything. This ensure to keep found ShowCache files between updates that are part of the TV guide.

cast, premere, new, live, finale extra description details is now entirely managed through regular xmltv tags fully compatible with both TVH and Kodi. As such there is no more need to keep a duplicate of the values.

th0ma7 · 2021-06-03T01:43:34Z

@edit4ever I believe this PR is good for a review as you see fit. Honestly I never been able to get such a good result until I found your EPG grabber! And now with my proposed additions I get all I ever hoped from any TV guide.

Hopefully you'll consider merging the various patches proposed part of this PR. I can provide additional info as you see fit further explaining the changes I've done.

Ran such as:

$ python3 zap2epg.py

I was able to achieve the following results (see attached images)

Using the following configuration:

<settings version="2">
  <setting id="zipcode">***-CAD-Postal-Code-***</setting>
  <setting id="lineupcode">lineupId</setting>
  <setting id="lineup">Local Over the Air Broadcast</setting>
  <setting id="device">-</setting>
  <setting id="days">14</setting>
  <setting id="redays">1</setting>
  <setting id="slist"></setting>
  <setting id="stitle">false</setting>
  <setting id="xdetails">true</setting>
  <setting id="xdesc">true</setting>
  <setting id="epgenre">3</setting>
  <setting id="epicon">1</setting>
  <setting id="tvhoff">false</setting>
  <setting id="usern">admin</setting>
  <setting id="passw">*********</setting>
  <setting id="tvhurl">192.168.***.***</setting>
  <setting id="tvhport">9981</setting>
  <setting id="tvhmatch">false</setting>
  <setting id="chmatch">true</setting>
  <setting id="desc01">10</setting>
  <setting id="desc02">18</setting>
  <setting id="desc03">6</setting>
  <setting id="desc04">13</setting>
  <setting id="desc05">6</setting>
  <setting id="desc06">15</setting>
  <setting id="desc07">2</setting>
  <setting id="desc08">9</setting>
  <setting id="desc09">2</setting>
  <setting id="desc10">14</setting>
  <setting id="desc11">6</setting>
  <setting id="desc12">11</setting>
  <setting id="desc13">6</setting>
  <setting id="desc14">12</setting>
  <setting id="desc15">6</setting>
  <setting id="desc16">20</setting>
  <setting id="desc17">6</setting>
  <setting id="desc18">19</setting>
  <setting id="desc19">0</setting>
  <setting id="desc20">0</setting>
</settings>

Where movie from guide used as example is as follow in xmltv:

        <programme start="20210603010000 0000" stop="20210603030600 0000" channel="63040.zap2epg">
                <episode-num system="dd_progid">MV00047402.0000</episode-num>
                <title lang="en">Témoin sous surveillance</title>
                <desc lang="en">Un policier se cache parmi la communauté amish afin de protéger un enfant qui a été témoin d&apos;un meurtre.
13+ | 1985 </desc>
                <length units="minutes">126</length>
                <date>1985</date>
                <icon src="https://zap2it.tmsimg.com/assets/p8671_b_h9_at.jpg" />
                <previously-shown  />
                <rating>
                        <value>13+</value>
                </rating>
                <credits>
                        <actor role="Det. Capt. John Book" src="https://zap2it.tmsimg.com/assets/25704_v9_bb">Harrison Ford</actor>
                        <actor role="Rachel Lapp" src="https://zap2it.tmsimg.com/assets/1133_v9_bb">Kelly McGillis</actor>
                        <actor role="Daniel Hochleitner" src="https://zap2it.tmsimg.com/assets/3384_v9_ba">Alexander Godunov</actor>
                        <actor role="Chief Paul Schaeffer" src="https://zap2it.tmsimg.com/assets/34191_v9_ba">Josef Sommer</actor>
                        <actor role="Samuel Lapp" src="https://zap2it.tmsimg.com/assets/81259_v9_bb">Lukas Haas</actor>
                        <actor role="Eli Lapp" src="https://zap2it.tmsimg.com/assets/29513_v9_ba">Jan Rubes</actor>
                        <actor role="Det. Lt. James McFee" src="https://zap2it.tmsimg.com/assets/650_v9_ba">Danny Glover</actor>
                        <actor role="Elaine" src="https://zap2it.tmsimg.com/assets/79320_v9_bb">Patti LuPone</actor>
                        <actor role="Moses Hochleitner" src="https://zap2it.tmsimg.com/assets/71706_v9_bb">Viggo Mortensen</actor>
                </credits>
                <category lang="en">Movie</category>
                <category lang="en">Crime drama</category>
        </programme>

th0ma7 · 2021-06-08T00:30:12Z

@edit4ever while trying to figure the easiest way to integrate your project I ended looking into the option of an embed-python script. This allowed merging zap2epg.py into tv_grab_zap2epg creating a unique TV grabber script. I've been testing it since a few days and it runs really well directly through TVH.

From there I used TVH default epggrab directory tree and moved the configuration file (now also renamed to zap2epg.xml for clarity), cache and logs under it such as:

$ tree epggrab
epggrab
├── cache
│   ├── ...
│   ├── SH03890063.json
│   ├── SH03891010.json
│   └── xmltv.xml
├── conf
│   └── zap2epg.xml
└── log
    └── zap2epg.log

What's in theory left to do would be:

to add a bit of code to manage the BaseDir variable if ran on raspi or synology which should be relatively trivial to do using uname -a output or similar.
Adjust the Kodi part to use the new config file name and directory structure
On my end, simply add the tv_grab_zap2epg and zap2epg.xml file to the SynoCommunity tvheadend package.

Again, trying to be as transparent as possible and documenting the changes I'm proposing. I will happily work things through with you if you are interested in order to get this merged. Otherwise just let me know as I can simply fork & rename on my side and resync as needed. Let me know what you see fit.

Again, thnx for you great project, cheers!

edit4ever · 2021-06-09T16:48:52Z

Thank you for these updates - I have been buried in a big project and still have some work to do on it. As soon as I get a chance, I will test and merge these PRs.

I appreciate the effort as I don't have much time to support this project lately.

Idea here is to set a default path being for Raspberry Pi and an alternate when detected running on a Synology NAS where `uname -a` Linux th0ma7-nas 4.4.59+ #25426 SMP PREEMPT \ Mon Dec 14 18:48:50 CST 2020 x86_64 GNU/Linux \ synology_apollolake_918+ Note: The default RasbPi path may need adjusting but I can't test.

th0ma7 · 2021-06-12T12:17:45Z

@edit4ever this is great! Let me know if I can do anything to help or if you'd like to chat to see what's may need additional adjustments.

Note that I can't test the Kodi part and haven't touched it at all.
So it may be currently broken due to path and filename change originally pointing to settings.xml

Script now allow all regular `baseline` capabilities from the XMLTV standard (http://wiki.xmltv.org/index.php/XmltvCapabilities) As a result, now added the following new options: --quiet --output FILENAME --days X --offset X --config-file FILENAME

th0ma7 · 2021-06-13T12:34:58Z

Honestly the code base is really good as it allowed me to easily add multiple functionalities. It already achieve above and beyond all of my expectations and much more than any other solution I tried previously. Also, with the latest additions it now makes the grabber to follow xmltv capabilities guidelines: http://wiki.xmltv.org/index.php/XmltvCapabilities

I still have a few ideas but not sure how much more mileage I'm willing to do for now. My focus is now more towards getting this merged in a form or another so I can start looking into integrating in the SynoCommunity tvheadend package and update the package repository.

Ideas for a later time:

Add a configuration setting for "credits"
Rework the code so it can run without any configuration file
Migrate the channel name exception variable to the configuration file
Add french channels exception variable in configuration channel in order to tag 'lang=fr' to title and descriptions as needed
Add grabber extra capabilities such as --extradetails, --extradescription, --credits.

Here is an example of the TVH configuration using extra arguments field:

th0ma7 · 2021-06-13T13:20:47Z

I was able to address http connection to TVH with commit da57f1d solving issue #36

It works both anonymously and with username/password:

spksrc@spksrc-debian10:~/script.module.zap2epg$ head -20 epggrab/log/zap2epg.log 
2021/06/13 13:18:45 Running zap2epg-3.0
2021/06/13 13:18:45 Kodi settings version: 2
2021/06/13 13:18:45 	Country: Canada [CAN]
2021/06/13 13:18:45 	Postal code: J3B1M4
2021/06/13 13:18:45 	TV Guide duration: 14 days
2021/06/13 13:18:45 	TV Guide Start: 1623589200
2021/06/13 13:18:45 	Lineup: Local Over the Air Broadcast
2021/06/13 13:18:45 	Configuration file: /home/spksrc/script.module.zap2epg/epggrab/conf/zap2epg.xml
2021/06/13 13:18:45 	Caching directory path: /home/spksrc/script.module.zap2epg/epggrab/cache
2021/06/13 13:18:45 No channel list found - adding all stations!
2021/06/13 13:18:45 Tvheadend anonymous access...
2021/06/13 13:18:45 Starting new HTTP connection (1): 192.168.80.10:9981
2021/06/13 13:18:45 http://192.168.80.10:9981 "GET /api/channel/grid?all=1&limit=999999999&sort=name&filter=%5B%7B%22type%22:%22boolean%22,%22value%22:true,%22field%22:%22enabled%22%7D%5D HTTP/1.1" 200 2257
2021/06/13 13:18:45 Accessing Tvheadend channel list from: http://192.168.80.10:9981
2021/06/13 13:18:45 14 Tvheadend channels found...
2021/06/13 13:18:45 Checking for old cache files...
2021/06/13 13:18:45 Deleting old cache: 1623600000.json.gz
2021/06/13 13:18:45 Deleting old cache: 1623654000.json.gz
2021/06/13 13:18:45 Deleting old cache: 1623643200.json.gz
2021/06/13 13:18:45 Deleting old cache: 1623621600.json.gz

th0ma7 · 2021-06-17T11:33:42Z

I've completed the integration of the code into the "next" tvheadend package for Synology NAS.
Again, your consideration to get this work merged would be much appreciated. My synocommunity PR is now pretty much pending on this to confirm final download URL and version to use from your github source (currently pointing to mine through this PR temporarily).

Also, I noticed the following error when ran anonymously on 127.0.0.1.
The error was cause by restricting access to local LAN only for * user (thus loopback wasn't authorized).
The JSON query failed such as:

2021/06/17 07:17:15 Tvheadend anonymous access...
2021/06/17 07:17:15 Starting new HTTP connection (1): 127.0.0.1:9981
2021/06/17 07:17:15 http://127.0.0.1:9981 "GET /api/channel/grid?all=1&limit=999999999&sort=name&filter=%5B%7B%22type%22:%22boolean%22,%22value%22:true,%22field%22:%22enabled%22%7D%5D HTTP/1.1" 401 250
2021/06/17 07:17:15 Accessing Tvheadend channel list from: http://127.0.0.1:9981
2021/06/17 07:17:15 Exception: main
Traceback (most recent call last):
  File "<string>", line 794, in mainRun
  File "<string>", line 130, in tvhMatchGet
  File "/volume1/@appstore/tvheadend/env/lib/python3.8/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/var/packages/python38/target/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/var/packages/python38/target/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/var/packages/python38/target/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm guessing that a better error capturing would be helpful to provide user with appropriate message such as Unable to access URL or similar.

edit4ever · 2021-07-01T18:33:53Z

merging zap2epg.py and the grabber breaks the kodi addon.

If you want to go down this route - then it should be it's own branch or fork.

Some of your updates/fixes would be good to integrate into the kodi version - but that merge isn't one of them.

Might I suggest you spin it off as a standalone grabber for TVH - my version was really designed for a beginner with a kodi setup and interface for simplicity. I can see value in your version as a standalone - but it really is a different animal because it assumes the user is knowledgeable and working with the command line for setup.

Refers to issue: edit4ever#34 (comment)

This allow running the script using hts user on usual TVH docker images.

th0ma7 · 2021-07-02T11:51:46Z

@edit4ever

merging zap2epg.py and the grabber breaks the kodi addon.

It made so much sense to me to merge them to ease the integration on other platforms and allow supporting greater xmltv capabilities. But sadly I have no way to test it out on RaspberryPi. There must be a qemu/vm that can do the trick to which I could then help out as you see fit.

If you want to go down this route - then it should be it's own branch or fork.

Might I suggest you spin it off as a standalone grabber for TVH - my version was really designed for a beginner with a kodi setup and interface for simplicity. I can see value in your version as a standalone - but it really is a different animal because it assumes the user is knowledgeable and working with the command line for setup.

Fine by me but it was really much hoping not to. To me its just a matter of well describing the howto and that should do it.

Some of your updates/fixes would be good to integrate into the kodi version - but that merge isn't one of them.

Sad of having a duplicate project sharing 99% of the code and now diverting. I would much rather giving you a hand on fixing any remaining issues instead but I totally understand your disinterest of doing so.

Code is resuming entirely starting from this PR: edit4ever/script.module.zap2epg#37 This is a fork from edit4ever Python3 branch of script.module.zap2epg Reusing same LICENSE as original.

th0ma7 · 2021-07-02T12:46:38Z

Closing the pull request.
Let me know if you change your mind.
In the meantime closing and created a new fork at https://github.com/th0ma7/tv_grab_zap2epg

edit4ever · 2021-07-02T13:25:40Z

I'm certainly not against merging - just not sure I have the time t sort it out right now. Also I want to make sure the Kodi addon works as a basic setup with GUI for people who don't work in the linux backend.

If you're interested in trying to see how both can work together - the easiest test system is using VMware and running LibreELEC. You can download the latest ova file from the repo below to install as a VM:

https://test.libreelec.tv/

The biggest challenge I see is that Kodi addons tend to work as python modules - and your merged grabber is no longer seen as a python file. The default.py file is what the zap2epg addon runs when selected in kodi - so it may just be adapting that to work with the merged file.

Thanks again for your work - it's definitely worth merging a lot of your tweaks even if we can't sort the merged file idea.

th0ma7 added 4 commits June 1, 2021 00:31

zap2epg: Convert escape chars to plain html where neeeded

56e63f1

zap2epg: Manage teletext, last-chance and premiere tags

f71c845

zap2epg: Add length tag

d097291

zap2epg: Add channel name to display-name tag

ccfc9e5

th0ma7 mentioned this pull request Jun 1, 2021

Convert to using python3 & force HTML for escape characters #35

Closed

5 tasks

th0ma7 added 6 commits June 1, 2021 11:31

zap2epg: Discard <> from safeTitle as managed through convHTML()

765e64d

zap2epg: Add <credits> tags with corresponding fields

3daa50f

zap2epg: Remove cast, premere, new, live, finale

6b9fdcc

cast, premere, new, live, finale extra description details is now entirely managed through regular xmltv tags fully compatible with both TVH and Kodi. As such there is no more need to keep a duplicate of the values.

zap2epg: Remove non-essential english info to be bilingual

c72dc82

th0ma7 mentioned this pull request Jun 3, 2021

Add <credits tags #39

Closed

th0ma7 added 3 commits June 7, 2021 21:20

zap2epg: Depricated API: logging.warn -> logging.warning

4f70b2b

zap2epg: Convert to embed-python as standalone TV grabber

1337e2f

zap2epg: Update project description to provide extra details

0fb8a11

th0ma7 added 2 commits June 12, 2021 12:01

zap2epg: Rename "First Aired" to "Premiere" being bilingual fr|en

3198a9d

th0ma7 added 5 commits June 12, 2021 17:22

zap2epg: Add zipcode to list of available arguments

0822f03

zap2epg: Document tv_grab_zap2epg capabilities

c2c5ece

zap2epg: Readme layout adjustments

1b07696

zap2ept: Small fixes for various options and document usage on DSM

cb177bf

th0ma7 added 2 commits June 13, 2021 13:13

zap2epg: Fix TVH login (issue edit4ever#36)

da57f1d

zap2epg: gridtime is float, convert to int first to remove trailing .0

288289c

This was referenced Jun 13, 2021

Add requests to python3 package SynoCommunity/spksrc#4678

Closed

[WIP] Add zap2epg TV grabber to tvheadend SynoCommunity/spksrc#4680

Merged

zap2epg: Set tvhmatch to true by default

65336f7

th0ma7 mentioned this pull request Jun 30, 2021

Direct import to tvheadend? #34

Closed

th0ma7 added 4 commits July 1, 2021 20:48

zap2epg: Convert channel name to plain HTML

8fe90c5

Refers to issue: edit4ever#34 (comment)

zap2epg: Allow running under $HOME/zap2epg using default hts user

3ad0e4f

This allow running the script using hts user on usual TVH docker images.

zap2epg: Change description to match other similar in TVH

af8c85e

zap2epg: Redirect id errors to /dev/null

9830e30

th0ma7 closed this Jul 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python3 th0ma7 updates #37

Python3 th0ma7 updates #37

th0ma7 commented Jun 1, 2021 •

edited

Loading

th0ma7 commented Jun 3, 2021

th0ma7 commented Jun 8, 2021

edit4ever commented Jun 9, 2021

th0ma7 commented Jun 12, 2021

th0ma7 commented Jun 13, 2021

th0ma7 commented Jun 13, 2021

th0ma7 commented Jun 17, 2021

edit4ever commented Jul 1, 2021

th0ma7 commented Jul 2, 2021

th0ma7 commented Jul 2, 2021

edit4ever commented Jul 2, 2021

Python3 th0ma7 updates #37

Python3 th0ma7 updates #37

Conversation

th0ma7 commented Jun 1, 2021 • edited Loading

Other improvements

Notes

th0ma7 commented Jun 3, 2021

th0ma7 commented Jun 8, 2021

edit4ever commented Jun 9, 2021

th0ma7 commented Jun 12, 2021

th0ma7 commented Jun 13, 2021

th0ma7 commented Jun 13, 2021

th0ma7 commented Jun 17, 2021

edit4ever commented Jul 1, 2021

th0ma7 commented Jul 2, 2021

th0ma7 commented Jul 2, 2021

edit4ever commented Jul 2, 2021

th0ma7 commented Jun 1, 2021 •

edited

Loading