Bug #987
closedonly imports a small number of available entries
Added by oldmanuk over 12 years ago. Updated over 12 years ago.
100%
Description
For some reason xmltv2vdr is only importing a very small number of the available EPG entries from my xmltv file
I have a file-based grabber called 'echo' that imports a previously prepared file called 'echo.xmltv' (acquired via tv_grab_uk_rt). I have numerous channel mappings configured, but for some reason xmltv2vdr only imports 2MB out of the available 60MB of data.
Is it related to the generated 'weak' eventid? Is that causing events to be considered duplicates?
All config files and sample xmltv are uploaded here:
http://dl.dropbox.com/u/951543/bug.tar.bz2
Updated by oldmanuk over 12 years ago
by contrast, here is a copy of the 60MB epg.data generated after running ./xmltv2vdr.pl (perl script) against the same xmltv file (although 1 day later):
Updated by Joe_D over 12 years ago
- Assignee set to Joe_D
- % Done changed from 0 to 20
I need your channels.conf, please!
Updated by oldmanuk over 12 years ago
Sure. Here you go: https://gist.github.com/2730367
Updated by Joe_D over 12 years ago
Please try the latest GIT
you have errors in your xmltv-file (checked with tv_validate_file):
file contains unexpected control characters, misencoded windows-1252?
look here "and Do Nothing Til You Hear fr"
No programme entries found for showcase.movies.sky.com
No programme entries found for chsuk.tv
No programme entries found for hd.1.boxoffice.sky.com
No programme entries found for hd.2.boxoffice.sky.com
No programme entries found for sonytv.com
No programme entries found for hd.showcase.movies.sky.com
No programme entries found for tsod.plus-1.sonytv.com
No programme entries found for bbc3.interactive.sport.bbc.co.uk
2 errors found.
Updated by oldmanuk over 12 years ago
Yeah I saw the errors on this latest xmltv but didn't think it was an issue. I've removed those channels that have no data from my config and will raise a defect against tv_grab_uk_rt for the unexpected control characters.
Will grab latest git and give it a go shortly.
Updated by oldmanuk over 12 years ago
Thanks for looking at this.
Definite improvement with fba8c36e
39M epg.data this time, but still some large gaps in the EPG
xmltv2vdr.log and epg.data attached http://dl.dropbox.com/u/951543/bug2.tar.bz2
Updated by Joe_D over 12 years ago
- % Done changed from 20 to 40
Please pay attention to the log and errors:
May 19 17:32:06 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid 2.jr.nickelodeon.co.uk
May 19 17:32:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid animalplanet.discoveryeurope.com
May 19 17:32:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid attheraces.co.uk
May 19 17:32:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid b4utv.com
May 19 17:32:20 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bbc1.interactive.sport.bbc.co.uk
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bbc2.interactive.sport.bbc.co.uk
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bbc3.interactive.sport.bbc.co.uk
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bestdirect.tv
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bid.tv
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid blighty.uktv.co.uk
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bliss.co.uk
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bloomberg.com
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid boomerang.cartoonnetwork.com
May 19 17:32:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cartoonito.cartoonnetwork.com
May 19 17:32:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cbbc.bbc.co.uk
May 19 17:32:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cbeebies.bbc.co.uk
May 19 17:32:33 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid chartshow.com
May 19 17:32:34 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cinemagic.disney.co.uk
May 19 17:32:34 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cinemoi.tv
May 19 17:32:34 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid citv.itv.co.uk
May 19 17:32:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid communitychannel.org
May 19 17:32:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid crimeandinvestigation.co.uk
May 19 17:32:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid crime.movies.sky.com
May 19 17:32:39 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid discoveryeurope.com
May 19 17:32:39 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid disneychannel.disney.co.uk
May 19 17:32:42 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid eden.uktv.co.uk
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid essentialflava.com
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid euronews.com
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid europe.cnbc.com
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid europe.cnn.com
May 19 17:32:45 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid extra.gemstv.co.uk
May 19 17:32:45 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid extreme.com
May 19 17:32:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid gemstv.co.uk
May 19 17:32:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid god.tv
May 19 17:32:54 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.animalplanet.discoveryeurope.com
May 19 17:32:56 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.cartoonnetwork.com
May 19 17:32:58 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.cinemagic.disney.co.uk
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.crimeandinvestigation.co.uk
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.crime.movies.sky.com
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.dave.uktv.co.uk
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.discoveryeurope.com
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.disneychannel.disney.co.uk
May 19 17:33:00 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.eden.uktv.co.uk
May 19 17:33:01 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.f1.sports.sky.com
May 19 17:33:06 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.nationalgeographic.co.uk
May 19 17:33:07 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.news.sports.sky.com
May 19 17:33:07 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.nickelodeon.co.uk
May 19 17:33:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.thehistorychannel.co.uk
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.watch.uktv.co.uk
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.wild.nationalgeographic.co.uk
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.world.nhk.or.jp
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.xd.disney.co.uk
May 19 17:33:11 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid history.discoveryeurope.com
May 19 17:33:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid horseandcountry.tv
May 19 17:33:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid investigation.discoveryeurope.com
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid jr.nickelodeon.co.uk
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid junior.disney.co.uk
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid kerrang.com
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid kiss-tv.kiss100.com
May 19 17:33:17 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid magictv.co.uk
May 19 17:33:17 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid military.thehistorychannel.co.uk
May 19 17:33:18 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid motors.tv
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid mutv.manutd.com
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid mydancenation.com
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid mystyle.com
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid nationalgeographic.co.uk
May 19 17:33:21 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid nickelodeon.co.uk
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid phoenixcne.com
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid popfun.co.uk
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid price-drop.tv
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid q4music.com
May 19 17:33:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid racinguk.tv
May 19 17:33:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid really.uktv.co.uk
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid sat.travelchannel.co.uk
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid science.discoveryeurope.com
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid scuzz.tv
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid setasia.tv
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid shed.discoveryeurope.com
May 19 17:33:30 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid smashhits.net
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid star-news.startv.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid star-plus.startv.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid supercasino.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid theactivechannel.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid thebox.co.uk
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid thehistorychannel.co.uk
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tinypop.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid too.cartoonnetwork.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid toons.nickelodeon.co.uk
May 19 17:33:38 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid travel.discoveryeurope.com
May 19 17:33:38 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.animalplanet.discoveryeurope.com
May 19 17:33:38 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.boomerang.cartoonnetwork.com
May 19 17:33:46 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.cinemagic.disney.co.uk
May 19 17:33:48 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.crimeandinvestigation.co.uk
May 19 17:33:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.discoveryeurope.com
May 19 17:33:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.disneychannel.disney.co.uk
May 19 17:33:51 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.eden.uktv.co.uk
May 19 17:33:57 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.history.discoveryeurope.com
May 19 17:33:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.junior.disney.co.uk
May 19 17:34:02 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.nationalgeographic.co.uk
May 19 17:34:02 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.nickelodeon.co.uk
May 19 17:34:04 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.sat.travelchannel.co.uk
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.science.discoveryeurope.com
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.sky-three.sky.com
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.thehistorychannel.co.uk
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.tinypop.com
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.toons.nickelodeon.co.uk
May 19 17:34:07 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.xd.disney.co.uk
May 19 17:34:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid turbo.discoveryeurope.com
May 19 17:34:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tv5.org
May 19 17:34:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid uk.cartoonnetwork.com
May 19 17:34:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid weddingtv.com
May 19 17:34:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid wild.nationalgeographic.co.uk
May 19 17:34:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid xd.disney.co.uk
May 19 17:34:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid zeeuk.com
May 19 17:34:14 [9544] xmltv2vdr: 'echo' skipped 50574 xmltv events
May 19 17:34:14 [9544] xmltv2vdr: 'echo' processed 3463 xmltv events - see ERRORs above!
May 19 17:34:16 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid S28.2E-2-2057-7715
May 19 17:34:16 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid S28.2E-2-2057-7710
May 19 17:34:18 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid S28.2E-2-2401-5104
May 19 17:34:18 [9544] xmltv2vdr: 'echo' ERROR channel -0-0-0 not found in channels.conf
May 19 17:34:18 [9544] xmltv2vdr: 'echo' processed 7013 vdr events - see ERRORs above!
Try to fix the mapping errors in the plugin setup menu.
Updated by oldmanuk over 12 years ago
yeah, sorry I meant there were still gaps in the channels that I had setup mappings for (e.g. Channel 4).
As its quite time consuming setting up all the mappings, I hadn't bothered to do them for all channels yet. I'll either remove these channels from my grabber or go through and map them. I'll report back once I've done this.
Updated by oldmanuk over 12 years ago
OK, so I tirelessly went through and mapped all the channels, stopped VDR, deleted both epg.data and epg.db and started vdr
As before, the EPG was partially imported, but large gaps still exist on latest git.
e.g.,
This should appear at 06:00 until 09:15 on Tue 22nd:
<programme start="20120522060000 +0100" stop="20120522091500 +0100" channel="bbc1.bbc.co.uk"> <title>Breakfast</title> <desc lang="en">Round-up of national and international news, plus the latest from the money markets.</desc> <category lang="en">News and current affairs</category> <video> <aspect>16:9</aspect> <quality>SDTV</quality> </video> <subtitles type="teletext" /> </programme>
But, it is missing from the EPG:
Here are the logs, xmltv file and epg.db again:
http://dl.dropbox.com/u/951543/bug3.tar.bz2
Updated by Joe_D over 12 years ago
- Status changed from New to Confirmed
- % Done changed from 40 to 60
Thanks for eliminating all side-errors. Please try the latest GIT, i found a bug in the internal sql index handling.
Updated by oldmanuk over 12 years ago
Many thanks! Latest git tested and working well. Issue fixed.
Great work.
Updated by oldmanuk over 12 years ago
Also, it might be worth putting out a 0.1.2 release/.tar.gz to include this fix, so that distros pick it up.
Updated by Joe_D over 12 years ago
- Status changed from Confirmed to Closed
- % Done changed from 60 to 100