Project

General

Profile

Actions
Copy link

Bug #987

closed

only imports a small number of available entries

Added by oldmanuk over 12 years ago. Updated over 12 years ago.

Status:
Closed
Priority:
High
Assignee:
Joe_D
Target version:
-
Start date:
05/18/2012
Due date:
% Done:

100%

Estimated time:

Description

For some reason xmltv2vdr is only importing a very small number of the available EPG entries from my xmltv file

I have a file-based grabber called 'echo' that imports a previously prepared file called 'echo.xmltv' (acquired via tv_grab_uk_rt). I have numerous channel mappings configured, but for some reason xmltv2vdr only imports 2MB out of the available 60MB of data.

Is it related to the generated 'weak' eventid? Is that causing events to be considered duplicates?

All config files and sample xmltv are uploaded here:
http://dl.dropbox.com/u/951543/bug.tar.bz2

Actions
Copy link
 #1

Updated by oldmanuk over 12 years ago

p.s., this is on latest git b05fade92

Actions
Copy link
 #2

Updated by oldmanuk over 12 years ago

by contrast, here is a copy of the 60MB epg.data generated after running ./xmltv2vdr.pl (perl script) against the same xmltv file (although 1 day later):

http://dl.dropbox.com/u/951543/epg.data.bz2

Actions
Copy link
 #3

Updated by Joe_D over 12 years ago

  • Assignee set to Joe_D
  • % Done changed from 0 to 20

I need your channels.conf, please!

Actions
Copy link
 #4

Updated by oldmanuk over 12 years ago

Actions
Copy link
 #5

Updated by Joe_D over 12 years ago

Please try the latest GIT

you have errors in your xmltv-file (checked with tv_validate_file):

file contains unexpected control characters, misencoded windows-1252?
look here "and Do Nothing Til You Hear fr"
No programme entries found for showcase.movies.sky.com
No programme entries found for chsuk.tv
No programme entries found for hd.1.boxoffice.sky.com
No programme entries found for hd.2.boxoffice.sky.com
No programme entries found for sonytv.com
No programme entries found for hd.showcase.movies.sky.com
No programme entries found for tsod.plus-1.sonytv.com
No programme entries found for bbc3.interactive.sport.bbc.co.uk
2 errors found.

Actions
Copy link
 #6

Updated by oldmanuk over 12 years ago

Yeah I saw the errors on this latest xmltv but didn't think it was an issue. I've removed those channels that have no data from my config and will raise a defect against tv_grab_uk_rt for the unexpected control characters.

Will grab latest git and give it a go shortly.

Actions
Copy link
 #7

Updated by oldmanuk over 12 years ago

Thanks for looking at this.

Definite improvement with fba8c36e

39M epg.data this time, but still some large gaps in the EPG

xmltv2vdr.log and epg.data attached http://dl.dropbox.com/u/951543/bug2.tar.bz2

Actions
Copy link
 #8

Updated by Joe_D over 12 years ago

  • % Done changed from 20 to 40

Please pay attention to the log and errors:

May 19 17:32:06 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid 2.jr.nickelodeon.co.uk
May 19 17:32:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid animalplanet.discoveryeurope.com
May 19 17:32:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid attheraces.co.uk
May 19 17:32:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid b4utv.com
May 19 17:32:20 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bbc1.interactive.sport.bbc.co.uk
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bbc2.interactive.sport.bbc.co.uk
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bbc3.interactive.sport.bbc.co.uk
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bestdirect.tv
May 19 17:32:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bid.tv
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid blighty.uktv.co.uk
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bliss.co.uk
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid bloomberg.com
May 19 17:32:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid boomerang.cartoonnetwork.com
May 19 17:32:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cartoonito.cartoonnetwork.com
May 19 17:32:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cbbc.bbc.co.uk
May 19 17:32:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cbeebies.bbc.co.uk
May 19 17:32:33 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid chartshow.com
May 19 17:32:34 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cinemagic.disney.co.uk
May 19 17:32:34 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid cinemoi.tv
May 19 17:32:34 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid citv.itv.co.uk
May 19 17:32:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid communitychannel.org
May 19 17:32:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid crimeandinvestigation.co.uk
May 19 17:32:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid crime.movies.sky.com
May 19 17:32:39 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid discoveryeurope.com
May 19 17:32:39 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid disneychannel.disney.co.uk
May 19 17:32:42 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid eden.uktv.co.uk
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid essentialflava.com
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid euronews.com
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid europe.cnbc.com
May 19 17:32:44 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid europe.cnn.com
May 19 17:32:45 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid extra.gemstv.co.uk
May 19 17:32:45 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid extreme.com
May 19 17:32:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid gemstv.co.uk
May 19 17:32:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid god.tv
May 19 17:32:54 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.animalplanet.discoveryeurope.com
May 19 17:32:56 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.cartoonnetwork.com
May 19 17:32:58 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.cinemagic.disney.co.uk
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.crimeandinvestigation.co.uk
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.crime.movies.sky.com
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.dave.uktv.co.uk
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.discoveryeurope.com
May 19 17:32:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.disneychannel.disney.co.uk
May 19 17:33:00 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.eden.uktv.co.uk
May 19 17:33:01 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.f1.sports.sky.com
May 19 17:33:06 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.nationalgeographic.co.uk
May 19 17:33:07 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.news.sports.sky.com
May 19 17:33:07 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.nickelodeon.co.uk
May 19 17:33:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.thehistorychannel.co.uk
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.watch.uktv.co.uk
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.wild.nationalgeographic.co.uk
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.world.nhk.or.jp
May 19 17:33:10 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid hd.xd.disney.co.uk
May 19 17:33:11 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid history.discoveryeurope.com
May 19 17:33:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid horseandcountry.tv
May 19 17:33:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid investigation.discoveryeurope.com
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid jr.nickelodeon.co.uk
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid junior.disney.co.uk
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid kerrang.com
May 19 17:33:15 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid kiss-tv.kiss100.com
May 19 17:33:17 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid magictv.co.uk
May 19 17:33:17 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid military.thehistorychannel.co.uk
May 19 17:33:18 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid motors.tv
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid mutv.manutd.com
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid mydancenation.com
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid mystyle.com
May 19 17:33:19 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid nationalgeographic.co.uk
May 19 17:33:21 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid nickelodeon.co.uk
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid phoenixcne.com
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid popfun.co.uk
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid price-drop.tv
May 19 17:33:22 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid q4music.com
May 19 17:33:23 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid racinguk.tv
May 19 17:33:24 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid really.uktv.co.uk
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid sat.travelchannel.co.uk
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid science.discoveryeurope.com
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid scuzz.tv
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid setasia.tv
May 19 17:33:25 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid shed.discoveryeurope.com
May 19 17:33:30 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid smashhits.net
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid star-news.startv.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid star-plus.startv.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid supercasino.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid theactivechannel.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid thebox.co.uk
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid thehistorychannel.co.uk
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tinypop.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid too.cartoonnetwork.com
May 19 17:33:37 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid toons.nickelodeon.co.uk
May 19 17:33:38 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid travel.discoveryeurope.com
May 19 17:33:38 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.animalplanet.discoveryeurope.com
May 19 17:33:38 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.boomerang.cartoonnetwork.com
May 19 17:33:46 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.cinemagic.disney.co.uk
May 19 17:33:48 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.crimeandinvestigation.co.uk
May 19 17:33:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.discoveryeurope.com
May 19 17:33:49 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.disneychannel.disney.co.uk
May 19 17:33:51 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.eden.uktv.co.uk
May 19 17:33:57 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.history.discoveryeurope.com
May 19 17:33:59 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.junior.disney.co.uk
May 19 17:34:02 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.nationalgeographic.co.uk
May 19 17:34:02 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.nickelodeon.co.uk
May 19 17:34:04 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.sat.travelchannel.co.uk
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.science.discoveryeurope.com
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.sky-three.sky.com
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.thehistorychannel.co.uk
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.tinypop.com
May 19 17:34:05 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.toons.nickelodeon.co.uk
May 19 17:34:07 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tsod.plus-1.xd.disney.co.uk
May 19 17:34:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid turbo.discoveryeurope.com
May 19 17:34:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid tv5.org
May 19 17:34:08 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid uk.cartoonnetwork.com
May 19 17:34:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid weddingtv.com
May 19 17:34:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid wild.nationalgeographic.co.uk
May 19 17:34:12 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid xd.disney.co.uk
May 19 17:34:13 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid zeeuk.com
May 19 17:34:14 [9544] xmltv2vdr: 'echo' skipped 50574 xmltv events
May 19 17:34:14 [9544] xmltv2vdr: 'echo' processed 3463 xmltv events - see ERRORs above!
May 19 17:34:16 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid S28.2E-2-2057-7715
May 19 17:34:16 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid S28.2E-2-2057-7710
May 19 17:34:18 [9544] xmltv2vdr: 'echo' ERROR no mapping for channelid S28.2E-2-2401-5104
May 19 17:34:18 [9544] xmltv2vdr: 'echo' ERROR channel -0-0-0 not found in channels.conf
May 19 17:34:18 [9544] xmltv2vdr: 'echo' processed 7013 vdr events - see ERRORs above!

Try to fix the mapping errors in the plugin setup menu.

Actions
Copy link
 #9

Updated by oldmanuk over 12 years ago

yeah, sorry I meant there were still gaps in the channels that I had setup mappings for (e.g. Channel 4).

As its quite time consuming setting up all the mappings, I hadn't bothered to do them for all channels yet. I'll either remove these channels from my grabber or go through and map them. I'll report back once I've done this.

Actions
Copy link
 #10

Updated by oldmanuk over 12 years ago

OK, so I tirelessly went through and mapped all the channels, stopped VDR, deleted both epg.data and epg.db and started vdr

As before, the EPG was partially imported, but large gaps still exist on latest git.

e.g.,

This should appear at 06:00 until 09:15 on Tue 22nd:

  <programme start="20120522060000 +0100" stop="20120522091500 +0100" channel="bbc1.bbc.co.uk">
    <title>Breakfast</title>
    <desc lang="en">Round-up of national and international news, plus the latest from the money markets.</desc>
    <category lang="en">News and current affairs</category>
    <video>
      <aspect>16:9</aspect>
      <quality>SDTV</quality>
    </video>
    <subtitles type="teletext" />
  </programme>

But, it is missing from the EPG:

Here are the logs, xmltv file and epg.db again:
http://dl.dropbox.com/u/951543/bug3.tar.bz2

Actions
Copy link
 #11

Updated by Joe_D over 12 years ago

  • Status changed from New to Confirmed
  • % Done changed from 40 to 60

Thanks for eliminating all side-errors. Please try the latest GIT, i found a bug in the internal sql index handling.

Actions
Copy link
 #12

Updated by oldmanuk over 12 years ago

Many thanks! Latest git tested and working well. Issue fixed.

Great work.

Actions
Copy link
 #13

Updated by oldmanuk over 12 years ago

Also, it might be worth putting out a 0.1.2 release/.tar.gz to include this fix, so that distros pick it up.

Actions
Copy link
 #14

Updated by Joe_D over 12 years ago

  • Status changed from Confirmed to Closed
  • % Done changed from 60 to 100
Actions
Copy link

Also available in: Atom PDF