Bug #1634
closedeepg crash when scanning Sky UK EPG
100%
Description
I've been recording an hour of the "EPG Background Audio" channel as a way of getting the 7 day Sky UK EPG merged into VDR. In doing this, I notice regular segfaults:
Core was generated by `/home/nickb/src/vdr/vdr --config=/tv/vdr/config --cachedir=/tv/vdr/cache --vide'. Program terminated with signal 11, Segmentation fault. #0 sky_huffman_decode ( Data=Data@entry=0x7fcf8e7fae47 "\004uк\021=\033A\214\067\023\307\364", <incomplete sequence \360>, Length=14, DecodeText=DecodeText@entry=0x7fcf8e7f9a20 "") at eepg.c:651 651 sNodeH *nH, H=(Format==SKY_IT)?*sky_tables[0]:*sky_tables[1]; (gdb) list 646 return NULL; 647 } 648 649 int sky_huffman_decode (const u_char * Data, int Length, unsigned char *DecodeText) 650 { 651 sNodeH *nH, H=(Format==SKY_IT)?*sky_tables[0]:*sky_tables[1]; 652 int i; 653 int p; 654 int q; 655 bool CodeError; (gdb) print sky_tables[0] $1 = (sNodeH *) 0x0 (gdb) print sky_tables[1] $2 = (sNodeH *) 0x0 (gdb) quit root@satellite:/#
So it looks like the sky_tables array is being initialised too late here. My suggested fix is to initialise the sky_tables at the point in time that the filter is added, just like for Freeview:
diff --git a/eepg.c b/eepg.c index 6bf997a..5b483ad 100644 --- a/eepg.c +++ b/eepg.c @@ -2687,10 +2687,11 @@ void cFilterEEPG::ProcessNextFormat (bool FirstTime = false) case MHW2: AddFilter (0x231, 0xc8); //MHW2 Channels & Themes break; case SKY_IT: case SKY_UK: + InitDictionary (); AddFilter (0x11, 0x4a); //Sky Channels break; case FREEVIEW: //Freeview, CONT mode //TODO streamline this for other modes InitDictionary (); AddFilter (pid, 0x4e, 0xfe); //event info, actual(0x4e)/other(0x4f) TS, present/following
With this patch in place, I no longer have regular crashes.
Files
Updated by dimeptr about 11 years ago
You do not have to record the channel to get the data into VDR.
I will test your patch in the recording scenario.
Updated by dimeptr about 11 years ago
- Status changed from New to Assigned
- Assignee set to dimeptr
Updated by cheesemonster about 11 years ago
If I don't record the channel, or at least tune explicitly to the channel, then I do not get any EPG data because VDR seems to move on to a different transponder before the data is received. I've left it for 48 hours and never received any EPG updates other than now/next.
Updated by dimeptr about 11 years ago
The data can not be updated if you do not tune into it specifically, but you do not need to record it.
I have tested the tuning a lot of times without a crash, so I will have to test with recording.
Updated by cheesemonster about 11 years ago
Recording is an easy way to schedule the channel tune and therefore grab the EPG updates on a daily basis without having to remember to manually tune in every day myself.
Updated by dimeptr about 11 years ago
I have been recording the channel the whole day now, without a problem. Which version are you using ?
Updated by cheesemonster about 11 years ago
I am using the latest git version of eepg on vdr 2.1.2. For me the problem happens fairly consistently when doing something like recording on Sky UK and then changing channel on another DVB-S device. Pretty much always happens when viewing on one DVB-S receiver and a recording kicks off on a second DVB-S receiver.
Updated by dimeptr about 11 years ago
I have tried recording few times on two devices without any problem. Can you do a backtrace of the segfault? The problem with adding InitDictionary () where you suggest is that it will be called many times, and each time the files will be reloaded. And also I would like to get to the root of the problem, since I can not reproduce it.
Updated by cheesemonster about 11 years ago
I have to wonder why, when the tables are already embedded into the application, should there be a InitDictionary() method that reads the data, previously written from those tables, from disk? Why not just access the tables that already exist?
Here's the stack trace:
```
Core was generated by `/home/nickb/src/vdr/vdr --config=/tv/vdr/config --cachedir=/tv/vdr/cache --vide'.
Program terminated with signal 11, Segmentation fault.
#0 sky_huffman_decode (Data=Data@entry=0x7fcf8e7fae47 "\004uк\021=\033A\214\067\023\307\364", <incomplete sequence \360>, Length=14,
DecodeText=DecodeText@entry=0x7fcf8e7f9a20 "") at eepg.c:651
651 sNodeH *nH, H=(Format==SKY_IT)?*sky_tables0:*sky_tables1;
(gdb) bt
#0 sky_huffman_decode (Data=Data@entry=0x7fcf8e7fae47 "\004uк\021=\033A\214\067\023\307\364", <incomplete sequence \360>, Length=14,
DecodeText=DecodeText@entry=0x7fcf8e7f9a20 "") at eepg.c:651
#1 0x00007fcf993d5c89 in cFilterEEPG::GetTitlesSKYBOX (this=0x7fcf3c3b0010, Data=0x7fcf8e7fae30 "\242\360\316\005y\301", Length=205) at eepg.c:2287
#2 0x00007fcf993dd2e5 in cFilterEEPG::Process (this=0x7fcf3c3b0010, Pid=<optimised out>, Tid=<optimised out>, Data=0x7fcf8e7fae30 "\242\360\316\005y\301",
Length=209) at eepg.c:2926
#3 0x00000000004e583c in cSectionHandler::Action (this=0x964140) at sections.c:211
#4 0x000000000050a6cd in cThread::StartThread (Thread=0x964140) at thread.c:262
#5 0x00007fcf9b9bff8e in start_thread (arg=0x7fcf8e7fc700) at pthread_create.c:311
#6 0x00007fcf9a3e3a0d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
(gdb)
```
Updated by dimeptr about 11 years ago
Well the idea for the original makers of the plugin was that they have files in order to decipher the Huffman tables. I was also thinking if it is still needed, but it is in todo.
Back to the problem, it is not possible to get to GetTitlesSKYBOX without adding the filter AddFilter (0x30, SI::TableIdSKYTitlesA0, 0xfc);
which is after a succesfull InitDictionary ()
Updated by cheesemonster about 11 years ago
Updated by dimeptr about 11 years ago
You are right, however your call to InitDictionary() is before the call that it is now, so it should segfault anyway.
Updated by cheesemonster about 11 years ago
The only error I'm seeing in the logs is:
/var/log/syslog.1:Dec 1 20:04:28 satellite vdr: [19032] EEPG: Error, wrong format detected in GetThemesSKYBOX. Format = 6.
/var/log/syslog.1:Dec 1 20:12:53 satellite vdr: [19032] EEPG: Error, wrong format detected in GetThemesSKYBOX. Format = 6.
/var/log/syslog.1:Dec 1 20:21:15 satellite vdr: [19032] EEPG: Error, wrong format detected in GetThemesSKYBOX. Format = 6.
/var/log/syslog.1:Dec 1 20:29:40 satellite vdr: [19032] EEPG: Error, wrong format detected in GetThemesSKYBOX. Format = 6.
/var/log/syslog.1:Dec 1 20:46:30 satellite vdr: [19032] EEPG: Error, wrong format detected in GetThemesSKYBOX. Format = 6.
6 == FREEVIEW
OK, I'll comment out the InitDictionary() change and add in some logging around this stuff to see what's going on.
Updated by cheesemonster over 10 years ago
Is EEPG safe when two threads are updating at the same time?
Looking though syslog, I see that the segfaults are consistent with this circumstance of simultaneous updates from two threads.
Apr 6 11:13:26 satellite vdr: [17999] EEPG: PMT scan idle Apr 6 11:13:26 satellite vdr: [17999] EEPG: Sky UK Extended EPG detected on pid 55. Apr 6 11:13:26 satellite vdr: [17999] EEPG: Filter Pid:0x11,Tid:0x4a added. Apr 6 11:13:26 satellite vdr: [17999] | ID | Channel ID | Channel Name | FND | Sky Num. | Apr 6 11:13:26 satellite vdr: [17999] |------|----------------------------|------------------------|-----|----------| Apr 6 11:13:26 satellite vdr: [17999] | 2102 | S28.2E-2-2047-6441 | | NO | 958 | Apr 6 11:13:26 satellite vdr: [17999] | 2104 | S28.2E-2-2047-6451 | | NO | 956 | Apr 6 11:13:26 satellite vdr: [17999] | 2152 | S28.2E-2-2047-6461 | | NO | 963 | Apr 6 11:13:26 satellite vdr: [17999] | 2155 | S28.2E-2-2047-6471 | | NO | 955 | Apr 6 11:13:26 satellite vdr: [17999] | 2004 | S28.2E-2-2047-6421 | | NO | 951 | Apr 6 11:13:26 satellite vdr: [17999] | 2075 | S28.2E-2-2050-6940 | BBC Two HD | YES | 142 | Apr 6 11:13:26 satellite vdr: [17999] | 2081 | S28.2E-2-2050-6943 | BBC One NI HD | YES | 979 | Apr 6 11:13:26 satellite vdr: [17999] | 2077 | S28.2E-2-2050-6951 | BBC Three HD | YES | 210 | Apr 6 11:13:26 satellite vdr: [17999] | 2078 | S28.2E-2-2050-6952 | CBBC HD | YES | 633 | Apr 6 11:13:26 satellite vdr: [17996] EEPG: PMT scan idle Apr 6 11:13:26 satellite vdr: [17996] EEPG: Sky UK Extended EPG detected on pid 55. Apr 6 11:13:26 satellite vdr: [17999] | 5617 | S28.2E-2-2301-52100 | Travel Ch +1 | YES | 263 | Apr 6 11:13:26 satellite vdr: [17999] | 3101 | S28.2E-2-2311-52080 | | NO | 419 | Apr 6 11:13:26 satellite vdr: [17999] | 5285 | S28.2E-2-2311-52085 | | NO | 857 | Apr 6 11:13:26 satellite vdr: [17999] | 3147 | S28.2E-2-2312-53147 | NHK World HD | YES | 507 | Apr 6 11:13:26 satellite vdr: [17999] | 5314 | S28.2E-2-2312-53148 | RT HD | YES | 518 | Apr 6 11:13:26 satellite vdr: [17999] | 3508 | S28.2E-2-2312-53150 | | NO | 358 | Apr 6 11:13:26 satellite vdr: [17999] | 4549 | S28.2E-2-2312-53152 | | NO | 803 | Apr 6 11:13:26 satellite vdr: [17999] | 3914 | S28.2E-2-2312-53153 | | NO | 834 | Apr 6 11:13:26 satellite vdr: [17996] EEPG: Filter Pid:0x11,Tid:0x4a added. Apr 6 11:13:26 satellite vdr: [17996] | ID | Channel ID | Channel Name | FND | Sky Num. | Apr 6 11:13:26 satellite vdr: [17996] |------|----------------------------|------------------------|-----|----------| Apr 6 11:13:26 satellite vdr: [17996] | 1519 | S28.2E-2-2052-4629 | | NO | 719 | Apr 6 11:13:26 satellite vdr: [17996] | 1520 | S28.2E-2-2052-4630 | | NO | 720 | Apr 6 11:13:26 satellite vdr: [17996] | 1521 | S28.2E-2-2052-4726 | | NO | 721 | Apr 6 11:13:26 satellite vdr: [17996] | 1522 | S28.2E-2-2052-4727 | | NO | 722 | Apr 6 11:13:26 satellite vdr: [17996] | 1523 | S28.2E-2-2052-4728 | | NO | 723 | Apr 6 11:13:29 satellite vdr: [17996] | 6127 | S28.2E-2-2063-20745 | | NO | 131 | Apr 6 11:13:29 satellite vdr: [17996] | 6011 | S28.2E-2-2063-20701 | | NO | 103 | Apr 6 11:13:29 satellite vdr: [17999] | 6127 | S28.2E-2-2063-20745 | | NO | 131 | Apr 6 11:13:29 satellite vdr: [17999] | 6011 | S28.2E-2-2063-20701 | | NO | 103 | Apr 6 11:13:29 satellite vdr: [17999] | 4098 | S28.2E-2-2035-3898 | | NO | 763 | Apr 6 11:13:29 satellite vdr: [17999] | 4097 | S28.2E-2-2039-3897 | | NO | 762 | Apr 6 11:13:29 satellite vdr: [17996] | 4098 | S28.2E-2-2035-3898 | | NO | 763 | Apr 6 11:13:29 satellite vdr: [17996] | 4097 | S28.2E-2-2039-3897 | | NO | 762 | Apr 6 11:13:30 satellite vdr: [18005] EEPG: PMT scan idle Apr 6 11:13:30 satellite vdr: [18005] EEPG: FreeView Extended EPG detected on pid f02. Apr 6 11:13:30 satellite vdr: [18005] EEPG: Filter Pid:0xf02,Tid:0x4e,Mask:0xfe added. Apr 6 11:13:30 satellite vdr: [18005] EEPG: Filter Pid:0xf02,Tid:0x50,Mask:0xf0 added. Apr 6 11:13:30 satellite vdr: [18005] EEPG: Filter Pid:0xf02,Tid:0x60,Mask:0xf0 added. Apr 6 11:13:30 satellite vdr: [18005] EEPG: Filter Pid:0x39,Tid:0x50,Mask:0xf0 added. Apr 6 11:13:30 satellite vdr: [18005] EEPG: Filter Pid:0x39,Tid:0x60,Mask:0xf0 added. Apr 6 11:13:30 satellite vdr: [17996] | 6210 | S28.2E-2-2054-10210 | | NO | 103 | Apr 6 11:13:30 satellite vdr: [17996] | 6220 | S28.2E-2-2054-10220 | | NO | 103 | Apr 6 11:13:30 satellite vdr: [17996] | 6371 | S28.2E-2-2054-10221 | | NO | 103 | Apr 6 11:13:30 satellite vdr: [17999] | 6210 | S28.2E-2-2054-10210 | | NO | 103 | Apr 6 11:13:30 satellite vdr: [17999] | 6220 | S28.2E-2-2054-10220 | | NO | 103 | Apr 6 11:13:30 satellite vdr: [17999] | 6371 | S28.2E-2-2054-10221 | | NO | 103 | Apr 6 11:13:36 satellite vdr: [17999] EEPG: Filter Pid:0x30,Tid:0xa0,Mask:0xfc added. Apr 6 11:13:36 satellite vdr: [17999] EEPG: sky_huffman_decode: sky_tables[1] is NULL Apr 6 11:13:36 satellite vdr: [17999] EEPG: Warning, could not huffman-decode title-text, skipping title. Apr 6 11:13:36 satellite vdr: [17999] EEPG: sky_huffman_decode: sky_tables[1] is NULL Apr 6 11:13:36 satellite vdr: [17999] EEPG: Warning, could not huffman-decode title-text, skipping title.
Apr 6 10:48:52 satellite vdr: [12365] EEPG: setstatus 0 Apr 6 10:48:52 satellite vdr: [12365] EEPG: setstatus 1 Apr 6 10:48:52 satellite vdr: [12365] EEPG: Filter Pid:0x0,Tid:0x0 added. Apr 6 10:48:52 satellite vdr: [12368] EEPG: setstatus 0 Apr 6 10:48:52 satellite vdr: [12368] EEPG: setstatus 1 Apr 6 10:48:52 satellite vdr: [12368] EEPG: Filter Pid:0x0,Tid:0x0 added. Apr 6 10:48:53 satellite vdr: [12371] EEPG: PMT scan idle Apr 6 10:48:53 satellite vdr: [12371] EEPG: Ended all processing Apr 6 10:48:56 satellite vdr: [12365] EEPG: PMT scan idle Apr 6 10:48:56 satellite vdr: [12365] EEPG: Sky UK Extended EPG detected on pid 55. Apr 6 10:48:56 satellite vdr: [12368] EEPG: PMT scan idle Apr 6 10:48:56 satellite vdr: [12368] EEPG: Sky UK Extended EPG detected on pid 55. Apr 6 10:48:56 satellite vdr: [12365] EEPG: Filter Pid:0x11,Tid:0x4a added. Apr 6 10:48:56 satellite vdr: [12368] EEPG: Filter Pid:0x11,Tid:0x4a added. Apr 6 10:48:56 satellite vdr: [12365] | ID | Channel ID | Channel Name | FND | Sky Num. | Apr 6 10:48:56 satellite vdr: [12365] |------|----------------------------|------------------------|-----|----------| Apr 6 10:48:56 satellite vdr: [12365] | 1519 | S28.2E-2-2052-4629 | | NO | 719 | Apr 6 10:48:56 satellite vdr: [12365] | 1520 | S28.2E-2-2052-4630 | | NO | 720 | Apr 6 10:48:56 satellite vdr: [12365] | 1521 | S28.2E-2-2052-4726 | | NO | 721 | Apr 6 10:48:56 satellite vdr: [12365] | 1522 | S28.2E-2-2052-4727 | | NO | 722 | Apr 6 10:48:56 satellite vdr: [12365] | 1523 | S28.2E-2-2052-4728 | | NO | 723 | Apr 6 10:48:56 satellite vdr: [12365] | 3809 | S28.2E-2-2056-3902 | Dave HD | YES | 213 | Apr 6 10:48:56 satellite vdr: [12365] | 3810 | S28.2E-2-2056-3903 | Watch HD | YES | 214 | Apr 6 10:48:56 satellite vdr: [12365] | 3825 | S28.2E-2-2056-3909 | alibi HD | YES | 229 | Apr 6 10:48:56 satellite vdr: [12365] | 5300 | S28.2E-2-2056-4041 | | NO | 854 | Apr 6 10:48:56 satellite vdr: [12368] | ID | Channel ID | Channel Name | FND | Sky Num. | Apr 6 10:48:56 satellite vdr: [12368] |------|----------------------------|------------------------|-----|----------| Apr 6 10:48:56 satellite vdr: [12368] | 1519 | S28.2E-2-2052-4629 | | NO | 719 | Apr 6 10:48:56 satellite vdr: [12368] | 1520 | S28.2E-2-2052-4630 | | NO | 720 | Apr 6 10:48:56 satellite vdr: [12368] | 1521 | S28.2E-2-2052-4726 | | NO | 721 | Apr 6 10:48:56 satellite vdr: [12368] | 1522 | S28.2E-2-2052-4727 | | NO | 722 | Apr 6 10:48:56 satellite vdr: [12368] | 1523 | S28.2E-2-2052-4728 | | NO | 723 | Apr 6 10:49:00 satellite vdr: [12365] | 6210 | S28.2E-2-2054-10210 | | NO | 103 | Apr 6 10:49:00 satellite vdr: [12365] | 6220 | S28.2E-2-2054-10220 | | NO | 103 | Apr 6 10:49:00 satellite vdr: [12365] | 6371 | S28.2E-2-2054-10221 | | NO | 103 | Apr 6 10:49:00 satellite vdr: [12368] | 6210 | S28.2E-2-2054-10210 | | NO | 103 | Apr 6 10:49:00 satellite vdr: [12368] | 6220 | S28.2E-2-2054-10220 | | NO | 103 | Apr 6 10:49:00 satellite vdr: [12368] | 6371 | S28.2E-2-2054-10221 | | NO | 103 | Apr 6 10:49:06 satellite vdr: [12365] EEPG: Filter Pid:0x30,Tid:0xa0,Mask:0xfc added. Apr 6 10:49:06 satellite vdr: [12368] EEPG: Filter Pid:0x30,Tid:0xa0,Mask:0xfc added. Apr 6 10:49:06 satellite kernel: [1009472.476878] section handler[12365]: segfault at 10 ip 00007f64d49e5c54 sp 00007f64d42d0a60 error 4 in libvdr-eepg.so.2.1.6[7f64d4999000+6f000]
Updated by dimeptr over 10 years ago
cheesemonster wrote:
Is EEPG safe when two threads are updating at the same time?
Looking though syslog, I see that the segfaults are consistent with this circumstance of simultaneous updates from two threads.
I agree that this might be the case, I will have to work harder to reproduce the issue in order to fix it.
Updated by cheesemonster over 10 years ago
OK, I've figured this out. The problem is that 'Format' is a global variable defined in util.h and should perhaps be a class local variable in cFilterEEPG because it appears to be referenced and set by functions within cFilterEEPG.
What I suspect is happening is that because I have a mix of Terrestrial receivers and Satellite receivers, the value of Format is switching between FREEVIEW (6) and SKY_UK (4) and at the point in time InitDictionary() is called, Format is set to the wrong value (FREEVIEW in this case), so the sky_tables are never loaded. Periodically it does work, but that's just down to timing.
Apr 7 12:26:07 satellite vdr: [5092] EEPG-nickb: SI::TableIdSKYChannels - calling GetChannelsSKYBOX Apr 7 12:26:07 satellite vdr: [5092] EEPG-nickb: SI::TableIdSKYChannels - calling InitDictionary - Format=6 Apr 7 12:26:07 satellite vdr: [5092] EEPG-nickb: SI::TableIdSKYChannels - adding SI::TableIdSKYTitlesA0 Apr 7 12:26:07 satellite vdr: [5092] EEPG: Filter Pid:0x30,Tid:0xa0,Mask:0xfc added.
This log snippet shows that the functions are being called in the correct order, as you previously described:
Apr 7 12:03:56 satellite vdr: [599] EEPG: Sky UK Extended EPG detected on pid 55. Apr 7 12:03:56 satellite vdr: [599] EEPG-nickb: Adding SI::TableIdSKYChannels filter Apr 7 12:03:56 satellite vdr: [599] EEPG: Filter Pid:0x11,Tid:0x4a added. Apr 7 12:03:56 satellite vdr: [599] EEPG-nickb: SI::TableIdSKYChannels - calling GetChannelsSKYBOX Apr 7 12:03:56 satellite vdr: [599] | ID | Channel ID | Channel Name | FND | Sky Num. | Apr 7 12:04:03 satellite vdr: [599] EEPG-nickb: SI::TableIdSKYChannels - calling GetChannelsSKYBOX Apr 7 12:04:06 vdr: last message repeated 60 times Apr 7 12:04:06 satellite vdr: [599] EEPG-nickb: SI::TableIdSKYChannels - calling InitDictionary Apr 7 12:04:06 satellite vdr: [599] EEPG-nickb: SI::TableIdSKYChannels - adding SI::TableIdSKYTitlesA0 Apr 7 12:04:06 satellite vdr: [599] EEPG: Filter Pid:0x30,Tid:0xa0,Mask:0xfc added. Apr 7 12:04:06 satellite vdr: [599] EEPG-nickb: SI::TableIdSKYTitles: calling GetTitlesSKYBOX Apr 7 12:04:06 satellite vdr: [599] EEPG: sky_huffman_decode: sky_tables[1] is NULL Apr 7 12:04:06 satellite vdr: [599] EEPG: Warning, could not huffman-decode title-text, skipping title.
Updated by dimeptr over 10 years ago
Your find makes sense. I will modify it this weekend and send you a patch to test, so hopefully we can fix this.
Updated by cheesemonster over 10 years ago
I wrote up a fix yesterday and am currently testing it out. So far so good, InitDictionary is now being called with the correct Format setting and I'm no longer getting segfaults. I'll attach a patch later this week.
Updated by cheesemonster over 10 years ago
- File format-filter.patch format-filter.patch added
Attached is the patch.
Updated by cheesemonster over 10 years ago
The patch appears to have had two other noticeable effects:
- I no longer get epg.data entry corruption -- some channels used to be filled with arbitrary event data from other channels.
- A memory leak I was seeing in VDR no longer occurs -- the process would often grow to consume between 1 and 8GB RAM.
Updated by dimeptr over 10 years ago
- File format-filter2.patch format-filter2.patch added
Can you test with this patch, I have changed the format to enum since it helps finding bugs like the missing implementation in eepg.c lines 2916 and 2927
Updated by cheesemonster over 10 years ago
I've tested the patch and it is working well. I have had single thread and multiple threads scanning the Sky and Freesat tables simultaneously and experienced no crashes.
Updated by dimeptr over 10 years ago
- Status changed from Assigned to Closed
- % Done changed from 0 to 100
Applied in changeset d7dc6141c91b48c410e9cc5734ca9a2adecc278e.