Project

General

Profile

epgd

This daemon is used to download EPG data from the internet and manage it in a mysql database.

Written by: Jörg Wendel (vdr at jwendel dot de)
Homepage: http://projects.vdr-developer.org/projects/vdr-epg-daemon
Latest version at: http://projects.vdr-developer.org/git/vdr-epg-daemon.git

This software is released under the GPL, version 2 (see COPYING).
Additionally, compiling, linking, and/or using the OpenSSL toolkit in
conjunction with this software is allowed.

Contents:

Contents
Description
Get the source from git
Requirements
MySQL Setup
epgd Installation
epgd Configuration
Starting epgd and init-Scripts
MySQL Hints
Considerations

Description:

epgd is part of the team epgd+epg2vdr+scraper2vdr to effectively retrieve, store and import epg data to vdr. It is designed to handle large amount of data and pictures in a distributed environment with one epg-server and many possible vdr-clients - therefore it relays on mysql.

Though it is possible to use epgd alone with mysql it only makes sense to use it as backend to the vdr-plugin epg2vdr. That being said you need to install, setup and configure mysql, epgd and epg2vdr in order to get a working environment.

Get the source from git:

get the source - probably done, when you reading this locally:
git clone git://projects.vdr-developer.org/vdr-epg-daemon.git

update the source:
cd /to/your/source/epgd
git pull

throwing away local changes and update to latest source:
cd /to/your/source/epgd
git checkout -f master
git reset --hard
git pull

Requirements:

- libarchive
  - libcurl 7.10+
  - libxslt 1.1.24+
  - libxml2 
  - libmysql >= 5.07
  - libz

Ubuntu (12.10):

- libarchive12, libarchive-dev
  - libz, libz-dev
  - libcurl4-nss-dev (oder libcurl3-nss-dev)
  - libxslt1.1 libxslt1-dev libxml2 libxml2-dev
  - libmysqlclient-dev libmysqlclient18
  - libjpeg62-dev libjpeg62

My-SQL Setup:

You need a running mysql database. Consult the manual of your linux-flavour how to install and configure mysql. Remember to put your db-files in a large enough filesystem: with many channels, many pre fetched days and many pictures you will easily need 3 or more Gb of disk space. When mysql is up and running you need to prepare the database and access-rights for use with epgd:

login as root:

#> mysql -u root -p
 CREATE DATABASE epg2vdr charset utf8;
 CREATE USER 'epg2vdr'@'%' IDENTIFIED BY 'epg';
 GRANT ALL PRIVILEGES ON epg2vdr.* TO 'epg2vdr'@'%';
 GRANT ALL PRIVILEGES ON epg2vdr.* TO 'epg2vdr'@'localhost' IDENTIFIED BY 'epg';
 FLUSH PRIVILEGES;

adjust the bind address in my.cnf:

bind-address         = 127.0.0.1

in youre my.cnf (mostly found here: /etc/mysql/my.cnf) to the address where
the mysql serve should listen at. Or comment out if it should listen at all interfaces.

switch of binary logging:

comment line starting with

log_bin  = ...

in youre my.cnf

login as epg2vdr for maintenance:

#> mysql -u epg2vdr -pepg -Depg2vdr

- or remote
#> mysql -u epg2vdr -pepg -Depg2vdr -h <host>

If you have problems with setting up and granting access rights to your epg2vdr user read "MySQL Hints" near the end of this file.

howto show configured users:

 use mysql
 SELECT * FROM user;

epgd Installation:

epgd is source distributed only. So you have to build your binary and make some adjustments to your system. So we try to help with some stuff in the contrib directory, it's not a bad idea to learn something about your linux's init-system and init-scripts.

- Unpack
- Modify Make.config (BINDEST and PLGDEST) as you like
- Call "make"
- Call "make plugins"
- Call "make install"
- Call "make install-plugins"
- Create directory /etc/epgd
- cp configs/* /etc/epgd
- merge the config parts of the epgd plugins (./PLUGINS/*/configs) to /etc/epgd
- > merge /etc/epgd/channelmap.conf and /etc/epgd/epgd.conf
- > copy ./PLUGINS/*/configs/*.xsl to /etc/epgd
- modify config (/etc/epgd/channelmap.conf and /etc/epgd/epgd.conf) like your needs
- Create the database (see below)
- Start epgd (see below)

If you don't know how to merge you can start of with

#> cat ./PLUGINS/*/configs/channelmap.conf >>/etc/epgd/channelmap.conf
#> cat ./PLUGINS/*/configs/epgd.conf >>/etc/epgd/epgd.conf

This is only true if you use all plugins. If you use only one of them replace "*" with the plugin you use.

Look carefully through your generated files!

Starting epgd and init-Scripts:

There are many ways to start epgd. For a first try or debugging sessions you may want to start it simply by typing at the root command-prompt (paths have to be adjusted according to your changes to BINDEST and PLGDEST in Make.config):

#> export LANG="de_DE.UTF-8"
#> ulimit -c unlimited #so you can torture the developers with back traces
#> /usr/local/bin/epgd -n -p /usr/local/lib/epgd/plugins

epgd is configured in /etc/epgd/epgd.conf. But you can overwrite the following options on the command line:

-n              don't daemonize
    -t              log to stdout
    -c &lt;config-dir&gt; use config in &lt;config-dir&gt;
    -p &lt;plugin-dir&gt; load plugins from &lt;plugin-dir&gt;
    -l &lt;log-level&gt;  set log level

For production use you should start epgd after net- and mysql services via your init-system. Some start-scripts have been put to ./contrib hopefully serving your needs (or at least giving you an idea of how to proceed).

epgd Configuration:

DbHost
    ip/name of mysql server host
DbPort
    port of the mysql server
DbName
    name of the database
DbUser
DbPass
DaysInAdvance #
    Download (insert) EPG information for the next # days
DaysToUpdate #
    Update # already insert days
Updatetime (hours)
    Perform automatic update (default: 2 hours)
XmlStoreToFs
    Sorer XML files to the filesystem, for debugging (default 0)
Get EPG images
    Download images with the EPG information
MaxImagesPerEvent
    How many pictures per event should be downloaded
EpgImageSize
    0 = 174x130
    1 = 329x245
    2 = 525x400
CachePath
    path to cache the xml files if XmlStoreToFs or SeriesStoreToFs enabled (default /var/cache/epgd)
SeriesEnabled
    get series from constable.net (default 1)
SeriesPort
    (default 2006)
SeriesStoreToFs
    (default 0)
SeriesUrl
    (default eplists.constabel.net)
UseProxy
      Enable HTTP Proxy (default 0)
HTTPProxy
      Name of the proxy
Proxy User Name
      Username of your proxy
Password
      Password of your proxy
Log level #
      Logging level 1-5 (Errors, Infos, Debug ...) (default 1)

When choosing a loglevel consider carefully the impact on i/o-performance of your system. If you are using some kind of pattern driven logfile parsing software like logchecker you may want to use ./contrib/epgd.ignore

MYSQL HINTS:

- If you cannot figure out why you get Access denied, remove all entries from the user table
that have Host values with wildcards contained (entries that match '%' or '_' characters).
A very common error is to insert a new entry with Host='%' and User='some_user',
thinking that this allows you to specify localhost to connect from the same machine.
The reason that this does not work is that the default privileges include an
entry with Host='localhost' and User=''. Because that entry has a Host value 'localhost'
that is more specific than '%', it is used in preference to the new entry when connecting
from localhost! The correct procedure is to insert a second entry with Host='localhost'
and User='some_user', or to delete the entry with Host='localhost' and User=''.
After deleting the entry, remember to issue a "FLUSH PRIVILEGES" statement to reload the grant tables.

Considerations:

By design epgd can handle multiple clients. If you looking for a solution for your vdr-network plan your implementation carefully. Which clients should be able to connect? Which epg-solution in in use on each client? Has any of the clients patches or plugins manipulating epg running? And now the most important hint: read the epg2vdr README before you continue.

epgd can generate large amounts of date, as well db-data as logging-data - so don't be surprised. Manipulating data (read, write, move around) does not only need appropriate disk space, but also appropriate I/O-performance.

If you run into problems and want to start from a clean base, stop epgd and drop all data (take a look at the scripts in ./scripts) and restart epgd.

Description:

epgd is part of the team epgd+epg2vdr+scraper2vdr to effectively retrieve, store and import epg data to vdr. It is designed to handle large amount of data and pictures in a distributed environment with one epg-server and many possible vdr-clients - therefore it relays on mysql.

Though it is possible to use epgd alone with mysql it only makes sense to use it as backend to the vdr-plugin epg2vdr. That being said you need to install, setup and configure mysql, epgd and epg2vdr in order to get a working environment.

Get the source from git:

get the source - probably done, when you reading this locally:
git clone git://projects.vdr-developer.org/vdr-epg-daemon.git

update the source:
cd /to/your/source/epgd
git pull

throwing away local changes and update to latest source:
cd /to/your/source/epgd
git checkout -f master
git reset --hard
git pull

Requirements:

- libarchive
  - libcurl 7.10+
  - libxslt 1.1.24+
  - libxml2 
  - libmysql >= 5.07
  - libz

Ubuntu (12.10):

- libarchive12, libarchive-dev
  - libz, libz-dev
  - libcurl4-nss-dev (oder libcurl3-nss-dev)
  - libxslt1.1 libxslt1-dev libxml2 libxml2-dev
  - libmysqlclient-dev libmysqlclient18
  - libjpeg62-dev libjpeg62

My-SQL Setup:

You need a running mysql database. Consult the manual of your linux-flavour how to install and configure mysql. Remember to put your db-files in a large enough filesystem: with many channels, many pre fetched days and many pictures you will easily need 3 or more Gb of disk space. When mysql is up and running you need to prepare the database and access-rights for use with epgd:

- login as root: 

#> mysql -u root -p
 CREATE DATABASE epg2vdr charset utf8;
 CREATE USER 'epg2vdr'@'%' IDENTIFIED BY 'epg';
 GRANT ALL PRIVILEGES ON epg2vdr.* TO 'epg2vdr'@'%';
 GRANT ALL PRIVILEGES ON epg2vdr.* TO 'epg2vdr'@'localhost' IDENTIFIED BY 'epg';
 FLUSH PRIVILEGES;

- adjust the bind address in my.cnf:
adjust the line
bind-address = 127.0.0.1
in youre my.cnf (mostly found here: /etc/mysql/my.cnf) to the address where
the mysql serve should listen at. Or comment out if it should listen at all interfaces.

- switch of binary logging:
comment line starting wiht 'log_bin' in youre my.cnf

- login as epg2vdr for maintenance:

#> mysql -u epg2vdr -pepg -Depg2vdr

- or remote
#> mysql -u epg2vdr -pepg -Depg2vdr -h <host>

If you have problems with setting up and granting access rights to your epg2vdr user read "MySQL Hints" near the end of this file.

- show all users:

 use mysql
 SELECT * FROM user;

epgd Installation:

epgd is source distributed only. So you have to build your binary and make some adjustments to your system. So we try to help with some stuff in the contrib directory, it's not a bad idea to learn something about your linux's init-system and init-scripts.

- Unpack
- Modify Make.config (BINDEST and PLGDEST) as you like
- Call "make"
- Call "make plugins"
- Call "make install"
- Call "make install-plugins"
- Create directory /etc/epgd
- cp configs/* /etc/epgd
- merge the config parts of the epgd plugins (./PLUGINS/*/configs) to /etc/epgd
- > merge /etc/epgd/channelmap.conf and /etc/epgd/epgd.conf
- > copy ./PLUGINS/*/configs/*.xsl to /etc/epgd
- modify config (/etc/epgd/channelmap.conf and /etc/epgd/epgd.conf) like your needs
- Create the database (see below)
- Start epgd (see below)

If you don't know how to merge you can start of with

#> cat ./PLUGINS/*/configs/channelmap.conf >>/etc/epgd/channelmap.conf
#> cat ./PLUGINS/*/configs/epgd.conf >>/etc/epgd/epgd.conf

This is only true if you use all plugins. If you use only one of them replace "*" with the plugin you use.

Look carefully through your generated files!

Starting epgd and init-Scripts:

There are many ways to start epgd. For a first try or debugging sessions you may want to start it simply by typing at the root command-prompt (paths have to be adjusted according to your changes to BINDEST and PLGDEST in Make.config):

#> export LANG="de_DE.UTF-8"
#> ulimit -c unlimited #so you can torture the developers with back traces
#> /usr/local/bin/epgd -n -p /usr/local/lib/epgd/plugins

epgd is configured in /etc/epgd/epgd.conf. But you can overwrite the following options on the command line:

-n              don't daemonize
    -t              log to stdout
    -c &lt;config-dir&gt; use config in &lt;config-dir&gt;
    -p &lt;plugin-dir&gt; load plugins from &lt;plugin-dir&gt;
    -l &lt;log-level&gt;  set log level

For production use you should start epgd after net- and mysql services via your init-system. Some start-scripts have been put to ./contrib hopefully serving your needs (or at least giving you an idea of how to proceed).

epgd Configuration:

DbHost
    ip/name of mysql server host
DbPort
    port of the mysql server
DbName
    name of the database
DbUser
DbPass
DaysInAdvance #
    Download (insert) EPG information for the next # days
DaysToUpdate #
    Update # already insert days
Updatetime (hours)
    Perform automatic update (default: 2 hours)
XmlStoreToFs
    Sorer XML files to the filesystem, for debugging (default 0)
Get EPG images
    Download images with the EPG information
MaxImagesPerEvent
    How many pictures per event should be downloaded
EpgImageSize
    0 = 174x130
    1 = 329x245
    2 = 525x400
CachePath
    path to cache the xml files if XmlStoreToFs or SeriesStoreToFs enabled (default /var/cache/epgd)
SeriesEnabled
    get series from constable.net (default 1)
SeriesPort
    (default 2006)
SeriesStoreToFs
    (default 0)
SeriesUrl
    (default eplists.constabel.net)
UseProxy
      Enable HTTP Proxy (default 0)
HTTPProxy
      Name of the proxy
Proxy User Name
      Username of your proxy
Password
      Password of your proxy
Log level #
      Logging level 1-5 (Errors, Infos, Debug ...) (default 1)

When choosing a loglevel consider carefully the impact on i/o-performance of your system. If you are using some kind of pattern driven logfile parsing software like logchecker you may want to use ./contrib/epgd.ignore

MYSQL HINTS:

- If you cannot figure out why you get Access denied, remove all entries from the user table
that have Host values with wildcards contained (entries that match '%' or '_' characters).
A very common error is to insert a new entry with Host='%' and User='some_user',
thinking that this allows you to specify localhost to connect from the same machine.
The reason that this does not work is that the default privileges include an
entry with Host='localhost' and User=''. Because that entry has a Host value 'localhost'
that is more specific than '%', it is used in preference to the new entry when connecting
from localhost! The correct procedure is to insert a second entry with Host='localhost'
and User='some_user', or to delete the entry with Host='localhost' and User=''.
After deleting the entry, remember to issue a "FLUSH PRIVILEGES" statement to reload the grant tables.

Considerations:

By design epgd can handle multiple clients. If you looking for a solution for your vdr-network plan your implementation carefully. Which clients should be able to connect? Which epg-solution in in use on each client? Has any of the clients patches or plugins manipulating epg running? And now the most important hint: read the epg2vdr README before you continue.

epgd can generate large amounts of date, as well db-data as logging-data - so don't be surprised. Manipulating data (read, write, move around) does not only need appropriate disk space, but also appropriate I/O-performance.

If you run into problems and want to start from a clean base, stop epgd and drop all data (take a look at the scripts in ./scripts) and restart epgd.