InstallTileRendering < JBrowse

JBrowse > InstallTileRendering

TWiki webs: Main | TWiki | Sandbox Log In or Register

See GettingStarted

Most of the information below is out of date. We might bring some parts of it back in the future, so I'm leaving it here, but GettingStarted is a better place to start. -MS 4/13/07

THE GOAL

This is just a log of how I installed everything necessary to run generate-tiles.pl on the SixtyFourBitCluster. This is here for reference in case I need to look something up or if the procedure needs to be repeated on some other system, but your mileage may vary.

generate-tiles.pl is just a Perl script that uses parts of frameworks of GenericGenomeBrowser (GBrowse), BioPerl, GD, MySQL, and a few other things to pre-render tiles for the AJAX-based genome browser we are building (see AndrewUzilov's "Current Projects"). But you can also use these instructions to set up a GenericGenomeBrowser (GBrowse) server, provided you also install and configure Apache to actually serve up the webpage, which is really easy. However, better instructions for this already exist.

PREREQUISITES

The following things were on the cluster already. If you don't have them on your system, you must install and configure them before proceeding:

Perl 5.005 or later
CPAN (Perl module for automatic download and installation of, well... Perl modules)

If CPAN is giving you trouble, delete the file /usr/lib/perl5/5.8.5/CPAN/Config.pm (or equivalent on your system) and run:

$ cpan

which should launch a configuration script where you can re-enter all the CPAN settings from scratch.

THE INSTALLATION PROCESS

I RECOMMEND DOING ALL OF THIS AS ROOT!

TO MAKE A LONG STORY SHORT...

Here are the exact steps, right down to the very last shell command, of how I set up each SunGridEngine exec node of the SixtyFourBitCluster to do tile rendering for Drosophila melanogaster:

InstallTileRenderingExactSteps

Log into each node as root and execute the above steps, and you will be ready to go - that means you will be able to submit a SunGridEngine job that runs generate-tiles.pl on that node to render tiles, writing output to /mnt/nfs/dmel. Note that the XML file will not be written, because we are rendering the tracks in many little pieces - generating XML is something you should do in a separate, single step, but without the time-consuming rendering (that's why generate-tiles.pl has all those options like -m 3 that you can use to generate only the XML file, or render without generating XML).

But the longer, more descriptive diatribe below may also be helpful, since it actually explains why we are doing some things are how we can do them differently.

SET UP MYSQL

In a nutshell (see MySQL installation instructions for more detail), we installed pre-compiled binaries for the AMD 64-bit architecture as follows (note: this won't install the man files, so look elsewhere for that)...

$ su -

Prepare the files:

$ cp mysql-max-5.0.18-linux-x86_64-glibc23.tar.gz /usr/local/src

$ cd /usr/local/src

$ tar xvfz mysql-max-5.0.18-linux-x86_64-glibc23.tar.gz

$ cd mysql-max-5.0.18-linux-x86_64-glibc23

$ ln -s `pwd` /usr/local/mysql

Add user (skip if you already have a mysql user):

$ groupadd mysql

$ useradd -g mysql mysql

Set up the grant tables:

$ scripts/mysql_install_db --user=mysql

IMPORTANT NOTE: the above command loads the default paths (for database directory, socket file, log file, etc.) from /etc/my.cnf, so make sure that the paths in that directory are the ones that you want, and make sure that the mysql user has ownership and read/write access to all those paths! This is critical. You must also change the socket and pid-file paths in /etc/my.cnf to be /tmp/mysql.sock and /tmp/mysql.pid, respectively, because that's where the mysql command will look for them when you run it without parameters, and that's what TiledImage will expect. Here is my personal /etc/my.cnf file (the lines commented out were the defaults on a Centos installation):

  # Some default paths below were changed to fix permission problems ('mysql' user could
  # not write to those paths). [AVU 2/6/2006]

  [mysqld]
  datadir=/var/lib/mysql
  #socket=/var/lib/mysql/mysql.sock
  socket=/tmp/mysql.sock

  # Default to using old password format for compatibility with mysql 3.x
  # clients (those using the mysqlclient10 compatibility package).
  old_passwords=1
     
  [mysql.server]
  user=mysql
  basedir=/var/lib

  [mysqld_safe]
  #err-log=/var/log/mysqld.log
  #pid-file=/var/run/mysqld/mysqld.pid
  err-log=/var/lib/mysql/mysqld.log
  pid-file=/tmp/mysqld.pid

So, to set the permissions for the relevant paths from the file above (usually everyone can read/write to /tmp), I would run:

$ chown -R mysql:mysql /var/lib/mysql

$ chmod -R u+rw /var/lib/mysql

Start up the MySQL daemon:

$ bin/mysqld_safe --user=mysql &

Note that if you didn't make sure that the mysql user has ownership to paths in /etc/my.cnf as above, this will fail and write to the error log, the path of which is... well, specified by /etc/my.cnf, or failing that, probably /tmp/mysql.err. If mysql is giving you trouble about the location of the socket file (which happens on some installations), just symlink the expected location to the location where it actually is, e.g.:

$ ln -s /tmp/mysql.sock /var/lib/mysql/mysql.sock

Note that if you restart the host, you'll have to restart the daemon by hand. See MySQL instructions (in the above example, located in /usr/local/mysql/docs/mysql.info) on how to configure automatic startup at boot.

Also note that this database is extremely insecure, as none of the users have passwords by default. You can set up a username and password (see this and this) and have the two components that use MySQL - that is, TiledImage and the GBrowse DBI MySQL adaptor - use a username and password for database access. The username and password for TiledImage have to be manually specified in TiledImage.pm (although this may be fixed in near future) where the DBI->connect() function call is executed. The username and password for the DBI MySQL adaptor (if you are using it, and not the memory adaptor) is specified in the .conf file.

If you are just running this database locally for the purposes of generating tiles and aren't worried about security, you can simply grant all privileges to the user ''@'localhost' (i.e. a no-name user on the local machine), which TiledImage runs the query as by default. This is done by:

$ mysql --user=root

mysql> GRANT ALL PRIVILEGES ON *.* TO ''@'localhost';

N.B.: the MySQL root user is not the same thing as your system's root user, so don't make that confusion, and the same goes for all the other MySQL users. When you just run mysql without a --user parameter, it will try to log into MySQL using your system username, but the --user parameter can override that. So any system user that has access to the mysql binary can log in as the MySQL root user (just like above) as long as you have the MySQL root password (that is, if you set one up).

Lastly, for TiledImage, you will need to add the database gdtile to MySQL before you can use it, which can be done by loading the TiledImageMySQLDatabaseSchema. There's probably a more elegant way to do it, but personally I just start:

$ mysql

or, if you're not root (because you need to be the MySQL root user to do some things):

$ mysql --user=root

and copy and paste the entire text of the schema into the mysql prompt.

SET UP BIOPERL AND OTHER PERL MODULES

Use CPAN to install pre-requisite modules by invoking:

$ cpan

as root on the command line, then typing in:

> install MODULE_NAME

in the CPAN shell prompt. This is what you need to install, in this order:

  CGI
  CGI::Session
  DBI
  DBD::mysql (the tests for this MIGHT FAIL - see below)
  Digest::MD5
  Text::Shellwords

Note that the DBD::mysql module might fail the tests if you set up a password for the mysql or root user, since the MySQL daemon requires a password that the tests don't provide, but that is OK - you can either just force install DBD::mysql in the CPAN interactive prompt and take it on faith, or provide a password for the tests (how to do that I'm not sure, although it's possible). Or alternately, you could remove all passwords from MySQL for the purpose of the tests and have it run completely unsecured.

Now to configure BioPerl, for which we will be using version 1.5.1-rc3 (get the file bioperl-1.5.1-rc3.tar.gz) from here. There are other versions out, but this writeup was written for 1.5.1-rc3, although the instructions here might be applicable to other versions also.

We must first use CPAN to install Bundle::BioPerl (the BioPerl Bundle), but heed this warning...

PROBLEMS MAY OCCUR HERE: I had difficulty installing the BioPerl bundle on a node with a complete re-install of CentOS. The libPNG and GD libraries should be compiled from scratch before you start installing the BioPerl bundle using CPAN! Get the libPNG source here and the GD source here. You will need zlib libraries for both, but you should have the latest ones already if you followed the Installing CentOS On Cluster instructions. If you have trouble, I would recommend trying this:

Compile and install libPNG from source to /usr/local/, or whichever directory does NOT have the .a files for zlib, since that seems to cause problems.

Do the same for GD.

Go back into CPAN and force install GD, GD::SVG, XML::Twig, then Bundle::BioPerl, all in that order. Ignore the failed tests.

The bundle is installed as the other modules above.

Accept all defaults and install all modules that you are prompted for. I would recommend against running the SOAP::Lite tests (not running them is the default anyway) if you are behind a strict firewall, since it will fail due to that.

Now, at this point you can install the actual bulk of BioPerl using CPAN also... but we won't do that, because the version of BioPerl that we want isn't on CPAN, so we're going to do if from source.

Unpack bioperl-1.5.1-rc3.tar.gz (or your favorite BioPerl version archive) and cd into the resulting directory. Run:

$ perl Makefile.PL

Say "install ALL scripts" and "yes to BioGFF tests" when prompted. You may fail the BioGFF tests if your MySQL database is under some sort of security protection, which is OK, as long as you know it won't interfere with TiledImage later. The makefile will be generated, and you will get a list of missing modules at the end. For me, this list was:

  GD::SVG (need to 'force install' this one)
  Ace
  XML::SAX::Writer (need to 'force install' this one)
  Clone
  Class::AutoClass

If you think you will need any of these, you can install them using CPAN. I only installed GD::SVG, since a GBrowse feature needs it. However, the feature doesn't seem to work, and I haven't yet tracked down whether this is due to the same reason that I had to 'force install' the module.

Run:

$ make

$ make test

and, if all (or at least most... or at least the releveant) tests pass, run:

$ make install

which, for me, installed everything to /usr/lib/perl5/site_perl/5.8.5/.

Lastly, we can use CPAN to easily install some optional modules for GBrowse and BioPerl, which are:

  XML::Parser  (update: this is apparently already installed if you did all the above steps)
  XML::Writer  (update: this is apparently already installed if you did all the above steps)
  XML::Twig    (need to 'force install' this one)
  XML::DOM     (update: this is apparently already installed if you did all the above steps)
  LWP          (update: this is apparently already installed if you did all the above steps)
  Bio::Das
  MOBY         (I don't think this is on CPAN... see http://www.biomoby.org/GettingTheCode.html if so inclined)

and out of which I installed all except for MOBY.

SET UP GBROWSE

$ cp Generic-Genome-Browser-1.62.tar.gz /usr/local/src

$ cd /usr/local/src

$ tar xvfz Generic-Genome-Browser-1.62.tar.gz

$ cd Generic-Genome-Browser-1.62

$ perl Makefile.PL

The above sets up the following default paths and settings on Red Hat-based operating systems (such as Centos):

   APACHE /usr/local/apache
     CONF /etc/httpd/conf
   HTDOCS /var/www/html
   CGIBIN /var/www/cgi-bin
      LIB 
      BIN 
    DO_XS 1
  NONROOT

which you can change as explained here.

$ make

$ make test (you should pass all the tests)

$ make install

CHECKING OUT generate-tiles.pl USING THE SUBVERSION SYSTEM

The script generate-tiles.pl that you will need is part of the TiledImage Subversion project, which you can check out of the Subversion depository as described here. The write-up of how to use the script will come eventually.

A SPECIAL NOTE ON Bio::DB::GFF::Adaptor::mysql - NOTE TO SELF: MOVE THIS TO WRITE-UP OF HOW TO USE THE SCRIPT

You have the option of loading the contents of your GFF file into a database (for the purposes of this example, MySQL). Sometimes this is necessary, as with the latest Drosophila melanogaster Chado release in GFF 3 format. The problem is that the bulk_load_gff.PLS script on the SixtyFourBitCluster does not seem to work properly (terminates because unable to read one of the temp files it creates... which are created just fine, but seem to disappear too early). But it does work on xander, so what we do is generate the MySQL database containing GFF data for Drosophila on xander, then back it up using:

$ mysqldump fly > NAME_OF_SAVE_FILE.sql

where "fly" is the database name. What this does is generate and write to standard out (hence the redirect to a savefile) all the SQL syntax statements that you need to load every little piece of data into the database from scratch.

So, on the recipient end, you should create the database:

$ mysql --user=root

mysql> CREATE DATABASE fly;

mysql> GRANT ALL ON fly.* TO ''@'localhost';

and then just pipe the contents of the savefile to MySQL like this:

$ cat NAME_OF_SAVE_FILE | mysql

which will execute every single SQL statement in that file, thus creating the necessary tables and filling them out with the data you backed up using mysqldump.

-- AndrewUzilov - 02 Mar 2006