Most of the information below is out of date. We might bring some parts of it back in the future, so I'm leaving it here, but GettingStarted
is a better place to start. -MS 4/13/07
This is just a log of how I installed everything necessary to run generate-tiles.pl
on the SixtyFourBitCluster
. This is here for reference in case I need to look something up or if the procedure needs to be repeated on some other system, but your mileage may vary.
is just a Perl script that uses parts of frameworks of GenericGenomeBrowser
, GD, MySQL
, and a few other things to pre-render tiles for the AJAX-based genome browser we are building (see AndrewUzilov
's "Current Projects"). But you can also use these instructions to set up a GenericGenomeBrowser
(GBrowse) server, provided you also install and configure Apache to actually serve up the webpage, which is really easy. However, better instructions
for this already exist.
The following things were on the cluster already. If you don't have them on your system, you must install and configure them before proceeding:
- Perl 5.005 or later
- CPAN (Perl module for automatic download and installation of, well... Perl modules)
If CPAN is giving you trouble, delete the file /usr/lib/perl5/5.8.5/CPAN/Config.pm
(or equivalent on your system) and run:
which should launch a configuration script where you can re-enter all the CPAN settings from scratch.
THE INSTALLATION PROCESS
I RECOMMEND DOING ALL OF THIS AS ROOT!
TO MAKE A LONG STORY SHORT...
Here are the exact steps
, right down to the very last shell command, of how I set up each SunGridEngine
exec node of the SixtyFourBitCluster
to do tile rendering for Drosophila melanogaster:
Log into each node as root and execute the above steps, and you will be ready to go - that means you will be able to submit a SunGridEngine
job that runs generate-tiles.pl
on that node to render tiles, writing output to /mnt/nfs/dmel
. Note that the XML file will not be written, because we are rendering the tracks in many little pieces - generating XML is something you should do in a separate, single step, but without the time-consuming rendering (that's why generate-tiles.pl
has all those options like -m 3
that you can use to generate only
the XML file, or render without generating XML).
But the longer, more descriptive diatribe below may also be helpful, since it actually explains why
we are doing some things are how we can do them differently.
SET UP MYSQL
In a nutshell (see MySQL
installation instructions for more detail), we installed pre-compiled binaries for the AMD 64-bit architecture as follows (note:
this won't install the man
files, so look elsewhere for that)...
$ su -
Prepare the files:
$ cp mysql-max-5.0.18-linux-x86_64-glibc23.tar.gz /usr/local/src
$ cd /usr/local/src
$ tar xvfz mysql-max-5.0.18-linux-x86_64-glibc23.tar.gz
$ cd mysql-max-5.0.18-linux-x86_64-glibc23
$ ln -s `pwd` /usr/local/mysql
Add user (skip if you already have a mysql
$ groupadd mysql
$ useradd -g mysql mysql
Set up the grant tables:
$ scripts/mysql_install_db --user=mysql
the above command loads the default paths (for database directory, socket file, log file, etc.) from /etc/my.cnf
, so make sure
that the paths in that directory are the ones that you want, and make sure
that the mysql
user has ownership and read/write access to all those paths!
This is critical. You must also change the socket
paths in /etc/my.cnf
to be /tmp/mysql.sock
, respectively, because that's where the mysql
command will look for them when you run it without parameters, and that's what TiledImage
will expect. Here is my personal /etc/my.cnf
file (the lines commented out were the defaults on a Centos installation):
# Some default paths below were changed to fix permission problems ('mysql' user could
# not write to those paths). [AVU 2/6/2006]
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
So, to set the permissions for the relevant paths from the file above (usually everyone can read/write to /tmp
), I would run:
$ chown -R mysql:mysql /var/lib/mysql
$ chmod -R u+rw /var/lib/mysql
Start up the MySQL
$ bin/mysqld_safe --user=mysql &
Note that if you didn't make sure that the mysql
user has ownership to paths in /etc/my.cnf
as above, this will fail and write to the error log, the path of which is... well, specified by /etc/my.cnf
, or failing that, probably /tmp/mysql.err
. If mysql
is giving you trouble about the location of the socket file (which happens on some installations), just symlink the expected location to the location where it actually is, e.g.:
$ ln -s /tmp/mysql.sock /var/lib/mysql/mysql.sock
Note that if you restart the host, you'll have to restart the daemon by hand. See MySQL
instructions (in the above example, located in /usr/local/mysql/docs/mysql.info
) on how to configure automatic startup at boot.
Also note that this database is extremely insecure
, as none of the users have passwords by default. You can set up a username and password (see this
) and have the two components that use MySQL
- that is, TiledImage
and the GBrowse DBI MySQL
adaptor - use a username and password for database access. The username and password for TiledImage
have to be manually specified in TiledImage.pm
(although this may be fixed in near future) where the DBI->connect()
function call is executed. The username and password for the DBI MySQL
adaptor (if you are using it, and not the memory adaptor) is specified in the .conf
If you are just running this database locally for the purposes of generating tiles and aren't worried about security, you can simply grant all privileges to the user ''@'localhost'
(i.e. a no-name user on the local machine), which TiledImage
runs the query as by default. This is done by:
$ mysql --user=root
mysql> GRANT ALL PRIVILEGES ON *.* TO ''@'localhost';
root user is not the same thing as your system's root user, so don't make that confusion, and the same goes for all the other MySQL
users. When you just run mysql
without a --user
parameter, it will try to log into MySQL
using your system username, but the --user
parameter can override that. So any system user that has access to the mysql
binary can log in as the MySQL
root user (just like above) as long as you have the MySQL
root password (that is, if you set one up).
Lastly, for TiledImage
, you will need to add the database gdtile
before you can use it, which can be done by loading the TiledImageMySQLDatabaseSchema
. There's probably a more elegant way to do it, but personally I just start:
or, if you're not root (because you need to be the MySQL
root user to do some things):
$ mysql --user=root
and copy and paste the entire text of the schema into the mysql
SET UP BIOPERL AND OTHER PERL MODULES
Use CPAN to install pre-requisite modules by invoking:
as root on the command line, then typing in:
> install MODULE_NAME
in the CPAN shell prompt. This is what you need to install, in this order:
DBD::mysql (the tests for this MIGHT FAIL - see below)
Note that the DBD::mysql
module might fail the tests if you set up a password for the mysql
user, since the MySQL
daemon requires a password that the tests don't provide, but that is OK - you can either just force install DBD::mysql
in the CPAN interactive prompt and take it on faith, or provide a password for the tests (how to do that I'm not sure, although it's possible). Or alternately, you could remove all passwords from MySQL
for the purpose of the tests and have it run completely unsecured.
Now to configure BioPerl
, for which we will be using version 1.5.1-rc3 (get the file bioperl-1.5.1-rc3.tar.gz
) from here
. There are other versions out, but this writeup was written for 1.5.1-rc3, although the instructions here might be applicable to other versions also.
We must first use CPAN to install Bundle::BioPerl (the BioPerl Bundle), but heed this warning...
PROBLEMS MAY OCCUR HERE:
I had difficulty installing the BioPerl
bundle on a node with a complete re-install of CentOS
. The libPNG and GD libraries should be compiled from scratch before
you start installing the BioPerl bundle using CPAN! Get the libPNG source here
and the GD source here
. You will need zlib
libraries for both, but you should have the latest ones already if you followed the Installing CentOS On Cluster
instructions. If you have trouble, I would recommend trying this:
- Compile and install libPNG from source to /usr/local/, or whichever directory does NOT have the .a files for zlib, since that seems to cause problems.
- Go back into CPAN and force install GD, GD::SVG, XML::Twig, then Bundle::BioPerl, all in that order. Ignore the failed tests.
The bundle is installed as the other modules above.
Accept all defaults and install all modules that you are prompted for. I would recommend against running the SOAP::Lite tests (not running them is the default anyway) if you are behind a strict firewall, since it will fail due to that.
Now, at this point you can install the actual bulk of BioPerl
using CPAN also... but we won't do that, because the version of BioPerl
that we want isn't on CPAN, so we're going to do if from source.
(or your favorite BioPerl
version archive) and cd
into the resulting directory. Run:
$ perl Makefile.PL
Say "install ALL scripts" and "yes to BioGFF tests" when prompted. You may fail the BioGFF tests if your MySQL
database is under some sort of security protection, which is OK, as long as you know it won't interfere with TiledImage
later. The makefile will be generated, and you will get a list of missing modules at the end. For me, this list was:
GD::SVG (need to 'force install' this one)
XML::SAX::Writer (need to 'force install' this one)
If you think you will need any of these, you can install them using CPAN. I only installed GD::SVG, since a GBrowse feature needs it. However, the feature doesn't seem to work, and I haven't yet tracked down whether this is due to the same reason that I had to 'force install' the module.
$ make test
and, if all (or at least most... or at least the releveant) tests pass, run:
$ make install
which, for me, installed everything to /usr/lib/perl5/site_perl/5.8.5/
Lastly, we can use CPAN to easily install some optional modules for GBrowse and BioPerl
, which are:
XML::Parser (update: this is apparently already installed if you did all the above steps)
XML::Writer (update: this is apparently already installed if you did all the above steps)
XML::Twig (need to 'force install' this one)
XML::DOM (update: this is apparently already installed if you did all the above steps)
LWP (update: this is apparently already installed if you did all the above steps)
MOBY (I don't think this is on CPAN... see http://www.biomoby.org/GettingTheCode.html if so inclined)
and out of which I installed all except for MOBY
SET UP GBROWSE
$ cp Generic-Genome-Browser-1.62.tar.gz /usr/local/src
$ cd /usr/local/src
$ tar xvfz Generic-Genome-Browser-1.62.tar.gz
$ cd Generic-Genome-Browser-1.62
$ perl Makefile.PL
The above sets up the following default paths and settings on Red Hat-based operating systems (such as Centos):
which you can change as explained here
$ make test
(you should pass all the tests)
$ make install
CHECKING OUT generate-tiles.pl USING THE SUBVERSION SYSTEM
The script generate-tiles.pl
that you will need is part of the TiledImage
Subversion project, which you can check out of the Subversion depository as described here
. The write-up of how to use the script will come eventually.
A SPECIAL NOTE ON Bio::DB::GFF::Adaptor::mysql - NOTE TO SELF: MOVE THIS TO WRITE-UP OF HOW TO USE THE SCRIPT
You have the option of loading the contents of your GFF file into a database (for the purposes of this example, MySQL
). Sometimes this is necessary, as with the latest Drosophila melanogaster Chado release in GFF 3 format. The problem is that the bulk_load_gff.PLS
script on the SixtyFourBitCluster
does not seem to work properly (terminates because unable to read one of the temp files it creates... which are created just fine, but seem to disappear too early). But it does work on xander
, so what we do is generate the MySQL
database containing GFF data for Drosophila on xander
, then back it up using:
$ mysqldump fly > NAME_OF_SAVE_FILE.sql
where "fly" is the database name. What this does is generate and write to standard out (hence the redirect to a savefile) all the SQL syntax statements that you need to load every little piece of data into the database from scratch.
So, on the recipient end, you should create the database:
$ mysql --user=root
mysql> CREATE DATABASE fly;
mysql> GRANT ALL ON fly.* TO ''@'localhost';
and then just pipe the contents of the savefile to MySQL
$ cat NAME_OF_SAVE_FILE | mysql
which will execute every single SQL statement in that file, thus creating the necessary tables and filling them out with the data you backed up using mysqldump
- 02 Mar 2006
Copyright © 2008-2013 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback