AJAX GBrowse wish list

Feature and improvement wishlist for AJAX GBrowse (Prototype GBrowse, demo at http://genome.biowiki.org).

This is also the development outline; tasks are listed approximately in the order they need to be done.

High-level goals

At a high level, the goals of the project are as follows:

1. to reproduce the core functionality of GBrowse (particularly search and linkout);
2. to enhance useability via use of AJAX;
3. to enable users to upload features.

These are all "key deliverables" which we have, among other things, promised to do in our grant proposal.

Immediate priorities

• Search on feature names/IDs

• quantitative (wiggle) tracks (e.g., conservation, ChIP on chip)

• subfeatures (e.g., transcript models with intron/exon/CDS structure)

• community annotation (track upload)

Specific features/improvements to implement

Items with this label correspond to the high-level goals. Others are typically things that people have suggested at some point, but do not correspond to the "key deliverables".

Documentation

• We need to start using JSDoc for commenting Java Script and perldoc (?) for commenting Perl. Documentation generated in this manner should go on our documentation page.

• The entire documentation page is one gigantic "to do" list.

Client and server in tandem

GBrowse-like feature display

• Mouseover popup menus for features
  • to show more options, e.g. open another window with feature details, highlight/mark feature, etc.
    • For now, we can hack this together by having generate-tiles.pl output XML for each tile with feature coordinates (relative to each tile), then the client can convert that to divs with mouseover events and append them to relevant tile div. Caching is also easy in this setup (tile divs are already cached, so just fetch their features from appropriate XML files).

Search facility

• Search facility
  • search service on server, returns XML blob that is rendered by client as a GBrowse-like table (appended to client as a Component)

Annotation upload

• Annotation upload
  • need ability to easily upload features onto server, have it render them for you and display in the browser
    • able to http-POST a file in pretty much any common annotation format (GFF, BED, etc). See UCSC genome browser FAQ: adding custom tracks.
    • graphical front end for track configurations (pick how you will have your features displayed without writing config files)
    • draw/edit features (WYSIWYG genome editor), convert to annotation and post to server
  • Multiple people working on same feature set will require...
    • tracking system/logging of changes (accountability for modifying public data)
    • user accounts and privileges/rights/access control lists (who gets to change what info in a community project)
    • quality control/moderation
  • Classic multiuser/concurrency issues:
    • locking data that is being edited/in use
    • making sure you have the latest data (i.e. client will need to check for "dirty" tiles/features in its cache - tiles that were changed during the current session must be updated)

• Feature upload and community annotation: implementation details (they are tied at the hip)
  • Feature Upload Requirements
  • Feature Upload Design
  • What database, schema, protocol to use?
    • Something like Google AJAXSLT to vivify features thru XSL transformations, a la Google Maps
    • Google Maps-like API for affixing notes to tracks (examples of use of Google Maps)
    • DAS might play nicely with XSLT and SVG... as long as the JavaScript client doesn't have to do any layout (i.e. bumping)
    • JSON has been suggested

Lower-priority display functionality

• Labels are too dense when zoomed out too far; rendering process needs to drop the labels automatically above some density cutoff (the current GBrowse already does this, but how?)
  • could be fixed elegantly by fixing other "feature label" problems below - but for now, we should at least fix "ridiculous" layout problems by hardcoding a track height cutoff

• An overview track of the entire landmark
  • would be nice if you could drag a box to select a desired region, and the browser would zoom in/center on it for you
  • this would actually be nice on the main view also, not just on the overview - it should be trivial to port code from overview to the main view

• Feature labels
  • Current problem: the feature label binds to the beginning of the feature, which is useless if the feature scrolls out of view. Also, labels run off-screen at the very end of the landmark.
  • Solutions:
    • don't render labels into the tile (but leave space for it), but either to separate graphic, or print it to tile XML - then client can compute where to put it optimally

Other client-server wishlist items

• For plot-type tracks, must have ability to display as some other feature type (e.g. boxes) everything above some threshold.
  • Either pop up the toggled glyphs over the existing track, or pop open a new track.
  • ...or replace existing track (i.e. toggle between plot view and box view).

• Location of config XML file should be encoded in the URL instead of hardcoded on the server side.

Blue-sky items

• Synteny display/multiple genome browsing
  • could display multiple genomes as tracks, scrolled in tandem
    • currently, client assumes all tracks have same genomic coordinates - need to add code to track that separate tracks map to separate genomic coordinates (add a "synteny view" mode)

• Rendering the view to SVG for publication figures, etc.

• Dynamic auto-shifting of track heights to minimize whitespace
  • Amount of whitespace per tile conveyed in XML? Or retrieved from server via XMLHttpRequest function? (kept track of in "current view properties" client-side data structure)
  • We really want to keep the tracks "sliding" dynamically (a la Google Maps automated panning/scrolling) as the users pans left and right, nice and smooth

• remember the dotplot tracks?

Client-side only

• Stuff that should be done ASAP:
  • Preserve vertical state across zoom levels (also, vertical centering).
  • Open/close entire control panels, like in the original GBrowse (also, rearrange them, possibly by dragging something).
  • Rearrange tracks - done, but could use a pretty GUI around it, e.g. change order by dragging to rearrange the track buttons (trivial (?) if one uses a drag-and-drop library like Scriptaculous or Dojo).

• Flashy things (functionality more-or-less exists, but could be spiffed up)
  • Make the view height control a slider. Arguably easy/fun with Scriptaculous, Dojo, Rico, etc.
  • Make transparency control a slider.

• Make the vertical-only drag bar a proper slider (i.e. like the one application windows have: with up/down button, draggable thingie, clickable empty space, etc.).
  • No need to do this horizontal-only, that function will be done by the overview track.

• JavaScript optimizations (it's already getting bulky, and there are many things to come) - especially memory use!
  • some profiling should be done

• Dynamic construction of ComponentInterface from some set of included components
  • layout of what Components to use and in what order to be specified in XML config file

• Dynamic construction of the DOM tree (no more hardcoding things in the XHTML)
  • layout of what Components to use and in what order to be specified in XML config file

• Client state encoded in URL (allowing for bookmarking in Web browser, etc.)

• A history Component (to fix the problem that Ajax applications commonly break the Web browser's "back/forward" buttons)
  • it might be nice for the user to have a log of everything they've done, and ability to jump to any part of it... a la Adobe Photoshop's "History" box feature

• Automated bug reporting

• Pop-up arrow marks to show direction of transcription
  • often, there is an arrow on the end of a feature showing direction that it is transcribed in... but when you scroll that out of view, it goes away, so it'd be nice if something popped up making up for that info

• Lightweight JavaScript rendering of features on top of tiles (push-pins, etc.)

• Keyboard shortcuts a la GMail?

• Animate short jumps to a new view, a la Google Maps (pan the view smoothly, or something fancy like that)

• Better (or at least some) error-checking of config XML

• More internal error-checking (e.g. liberal use of try/catch blocks)
  • unexpected failures should submit an automated bug report (e.g. catch blocks should fire a bugAlert() call)

• Porting to other browsers, as development is currently for Firefox only (this one is going to be a doozy... will hold this off until "v1.0", as Ian says)
  • or maybe it is wiser to start doing this now, bit-by-bit? otherwise porting the whole thing will be a leviathan of an effort

• If we have any customizable user settings that can be saved... maybe save them using cookies (OK, this is really frivolous and far out there... that's why it's last in the list)

Server-side only

• Just-in-time server-side rendering. Pretty straightforward since we already have a persistent primitives database in Tiled Image. Probably need mod-perl to make it efficient. (Since we hypothesize that this will effectively remove the bottleneck to displaying large genomes, it's arguably the most important task right now!)

• Serving up a human chromosome: (i) as proof of concept and (ii) as a testbed for Bioontology stuff
  • Add more genomes to demo (e.g. HIV, anthrax)

• Rendering optimizations:
  • Slow step is database access, not fill - but would be nice to get some hard benchmark data
    • need to do more profiling of Tiled Image and database access
  • Gridline primitives are filling half the database, but are the same for each tile, so either:
    • put them in database once, then reuse for each tile (database is now half the size), or
    • don't render at all, instead make a background PNG with tile gridlines and just apply that to all tiles (tiles with features will have to have transparency)
  • Key for primitives on tile number, instead of bounding box (1 key is better than 4); for primitives that cover more than one tile, two ways to store:
    • Store primitive multiple times, one for each tile (wastes space, though)
    • Store each primitive once with a unique key, then have a second table mapping tiles to list of primitive keys; more space-efficient, but for each tile, you have to look up primitive keys, then look those up in another table
    • Or maybe create hash function to convert 4 keys to 1 integer key (or a string, but keying on strings is inefficient, no?)
    • Why do we need the Y-coord bounding box keys, though? For the general case they make sense, but this is a restricted case: the BB is completely defined by X coords (we do ALL Y coords for that range)... so there go 2 keys right there.
  • Try later version of MySQL, or PostgreSQL
  • Render larger tiles, then split up into smaller tiles - remember THIS?

• Code cleanup
  • Instead of having separate TiledImagePanel Perl module, incorporate the code into BioPerl's existing Panel module.
  • Fix up signal handlers in TiledImage (something there wasn't working right, needs to be remedied but it's been months since I've looked at it [AVU])

• Use a lightweight image server (e.g. Public File) for tile serving
  • but will this work with on-demand tile rendering, as tile requests would have to pass through a wrapper script first? no longer just an image server...

Ideas to investigate/ponder

• Use Java Script Canvas library; server will serve up only primitives (i.e. just the layout), all rendering will be done client-side (also can do a lot of post-layout that way, e.g. toggling feature labels on/off, recoloring/highlighting features).

• Have the client do all the layout, including reducing server rendering overhead by assembling the glyphs on the client.
  • Express each feature as a div element (or some such).
  • The server will render just a tiny "slice" (what we called "brick") of the feature glyp; the client will tile it within the div using background-image and background-repeat CSS properties.
    • So the feature will still be one DOM node, unlike with Canvas - less overhead.
    • For complex features that have no periodicity/can't be tiled, the entire feature can be rendered on server and served up as a single PNG.
  • Advantages:
    • Server side overhead drastically reduced, as features with periodicity do not have to be rendered at all - only the repeating element must be rendered for each glyph of that type.
    • Solves track label issues.
    • Gives client more freedom for layout (variable track heights can be done), also opening the possibility of using it as an editor.
    • Solves the problem of community annotation/upload changing the underlying data and forcing server to re-do the layout even if a minor feature was changed.
  • Disadvantages:
    • More JavaScript to write.

• Use SPARQL for mashups.

• We really need to consider the case when a user adds just ONE feature to an existing track... this feature may change the layout significantly (e.g. it may affect the entire track height), so how do we handle the problem of re-doing the layout of the entire track due to a change like that?
  • This may be an argument for client-side rendering - i.e. there is no fixed "track height", the client determines that dynamically for whatever local region it is looking at... the server-side just stores genomic feature coordinates, the client does the layout "live" and only locally. This also enables other fancy thing, e.g. dynamically changing track height to be tight to features.
  • IH: solutions include (i) forcing the client (or the user) to do layout; (ii) having a "publish" button, like on blogspot, that triggers a fresh layout on the server. These could work in parallel (i.e. for minor edits, the user must do layout; they can also request a completely fresh layout from the server, but this takes time; bulk uploads would get a fresh layout by default)

Known bugs/problems

Please use the SourceForge bug tracker (category GBrowse-ajax) to report bugs and problems.

AndrewUzilov says: exception to the above... I've come to the conclusion that the stuff below shouldn't be submitted using the SourceForge bug tracker, as it consists of either silly coding TODO's or ruminations on possible points of failure, not actual bugs. Some of these may be minor annoyances not even worth tackling.

Client-side only

• Shaky centerpoint when changing zoom levels?
  • probably artifact of imprecision... not worth fixing?

• Make mouse dragging release when leaving window (otherwise weird, weird things happen when cursor re-enters window)

• Need to prevent selection (highlighting) of items in divs somehow

• Cursor doesn't change to "grab hand" when dragging on certain platforms (i.e. Firefox on Win XP... I think mabye 1.0.7 or some other earlier version... maybe a non-issue for later versions?)

Server-side only

• No database cleanup when you Ctrl+C generate_tiles.pl... should have a signal handler to clean that stuff up so that database doesn't get clogged after botched runs.

• Ends of plus-strand features in "Genes" track don't render.

• Some way to get units on the ruler to display properly without hacking "arrow.pm" - we should use as much of the current BioPerl framework as possible

• Aggregators for Drosophila are not working properly.

• Loading the Drosophila data from the FlyBase Chado DB r4.2.1 release GFF file into MySQL seems finicky... I mean, are those labels even correct? How do we make this work with the aggregators?

Client and server in tandem

• ruler tiles follow a different naming convention than regular tiles, which is annoying... fix it!

• What will happen when there is no track name?

• What's this numtracks tag in the XML?

• config XML "nested zoom levels" schema is stupid and should be changed

• Many problems with "classic/original GBrowse button", e.g. it should pop up only tracks that are currently open
  • this is really tempting to not fix, as it is for demo only... maybe even a non-issue