Now that we’ve got node connected to our postgis database, we need to pull the route information into our sencha mobile web app.

I’ve pushed this version to the github repo, but I haven’t deployed it to nodejitsu, just yet.  My postgis hosting service requires that ssl be enabled, which is supported by the native version of the postgres module.  Unfortunately I can only get the pure js api to work on nodejitsu.

Rather than using jQuery or even Ext to make ajax requests, I’m going to be using socket.io, which is a websocket like implementation for node.

passing around data with socket.io

We are triggering the ‘get-routes’ socket.io event on the backend, which takes a lat/lon and returns a list of routes to the callback.  This allows the view to be updated asynchronously by adding the new routes to the store and refreshing the display.

var refreshRoutes = function(lat, lon) {
                socket.emit('get-routes', lat, lon, function(routes) {
                    Ext.each(routes, function(route, index) {
                        App.routeStore.add(route);
                    });
                    App.routeList.refresh();
                });
            };

sencha geolocation and data request

I’ve added in the geolocation api that is part of sencha.  The app is refreshing the routes when it has a lock on the gps.  Sencha’s geolocation api makes this sort of thing easy.  Note that I’m not really handling any errors here, and they definitely could crop up. The following goes into the app’s onReady method.

var geo = new Ext.util.GeoLocation({
            autoUpdate: false,
            listeners: {
                locationupdate: function(geo) {
                    refreshRoutes(geo.latitude, geo.longitude);
                },
                locationerror: function(geo, bTimeout, bPermissionDenied, bLocationUnavailable, message) {
                    if (bTimeout) {
                        alert('Timeout occurred.');
                    } else {
                        alert('Error occurred.');
                    }
                }
            }
        });
        geo.updateLocation();

project status and next steps

I’ve got the routes view rendering and I’ve added disclosure icons that switch the focus to the comments view.  I gave up on putting the map on the start screen because I had some trouble getting those components to share a card.  The next arrival bar on the bottom is mocked up for now and will require additional backend work.

I’m also thinking of scrapping comments and just focusing on the other two components.  That would free up some space on the toolbar and reduce the scope of the project.  My next step is to split the chat channels based on trimet routes and wire up that view.

I’ve written my spatial query and it returns a an array of the 10 route lines that are closest to a point. My query retrieves the route number, description and distance in decimal degrees. I’ve projected the trimet data to the WGS84 projection, which is what we usually get from a GPS.

a note about projections and web mapping

I’m sure that the GIS people are nervous about my cavalier treatment of projections. I didn’t mention anything about spheres vs spheroids or anything like that. If we were doing serious analysis it would be unacceptable to pay so little attention to the underlying data. However we are building a web app where speed and ease of use trumps accuracy. There are compromises that have to be made for this sort of thing. It is good to be aware of the tradeoffs, but I’m not too concerned.

a trimet node module and git branches

I’ve added a some node module to the repo to hold the server side application code for interacting with postgis and the trimet data. I haven’t worried too much about organizing my code, yet, but it seems like a good time to impose some structure. I’ve also added a test directory and an integration test that actually runs the query against our database and verifies that we get the expected results. This will come in handy as things get more complicated.

git flow and development branches

You may also notice that the git repo contains a develop branch in addition to our master. The master branch holds our current stable deployed code, and new code gets added to develop.

When I am ready to release I merge the develop branch into master to be deployed.  Git makes it easy to branch and merge.  I’m also using the flow extension to git to make it easy to manage the branches.  I highly recommend this approach.

trimet node.js module

In the development branch, I’ve added a new javascript file called lib/trimet.js.  It exports one function to our server that takes a latitude and longitude and returns a list of routes from the postgis database.  It is using the node-postgres module by Brian Carlson and used in production at places like yammer.

This integration test demonstrates the use of trimet.getRoutesByPoint, our first module function. It uses mocha, an excellent node test framework that I wrote about in article on asynchronous tests with node and redis.

preventing sql injection attacks

Note that I am creating the query string by concatenating the coordinates. This opens us up to sql injection attacks and should raise a red flag for security. The node postgres module supports parameter replacement, but I was running into trouble with that.

In order to prevent random sql from being executed on our database, I’m testing to make sure our latitude and longitude are actually numbers before passing them to the module. Really this method should fail if we pass it anything but numbers. We’ll address that later.

var trimet = require('../lib/trimet.js'),
    should = require('should');

describe('trimet: get nearest routes from coordinate', function() {
    describe('getRoutesByPoint', function() {
        it('should return some points', function(done) {
            var lat = 45.499541,
                lon = -122.613639,
                expectedRoutes = [{
                    route: 9,
                    description: 'Powell/Broadway',
                    distance: 0.00200889736828729
                }, {
                    route: 14,
                    description: 'Hawthorne',
                    distance: 0.00229043846550735
                }];

                trimet.getRouteByPoint(lat, lon, function(err, data) {
                    should.not.exist(err);
                    should.exist(data);
                    data.should.eql(expectedRoutes);
                    done();
                });
            });
        });
    });

Now that I’ve got the comment view more or less working, it’s time to link it up with actual trimet routes.  I think we need an initial screen to show to users when they first bring up the app.  I imagine that people will be either riding the bus/max, or they will be waiting at the stop.  Seems like we need to start by showing routes that are closest the user’s current location.

performing spatial queries in postgis

Our backend needs to support a method that takes a latitude and longitude in the commonly used WGS84 projection and return the routes that are closest to that point.  Just getting the nearest stops isn’t useful because the rider may be actually travelling the route and not anywhere near a stop.  Luckily postgis makes it easy to select lines based on proximity to a point.  Please see the note from Regina in the comments for updated postgis functions.

SELECT "RTE" as route,"RTE_DESC" as description,"DIR_DESC" as direction,
	distance(PointFromText('POINT(-122.613639 45.499541)', 4326), the_geom) as distance
from tm_routes order by distance limit 10;

This query returns the following results when applied against the postgis table I created from the trimet shapefile projected to WGS84. Note that postgres requires that we double quote the column names because they are capitalized.

 route |      description       |           direction            |      distance       
-------+------------------------+--------------------------------+---------------------
     9 | Powell/Broadway        | To Powell & 98th or Gresham TC | 0.00200889736828729
     9 | Powell/Broadway        | To Saratoga & 27th             | 0.00200890251737265
    14 | Hawthorne              | To Foster & 94th               | 0.00229043846550735
    14 | Hawthorne              | To Portland City Center        | 0.00229044244641688
    71 | 60th Ave/122nd Ave     | To Foster & 94th               | 0.00460586966268184
    71 | 60th Ave/122nd Ave     | To Clackamas Town Center       | 0.00460588437600594
     4 | Division/Fessenden     | To Gresham TC                  | 0.00575370636240205
     4 | Division/Fessenden     | To St Johns                    | 0.00575370636240205
    75 | Cesar E Chavez/Lombard | To St. Johns                   | 0.00898150226029743
    75 | Cesar E Chavez/Lombard | To Milwaukie                   | 0.00898150543661629
(10 rows)

grouping results in sql

The routes returned look great for my current location. The data is accurate enough to differentiate between routes that run on the same street in different directions. If I were waiting at a bus stop, it would be a safe bet to assume that the bus I want is on the same side of the street as me. I still want to return other routes, but at this point I don’t really care what direction they are going. I want to group the results and just return a single row for each route. We can achieve that goal by tweaking the sql. We just need to apply an aggregate to the distance and group the results by route. We could filter the results by distance, but there isn’t really a need at this point.

SELECT "RTE" as route, "RTE_DESC" as description,
	min(distance(PointFromText('POINT(-122.613639 45.499541)', 4326), the_geom)) as distance
from tm_routes group by route, description
order by distance limit 10;

route |      description       |      distance       
-------+------------------------+---------------------
     9 | Powell/Broadway        | 0.00200889736828729
    14 | Hawthorne              | 0.00229043846550735
    71 | 60th Ave/122nd Ave     | 0.00460586966268184
     4 | Division/Fessenden     | 0.00575370636240205
    75 | Cesar E Chavez/Lombard | 0.00898150226029743
    66 | Marquam Hill/Hollywood | 0.00898150861891694
    17 | Holgate/NW 21st        | 0.00924074516961275
    10 | Harold St              |  0.0148568363291083
    15 | Belmont/NW 23rd        |  0.0170180326549672
    19 | Woodstock/Glisan       |   0.020370095002805
(10 rows)

putting it all together

I’m happy with these results. The next step is to wrap it in a method and expose it on our backend. Then I’ll write a new Sencha touch view to request the data based on the coordinate returned from the geolocation api in the browser. Should be a snap.

So far I haven’t spent much development time working on the backend for seatMate.  I mentioned that I was using node.js to serve up the static files on my development server and nodejitsu.  The backend also saves and retrieves comments from a list stored in redis.

backend architecture choices, async events with node.js

The great thing about node and redis is that they support asynchronous operations in the same way that javascript handles events in the browser.  To make things even more responsive, I’m using socket.io to pass an object to a method on the server, rather than using an http post when the client submits a comment .  socket.io acts like a websocket, but degrades to flash sockets, long polling and other less fancy methods to support all browsers including IE6.

In a previous post, I wrote more about node and redis.

pushing events back to the browser with socket.io

The benefit to using socket.io and node  is that I can trigger events in the browser from the server without doing any polling from the client.  This will be handy when it comes to the chat and instant messaging components, which update in real time.  When a user sends a message, the backend can immediately relay it to any other device connected to the channel causing it to appear in their browser.

using redis to store and relay information

redis is the other technology I added to the project.  It is one of the new nosql datastores, but it also supports other features that will make it useful here.  redis is a data structure storage engine and can handle strings, hashes, sets and sorted sets.  It also has a publisher/subscriber functionality that makes it a snap to implement a messaging queue.

For the comment view, I’m pushing incoming messages into a redis list and retrieving the last 15 messages when rendering the comment view.  All these operations are extremely fast, even if the list gets huge.  No need to talk about the big o, but it will stay fast at scale.

performing spatial queries on a geodatabase with postgis

The next step is to actually tie the comments to specific bus routes and busses.  For the contest I have to use TriMet data.  The stops, routes, and possibly the real time transit feeds are exactly what I need.  To start matching up users with routes, I need to be able to select the route that is closest to the location of the the rider.

It would be trivial to perform this sort of operation with points, although scaling might get difficult.  Calculating the distance to a line from a point is an easy to do with postgis, the geodatabase extension to postgresql.  My next step is to create a method that queries postgis for routes that are closest to the location of the device as reported by the geolocation api.

In cases where the user is on a bus that travels the same path as our routes this operation will return multiple rows and, I will have to use other data to disambiguate the results.  Just grabbing the closest route is good enough for now, though.

After spending a day watching Sencha Touch screencast videos, I’ve got my comment view cleaned working and deployed to nodejitsu at http://seatmate.nodejitsu.com.  Basically the same code should make the chat and instant messaging views function as well.  Feel free to file bug reports on the seatMate github issue tracker.

I’ve got a Sencha list view with an xtemplate bound to a json store.  There is a form panel that dispatches to a controller for posting messages to a single comment channel on the backend.  When the page loads, the datastore is prepopulated with the last 10 or so  comments on the channel.  The list view scrolls independently of the rest of the page.

New comments entered through the form are added to the view immediately, but comments posted on other devices are not displayed automatically.  That would be what makes the feature that differentiates the comment view from the chat view.  I also need to tweak the css to make the comments smaller and have more of them on the screen at once.

The app works on Android and iOS browsers, as well as desktop webkit browsers like Chrome, but has some trouble with Firefox.  Unfortunately that is a problem for all Sencha Touch apps.

Development was good considering I also had to learn Sencha/ExtJS.  Unfortunately I don’t have any javascript unit tests.  I have a hard time doing TDD when I’m using a new framework, but I’d be able to do it next time.

I’m going to finish the app in Sencha, but I feel like it isn’t the best approach for mobile web app development.  I’m writing a lot of code for what I’m getting and I prefer to build my interfaces in html rather than javascript.  The framework does provide a slick product and handles a lot of the mobile pain, though.

I’d also prefer a solution that allowed the same app to run on mobile, tablets and desktops with responsive design.  Unfortunately sencha touch apps are relegated to the mobile ghetto.  But I feel like I’m making good progress overall.

creating a mobile web app from a mockup

I’ve started on the interface using sencha touch, a mobile web framework.  Sencha Touch is part of the ExtJS family of javascript libraries.  I’ve hard good things about sencha and GeoExt, so I’m excited to give it a shot.  The style is similar to how we used to develop web apps at the company I work for, but we’ve since moved on to using knockout.js.

the sencha touch approach

Sencha lets you build up a mobile web frontend programmatically rather than starting from existing html.  You create objects that represent forms, tabs, carousels and everything else you need to develop a smartphone or tablet app.  Sencha produces html5 and supports many of the animations and transitions that are used in native iOS and android apps.

I’ve made good progress and sencha seems robust and flexible.  My main concern is that the workflow is clumsy.  Rather than writing html to create the interface, you describe it in javascript.  I would rather create the page freely and then add behavior.

Using code inspired by several of the online demos, I’ve put together a sencha app with an initial stab at a chat/comment view.  I have a comment entry field and a submit button as well as a table of comments.  The other tabs will follow the same general pattern, but aren’t built yet.

 serving the static html, css and js files

Since the files are static html I can view them directly in a browser without spending time setting up a webserver.  However, I’m using a snippet of javascript for node.js that serves up the contents of the static directory on port 8000.  I’ve pushed this version to the repo and deployed it to nodejitsu.

deployed to hosting provider

You can check it out at http://seatmate.nodejitsu.com/

I’ve also attached a few more mockups of the other screens.  They are in landscape and portrait mode.

first mockup for comment tab

For usability’s sake, web apps tend to use familiar components.  There are a number of design templates that make it easy to get a an application up and running without spending a lot of time wrestling with html and css.

One of the problems with using these templates, is that your application will end up looking like every app that uses the same template.  At some point I will make the application look more unique, so I’ve added “not cookie cutter” to the requirements.

Here is my first drawing for the comment component of seatMate.  It is basic, but should provide enough to get started.  Please pardon my messy drawing!

repo

I’ve added this mockup and an initial skeleton for the app to the seatMate github repo.

2012 TriMet App Contest

I’ve decided to compete in the TriMet App Contest.  The deadline is February 1st, 2012, and I am starting this weekend.  The app must use TriMet and CivicApps open datasets and be released as an open source project.

Rather than keeping my app under wraps until the end date, I’ve decided to do all the design and development in public.  I hope the other entrants will use my code and methods to develop their own cool apps.

My App: seatMate

For this contest I will build a chat and messaging app for TriMet passengers who are riding the same bus or route.  It will be targeted at riders with tablets and smartphones and use the TriMet route and arrival data to enable social interactions between passengers.

Leave a message/comments

Passengers will be able to leave message or comments that will be visible to other people riding the same bus or route.  They could be used for lost and found or “missed connection” type communication.  They will be public, archived and searchable.

Chat

There will also be the option of partaking in a chatroom or private instant messaging with other people riding the same bus or route.  There should be some expectation of privacy/anonymity for users.

Flirting

The third element of the app will be to facilitate romantic interaction between riders that opt in.  People that are interested in meeting other passengers can activate a “flirt” mode, that will enable private messaging between interested parties.  Users will only be identified by age/gender/sexual preference.

Requirements

seatMate will  be a mobile html5/javascript web application and also available as a native Android/iOS app from the app stores.  It should:

• be fast and interactive
• use geolocation to determine route information
• have push notifications or sms
• allow image uploading/sharing
• deployable to cloud hosting (ideally at the freemium level)

Links

repo

The contest specifies that the app must be available as a public repository.  I’ll be using git and host the code on github.  This will allow for public issue tracking and enable anyone to fork the repo and submit code through github pull requests.

https://github.com/eknuth/seatMate

trello

For project management, I’ll be keeping big picture task information in Fog Creek’s trello app.  It’ll be public and allow voting and comments.

https://trello.com/board/seatmate/4f081075ffd434f16d04b1bc

blog

I will be blogging about the process on this site using the seatMate tag.

http://iknuth.com/tag/seatMate/

Feel free to follow along and participate as much as you like.  I’m hoping that this will be a fun month!

overview

node.js

Node is currently the cool kid of the web dev scene and is starting to take over.  Node is the project that lets you run javascript on the server side in a super fast v8 based vm.  Out of the gate, it is evented/async, non blocking and ridiculously fast.  Javascript has a lot of haters, but it is undeniably an interesting and powerful language.   The node community is thriving and mature.  The tools are awesome and it is a blast to use.

redis

redis is amazing. It is one of the stars of the nosql movement. It is a server that stores strings, lists, sets and hashes. If that isn’t even it also lets you do publisher/subscriber to trigger events. It is a natural fit for node and if you try it, you’ll like it.

mocha

Mocha is the latest generation of node/js test frameworks by TJ Holowaychuk. It is fast and allows for incredibly easy asynchronous testing. It supports standard assertions as well as bdd style tests using should.

setup node and mocha

This gist uses homebrew to install node. OSX has great support for node and homebrew is by far the best package manager. npm is among other things the command to install node packages. We’re using the -g flag to npm to install the package globally, otherwise it would be installed for just this project.

# install node 
brew install node

# install redis
brew install redis

# install npm
curl http://npmjs.org/install.sh | sh

# install mocha and should for testing
npm install mocha should -g

# install node-redis to talk with redis
npm install redis

# create the first test
mkdir test
touch test/test_redis.js

Now, if you type “mocha” at the command line, it will execute the non existent tests in our test directory and return the results. Since we have no tests, it should look something like this:

Async testing with mocha and should

The following gist is a basic asynchronous test using mocha and the should bdd style assertion library. Async tests can be challenging because you are testing things that happen at various times. We are going to set a key and make sure the callback function gets called without an error.

Generally that is hard, but mocha gives us a sweet little callback called done that we can pass to our redis set command. It ensures that the callback happens successfully and no errors are passed. If you’ve done async testing in the pass, you will probably be amazed at how painless it is to do it like this.

In the second test we are going to retrieve the key we just set and explicitly test the value in the callback. To make sure the async happens as we expect, we manually call done() after our assertions.

var redis = require("redis"),
    should = require('should'),
    client = redis.createClient();

describe('strings in redis', function() {
    var keyToSet = 'bingo',
        valueToSet = 'the dog';
    describe('#set()', function() {
        it('should save without error', function(done) {

            client.set(keyToSet, valueToSet, done);

        });
    });

    describe('#get()', function() {
        it('should be able to get the value we set', function(done) {

            client.get(keyToSet, function(err, data) {
                should.not.exist(err);
                data.should.equal(valueToSet);
                done();
                
            });
        });
    });
});

Just remember to say “no” to coffee script.

django and s3

This project doesn’t necessarily require heroku, but I’ve been interested in the service as an easy way to deploy apps. I really like the idea of not having to set up a server with nginx, postgres, gunicorn, etc. The convenience of deploying just by pushing from git is awesome.  

One issue I ran into was a problem with serving static content like css, javascript and image files from heroku. I’m sure it is just a configuration problem, but this solution of using s3 to serve those files is the way you want to go in the long run.

You can clone my sample project from github.

git clone git@github.com:eknuth/django-heroku-s3-bootstrap-demo.git

Django on Heroku

Heroku used to be limited to ruby apps, but it has recently become much more flexible with the addition of the cedar stack. Now it supports python 2.7 and uses pip to install modules from a standard requirements.txt file.  They have also started offering a free 5mb postgresql db to go along with their current freemium offering, which is fantastic.  That doesn’t sound like a lot of space, but you’d be surprised how much information you can cram into 5 mb.

Initial Setup

You can follow the django instructions from heroku here:
http://devcenter.heroku.com/articles/django

If you are using macosx, or virtualenv uses python 2.6 by default, I would suggest specifying 2.7 when you create the environment.  Heroku is using 2.7 and will complain about pip installing importlib which was not included by default in python 2.6.

virtualenv --no-site-packages . -p /usr/local/bin/python2.7

Open the site on heroku using the “heroku open” command and verify that you get the django welcome screen.  You should have no urls configured.

Admin and Static Files

You should now have a working empty project.  Let’s turn on the django admin app and fire up the development server.

As a side note, this config will import settings from settings_local.py, if it exists and overwrite the ones in settings.py. This local settings file is not committed to the repo and won’t get deployed to heroku.

Follow the instructions to install the django admin app and then run manage.py syncdb and then runserver.  If everything went ok, you should be able to log in to your admin console.

Now, commit and push your app to heroku.  Run the manage.py commands on heroku and magically your admin console is live on the web.

Twitter Bootstrap and view

Now, lets getting started with a template and a home view.  We can use the totally awesome twitter bootstrap project.  They have released a site/webapp html template, css and javascript package that provides everything you need to get started.  It is great for those that are less inclined to design.

Create a templates folder in your project root.  Clone the project from github and copy hero.html from the examples directory into your new templates directory.

@render_to

Now let’s create a view and wire up that template to a url.  Now is the time to add on a django project that will make things easier for us.  I really love the render_to decorator from django-annoying.  Run “pip install django-annoying”, and it will be ready to go.  Don’t forget to save your updated requirements with a “pip freeze >requirements.txt”.

relative paths

Add the template dir and a little snippet of python to use relative paths. This will make life easier in the future.
In the urls.py file, uncomment the following home view: url(r’^$’, ‘om.views.home’, name=’home’), And create a views.py file to grab the template and render it.

Now when you open the app on your development server, you get an actual “hello world” page. Unfortunately we are missing the css file, so it is unstyled. In your root project, create a static folder and inside that create another folder named css. Copy bootstrap.css into static/css.

To reference this css file, edit hero.html and change the line with bootstrap.css to this:

<link href="{{ STATIC_URL }}css/bootstrap.css" rel="stylesheet">

In order to make the STATIC_URL template tag work, we need to update STATICFILES_DIRS in settings.py to:

STATICFILES_DIRS = (        os.path.join(SITE_ROOT, 'static'),)


We also need add some template context processors with this:

TEMPLATE_CONTEXT_PROCESSORS = (
'django.contrib.messages.context_processors.messages',
'django.contrib.auth.context_processors.auth',
'django.core.context_processors.static',
)


When we open the page served from our development server, we are good to go. The page is styled and looks awesome.

Static Files on Heroku

Unfortunately when we push to heroku and open the page, we are back to the our css free look. Seems like heroku has no problem serving up the admin static files, but it has a problem finding our static directory. I’m sure it is just a matter of configuration, but honestly we really don’t want to use our application server to serve up static files anyway. Images, css and javascript should all be coming out of something fast and light.

Enter S3

Amazon provides an excellent service for doing just that. Amazon’s web services and the simple storage system are cheap and easy to use. There is also an excellent module for using s3 with django’s static files app and the collectstatic management command with s3 as a backend. django-storages makes it incredibly simple to use s3 for our static files.

django-storages and boto

First we want to install the module with “pip install django-storages” and add ‘storages’ to your INSTALLED_APPS tuple and set your storage type with:

STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

We will also need to install boto, the python module for dealing with s3. Use “pip install boto” for that.
Log in to the amazon web services management console and create a new bucket. Add the name of the bucket and your access keys to settings.py like this:

AWS_ACCESS_KEY_ID = 'ACCESS KEY'
AWS_SECRET_ACCESS_KEY = 'SECRET'
AWS_STORAGE_BUCKET_NAME = 'django-blog


set up the s3 bucket

Look at the bucket properties and grab the url for the bucket. Update your STATIC_URL setting to point to the bucket. In this case it will look like this:

STATIC_URL = 'https://s3.amazonaws.com/django-blog/'


deployment

Now run manage.py collectstatic and you will see all the files being copied to your bucket. Freeze your requirements file and commit your repo. Now when you push to heroku, you should see your website with the bootstrap.css file pulled from s3.