Monday, October 14, 2013

Automatically create super user when running syncdb

MVP of the new world changing app is getting sculpted, you are still some weeks away from using migrations (aren't we always) and DB gets deleted even before it has time to say hello to it's neighbors on the disk, only to be recreated with another set of tables. Well, it is all fine and dandy for syncdb is smooth as silk expect for that annoying prompt to create super user! If I had a customer for every time I typed in admin/admin/admin/admin on terminal, I would receive VC funding faster than I can say funding.

But since we are not even getting a feature done leave aside getting a customer by doing this boring shit, here is how you get rid of it. You use fixtures. Next time you have created the super user, dump the auth tables thus:
$ django-admin.py dumpdata auth > initial_data.json
And next time run syncdb like this:
$ django-admin.py syncdb --noinput

And it is smooth as a chin shaved with Gillette Mach 3. Or Philips if that is what you prefer.

For better organization, do not forget to move it to a fixtures directory and add that directory to FIXTURES_DIR variable in settings.

Happy DB deleting!

Credits: Stackoverflow

Wednesday, September 25, 2013

Supporting IP address ranges in Django Internal IPs setting

Django internal ips is one of those pesky little details that turn out to be the reason for your css not loading in your web app after you restarted your machine. Here is the chain of events:

Restart -> DHCP -> IP change -> no longer an internal IP -> DEBUG is false -> your templates render as if on production -> you need css instead of less.

Unfortunately internal_ips can only take individual IP addresses and not a range. So if you decide to package your app into a nice vagrant VM for distributing to your colleagues, when they run and try to access it, again the same problem.

Solution is simple. After all it is Python we are talking about. :)

Inspired by http://dancarroll.org/blog/2011/01/debugging-django-dev-server/, I included a container derived from list that answers yes to the membership question whenever the given key matches one of the patterns in the list.

from fnmatch import fnmatch
class globlist(list):
    def __contains__(self, key):
        for pat in self:
            if fnmatch(key, pat): return True
        return False
# Put the whole internal range as internal ips
INTERNAL_IPS = globlist(['127.0.0.1','192.168.1.*'])

Voila!

Sunday, June 16, 2013

HATEOAS

Got one version of the HATEOAS working but it doesn't look like it will go very far.

Basically while initializing the collection of pages in a publication, I do it manually and use the URLs provided by the API to set the URL on each of the pages model. Later when a publication is requested, I trigger the fetch on all the page models.

The problem is there are usually a lot of pages and all these calls take time. It looks funny when the starting view contains the default title for all the pages and slowly they are replaced by the actual title as they are loaded. I think it would be a better experience to show only few pages to start with and keep populating the list as additional pages are loaded. Perhaps it is better to modify the API or ignore the pages URL all together.

Saturday, June 15, 2013

Handling HATEOAS with Marionette

Till few days ago, I was an innocent web developer building a REST API. Then I came to know about HATEOAS and my life was never the same again.

HATEOAS essentially means that the API consumer should be able to navigate the API simply on the basis of information provided by the API and should not require out of band information. One example is the end point discovery. When you fetch resource A that is related to several instances of resource B, the url to API endpoints for resource B should be included in the server response.

Currently my Publication object returns with the links for its constituent pages. Now I am trying to figure out the best way of taking these URLS and using them in backbone to load the individual pages.

Next week's aim is to spend time on Page editor and finally get the template thing working. Finally we are coming to the point of this whole exercise. Warm up is done and the real play begins!

Friday, June 14, 2013

Bootstrap tabs in a Marionette view

Not much progress today but after couple of blind alleys, managed to get the bootstrap tabs working in a Marionette Layout. It mostly works out of the box since there is no removing elements from the dom business here unlike modal dialogs. So you just wrap the jquery plugin for use in rjs and then correctly markup the html in template.

Thursday, June 13, 2013

PushState, RequireCSS and RequestResponse

Today was the day of lots of new things. As I mentioned yesterday, I was in the market for something that will help me load my CSS.

While I agree that requirejs is a great tool to manage js dependencies, lately I am feeling that it leaves the job half done. A modular piece of JS, in many cases, will come with accompanying CSS and html templates. For example, almost every jquery plugin has a JS and a CSS file. Requirejs provides no API for declaring dependency on a whole module.

One way to achieve this with rjs is as following. First find a rjs plugin that loads html templates and css. Then write a wrapper module declaring dependency on both js and css. Now use this wrapper module wherever you needed the original. But this means writing a wrapper for every plugin or library you want to use. Given that rjs wants one module per file, this can mean a lot of files! One way of working around that is:

define('external', [], function() {
    var exports = {};
    exports.plugin1 = {
      ...
    };

    exports.plugin2 = {
      ...
    };

    return exports;
});
Then we can add a dependency on 'external' and use the required module. But now every external module will be loaded as a dependency for your module which is again not very palatable.

Loading CSS using RequireJS

For loading CSS, there are some plugins available. The primary issue is that a lot of modern browsers seem to be lacking a onload event on link elements. This means while it is easy to add a link element in the dom and thus load the CSS, there is no way to know when the sheet has loaded. One workaround is to keep polling the dom stylesheets to see if the new sheet has come in. The other approach is more clever. You create a script tag and use the css url as src. Browser loads the css and tries to interpret as a script and throws an error. So the onerror event on script tag can be treated as the onload for the css file.

There are bunch of drawbacks. One is that onerror gets called for 404 errors also. This means that the sheet was not found. This can be overcome with checking for the presence of stylesheet in the dom.

The other issue is that invalid script tag throws errors in the console. The workaround for that is to use img tag instead of script tag. With img tag, there is a non-threatening warning in the console and that's it.

There are couple of rjs plugins available that implement the above ideas: RequireCSS & CSSLoader. I am using a fork of RequireCSS that uses the img tag technique described above.

RequestResponse

RequestResponse is a simple object on which you can do two things:
  • set handlers for particular requests by name
  • make a particular request. The handler for the request will be executed and response returned.
Marionette applications come with a RequestResponse object on them. One use I am making of them is to make a central datastore for my models and collections. All other parts of application request the data from this central place. Updates are also done in the datastore. Other components use events to indicate what needs to be done.

This means any changes to a model or collection are immediately propagated to all the views. It also helps in supporting both the option of getting data: through a bootstrap variable dumped by the server in initial file or through a fetch from server.

// datastore module

var store = {};
// The library for the current user
function getLibrary(){
 if(!store.library){
  if(typeof pubs === 'undefined'){
   store.library = new Library();
   store.library.fetch();
  }
  else{
   store.library = new Library(pubs);
  }
 }
 return store.library;
}

var reqres = new Backbone.Wreqr.RequestResponse();
reqres.setHandlers({
 "collection:library": getLibrary,
 "model:publication": getPublication
});

return reqres;


// using the datastore in a view
// Request the library collection to be displayed
this.collection = DataStore.request("collection:library");

This pattern is useful when trying to build support for pushstate in your application. In case of pushstate, the view can either be rendered directly from the server or it might be reached through some interaction on the frontend. If the view is being rendered on the server side, it makes sense to provide the bootstrap data along with the template thus avoiding a extra call for data fetch. On the other hand, if we reach the view through some frontend interactions, we have to fetch the data from server through an AJAX call. A central datastore encapsulates the logic of doing the right thing in both the cases.

Wednesday, June 12, 2013

We have routers

I am now beginning work on the second major screen of the app which is the writingdesk. So the first thing to do was to get a router for our app.

Router in Backbone is the same thing as the router in Django or other frameworks. It's job is to load the right view depending on the URL being visited. The URLs can either look like 'http://example.com/#dashboard' or 'http://example.com/dashboard'. If you, like me, find the first ones ugly, it is easy to switch to second kind. Just pass pushState as true when instantiating the router.

The catch is however that since the user can now bookmark this url or hit refresh, your server will also need to be ready to serve this URL if asked for. That was easy to achieve with my limited number of views and screens.

Currently I am trying to be figure out a good way for 1) loading of CSS files 2) loading a large template for the second screen. I had hoped that requirejs would have a plugin for css files as well but turns out that detecting when a CSS file has loaded is not possible with much reliability. Hence there is no official css plugin for requirejs.

Tuesday, June 11, 2013

You cannot REST all the time

In any app of reasonable size, there are going to be API endpoints other than the REST API points corresponding to resources. For example, there may be a special endpoint for importing files which may lead to creation of one or more resources.

It is however a good idea to have these endpoints speak the same language of codes and return values as the REST endpoints thus making the whole thing more consistent. With some tweaks to the aging FileUploader, which included adding support for csrf token in AJAX request, treating 201 as a success code, I have my import endpoint pretty close to create endpoint.

I also gave up a little more of my fear and removed the second call to the server for fetching the models in case an event of imported:publicaiton is raised. Now we are living the JS dream where we speak as infrequently to server as possible and be very snappy.

Time has also come for our app to get some routes.

Monday, June 10, 2013

Regarding callbacks when saving a Backbone model

This one stumped me for a while. Backbone allows you to specify success and error callbacks when saving a model. But if you go by the documentation, this is how you create and save a new model:

submitForm: function(e){
 e.preventDefault();
 
 var pub = new Publication({
  'title': this.$('input[name="title"]').val()
 });
 pub.save({
  success: this.onSuccess
 }) 
},

onSuccess: function(model, response, options){
 Vent.trigger('imported:publication');
 this.$('form').trigger('reset');
},

But with the above, while the model will be correctly saved, the success callback will not be called. Reason being, the first argument to the save call is mandatory unlike what the documentation says. So something like the following will work:

 pub.save({}, {
  success: this.onSuccess
 }) 

Notice the empty object being passed as the first argument. You can also pass null if you like that but I still get nightmares of NullPointerException from my Java days. So I try and steer clear of it.

The next problem you may run into is that `this` does not seem to be set to correct value in the callback. In general, any callbacks being called from non-backbone or marionette code will suffer from this problem. This can be handled by a call to bindAll while initializing the view:

initialize: function(){
 _.bindAll(this);
},

Another thing you will sooner or later start missing here is the lack of a 'onComplete' callback that gets called irrespective of success and failure. This callback is useful for things like enabling the submit button that you may have disabled on submission or removing the waiting icon.

Fortunately, the save method returns a jQuery deferred object. So we are free to set our own onComplete callbacks on it:

formSubmit: function(e){
 e.preventDefault();
 
 this.$('button').hide();
 
 var pub = new Publication({
  'title': this.$('input[name="title"]').val()
 });
 pub.save({}, {
  success: this.onSuccess
 })
 .complete(this.onComplete);
 
},

onSuccess: function(model, response, options){
 Vent.trigger('imported:publication');
 this.$('form').trigger('reset');
},

onComplete: function(){
 // Display the submit button again
 this.$('button').show();
}

In fact we can do the same for the success and error callbacks as well.

pub.save().success(this.onSuccess).complete(this.onComplete);


While passing a success callback or using the deferred to setup a success callback are identical, in case of error callback, if you do not specify one in the arguments, an error event is triggered.

Sunday, June 9, 2013

Moving towards a fully working Marionette app

Starting from the ItemView, I have finally converted my application to use marionette application at the top. With this, the overall structure has started falling in place.

One benefit of committing to a framework is that it can force you to some good practices. While building my API using Django view, I was constantly using @csrf_exempt decorator so that I don't have to deal with passing the csrf token while making AJAX requests. I said I will come back to it but never did. However with Django Rest Framework, this was not a possibility and so I ended up handling the issue.

As it turns out, it is surprisingly simple. The CSRF token is included in a cookie by Django whenever csrf protection is on. You just need to extract that cookie and include the CSRF token as a header in request. All the code required to do this is provided in the Django official documentation.

So with CSRF hurdle crossed, I was able to create a new Publication via my Marionette app via the API.

Looks like it is time to go back to DRF and check out the Pages level API.

Central Message Bus

So armed with Marionette and Eclipse templates, I made some good progress on the dashboard. The additional pieces provided by Marionette are not only helping in reducing boilerplate code, they are also helping me think more clearly about my application.

The dashboard now comprises of several independent views that talk to each other via a message bus. A message bus or event aggregator is something on which any object in your app can trigger an event or listen for an event. So instead of directly subscribing for events on each other, they now only talk to the message bus. This decouples them and makes them reusable.

For example, the Publication Importer view I wrote is reusable in two ways. One is that it can be embedded anywhere in the app and it should work. It is also a blueprint for how to write import views for other resources. It looked like I can factor out a base class from the view that can be configured/overridden for building any number of import views.

I started with converting the basic views to Marionette and I am slowly building up to a Marionette application. Tomorrow I hope to get to the point where I can start working on the main character in this cast - the writing desk.

Eclipse Templates FTW

What do you do when you find yourself writing the same skeleton code in almost every file again and again? Since I switched over to AMD style modules for javascript, using requirejs, following is the common patterns in every file:


define([
        "jquery",
        "underscore",
        "marionette",
        ],
function($, _, Marionette){
 "use strict";
 
 var dashboard = Marionette.CompositeView.extend({
          ....

 });
 
 return dashboard;
 
});

Wouldn't it be great if this scaffolding could be generated with couple of key strokes? So I started digging into Eclipse and out came the templates. Not only most of the perspectives come with pre-built templates, you can define your own.

Defining a new template is simple. Go to Preferences and filter for templates in the left hand sidebar.



Click on New and create a new template:


Remember to give it a name you can easily remember. In order to use the template, you type the name and hit CTRL + Space and the name is replaced by the template. Best part is that when you change a $variable, all instance of it are updated. So in the above, when you replace object by the name of your JS Module, the return statement will also get updated automatically.

Marionette reduces the boilerplate code one has to write but some amount of it in inevitable. For those cases, there are templates. More than the time saving, it is the feeling of escaping the mundane that makes these very useful.

Saturday, June 8, 2013

Backbone is out, Marionette is in.

I know you are thinking that didn't take long. I agree that I am guilty of not making up my mind and then sticking with it. But not in this case.

As I said in my previous post, Backbone means a lot of decisions. Good thing is that it is already couple of years old and very successful. Which means it has drawn the attention of many very capable developers who not only saw the same issues but also decided to do something about them.

Now some of them went ahead and wrote their own libraries and frameworks like Spine.js, Angularjs and many many more. Each comes with their own pros and cons and many of them are more advanced. But there were others who decided to build on the foundation of Backbone itself. Marionette falls into that category.

The primary motivation behind Marionette, apart from improving the spelling skills of developer, is to reduce the boilerplate code in Backbone app. Though the name marionette seems to recover most of the reduced code.

But anyway, I have started to convert my 1 day old Backbone based dashboard to Marionette. So far so good. I will be back with more details tomorrow.

Thursday, June 6, 2013

Backbone is too many decisions

The road to a working backbone.js app is filled with decisions. I am finally understanding what is meant by it being lightweight and remaining out of your way. Combine it with Javascript where everything goes anywhere and you have the opportunity for perfect mess.

Anyway, as I am working through those decisions, I am also looking at the possibility of keeping couple of Angular apps I built earlier. If this works, I should be able to use Angular for some compact parts of the app and use Backbone for the rest.

But for now, we can rejoice as the wiring up of the Dashboard view is done and our very first Backbone powered app is ready to be used. Of course, very little can be done with it for the time being but that will change very soon.

Wednesday, June 5, 2013

We have an API!

So after the initial few scares, DRF ride proved to be smooth and we have a reasonable looking Publication level API. It is interesting how building an API makes you think about certain representations you chose in your model.

For example, motivated by the presence of Metadata EPub3 that can be several level deep, I decided to keep all the metadata as a single dict stored as JSON in the db. This allows me to capture metadata any level deep. What do I mean by deep metadata? An EPub file can not only have metadata like Title, it can have metadata explaining some property of the title. For example, title may have a language and then language may have a family.

However this means that all this data is not easily available for direct access and manipulation. So I had setup the access to the title of the Publication through a property.


@property
def title(self):
    return self.mtdt['title']['text']

This was nice and dandy since DRF allows you to use any function on the model as a source for a field till it takes only self as an argument. But as soon as I tried to create a new pub through API, I ran into a problem. I knew how to setup readonly properties but not how to update them. It turns out, it is quite easy:
# define a setter for the property
@title.setter
def title(self, val):
    if 'title' in self.mtdt:
        self.mtdt['title']['text'] = val
    else:
        self.mtdt['title'] = {'text': val}

There is a similar deleter decorator as well.

So with this, title is now a read and write property on the model and can be used seamlessly through the API. However I realized that it makes sense to give the Publication its own title that is separate from the title of the book. Kind of like project title. And that it what I did. I am also thinking of opening up the metadata dict and store the individual items as generic metadata attached to the publication. But that is for some other time.

Now that we have a rudimentary API for the Publications going, it is time to head back to the ugly world of Javascript and see if Backbone fulfills its promise.

Tuesday, June 4, 2013

Introspecting Django Models

कभी न कभी सभी को अपने गिरेबान में झांकना ही पड़ता है।
(Everyone is forced to look into their own affairs one day.)

We developers like to call it introspection. As if we are going to find something other than what we typed in ourselves! Thoreau went to a jungle to introspect. I am sure he was looking for a Python. Introspecting with Python is fun.

Anyway, it so happens that the simple matter of serializing JSON data to JSON was stuck on the matter of field instance vs field value. When you declare a field on a Django model and then create an instance of that model and do the following:

class DemoModel(models.Model):
    title = models.CharField(max_length=255)

model = DemoModel({'title': 'Introspecting'})
getattr(model, 'title') # 'introspecting'

You get the value of the field that it now holds.

But what if you need to access the methods and attributes of the CharField class. For that you need the instance of the CharField class. You can get that through the _meta attr on the model like this:

fld = model._meta.get_field('title')

# Now you can access the settings done on the class
getattr(fld, 'max_length') # prints 255

There are a bunch of other methods available that let you introspect everything from simple fields to foreign keys and many to many relations.

So now we can introspect from the comforts of our homes. No need to head out to woods.

Monday, June 3, 2013

Getting a Hierarchical URL structure going with DRF

Indians love hierarchy. Without hierarchy, our great nation will come to a stop. We wouldn't know what to do, whom to flatter and whom to diss. We carry over the same love of hierarchy to our technical architecture as well. If there is one thing you can be absolutely sure about DRF, it is that the author could not have been India. It is 2.3.4 and it still doesn't support hierarchical URLs out of box.

There are bunch of ways you can represent related resources in the API response. Out of them all, I liked the idea of returning a URL that you can use to fetch the related resource most appealing. So when you fetch a Publication, you get a list of URLs as part of it. Each URL represents one page. Now these URLs can be top level like:
/api/v1/pages/:pageid:

But since in our system, a page is always associated with a Publication, following seems more natural:
/api/v1/publications/:pubid:/pages/:pageid:

Trouble is that the built in HyperlinkedRelatedField cannot handle the second type of URL. Here is why. The URLs for related objects are generated by reverse lookup of views. Given a view_name, you can find the URL regex from URLconf. Then you fill in the captured parameter in the URL with the corresponding values for the given object and you have your URL.

However, HyperlinkedRelatedField only lets you specify only one captured argument and also mandates that it should be the same name as the corresponding field on the model. So if you are capturing the primary key which is the most common case, it would need to be called 'id' or 'pk'. This cannot work if you are capturing more than one primary keys from the URL since you cannot call both of them 'pk'. So we write our own MultilevelHyperlinkedField: 

class MultilevelHyperlinkedField(serializers.HyperlinkedRelatedField):
    '''
    Generate multilevel urls for foreign key relations
    '''
    def get_attr(self, obj, prop):
        '''
        Follow the dots
        '''
        value = obj
        for component in prop.split('.'):
            value = get_component(value, component)
            if value is None:
                break
        
        return value
    
    def get_url(self, obj, view_name, request, format):
        kwargs = {k: self.get_attr(obj, v) for k,v in self.lookup_field}
        logger.debug(kwargs)
        return reverse(view_name, kwargs=kwargs, request=request, format=format)
    
    def get_object(self, queryset, view_name, view_args, view_kwargs):
        qs_kwargs = {v: view_kwargs[k] for k,v in self.lookup_field}
        return queryset.get(**qs_kwargs)


Now we can build URLs as deep as we want:
pages = MultilevelHyperlinkedField(source='items', 
                                   many=True, 
                                   read_only = True, 
                                   view_name='page_details',
                                   lookup_field=[('pubid','publication.pk'),
                                                     ('pageid', 'pk')])

lookup_field, that can only be a single name in case of HyperlinkedRelatedField, is now a list of tuples. Each tuple provides a mapping between the name under which it is captured in the URL and the name of the field on the model. Notice that the name of the field on the model can be across models using the dot notation. This allows us to generate URLs as we like them.

Once we cross this hurdle, things start looking better for a while before they get ridiculous. I mean, how difficult would you imagine it would be, to serialize JSON? After all JSON is what we are serializing into! Turns out, not as easy as you would imagine. This sounds just too bureaucratic to be happening in my code. We hope to cut the red tape as soon as possible.

Sunday, June 2, 2013

Building a REST API using Django Rest Framework

It was a God forsaken day when I decided to build the front end of Instascribe using Backbone. I was so happy with my existing Spaghetti DOM code. It made me feel like a gladiator in the Roman Colosseum. But then good times never last.

As soon as I started using Backbone, it wanted REST. So much for a robust framework. It is thus that I found myself in need of a RESTful API for my Django app. My previous attempt at an API had produced a jungle of api endpoints that was as diverse as Amazon. The return values of the endpoints were also all over the map, sometimes returning html, sometimes json. (It is but natural that one would use the format that is easiest for the job at hand.)

But they say, to get REST, you need to be consistent. As I started cleaning up my code, I realized that there are other important things related to APIs that I have not yet thought about. Things like permissions, rate control, discovery. So I looked around and found django-tastypie and django-rest-framework. While Django tastypie sounded delicious and I was already feeling ravenous having lost my spaghetti, it lost the battle to django-rest-framework on account of its hydrate and dehydrate functions. No way I am going to write functions that are called hydrate and dehydrate in my web application code. I found the class based approach of writing serializers in django-rest-framework a lot more intuitive and cleaner.

The examples in the documentation of drf are varied and well written giving the impression that it will work fine for my use case (which, for some strange reason, almost always turns out to be that corner case that is still pending implementation).

So now I am busy building my new API using django-rest-framework. It is time to REST!

Saturday, June 1, 2013

Cleaning and Validating POST request data in Django

Users are evil. They send us bad data all the time. Sometimes they try to delete our database by naming their children strangely. Sometimes they send us their phone number where an email address was expected. They want our servers to crash and bad things to happen. We must be paranoid when dealing with their data.

When using Django forms, we get the help of cleaners and validators to keep the bad guys out. But what to do when we need to do this new fangled API stuff? User data coming directly into our views! Oh the horrors! Users must be getting wet eyes from all the evil laughter.

But in Django, we like to keep thing DRY. And so here is what we do. We can use a Form only for the validation and cleaning of data. Nothing mandates that we have to use the generated HTML for displaying the form.

Let us say, we have a AJAX request bringing in data from a registration form. So we create a Form as follow:

class AJAXRegisterForm(forms.Form):
  '''Form for validating the AJAX request'''
  username = forms.CharField(max_length=64)
  email = forms.CharField(max_length=32)
  evil_plan = forms.CharField(max_length=255)

This is how we proceed in the views:
def user_register(request):
  if request.method == 'POST': 
    regform = AJAXRegisterForm(request.POST)
      if regform.is_valid():
        # The cleaned form data is available here in 
        # form.cleaned_data dictionary
        # We can now safely add the user to our spam list
        # If required, we can add custom validators to the form
      else:
        # Send back an appropriate insult telling them 
        # that their evil plan failed
        ret = {'warning': 'Do not mess with Django Warriors'}
        HttpResponseServerError(json.dumps(ret), 
                                mimetype="application/json")
And thus keep the evil users at bay.

Friday, May 31, 2013

In the Trenches with a Python on my Backbone

Five years have gone by in a jiffy and once again I find myself in the trenches. A Python gives me Angular looks and an Apache aims for my Backbone as I try to Bootstrap some magic into this world.

The magic will be here in an instant O dear scribe, if only you get me some coffee and a script.