Backbone is out, Marionette is in.

I know you are thinking that didn't take long. I agree that I am guilty of not making up my mind and then sticking with it. But not in this case.

As I said in my previous post, Backbone means a lot of decisions. Good thing is that it is already couple of years old and very successful. Which means it has drawn the attention of many very capable developers who not only saw the same issues but also decided to do something about them.

Now some of them went ahead and wrote their own libraries and frameworks like Spine.js, Angularjs and many many more. Each comes with their own pros and cons and many of them are more advanced. But there were others who decided to build on the foundation of Backbone itself. Marionette falls into that category.

The primary motivation behind Marionette, apart from improving the spelling skills of developer, is to reduce the boilerplate code in Backbone app. Though the name marionette seems to recover most of the reduced code.

But anyway, I have started to convert my 1 day old Backbone based dashboard to Marionette. So far so good. I will be back with more details tomorrow.

Backbone is too many decisions

The road to a working backbone.js app is filled with decisions. I am finally understanding what is meant by it being lightweight and remaining out of your way. Combine it with Javascript where everything goes anywhere and you have the opportunity for perfect mess.

Anyway, as I am working through those decisions, I am also looking at the possibility of keeping couple of Angular apps I built earlier. If this works, I should be able to use Angular for some compact parts of the app and use Backbone for the rest.

But for now, we can rejoice as the wiring up of the Dashboard view is done and our very first Backbone powered app is ready to be used. Of course, very little can be done with it for the time being but that will change very soon.

We have an API!

So after the initial few scares, DRF ride proved to be smooth and we have a reasonable looking Publication level API. It is interesting how building an API makes you think about certain representations you chose in your model.

For example, motivated by the presence of Metadata EPub3 that can be several level deep, I decided to keep all the metadata as a single dict stored as JSON in the db. This allows me to capture metadata any level deep. What do I mean by deep metadata? An EPub file can not only have metadata like Title, it can have metadata explaining some property of the title. For example, title may have a language and then language may have a family.

However this means that all this data is not easily available for direct access and manipulation. So I had setup the access to the title of the Publication through a property.

@property
def title(self):
    return self.mtdt['title']['text']

This was nice and dandy since DRF allows you to use any function on the model as a source for a field till it takes only self as an argument. But as soon as I tried to create a new pub through API, I ran into a problem. I knew how to setup readonly properties but not how to update them. It turns out, it is quite easy:

# define a setter for the property
@title.setter
def title(self, val):
    if 'title' in self.mtdt:
        self.mtdt['title']['text'] = val
    else:
        self.mtdt['title'] = {'text': val}

There is a similar deleter decorator as well.

So with this, title is now a read and write property on the model and can be used seamlessly through the API. However I realized that it makes sense to give the Publication its own title that is separate from the title of the book. Kind of like project title. And that it what I did. I am also thinking of opening up the metadata dict and store the individual items as generic metadata attached to the publication. But that is for some other time.

Now that we have a rudimentary API for the Publications going, it is time to head back to the ugly world of Javascript and see if Backbone fulfills its promise.

Introspecting Django Models

कभी न कभी सभी को अपने गिरेबान में झांकना ही पड़ता है।

(Everyone is forced to look into their own affairs one day.)

We developers like to call it introspection. As if we are going to find something other than what we typed in ourselves! Thoreau went to a jungle to introspect. I am sure he was looking for a Python. Introspecting with Python is fun.

Anyway, it so happens that the simple matter of serializing JSON data to JSON was stuck on the matter of field instance vs field value. When you declare a field on a Django model and then create an instance of that model and do the following:

class DemoModel(models.Model):
    title = models.CharField(max_length=255)

model = DemoModel({'title': 'Introspecting'})
getattr(model, 'title') # 'introspecting'

You get the value of the field that it now holds.

But what if you need to access the methods and attributes of the CharField class. For that you need the instance of the CharField class. You can get that through the _meta attr on the model like this:

fld = model._meta.get_field('title')

# Now you can access the settings done on the class
getattr(fld, 'max_length') # prints 255

There are a bunch of other methods available that let you introspect everything from simple fields to foreign keys and many to many relations.

So now we can introspect from the comforts of our homes. No need to head out to woods.

Getting a Hierarchical URL structure going with DRF

Indians love hierarchy. Without hierarchy, our great nation will come to a stop. We wouldn't know what to do, whom to flatter and whom to diss. We carry over the same love of hierarchy to our technical architecture as well. If there is one thing you can be absolutely sure about DRF, it is that the author could not have been India. It is 2.3.4 and it still doesn't support hierarchical URLs out of box.

There are bunch of ways you can represent related resources in the API response. Out of them all, I liked the idea of returning a URL that you can use to fetch the related resource most appealing. So when you fetch a Publication, you get a list of URLs as part of it. Each URL represents one page. Now these URLs can be top level like:

/api/v1/pages/:pageid:

But since in our system, a page is always associated with a Publication, following seems more natural:

/api/v1/publications/:pubid:/pages/:pageid:

Trouble is that the built in HyperlinkedRelatedField cannot handle the second type of URL. Here is why. The URLs for related objects are generated by reverse lookup of views. Given a view_name, you can find the URL regex from URLconf. Then you fill in the captured parameter in the URL with the corresponding values for the given object and you have your URL.

However, HyperlinkedRelatedField only lets you specify only one captured argument and also mandates that it should be the same name as the corresponding field on the model. So if you are capturing the primary key which is the most common case, it would need to be called 'id' or 'pk'. This cannot work if you are capturing more than one primary keys from the URL since you cannot call both of them 'pk'. So we write our own MultilevelHyperlinkedField: 

class MultilevelHyperlinkedField(serializers.HyperlinkedRelatedField):
    '''
    Generate multilevel urls for foreign key relations
    '''
    def get_attr(self, obj, prop):
        '''
        Follow the dots
        '''
        value = obj
        for component in prop.split('.'):
            value = get_component(value, component)
            if value is None:
                break
        
        return value
    
    def get_url(self, obj, view_name, request, format):
        kwargs = {k: self.get_attr(obj, v) for k,v in self.lookup_field}
        logger.debug(kwargs)
        return reverse(view_name, kwargs=kwargs, request=request, format=format)
    
    def get_object(self, queryset, view_name, view_args, view_kwargs):
        qs_kwargs = {v: view_kwargs[k] for k,v in self.lookup_field}
        return queryset.get(**qs_kwargs)

Now we can build URLs as deep as we want:

pages = MultilevelHyperlinkedField(source='items', 
                                   many=True, 
                                   read_only = True, 
                                   view_name='page_details',
                                   lookup_field=[('pubid','publication.pk'),
                                                     ('pageid', 'pk')])

lookup_field, that can only be a single name in case of HyperlinkedRelatedField, is now a list of tuples. Each tuple provides a mapping between the name under which it is captured in the URL and the name of the field on the model. Notice that the name of the field on the model can be across models using the dot notation. This allows us to generate URLs as we like them.

Once we cross this hurdle, things start looking better for a while before they get ridiculous. I mean, how difficult would you imagine it would be, to serialize JSON? After all JSON is what we are serializing into! Turns out, not as easy as you would imagine. This sounds just too bureaucratic to be happening in my code. We hope to cut the red tape as soon as possible.

Building a REST API using Django Rest Framework

It was a God forsaken day when I decided to build the front end of Instascribe using Backbone. I was so happy with my existing Spaghetti DOM code. It made me feel like a gladiator in the Roman Colosseum. But then good times never last.

As soon as I started using Backbone, it wanted REST. So much for a robust framework. It is thus that I found myself in need of a RESTful API for my Django app. My previous attempt at an API had produced a jungle of api endpoints that was as diverse as Amazon. The return values of the endpoints were also all over the map, sometimes returning html, sometimes json. (It is but natural that one would use the format that is easiest for the job at hand.)

But they say, to get REST, you need to be consistent. As I started cleaning up my code, I realized that there are other important things related to APIs that I have not yet thought about. Things like permissions, rate control, discovery. So I looked around and found django-tastypie and django-rest-framework. While Django tastypie sounded delicious and I was already feeling ravenous having lost my spaghetti, it lost the battle to django-rest-framework on account of its hydrate and dehydrate functions. No way I am going to write functions that are called hydrate and dehydrate in my web application code. I found the class based approach of writing serializers in django-rest-framework a lot more intuitive and cleaner.

The examples in the documentation of drf are varied and well written giving the impression that it will work fine for my use case (which, for some strange reason, almost always turns out to be that corner case that is still pending implementation).

So now I am busy building my new API using django-rest-framework. It is time to REST!

Cleaning and Validating POST request data in Django

Users are evil. They send us bad data all the time. Sometimes they try to delete our database by naming their children strangely. Sometimes they send us their phone number where an email address was expected. They want our servers to crash and bad things to happen. We must be paranoid when dealing with their data.

When using Django forms, we get the help of cleaners and validators to keep the bad guys out. But what to do when we need to do this new fangled API stuff? User data coming directly into our views! Oh the horrors! Users must be getting wet eyes from all the evil laughter.

But in Django, we like to keep thing DRY. And so here is what we do. We can use a Form only for the validation and cleaning of data. Nothing mandates that we have to use the generated HTML for displaying the form.

Let us say, we have a AJAX request bringing in data from a registration form. So we create a Form as follow:

class AJAXRegisterForm(forms.Form):
  '''Form for validating the AJAX request'''
  username = forms.CharField(max_length=64)
  email = forms.CharField(max_length=32)
  evil_plan = forms.CharField(max_length=255)

This is how we proceed in the views:

def user_register(request):
  if request.method == 'POST': 
    regform = AJAXRegisterForm(request.POST)
      if regform.is_valid():
        # The cleaned form data is available here in 
        # form.cleaned_data dictionary
        # We can now safely add the user to our spam list
        # If required, we can add custom validators to the form
      else:
        # Send back an appropriate insult telling them 
        # that their evil plan failed
        ret = {'warning': 'Do not mess with Django Warriors'}
        HttpResponseServerError(json.dumps(ret), 
                                mimetype="application/json")

And thus keep the evil users at bay.