StackToHeap

No Overflow

Getting Started With Ansbile for a Django Project

| Comments

On a Django project, I have been using Chef for configuration management. I am totally happy with Chef. It gets the job done. The community cookbooks are great.

But there’s a new (relatively) kid in the block, by the name of Ansible, and I wanted to try it out for three main reasons:

1) It’s way simpler than existing solutions like Chef and Puppet. From reading the docs, and seeing its yaml based playbooks, it does seem simple to write. My experience with it, enough to write this blog post, does show that it simple, flexible and well thought out.

2) It is written in Python and can be pip installed. That means no more dependency on Ruby when you are working on a Python project. Also, it provides some Python / Django stack specific modules like the django_manage and supervisorctl modules already.

3) It doesn’t need any client software on the machines we intend to bootstrap. All that it would require is Python 2.4 or later ( not Python 3). It gets its job done through ssh. That is great news! Since many distributions have, and need, a relatively recent version of Python 2.X, this means that you don’t need to install ANYTHING on the box being configured to use Ansible.

Installing Ansible

I decided to install it through pip on Mac OS X. Other installation methods include using the package manager of your OS, or running from source.

1
2
3
4
pythonbrew venv create test-ansible
pythonbrew venv use test-ansbile
pip install ansible
pip freeze > requirements.txt

I use pythonbrew for managing my pythons and my virtualenvs, as mentioned in one of my previous blog posts - Why use virtualenv when there is pythonbrew

Vagrant

Of course, the next step is to get your Vagrant box up, and test your Ansible provisioning on it. The amazing authors of Vagrant have also provided a provisioner for Ansible, so using Ansible with Vagrant is simpler than it already is.

Do a vagrant init and configure it to provision with ansible. Your Vagrantfile may look something like below:

1
2
3
4
5
6
7
8
9
10
11
Vagrant.configure("2") do |config|
  config.vm.box = "debian-7.1.0"

  config.vm.network :private_network, ip: "192.168.33.10"

  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "site.yml"
    ansible.inventory_path = "hosts"
    ansible.verbose = "v"
  end
end

If you have done Chef/Puppet provisioning with Vagrant before, this should be familiar. All that the above does is to enable ansible provisioning and specifies the main playbook - site.yml. While Vagrant can automatically handle your inventory, I like to explicitly specify the hosts. For now, the hosts file would look like:

1
2
3
4
5
[webservers]
192.168.33.10

[dbservers]
192.168.33.10

where webserver and dbservers are host groups.

Directory Layout

I decided to follow the directory layout and structure as prescribed in the Ansible best practises

Ansible Playbooks

I have setup a top level playbook site.yml, as mentioned in the VagrantFile, which just includes playbooks for different server roles. It looks something like:

1
2
3
4
---
# file: site.yml
- include: webservers.yml
- include: dbservers.yml

The webservers.yml and dbservers.yml look similar to what’s described in the best practises link above.

Common tasks

Common tasks like updating the apt cache, installing build-essentials and installing packages like git can be placed in the common role’s tasks. The playbook for this would look something like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
---
# file: roles/common/tasks/main.yml

- name: update apt cache
  apt: update_cache=yes cache_valid_time=3600
  tags: apt
  sudo: yes

- name: install build essentials
  apt: name=$item state=installed
  with_items:
    - autoconf
    - binutils-doc
    - bison
    - build-essential
    - flex
  sudo: yes

- name: install git
  apt: name=git state=latest
  sudo: yes

Vagrant provision

We are now ready to provision:

1
$ vagrant provision   # vagrant up

Change ansbile verbosity by setting the ansible.verbose property appropriately in your VagrantFile.

That’s all for this blog post. I will add more posts as I continue to configure the entire stack with Ansible. I am loving Ansible already. It takes the best of Puppet / Chef and comes with a very simplified way of doing configuration management. As I proceed, there will be some roadblocks, but I am looking forward to them as well.

I have setup a repo on Github - django-ansible.

Using Git at a Hackathon Where Git:// Port Is Blocked

| Comments

Stuck in a network where the git:// protocol is blocked ( port 9418)?

I was. At a hackathon and having to do bower install, where a lot of packages were installed through “git://” from Github. Unfortunately, the port was blocked and the installation was not going through.

One option under such scenarios is to ask git to use the https:// protocol ( hopefully, that one is open ) instead of the git://.

It can be simply done as follow:

1
git config --global url."https://".insteadOf "git://"

This is not just for changing the protocol, but for replacing the beginning part of ANY url with the one you specify. In our case, we are replacing urls beginning with git:// to https://.

A simple solution, instead of having to get the port opened, etc. Just remember to unset this config once done.

Building a Debian Wheezy Vagrant Box Using Veewee

| Comments

For a Django website that I am working, which will be hosted on Debian 7.1, I needed a vagrant box that I can use during development. Unfortunately, such a box was not available at vagrantbox.es or elsewhere that I could trust, so I decided to build the box myself.

I did come across a nice blog post describing how to do the same, but it seemed like too many manual and repetitive steps ( across boxes ). Surely, there must be a tool to make it easier to create a vagrant box for any distribution?

It turns out that there is one - Veewee

Installing veewee was pretty straightforward and can be followed from the docs

Since I wanted to create a debian 7.1 box, I used the following command to find the template I wanted:

1
bundle exec veewee vbox templates | grep -i debian

Turns out I need Debian-7.1.0-amd64-netboot. So the step to define your box is as simple as:

1
bundle exec veewee vbox define 'debian-7.1.0' 'Debian-7.1.0-amd64-netboot'

At this point, you can change the defintions files to customize your box as needed. Going through the steps defined in various script files, you will also notice lot of similarities between what these scripts do and the manual steps described in the blog post mentioned earlier. I didn’t really change any of the defaults, as this was my first try with the tool, and I wanted to see how it proceeds.

The next step is, of course, building the box:

1
bundle exec veewee vbox build 'debian-7.1.0'

This downloads the required ISO for you and builds the box as per your definitions. This would then open up VirtualBox (in my case) and setup the virtual box for you.

The last step is to export the box so that you can actually use it with Vagrant:

1
bundle exec veewee vbox export 'debian-7.1.0'

Neat, and simple! Of course, the meat of what veewee offers is in building custom boxes, and this just scratches the surface.

Things That Trip Newbies in Powershell - Pipeline Output

| Comments

After the previous posts that pointed out the gotchas with powershell, now is the time to look at a more confusing aspect of Powershell. One that almost every newbie encounters, and gets immensely frustrated trying to figure out what’s happening.

I will be talking about pipeline output and return values from functions in this post.

Let’s start with an example that illustrates the confusing aspect:

1
2
3
4
5
6
7
8
9
10
function CheckParameterIsOne($paramter) {
    echo "Checking parameter is one"
    return $parameter -eq 1
}

if(CheckParameterIsOne(2)){
    echo "Parameter was one"
} else {
    echo "Parameter was not one"
}

Pardon the contrived example, but this is something that I come across a lot. A function that checks for a condition, or returns some value, and also “echoing” what it is doing.

It would be normal to expect the above to print Paremeter was not one. But the thing is, the function will always be true and you will always get Parameter was one printed, irrespective of the parameter. What’s more, the line Checking parameter is one is NEVER printed. What’s happening?

Functions in Powershell return values to the pipeline. And the values returned are not only through return, but any uncaptured object is returned to the pipeline.

That is, the following two are equivalent:

1
2
3
4
5
6
7
8
9
function CheckParameterIsOneWithReturn($paramter) {
    echo "Checking parameter is one"
    return $parameter -eq 1
}

function CheckParameterIsOneWithoutReturn($paramter) {
    echo "Checking parameter is one"
    $parameter -eq 1
}

Also, since echo, the alias for Write-Output ( another gotcha - it is not an alias for Write-Host) send the object to the pipeline, the echo "Checking parameter was one" is also passed to the pipeline along with the boolean from the condition.

Since the function ALWAYS returns two objects to the pipeline - effectively something like - @("Checking parameter was one", "$true") - the function’s output is always a truthy value.

Moreover, the returned object are “captured” by the if condition, and that is the reason why you never the “Checking parameter is one” output.

Fix the function by converting it to just:

1
2
3
function CheckParameterIsOneWithReturn($paramter) {
    return $parameter -eq 1
}

This will return a $true or $false and will behave as expected.

Git Learns to Ignore Better - New in 1.8.2

| Comments

The new version of git - 1.8.2 is here and with it, git is better at ignoring things. The files and directories in your repository, that is.

From the 1.8.2 release notes you can see the following:

The patterns in .gitignore and .gitattributes files can have **/, as a pattern that matches 0 or more levels of subdirectory. E.g. “foo/**/bar” matches “bar” in “foo” itself or in a subdirectory of “foo”.

This is great for ignoring, say lib folders within your src folder, but not ignoring lib folders elsewhere, especially, the root of your repsoitory. All the gross ways of achieving something like the above can be, well, ignore, and we can start using this.

Wait! 1.8.2 has even more in the art of ignoring thing:

“git check-ignore” command to help debugging .gitignore files has been added.

This seems neat! I have seen so many people struggle with trying to understand what is being ignored and what is not. Let’s see how this new command can help.

You can pass path(s) to the git check-ignore command and it will tell you the pattern from .gitignore or elsewhere that ends up ignoring the path(s).

Let’s try it out:

1
2
3
4
5
git init .
touch afile
mkdir bin
touch bin/abin.dll
echo "bin/" > .gitignore

Now if you do git check-ignore bin or git check-ignore bin/abin.dll, the command will spew back the paths, confirming that they are ignored. Of course, the git check-ignore command can go even further:

1
2
$ git check-ignore bin/a.dll --verbose
.gitignore:1:bin/   bin/a.dll

When you use the --verbose flag, it tells you the source of the ignore pattern, the line number, the pattern itself and the path being ignored because of the pattern.

While most of the times our git ignores are pretty simple like bi/ or *.dll, but for the times when things go out of hand, it makes sense to use the git check-ignore command to debug what is going on.

Git Learns Pre-push Hook - New in 1.8.2

| Comments

The 1.8.2 version of Git is here and one of the most sought features in git finally reaches the fans - the pre-push hook.

Go ahead, install the new version of git and do a git init or git init --bare. Under the hooks folder, you should see a sample for the brand new pre-push hook. Yay!

This is what the top comments of the pre-push script say:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/sh

# An example hook script to verify what is about to be pushed.  Called by "git
# push" after it has checked the remote status, but before anything has been
# pushed.  If this script exits with a non-zero status nothing will be pushed.
#
# This hook is called with the following parameters:
#
# $1 -- Name of the remote to which the push is being done
# $2 -- URL to which the push is being done
#
# If pushing without using a named remote those arguments will be equal.
#
# Information about the commits which are being pushed is supplied as lines to
# the standard input in the form:
#
#   <local ref> <local sha1> <remote ref> <remote sha1>
#
# This sample shows how to prevent push of commits where the log message starts
# with "WIP" (work in progress).

That seems pretty simple, and in line with other hooks. The example is also pretty useful, helping you block pushes containing commits with message starting with “WIP”.

So what are the scenarios for using this? Come on, do you really need a reason to use something as good as this? Well anyways, one thing that comes into mind is automatically running your local builds / tests just before pushing the commits to remote. If they fail, don’t push.

Why Use Virtualenv When There Is Pythonbrew?

| Comments

One of the first things that people coming from the Ruby / Rails world and entering the amazing world of Python / Django ( ahem ) want to know is what is the equivalent of rvm? Or to be more generic, how does one manage their “pythons” and the manage their “gemsets”

I see that newcomers to Python are still being pointed to virtualenv and then to the virtualenvwrapper which apparently makes working with virtualenv much easier. As someone coming back to the world of Python, from the days of installing all the packages for every project that your are working on system wide, I too, was led towards the aforementioned duo.

Until I came across pythonbrew, that is. It describes itself rather modestly as “..a program to automate the building and installation of Python in the users $HOME”, and claims to be inspired by rvm itself.

What many don’t seem to realize is that pythonbrew itself is a wrapper to virtualenv and allows the ability to manage virtual environments in addition to your pythons. The interface is consistent and it does more than what virtualenv or virtualenvwrapper do.

This is how you create and use virtual environments with pythonbrew:

1
2
pythonbrew venv create proj
pythonbrew venv use proj

This is how you do it with virtualenv:

1
2
virtualenv proj
source bin/activate

Bit better with virtualenvwrapper:

1
2
3
mkvirtualenv proj #this also activates it
#later
workon proj

Can you see the consistency in interface with pythonbrew that I am talking about? One you start with pythonbrew, you know where to go next. What the next step would be. Maybe I am spoilt by rvm, but pythonbrew just seems to be right approach here for me. The fact that it uses virtualenv under the wraps means you can always go down a level and use it directly.

With the 2.7.x python and 3.x python being used simultaneously, I surely hope people do need easy ways to install and use multiple pythons, in addition to managing the packages.

I have been using pythonbrew in my Django project and have had absolutely no problems with it. I had no problems setting up Pycharm with it, as well. It is exactly what I needed, without the complexity of the virtualenv interface, with a consistent way of doing things, and similarity to rvm.

So my question to pythonistas is, why do you still use / recommend virtualenv ( alone ) when there is pythonbrew? Or am is just reading old docs and blogs?

Things That Trip Newbies in Powershell - Encodings

| Comments

This is a follow up to the previous post on things that trip newbies in Powershell

In this post, I will be concentrating on Encodings. When I began using Powershell, it managed to confuse, trick and irritate me a lot. Encodings issues continue to be problematic with Powershell even in v3.0.

Let me start off with a common enough usage that might come to bite you. Say you have a git repo. You can want to pipe some content to .gitignore to ignore some files and directories. Easy right?

You can do:

1
"*.dll" > .gitignore

As any git user would know, that should ignore any dll file in your repo. But will it? Go ahead try it out.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
PS> git init .
PS> touch a.dll
PS> "*.dll" > .gitignore
PS> git add .
PS> git status
# On branch master
#
# Initial commit
#
# Changes to be committed:
#   (use "git rm --cached <file>..." to unstage)
#
#       new file:   .gitignore
#       new file:   a.dll

What??? Shouldn’t a.dll be ignored. Let’s see what’s going on here:

1
2
3
4
5
6
7
8
PS> git diff --cached
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..2f14638
Binary files /dev/null and b/.gitignore differ
diff --git a/a.dll b/a.dll
new file mode 100644
index 0000000..e69de29

So git is treating the .gitignore file as a binary. This is where encodings come into the picture.

Doing "*.dll" > .gitignore is pretty much the same as doing "*.dll" | out-file .gitignore. The default encoding that is used here is UCS-2. Many tools like git are not happy with it, at least by default.

We have this, from the docs:

By default, the Out-File cmdlet creates a Unicode file. This is the best default in the long run, but it means that tools that expect ASCII files will not work correctly with the default output format. You can change the default output format to ASCII by using the Encoding parameter

To make git happy, you can convert the .gitignore file to, say ascii. Actually, the out-file cmdlet provides ability to do just that.

1
 "*.dll" | out-file -Encoding ascii .gitignore

Now, the dlls should be ignored as expected. Unfortunately, there is no way to change how > behaves in this aspect!

Using TypeConverters to Get AppSettings in .NET

| Comments

The most easiest way to get AppSettings in .NET is to use the ConfigurationManager. It goes like:

1
var debugSetting = ConfigurationManager.AppSettings["IsDebug"];

The AppSettings property is just a NameValueCollection - an association of string keys and string values. You pass in the string key whose value you want and you get the string value back.

So, if your App.config were like:

1
2
3
4
5
6
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="IsDebug" value="true" />
  </appSettings>
</configuration>

The debugSetting will have the string value “true”. But most often, you don’t want just the string. Even in this example, you want a boolean value to see if debug is enabled or not (contrived, yes). So, you end up doing something like:

1
2
3
4
var debugSetting = Convert.ToBoolean(ConfigurationManager.AppSettings["IsDebug"]);
if(debugSetting){
    //do something nasty
}

Same goes for integers, urls, and so many other types. Get back the string, convert it to the type that is needed, and consume it.

Is there a better way to all this, which is also pretty simple? TypeConverters to the rescue!

We can define an AppSettings helper like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public static class AppSettings
{
    public static T Get<T>(string key)
    {
        var appSetting = ConfigurationManager.AppSettings[key];
        if (string.IsNullOrWhiteSpace(appSetting)) throw new AppSettingNotFoundException(key);

        var converter = TypeDescriptor.GetConverter(typeof(T));
        return (T)(converter.ConvertFromInvariantString(appSetting));
    }
}

//usage
var debugSetting = AppSettings.Get<bool>("Debug");

Here we use TypeDescriptor.GetConverter() method to get a TypeConverter for the type T we want to convert to. Then we get the converted value from the converter by calling its ConvertFromInvariantString, the input to which is of course the value from the appSetting.

This helper can now be used for various types which have converters. Consider a sample App.config:

1
2
3
4
5
6
7
8
9
10
<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <appSettings>
    <add key="IsDebug" value="true" />
    <add key="MyEnumValue" value="1" />
    <add key="MyEnumValue2" value="A" />
    <add key="ServiceEndpoint" value="http://www.example.com" />
    <add key="MyFavColor" value="Red" />
  </appSettings>
</configuration>

With that config, you can easily get the values you need, in the type that you will use them in:

1
2
3
4
5
var isDebug = AppSettings.Get<bool>("IsDebug");
var myEnumValue = AppSettings.Get<TestEnum>("MyEnumValue");
var myEnumValue2 = AppSettings.Get<TestEnum>("MyEnumValue2");
var myFavColor = AppSettings.Get<Color>("MyFavColor");
var serviceEndpoint = AppSettings.Get<Uri>("ServiceEndpoint");

The above example works for bool, TestEnum, Color and Uri types. Easy!

Precompiling Razor Views in ASP.NET MVC 3

| Comments

Before I start talking about how to go about precompiling razor views in ASP.NET MVC 3, let’s talk about the why.

The foremost reason is performance. Razor views are compiled at runtime. In projects with many views, this can be slow down the app start-up time. For better performance, it is ideal to precompile your razor views.

Even if performance if not a criterion and you don’t have that many views anyway, you also get the advantage of making sure that your razor views do compile at build time rather than at runtime. By default, in MVC 3, you get compile errors from the razor view only when you hit that view from your app. You get no idea, even if you have some silly typo ( though your IDE might be complaining ), till you hit the app. It is nice to have the safety net of compiling your razor views at build time.

Note that precompiling razor views, as will be described here, is NOT the came as compiling them. MVC 3 has the MvcBuildViews feature whereby, by changing the default <MvcBuildViews>false</MvcBuildViews> to true in you csproj file, you can compile your razor views at compile time. But this can only be used for catching compilation errors in your views, and it’s not the same as precompilation. If you don’t want precompilation, you can live with this. It is a simple one line change in your csproj file, and you get the safety net that we talked about. Precompiling your views is slightly more complex. But, MvcBuildViews will slow down your build ( maybe that is the reason why it is not enabled by default!). It is just not as efficient as the precompilation technique that we are going to talk about. In our project, with lots and lots of views, the build time with precompilation was faster than the build time with MvcBuildViews by a whopping 50%. Thats a lot you can shave off your build times.

Another reason to use precompilation is that you don’t need to deploy all those .cshtml files to your web server. This will reduce deployment times, especially, again, for projects with many view files.

Hope these reasons convinced you that precompiled razor views is the best thing since sliced bread. I will talk about the specific approach that I took to achieve this. We will be using the amazing library called RazorGenerator.

Go ahead and install the Nuget packages RazorGenerator.Mvc and RazorGenerator.MsBuild into your MVC project.

The RazorGenerator.Mvc package brings in the PrecompiledMvcEngine and registers it as the view engine in your mvc project. It does so in the RazorGeneratorMvcStart class found under App_Start folder:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[assembly: WebActivator.PostApplicationStartMethod(typeof(MvcApplication1.App_Start.RazorGeneratorMvcStart), "Start")]

namespace MvcApplication1.App_Start {
    public static class RazorGeneratorMvcStart {
        public static void Start() {
            var engine = new PrecompiledMvcEngine(typeof(RazorGeneratorMvcStart).Assembly) {
                UsePhysicalViewsIfNewer = HttpContext.Current.Request.IsLocal
            };

            ViewEngines.Engines.Insert(0, engine);

            // StartPage lookups are done by WebPages. 
            VirtualPathFactoryManager.RegisterVirtualPathFactory(engine);
        }
    }
}

It uses WebActivator to add the PrecompiledMvcEngine as a view engine in your project. Note also the line UsePhysicalViewsIfNewer = HttpContext.Current.Request.IsLocal. That enables you to develop with precompiled razor views as you would have done before. When you make changes to a razor file, you don’t have to compile your project ( and hence precompiling the razor views ). Just refresh the app and see the change. As is obvious from the line, if the actual .cshtml file are newer, it will use them for local requests, instead of the precompiled version. This means seamless transition to precompile views. Neat!

Now, let’s get to the exciting part - hooking up the actual precompilation of views at build time. This is where the RazorGenerator.MsBuild package comes into the picture. It provides msbuild targets to precompile your razor views that you can hookup into your csproj.

You want to import the RazorGenerator.targets that comes with the package in your csproj:

1
2
<Import Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets" Condition="false" />
<Import Project="$(SolutionDir)\packages\RazorGenerator.MsBuild.1.5.0.0\tools\RazorGenerator.targets"/>

The second import is the one being added, and I add it below the one importing the WebApplication related targets.

Next, you have to add a property that the target imported from the RazorGenerator.targets files uses to check if precompilation is to be done or not:

1
2
<MvcBuildViews>false</MvcBuildViews>
<PrecompileRazorFiles>true</PrecompileRazorFiles>

PrecompileRazorFiles is the property being added and I have added it just below MvcBuildViews in the csproj for obvious reasons.

That’s it! You can now build your application and the views will be precompiled. Try introducing a compile error in one of your razor views and you should get the error when compiling your solution. The precompiled files will be generated under obj\CodeGen.

There is one more thing that you have to do if you don’t want to include the views as part of your deployment package. By default, the .cshtml files will be created as Content type. The web applications target looks for all content types and includes them in the _PublishedWebsites folder. So you have to change the Build Action for your views to None so that they don’t get included. This can be easily automated without changing the view properties. Just add the below to your .csproj

1
2
3
4
5
6
7
<Import Project="$(SolutionDir)\packages\RazorGenerator.MsBuild.1.5.0.0\tools\RazorGenerator.targets"/>
<Target Name="BeforeBuild">
    <ItemGroup>
      <Content Remove="Views\**\*.cshtml" />
      <None Include="Views\**\*.cshtml" />
    </ItemGroup>
</Target>

I am adding the BeforeBuild target right after the previouly added line to import the RazorGenerator.targets file. The BeforeBuild target is a hook provided to do things as a pre-build event. Here we remove all .cshtml from Content itemgroup and add them all to the None group. Now, when the web applications target is executed, the views will not be copied over as they are not Content type anymore.

That’s all there is to it.

Wait, what about using the RazorGenerator Visual Studio extension which generates the backing .generated.cs file when you save the file. I decided not to go with that approach because using the msbuild targets is much cleaner. With the extension approach, not only do you need an extension to visual studio that every team member has to be told to install, all view files, old and new, have to be edited to specify a custom tool. Also, the .generated.cs files pollute you solution while navigating. Too much of an hassle, and I don’t recommend that approach. Using the steps given above, you can achieve precompilation of your razor views without any changes to the way you have been developing so far.