RSS
 

Application Deployment – web and other with php-packager

30 May

Introduction

Deployment of applications on a server is always an issue for developers of applications that need to be installed in some way. Applications like this include specifically web applications built with or on top of Drupal, SilverStripe, Symfony or other PHP web application frameworks. But this is not limited to PHP. Linux server applications can also be Java EE or other JVM applications that run on a server or even C(++), Python, Ruby (on Rails) and any other software. There are usually files that have to be generated or/and copied from a development machine to a test machine and ultimately to a production machine. This process is usually error prone and tedious and needs a lot of documentation. With good documentation this can be reasonably easy, but it is never trivial and humans make mistakes. On top of that one often needs to compile ones own or third party applications and so on, because there is no package for that software yet, because it is far from trivial to roll your own packages. Whether that is a C++ application or a Java application doesn’t really matter. However, we’ve learned that compiling your source code on a production machine is really bad practice, despite it being the easiest way to do things. The main reason for this being bad is that this implies that there is a compiler on a production machine. This is a security risk because if an attacker gains any kind of access to the machine, they can compile their own software on it and run it at their will and possibly compromise the production machine.

However, to compile and/or deploy ones applications there are usually a series of steps that can be bundled into a script. For a while I was using deployment scripts so that I could track a version number within git tags. These deployment scripts were individual to specific applications and never easy to write. Some stuff was generic and other stuff very specific to the application.

PHP-Packager was born

So, I’ve been conversing with my colleague about how we should go about deploying our applications in a repeatable manner. While there are things like Juju, Puppet, Maven, Gradle and so on, we are using mainly Linux as a platform for servers and specifically a Debian based distribution you might have heard of called Ubuntu. As you might know Ubuntu and other Debian based distros (actually also Red Hat/CentOS/Fedora as well) manage their software with packages and ultimately with APT or YUM and the like. For Ubuntu and other Debian based distributions like Linux Mint these are usually .deb packages with dependencies and install/uninstall scripts. For Red Hat based distros, this is usually .rpm.

Finally, we decided to make our own APT repositories for all our VMs, even though we had feared and avoided it in the past. One test and one production repository. Technically, you can make a development repository as well, but for now that would be overkill. For that we needed a mechanism to package and deploy our applications to the repository while still keeping track of versions in our source code repository so we could roll back to a known version. For version control we use Git and for that reason we decided the current Git commit should simply be tagged with the current version over which we have some power, but we don’t want to keep track of every minor version manually.

There is already a solution out there to make packages. It is called FPM, by jordansissel (https://github.com/jordansissel/fpm). It greatly simplifies the process of making packages for all major distributions. However, since I am most comfortable with PHP as a scripting language and not Ruby, Python or even Groovy, I came up with the idea to wrap it with PHP and some little convenience automation, so I don’t have to do the generic stuff. PHP is a powerful, object oriented language and is one of the easier high-level scripting languages. It is itself open source and free and has many other advantages. The name seems to imply that it is only for the web, but apart from the opening <?php tag there is not much more that enforces it to be used only for web apps. It can easily be used from the command line as well.

PHP-Packager or packager for short, is a little script/application that allows for as-easy-as-it-can-get deployment of software via packages. Packages can have dependencies and basically bundle a set of files and scripts triggered by events/actions.

PHP-Packager requires one folder and at least one file within your repository. You can read more details on how to use it on its Github page at https://github.com/geekdenz/packager.

There is more potential for it like more generic tools which one might need in the deployment process, but currently it has the following features:

  1. Package files into a bundled deb or rpm package.
  2. Add after-install and before-remove php scripts for custom procedures like setting up a database for an application or creating symlinks or other files.
  3. Tag your git repository with a version tag. The version is kept track of and can be changed at any time in version.txt in the root of your application’s source code.
  4. Automatically upload the deb or other package to a repository with scp.

There are probably many other things that could be simplified or improved. But this is the current state of this application.

 

SilverStripe

24 May

Introduction

Today and yesterday, I attended the SilverStripe training course lead by SilverStripe CEO Sam Minnee. It was a pleasant experience, a well paced session and I got a lot out of it. Even though it was the first SS training course provided ever, it was well thought through and gave a lot of information that would have been tedious to figure out by my own research. Despite this, there is plenty and good documentation about SS on the web, which was the driver for me to influence the decision to use SS for other applications. But listening to Sam brought about a lot of little tricks even for developing PHP in general and SS specific things that otherwise I would probably not have figured out.

The course was two days in total and there were 10 people attending. A fairly small number for such a famous CMS that has been taken up by the government as the primary web development platform.

Sam introduced us to the Common Web Platform (CWP) version of SS. It is a common set of extensions for SS which can also be acquired by downloading them from SS directly. However, it also includes a common way of deploying SS websites.

People

Sam Minnee, the now CEO of the company, is a pleasant and competent guy even with the technical details of SS. He is the initiator of the project and seems to know almost everything there is to SS. He used to  be the CTO before he became CEO.

However, no other SS staff were there apart from Sam Minnee.

SilverStripe Framework & CMS

The SS framework and CMS are extensions of the framework itself. The framework is a simple base application which all SS sites are based off of.

We worked with the framework to do the following:

  1. Extend a simple site by adding Staff pages and a Staff Holder page which lists the staff in a company. This was really easy to do and abstracted in a nice way, so it can be applied to more concepts, such as maybe Map pages or animal and plant species pages for example.
  2. Media/Image handling
  3. Templates: the simple templating engine was introduced by giving examples tied to the data objects/classes in the model.
  4. Forms and data binding to them: an example was given with LinkedIn like endorsements
  5. Tasks: tasks are things that could be run with a cron job or to do simple debugging to learn what code does
  6. Debugging: We were introduced to simple debugging functions and how to improve PHP’s own debugging features in php.ini to avoid the blank page problem in PHP.
  7. Functions and their relation to pages and their data.
  8. Look into the data model and DRM: Data Relational Mapping is designed quite well in SS and it is very easy to use, so one doesn’t have to create database tables and properties any longer. It has built in security for handling SQL injections and other attacks, while the programmer still has the power to do custom WHERE clauses.
  9. Data types: The basic data types have been introduced as well as how to find out more about them.
  10. CRUD: Create Read Update Delete functionality is built into SS and it exposes a powerful admin and user interface in the backend. However, this functionality can be exposed to the front-end application as well through the powerful forms API.
  11. Custom Data Validators: are easily implemented. Even custom PHP code can be used to validate user input. So there are no limits, even regular expressions and more sophisticated logic can be used.
  12. Composer: composer is a PHP dependency management tool, similar to Apache Ivy, Maven or Gradle for Java. It can be used to manage the extensions that are available in SS and PHP in general. Even SS itself is managed with composer and available as a package.
  13. Security: How to manage users and permissions with SS very easily.

Extensions

SS provides a staggering amount of extensions. A few to mention are:

  1. Content Connector – Squiz, Drupal and other CMS can be queried to import their content. The pages can even be scraped to get the content of remote sites. This is a powerful tool and could be used to migrate existing sites. The extension for Squiz is called silverstripe-matrix-connector.
  2. There is a sophisticated workflow module within the CWP.

GitLab

GitLab is a platform for code hosting. So similarly to GitHub or Indefero it is a web application for managing git repositories more easily. One should definitely give it a test run because I believe it also has issue management and similar features.

DeployNaut

DeployNaut is a fairly new application to ease and speed up SS deployment of sites. It runs commands etc automatically, so that a SS site can be deployed in a safe, easy and repeatable fashion.

Deployment

Sites can be deployed in different ways. However, the most interesting way to do this is using DeployNaut, a custom SS deployment solution. With this the code can be submitted to a GitLab code repository and then from within that web application, sites can be easily deployed (with a click of a button). However, DeployNaut is not yet accessible but I believe it is planned on making it so and providing it free of charge and Open Source in the future.

PHP Debugging

These settings in php.ini make debugging easier. For example the white page problem can be avoided like this.

display_errors = On
error_reporting = E_ALL | E_STRICT
 
 

Google Code Jam Problem – Problem B. Baby Height – Mistake?

01 Apr

Just having a couple of training sessions on Google Code Jam problems I noticed the following.  I tried my luck with this problem: http://code.google.com/codejam/contest/2334486/dashboard#s=p1 .

I got it wrong quite a few times. Mostly because of the fact that I misread one sentence: “If the algorithm produces a range whose endpoints fall on fractional inches, your program should shrink the range until both endpoints can be expressed in whole inches.”.

I misread this and ended up using ints instead and letting truncation take care of the rounding down. However, this means that the lower end of the range should be rounded up. Admittedly, my mistake.

However!  I still couldn’t get it right after correcting my mistake.  Why?  Because I believe there is an error in the question. It also states:
“Each integer denoting feet will be at least 1 and at most 9.
Each integer denoting inches will be at least 0 and at most 11.”.

This, to me, means that we cannot have more than 9’11″.  I implemented this range, 1’0″ to 9’11″ in my program but still got “wrong” results.  I then took out the logic to enforce this limit and voila, my output succeeded!  So, obviously the question rather than my final answer is wrong.

I think it’s really bad to have a wrong question like this up.  It was frustrating for me, because I doubted my reading of the question before even thinking about the possibility of Google making such a mistake.  Sure, mistakes happen, but has no one noticed this yet?

 
 

Which Technology To Choose?

03 Jan

Since this seems to be interesting for a lot of developers, I thought I’d share my thoughts:

Introduction

Which technologies to choose is getting increasingly more difficult for developers, web and embedded alike. Questions like “Why Java?”, “Why HTML 5 (HTML + JavaScript + CSS Modern Extensions)?’, “Why C++?”, “Why PHP?”, “Why Python?” or “Why not something else?” are not easy to answer. There are many advantages and disadvantages in using any of these programming languages and environments. Within these languages there are even more possibilities of choosing development tools called frameworks. When talking of frameworks, I mean jQuery (JavaScript), OpenLayers (JavaScript), (Geo)Ext (JavaScript), Twitter Bootstrap (HTML5), Drupal (PHP), Symfony (PHP), Java EE, Wicket (Java), Hibernate (Java), GWT (Java -> JavaScript), just to name a few I’ve had the pleasure (and pain) to work with. More and more I’m coming to the following conclusions:

Choosing The Path Of Least Resistance

Maximum Gain With Minimum Pain

This has always been my main goal with the development tools I choose. After all, computers are there to make our lives easier and not harder. You might be surprised of some of the choices I’ve made with OS and tools. While choosing the Windows or MacOS platforms seems like the easier path, Linux and yes, the command line, actually have proven easier or at least more powerful by now. So, what seems the easiest at first might be harder in the long run and vice versa. Also, Linux tooling for web or any kind of development is far better documented, supported and a lot cheaper (free). So, if you are like me (and most other people) and have more time than money, it is probably worth your while to learn and invest (time, at least) into Open Source. It has had a great return in terms of moral and now even monetary rewards. There are companies out there that invest their staff’s’ time and resources into Open Source. Among those not least Google and Facebook, the biggest Internet companies in the world!

Frameworks

Frameworks might in fact make life harder rather than easier. Sure, I use many framework in my work, but sometimes it seems like learning and choosing a framework sometimes takes longer than writing your own and using it. At least attempting to write your own framework(s) has proven as a good exercise for learning how to use them and vice versa, choosing a framework and using it might give you ideas on what to do and more importantly on what not to do. Now, I haven’t written my own complete framework as yet, but I have started doing so. I guess once you start your own framework, you are never finished. The main framework I am working on since about 2 years has been a client side spatial portal application framework called the LcrToolkit (Landcare Research Toolkit, used on http://ourenvironment.scinfo.org.nzhttp://smap.landcareresearch.co.nz and others to come out soon). Thinking back now, I could have come up with a more original name for it, but that is just a side note. It is a very specialised framework for putting up spatial portal applications. Learning to use a framework is a big time commitment, whether this is Microsoft .NET, PHP (Symfony, Drupal, …) or Java’s many frameworks. So, if it were for saving time, some people say, use .NET, because some people believe the ads that the time investment will be minimal. This (in my opinion) is wrong! Learning PHP and JavaScript frameworks can be achieved much faster, because these languages (at least the basics) are very simple AND they are openly developed with tons of documentation and since they are free, there is a lot more support from developers and advocates. Not only do these developers innovate and support free products because they are super nice people and want to make the world a better place, they develop Open Source products, because these products live longer and if they are good, attract a lot of developers. This is not only because Open Source is better, but also because developers get credit for what they have achieved. Furthermore, Open Source is very competitive and therefore secure. To get your name out there as an Open Source developer, you have to be (one of) the best. That means you have to compete, BUT always in a friendly and respective manner. To be the best, you have to be prepared to fail to succeed. You have to be open to being corrected and that is not always easy to handle. To get back to Frameworks: There are a lot of them out there and I have named a few already. It seems like there are simply too many. The more I research frameworks, the more time I could spend trying them all out. The possibilities are endless. So, one must put a vigorous filter into ones research. Lately, I’ve been wanting to implement a testing framework inside the LcrToolkit and eventually went with qUnit (JavaScript), not because I think it is the best, but simply because I do not have time to try them all. qUnit is simple, well documented and works just fine. Surely, there must be more advanced testing frameworks for JavaScript, but qUnit is easy to pick up and that is the main criteria these days for me. Things like Hibernate and JPA don’t really appeal to me. They are bloated, take a lot of time to learn and don’t really gain you anything. The only benefit you get from ORMs is that you are “independent” of a particular database management system. Just use PostgreSQL for Christ’s sake. It is the most advanced database out there, and yes I mean the most advanced DBMS. It is superior to even Oracle and far better than MySQL or Oracle’s DBMS. Not that Oracle is a bad company… anyway, I’m missing the point. Frameworks: There are so many that one cannot grasp all of them. I’ve been a bit of a grasshopper on frameworks. I’ve worked with Drupal, Joomla, CiviCRM, GWT, Wicket and all of the above and have taken away that frameworks are great, BUT cannot replace good developers. You need to be thinking all the time about security, look and functionality of your app, whether it is a web app, website or even “just”  an embedded program.

To be continued… ??

 

OpenLayers – Questions and Answers

09 May

Here I would like to try to answer questions about OpenLayers. Please ask in a comment.

  1. How to enable animated zooming as in http://www.tomtom.com/livetraffic/ (though it seems they implemented their own method)

    View http://openlayers.org/dev/examples/google-v3.html or http://dev.openlayers.org/sandbox/emanuel/animatedZooming/demo.html. The performance of those sites might not be as good as Tomtom’s because they might not have as fast WMS servers or the tiles are not cached efficiently. If you need fast tile seeding/caching, have a look at MapServer and in particular MapCache on the server side. Both of which are CGI programs implementing WMS.

  2. How to restrict the map to a certain area by giving lon and lat values.

    View http://openlayers.org/dev/examples/restricted-extent.html or http://openlayers.org/dev/examples/all-overlays.html (line 18; view source). Generally, I think the maxExtent property of the Map class will do the trick.

 

Selling Domains

21 Apr

Because I want to concentrate on development of fewer web applications rather than incept new platforms and realize them, I decided to sell the following domains:

designingright.com
glaslampen.com
glasslamps.eu
inceptyou.com
landcafe-mulmshorn.de
open4geo.com
programmingreference.net
pyramidenlampen.de
pyramidlamps.com
pyramidlamps.net
uinstruct.net
youinstruct.net

 
 

jQuery Plugin Template

19 Apr

Here my simple and short jQuery plugin template. Hope it helps with plugin development. It has all the recommended settings (from docs.jquery.com/Plugins/Authoring):

(function($) {
    $.fn.pluginNamespace = function(method) {
        var defaults = {
                // settings here
            },
            settings = null,
            methods = {
                init: function(options) {
                    settings = $.extend(defaults, options);
                }
            };
        if ( methods[method] ) {
            return methods[method].apply( this, Array.prototype.slice.call( arguments, 1 ));
        } else if ( typeof method === 'object' || ! method ) {
            return methods.init.apply( this, arguments );
        } else {
            $.error( 'Method ' +  method + ' does not exist on jQuery.tooltip' );
        }
    };
})(jQuery);
 
 

SMTP Postfix Server Setup for your home development

18 Mar

EDIT:

This is much easier if not trivial to achieve with your ISP’s smtp server and configuring postfix as a smart host and simply supplying your ISP’s smtp server as the smart host. A simple:

sudo apt-get install postfix

and then configuring you server as a smart host relaying mail to smtp.yourisp.com.

If you already installed postfix, you can follow the following:

Note that You might not want to use dnsexit if security is a concern for you. I stumbled upon this now (28 July 2012) and realised that if you use dnsexit, you are basically giving them your credentials. So, be warned. I used my own server for this.

Source: http://www.dnsexit.com/support/mailrelay/postfix.html

This document describes 5 easy steps to setup your Postfix email server as smarthost to relay on DNS Exit mail relay server for all email sending.

The mail relay service from dnsExit.com can be found at here

1. Create password maps file (it assigns username/passwords to
specified mail servers). You can choose any name, let's say it
is /etc/postfix/relay_passwd. It's content should be as follows:
 relay.dnsexit.com USERNAME:PASSWORD 

 Note: Replace USERNAME and PASSWORD with your DNS EXIT mail
relay username and PASSWORD.   

2. Set proper permissions for that file:
# chown root:root /etc/postfix/relay_passwd
# chmod 600 /etc/postfix/relay_passwd     

3. Create hash from maps file (remember to do it each time
you change your maps file):
#  postmap /etc/postfix/relay_passwd   

4. Configure your /etc/postfix/main.cf:
relayhost = [relay.dnsexit.com]  # this is the important line!
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/relay_passwd
smtp_sasl_security_options =    

Note: If your ISP blocks outgoing port 25. You can choose to
use alternative SMTP ports by appending the port at the end:
relayhost = [relay.dnsexit.com]:26

5. Reload or restart your postfix:
# /etc/init.d/postfix restart

The thing to note here is that you can easily use your ISP’s smtp server as the relayhost. You can find out easily what your ISP’s smtp server is by going to their support pages.

 
2 Comments

Posted in Howto, Linux

 

Howto: Install Apache Tomcat 6 on Ubuntu 11.10

21 Feb
  1. Download Tomcat 6 from http://tomcat.apache.org/download-60.cgi or a mirror listed there.
  2. Extract it in the /usr/local directory:
    cd /usr/local/
    sudo tar xvfz ~/apache-tomcat-6.0.35.tar.gz # or wherever you downloaded tomcat to
    sudo ln -s apache-tomcat-6.0.35 apache-tomcat-6 # or whatever your tomcat version is

This post is incomplete, but might get you going. More docs can be found @ http://www.eclipse.org/forums/index.php/t/272457/

 
No Comments

Posted in Howto, Java, Linux

 

HTML5 Mastery – Anselm Bradford & Paul Haine

19 Feb

In HTML5 Mastery Anselm Bradford and Paul Haine present the complete set of new elements that HTML5 has to offer.

This is for the beginner as well as the advanced level web developer/designer to learn the new techniques for developing standard compliant web sites/pages.

If you are a web developer and know the basics of HTML and want to take that leap forward into a new era of web standards, this book is for you. Anselm and Paul describe all of the elements that have been added by the time of writing in a very efficient and concise way. Examples of use and use cases make the context in which these elements should be used very clear and explanations are kept simple in order to maximise the readers understanding. If you are looking for a book that gives a complete overview of HTML5 you have come to the right place. If you are looking for a book going into detail on the new JavaScript APIs, this book gives you an idea of what you can do but is by no means the focus of this book. The focus is more on HTML5 as in its markup elements. So, in essence, it serves the title well.

Chapter 1

Here the authors describe how HTML5 came about and what the history behind it is. The reader gets a general introduction to an HTML5 documents basic structure and its ease of implementation compared to previous standards.

Chapter 2

In Chapter 2 "Using the right tag for the right job" the old and new elements are described in detail. It is filled with a lot of information on how to effectively use the correct element in the right situation. If you need a refresher on old elements and need to know new elements, this chapter is for you.

Chapter 3

This is all about semantics. The aim of the standard is to provide for a semantic web that is better searchable and discoverable. The main thing I got out of this chapter is that HTML should be used to optimize content and not for styling. The good part here are a lot of diagrams that visually explain the concepts and outline the tags and their usage.

This is not a complete summary by all means. I merely try to give an overview and make you aware of this book’s use. This book is useful for designers and web developers alike. What I got out of it is a lot of information that I wouldn’t have even looked up because I wouldn’t have thought of it. It is definitely a must read for web oriented people who want to embrace the advances of HTML in the 21st century’s second decade. If you would like a complete reference and understanding of markup, this book is for you. If you are a pure programmer and want to know more about HTML5′s canvas and video element’s API, this book only scratches the surface. However, all you need to know is what is possible and this book is good at telling you that.