Tag Archive > python

Ubuntu, Django, Postgresql and Nginx, a rock solid web stack.

» 02 May 2012 » In Uncategorized » 3 Comments

Lately I’ve been working on developing a web application project,   I guess everyone involved in such process had to make the uneasy decision to choose the right Web Stack to run the application.  Multiple factors usually drive the final decision,total cost of ownership (TCO), scalability, stability and performances. The TCO factor can be easy overcome with FLOSS (Free and open source software), scalability is a very broad term and concept  that doesn’t have any clear answer. Stability can be achieved with software that is  widely used and widely reviewed and tested for a long time, never pick something new just because it’s cool. Doesn’t matter if the website serve millions of users or just a few hundred, slow or bad performances can make the user experience a nightmare. Performance enhancement can be  achieved with, Memory caching system, Low memory footprint web servers, good and slim database design, query optimization to work under heavy loads. Please  be aware that applications, services and workloads differ from case to case, you might need more write capacity rather than read, serving multimedia files or whatever else, don’t blame me if you have troubles with your production application after following this how-to. For me and some other, this setup works quite well, I hope the same for you :-) .

This tutorial covers the set up a free, full-featured web development framework based on Ubuntu Server 12.04(LTS), Django 1.4 stable, Postgresql database V. 9.1, Nginx web server V.1.1.19 and Gunicorn.

Ubuntu Server
Linux for high-volume server workloads
http://www.ubuntu.com/business/server/overview

Django
The web framework for perfectionists with deadlines
www.djangoproject.com

Postgresql
A powerful, open source object-relational database system.
http://www.postgresql.org

Nginx
A web server with a strong focus on high concurrency, performance and low memory usage.
http://wiki.nginx.org/Main

Gunicorg
A WSGI HTTP Server, the link between Django an Nginx.
http://gunicorn.org/

Postgresql installation and setup


apt-get install postgresql python-psycopg2

sudo -u postgres psql template1

ALTER USER postgres with encrypted password 'a_strong_password';

template1=# CREATE DATABASE test;

template1=# CREATE USER test WITH PASSWORD 'test';

template1=# GRANT ALL PRIVILEGES ON DATABASE test to test;

In /etc/postgresql/9.1/main/pg_hba.conf

Change

local   all             postgres                                peer

To

local   all             postgres                                md5

local   all             all                                     peer

to

local   all             all                                     md5

Restart Postgresql server with:

/etc/init.d/postgresql restart

To thest the local connection to the database please use:

psql -d postgres -U postgres

and

psql -d test -U test

Django installation and setup.

apt-get install aria2c

aria2c http://www.djangoproject.com/download/1.4/tarball/

tar xvfz Django-1.4.tar.gz

python setup.py install

mkdir /var/djangoweb/

django-admin.py  startproject myproject

django-admin.py  startproject myproject

cd myproject/

mkdir /var/djangoweb/myproject/static

ln -s /usr/local/lib/python2.7/dist-packages/django/contrib/admin /var/djangoweb/myproject/static/

in myproject/settings.py

Add

import os.path
PROJECT_DIR = os.path.dirname(__file__)

Database setup in django

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2', # Add 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'test',                      # Or path to database file if using sqlite3.
        'USER': 'test',                      # Not used with sqlite3.
        'PASSWORD': 'test',                  # Not used with sqlite3.
        'HOST': '',                      # Set to empty string for localhost. Not used with sqlite3.
        'PORT': '',                      # Set to empty string for default. Not used with sqlite3.
    }
}

Then go to the INSTALLED_APPS section and uncomment
django.contrib.admin line.

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # Uncomment the next line to enable the admin:
    # 'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
)

and # ‘django.contrib.admin’,

INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    # Uncomment the next line to enable the admin:
    'django.contrib.admin',
    # Uncomment the next line to enable admin documentation:
    # 'django.contrib.admindocs',
)

in myproject/urls.py uncomment the following lines.


from django.contrib import admin
admin.autodiscover()

url(r'^admin/', include(admin.site.urls)),

The  urls.py  file have to look like this.

from django.conf.urls import patterns, include, url

# Uncomment the next two lines to enable the admin:
from django.contrib import admin
admin.autodiscover()

urlpatterns = patterns('',
    # Examples:
    # url(r'^$', 'myproject.views.home', name='home'),
    #url(r'^myproject/', include('myproject.foo.urls')),

    # Uncomment the admin/doc line below to enable admin documentation:
    # url(r'^admin/doc/', include('django.contrib.admindocs.urls')),

    # Uncomment the next line to enable the admin:
    url(r'^admin/', include(admin.site.urls)),
)

Now the basic parameters are defined, we must perform the first database synchronization,
go on top of the project root and issue the “python manage.py syncdb” command.

You’ll see the creation of the first Django tables inside the database,  you also need to create
the administrator user.

Creating tables ...
Creating table auth_permission
Creating table auth_group_permissions
Creating table auth_group
Creating table auth_user_user_permissions
Creating table auth_user_groups
Creating table auth_user
Creating table django_content_type
Creating table django_session
Creating table django_site
Creating table django_admin_log

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes
Username (leave blank to use 'root'): phillip
E-mail address: phillip@foo.com
Password:
Password (again):
Superuser created successfully.
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

At this point the Django interaction with the database looks working properly.

In order to get access to Django administration page you need to run the
embedded python web server.


python manage.py runserver 0.0.0.0:80
Validating models...

0 errors found
Django version 1.4, using settings 'myproject.settings'
Development server is running at http://0.0.0.0:80/
Quit the server with CONTROL-C.

Point the browser to:

http://server_ipaddress/admin/

Gunicorn installation and setup

Install gunicorn


apt-get install git python-setuptools
git clone git://github.com/benoitc/gunicorn.git
cd gunicorn/
python setup.py install

inside the root project folder myproject

vim gunicorn.conf.py

bind = "127.0.0.1:8888"
workers = 2
#recommended (2 x $num_cores) + 1

To run gunicor go in the root project folder and run:

gunicorn_django -c gunicorn.conf.py

To Daemonize the Gunicorn process add -D

gunicorn_django -c gunicorn.conf.py -D

Yum must get:

gunicorn_django -c gunicorn.conf.py
2012-05-01 19:53:28 [6176] [INFO] Starting gunicorn 0.14.2
2012-05-01 19:53:28 [6176] [INFO] Listening at: http://127.0.0.1:8888 (6176)
2012-05-01 19:53:28 [6176] [INFO] Using worker: sync
2012-05-01 19:53:28 [6180] [INFO] Booting worker with pid: 6180
2012-05-01 19:53:28 [6181] [INFO] Booting worker with pid: 6181

Nginx installation and configuration


apt-get install nginx-full  openssl

rm /etc/nginx/sites-enabled/default

touch /etc/nginx/sites-available/django

ln -s /etc/nginx/sites-available/django  /etc/nginx/sites-enabled/django

In /etc/nginx/sites-enabled/django please add:


server {

listen   server_ipaddress:80;
#server_name *.site1;

access_log /var/log/nginx/djangosite1.access.log;
error_log /var/log/nginx/djangosite1.error.log;

location / {
        proxy_pass http://127.0.0.1:8888;

}

#location /static {
#         autoindex on;
#         root /root/blog/public_html/;
#    }

location /static/admin {
         autoindex on;
         root /var/djangoweb/myproject/static/admin;
    }

}

/etc/init.d/nginx restart

Django admin web interface.

Now you can point the browser to  http://server_ipaddress/admin/, be aware the myproject directory is just the schema container for the project, no applications are yet configured or installed. I strongly advise to follow “Writing your first Django app” how-to, do not copy paste the code.

If you want to find more about django development please take a look to:

Django From the Ground Up video series
DjangoCon 2011
The Django book

I hope this document will be useful for who is struggling with Django setup. Stay tuned.

Did you like this? Share it:
Scridb filter

Continue reading...

Tags: , ,

Dirt cheap USB Temperature Sensor with python SMS alerting system.

» 12 April 2012 » In Uncategorized » No Comments


In the last few weeks I had to design an effective and cheap temperature monitoring system for the server room. After searching a bit on the web I found this USB Thermometer Temperature dongle on Ebay, only 10 bucks with free shipping. As soon the sensor arrived, I plugged it into one Ubuntu based server and surprisingly worked immediately, below you can see the dmesg output.

[12737.064004] usb 2-1.3: new low speed USB device using ehci_hcd and address 6
[12737.161227] usb 2-1.3: configuration #1 chosen from 1 choice
[12737.164379] input: RDing TEMPer1V1.1 as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/2-1.3:1.0/input/input6

Along with monitoring the temperature I wanted to have an alarming system that sends alarm SMS message to mobile phones when the temperature has passed a certain threshold. So I subscribed to textmagic, an SMS gateway provider. Textmagic has a good set of API, and of course they have the python API which I used to write my Pytemp2sms.py script.

USB dongle software and settings.

wget http://www.isp-sl.com/pcsensor-0.0.1.tgz
tar xvfz pcsensor-0.0.1.tgz
cd pcsensor-0.0.1
cp 99-tempsensor.rules /etc/udev/rules.d/
cp pcsensor /usr/local/bin/

Reboot the server.

To test the dongle run the pcsensor command .
You must get this kind of output.

2012/04/11 14:30:50 Temperature 60.58F 15.88C

SMS alarm.

Textmagic python API installation.

https://code.google.com/p/textmagic-sms-api-python/wiki/UserManual


apt-get install python-setuptools
easy_install PyTextMagicSMS

Pytemp2sms script.

Script download: http://bailey.st/code/bitsofpy/Pytemp2sms/


#!/usr/bin/python
import sys
import subprocess
import fileinput
import textmagic.client

#Temperature threshold in C
threshold = 5
#Balance threshold
credits = 100
#TextMagic account details, before using textmagic service
#you need to register and obtain an API password
#(different from your login password)
#
#https://www.textmagic.com/app/wt/account/api/cmd/password
username = "yourusername"
password = "APIpassword"
#Owner of the Textmagic sms account.This number will receive
#the credit balance .
owner = "+44phonenumberoftheowneraccount"

'''License: This software is distributed under
   the terms  and  conditions  of GPL  - GNU
   General  Public  License.'''

def usage():

    print " ------------------------------------------"
    print "|              Pytemp2sms                  |"
    print "|                                          |"
    print "|Temperature Sensor with sms alert system. |"
    print "|                                          |"
    print "|www.bailey.st                             |"
    print "|phillip@bailey.st                         |"
    print " ------------------------------------------ "

    print "\n"
    print "Usage instructions:"
    print "\n"
    print "Pytemp2sms.py checktemp | checkbalance"
    print "\n"

def checktemp():

    output = subprocess.Popen(["/usr/local/bin/pcsensor"], stdout=subprocess.PIPE).communicate()[0]
    output2 = output.split()[4]
    temp = output2[:2]

    if int(temp) > threshold:

        for phonebook in fileinput.input("/usr/local/bin/phonebook.txt"):

            client = textmagic.client.TextMagicClient(username, password)
            client.send("WARNING - Server room temperature above threshold!!! PLEASE CHECK!!!", phonebook)

    else:
        sys.exit()

def checkbalance():

    client = textmagic.client.TextMagicClient(username, password)
    balance = client.account()['balance']
    print balance

    if balance < credits:
        client.send("WARNING - Textmagic credit balance below threshold, please top the account", owner)

    else:
            sys.exit()

if len(sys.argv) < 2:
    usage()
    sys.exit()

if "checktemp" in sys.argv[1]:
        checktemp()

if "checkbalance" in sys.argv[1]:
        checkbalance()

Phonebook file.

The file phonebook.txt must contain all the mobile numbers that need to receive the alarm sms. Please follow this syntax:

+44numberone
+44numbertwo
+44numberthree

Crontabs.


*/30 * * * * /usr/local/bin/Pytemp2sms.py checktemp >/dev/null 2>&1
*/60 * * * * /usr/local/bin/Pytemp2sms.py checkbalance >/dev/null 2>&1
Did you like this? Share it:
Scridb filter

Continue reading...

Tags:

Bits of python: import a CSV file into a MySQL database.

» 22 February 2012 » In Uncategorized » 1 Comment

This is my first blog post of the series “Bits of python”, a journey into the Python programming language done with simple examples. From time to time I’ll write about Python and Django and how they interact with Unix and Linux systems. All the scripts available are stripped versions of my daily work, mostly file parsers, databases administration and reporting tools. This first example is about how to import a CSV (Comma-separated values) files into a MySQL database and can be very handy when you work with spreadsheets and databases.

The python modules used in this example are, the CSV  and the python-mysqldb, the first is already present into python and for the second one you can install it just with apt-get install python-mysqldb.

The sample code consist in three files, presidents.csv, presidents.sql and importCSV.py, can be downloaded from here, once downloaded the archive can be unpacked with tar xvfz csvintomysql.tar.gz . Now it’s time to take a look to our file, let’s go into csvintomysql directory, cd csvintomysql.

The presidents.csv file is the list of all the presidents of the United States of America and it look like this (the list goes more long):

Presidency , President, Took office, Left office, Party, Home State

1,George Washington,1789-04-30,1797-03-04,Independent ,Virginia
2,John Adams,1797-03-04,1801-03-04,Federalist ,Massachusetts
3,Thomas Jefferson,1801-03-04,1809-03-04,Democratic-Republican ,Virginia
4,James Madison,1809-03-04,1817-03-04,Democratic-Republican ,Virginia
5,James Monroe,1817-03-04,1825-03-04,Democratic-Republican ,Virginia

As you can see there are several fields delimited by a comma that need to be exported and inserted into the Mysql database using the python CSV module.

The presidents.sql file is the series of MySQL instruction to create a database called csvdb and a table called presidents. In order to keep simple the creation of the database you need to log into MySQL as root and from the database prompt type:


mysql> source presidents.sql
Query OK, 1 row affected (0.00 sec)

Database changed
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.02 sec)

The importCSV.py is the Python script that parse and insert the CSV data into the database, the connection credential are already configured to connect to the database csvdb with csv as user and csv as password. Please note, if you want to modify the credential you need to change the settings in the first place, precisley in the presidents.sql file. Do not use this script in production environment!!!!! till you changed the Mysql login credential for the csv user.

#!/usr/bin/python
#simple python csv parser that export Comma
#Separated Value data into a MySQL database.
#phillip@bailey.st
import csv
import MySQLdb
# open the connection to the MySQL server.
# using MySQLdb
mydb = MySQLdb.connect(host='localhost',
    user='csv',
    passwd='csv',
    db='csvdb')
cursor = mydb.cursor()
# read the presidents.csv file using the python
# csv module http://docs.python.org/library/csv.html
csv_data = csv.reader(file('presidents.csv'))
# execute the for clicle and insert the csv into the
# database.
for row in csv_data:

    cursor.execute('INSERT INTO PRESIDENTS(PRESIDENCY ,PRESIDENT \
            ,TOOKOFFICE ,LEFTOFFICE ,PARTY ,HOMESTATE)' \
            'VALUES(%s, %s, %s, %s, %s, %s)',  row)
#close the connection to the database.
cursor.close()
print "Import to MySQL is over"

Line 5 and 6 are the import module declaration.

Line 9 to 12 are the MySQL connection credential.

Line 13 prepare the cursor to handle the connection.

Line 16 the csv.reader will read and parse the presidents.csv file.

Line 19 a for loop will be performed.

Line 21 will execute the insert into PRESIDENTS table query.

Line 25 will close the database connection.

You can run the script with:

root@:# python importCSV.py
Import to MySQL is over

After the

mysql> use csvdb;

select * from PRESIDENTS;


+----+---------------------+------------+-----------------------------+------------+------------+-------------------------------------------+---------------+
| id | cur_timestamp       | PRESIDENCY | PRESIDENT                   | TOOKOFFICE | LEFTOFFICE | PARTY                                     | HOMESTATE     |
+----+---------------------+------------+-----------------------------+------------+------------+-------------------------------------------+---------------+
|  1 | 2012-02-22 23:45:05 | 1          | George Washington           | 1789-04-30 | 1797-03-04 | Independent                               | Virginia      |
|  2 | 2012-02-22 23:45:05 | 2          | John Adams                  | 1797-03-04 | 1801-03-04 | Federalist                                | Massachusetts |
|  3 | 2012-02-22 23:45:05 | 3          | Thomas Jefferson            | 1801-03-04 | 1809-03-04 | Democratic-Republican                     | Virginia      |
|  4 | 2012-02-22 23:45:05 | 4          | James Madison               | 1809-03-04 | 1817-03-04 | Democratic-Republican                     | Virginia      |
|  5 | 2012-02-22 23:45:05 | 5          | James Monroe                | 1817-03-04 | 1825-03-04 | Democratic-Republican
+----+---------------------+------------+-----------------------------+------------+------------+-------------------------------------------+---------------+

I hope the example will be useful to understand how to work with spreadsheets and CSV data sets. If you have doubts or questions, please leave a comment on the post. All the code is freely available at http://bailey.st/code/bitsofpy/csvintomysql/

Best.

Did you like this? Share it:
Scridb filter

Continue reading...

Tags: