Skip to content

Setting up a Python-CGI – Appendix

In my previous post, I demonstrated how you can set up a simple Python-CGI.

There, we changed the default setup and made our CGI accessible directly at:

localhost!

But maybe you don’t want to change the default setting or want to host a bunch of applications. Then it is necessary to make it accesible with a link like:

localhost/mycgi

When you already followed my previous post, you’ve got a running Python-CGI setup (cgi-bin, html).

Making this also accessible in our favoured syntax is really pretty easy. We just have to define a new configuration file (mycgi_http.conf) and have to symlink this file to Apache2 (/apache2/conf.f)

Let’s do it:

  1. Make the new configuration file:
     gedit /home/user/cgi-projects/mycgi/mycgi_http.conf
  2. Add the following lines in there:
    #<VirtualHost *:80>
     ScriptAlias /mycgi/cgi-bin /home/user/cgi-projects/mycgi/cgi-bin
     Alias /mycgi /home/user/cgi-projects/mycgi/html/
    #</VirtualHost>

    I made it deliberately simple, but you could make it also as complex as in the previous example! But I think you can see the significant difference!

  3. Symlink this file to Apache2:
     sudo ln -s /home/user/cgi-projects/mycgi/mycgi_http.conf /etc/apache2/conf.d/mycgi_http.conf
  4. Restart Apache2:
     sudo /etc/init.d/apache2 restart
  5. Test it, finally: http://localhost/mycgi

When you have done everything correctly you should see the same index-file as in the previous example! Always remember to replace ‘user’ with your login!

Advertisements

Setting up a Python-CGI

This is just a simple description on how to setup a basic Python-CGI!
Make sure to have ‘Apache2’ and Python (min V2.7) installed and you can start!

Set up a new Apache2 confiuration and test it

  1. Create a folder for your CGI:
     mkdir /home/user/cgi-projects/
     mkdir /home/user/cgi-projects/mycgi/
     mkdir /home/user/cgi-projects/mycgi/html
  2. Add a index.html file there:

    echo '<b>Cool! The Apache2 configuration has been changed!</b>' > /home/user/cgi-projects/mycgi/html/index.html
  3. Make a copy of the Apache2 default http-configuration file:
    sudo cp /etc/apache2/sites-available/default /etc/apache2/sites-available/mycgi
  4. Open it in an editor (gedit in my case):
     sudo gedit /etc/apache2/sites-available/mycgi
  5. Change the default directory … from:
      <Directory /var/www/>

    …to…

     <Directory /home/user/cgi-projects/mycgi/html/>
     
  6. Define this file (mycgi) as apache2 httpd configuration file:
     sudo a2dissite default && sudo a2ensite mycgi
  7. Restart Apache2:
     sudo /etc/init.d/apache2 restart
  8. Test it on: http://localhost

When you can see now the text Cool! The Apache2 configuration has been changed! you have done good so far. If not…check if you have adapted everything correctly…escpecially replacing ‘user’ with your login!

Building a basic CGI from that setup

  1. Add a CGI-folder to the project:
     mkdir /home/user/cgi-projects/mycgi/cgi-bin
  2. Add a python-file in there and open/save it:
    gedit /home/user/cgi-projects/mycgi/cgi-bin/mycgi.py
     
  3. Add these lines to that file:
     #!/usr/bin/env python
     print "Content-type: text/html\n"
     print "Ha! It even works with a python file!"
  4. Define permissions of this file (to make it accessible for Apache2):
    chmod +x /home/user/cgi-projects/mycgi/cgi-bin/mycgi.py
  5. Open the http configuration file again and adapt the cgi-bin definition:
    sudo gedit /etc/apache2/sites-available/mycgi
  6. Change the default directory … from:
     <ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/>

    …to…

    <ScriptAlias /cgi-bin/ /home/user/cgi-projects/mycgi/cgi-bin/>
     
  7. Restart Apache2:
    sudo /etc/init.d/apache2 restart
  8. Test it on: http://localhost/cgi-bin/mycgi.py

That’s it! You’ve got a basic Python-CGI on your local machine!

Use the CGI to return a JSON-file

This is my basic aim on the cgi-scripting, so I will describe this finally!

  1. Make a new python-file (json.py):
     gedit /home/user/cgi-projects/mycgi/cgi-bin/json.py
  2. Add these line to the file:
    #!/usr/bin/env python
    import simplejson as json
    print "Content-type: application/json\n"
    data = { 'a':'A', 'b':[2, 4], 'c':3.0 }
    #This way...
    #print json.dumps(data)
    #..or this way
    print(json.JSONEncoder().encode(data))

    Following some descriptions in the web, I made the failure to forget the ‘\n’ at the end of the first printed line.
    Consequential error message: ‘Premature end of script headers: json.py’

  3. Remember to change permissions:
     chmod +x /home/user/cgi-projects/mycgi/cgi-bin/json.py
  4. Testing it on: http://localhost/cgi-bin/json.py

Request the JSON-file from your CGI

Finally:

  1. Change the data output of json.py to:
     data = {'elements':['Content1', 'Content2', 'Content3']}
  2. Change the index.html file of your Apache2 configuration to that:
    <!DOCTYPE html>
    <html>
    <head>
     <title>CGI-Test</title>
     <meta charset="utf-8" />
     <script src="http://d3js.org/d3.v3.min.js"></script>
     <style></style>
    </head>
    <body>
     <script>
     d3.json("http://localhost/cgi-bin/json.py", function(error, data){
       d3.select('body').selectAll('div').data(data.elements).enter()
         .append('div').text(function(d){return d})
     })
     </script>
    </body>
    </html>

With this JavaScript implementation, we call the CGI-script and use the returned JSON-file to build add some contents to the html-site!

Acknowledgement:

These sites helped me a lot:

So thank you for your knowledge sharing!!!

Visualising GeoJSON-files on GitHub

Some weeks ago GitHub launched a very cool new tool, that enables you to visualise GeoJSON- and TopoJSON-files directly!

Get more informations in the corresponding help and blog posts!

So far, I found it cool but did not really care about it … until today!

I just tried it, played a little bit with all the functionalities and defined some Test-Datasets, that conclude (in my opinion) all necessary aspects.

You can find them within the corresponding Repository!

All Geometries of GeoJSON-Specification
github_maps

All possible styling options for markers
github_maps2

Compile Java on command line and Include packages

As I’m getting more and more a friend of pure commandline programming, I tried to compile a Java-file directly!
But this file has references to external classes, provided in *.jar files! What made me struggling a little bit!

Like always…Googling that helped a lot to find the answer!

So that’s the trick:


javac -cp '.:package_1.jar:package_2.jar' MyJavaFile.java

Notice: That works only on Linux systems

Keep node.js up to date!

Installing topojson on Ubuntu is pretty simple, right?
But what if you’d like to update it and afterwards it is completely wrecked!?! That happened to me yesterday and I’d like to share my experiences with you! Especially you, who is not experienced in working with the node package manager ‘npm’, as I am (or at least was before I had this problem)!

I reported the issue to the developer of topojson (Mike Bostock), but I already had the feeling that it is not a problem on topojson! As he did not report any idea, I knew I have to do it on my own.

1st solution

After many trials and errors, trying to update other node-modules and some resignations … I remembered that I have read ‘”pre” versions of node cannot be installed’, somewhere. That does not fit to my failure, but maybe it is basically not good to use a ‘pre’ version of node.js.
So I installed a node version manager ‘nvm‘ and updated node to the latest, not ‘pre’, version:

cd ~/Software
git clone git://github.com/creationix/nvm.git ~/nvm
. ~/nvm/nvm.sh
nvm install v0.10.13
nvm use v0.10.13

Now I can simply install topojson as I did before!

sudo npm install -g topojson

But sadly, this helps just for the moment. Opening a new terminal…forces me now to start nvm, before I can call topojson:

. ~/nvm/nvm.sh
nvm use v0.10.13

It seems, that topojson is now linked to the ‘node’ version that I’ve installed with ‘nvm’.

2nd solution

Therefore, I re-install node completely. So I

  • downloaded node-version 0.10.13, here
  • installed it normally:
    ./configure
    make
    sudo make install
    node -v
    
  • and re-installed topojson
    npm install -g topojson
    

Now, everything is clean again and I can call topojson directly after opening a new terminal, without initialising ‘nvm’!

Styling a readme.md

Styling a readme.md file is pretty simple.

Here you can find a readme.md file that is also a cheat sheet, containing the basic styling elements of readme files (that I know so far):

readme-styling

A similar file, but with focus on the question ‘What should be described in a readme file’ can be found here!

Tips on not described styling elements are welcome!

 

Short evaluation of neighbor detection using topojson

This will be only a quick and (maybe) dirty evaluation of the neighbor detection that I’ve implemented: http://wwwpub.zih.tu-dresden.de/~rklammer/html/vorlagen/RKMapping_neighbor_finding.html

I’ve written a simple function that evaluates the neighbors of a feature, by using the ArcGeometries extracted from a topojson file, that I’ve described in previous posts (1, 2).

After that, I found out, that there is also a function for neighbor detection included to topojson.js … but … it seemed to be a bit slower than my version … this has to be analysed!

My method

Pretty simple…just do this for each feature:

  1. loop through the geometry of a feature
  2. extract all arcs (at least only there id’s)
  3. analyse the members of each arc (arc has to have the info about its members included for that)
  4. 1 member = no neighbor – 2 members = a neighbor

Actually, I am a noob JavaScript-programmer and do not understand the syntax of topojson’s neighbor detection to 100%…so I will not try to describe that!

Analysing the performance

The analysis will be a little bit dirty, as I will not calculate an arithmetic mean of many test results. This is possible, as I am more interested in the relative time differences, between datasets of different feature amount, than in their absolute values! As you can see (and test by yourself) in the interactive test scenario, I used:

  • municipalities:
    • clipped –> 2946 features
    • full        –> 11626 features
  • states (full) –> 16 features
  • counties (full) –> 402 features

Now…lets have a look at the table!

 
topojson
.neighbors()
getArcs()
get
Arcs
Of
Neighbores()
States (full)
(16 features
1,164
seconds)

full_states
0,008
seconds
0,058
seconds
0,002
seconds
Counties (full)
(402 features
2,985
seconds)

full_counties
0,185
seconds
0,277
seconds
0,008
seconds
Municipalities (clipped)
(2946 features
3,083
seconds)

clipped_muni
1,005
seconds
0,191
seconds
0,013
seconds
Municipalities (full)
(11626 features
26,638
seconds)

full_muni
18,513
seconds
0,940
seconds
0,016
seconds

Basically, my own function seems to be faster than the topojson function. But I need the ArcCollection for being able to detect the members of each arc…so I have to include the processing time of that too. That is why the table contains a column for the getArcs() as well for the getArcsOfNeigbors() functions.

Processing, only 16 german states, took all together 0,06s which is compared to 0,008s way more processing time. That means topojson is ca. 85% faster than my function.

This seemingly advantage is also detectable for the processing of 402 county features. There we have 0,285s (total) against 0,185s, what is only ca. 35% faster then my function.

I said seemingly, because the next two datasets show the difference in performance between both functions. The processing of the ArcCollection still takes way more time than following neighbor detection, but all in all it takes 0,204s while topojson.js takes 1,005s. Now my function is ca. 80% faster.

This first indication is manifested by the last, hughest dateset (11626 features). Here, topojson.js takes long 18,513s while my function takes only ca. 1s. That is a ledge of ca. 95%…WOW!

But that seems to be a little bit strange. Topojson’s processing time is rising uproportionally and I do not have a clue why. The processing time of my function, in contrast, grows proportionally…at least the getArcsOfNeighbors() function, which processes super fast!

Finally, let’s have alook at the growth:

Number of features:

  • 2512% (16 – 402)
  • 733% (402 – 2946)
  • 383% (2946 – 11626)

Processing time – topojson.neighbor():

  • 2313% (0,008 – 0,185)
  • 543% (0,185 – 1,005)
  • 1842% (1,005 – 18,513)

Processing time – getArcs() + getArcsOfNeighbors():

  • 475% (0,06 – 0,285)
  • – 29% (0,285 – 0,204)
  • 490% (0,204 – 1,000)

There you can also see the unnormal growth of processing time, for the topojson function. It is not associated with the growth of number of features…a growth of 1842% is almost five times the feature growth (383%)!

On the other hand my implementation does has also have an unproportional growth. At least the getArcs() function processes 2946 municipality features faster then 402 county features. It is a fact but not as drastic as this exemplary values indicate…average mean for 10 test processings:

  • Counties (full): 0,157s
  • Municipalities (clipped): 0,148s

That is strange and I have to investigate on that more deeply…