By Per Buer, CEO of Varnish Software

Varnish is an open source web accelerator created by Varnish Software, in Norway. Varnish accelerates your site by sitting in front of your web server, typically Apache, caching the content. It will keep the most recent requested pages in memory and serves them from memory, producing high performance responses on a Web site, both in term of capacity and response time. When content is served from memory this happens a lot faster then anything

It utilizes a number of novel techniques to achieve its performance.  One of the most characteristic is how configuration is handled. Varnish is configured in a domain specific language called Varnish Configuration Language (VCL). The VCL configuration is translated to C, compiled, loaded and executed. This gives you two immediate advantages, speed and flexibility. The speed part is easy to understand. The flexibility comes from the fact that you can write policies describing how specific pages or requests should be handled. This is best described by an example;

if (req.url ~ "^/fun/" && 

   (req.http.referer && req.http.referer !~ "^http://www.example.com/")

    ) 

    { error 403 "No hotlinking please";

The code should be easy to understand. If the URL is begins with /fun and you have a referer header this referer must bu www.example.com or we won’t deliver you the content.

Another, and for more complex, example of the power of VCL can be seen here (http://kristian.blog.linpro.no/tag/security-vcl/) where a set of developers took the mod_security rule set and transformed it into VCL. You are then able to execute the mod_security at blistering speeds with only a few limitations.

VCL can be extended by embedding C into the configuration. This is not for the faint hearted, but can give you the necessary power to work around the language limitations. Users have been using this for enabling varnish to log to syslog, to server content based on your IP’s origin (Geo-ip) and to manipulate HTTP headers in ways VCL doesn’t allow. A plugin architecture, which takes out some of the rough edges is being designed but no release date is known at this point.

Varnish does also employ a completely new scheme for logging. In stead of logging directly to log files Varnish logs everything to a segment in shared memory. This means that Varnish doesn’t get slowed down waiting for disks. Logging to disk can then be done by a separate program – having full access to what happens through the shared memory segment.

When comparing Varnish with from other Web caches you’ll see that it is incredibly fast. Under synthetic load the Varnish team has managed to push a three year old single quad core server into delivering a staggering 143000 requests per second. I must stress that this is a synthetic benchmark and it does not take into account the slow clients, misbehaving browsers and network problems of the real world. In your sizing rule should be one server per gigabit of traffic, more or less. That’s more traffic than most of us ever see, turning tuning and scaling mostly to a non-issue.

Dinside.no page download speed Pre and Post Varnish

In addition to giving you enough capacity to survive the most intense Slashdotting, Varnish’s speed will also give other advantages. Compared with an un-cached site, the pages will appear much faster – especially if you have many objects on each page. The graph measured the time it took to download one of Norway’s online IT newspapers from a installation of 14 servers running Squid to two servers running Varnish. As you can see the latency dropped quite dramatically and this was noticed by its readers which started commenting on it in the forums.

An alternative to Varnish is buying services from a content distribution network (CDN). Depending on whether you need the distributed aspect of the CDN the comparison might or might not be valid. A CDN will probably have servers closer to you reader than your data center – but their servers might not be as fast as Varnish is. Your mileage may vary.

Varnish is administered by a command line interface. You use telnet to enter the a special console where you have access to commands for loading and reload configuration, inspecting statistics. There is a “varnishadm” tool which gives you the ability to execute commands on the command line without the use of telnet. I, personally, like this much better because I have access to the key bindings and line editing capabilities in bash.

Companies such as creativecommons.org, Twitter and LinkedIn use Varnish in production, and there is commercial support and an enterprise offering from Varnish Software for enterprise customers who don’t want to rely completely on the community.

For further reading I recommend this introduction to Varnish. It will give you a good idea of how you should proceed. Introduction to Varnish http://varnish-cache.org/wiki/Introduction

Preparing Varnish and Wordpress for Slashdotting http://varnish-cache.org/wiki/VarnishAndWordpress

About the Author

Per Buer is the CEO of Varnish Software, a spin-off of Redpill Linpro, the company which has been developing Varnish for the last three years. Per was working as programmer, then a sysadmin, before being caught in management positions.

By Melissa Soriano, Engineer, NASA Jet Propulsion Laboratory

In this article, I will describe how to bring up Debian Linux on an Embedded System. 
The hardware chosen for the embedded system is a PowerPC 440EPxX.  I have chosen Debian for the operating system of choice to load on the embedded hardware because of my prior experience with it and because it is a free distribution that continues to be actively maintained and supported.  Linux is a powerful operating system and is valuable on an embedded hardware platform for real-time applications.  An example is real-time digital signal processing.  Alternatives to Debian include Montavista and Wind River, which are both commercial Linux distributions.

1. Description of the hardware

The hardware is a custom digital signal processing printed circuit board which includes a PPC440EPx embedded processor and five Xilinx Virtex 5 Field Programmable Gate Arrays (FPGAs).  The PPC440EPx was designed with an RS-232 interface, a JTAG debug and flash programming interface, 2 Gigabit Ethernet interfaces, and one status LED. Initially, the system is completely bare of software and contains no operating system.  The goal is to bring up Debian Linux on this embedded hardware platform, and the same process applies to most embedded platforms.

2. Compiling the custom bootloader (uboot) and Linux kernel

The bootloader is a small program that is used to load the operating system.  Popular bootloaders include the IBM PowerPC Initialization and Boot Software (PIBS) and the U-Boot Universal Bootloader from DENX.  In this example, U-boot was used.  The board was designed with a JTAG interface to the PPC440EPx.  An Abatron BDI debugger can be used to load U-boot and associated environmental variables onto the Flash using the JTAG interface.  In this system, the Flash is a Mirrorbit.  The U-boot source is available from DENX.  In the include/configs directory is sequoia.h, which includes detailed configuration information about the system clock, Flash, base addresses, serial ports, ethernet ports, and usb ports.  This file is specific to the Sequoia reference board and will require modification based on the actual system.  Recompilation of u-boot was necessary.  A standard Debian Linux filesystem was used along with a custom version of the Linux DENX kernel (linux-2.6.25-svn1995-dirtyxx).  The latest version of DENX Linux can be obtained from git.denx.de.  Please note that DENX support for sequoia was started during development of the 2.6.19 kernel.  Most of the relevant configuration is handled by arch/ppc/configs/sequoia_defconfig.  This file was modified for the current system and the Linux kernel was recompiled using DENX’s Embedded Linux Development Kit (ELDK).

export ARCH=ppc
export CROSS_COMPILE=ppc_4xxFP-
PATH=$PATH:/opt/eldk/usr/bin:/opt/eldk/bin
make sequoia_defconfig
make menuconfig
make uImage

3. Programming  Flash memory with the bootloader (uboot)

The Abatron must be configured for the intended processor (in this case, PPC440EPx) using a configuration file.  This file includes information about the translation lookaside buffer (TLB), the caches, the peripheral bus configuration registers, the type of Flash, the target CPU type, and the Abatron’s host.  In this example, the bdi configuration file for the AMCC 440EPx “Sequoia” reference board (evkit440epx.cfg) was used as a starting point and modified based on differences between the “Sequoia” reference and the actual system.  The most common modifications will likely be to the host section.  If this section is correct, you should be able to power up the Abatron, telnet to it from your host, and get a prompt back.  Connect the Abatron JTAG to the system’s JTAG and at the Abatron prompt, erase the flash and program U-Boot and its associated environmental variables.

440EPX>reset
440EPX>erase440EPX>prog 0xfff60000 env.bin
Programming env.bin , please wait ....
Programming flash passed
440EPX>prog 0xfffa0000 u-boot.bin
Programming u-boot.bin , please wait ....
Programming flash passed
440EPX>md 0xfffa0000
fffa0000 : 27051956 552d426f 6f742032 3030392e  '..VU-Boot 2009.
fffa0010 : 30382028 53657020 31372032 30303920  08 (Sep 17 2009
fffa0020 : 2d203137 3a30373a 31332900 00000000  - 17:07:13).....
fffa0030 : 00000000 00000000 00000000 00000000  ................
fffa0040 : 00000000 00000000 00000000 00000000  ................
fffa0050 : 00000000 00000000 00000000 00000000  ................
fffa0060 : 00000000 00000000 00000000 00000000  ................
fffa0070 : 00000000 00000000 00000000 00000000  ................
fffa0080 : 00000000 00000000 00000000 00000000  ................
fffa0090 : 00000000 00000000 00000000 00000000  ................
fffa00a0 : 00000000 00000000 00000000 00000000  ................
fffa00b0 : 00000000 00000000 00000000 00000000  ................
fffa00c0 : 00000000 00000000 00000000 00000000  ................
fffa00d0 : 00000000 00000000 00000000 00000000  ................
fffa00e0 : 00000000 00000000 00000000 00000000  ................
fffa00f0 : 00000000 00000000 00000000 00000000  ................440EPX>reset
440EPX>go

If you are connected to the system’s serial console, you will now see U-Boot come up.  Interrupt the autoboot to gain access to the U-Boot command line.

U-Boot 2009.08 (Nov 16 2009 - 15:54:12)

CPU:   AMCC PowerPC 440EPx Rev. A at 533.333 MHz (PLB=133, OPB=66, EBC=66 MHz)
       No Security/Kasumi support
       Bootstrap Option C - Boot ROM Location EBC (16 bits)
       Internal PCI arbiter enabled, PCI async ext clock used
       32 kB I-Cache 32 kB D-Cache
Board: Sequoia - AMCC PPC440EPx Evaluation Board, Rev. FF, PCI=66 MHz
I2C:   ready
256 MB

FLASH: 64 MB
NAND:  No NAND device found!!!
0 MiB
USB:   Host(int phy) Device(ext phy)
DTT:   1 FAILED INIT
Net:   ppc_4xx_eth0

Type run flash_nfs to mount root filesystem over NFS

Hit any key to stop autoboot:  0
=>

4. Booting into Linux

U-Boot can now be used to boot into Linux.  U-Boot has many configuration variables which must be set correctly.  In particular, the ethaddr, ipaddr, and serverip U-Boot environmental variables must be set in order to tftp the Linux kernel image. It is a good idea to make a memory map of the system’s flash.  In this example, the Linux kernel image will be booted using the network using tftp protocol to a load address of 400000.  Any arbitrary addresses in the FLASH may be used as long as the load address does not overlap with the location of U-Boot and its associated environmental variables.The file system is accessed using Network File System (NFS) and is mounted on the host.

=> setenv ethaddr 00:10:EC:01:05:E6
=> setenv ipaddr 192.168.100.51
=> setenv serverip 192.168.100.206
=> setenv hostname myproject
=> saveenv

=> setenv bootargs ip=${ipaddr}:${serverip}:${gateway}:${netmask}:${hostname}:${netdev}:off panic=1 console=ttyS0,115200
=> setenv bootcmd 'tftp 400000 uImage.mine; bootm 400000'

The saveenv command saves the environmental variables to persistent storage.  You can now reboot the system.  This time do not interrupt the autoboot and watch Linux come up.

U-Boot 2009.08 (Nov 17 2009 - 09:05:32)

CPU:   AMCC PowerPC 440EPx Rev. A at 533.333 MHz (PLB=133, OPB=66, EBC=66 MHz)
       No Security/Kasumi support
       Bootstrap Option C - Boot ROM Location EBC (16 bits)
       Internal PCI arbiter enabled, PCI async ext clock used
       32 kB I-Cache 32 kB D-Cache
Board: Sequoia - AMCC PPC440EPx Evaluation Board, Rev. FF, PCI=66 MHz
I2C:   ready
256 MB
FLASH: 64 MB
NAND:  No NAND device found!!!
0 MiB
USB:   Host(int phy) Device(ext phy)
DTT:   1 FAILED INIT
Net:   ppc_4xx_eth0

Type run flash_nfs to mount root file system over NFS

Hit any key to stop autoboot:  0
miiphy_reset
PHY ADDR 0x0 1040
MII devices: 'ppc_4xx_eth0'
Current device: 'ppc_4xx_eth0'
PHY OUI = 0x01C1, Model = 0x02, Rev = 0x01
0
in 4xx_enet.c
Waiting for PHY auto negotiation to complete.... done
ENET Speed is 1000 Mbps - FULL duplex connection (EMAC0)
Using ppc_4xx_eth0 device
TFTP from server 192.168.100.206; our IP address is 192.168.100.51
Filename 'uImage.mine'.
Load address: 0x400000
Loading: #################################################################
         ######################################
done
Bytes transferred = 1502956 (16eeec hex)
## Booting kernel from Legacy Image at 00400000 ...
   Image Name:   Linux-2.6.25-svn1995-dirty2
   Image Type:   PowerPC Linux Kernel Image (gzip compressed)
   Data Size:    1502892 Bytes =  1.4 MB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
Linux version 2.6.25-svn1995-dirty2 (soriano@host) (gcc version 4.2.2) #18 Thu Nov 19 09:50:36 PST 2009
AMCC PowerPC 440EPx Sequoia Platform

Linux configures the Ethernet physical device interface.  You can now login to myproject and use Debian Linux on the embedded system.

soriano@host:~$ telnet 192.168.100.51
Trying 192.168.100.51...
Connected to myproject.lab.
Escape character is '^]'.
Debian GNU/Linux 5.0
myproject login: ops
Last login: Thu Nov 19 17:52:22 UTC 2009 on ttyS0
Linux myproject 2.6.25-svn1995-dirty2 #18 Thu Nov 19 09:50:36 PST 2009 ppc

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
ops@myproject:~$

This is a basic overview of how a developer can load Debian Linux onto an embedded system.  Debian Linux is a free open-source operating system that can be used in conjunction with embedded systems for real-time applications.

More Resources

u-boot: http://www.denx.de/wiki/U-Boot
Abatron: http://www.abatron.ch/
Debian Linux http://www.debian.org/

About the Author

Rich Sharples, Director of Product Management for Applications Platforms and Developer Tools at Red Hat, spoke with TheBitsource.com about the recent updates in Java Enterprise Edition Six, and a look into upcoming technologies in the Java platform.

Java EE 6 includes some new features such as managed beans, contexts and dependency injection, bean validation, JavaServerFaces 2.0, and the Web Profile.

Rich has been involved with developing, designing and evangelizing Java and Java-based enterprise software sine 1998. He spent 8 years in the Software Group at Sun Microsystems before joining Red Hat.

Q: How much of Java EE has changed between Java EE 5 and 6

Rich Sharples: Java EE 6 is a major revision in every respect. Some quick analysis of the EE 5 and EE 6 specifications :
•    Java EE 6 introduces 8 new specifications / APIs – Java EE 5 introduced the same number
•    Of the 23 other APIs (from Java EE 5) – 11 have changed (either bumped up a major or minor version) and one has moved to Java SE. Java EE 5- changed 10 out of 16 existing specs. / APIs
•    The page count has increased by about 6%, Java EE 5 actually reduced the page count by 40% (compared to J2EE 1.4). If you include the new Java EE 6 Web Profile – the page count increase is about 30%.

What the quick analysis doesn’t tell you is the significance of some of the changes. The major goal of Java EE 5 was to  increase developer productivity and adoption of EE. This was addressed via focusing on POJO programming, annotations and convention-over-configuration. The way in which EE 6 attempts to address some of the complexity of the EE platform is far more ambitious.

First there is the introduction of the Java EE 6 Web Profile. This is a subset of Java EE 6 and focuses on the core technologies and APIs required for building modern Web Applications. The specification only references 15 specs/APIs so it significantly reduces the potential learning curve for new developers moving to the Java EE platform. The idea of profiles has been discussed for many years and Java EE has lost some mind share to lighter-weight offerings built on Tomcat.  Developers now have a reliable standard profile and will be able to choose offerings from a number of vendors and open source projects.

The second major area that has changed is the introduction of two APIs that for the first time bring a consistent programming model to Java EE – JSR-299 (Contexts and Dependency Injection) and JSR-303 (Bean Validation).

CDI (Contexts and Dependency Injection) essentially standardizes the best design patterns distilled in popular, contemporary frameworks such as Seam, Spring and Guice and some of the earlier but far more limited advances in Resource Injection in Java EE 5. As well as standardizing a powerful, type safe CDI allows developers to manage the scope, state, life-cycle and context for objects in a much more declarative fashion – in line with modern programming paradigms and good practices.

Bean Validation simplifies how data is validated across presentation, business and data access tiers of an Enterprise Java application – removing the burden of maintaining duplicated boiler-plate code and potential inconsistencies.

In short these two specifications make it much simpler for developers to produce well structured and maintainable code using Java EE technologies like EJBs and JavaServer Faces.
Q: What is Weld and what does it enable developers to do? Why was it chosen to be integrated into JEE 6?

Rich Sharples: Weld is the Reference Implementation of JSR-299 (Contexts and Dependency Injection). Red Hat is the specification lead for 299 so was responsible for producing the Reference Implementation (Weld) and the TCK. Weld is container neutral and can be run in JBoss AS, GlassFish, Tomcat and ?Jetty?.

Weld is also the core of Seam 3.0 which is a powerful open source framework for building sophisticated Rich Internet Applications using the Java EE platform.

Q: Has anything been removed in JEE 6?

Rich Sharples: The only specification that was removed from Java EE 6 spec. was the Java Beans Activation Framework (JAF). JAF is now part of Java SE – so it’s basically moved down a layer – it hasn’t really been removed.

Java EE has always provided strict compatibility with previous versions and this release was no exception. But it’s worth noting that for the first time a major feature has been marked as deprecated (meaning it will be removed in a future version). In this case EJB CMP 1.1 Entity Beans have been marked deprecated – CMP was never a popular technology and it’s unlikely that it would be considered for any new development.

Q: How does JEE 6 make Web application development easier or more effective?

Rich Sharples: CDI and Bean Validation provide a much cleaner and consistent programming model across tiers of a Web Application. CDI builds some core capabilities for Dependency Injection and Context  Management into the platform and does it in a consistent and standard way.

The new 2.0 version of Java Server Faces (very popular for building Web UIs) has undergone some significant changes. Major changes include Facelets as first class view declaration language (overcoming the limitations of JSP); much simpler creation of custom component, native support for Ajax and improved state and event management.

Finally the new Java EE 6 Web Profile – provides a much easier on-ramp for developers coming from other platforms and technology.

Q: Are there any changes to the JVM in JEE6 that result in better performance or manageability?

Rich Sharples: There are many JVMs from different vendors and each have their own performance tricks and enhancements, Red Hat contributes to and fully supports the OpenJDK and has an interest in making Linux the best and most efficient place to run Java.

Q: How will JDK7 fit into the JEE6 Roadmap, or will it be released in JEE 7?

Rich Sharples: There is a staggering of SE and EE release. Java EE typically requires a specified level of Java SE – in the case of Java EE 5 – SE 5 was required though many Java EE 5 products will also use Java SE 6. In the case of Java EE 6, Java SE is required but many products will, over time, support both Java SE 6 and Java SE 7. So it’s likely that Java SE 7 will be supported by Java EE 6 products but Java EE 6 will not require Java SE 7.

Q: Will there be support for the Da Vinci Machine Project in Java EE 7?

Rich Sharples: The Java SE 7 JSR still does not exist – all we can do is speculate what will be in Java SE 7. Irrespective of what will be in Java SE 7 – there have already been significant advances in binding non-Java languages to the Java Virtual Machine. Jruby, Scala, Groovy / Grails, Django, JavaScript, Jython are some of the more established examples. There has also been some experimentation in integrating non-Java languages into Java EE environments – Red Hat’s own TorqueBox is a good example. Torquebox provides a first class Ruby on Rails experience for developers with the power and flexibility of running on a proven enterprise platform – JBoss App Server 5.0.

More Resources

Rich Sharples Blog http://blog.softwhere.org
JBoss World http://www.redhat.com/promo/summit/2010/
Java EE Home http://java.sun.com/javaee/

By Ryan Compton

“We basically just killed Moblin” -Pete Kronowitt

Last week in Barcelona it was announced that Nokia and Intel will merge the better parts of Maemo and Moblin into a new product aimed at creating a unified framework for mobile device development, MeeGo. Today at SCALE 8x I spoke with Intel’s Pete Kronowitt about the project and what it means for the open source community.

What is MeeGo?

MeeGo is a new platform designed to run on mobile devices under the auspices of the Linux Foundation. The MeeGo OS is specialized to run efficiently on your netbook, smartphone, or carputer and will deliver a full Internet experience (ie Java, Flash, Silverlight, Adobe Air) on each of those machines. A stylish user interface provides quick access to commonly used applications with an emphasis on social networking tools. MeeGo is aggressively stack optimized for execution with a small footprint and speed. You can go from the “ON” button to the internet in 20 seconds on a low end netbook with a solid state drive.

What does MeeGo bring to open source?

Meego allows a developer to build an application in the well understood Qt development environment and run it with the core OS designed for Moblin. This provides a straightforward way to get your application running on any equipment supported by MeeGo allowing for more expansion of open source software into the exploding mobile device market. There were 17 publicly committed Linux vendors for Moblin and most have provided positive feedback about the MeeGo announcement.

What about the history of Meego?

Near the end of 2007 Moblin was released with the aim of bringing open source to mobile devices. Not too long afterward ASUS came out with a triumphantly successful line of netbooks so we got on board with that to. We were able to get OpenHand UK to bring us Clutter while collaboration with Nokia brought a couple hundred more developers in and we were able to release MeeGo in Q1 2010.

How can the open source community start developing?

The vast majority of the core OS has been built with a considerable amount of work by engineers at Intel and Nokia. Right now we’d like to see people writing apps for release through Intel’s AppUp or Nokia’s Ovi. The major push is for development to happen in Qt but there will be continued support for Clutter and GTK.

What browser will MeeGo use?

Look for the developer preview in March.

How do you get all those UI graphics and still have a reasonable battery life?

A major challenge is making sure support exists for the broad developer community. Since the community utilizes both multimedia frameworks, MeeGo suppports both gstreamer and Helix. More important to extend battery life is ensuring the entire software platform is optimized. Tools play an important role to analyze misbehaving applications. Intel created a widely used tool called PowerTOP which provides tuning suggestions to lower power consumption. Definitely check out PowerTop.

Can you bring up a terminal?

Yes. (Note that bringing up a terminal in front of the SCALE audience results in much applause with oohs and ahhs)

Anything else you’d like to say?

Not too long ago you would watch Star Trek and Captain Kirk would say “Computer” and then ask the computer to do just about anything. We’re basically there now.

If you’d like to speak with Pete he can found playing guitar at the Talking Stick in Venice, California on February 23rd.

More Resources:

Meego Home Page at the Linux Foundation http://www.linuxfoundation.org/lp/page/meego

Ryan Compton’s Homepage http://www.math.ucla.edu/~rcompton/

PyCon 2010:  Event-Wrap Up

PyCon 2010 was  the largest PyCon to date, at over 1000 attendees, and had the the largest percentage of women in attendance in years past, at about 10%. This year’s PyCon presented many different interesting topics and concepts in Python development such as the core system, (which focuses more on compiler/interpreter design), testing, and Web application development. There was something for everyone at PyCon, for beginners, advanced users, and even for those with no previous experience of Python.

The Community

The Python community at PyCon represents a largely interactive community. There was a large academic presence at Pycon, in addition to high presence from Google as well, being that Python is one of their top three used programming languages. Eldarion, a development shop of Python Web applications, who houses the Pinax project was also a contributing company to the event. The event had a significantly friendly tone to it. Most people were chatting casually about various projects and current events in Python, and there was a much less corporate feel to the event than other technology conferences.

Interactivity

The most interesting thing to me was the amount of interactivity at Pycon. If you wanted to contribute to an open source project, the development sprints (which are still underway) were a fantastic way to get involved. Regardless of your experience level, provided you know how to write programs in Python, you could get involved. Even if you didn’t and were a complete Python newbie, in most cases someone was still more than willing to sit down and show you the ropes on a given project. I was able to implement my first Django Web application while at PyCon, which to me was of a benefit in itself, and I look forward to checking out TurboGears as well for designing Web applications.

Keynotes and Lighting Talks

Mark Shuttleworth, Founder of Ubuntu was a highlighted keynote speaker at the conference, who gave a talk on software quality, cadence, and design. In his keynote, he spoke mainly about the importance of writing testable code, and something he called “STFU protocol”, which essentially means as a developer, have  users test your software, and design software with the user in mind. This keeps a focus on accelerating the pace of development in open source software, and benefits the entire ecosystem. He stated that, “when you make a [software] release, you energize the [open source] community”.

Other great talks although they weren’t able to go into much details were the lightweight talks. Particularly, I found the Haystack project, presented by Daniel Lindsley to be of particular interest, primarily due to the prevalence of search in Web applications as well as it’s plug-able nature into Apache Solr and other search engines.

Talks

The talks at PyCon were exceptional. Many talks were geared toward the core Python core system, and CPython implementations, including talks on Understanding the Python Global Interpreter Lock (GIL), PyPy, Unladen Swallow, and other CPython implementations. Other talks took a less core Python perspective and focused Python Web application development, primarily with Django and TuboGears. SourceForge.net delivered an interesting presentation on how they are rebuilding their Web site using Python, TurboGears and MongoDB. Their main concept for using these technologies was that they feel them to be the best available software technologies currently available.

There was also some bleeding edge topics on data stores, including Tobias Iverson’s talk on Neo4j, which is a graph-based databases rather than a relational database. This was a new cutting edge topic and primarily focuses on the aspect of efficiently scaling to very large numbers in terms of storing and accessing data (billions?).

Collin Winter gave an excellent talk on Unladen Swallow, and how they are making Python faster by utilizing LLVM and other clever methods.  One of the most notable announcements in this talk is that many of the optimizations developed in Unladen Swallow will be integrated into Python 3.

For the practical person writing simple (or complex) command line tools, Catherine Devlin gave a talk on how to create easy command-line application using cmd and cmd2. cmd is a standard Python library for creating command line applications. cmd2 is a drop-in replacement for cmd that also includes a searchable command history, interacting with the shell, and redirecting to files, for starters. The Python command-line SQL client, sqlpython, utilizes cmd2 as it’s interactive shell.

Most of the users of Python are not core developers, and although there were many core developers at the conference, this conference could easily apply to anyone who just loves programming and learning more. There were many topics just on how to code “cool things” in Python, so it is a conference for anyone who enjoys writing programs and solving problems, regardless of experience level.

With high-profile companies such as Youtube, SourceForge.net, and Lolapps using Python to power their applications, as well as being able to do powerful and fun things quickly. PyCon offers a lost of support to developers wanting to attend. It also is gathering momentum as a development utility for testing. Python testing suites were a prevalent topic at the conference, with a  from Sauce Labs, a company specializing in Web testing and Selenium support. Frank Wierzbicki, also of Sauce Labs gave a keynote talk on the state of Jython, the Python implementation in Java.

PyCon videos have been made to all on blip.tv and most presentations now have their slides posted on the PyCon Web site.

Fun and Interesting Resources

Many users found enjoyment on the PyCon IRC channel on freenode: #pycon
The PyCon Live Stream http://pycon.djangodose.com/
PyCon OHWar http://pycon.ohwar.com/
PyCon Videos on Blip.TV (Getting released gradually for 2010) http://pycon.blip.tv/
PyCon on Micro-community (with videos) http://python.mirocommunity.org/
Grig Gheorghiu’s PyCon Take Aways (A different perspective) http://agiletesting.blogspot.com/2010/02/my-top-10-pycon-takeaways.html.

Sanjiv Singh is a Python developer from New Delhi, India who contributes to the TurboGears and Toscawidgets projects. Singh recently gave a talk on “TurboGears2 Geopspatial Framework at PyCon 2010, which is a framework for developing Geographic Information System Applications, such as the Blind Audio Tactile System.

Q: What is the relationship between a GIS application and a Geospatial framework?

Sanjiv Singh: A GIS application is any software application that deals with information pertaining to “location” or “geometry” of objects and is capable of:
(a) processing such information, such as finding out objects located within 100m raduis of another
(b) displaying such information, such as showing objects on a map

A Geospatial Framework, on the other hand, is a collection of programs that work in an integrated manner towards easily developing GIS applications. The word Geospatial is a more formal term for what we commonly refer to as GIS.

Q: Did you develop the Turbogears Geospatial Framework?

Sanjiv Singh: Yes and No. The framework consists of many different python gis applications / libraries such as featureserver, mapfish, tilecache, Shapely, GeoJSON, etc. These are developed and maintained by other developers. However, I integrated these applications with the TurboGears2 Web Framework so that TurboGears developers may now develop GIS applications easily. I also contributed significantly to GeoAlchemy which is an extension of SQLAlchemy, the python database toolkit for GIS applications. GeoAlchemy now forms the ORM layer of the TG2 Geospatial Framework.

Q: Why didn’t you choose Django, which has the GeoDjango framework readily available?

Sanjiv Singh: GeoDjango is a matured GIS framework, however, it is meant for Django applications. Having developed several applications using TurboGears, I did not want to re-implement existing applications or to learn another framework. So I used several Python gis libraries with my TurboGears applications to make them do Geospatial work. So, in order to make it easier for other TurboGears developers to use GIS I integrated these libraries through a new TG2 extension called tgext.geo. This extension along with GeoAlchemy now works as a Geospatial Framework.

More Resources

Sanjiv Sighn’s Open Source Projects http://www.ohloh.net/accounts/sanjiv
Sanjiv Singh’s PyCon Presentation (Including Slides) http://us.pycon.org/2010/conference/schedule/event/16/
Sanjiv Singh’s Blog http://sanjiv-singh.blogspot.com/

TurboGears http://turbogears.org/
Toscawidgets http://toscawidgets.org/

By Ron Evans (@deadprogram) of deadprogrammersociety.com

It was a beautiful sunny day here in Los Angeles, and I had spring already in my step, because I was headed to LARubyConf 2010. It was almost time for the main event, and there was lots of good stuff in store. The proceedings began with some brief introductions from the hardest working man in technology conference video, Coby Randquist (@kobier), then on with the show.

Oh S***: How to bring a big Rails website down (and how not to)
Tim Morgan (@riscfuture)

Tim’s presentation was named “Oh S***: How to bring a big Rails website down (and how not to)”. It was an excellent collection of real-world experiences from Tim’s work at Scribd with database problems, and their eventual solutions.

Scribd handles a lot of documents. Really, they do. In fact, they have over 50,000 new docs uploaded every day. With a site this big, it is easy to break it, and that does happen. Most problems stem from the way Ruby on Rails handles the database interactions, which can become troublesome in a number of cases. Having caused unintentional havoc, Tim was schooled by his ‘masterful DBA’ on some key rules of thumb for keeping out of trouble.

Database Rules of Thumb

1. If you add something database related to your models, make sure you look at the actual SQL that is generated when using it.
2. When you look at the generated SQL and do not understand it, use EXPLAIN
3. If what EXPLAIN reports is not clear, ask someone

There were several excellent examples presented for working with Ruby on Rails and MySQL. This included both correcting issues as well as performance optimization techniques. If you are using PostgreSQL or some other data store, then many of these specific techniques may not directly apply, but it was still an excellent exercise going thru them in detail.

Alternative Data Structures in Ruby
Tyler McMullen (@tbmcmullen)

Next up was Tyler McMullen, also from Scribd, to talk about some different ways to represent your data besides just putting it into a database-like structure. By choosing a more appropriate data representation and algorithms, you can use well known techniques to solve some problems that are very performance intensive far better than just using databases.

The four structures discussed were Bloom Filters, BK Trees, Splay Trees, and Trie. As one friend commented to me afterwards, it was like a high-compression braindump at lightning-talk speed. For audience members who were not versed in the power of algorithms, or even those who were, Tyler hit some great points by not only showing how each worked, but also illustrating a real world example for each of when it could be used.

Use a bloom filter for problems like quickly handle requests for non-existing on a file server. A BK-tree is useful when you need to find best match, like spelling correction. Splay trees are great for improving performance for caches and garbage collectors. And a Trie is useful for problem like auto-completion. In fact, Tyler showed a cool example of auto-completion implemented in Rack in 20 lines of Ruby code.

Want to see code? Check out http://github.com/tyler for some ruby extensions with these structures and algorithms.

Teaching Ruby To Kids
Sarah Mei (@sarahmei)

Now we got to hear from Sarah Mei, a developer at Pivotal Labs and an expert on teaching programming to kids. As Sarah put it very amusingly, “most programming instruction = fail”. And this, despite the fact that programming is becoming part of basic literacy.

She challenged the audience to get involved with teaching kids to program. After all, Ruby is the programming language for extroverts, and teaching is not “Rocket Science”. We all already have the tools and techniques we need to get started. She then proceeded to give a bunch of specific pointers, largely based around the same principles as Agile development.

Some great tools to teach programming to kids are Shoes, Hackety Hack, and the upcoming Small Ruby which is Sarah’s own implementation of Logo in Ruby. Other tools of interest are Alice, and Kodu.

You need to find a balance between theory and practice. A child’s aptitude for math may = computer science, but aptitude at language = programming. The youngest age to teach programming is probably middle school. Pairing the kids together works well, seems to be better than pairing a teacher/student. What is the best way to get more girls programming? Starting at an earlier age.

Sarah gave a very entertaining and important talk. Each of us has a part to play in the education of the next generation of programmers, so let’s get to work!

Everything You Ever Wanted To Ask About Threads And Fibers But Were Afraid To Ask
Joe Damato (@joedamato) & Aman Gupta (@tmm1)

For those who were confused by the double take, Joe and Aman decided to pair present in each other’s scheduled talks. This was a pretty cool way to handle the massive amount of information that they have on Ruby performance and internals.

So, what is a thread? It is just a set of execution state, with instruction and stack pointers to allow for suspending and resuming the thread as needed. There are three kinds of threads: green threads, native threads, and hybrid threads.

Ruby 1.8 users green threads, but apparently does it wrong. On the other hand, Ruby 1.9 uses native threads. Erlang uses hybrid threads.

They also talked about 3 very powerful tools for performance profiling: strace (dtrace on OSX), google pref-tools, and ltrace. Using these tools, they have identified some major performance improvements in Ruby 1.8, and have these available at http://github.com/ice799 and http://github.com/tmm1/ruby187

One reason these patches are not part of Ruby’s MRI is that they are only for Ruby 1.8 and are also x86 specific.

Joe and Aman delivered when it came down to a very detailed discussion of how Ruby handles threads, as well. But their afternoon talk would be even more hardcore.

New Relic With The Best Sponsored Message Ever – Really!
Paul McCready (@kingofslugs) from New Relic gave the coolest and most fun sponsored demo I have ever seen. He showed some awesome pictures of human-powered flight (Go, Gossamer Albatross!) and in an incredibly entertaining way tied it into his brief pitch as to the benefits of New Relic’s application profiling service. Furthermore, he really knew what he was talking about. Good job, Paul! Someone give that man a raise.

After a long lunch filled with fun conversations between all the amazing Rubyists assembled, it was time for the second half.

Civic Hacking
LuigiMontanez (@LuigiMontanez)

Luigi Montanez works at the Sunlight Foundation, a non-profit organization dedicated to government transparency via open source software and APIs. They have done some really excellent things, and have started getting more developers in the community involved. Of course, they need you to get involved, which was the immediate gauntlet throw by Luigi in his talk.

So what is civic hacking? It is when the government releases data, and citizens then develop peaceful application using that public data.

He discussed the Obama iPhone app, which was a key tool in organizing people during the run-up to the last presidential election. Interestingly, the app was not created by Obama’s campaign. Instead, it was built entirely by volunteers, and was only show to the Obama staff after it has been created and was about to be released. This is a excellent example of civic hacking in action.

The concept is simply this: Open Source + Open Data = Better Government. Right now, Federal, State, and local government has not done a very good job of publishing data online, let alone providing APIs with which to access this data.

Apps For America was a contest sponsored by the Sunlight Foundation, and was very effective as getting many developers to create cool and use applications that mashup or present government data in a useful way. Fillibusted, and This We Know are just a few of the great things created by open source civic hackers as part of Apps For America, and they are planning Apps For America 2 right now.

Luigi also discussed the Aaron Swartz incident. Aaron Swartz, of Reddit fame, brought the attention of the Man on himself, when he helped out to make publicly availble information about U.S. federal court cases actually public, from a system called PACER. He was vindicated, but it points out even more strongly the need for making public data accessible to the public. Now there is a Firefox plugin called RECAP that grabs data from PACER and uploads to archive for free access, helps bypass the paywall, and makes new data available.

There are many ways to get started civic hacking. The main thing is to get involved. Because today in the internet age, we do not need large bureaucracy to support our engagement with government. Power to the people, right on!

Garbage Collection and the Ruby Heap
Aman Gupta (@tmm1) & Joe Damato (@joedamato)

The second part of Aman & Joe’s detailed retrospective of the internals of Ruby, covered two things you would know nothing about if you had never programmed in any other language than Ruby. Lucky for all Ruby programmers, all aspects of memory management are hidden from view.

After a detailed description of stacks vs. heaps, and descriptions of how Ruby handles the many internal references, they discussed Ruby memory leaks. Since you do not handle memory directly in Ruby, they are not classic memory leaks. Instead, they are reference leaks. If an object has any objects that contains a reference to it, it can never be freed. Examples are assignment of an object to instance variable, or adding an object to a hash or array.

One very interesting technique they mentioned was loading profile data into MongoDB so you can do queries on it using memprof, a new profiling tool that Aman & Joe have created. Memprof actually outputs JSON, which is MongoDB’s data storage format, which is what makes this so very easy to do.

Joe and Aman are amazingly knowledgeable about the hardcore internals of Ruby performance, and have posted their slides online, so if you want to get down and dirty with Ruby’s memory management implementation, I highly recommend checking it out. They are also getting ready to offer their memprof as a web-based service (http://memprof.com) to make the process of analyzing the massive quantities of data easier.

A New Look at Software Development: What will the next 10 years bring?
Dave Astels (@dastels)

Dave Astels is a true software genius, having been instrumental in creation of the canonical Ruby testing framework RSpec. Now working with Ruby thinktank and hosting gurus Engine Yard, he has been thinking about a few interesting ideas regarding the next generation of software development. Or, as he put it quite humorously “the alternate title for this talk is ‘Programming: You’re Doing It Completely Wrong’”

What Dave was talking about, was that the difficulty in creating software is only getting worse. Software is more complex, bigger, more distributed, parallel, and even more life critical. We cannot continue building software over the next 10 years in the same way that we have for the last 50 years.

That’s right, 50 years. The theoretical basis for all the languages that we use today were mostly determined in the 1950’s. Actually, Smalltalk (‘71) is one of the conceptual new kids on the block. Think you are modern with your functional languages like Erlang? Sorry, based on ML (‘73).

As far as programming tools, Dave cannot understand why we are still using file system based tools like Emacs and Vim. Not that he likes any current IDE’s better. But clearly there are other ways to work with code, other than editing text files on disk.

We need languages and tools that have parallelism & distributed processing baked in and that actively prevent bugs. The programming idioms that have become second nature to us as everyday tools don’t work.

Dave wants to see intelligent tools, that are more cooperative with the user, and do more for the developer. Perhaps even written using Rubinius. AHe says we need a new way of programming, that incorporates parallel strategies, problem decomposition, data structure design, and algorithmic organization.

It was a very thought provoking talk, and I am hoping that some of the concepts take root and grow.

Mobile Ruby
Sarah Allen (@ultrasaurus)

Sarah Allen is CTO of software consultancy Blazing Cloud, where she has been doing mobile development, and is the creator of the “Ruby Conf 1.0″ app on the iTunes app store. That app was written using the Rhodes mobile framework, which was the main topic of Sarah’s talk.

“Writing mobile apps are like getting tattoos. some people like the pain, but most people just want the effect”, Sarah told the amused audience. “And Brand Transcends Platform,” she explained as she was juggling 3 handsets for the live portion of her demo. As the only presenter to show actual running code, I salute her!

Content is what really matters when developing a mobile app. Not so much the specifics of UI experience. It is key to remember that “Mobile != Pocket Desktop.” Also the promise of “Write Once, Run Anywhere” does not really exist for mobile development, but at least we can achieve “Run Anywhere.”

The rest of her presentation consisted of looking through the source code for the Ruby Conf 1.0 app. Since Rhodes is loosely based on Ruby on Rails, and the code you write is Ruby, most Rails developers will be able to recognize a lot of similarities. There are important differences, however. One interesting thing about Rhodes, is that there is actually a very small Ruby VM running on the mobile device when you compile your app togther. Apparently the most recent version also supports the popular Net:HTTP lib, so it is possible to create some pretty interesting applications that talk to remote services without using the RhoSync server to do so.

There have been a number of apps approved on the iTunes app store that were developed using Rhodes, including the Wikipedia mobile app. Rhodes is definitely worth taking a look, if you a Rubyist who wants to get into mobile development without learning Objective-C.

The Big Rewrite – Doing It Right
Rich Kilmer (@rich_kilmer)

The conference keynote was given by RubyCentral founder and longtime Rubyist (10 years!) Rich Kilmer. Rich runs a consultancy called InfoEther, and a few months back his colleague Chad Fowler (@chadfowler), also a RubyCentral founder and member of InfoEther, wrote a blog post against the so-called “Big Rewrite”. Rich’s entire presentation was effectively a rebuttal against Chad’s post. Did I mention that Bob Martin (@unclebobmartin) was a speaker at RailsConf 2007, and the topic of his talk was speaking vehemently against the “Big Rewrite”? Oh, we Rubyists are a wild and crazy bunch!

Rich’s presentation was quite excellent. His really knows not only Ruby, but has a fantastic grasp of how to build scalable, loosely coupled systems, based on a lot of real-world experience. So his plan for success is based on an actual big rewrite being done for a client by his consultancy, which is actually underway at the time of his presentation. And this is no mere incremental change, this is a total bottom to top rewrite.

Actually doing the “Big Rewrite” consists of understanding the drivers for the rewrite, preparing for the rewrite, executing the rewrite, then flipping.

The drivers, or reasons for doing the rewrite, are that it must be business driven, it must NOT be technology driven, you don’t call it a rewrite, and you must complete it in a major business cycle.

When preparing for the rewrite, you must drop a major release just before the rewrite, you must understand your problem domain, and you must break down the current system into logical sets of functionality.

Executing the rewrite requires implementing pieces as REST services, using central authentication, and implementing shared behaviors and using them. Furthermore,
you must automate system administration tasks (Puppet or Chef), use a standard worker framework (minion or resque), and dedicate resources to repeatable data migrations. Lastly, keep service code consistent & models clean, and make sure to use the right tool for the right job.

When ready to actually do the flip, you need to perform incremental migrations of historic data, prepare business users for possible disruption, and do practice runs of the flip scenario several times. Finally, enable “read only” of the old system during final client flip, do the flip, and provide a way to fall back if flip fails.

A few final points: use polyglot programming to handle needs with best language for certain tasks. if you need 20 lines of Lua or Perl, no big deal. Do it. You may need to add functionality to the existing system if the flip fails, or if you need to add functions during the big rewrite itself. And if two things should be named the same, but have different behavior, you probably actually have two different systems. You can do a big rewrite, but you need to stay within business cycle.

Wrap Up
It was a rewarding day of serious Ruby awesome here in Los Angeles. When it comes to enthusiasm, and vibrancy of the Los Angeles Ruby community, we are really coming into our own. That is certainly one reason that we had so many cool out of town presenters and attendees. Thank you to all of the presenters, attendees, and organizers who helped make LARubyConf a great conference. See you next year!

This guest blog post was written by Ron Evans (@deadprogram) from The Hybrid Group, a team of Los Angeles Ruby on Rails software developers. Ron is a contributor to several open source projects, occasional author and speaker, and iconoclast. Check out his blog Dead Programmer Society for many more posts about Ruby.

By Ryan Compton

I had the chance to speak with Stephen Spector of Xen.org about his work on this exciting virtualization project. We spoke about the basics of Xen, Xen development, and Xen in Antarctica, which is one of the goals for the Xen Around the World project.

What does Xen do?

Xen is a collaboration between industry giants, universities, and hobbyists to provide an open source (GPLv2) hypervisor for server virtualization. The hypervisor provides a layer between the guest and the hardware allowing for multiple operating systems to be run on the same server.

How does Xen differ from other hypervisor’s?

Xen is developed by a combination of corporations where each company builds their unique strength into the software. Intel optimizes hardware, Oracle virtualized databases and Citrix works with desktop virtualization. VMware, our major propitary competitor, develops all their software alone and thus relies on a smaller support base.

Why is it exciting to be working in virtualization now?

Only about 20% of servers are virtual right now and that number is growing rapidly. As high speed internet becomes more widely available cloud computing will only continue to grow at least as fast as it has in the past few years. Xen is already being used for Amazon’s EC2 and Citrix systems’s servers.

How can people become involved with the development of Xen?

First off, get to our friendly mailing lists and participate in the conversations. The Xen hypervisor  is a highly sophisticated construct that most hobbyists are not involved with, at this level changes must be approved by the hypervisor’s creator, Keir Fraiser. Most of the development that we see from individuals comes at the management space layer. We also have people that contribute by  simply answering questions on the message boards or explain how they use Xen.

I noticed you have a project “Xen Around the World“, where are some of the more interesting places you’ve found Xen?

The goal here is to get Xen in Antarctica. So far we’ve hit every other continent but we still can’t find a way to get there. There is one group using Xen in a fairly remote part of Africa but we can’t seem to find out exactly what they are doing there. There’s also a group of guys in Britain who run horse racing bets on an OS/2 network, they used Xen to keep their bets going while they updated their hardware.

Anything coming up?

We’ve got a conference in San Jose on April 28th and 29th. That’s a real good place to be if you want to get to know some of the fine details of the hypervisor’s memory mapping.

For more information Steve can be found traveling to any of the following locations.

Ryan Compton’s Homepage http://www.math.ucla.edu/~rcompton/

The PyCon team has begun to post the videos from PyCon 2010 on blip.tv. More videos will appear online as they become available. See what you’ve missed here http://pycon.blip.tv/.

Rick Capeland, SourceForge.net gave a presentation on, “How Python, TurboGears, and MongoDB are Transforming SourceForge.net”, at PyCon 2010 today in Atlanta, Georgia. Copeland discussed SourceForge’s desire to migrate off PHP code and start re-factoring their customer facing site, using a with the titled recipe of Python, TurboGears, and MongoDB. The PHP code was chosen by SourceForge to be the best technology at the time back in 1998, and SourceForge believes that a combination of Python and NoSQL is the best technology that the current times have to offer.

What was interesting was that Copeland goes on to explain that their reluctance to use Memcached due to the network overhead and object serialization and de-serialization. MongoDB can about the same performance as Memcached by utilizing local instance slaves, according to Copeland.

Copeland also recently released the open source project, Ming, which is an open source library for interfacing with MongoDB from Python. SourceForge plans to continue to re-factor more of their Web applications using the same technologies in the near future.

Copeland’s talk was interesting, and provides a different perspective on rebuilding a Web site from scratch. SourceForge is definitely breaking out the fun toys for their site redesign, and it will be interesting to see how quickly they can redesign their site and with how much manpower using these utilities.

What would also be interesting to see is a performance comparison of the site before and after the redesign on Python, TurboGears, and MongoDB.

More Resources

Rick Capland’s Talk (With Slides) http://us.pycon.org/2010/conference/schedule/event/110/
MongoDB http://www.mongodb.org/display/DOCS/Home
TurboGears http://turbogears.org/