Boredom & Laziness

WalkDir 0.3 released (for more Python versions, thanks to Shining Panda!)

2012-01-31T22:44:00.001+10:00

WalkDir is my Python support library that aims to make it as easy to work with filtered directory listings as it is to walk over entire directory trees with os.walk().

The module's design tries to take full advantage of Python's iterator model - most of its functionality is provided by pipelined iterators that accept os.walk() style iterables and expose the same interface themselves.

The only major functional change in version 0.3 is that these pipelined iterators now make sure they pass along the objects produced by the underlying iterators, and only use indexing operations to access the individual fields. Previously they would use tuple unpacking to access the directory details, which restricted the supported types to those with exactly 3 fields and also had the side effect of replacing the underlying objects with ordinary 3-tuples.

I changed this mainly due to a new OS interface that is likely to be coming in Python 3.3: an os.walk() variant that produces a 4-tuple rather than a 3-tuple. The 4th value will be a file descriptor for the directory making it easier (in conjunction with new file descriptor based APIs in the 3.3 os module) to write filesystem modification code that is robust against symlink attacks. By passing the underlying objects through unmodified, WalkDir is now compatible with this API - all the path based filtering will still work, but the file descriptor values will also be passed along correctly.

For those that haven't seen any of my previous comments on WalkDir, the other parts of the API are just there for convenience - one factory function that constructs pipelines for you, and 3 terminal iterators that flatten out the os.walk() style triples into a simple series of paths (all paths, just the visited directories or just the file paths).

The other notable change in 0.3 is the list of officially supported versions. Previously, the module was only known to work on 2.7 and 3.2+ (since they're the versions I have on my home development machine). However, thanks to a free open source account provided by the folks at Shining Panda, WalkDir 0.3 is known to work on Python 2.6, 2.7 and 3.1+ (I even test it on PyPy and Stackless, just because I can). After pushing a broken package to PyPI for 0.2, I even have a sanity check I can run that ensures the module can be downloaded with pip and then imported on all the supported versions.

Using the SOPA protests to highlight related problems in Australia

2012-01-18T19:47:00.001+10:00

I figure this is the easiest place to publish the message I just sent to Larissa Waters, the Greens Senator that is one of Queensland's representatives in the Federal Senate. I also wrote to Yvette D'ath (our local MHR) a few days ago, but I didn't keep a copy of that one. Will this achieve anything? Probably not, but hey, at least I tried (and if none of their constituents ever write to them about it, our reps are quite entitled to assume we're all OK with them selling out the county to legacy US media interests):

Senator Waters,

With today being the day Wikipedia and a wide range of other sites have either gone dark or taken other action to protest draconian internet censorship legislation making its way through the US Congress, it seems an opportune time to highlight our own government's ongoing concerning behaviour on that front.

Of particular concern is their continuing refusal to release details of a secretive meeting between government representatives and representatives of the same organisations that are behind the draconian US bills currently being protested. The government even deliberately excluded representatives of a number of community interest organisations that sought to attend these discussions.

These legacy media companies (aka horse drawn carriage manufacturers) are flailing around wildly as the rise of free and open digital communications networks (aka automobiles) threatens the cherished gatekeeper role they have enjoyed for the past few decades as media distributors. They have failed to adapt, and are increasingly being bypassed as artists, writers, musicians, comedians and other media creators find ways to use the power of the internet to connect more directly with their fans. These direct connections are great for both artists and fans, but place the intermediaries like YouTube, Apple iTunes, Amazon, BandCamp, Flickr, etc, in the role of service providers to the artists and fans rather than gatekeepers to widespread distribution. Unfortunately, instead of going gracefully into that good night, these organisations are investing inordinate sums of money worldwide in lobbying for legislation that would make the permissive, open practices of most of these new service providers a recipe for prohibitively high legal liabilities, effectively making those practices unsustainable and thus breaking the internet as we know it today.

Australia already markedly shifted many intellectual monopoly policies to favour the interests of US copyright holders at the expense of Australian citizens when we signed the US-FTA some time ago. We have also participated in the secretive process of drafting the Anti-Counterfeiting Trade Agreement, which spends far more time considering digital copyright infringement than it does actual counterfeiting. The current negotiations over membership in the Trans-Pacific Partnership agreement raise legitimate fears that Australia's intellectual monopoly policy will be shifted even further towards the draconian position of the United State Trade Representative, even as those policies are being protested strongly within the US itself.

In line with your published policy on community participation in government, do the Greens plan to publicly question the government over their apparent willingness to place the interest of large US companies ahead of those of individual Australian citizens?

Regards,
Nick.

New Year Python Meme - December 2011

2011-12-22T23:15:00.000+10:00

I'm normally a curmudgeon about this kind of thing, but I enjoyed reading some of the other posts in this series Tarek kicked off, so I decided to make my own contribution.

1. What's the coolest Python application, framework or library you have discovered in 2011?
The move to Red Hat marked my entry into the world of web development (previously I'd merely been in interested observer of that world, rather than a participant). By far my favourite discovery since making that change is django-rest-framework - with that, I can use my web browser to browse early iterations of my server's REST API directly, without needing to write custom clients to process the JSON data from APIs that are still in a state of flux.

As a service, ReadTheDocs has been an absolute revelation - between that, code hosting & issue management sites like BitBucket and GitHub and of course PyPI itself, it's now possible for an open source project to have a quite respectable web presence without the developers needing to understand anything more than Sphinx, source control and the project they're working on.

2. What new programming technique did you learn in 2011?
REST would be the big one. I'd had some general exposure to the concept in the past, but there's no substitute for sitting down and building it into a product when it comes to understanding a programming or API design technique.

3. What's the name of the open source project you contributed the most in 2011? What did you do?
CPython, by far - kibbitzing on python-dev and python-ideas (and import-sig too these days), writing and reviewing several different PEPs, documentation updates, code reviews and patch applications, as well as working on my own things (including the still-in-progress integration work for the 'yield from' expression that's coming in 3.3).

I also recently started up 4 separate open source projects - 3 PyPI modules to hopefully address deficiencies I see in the current standard library offerings, plus the upstream open source project for my current development efforts at Red Hat:

contextlib2 (ContextStack has some potential as a new building block)
WalkDir (the idea here is to be the "itertools for os.walk()")
Shell Command (let Python handle control flow, the shell actual commands)
PulpDist (Bringing a semblance of order to small-scale rsync mirror networks)

4. What was the Python blog or website you read the most in 2011?
Planet Python.

5. What are the three top things you want to learn in 2012?
From a work point of view, getting my RHCSA (Red Hat Certified System Administrator) is at the top of the list. Coming up to speed on AMQP (Advanced Message Queuing Protocol) is a close second. Finally, I want to fill in more of the gaps in my very sketchy knowledge of web UI development (i.e. HTML/CSS/Javascript).

6. What are the top software, app or lib you wish someone would write in 2012?
I want to see the __preview__ namespace (in particular, the regex module) make it into Python 3.3. But that requires a volunteer to step up and write the PEP, write the code and generally champion the idea (if we have to wait for me to do it, there's no way it will happen before 3.4).

Want to do your own list? here's how:

copy-paste the questions and answer to them in your blog
tweet it with the #2012pythonmeme hashtag

Help improve the Python 3.3 Standard Library...

2011-12-16T23:40:00.001+10:00

... and hopefully help yourself with current programming projects, too.

Some recent programming activities left me underwhelmed by a few of the standard library's included batteries. This has already led to a significant revamp of the subprocess module documentation to steer new users away from the Popen Swiss army knife (unless they really need it) and to explain the commonly needed parameters more clearly. It still needs work (the notes and warnings are far too repetitive), but it at least introduces things in the right order now (high level convenience API that most people want first, lower level Popen API that some people need second).

However, for 3.3 I'd like to improve things even more in at least three areas: invocation of the system shell for administration tasks, better tools for traversing filesystem directories and programmatic management of deterministic resource cleanup (i.e. not relying on the garbage collector).

Accordingly, I have 3 projects up on PyPI (with docs on ReadTheDocs and source control and issue tracking on BitBucket):

WalkDir: os.walk() style iterators with file and directory filtering (both inclusion and exclusion), depth limiting and symlink loop detection, as well as convenience iterators to flatten os.walk() style iterators into a series of paths (either all walked paths, just the directories or just the files). I currently plan to make (at least some of) these part of the shutil module, but exactly what gets added will be based on the feedback I receive on this module and its API design.
Shell Command: Convenience APIs that combine subprocess invocation with string interpolation. Interpolated strings are escaped with shlex.quote() by default, with a custom conversion specifier ("!u", for unquoted) used to invoke the standard interpolation process. It also features an experimental API where I'm tinkering with the use of select.select() on subprocess pipes (I'm not sure it achieves a lot over simple blocking IO in its current form, though). The current plan for this API is that it will be added directly to the subprocess module (well, the stable and sensible parts will be, anyway - I still have my doubts about the select.select() experiment)
contextlib2: This module basically exists to let me publish and gather feedback on ContextStack, a proposed addition to contextlib for 3.3 that should make it easier to manage deterministic resource cleanup programmatically (i.e. without coupling it as directly to code layout as simple with statements do).

Feedback on any and all of these is appreciated, either here or on the respective issue trackers. It isn't a foregone conclusion that any of these APIs will be added at all, so examples of real world use cases would definitely be helpful.

Correcting ignorance: learning a bit about Ruby blocks

2011-10-08T06:03:00.001+10:00

Gary Bernhardt pointed out at PyCodeConf that I didn't know Ruby even half as well as I should if I wanted to really understand why Ruby programmers rave about blocks so much (I started this before his talk, but it touches on his key point about the centrality of blocks to Ruby's design, and Python's lack of a similarly endemic model for code interleaving). So I set about trying to fix that (at least, to the extent I can in 24 hours or so). Unsurprisingly (since I'm not interested in becoming a Ruby programmer at this point in time), I approached this task more in terms of what it could teach me about Python (and its limitations) rather than in figuring out the full ins and outs of idiomatic Ruby. So feel free to bring it up in the comments if you think I've fundamentally mischaracterised some aspect of Ruby here.

The first distinction: two kinds of function

It turns out the first distinction shows up at quite a fundamental level. Ruby has two kinds of function: named methods and anonymous procedures. The semantics of these are quite different, most notably that named methods create their own local namespace, while anonymous procedures just use the namespace of the method that created them (so they're almost like ordinary local code).

Python also has two kinds of function: ordinary functions and generator functions. The name binding semantics are identical, but the invocation style and semantics are very different. Lambda expressions and generator expressions provide syntax for defining these inside an expression, but under the hood the semantics are still the same as those of the statement versions.

The closest you can get to a Ruby style anonymous procedure in Python is to create a named inner function and declare every otherwise local variable explicitly 'nonlocal' (in Python 3 - nonlocal declarations aren't available in Python 2). Then all name binding operations in the inner scope would also affect those names in the outer scope.

Actually, make that three kinds of function

The named method vs anonymous procedure distinction actually doesn't fully capture Ruby's semantics. Blocks (which is what I was most interested in learning about), add a new set of semantics that don't apply to the full object versions: they not only use the namespace of the defining method for their local variables, but their parameters are pass-by-reference (so they can rebind names in the calling namespace) and their control flow can affect the calling method (i.e. a return from a block will cause the calling method to return, not just the block itself). While somewhat interesting, I don't think these are actually all that significant - the core semantic difference is the one between Ruby's anonymous closures and Python's generators, not the dynamic binding behaviour of blocks.

The implications: blocks versus coroutines

This initial difference in the object model for code execution has created a fundamental difference in the way the two languages approach the problem of interleaving distinct pieces of code. The Ruby way is to define a separate piece of the current function that can be passed to other code and invoked as if it was still inline in its original location, then resuming execution when the called operation is complete. The Python way is to suspend execution, hand control back to the invoking piece of code, and then resume execution of the current code block at a later time (as determined by the invoking code).

Hence, where Ruby has specific syntactic sugar for passing a block of code to another method (do-end), Python instead has syntactic sugar for various invocation styles for coroutines (iteration via for loops, transactional code via with statements).

It's also the case that coroutines are not (yet) as deeply bound into Python's semantics as blocks are into Ruby. Whereas Ruby had blocks from the beginning and defined key programming constructs in terms of them (such as iteration and transactional style code via blocks), Python instead is built around various task specific protocols that may *optionally* be implemented in terms of coroutines (e.g. for loops, the iterator protocol and generators, the with statement, the context manager protocol and the contextlib.contextmanager decorator applied to a generator).

Callback programming and hidden control flow

One interesting outgrowth of the Ruby approach is that callback programming actually becomes a fairly natural extension of the way the language works - since programming with blocks is callback style programming, the invoking code doesn't really care if the called method runs the passed in block immediately or at some later time. Whether you consider this a good thing or a bad thing is going to depend on how you feel about the merits and dangers of hidden control flow.

During the discussions that led to the introduction of the with statement in Python 2.5, Guido made a clear, conscious design decision: he wanted the possible flows of control through the function body to be visible locally inside a function, without being dependent on the definitions of other methods (raising exceptions, of course, being an exception - catching them, though, largely obeys this guideline). Most code is run immediately, code in if statements and exception handlers is run zero or one times, code in loops is run zero or more times, code in nested function definitions is executed at some later time when the function is called. The Ruby blocks design is the antithesis of this: your control flow is entirely dependent on the methods you call. The downside of wanting visible control flow, of course, is that iteration, transactional code and callback programming all end up looking different at the point of invocation. (If you read PEP 340, Guido's original proposal for what eventually became the with statement, and contrast it with PEP 343, the version that we finally implemented, you'll see that his original idea was a fair bit closer to Ruby's blocks in power and scope).

So Ruby's flexibility comes at a price: when you pass a block to a method, you need to know what that method does in order to know how it affects your local control flow. Naming conventions can help reduce that complexity (such as the .each convention for iteration), but it does move control flow into the domain of programming conventions rather than the language definition.

On the other hand, Python's choice of explicit control flow comes at a price in flexibility: callback programming looks starkly different to ordinary programming as you have to construct explicit closures in order to pass chunks of code around.

Two way data flow

With their functional API, blocks natively supported two-way data flow from the beginning: data was passed in by calling them, and then either returned as the result of the block or by manipulating the passed in name bindings.

By contrast, Python's generators were originally output only, reflecting their target use case of iteration. You could input some initial data via parameters, but couldn't readily supply data to a running calculation. This has started to change in recent years, as generators now provide send() and throw() methods to pass data back in, and yield became an expression in order to provide access to the 'send()' argument. However, these features do not, at this stage, have deep syntactic support - there's a fairly obvious mapping from continue to send() and break to throw() that would tie them into the for loop syntax, but this capability has not garnered significant support when it has been brought up (I believe because it doesn't really help with the last major code execution model that Python doesn't provide nice native support for: callback programming).

In Python 3.3, generators will gain the ability to return values, and better syntax for invoking them and getting that value, moving the language even further towards full coroutine support (see PEP 380 for details). However, that is merely the next step along the path rather than arrival at the destination.

Reinventing blocks

I think the folks who accuse us of (slowly) reinventing blocks have a valid point - Python really is on the path of devising ways to handle tasks neatly with coroutines (i.e. functions that can be suspended and transparently resumed later without losing any internal state) that Ruby handles via blocks (i.e. extracting arbitrary fragments of a function body and passing them to other code). The fact that generators were not built into Python from the outset but instead have been added later to make certain kinds of code easier to write does show through in a variety of ways - coroutine based code often doesn't play nicely with ordinary imperative code and vice-versa.

Ruby's way has a definite elegance to it (despite the hidden control flow). I think aspiring to that kind of elegance for callback programming in Python would be a good thing, even if the semantic model is completely different (i.e. coroutine based rather than block based). The addition of actual block functionality remains unlikely, however - if they were as powerful as Ruby blocks, then it would create two ways to do too many things (with no obvious criteria to choose between the current technique and the block based technique), but if they were strictly less powerful, then reusing Ruby's block terminology would likely be confusing rather than enlightening. For better or for worse, Python is now well down the path of coroutine based programming and we likely need to see how far we can take that model rather than trying to shoehorn in yet another approach.

Spinning up the pulpdist project

2011-09-28T02:23:00.000+10:00

One novel aspect of the pulpdist project is that it is starting with an almost completely blank slate from a technology point of view (aside from the decision to use Pulp as the main component of the mirroring network). Red Hat does have development standards for internal projects, of course (especially in the messaging space), but they're fairly flexible, leaving the individual tool development teams with a lot of options. If something ships with Fedora and/or RHEL, or is available under licensing terms that would be acceptable for inclusion in Fedora (and subsequently RHEL), then it's fair game.

This post focuses on the design of the management server. I'll write up a separate post looking at the currently planned design for the Pulp data transfer plugins.

Source Control

Unsurprisingly, Red Hat's internal processes are heavily influenced by Linux kernel processes. Accordingly, the source control tool of choice for new projects is Git. While I have a slight preference for Mercurial (due mainly to familiarity), I'm happy enough with any DVCS, so Git it is.

Primary Development Language

Python, of course. You don't hire a CPython core developer to get them to work on a Ruby or Perl project (although the current system I'm replacing was written in Perl). As a web application, there will naturally be some Javascript and CSS involved as well.

Web Framework

The main management application for pulpdist is going to be a full-scale web application. User profiles and authentication, database storage, communication with other web services, provision of a REST API, integration with the engineering tools messaging bus. Basically, micro-frameworks need not apply.

While I expect Pyramid/Pylons would also have been able to do the job, I decided to go with Django 1.3. This was heavily influenced by social factors: I know a lot of Django devs that I can bug for advice, but the same is not true for Pyramid. The complexity of the whole Pyramid/Pylons/TurboGears setup is also not appealing - while veteran web developers may find the "you decide" approach a selling point, Django's batteries included approach makes it far simpler to get started quickly, and decide as I go along which pieces I should keep, discard or replace.

I've heard some experienced Django developers muttering complaints about the class based views design in 1.3, but as someone coming in that is an experienced Python developer, but a relatively noobish web developer, the CBV approach seems eminently sensible, while the old function based approach looks repetitive and insane. Object oriented programming was invented for a reason!

I'll admit that my perception may be biased by knowing exactly how to make multiple inheritance work the way I want it to, though :)

Web Server

The management server doesn't actually have that much work to do, so the basic Apache+mod_wsgi configuration will serve as an adequate starting point (any heavy lifting will be done by the individual Pulp instances, and the main data traffic on those doesn't run through their web service). WSGI provides the flexibility to revisit this later if needed.

I've also punted on any web caching questions for now - the management server is low traffic and once the access to the Pulp sites is pushed out to a backend service, it should be fast enough at least for the early iterations.

Authentication & Authorisation

The actual user authentication task will be handed off to Apache and all management application access will be restricted to Kerberos authenticated users over SSL. Django's own permissions systems will be used to handle authorisation restrictions. (The experimental prototype will use Basic Auth instead, since it is the Apache/Django integration the prototype needs to cover, not the Apache configuration for SSL and Kerberos authentication)

Integration with Pulp's user access controls is via OAuth, but the design for configuration of user permissions in the Pulp servers is still TBD.

Database and ORM

Again, the management server isn't doing the heavy lifting in this application. The Pulp instances use MongoDB, but for the management server I currently plan to use the standard Django ORM backed by PostgreSQL. For the prototype instance, the database is actually just an SQLite3 file. I'm not quite sold on this one as yet - it's tempting to start playing with SQLAlchemy, since I've already had to hack around some of the limitations in the native ORM in order to store encrypted fields. OTOH, I already have a ton of things to do on this project, so messing with this is a long way down the priority list.

Schema and data maintenance is handled using South.

HTML Templating

The standard Django templating engine should be sufficient for my needs. As with the ORM, it's tempting to look into upgrading it to something like Jinja2, but once again 'good enough' is likely to be the deciding factor.

For data table display, I'm using Django Tables 2 and form display will use Django Uni-Form.

REST API

The REST API for the service is currently there primarily as a development aid - it lets me publish the full data model to the web as soon as it stabilises (and even while its still in flux), even if the UI for end users hasn't been fully defined. This is particularly useful for the metadata coming back from the Pulp server, since it doesn't need much post-processing to be included as raw data in the management server's own REST API. The JSON interface will also allow much of the backend processing to be fully exercised by the test suite without worrying about web UI details.

The design of the REST API was heavily influenced by this Lessons Learned piece from the RHEV-M developers. The Django Rest Framework means I can just define the data I want to display as a list or dictionary and the framework takes care of formatting it nicely, including rendering URLs as hyperlinks.

AMQP Messaging

I haven't actually started on this aspect in any significant way, but the two main contenders I've identified are python-qpid (which is what Pulp uses) and django-celery (which would also give me an internal task queue engine, which the management server is going to need - the prototype just does everything in the Django process, which is OK for experimentation on the LAN, but clearly inadequate long term when talking to multiple sites distributed around the planet). At this early stage, I expect the internal task management aspect is going to tip the decision in favour of the latter.

Testing Regime

As the foundation for the automated testing, I'm going with Django Sane Testing (mainly based on the example of other internal Django projects). Michael Foord's mock module lets me run at least some of the tests without relying on an external Pulp instance (fortunately, the namespace conflict with Fedora's RPM building utility 'mock' was recently resolved with the latter's support library being renamed to 'mockbuild').

Continuous integration is an open question at this point. Pulp uses Jenkins for CI and I'm inclined to follow their lead. The other main possibility is to use Beaker, Red Hat's internal test system originally set up for kernel testing (one key attraction Beaker offers is the ability to set up multi-server multi-site testing in a test recipe so I can run tests over the internal WAN).

Packaging

Tito is a tool for generating SRPMs and RPMs directly from a Git repository. For my own packages, this is the approach I'm using (with handcrafted spec files). For some strange reason, the sysadmins around here like it when internal devs provide things as pre-packaged RPMs for deployment :)

Packaging of upstream PyPI dependencies that aren't available as Fedora or RHEL packages is still a work in progress. I experimented with Tito and git submodules (which doesn't work) and git subtrees (which does work, but is seriously ugly). My next attempt is likely to be based on py2pack, so we'll see how that goes (I actually discovered that project by searching for 'cpanspec pypi' after hearing some of the Perl folks here extolling the virtues of cpanspec for easily packaging CPAN modules as RPMs).

I also need to switch to using virtualenv to get a clearer distinction between Fedora packages I added via yum install and stuff I picked up directly from PyPI with pip.

Mirror All The Things!

2011-09-09T12:54:00.001+10:00

After describing the project I'm working on to a few people at PyConAU and BrisPy, I decided it might be a good idea to blog about it here. I do have a bit of an ulterior motive in doing so, though - I hope people will point out when I've missed useful external resources or applications, or when something I'm planning to do doesn't make sense to the assorted Django developers I know. Yes, that's right - I'd like to make being wrong on the internet work in my favour :)

The project is purely internal at this stage, but I hope to be able to publish it as open source somewhere down the line. Even being able to post these design concepts is pretty huge for me personally, though - before starting with Red Hat a few months ago, I spent the previous 12 and a half years working in the defence industry, which is about as far from Red Hat's "Default to Open" philosophy as it's possible to get.

Mirror, Mirror, On The Wall

The project Red Hat hired me to implement is the next generation of their internal mirroring system, which is used for various tasks, such as getting built versions of RHEL out to the hardware compatibility testing labs (and, when they're large enough, returning the generated log files to the relevant development sites), or providing internal Fedora mirrors at the larger Red Hat offices (such as the one here in Brisbane).

There are various use cases and constraints that mean the mirroring system needs to operate at the filesystem level without making significant assumptions about the contents of the trees being mirrored (due to various details of the use cases involved, block level replication and approaches that rely on the transferred data being laid out in specific ways aren't viable alternatives for this project). The current incarnation of this system relies almost entirely on that venerable workhorse of the mirroring world, rsync.

However, the current system is also showing its age and has a few limitations that make it fairly awkward to work with. Notably, there's no one place to go to get an overview of the entire internal mirroring setup, and the direct use of rsync means it isn't particularly friendly with other applications when it comes to sharing WAN bandwidth and the servers involved are wasting quite a few cycles recalculating the same deltas for multiple clients. Hence, the project I am working on, which is intended to replace the existing system with something a bit more efficient and easier to manage, while also providing a better platform for adding new features.

Enter Pulp

Pulp is an open source (Python) project created by Red Hat to make it easier to manage private yum repositories. Via Katello, Pulp is one of the upstream components for Red Hat's CloudForms product.

The Pulp project is currently in the process of migrating from their original yum-specific architecture to a more general purpose Generic Content plugin architecture. It's that planned plugin architecture that makes Pulp a useful basis for the next generation internal mirroring system, which, at least for now, I am imaginatively calling pulpdist (referring to both "distribution with Pulp", since that's what the system does, and "distributed Pulp instances", since that's how the system will work).

The main components of the initial pulpdist architecture will be:

a front-end (Django 1.3) web app providing centralised management of the entire distribution network
custom importer and distributor plugins for Pulp to handle distribution of tree changes within the distribution network
custom importer plugins to handle the import of trees from their original sources and generation of any additional metadata needed by the internal distribution plugins
generic (and custom, if needed) plugins to make the trees available to the applications that need them

I'll be writing more on various details that I consider interesting as I go along. Initially, that will include my plan for the mirroring protocol to be used between the sites, as well as various decisions that need to be made when spinning up a Django project from scratch (while many of my specific answers are shaped by the target environment for internal deployment, the questions I needed to consider should be fairly widely applicable).

Open Source, Windows and Teaching Python to New Developers

2011-08-27T22:16:00.000+10:00

A few questions and incidents recently prompted me to reflect on why I don't help with CPython support on Windows, even though I use Windows happily enough on my gaming system. Since this ended up being a rather pro-Linux article and upfront disclosure is a good thing, I'll note that while I do work for Red Hat now, that's a very recent thing - my adoption of Linux as my preferred development platform dates back to 2004 or so. I work for Red Hat because I like Linux, not the other way around :)

The Availability of Professional Development Tools

I don't make a secret of my dislike of Windows as a hobbyist development platform. While Microsoft have improved things in recent years (primarily by releasing the Express editions of Visual Studio), there's still a huge difference between an operating system like GNU/Linux, which was built by developers for developers based on a foundation that was built by academics for academics, and Windows, which was built by a company that used deals with computer manufacturers to get it into end users' hands regardless of technical merit. Developers were forced to follow in order to reach that large installed user base. Those different histories are reflected in the different development cultures that surround the respective platforms.

To get the same tool chain that professional Linux companies use, you don't need to do anything special - Linux distributions include the tools used to create them. If you have a distribution, you have everything you need to build applications for that distribution, including documentation. With the open source nature of the platform and almost all of the software (the occasionally binary driver notwithstanding), there's a vast range of tools out there to help you get things done (although sorting through the mass can be a little tricky sometimes, since it can be hard to tell the difference between stuff that doesn't exist and stuff that exists, but hasn't been uncovered by your research).

As far as I'm aware, Mac OS X isn't quite as generous with freely available development utilities, but isn't all that far off the Linux approach (I'm not a Mac user or developer though, so there may be more hurdles than I am aware of - I recall some muttering about Apple beginning to charge a small fee for XCode. My opinion is based mostly on the fact that it seems pretty easy to find open source devs that use Macs). With the POSIX-ish underpinnings, many of the utilities from the *nix world also work in this environment.

The minimum realistic standard for professional Windows development, though, is an MSDN subscription (to get full access to the OS documentation and various utilities), along with a professional copy of Visual Studio. The tools available for free (including the Express editions of Visual Studio) are clearly second rate. Even when the tools themselves are OK, the licensing restrictions on the applications they create may make them practically useless (and MS have the gall to call the GPL viral - at least the gcc team don't restrict how you license and distribute the binaries it creates). So why should a hobbyist develop for a system that thinks they should pay substantial sums for the privilege of developing for it, instead of one that welcomes all contributors, providing not only the end product, but the ingredients and recipes all for free?

At the recent PyConAU sprints, one of the contributors (an existing Linux user that happened to have a Windows only laptop with them) became frustrated with getting all the necessary tools set up to work properly on Windows (configuring git+ssh for read/write access to a GitHub repo was one key point of irritation), and decided to dual boot Ubuntu on the machine instead. Twenty minutes later, she was up and running and hacking on the project she originally wanted to hack on. Granted, she already knew how to use Linux, but seriously, there's something fundamentally wrong with a platform when installing and dual-booting to a different OS is the easiest way to get a decent development environment up and running.

All that ends up putting cross-platform languages like Python in an interesting position: when developing with Python, you can often get away with not understanding the underlying details of your operating system, because the language runtime tries to provide a largely standardised interface on all platforms. However, many open source developers either don't use Windows at all, or genuinely dislike programming for it, so the burden of making things work properly on Windows falls on the shoulders of a comparatively small number of people, either those who genuinely like programming for the platform (yes, such people exist, I'm just not one of them), or those that are looking for any niche where they can usefully contribute and are happy enough to take on the task of improving Windows compatibility and support.

I don't have particularly hard numbers to back this up (other than the skew in core developer numbers vs overall OS popularity), but my intuition is that, at least for CPython, the user:core developer ratio is orders of magnitude higher for Windows than it is for Linux or Mac OS X.

The Implications for Teaching Python on Windows

Something cool that is going on at the moment is that a lot of folks are interested in the idea of teaching more people how to program with Python as the language used. However, the potential students (young and old) that they are wanting to teach often don't have any development experience at all and are using the most common consumer operating system (i.e. Windows). So good Windows support, and an easy installation experience are important considerations for these instructors. A request that is frequently made (with varying levels of respect and politeness), is that the official python.org Windows installer be updated to automatically adjust the PATH (or at least provide the option to do so), so that Python can be launched from the command line by typing "python" instead of something like "C:\Python27\python".

If educators want that right now their best bet is actually to direct their students towards the Windows versions of ActiveState's ActivePython Community Edition. ActiveState add a few things to the standard installer, like PATH manipulation and additional packages (such as pywin32). They also bundle PyPM, which is a decent tool for getting PyPI packages on to Windows machines (at least, I've heard good things about it - I haven't actually used it myself). (That said, I believe I may need to caveat that recommendation a bit: as near as I can tell from their website, PyPM has been deliberately disabled for their 64-bit Windows Community Edition installer. Still, even in that case, you can easily grab additional packages direct from PyPI via "pip install" on the command line)

Brian Curtin is working on adding optional PATH manipulation to the python.org installer for 3.3, and there's a chance such a change might be backported to the next maintenance releases for 3.2 and 2.7 (no promises, though). Even if it does make it in, it will still be a while before the change is part of a binary release (especially given that Brian has only just started tinkering with it).

This is clearly a nice thing for beginners, especially those that aren't in the habit of tinkering with their OS settings, but I do honestly wonder how much of a difference it will make in the long run. In many ways, software development is one long exercise in frustration. You decide you want to fix bug X. But it turns out bug X is really due to bug Y. You could work around Y just to fix X, but the bigger bug would still be there. But then you discover that fixing bug Y properly requires feature Z, which doesn't exist yet, so a workaround (even an ugly one) starts to sound pretty attractive. "Yak shaving" (the highly technical term for things that you're working on solely because they're a prerequisite for what you actually want to be working on) is so common it's almost the norm rather than the exception. The many and varied frustrations of trying to use Windows as a hobbyist open source developer also won't magically go away just because the python.org installer starts automating one environment variable update - as soon as people are introduced to sites like GitHub and BitBucket, they'll get to discover the joy that is SSH and source control on Windows. If they get past that hurdle, they'll likely start to encounter the multitude open source projects that don't even offer Windows installers (if the project supports Windows at all), because their Windows developer count stands at a grand total of zero.

Final Thoughts

I hope the people teaching Python to beginners on Windows and the folks working on improving Windows support don't take this article as an attack on their efforts. I find both goals to be quite admirable, and wish those involved all the success they can find. But there are reasons I abandoned Windows as a personal development platform ~7 years ago and taught myself to use Linux instead. As far as I can tell, most of those reasons remain valid today, even after Microsoft started releasing the Express versions of Visual Studio in an attempt to stem the flood of hobbyist developers jumping ship.

The other day I called the relative lack of Windows developers in open source a vicious cycle and I stand by that. If someone can learn to program, mastering Linux is going to be comparatively easy. For anyone seriously interested in open source development, using Linux (even in a virtual machine, the way I do on my gaming laptop) is by far the path of least resistance. Getting more Windows developers in open source requires that people care sufficiently about Windows as a platform that they don't just switch to Linux, but care about open source enough to start contributing at all, and that seems to be a genuinely rare combination.

Scripting languages and suitable complexity

2011-08-24T22:37:00.000+10:00

Steven Lott is a Python developer and blogger that I first came across via his prolific contributions to answering questions on Stack Overflow, and then later by reading his blog posts that appeared on Planet Python. His Building Skills in Python book is a resource I've recently started suggesting newcomers to Python take a look at to see if his style works for them.

Some time ago, he posted about what he called the "Curse of Procedural Design": the fact that, beyond a certain point, purely procedural code typically starts drowning in complexity and becomes an unmaintainable mess. Based on that, he then started questioning whether or not he was doing the right thing by teaching the procedural aspects of Python first and leaving the introduction of object oriented concepts until later in the book.

Personally, I think starting with procedural programming is still the right thing to do. However, one of the key points of object-oriented design is that purely procedural programming doesn't scale well. Even in a purely procedural language, large programs almost always define essential data structures, and functions that work on those data structures, effectively writing object oriented code by convention. Object support built into the language makes this easier but it isn't essential (as large C projects like the Linux kernel and CPython itself demonstrate).

Where languages like Java can be an issue as beginner languages is that by requiring that all code be compiled in advance and written in an object oriented style, they set a minimum level of complexity for all programs written in that language. Understanding even the most trivial program in Java requires that you grasp the concepts of a module, a class, an instance, a method and an expression. Using a compiled procedural language instead at least lets you simplify that a bit, as you only need to understand modules, functions and expressions.

But the scripting languages? By means of a read-eval-print loop in an interactive interpreter, they let you do things in the right order, starting with the minimum level of complexity possible: a single expression.

For convenience, you may then introduce the concept of a 'script' early, but with an appropriate editor, scripts may be run directly in the application (giving an experience very similar to a REPL prompt) rather than worrying about command line invocation.

More sophisticated algorithms can then be introduced by discussing conditional execution and repetition (if statements and loops), but still without any need to make a distinction between "definition time" and "execution time".

Then, once the concept of algorithms has been covered, we can start to modularise blocks of execution as functions and introduce the idea that algorithms can be stored for use in multiple places so that "definition time" and "execution time" may be separated.

Then we start to modularise data and the associated operations on that data as classes, and explore the ways that instances allow the same operations to readily be performed on different data sets.

Then we start to modularise collections of classes (and potentially data and standalone functions) as separate modules (and, in the case of Python, this can be a good time to introduce the idea of "compilation time" as separate from both "definition time" and "execution time").

Continuing up the complexity scale, modules may then be bundled into packages, and packages into frameworks and applications (introducing "build time" and "installation time" as two new potentially important phases in program execution).

A key part of the art of software design is learning how to choose an appropriate level of complexity for the problem at hand - when a problem calls for a simple script, throwing an entire custom application at it would be overkill. On the other hand, trying to write complex applications using only scripts and no higher level constructs will typically lead to an unmaintainable mess.

In my opinion, the primary reason that scripting languages are easier to learn for many people is that they permit you to start immediately with code that "does things", allowing the introduction of the "function" and "class" abstractions to be deferred until later.

Starting with C and Java, on the other hand, always requires instructors to say "Oh, don't worry about that boilerplate, you'll learn what it means later" before starting in with the explanation of what can go inside a main() function or method. The "compilation time" vs "execution time" distinction also has to be introduced immediately, rather than being deferred until some later point in the material. There's also the fact that such languages are actually usually at least two languages in one: the top level "compile time" language that you use to define your data structures, functions and modules and the "run time" language that you actually use inside functions and methods to get work done. Scripting languages don't generally have that distinction - the top level language is the same as the language used inside functions (in fact, that's my main criteria for whether or not I consider a language to be a scripting language in the first place).

Of Python and Road Maps (or the lack thereof)

2011-08-24T15:16:00.006+10:00

I gave my first ever full conference talk at PyCon AU this year, on the topic of how Python evolves over time.

One notable absence from that talk is a discussion of any kind of specific 'road map' for the language as a whole. There's a reason for that: no such document exists. Guido van Rossum (as the language creator and Benevolent Dictator for Life) is the only person who could write an authoritative one and he's generally been happy to sit back and let things evolve organically since guiding the initial phases of the Py3k transition (which is now ticking along quite nicely and the outlook for 2.x being relegated to 'supported-but-now-legacy' status within the 2013/14 time frame is promising). PEP 398, the release PEP for Python 3.3 comes close to qualifying, but is really just a list of PEPs and features, with no context for why those ideas are even on the list.

However, despite that, it seems worthwhile to make the effort to try to write a descriptive road map for where I personally see current efforts going, and where I think the Python core has a useful role to play. Looking at the Python Enhancement Proposal index for the lists of Accepted and Open PEPs can be informative, but it requires an experienced eye to know which proposals are currently being actively championed and have a decent chance of being accepted and included in a CPython release.

When reading the following, please keep in mind that the overall Python community is much bigger than the core development team, so there are plenty of other things going on out there. Others may also have differing opinions on which changes are most likely to come to fruition within the next few years.

Always in Motion is the Future

Unicode is coming. Are you ready?

The Python 2.x series lets many developers sit comfortably in a pure ASCII world until the day when they have to debug a bizarre error caused by their datastream being polluted with non-ASCII data.

That is only going to be a comfortable place to be for so long. More and more components of the computing environment are migrating from ASCII to Unicode, including filesystems, some network protocols, user interface elements, other languages and various other operating system interfaces. With the modern ease of international travel and the rise of the internet, even comfortable "we only speak ASCII in this office, dammit!" environments are eventually going to have to face employees and customers that want to use their own names in their own native, non-Latin character sets rather than being constrained by a protocol invented specifically to handle English.

The Python 3.x series was created largely in response to that environmental transition - in many parts of the world where English isn't the primary spoken language, needing to deal with text in multiple disparate character encodings is already the norm, and having to deal regularly with non-ASCII data is only going to become more common even in English dominant areas.

However, supporting Unicode correctly on this scale is something of a learning experience for the core development team as well. Several parts of the standard library (most notably the email package and wsgiref module) really only adjusted to the new Unicode order of things in the recent 3.2 release, and (in my opinion) we've already made at least one design mistake in leaving various methods on bytes and bytearray objects that assume the underlying data is text encoded as ASCII.

The conversion to Unicode everywhere also came at a measurable cost in reduced speed and increased memory usage within the CPython interpreter. PEP 393 ("Flexible String Representation") is an attempt to recover some of that ground by using a smarter internal representation for string objects.

Easier and more reliable distribution of Python code

Tools to package and distribute Python code have been around in various stages of usability since before the creation of distutils. Currently, there is a concerted effort by the developers of tools like pip, virtualenv, distutils2, distribute and the Python Package index to make it easier to create and distribute packages, to provide better metadata about packages (without requiring execution of arbitrary Python code), to install packages published by others, and to play nicely with distribution level package management techniques.

The new 'packaging' module that will arrive in 3.3 (and will be backported to 3.2 and 2.x under the 'distutils2' name) is a visible sign of that, as are several Python Enhancement Proposals related to packaging metadata standards. These days, the CPython interpreter also supports direct execution of code located in a variety of ways.

There's a significant social and educational component to this effort in addition to the technical underpinnings, so this is going to take some time to resolve itself. However, the eventual end goal of a Python packaging ecosystem that integrates nicely with the various operating system software distribution mechanisms without excessive duplication of effort by developers and software packagers is a worthy one.

Enhanced support for various concurrency mechanisms

The addition of the 'concurrent' package, and its sole member, 'concurrent.futures' to the Python 3.2 standard library can be seen as a statement of intent to provide improved support for a variety of concurrency models within the standard library.

PEP 3153 ("Asynchronous IO support") is an early draft of an API standardisation PEP that will hopefully provide a basis for interoperability between components of the various Python asynchronous frameworks. It's a little hard to follow for non-Twisted developers at this stage, since there's no specific proposed stdlib API, nor any examples showing how it would allow things like plugging a Twisted protocol or transport into the gevent event loop, or vice-versa.

While developers are still ultimately responsible for understanding the interactions between threading and multiprocessing on their platform when using those modules, there are things CPython could be doing to make it easier on them. Jesse Noller, the current maintainer of the multiprocessing package, is looking for someone to take over the responsibility for coordinating the related development efforts (between cross-platform issues and the vagaries of os.fork interactions with threading, this is a non-trivial task, even with the expertise already available amongst the existing core developers).

PEP 3143 ("Standard daemon process library") may be given another look for 3.3, although it isn't clear if sufficient benefit would be gained from having that module in the stdlib rather than published via PyPI as it is currently.

Providing a better native Python experience on Windows

Windows is designed and built on the basis of a mindset that expects most end users to be good little consumers that buy software from developers rather than writing it themselves. Anyone writing software is thus assumed to be doing so for a living, and hence paying for professional development tools. Getting a decent development environment set up for free can be so tedious and painful that many developers (like me!) would rather just install and use Linux for development on our own time, even if we're happy enough to use Windows as a consumer (again, like me - mostly for games in my case). The non-existent "BUNDLE ALL THE THINGS!" approach to dependency management and the lack of any kind of consistent software updating interface just makes this worse, as does the fact that any platform API handling almost always has to special-case Windows, since their libc implementation is one of the more practically useless things on the face of this Earth.

This creates a vicious cycle in open source, where most of the active developers are using either Linux or Mac OS X, so their tools and development processes are focused on those platforms. They're also typically very command line oriented, since the diverse range of graphical environments and the widespread usage of remote terminal sessions makes command line utilities the most reliable and portable. Quite a few new developers coming into the fold run into the Windows roadblocks and either give up or switch to Linux for development (e.g. by grabbing VirtualBox and running a preconfigured Linux VM to avoid any hardware driver issues), thus perpetuating the cycle.

Personally, I blame the cross-platform hobbyist developer hostile nature of the platform on the Windows ecosystem itself and generally refuse to write code on it unless someone's paying me to do so. Fortunately for single-platform Windows users and developers, though, there are other open source devs that don't feel the same way. Some of them are working on tools (such as the script launcher described in PEP 397) that will make it easier to get usable development environments set up for new Python users.

Miscellaneous API improvements in the standard library

There are some areas of the standard library that are notoriously clumsy to use. HTTP/HTTPS comes to mind, as does date/time manipulation and filesystem access.

It's hard to find modules that strike the sweet spot of being well tested, well documented and proven in real world usage, with mature, stable APIs that aren't overly baroque and complicated (yes, yes, I realise the irony of that criticism given the APIs I'm talking about replacing or supplementing). It's even harder to find such modules in the hands of maintainers that are willing to consider supporting them as part of Python's standard library (potentially with backports to older versions of Python) rather than as standalone modules.

PEP 3151 ("Reworking the OS and IO exception hierarchy") is an effort to make filesystem and OS related exceptions easier to generate and handle correctly that is almost certain to appear in Python 3.3 in one form or another.

In the HTTP/HTTPS space, I have some hopes regarding Kenneth Reitz's 'requests' module, but it's going to take more real world experience with it and stabilisation of the API before that idea can be given any serious consideration.

The regex module (available on PyPI) is also often kicked around as a possible candidate for addition. Given the benefits it offers over and above the existing re module, it wouldn't surprise me to see a genuine push for its inclusion by default in 3.3 or some later version.

A basic stats library in the stdlib (to help keep people from implementing their own variants, usually badly) is also a possibility, but again, depends on a suitable candidate being identified and put forward by people willing to maintain it.

The RFC treadmill

Many parts of the Python standard library implement various RFCs and other standards. While our release cycle isn't fast enough to track the newer, still evolving protocols, it's good to provide native support for the latest versions of mature, widely supported ones.

For example, support for sendmg(), recvmsg() and recvmsg_into() methods will be available on socket objects in 3.3 (at least on POSIX-compatible platforms). Additions like PEP 3144 ("IP Address Manipulation Library for the Python Standard Library") are also a potential possibility (although that particular example currently appears to be lacking a champion).

Defining Python-specific interoperability standards

Courtesy of the PEP process, python-dev often plays a role in defining and documenting interfaces that allow various Python frameworks to play nicely together, even if third party libraries are needed to make full use of them. WSGI, the Web Server Gateway Interface, is probably the most well-known example of this (the current version of that standard is documented in PEP 3333). Other examples include the database API, cryptographic algorithm APIs, and, of course, the assorted standards relating to packaging and distribution.

PEP 3333 was a minimalist update to WSGI to make it Python 3 compatible, so, as of 3.2, it's feasible for web frameworks to start considering Python 3 releases (previously, such releases would have been rather challenging, since it was unclear how they should talk to the underlying web server). It likely isn't the final word on the topic though, as the web-sig folks still kick around ideas like PEP 444 and WSGI Lite WSGI Lite. Whether anything actually happens on that front or if we keep chugging along with an attitude of "eh, WSGI gets the job done, quit messing with it" is an open question (and far from my area of expertise).

One area that is being actively worked on, and will hopefully improve significantly in Python 3.3, is the low level buffer interface protocol defined by PEP 3118. It turned out that there were a few flaws in the way that PEP was implemented in CPython, so even though it did achieve the basic goal of allowing projects like NumPy and PIL to interoperate without needing to copy large data buffers around, there are still some rather rough edges in the way the protocol is exposed to Python code via memoryview objects, as well as in the respective obligations of code that produces and consumes these buffers. More details can be found on the issue tracker if you're interested in the gory details of defining conventions for reliable management of shared dynamically allocated memory in C and C++ code :)

Enhancing and refining the import system

The importer protocol defined by PEP 302 was an ambitious attempt to decouple Python's import semantics from the underlying filesystem. It never fully succeeded - there's still a lot of Python code (including one or two components in the standard library!) that assumes the classical directories-on-a-disk layout for Python packages. That package layout, with the requirement for __init__.py files and restriction to a single directory for each package, is itself surprising and unintuitive for many Python novices.

There are a few suggestions in the works ultimately aimed not only at cleaning up how all this is implemented, but also at further decoupling the module hierarchy from the on-disk file layout. There are significant challenges in doing this in a way that makes life easier for developers writing new packages, while also allowing those writing tools that manipulate packages to more easily do the right thing, but it's an area that can definitely do with some rationalisation.

Improved tools for factoring out and reusing code

Python 3.3 will at least bring with it PEP 380's "yield from" expression which makes it easy to take part of a generator and split it out into a subgenerator without affecting the overall semantics of the original generator (doing this correctly in the general case is currently significantly harder than you might think - see the PEP for details).

I suspect that the next few years may bring some more tweaks to generators, generator expressions and context managers to make for loops and with statements even more powerful utilities for factoring out code. However, any adjustments in this area will be carefully balanced against the need for language stability and keeping things reasonably easy to learn.

Not Even on the Radar

In addition to the various suggestions mentioned in PEP 3099 ("Things that Will Not Change in Python 3000"), there are a couple of specific items I think are worth calling out as explicitly not being part of any realistic road map:

Major internal architectural changes for CPython

Questions like 'Why not remove the GIL?', 'Why not switch to a register based VM?', 'Why not add a JIT to CPython?' and 'Why not make PyPy the reference interpreter?' don't really have straightforward obvious answers other than "that's harder than you might think and probably less beneficial than you might hope", so I expect people to continue asking them indefinitely. However, I also don't see any significant changes coming on any of these fronts any time soon.

One of the key advantages of the CPython interpreter as a reference implementation is that it is, fundamentally, quite a simple beast (despite a few highly sophisticated corners). If we can do things, chances are that the other implementations are also going to be able to support them. By contrast, the translation stage in their toolchain means that PyPy can contemplate features like the existing JIT, or their current exploration of Software Transactional Memory for removing their Global Interpreter Lock in a useful way that simply aren't feasible with less sophisticated tools (or a much bigger development team).

Personally, I think the status quo in this space is in a pretty good place, with python-dev and CPython handling the evolution of the language specification itself, as well as providing an implementation that will work reasonably well on almost any platform with a C compiler (and preferably some level of POSIX compliance), while the PyPy crew focus on providing a fast, customisable implementation for the major platforms without getting distracted by arguments about possible new language features.

More feature backports to the 2.x series

Aside from the serious backwards compatibility problems accompanying the Unicode transition, the Py3k transition was also strongly influenced by the concept of paying down technical debt. Having legacy cruft lying around made it harder to introduce language improvements, since the additional interactions created more complexity to deal with, to the point where people just didn't want to bother any more.

By clearing out a bunch of that legacy baggage, we created a better platform for new improvements, like more pervasive (and efficient!) Unicode support, even richer metaclass capabilities, exception chaining, more intuitive division semantics and so on.

The cost, borne mostly by existing Python developers, is a long, slow transition over several years as people deal with the process of not only checking whether the automated tools can correctly handle their own code, but also waiting for all of their dependencies to be available on the updated platform. This actually seems to be going fairly well so far, even though people can be quite vocal in expressing their impatience with the current rate of progress.

Now, all of the major Python implementations are open source, so it's certainly possible for motivated developers to fork one of those implementations and start backporting features of interest from 3.x that aren't already available as downloads from PyPI (e.g. it's easy to download unittest2 to get access to 3.x unittest enhancements, so there's no reason to fork just for that kind of backport). However, anyone doing so will be taking up the task in full knowledge of the fact that the existing CPython development team found that process annoying and tedious enough that we got tired of doing it after two releases (2.6 and 2.7).

So, while I definitely expect a certain level of ongoing griping about this point, but I don't expect it to rise to the level of anyone doing the work to backport things like function annotations or exception chaining to the 2.x series.

Update 1: Added missing section about RFC support
Update 2: Fix sentence about role of technical debt in Py3k transition
Update 3: Added a link to a relevant post from Guido regarding the challenges of maintaining official road maps vs one-off "where are we now?" snapshots in time (like this post)
Update 4: Added missing section on defining Python-specific standards

Sharing vs broadcasting

2011-07-17T03:04:00.000+10:00

Even since I started playing with Google+ and its Circle mechanic, I've been trying to figure out why I don't like it. I thought it sounded great when I heard about it, but as soon as I started using it... meh :P

I still haven't quite figured it out (although I mostly think it's the neither-chicken-nor-fowl aspect, along with it adding back ~500 useless "suggestions" from random Python mailing list contacts that I had already purged once), but it's certainly helped me see the way I communicate online in a different light.

For a long time, my only real online presence was regular postings to Python mailing lists, commenting on various sites and the very occasional post here on B&L.

After enough of my friends joined, Facebook was added to the mix, but I do everything I can to keep that locked down so only acquaintances can see most things I post there.

After going to PyconAU last year, and in the leadup to PyCon US earlier this year, I started up my @ncoghlan_dev account on Twitter, got the "python" category tag on here added to Planet Python, and started to develop a bit more of an online public presence.

Here on the blog, I can, and do, tag each post according to areas of interest. As far as I know, the 'python' tag is the only one with any significant readership (due to the aggregation via Planet Python), but people could subscribe to the philosophy or metablogging tags if they really wanted to.

When it comes to sharing information about myself, there's really only a few levels based on how much I trust the people involved: Friends&Family, Acquaintances, General Public pretty much covers it. Currently I handle that via a locked down FB for Friends, Family & Acquaintances (with a "Limited Access" list for people that I exclude from some things, like tagged photos) and completely public material on Twitter and Blogger.

The public stuff is mostly Python related, since that's my main online presence, but I'm fairly open about political and philosophical matters as well. FB, by contrast, rarely sees any mention of Python at all (and I'm often a little more restrained on the political and philosophical front).

Where I think Circles goes wrong is that it conflates Access Control with Topic Tagging. When I publish Python stuff, I'm quite happy for it to be public. However, I'd also be happy to tag it with "python", just as I do here on the blog, to make it easier for my friends to decide which of my updates they want to see.

This is classic Publish/Subscribe architecture thinking. When Publishing, I normally want to be able to decide who *can* access stuff. That is limited by closeness, but typically unrelated to the topic. Having, tagging as a service to my subscribers, I am quite happy to do. When Subscribing, I want to be able to filter by topic.

If I publish something to, say, my Pythonistas circle, than that does more than I want. Yes, it publishes it to them, but it also locks out everybody else. The ways I know people and the degree to which I trust them do not align well with the topics I like to talk about. I've already seen quite a few cases where the reshared versions of public statuses I have received have been limited access.

The more I think about it, the more I think I'm going to stick with my current policy of using it as a public micro-blog and pretty much ignore the fact that the Circles page exists.

Effective communication, brain hacking and diversity

2011-07-10T17:10:00.000+10:00

Jesse Noller recently posted an interesting quote on Twitter:

"One who feels hurt while listening to harsh language may lose his mindfulness and not hear what the other person is really saying."

When you think about it, human language is a truly awe inspiring tool. By the simple act of creating certain vibrations in the air, marks on a page or electromagnetic patterns in a storage system, we're able to project our thoughts and feelings across space and time, using them to shape the thoughts and feelings of others.

While this ability to communicate is so thoroughly natural to most of us as humans that we typically take it for granted, it is actually an amazing world shaping capability deserving of our respect and attention.

And once we start giving it the attention it deserves, then we realise that we can judge the effectiveness of our own communication by looking at the communication that is subsequently reflected back at us. How well do those reflections mirror the thoughts and feelings that our own words were intended to create? It's the linguistic equivalent of running our code and seeing if it does what we wanted it to do.

All communication is a form of brain hacking, even when the only target is ourselves. We write lists of goals - formulating for our own benefit concrete plans of action that we can then tackle one step at a time. We write polemics, trying to engender in others some dim sense of the joy or outrage we feel with respect to certain topics. Sometimes we succeed, sometimes we fail. Sometimes we assume certain shared beliefs and understanding, so the point completely fails to come across to those without that common background.

For those closest to us, those with the most shared history, we have a rich tapestry of common knowledge to draw from. Movies we've all seen, books we've all read, events we all attended, discussions we were all part of - outsiders attempting to follow a transcript of our conversations would likely soon be utterly lost due to the shared subtext that isn't explicitly articulated (and the same is true for any group of close friends).

As groups get larger, the amount of truly common knowledge decreases, but there's still plenty of unwritten subtext that backs up whatever is explicitly articulated. In a certain sense, that unwritten subtext can be seen as the very definition of culture - it's the things you don't have to say because they're assumed. My past musings on the culture of python-dev are an example of this.

And that brings us to the point of considering the opening quote, diversity and questions of common courtesy. When speaking to friends, I can share truly awful jokes without offence because of the shared background information as to what is and isn't acceptable (and what things should be taken seriously). As the group being addressed gets larger, then the valid assumptions I can make about shared views of the world become fewer and fewer, so I have to start explicitly articulating things I would otherwise assume, and simply not say some things because I know (or at least strongly suspect) that they won't come across correctly to the audience I'm attempting to reach. Sometimes even addressing similar groups of people in a different context can change the assumptions as to what is a reasonable way to phrase things.

If a member of my target audience gets hung up on my wording or my choice of examples to the point where they miss the underlying message, then to a large degree, the responsibility lies with me as the originator of the communication. Now, I'm not a saint and make no pretence of being one. The rich fields of metaphors in English include many relating to subjects that are truly quite horrific or otherwise offensive to various groups of people. Sometimes I'm going to use that kind of phrasing without thinking about it, especially when talking rather than writing (my own innate tact filter is definitely set up to filter incoming communication, so applying tact in the outwards direction is a conscious process rather than something I do automatically). If such a miscommunication happens and someone points it out, then the onus is on me to admit that yes, my choice of words was poor and obscured my meaning rather than illuminating it. That's life, I make mistakes, and hopefully we can move on.

It's not entirely a one way street, though. Just as we apply contextual analysis to our understanding of historical writings, so it can be useful to apply the same approach to things that are said by current figures. Richard Dawkins recently made some ill-advised comments in relation to Skepchick's advice to men to avoid certain actions that make them look creepy (that's all she said, "Don't do this, it's creepy" and she copped flack for it, as if she'd said people doing it should be sent to prison or castrated or something equally extreme). Does the fact that Dawkins clearly didn't get why he was in the wrong make him a horrible human being or devalue his extensive contributions to our understanding of evolutionary biology*? No, it doesn't, any more than Isaac Newton's obsession with alchemy devalued his contributions to physics and mathematics. It just makes him a product of the culture that raised him. Hopefully he'll eventually realise this and publicly apologise for failing to give the matter due consideration before weighing in.

However, what really surprised me is the number of people that indicate they're shocked by his words, or questioning their support for his other activities just because he so vividly demonstrated his cluelessness on this particular topic. The world is a complicated place and the social dynamics of privilege, cultural blindspots and effectively encouraging diversity aren't one of the easiest pieces to comprehend. Hell, as a middle-class, 30-something, white, English-speaking, straight, cisgendered male living in Australia I'm quite certain that my own grasp of the topic is heavily coloured by the fact that on pretty much any of the typical grounds for discrimination I'm in the favoured majority (being an atheist is arguably the only exception, but that's far less of a problem here in Australia than it is in the US. Our Prime Minister is an acknowledged atheist and even the Murdoch media machine didn't really try to make much of an issue out of that before the last election). I do my best to understand the topic of diversity based on the experiences of those that actually have to deal with it on a daily basis, but it's still a far cry from seeing things first hand.

So, since I don't believe I can speak credibly to the topic of diversity directly, I instead prefer to encourage people to reflect on the value and nature of communication and community in general. Martin Fowler wrote an excellent piece about the challenge of creating communities that are welcoming to a diverse audience without rendering them bland and humourless (as the benign violations of expectations and assumptions that are at the heart of most humour often depend on the shared context that welcoming communities can't necessarily assume). This excellent video highlights the importance of focusing on actions (e.g. "This thing you said was inappropriate and you should consider apologising for saying it") rather than attributes (e.g. "You are a racist/misogynist/whatever"). If the latter is actually true, you're unlikely to change their mind and if it *isn't* true, you're likely to miss an opportunity to educate them as they get defensive and stop listening (refer back to that opening quote!).

I don't pretend to have all (or even any of) the answers, I just believe the entire topic of effective communication and all it entails is one worthy of our collective consideration, since effective communication is almost always a necessary precursor to taking effective action (e.g. on matters such as mitigating and coping with climate change).

In many respects though, the entire topic is really quite simple. To quote Abe Lincoln in one of my all time favourite movies:

Be excellent to each other.

----
* Seriously, read the popular science books on biology that Dawkins has written, especially "The Greatest Show on Earth". They're orders of magnitudes better than "The God Delusion", which is far too laden with angry and aggressive undertones to be an effective tool for communicating with anyone that doesn't already agree with the thesis of the book. In his biology books, his obvious love and passion for the subject matter comes to the fore and they're by far the better for it.

Note for Planet Python: even though this post is about communication rather than code, I have included the python tag since a couple of different diversity related issues have come up recently on python-dev and psf-members. It is no coincidence that "communication" and "community" share a common(!) root in "communis".

Sure it's surprising, but what's the alternative?

2011-07-09T22:12:00.001+10:00

Armin Ronacher (aka @mitsuhiko) did a really nice job of explaining some of the behaviours of Python that are often confusing to developers coming from other languages.

However, in that article, he also commented on some of the behaviours of Python that he still considers surprising and questions whether or not he would retain them in the hypothetical event of designing a new Python-inspired language that had the chance to do something different.

My perspective on two of the behaviours he listed is that they're items that are affected by fundamental underlying concepts that we really don't explain well even to current Python users. This reply is intended to be a small step towards correcting that. I mostly agree with him on the third, but I really don't know what could be done as an alternative.

The dual life of "."

Addressing the case where I mostly agree with Armin first, the period has two main uses in Python: as part of floating point and complex number literals and as the identifier separator for attribute access. These two uses collide when it comes to accessing attributes directly on integer literals. Historically that wasn't an issue, since integers didn't really have any interesting attributes anyway.

However, with the addition of the "bit_length" method and the introduction of the standardised numeric tower (and the associated methods and attributes inherited from the Complex and Rational ABCs), integers now have some public attributes in addition to the special method implementations they have always provided. That means we'll sometimes see things like:

1000000 .bit_length()

to avoid this confusing error:

>>> 1000000.bit_length()
  File "stdin", line 1
    1000000.bit_length()
                     ^
SyntaxError: invalid syntax

This could definitely be avoided by abandoning Guido's rule that parsing Python shall not require anything more sophisticated than an LL(1) parser and require the parser to backtrack when float parsing fails and reinterpret the operation as attribute access instead. (That said, looking at the token stream, I'm now wondering if it may even be possible to fix this within the constraints of LL(1) - the tokenizer emits two tokens for "1.bit_length", but only one for something like "1.e16". I'm not sure the concept can be expressed in the Grammar in a way that the CPython parser generator would understand, though)

Decorators vs decorator factories

This is the simpler case of the two where I think we have a documentation and education problem rather than a fundamental design flaw, and it stems largely from a bit of careless terminology: the word "decorator" is used widely to refer not only to actual decorators but also to decorator factories. Decorator expressions (the bit after a "@" on a line preceding a function header) are required to produce callables that accept a single argument (typically the function being defined) and return a result that will be bound to the name used in the function header line (usually either the original function or else some kind of wrapper around it). These expressions typically take one of two forms: they either reference an actual decorator by name, or else they will call a decorator factory to create an appropriate decorator at function definition time.

And that's where the sloppy terminology catches up with us: because we've loosely used the term "decorator" for both actual decorators and decorator factories since the early days of PEP 318, decorator implementators get surprised at how difficult the transition can be from "simple decorator" to "decorator with arguments". In reality, it is just as hard as any transition from providing a single instance of an object to instead providing a factory function that creates such instances, but the loose terminology obscures that.

I actually find this case to be somewhat analogous to the case of first class functions. Many developers coming to Python from languages with implicit call semantics (i.e. parentheses optional) get frustrated by the fact that Python demands they always supply the (to them) redundant parentheses. Of course, experienced Python programmers know that, due to the first class nature of functions in Python, "f" just refers to the function itself and "f()" is needed to actually call it.

The situation with decorator factories is similar. @classmethod is an actual decorator, so no parentheses are needed and we can just refer to it directly. Something like @functools.wraps on the other hand, is a decorator factory, so we need to call it if we want it to create a real decorator for us to use.

Evaluating default parameters at function definition time

This is another case where I think we have an underlying education and documentation problem and the confusion over mutable default arguments is just a symptom of that. To make this one extra special, it lies at the intersection of two basic points of confusion, only one of which is well publicised.

The immutable vs mutable confusion is well documented (and, indeed, Armin pointed it out in his article in the context of handling of ordinary function arguments) so I'm not going to repeat it here. The second, less well documented point of confusion is the lack of a clear explanation in the official documentation of the differences between compilation time (syntax checks), function definition time (decorators, default argument evaluation) and function execution time (argument binding, execution of function body). (Generators actually split that last part up even further)

However, Armin clearly understands both of those distinctions, so I can't chalk the objection in his particular case up to that explanation. Instead, I'm going to consider the question of "Well, what if Python didn't work that way?".

If default arguments aren't evaluated at definition time in the scope defining the function, what are the alternatives? The only alternative that readily presents itself is to keep the code objects around as distinct closures. As a point of history, Python had default arguments long before it had closures, so that provides a very practical reason why deferred evaluation of default argument expressions really wasn't an option. However, this is a hypothetical discussion, so we'll continue.

Now we get to the first serious objection: the performance hit. Instead of just moving a few object references around, in the absence of some fancy optimisation in the compiler, deferred evaluation of even basic default arguments like "x=1, y=2" is going to require multiple function calls to actually run the code in the closures. That may be feasible if you've got a sophisticated toolchain like PyPy backing you up but is a serious concern for simpler toolchains. Evaluating some expressions and stashing the results on the function object? That's easy. Delaying the evaluation and redoing it every time it's needed? Probably not too hard (as long as closures are already available). Making the latter as fast as the former for the simple, common cases (like immutable constants)? Damn hard.

But, let's further suppose we've done the work to handle the cases that allow constant folding nicely and we still cache those on the function object so we're not getting a big speed hit. What happens to our name lookup semantics from default argument expressions when we have deferred evaluation? Why, we get closure semantics of course, and those are simple and natural and never confused anybody, right? (if you believe I actually mean that, I have an Opera House to sell you...)

While Python's default argument handling is the way it is at least partially due to history (i.e. the lack of closures when it was implemented meant that storing the expressions results on the function object was the only viable way to do it), the additional runtime overhead and complexity in the implementation and semantics involved in delaying the evaluation makes me suspect that going the runtime evaluation path wouldn't necessarily be the clear win that Armin suggests it would be.

Updated to do list for Python 3.3

2011-06-29T15:27:00.000+10:00

I wrote a to-do list for 3.3 back in January. It's already obsolete, so inspired by Brett's recent update I have a new list of my own:

Finish PEP 394 (The "python" command on *nix platforms)

This is the PEP clarifying that the current collective recommendation from python-dev to Linux distributions (et al) is to stick with Python 2.x as the system python for the moment. It also involves adding a python2 symlink to the next release of 2.7 so that we follow our own advice.

PEP 3118 buffer API and memoryview fixes

On issue #10181 I've been doing the software architect thing in devising a way to address the flaws in the design of memoryview objects. Absent further contributions I may resort to coding it myself, otherwise I'll at least review whatever patches are put forward.

PEP 380 (yield from expressions)

Guido gave his approval to PEP 380 recently. This is almost ready to go in, but some work needs to be done on incorporating the tests neatly into the regression test suite.

Import engine

Import engine is a GSoC project I am mentoring that consolidates the scattered state for the import system into methods and attributes on a single object. While the "real" import system state will remain in the current locations for compatibility reasons, the import engine API will provide a more coherent interface to the whole mechanism. It will also allow creation of more limited engine instances for targeted imports (e.g. from plugin directories) that won't be confused by name conflicts with other libraries.

I'm hopeful Greg will get as far as writing the PEP itself before the end of the summer, otherwise that will become a follow-up activity.

CPython compiler enhancements

Eugene Toder put together a nice patch that refactors the CPython compiler to include an AST optimisation step as well as cleaning up several 2.x holdovers in the AST. Getting the AST compiler ready for Python 2.5 is one of the first things I ever hacked on as a core developer and it's still one of my main interest areas.

PEP 395 (module aliasing)

After starting to work with Django and discovering it commits the sin of putting a package subdirectory on sys.path, I have even more motivation to provide a module aliasing mechanism that doesn't run the risk of accidentally getting multiple copies of the same module loaded in a process. That kind of thing should only happen when you do it on purpose.

I need to review my proposal in PEP 395 to make sure it also covers the Django use case (and modify the proposal if it doesn't).

Core mentoring

There are still some coverage patches from the Pycon US sprints to be reviewed and committed, along with a couple from the core mentoring mailing list. I'm also likely to be leading the CPython sprints following PyconAU, so there'll be some more patches to review and commit as a result of that.

importlib bootstrapping

While the migration to importlib is definitely Brett's project, it's also one I keep a close eye on in case he wants any reviews or feedback on elements of the design.

PEP process tinkering

I'd still like to clean up PEP 0 by adding the "Consensus" state to PEP 1. However, it has dropped below the above items on the to-do list.

With statement tinkering

Screwing up @contextmanager for 3.2 gives me even more motivation to want implicit context managers and Alex Gaynor tells me he has a use case for a context manager that can skip the with statement body. As with the PEP process tinkering, these are still on the "yes, please" list, but I'm unlikely to get a chance to work on them any time soon.

Other miscellaneous features

Resurrecting the post-import hooks PEP would be nice and I'd also like to add some of the ABC registration viewing and monitoring features proposed on the issue tracker.

So, how long until 3.3 again? :)

Switching to Android

2011-06-16T00:33:00.000+10:00

So my new HTC Desire S arrived today which means it is time for an app downloading spree... (starting with just freebies for now while I figure out what I do and don't like, and delete some of the cruft that HTC/Telstra added)

Already grabbed:
Google Goggles
Google Sky
Google Reader
Dropbox
Firefox
KeepassDroid
OI File Manager
Shelves for Android (now actually *scanning* my book collection will be quite a project...)
Barcode Scanner
Compass

And a ton of standard apps from Google/HTC/Telstra for all the basics (Phone, SMS, Music, Mail/Gmail, FB, Twitter, Camera, Calendar, Clock, Weather, Calculator, Adobe PDF Reader, Maps/Navigation, LED Flashlight, etc).

Things I know I want but don't have yet:

Ebook reader (I used Stanza from Lexcycle on the iPhone, but they don't make an Android version)
Weight tracker (don't need anything fancy, just something that I can import old data into and will give me a time-weighted average)

I'm also open to suggestions for things I might want but just don't know it yet, and of course I'll have to track down a few idle time games.

Fixing GRUB2 update issues with Kubuntu 11.04

2011-06-13T22:35:00.000+10:00

After doing a dist-upgrade to 11.04 a while back, my Kubuntu machine refused to boot.

I eventually tracked this down to GRUB2 os-prober feature freaking out and trying to boot off the partition that held only the "/usr" directory rather than the one with the root and "/boot" directories. (Why, I have no idea. The latter is the first partition and the one that holds the MBR, so os-prober is clearly going to some effort to find and enforce the wrong partition).

After searching on Google and with a bit of experimentation, I was able to fix it by booting off a LiveCD, adding the line "GRUB_DISABLE_OS_PROBER=true" to "/etc/default/grub" and then running "sudo update-grub".

Musings on the culture of python-dev

2011-04-22T01:42:00.000+10:00

I mentioned at the end of my PyCon summary post that several people had told me that they find python-dev to be a hostile and unwelcoming environment, and that (after some reflection on the matter) I could actually see their point. We may be the model of civility compared to somewhere like the Linux Kernel Mailing List or (*shudder*) Blizzard's World of Warcraft forums, but mere civility is a far cry from being a consciously welcoming place.

I'll say up front that python-dev itself is unlikely to change any time soon. There are reasons it is the way it is, and I'll elaborate on some of them later in this post. In the meantime, python-ideas is available as a venue where outlandish ideas won't be rejected quite as abruptly, and the core-mentorship list has been set up specifically to provide a gentler introduction to the ways and means of core development without having to jump right in at the deep end by posting to python-ideas or python-dev. Questions about development with Python will continue to be redirected in fairly short order to python-list or python-tutor.

And, of course, what follows is just my opinion. Ask another veteran python-dev poster what they think, and you'll likely get a different answer. Ask an actual newcomer to (or lurker on) python-dev, and they'll probably have a different answer, too.

Python evolves too slowly!

You're changing the language too fast!

If I had to choose just one explanation for the frequency of abrupt responses in python-dev, the tension between the above two statements would be it. Compared to application-level open source projects, Python actually evolves quite slowly. 18+ months between minor version increments? 10 years between major versions? That's crazy! Canonical releases an entire new OS version every 6 months!

On the flip side, however, for a programming language definition, Python evolves quite fast. There's no such thing as a minor release for C or C++ - the last versions of those were C99 and C++98 respectively. We should see C++11 published by the end of the year, and C1X is still in development. CPython's own PEP 7 still mandates the use of C89 compatible constructs for portability reasons, even though C89 is older than one of our release managers.

Java hasn't had a major feature update since 2006, and even C# is only running at a new version every 2-3 years (with the last formally standardised version being 2.0 back in 2006).

Python also has a history of being more aggressive with deprecations than are languages backed by large corporate sponsors. We only have limited volunteer resources, so rather than letting old code bitrot (or else take up maintenance time when it breaks), we'd prefer to rip it out in favour of an improved alternative. However, "more aggressive" is still pretty slow - some deprecated features stuck around for nearly 10 years (until Python 3 came out) and even a "fast" deprecation has historically taken at least 3 years (x.y contains feature, x.y+1 deprecates it, x.y+2 removes it). It's highly likely that that deprecation period will be extended out by a release for the 3.x series, pushing the minimum lifetime of a new feature that later proves to be a mistake out to nearly 5 years.

The task of updating the language and the standard library is a balancing act between those two forces - we want to make life easier for programmers adopting Python for new activities, while preserving backwards compatibility for existing applications. This is why Python 3 is such a big deal - most decisions are made with the emphasis on the needs of existing Python programmers, but the decision to create Python 3 was largely for the benefit of future Python programmers. That means that all current Python programmers are lumped with the task of actually managing a disruptive transition that wasn't really designed for their immediate benefit. Obviously, the collective opinion of python-dev is that it will be worth the pain in the long run, but those anticipated benefits don't make the migration any easier to deal with right now.

We get quite a few people coming into python-dev and betraying quite quickly that they don't have any respect for the time frames involved in language (rather than application) development. Telling someone "You're wrong, but explaining the real reasons why would require that I distil decades of experience down into a single email post and I don't feel like taking the time to do that right now, since even if I tried you would ignore me anyway" tends to be difficult to phrase politely.

A lot of the rest of this post is really just elaborations on the theme of why a programming language needs to evolve more slowly than most other pieces of software.

Heart of the ecosystem, but far from the whole of it

I've elaborated on the cost of change before, when discussing the design principle "Status Quo Wins a Stalemate". The core point is that any significant change made by python-dev, even one that will ultimately be a positive one, imposes a high near-term cost as the implications of the change ripple out across the whole Python ecosystem. As noted above, newcomers can easily perceive this as high-and-mighty arrogance rather than the voice of experience.

What do you mean by "cognitive burden"?

Even without considering the near-term cost of changes, every addition to the language (and even the standard library) imposes a potential burden on anyone learning the language in the future. You can't just say, "Oh, I won't worry about learning that feature" if the code base you've been asked to maintain uses it, or if it is offered as an answer to a query posted on python-list or Stack Overflow or the like. The principle of "There Should Be One - and preferably only one - Obvious Way To Do It" is aimed squarely at reducing the cognitive load on people trying to learn the language. Quite clearly, the "only one" aspect is an ideal rather than a practical reality (two kinds of string formatting and three argument parsing libraries in the standard library all say "Hi!"), but in such cases we do try to indicate that the most recently added (and hopefully least quirky) approach is the preferred way to do it.

This idea is also encountered as the aphorism "Not every three line function needs to be a builtin". Again, new posters may not take kindly to being told that their idea simply doesn't cut it as a potential language addition.

Gee, how dumb are you lot? Why don't you just...?

Another favourite bugbear is posters that bounce into python-dev assuming that we're a collection of clueless idiots that can't see the obvious. Collectively, we do pay quite a bit of attention to what other language communities are doing, as well as having personal experience with what does and doesn't work in actual programming practice. There's a reason that "new" Python features are generally modelled on something that has been demonstrated to work elsewhere (e.g. list comprehensions et al inspired by Haskell, the with statement partially inspired by C++ RAII, the new string formatting syntax inspired by C# string formatting).

New posters that give the list a tiny bit of credit and do us the courtesy of at least asking "Has this been thought of or discussed before? Are there any problems with it that I haven't considered?" tend to get significantly more positive reactions than those that start with a tone closer to "Here is my awesome idea, and you are seriously dumb if you don't get it and decide to adopt it immediately!". Positive responses are even more likely if ideas are posted to the right list (i.e. python-ideas).

You do remember you didn't have to pay a cent for this, right?

A fortunately rare (but still annoying when it arises) source of negative reactions is the end user that comes in demanding to know why certain things aren't being done, when the answer is "Because nobody stepped up to either do it themselves, or to pay for someone else to do it". It's a pretty simple equation, really, and not demonstrating understanding of it suggests a complete disregard for the volunteer nature of so many of the contributions that have been made to Python over the years.

In the end, we're still just people

We like painting bikesheds (or, more to the point, we can't always help ourselves, even when we know better). We like to be right and "win" arguments (or sometimes simply take time to process and properly understand the point someone else is trying to make). Even the mailing list members that are paid to work with Python by our employers are typically still participating in python-dev and hacking on CPython in our spare time rather than as a job, so there's not a lot of tolerance for "noise" and "time wasting".

As I'm a firm believer in the phrase "Vigorous criticism is the only known antidote to error", there are limits to how much I would personally want the culture of python-dev to change. Moving "blue sky" dreaming to python-ideas, "how does the process work?" coaching to core-mentorship and VCS management issues to python-committers allow them to develop cultures more appropriate to those specific activities, allowing python-dev to really focus in on the "vigorous criticism" part of the story. Keeping that from crossing the line into excessive negativity and a total reluctance to change is an ongoing challenge, but hopefully an awareness of that danger and the occasional pause for reflection will be enough to keep things on the right track.

The benefits (and limitations) of PYC-only Python distribution

2011-04-09T22:47:00.001+10:00

This Stack Overflow question hit my feed reader recently, prompting the usual discussion about the effectiveness of PYC only distribution as a mechanism for obfuscating Python code.

PYC Only Distribution

In case it isn't completely obvious from the name, PYC only distribution is a matter of taking your code base, running "compileall" (or an equivalent utility) over it to generate the .pyc files, and then removing all of the original .py source files from the distributed version.

Plenty of Python programmers (especially the pure open source ones) consider this practice an absolute travesty and would be quite happy to see it disallowed entirely. Early drafts of PEP 3147 (PYC Repository Directories) in fact proposed exactly that - in the absence of the associated source file, a compiled PYC file would have been ignored.

However, such blatant backwards incompatibility aroused protests from several parties (including me), and support for PYC-only distribution was restored in later versions of the PEP (although "compileall" now requires a command line switch in order to generate the files in the correct location for PYC-only distribution).

Use Cases

As I see it, there are a couple of legitimate use cases for PYC-only distribution:

Embedded firmware: If your code is going onto an embedded system where space is at a premium, there's no point including both your source code and the PYC files. Better to just include the compiled ones, as that is all you really need
Cutting down on support calls (or at least making the ones you do get more comprehensible): Engineers and scientists like to tinker. It's in their nature. When they know just enough Python to be a danger to themselves and others, you can get some truly bizarre tickets if they've been fiddling with things and failed to revert their changes correctly (or didn't revert them at all). Shipping only the PYC files can help make sure the temptation to fiddle never even arises

Of the two, the former is by far the stronger use case. The latter is attempting a technical solution to a social problem and those rarely work out well in the long run. Still, however arguable its merits, I personally consider deterrence of casual modifications a valid use case for the feature.

Drawbacks

Stripping the source code out of the distribution does involve some pretty serious drawbacks. The main one is the fact that you no longer have the ability to fall back to re-compilation if the embedded magic cookie doesn't match the execution environment.

This restricts practical PYC-only distribution to comparatively constrained environments that can ensure a matching version of Python is available to execute the PYC files, such as:

Embedded systems
Corporate SOEs (Standard Operating Environments)
Bundled interpreters targeting a specific platform

Cross-platform compatibility of PYC files (especially for 32-bit vs 64-bit and ARM vs x86) is also significantly less robust than the cross-platform compatibility of Python source code.

Limitations

Going back to the SO question that most recently got me thinking about this topic, the big limitation to keep in mind is this: shipping only PYC files will not reliably keep anyone from reading your code. While comments do get thrown away by the compilation process, and docstrings can be stripped with the "-OO" option, Python will always know the names of all the variables at runtime, so that information will always be present in the compiled bytecode. Given both the code structure and the original variable names, most decent programmers are going to be able to understand what the code was doing, even if they don't have access to the comments and docstrings.

While there aren't any currently active open source projects that provide full decompilation of CPython bytecode, such projects have existed in the past and could easily exist again in the future. There are also companies which provide Python decompilation as a paid service (decompyle and depython are the two that I am personally aware of).

Alternatives

You can deter casual tinkering reasonably well by placing your code in a zip archive with a non-standard extension (even .py!). If you prepend an appropriate shebang line, you can even mark it as executable on POSIX based systems (see this post for more information).

You could also write your code in Cython or RPython instead of vanilla Python and ship fully compiled executable binaries.

There are minifier projects for Python (such as mnfy) that could be fairly readily adapted to perform obfuscation tricks (such as replacing meaningful variable names with uninformative terms like "_id1").

Climate change "skepticism": a text book case of the "Tragedy of the Commons"

2011-03-30T00:44:00.002+10:00

So, with the government moving to implement a carbon tax, the issue of climate change and attempts to mitigate it are once again a big deal here in Australia.

Inspired by a friend's comment comparing carbon taxes with fines for littering, along with a few of the responses to that comment, I started pondering the similarities and differences between the current attempts to deal with carbon dioxide emissions, and the effective cost internalisation schemes created in the 1990's that severely reduced industrial sulphur dioxide emissions, as well as various current laws that prohibit dumping of toxic waste in developed nations (and their unintended side effects).

(Fair warning before I wade in: I've simplified quite a few things, particularly on the acid rain front. This post is long enough as it is, without making it even longer with all the relevant caveats about other causes of acid rain and any other details I've glossed over as being irrelevant to the main point of the article. It's a general overview, not a peer reviewed scientific paper)

A tale of two gases
The key similarity between sulphur dioxide induced acid rain and carbon dioxide induced global warming is hopefully fairly obvious: they both represent a classic economic "externality", a cost borne by someone other than the person responsible for causing it. The industries and individuals emitting these pollutants don't suffer any immediate harmful consequence through the normal action of nature.

In situations like that, regulation in one form or another is the only effective tool we have to ensure that at least some portion of those external costs is reflected back on the responsible individuals. As noted in the link above, this approach proved extraordinarily effective in reducing acid rain in the US, drastically cutting sulphur emissions at a fraction of the originally predicted cost.

However, there are a few key differences that have contributed to the discussion over carbon dioxide taking a rather different path to that over sulphur dioxide:

It's hard to deny the harmful effects of sulphur dioxide emissions when plants, fish and insects are dying due to the excessive levels of acidity in rain and ground water. By contrast, it is very hard to convey clearly why a small rise in average global temperatures is going to be such a bad thing, or even how human carbon dioxide emissions contribute to that happening.
Acid rain from sulphur emissions is generally a local problem with local causes (from a continental perspective, anyway). Stop emitting sulphur dioxide in the US and acid rain stops falling in the US. Significantly reduce carbon dioxide emissions in Australia (or even the US) though, and we're likely still screwed if other countries don't follow suit.
The "sulphur" cycle (how long it takes for the sulphur emissions to be deposited back on the ground as acid rain) is significantly shorter than that of carbon, so efforts to reduce emissions will have an effect on water acidity levels in a reasonably short time frame

Comparisons with anti-waste dumping laws are similar: the effects of toxic waste dumping are generally both local, obvious and able to be cleaned up within years rather than decades, so it is easy to get support for laws prohibiting such practices, even if the end result of those laws is just to export the problem to a poorer country.

Manufacturing doubt and exploiting the "Grey Fallacy"
Acid rain is an easy problem to sell: the acid damages plants and objects directly, so the harm can be easily conveyed through pictures and videos. The problem of climate change, though is far, far harder to illustrate simply, since it is a signal buried in some very noisy weather patterns.

A fundamental issue is the fact that people don't really experience climate, but instead experience weather. The day to day and seasonal variations in the weather massively exceed the scale of the underlying trends being discussed by climate scientists. The fact that predicting the weather in a few days time is harder than predicting long term climate trends is actually massively counterintuitive, even though almost all scientists and engineers are familiar with the fact that random fluctuations at a small scale may average out to something almost perfectly predictable at a large scale (compare the unpredictable nature of quantum physics with the straightforward determinism of classical Newtonian mechanics).

We're also not really used to the idea of directly affecting the balance of the planet on a global scale. In absolute terms, the effects of the sun, the oceans and the plants on the carbon cycle all dwarf the human contribution. The idea that the non-human forcings were actually largely in balance with each other, and that the comparatively small human contribution is enough to tip that balance to the point of producing more carbon than can be absorbed by natural mechanisms is really quite a subtle one.

The scientific background needed to truly understand and come to grips with the hows and the whys of the IPCC predictions reminds me of this comic regarding the knowledge needed to even begin to understand the vagaries of string theory (make sure to click through on the comic itself to see the whole chain of images).

If there wasn't anyone around with a vested interest in maintaining the status quo (at least for a while longer), this likely wouldn't be a problem. Scientists, politicians, economists and engineers could proceed with the development of mitigation strategies, while also attempting to educate the general populace as to the reasons behind any actions taken. Since such vested interests do exist, however, their opposition makes it significantly harder to convince the lay public that there is a real problem here.

Because climate science is such a complex topic, it is actually quite hard to explain to a non-scientist just why the scientific consensus is as strong as it is. If you oversimplify, you veer into territory where statements are so oversimplified that they border on being false. However, if you're coming from the other angle and want to persuade people that the science "isn't settled", then you're no longer constrained by the need to be accurate and can just go with whatever sounds more intuitively plausible and/or better caters to people's natural inclinations and prejudices.

Sites like Skeptical Science do their best to clearly explain why the oft-repeated "skeptic" arguments are basically BS (and do it well), but to someone only following the Cliff Notes mainstream media version of the debate, the temptation is still very, very strong to just assume the truth lies somewhere between the two positions being debated.

Telling people to "do their own research" doesn't really help all that much in practice. Telling the BS from the valid science is itself a fine art that takes a great deal of skill and experience. Being a veteran of arguments with creationists (and even intelligent design advocates) is actually quite beneficial, since the global warming "skeptics" use many of the same rhetorical tricks as creationists do when attempting to deny the fact of evolution (incessant use of such deceptive tactics is actually one of the major hints that someone is trying to sell you a line rather than just stating the truth as they see it). The most common tactic used by both groups is to drag out a thoroughly rebutted argument for each new audience, in hopes that they aren't aware of the existence of a rebuttal.

For example, just as most people can't answer "How could a bombardier beetle possibly evolve?" off the top of their heads - I've actually forgotten all of the plausible answers myself - neither can they answer questions like "If the climate is supposed to be warming, why is it colder now than it was during the Middle Ages?". While that one is actually fairly straightforward to answer (ice cores and other data shows that the medieval warming was likely localised to Europe due to various currents in the Atlantic, but this merely shifted heat around, so that other parts of the world were correspondingly colder), there are dozens of other oft-repeated thoroughly debunked arguments, and it's basically impossible for a mere interested observer to remember them all. As it turns out, I had actually misremembered the correct explanation for the Medieval warm period, so the point above isn't quite right and invites an attack on the grounds that I don't know what I'm what I'm talking about. To some extent that's actually true - my opinion on climate science issues is based on a meta-analysis of the trustworthiness of various information sources (including the full IPCC reports), since I'm not inclined to spend a few decades studying up and redoing all the science myself. Fortunately, Skeptical Science has the full story for this question and many others (the MWP was actually colder than the latter half of this century, despite higher solar activity and lower volcanic activity), so correcting my error is the task of a few moments. And if you're still wondering about the bombardier beetle thing, TalkOrigins has the full story for that, too.

Fortunately, at the decision making level here in Australia, this part of the debate seems to be coming to a close, with even Toby Abbott (the leader of the opposition) at least paying lip service to the fact that human-induced increases in average global temperatures are now a fact of life. However, the mainstream media is still happy to "teach the controversy" in the hopes of picking up a few more eyeballs (or ears) to sell to their advertisers.

But the problem is so much bigger than us!
However, even once you get agreement that human-induced global warming due to excessive carbon dioxide emissions is a genuine problem, you then run into the "But we can't do anything about it!" attitude.

Many Australians (including some elected members of our Federal parliament) go "Oh, but the US/Europe/Brazil/Russia/India/China have a much bigger impact than we do. Doing anything will hurt our international competitiveness without really achieving anything, so we shouldn't bother."

Even the higher emission countries point fingers at each other. The US won't budge until China does. The BRIC countries are waiting for the US to make a move, and use US inaction as justification for similarly doing nothing in the meantime.

An absolutely textbook "Tragedy of the Commons" reaction. And we know how that story ends, don't we? In the long run everybody loses, nobody wins.

How do you fix it? Rules, regulations and social pressure. The "community of nations" isn't just words. The complex web of interdependencies that spans the world gives countries real power to exert influence on each other. Once some countries start to make a move towards serious carbon emission control strategies, then that becomes a bargaining chip to use in international negotiations. Will it work? Who knows. The only thing we know for sure is that the more carbon we pump into the atmosphere over the next few decades, the higher the impact from global warming will be (it's now guaranteed that there will be some impact - the carbon cycle is too long for us to prevent that at this late stage of the game).

Ideally you would want to adopt an emissions trading scheme along the lines of that used to curb sulfur emissions, but the wholehearted embrace of dodgy pseudoscience by far too many members of our Federal Opposition party spiked that particular barrel.

So a carbon tax it is (for now, anyway). The hysterical cries of "Oh my god, you've doomed the country!" ring loudly from those that will bear most of the direct costs of actually internalising the negative effects their carbon emissions have on the wider environment. Their motives are, to say the least, a little bit suspect. The political opportunism involved in our Federal Opposition leader backing them is disappointing, but unsurprising.

In the PM's words
There's a nice summary of the current situation in a Fairfax opinion piece that went out over Gillard's byline:

The longer we leave it the harder action on climate change gets. This reform road is a hard one to walk. Just as doing nothing is not an option, we need to be careful to ensure that we do not make decisions that will cost our economy and jobs.

There's a definite risk that the government's plans won't live up to their ambitions. However, the world's past experience with the sulfur dioxide doomsayers should arouse some deep skepticism and not directed towards those wanting to press forward with plans to curb carbon emissions.

Queensland's "unelected" leader of the opposition

2011-03-29T21:47:00.002+10:00

Following the declaration of Brisbane's Lord Mayor that he is running for state parliament at the next election, and will be the leader of the opposition for that campaign, there is a trope making the rounds that he is somehow "unelected".

Out of curiosity, I decided to see how his numbers in the Brisbane City Council elections in 2008 stacked up against Anna Bligh's numbers in the 2009 state election.

The internet being what it is, official sources for these numbers weren't too hard to dig up:
I found the Lord Mayoral results on the Brisbane City Council site
I found the Electorate of South Brisbane results on the Electoral Commission Qld site (with the preferences taken into account here).

Results:
Campbell Newman received 335,076 first preference votes for Lord Mayor (60% of the total).
After distribution of preferences, he received a total of 339,320 votes (66% of the total).
It's probably worth noting that the second placed candidate received 29% of the first round votes, increasing to 34% after distribution of preferences, so Newman actually did receive the majority of the remaining preferences, there just weren't many to go around.

Anna Bligh received 12,243 first preference votes in her electorate of South Brisbane (48% of the total).
After distribution of preferences, she received a total of 14,697 votes (65% of the total).

So, our State Premier holds her position on the back of the support of her party and around 15k people living in or near South Brisbane.

The new leader of the opposition will hold that position on the back of the support of his party and around 340k people living in the City of Brisbane.

Since the current population of Qld is estimated at 4.5 million people, effectively 7.5% of the state voted for Newman to be Brisbane's Lord Mayor, while only 0.3% voted for Bligh to represent the seat of South Brisbane.

And people are calling Newman an unelected leader?

(Yes, I'm aware that the nature of parliamentary governments based on geographic representation means that most constituents don't get to vote directly for their parliamentary leaders. The only purpose of this post is to point out how dumb that makes the "unelected" gibe sound when it is aimed at the directly elected Lord Mayor of a city the size of Brisbane)

Thoughts and Impressions following PyCon 2011

2011-03-20T06:20:00.000+10:00

I'm back home following my inaugural trip to PyCon, so it seems like a good time to record my impressions of the summits, conference and sprints.

I really enjoyed the whole experience - kudos to Van Lindbergh, Jesse Noller and the rest of the volunteer team for organising everything. I'm glad to see the "apprenticeship" setup for the conference leadership continuing, with Jesse (the deputy coordinator this year) stepping up to coordinate Santa Clara, with the future coordinator for Montreal assisting during those two years.

The personal connection side of things was brilliant. When it comes to the folks that were already on python-dev before I really started to follow it back in late 2003, I've been interacting and working with them online for 8+ years, and there are of course many others that have joined python-dev in the intervening time. I'd seen a few of them in photos and videos, and heard a few in videos and podcasts, but by and large, this was the first time I had been able to connect faces and voices to names on the screen. Very cool stuff, including getting to meet Raymond Hettinger (who accepted my first patches back in '04, among them the looks-insane-but-it-works string hack to speed up the 2.x decimal module) and of course Guido himself (who was the one who actually granted me commit rights, essentially for making sense while arguing with him about PEP 340/346/343).

Getting ready for Pycon was actually also the motivation behind restarting this blog and adding it to Planet Python, finally getting myself a Twitter account (@ncoghlan_dev) and (after getting home) hooking my DISQUS profile up to that. They're all aspects of taking a bit more of an active part in the wider Python community after getting a taste of it at PyconAU last year (despite the fact that I have yet to make it to a BrisPy meeting... Wednesday night just isn't a good night for me these days).

From a more technical perspective, there were a few things that I found particularly interesting:

1. PyPy is definitely starting to make the transition from "experimental platform for research into dynamic language optimisation" to "let's use this to make production code go faster". This shows not only in their benchmark results, but also in their efforts to update their website to be more production-user friendly and the effort to get more major projects running on it at the sprints, including those that stress the boundaries of the distinction between the language definition and CPython implementation details (*cough*SQLAlchemy*cough*). One of those efforts actually revealed a bug in one of the dark corners of the CPython implementation (folks in Hanover F at the sprints may have heard me swearing about my various attempts at fixing that one...)

2. There is definite interest in supporting Python 3 in more modules and packages, as well as improving the available information out there regarding published packages. There's likely to be at least one after-the-fact PEP to better explain one of the major C API changes that bit some sprinters attempting to forward port zc.buildout (I think that was the affected package), there is collaboration developing amongst the Linux distros (and others) to get more existing packages on Python 3 (join the python-porting list if that project interests you), there are a couple of new sites with improved information on the level of Python 3 support in various modules, and the team behind djangopackages are working on providing the same service for the whole of PyPI (and, no doubt, Python 3 support will end up being one of the points of comparison).

3. With distutils2 entering the standard library as "packaging" in 3.3 (to reflect the scope creep in the mission of the package, as well as to avoid name conflicts with future backports of distutils2 post 3.3 release), it was fascinating listening to the sprinters discussing how to take their clean 3.3 code from the standard library and backport it (as distutils2) to run on 3.2, 3.1, 2.7, 2.6, 2.5 and 2.4 without cluttering the stdlib version with backwards compatibility cruft. If their results are a match for their goals, then their new 2to3 and 3to2 inspired tool may end up becoming the basis for a general purpose Python "backport" transformation technique that is able to iteratively downgrade Python code to support earlier versions, while still allowing the use of clean, idiomatic code in the latest version.

4. The understanding of how best to leverage the Mercurial transition is still evolving on python-dev. My personal opinion has now developed towards one where I hope we will start using more feature clones (rather than branches within the main repository), with the main cpython repository only being used to accept feature-complete (or near complete) contributions. We're actually pretty close to supporting that model now, it just needs a few tweaks to the way builds are pushed to the buildbots to get us the rest of the way to being able to trial code on the full buildbot fleet without having to push it into the main repository first.

5. Collaboration efforts between the 5 biggest Python implementations (CPython, PyPy, Jython, IronPython, Stackless) continue to evolve. The PSF announced $10k in direct funding to PyPy at the start of the conference proper, the main python.org download page now includes links to the sites of the other 4 major implementations, more contributors to the other projects were given CPython push rights to allow standard library fixes to be contributed upstream rather than maintained in downstream forks, there are plans in progress to create a collaborative "speed.python.org" modelled on the existing PyPy benchmark site and Brett Cannon plans to revive the PEP about splitting the standard library out to a separate source control repository.

6. Brian Curtin has a nice write-up welcoming several newcomers that made their first submissions to the CPython bug tracker at the sprints. Brett's idea of improving test coverage as an introductory activity is a *great* idea, since such changes are relatively easy to get started with, relatively easy to review, low risk of breaking anything (except the buildbots) and involve actually writing code. I'll also note here that Eric Snow spent the sprints working on a more esoteric idea that came out of a python-ideas discussion: see what would be involved in providing a variant on exec() that allowed an actual function body to be executed in a designated namespace.

I was also surprised (and somewhat concerned) at the number of people that perceived python-dev as a hostile, unwelcoming place. On further reflection, I realised there was actually some merit to that point of view, but that's a topic for another post.

Python Language Summit - Highlights

2011-03-11T22:39:00.000+10:00

The language summit covered a fair bit more ground than the VM summit did, so this post only covers the topics that I personally found particularly interesting. My rough notes at least mention everything that was discussed.

Moar Speed
The question of speed, optimisations and benchmarking came up again. A lot of this was just retreading the ground from the VM summit with a wider audience. One thing that did become clearer is that the near-term points of contact for the speed.python.org concept are Maciej Fijałkowski from the PyPy team for the technical issues and Jesse Noller on the PSF side for the hosting/organisational issues (although I expect Jesse would welcome offers of assistance if anyone else, particularly existing PSF members, wanted to step up and help coordinate things).

Communicating the collective opinions of python-dev
Where are we going, what are we doing? The main communications channels for python-dev have historically been PEPs (including release PEPs), the What's New document for each release and of course the mailing list itself. The python-dev summaries project has been tried a couple of times, but generally burned out the people involved.

Doug Hellman (PSF communications officer) would like to try a setup where there is an official python-dev technical blog where major discussions and decisions (including the outcomes of PEPs) can be presented in easier to swallow chunks, giving the gist of significant decisions and discussions, with references back to the source PEPs and mailing list threads.

It's an interesting idea, but, as Guido pointed out, will likely require *new* people to step forward to do it that are interested in the idea of helping to provide a window into the goings-on of python-dev (hopefully the more interesting parts, where we aren't just arguing about the colour of the current bikeshed du jour). From a personal point of view, I know I've only just really started using *this* blog to talk about my own perspective on Pythonic stuff. Something that may be practical is for the python.org technical blog to highlight blog posts where existing core devs are talking about new and upcoming stuff on their personal blogs.

Doug Hellman is the point of contact for anyone interested in following up on this.

Policy on use of accelerator modules
There are a few unwritten policies regarding the use of implementation-specific accelerator modules to speed up parts of the standard library (such as "always test both versions", "the accelerated version should be API compatible with the Python version", "the interpreter should still work if the accelerated version is missing").

Brett Cannon has volunteered to write these down in an official policy PEP. While CPython is likely the main offender here, it will be suggested that other implmentations follow the same policy for their own accelerator modules. Patches to bring CPython more inline with this policy, include providing pure Python alternative of existing C-only modules, are definitely of interest.

Compatibility warnings
With the rise in significance of alternate implementations, some grey areas in the language definition (such as the reliance on refcounting semantics, abuse of the ability to store non-string keys in CPython namespaces, storing objects that implement the descriptor protocol in classes without considering the consequences) are potential sources for confusion when they break on other versions (or potentially even in future versions of CPython.

ResourceWarning was added a while back to cover the refcounting issue, and uncovered a few bugs in CPython and its test suite. The proposal is to add CompatibilityWarning as a peer exception to ResourceWarning and use it for cases where people are relying on CPython implementation accidents that aren't officially supported by the language definition.

Nobody has stepped forward to write the PEP for this as yet, but it may make an interesting sprint topic (I know at least Brett and I are around for the sprints, and there should be a few other CPython core devs kicking around).

Better exception info
ImportWarning will likely acquire a "module" attribute during the sprints (this is an easy one, since it will just reference the module's name). There are other expections that could probably do with having the repr() of critical bits of information stored separately on the exception object (e.g. KeyError, ValueError, IndexError) for easy programmatic access.

Using repr() helps avoid issues with reference cycles keeping things along longer than intended. However, the API for creating such enhanced exceptions would still need to be worked out, as well as how best to deal with cases where third party code has only populated the exception message without filling in the specific details. Technically, even ImportError isn't immune to that concern, as raising it is sometimes the responsibility of third party PEP 302 importers and loaders.

Python Language Summit - Rough Notes

2011-03-11T08:31:00.001+10:00

Same drill, different day, more people, more notes :)

Still just my interpetation, though. Will probably highlight a few things I find particularly interesting again tomorrow (as I did for the VM summit).

PSF Communications (Doug Hellman)
- currently writing about PSF funding and similar activities
- would like to include more technical material summarising python-dev discussions
- how best to go about that
- new blog, not existing PSF blog
- existing PSF board not in the position to do it
- Guido: core devs already do a lot via PEPs and mailing list, likely not keen to write blog as well
- may be better to get others to do it, willing to follow discussions of interest
- posts may be primarily pointers to other resources (e.g. PEPs, mailing list posts)
- all implementations
- major new releases should go on python.org as NEWS items

Warnings for things that may cause problems on different implementations
- ResourceWarning helps to pick up reliance on CPython specific refcounting
- CompatibilityWarning for reliance on non-strings in namespaces (e.g. classes)
- update Language Spec to clarify ambiguous situations
- like ResourceWarning, silence CompatibilityWarning by default
- what to do about builtin functions that currently aren't descriptors (i.e. doesn't change behaviour when retrieved from a class)
- e.g. make staticmethod objects directly callable
- big gray area in language spec - callables may not be descriptors
- perhaps change CPython builtin descriptors to warn about this situation
- another use case for CompatibilityWarning
- Guido not convinced the builtin function problem can be handled in general
- a better callable variant of staticmethod may be better, that allows the default descriptor behaviour to be easily stripped from any function
- doesn't want to require that all builtin functions follow descriptor protocol, since it is already the case that many callables don't behave like methods
- a better staticmethod would allow the descriptor protocol to be stripped, ensuring such functions can be safely stored in classes without changing behaviour

Standard Library separation
- see VM summit notes
- over time, migrate over to separate repository for standard lib, update
- need Python and C modules stay in sync
- buildbots for standard library
- challenge of maintaining compatibility as standard lib adopts new language changes
- need a PEP to provide guarantees that C accelerators are kept in sync (Brett Cannon volunteered to write test)
- bringing back pure Python alternatives to C standard library is encouraged, but both need to be tested
- accelerator modules should be subsets of the Python API
- Brett will resurrect standard library PEP once importlib is done
- full consolidation unlikely to be possible for 2.7 (due to CPython maintenance freeze)

Speed Benchmarking
- see VM summit notes
- really good for tracking performance changes across versions
- common set of benchmarks
- OSU OSL are willing to host it
- backend currently only compares two versions
- first step is to get up and running with Linux comparisons first, look at other OS comparisons later
- hypervisors mess with performance benchmarks, hence need real machines
- should set up some infrastructure on python.org (benchmark SIG mailing list, hg repository)
- eventually, redirect speed.pypy.org to new speed.python.org
- longer term, may add new benchmarks

Exception data
- need to eliminate need to parse error strings to get info from exceptions
- should be careful that checks of message content aren't overly restrictive
- PEP 3151 to improve IO error handling? (Guido still has some reservations)
- ImporError needs to name module
- KeyError, IndexError, ValueError?
- need to be careful when it comes to creating reference loops
- exception creation API also an issue, since structured data needs to be provided

Contributor Licensing Agreements
- Jesse and Van looking to get electronic CLAs set up
- will ensure adequately covers non-US jurisdictions

Google Summer of Code
- encouraging proposals under the PSF umbrella

Packaging
- distutils2 should land in 3.3 during the sprints
- namespace packages (PEP 382) will land in 3.3
- external name for backports should be different from internal name
- too late to introduce a standard top level parent for stdlib packages
- external backports for use in older versions is OK
- external maintenance is bad
- hence fast development cycles incompatible with stdlib
- want to give distutils2 a new name in stdlib for 3.3, so future backports based on version in 3.4 won't conflict with the standard version in 3.3

Python 3 adoption
- py3ksupport.appspot.com (Brett Cannon)
- supplements inadequate trope data on PyPI with manual additions
- Georg Brandl has graphical tracker of classification data on PyPI over time
- Allison Randall/Barry Warsaw have been doing similar dependency tracking and migration info for Ubuntu
- giant wiki page for Fedora Python app packaging
- good dependency info would provide a good ranking system for effectively targeting grants
- 3.python.org? getpython3.com? need to choose an official URL
- funding may help with PyPy migration
- IronPython will be looking at 3.x support once 2.7 is available (this week/next week timeframe)
- Jython focused on 2.6 now, may go direct to 3.x after that (haven't decided yet)
- PSF funding needs a specific proposal with specific developer resources with the necessary expertise and available time
- CObject->Capsule change is a compatibility issue for C extension modules
- Django targeting Python 3 support by the end of summer
- zc.buildout is a dependency of note that hasn't been ported yet (Pycon sprint topic)
- other migration projects being tackled at Pycon sprints (webop?)

Python upstream and distro packaging
- PEP 394 - recommendations for symlinks practices
- PEP 3147 and 3149 were heavily targeted at helping distros share directories across versions
- namespace packages (PEP 382)
- PEP 384 stable ABI (done for 3.2)
- better tools needed to help with migration to stable ABI

Baseline Python distro installs
- system python varies in terms of what is installed
- challenging to target, as available modules vary
- "build from source" is only a partial answer as some build dependencies are optional
- distros make some changes to support differences in directory layouts
- some changes affect Python app dependencies (e.g. leaving out distutils)
- conflict between "system Python" use case of what is needed to run distro utilities and "arbitrary app target" for running third party apps
- distributing separate Python under app control is not ideal, due to security patch management issues
- specific problems are caused by removal of stuff from base install (e.g. distutils)
- other problem is when distro uses old versions of packages (but virtualenv can help with that)
- may help if a "python-minimal" was used for the essential core, with "python" installing all the extras (including distutils, tkinter, etc)
- then have a further python-extras (or equivalent) that adds everything else the distro needs for its own purposes
- distros tend to work by taking a CPython build and then splitting it up into various distro packages
- to handle additions, would be good to be able to skip site-packages inclusion in sys.path (ala virtualenv).
- "-S" turns off too much (skips site.py entirely, not just adding site-packages to sys.path)
- "-s" only turns off user site-packages, not system site-packages

Python 3.3 proposed changes to strings to reduce typical memory usage
- PEP 393 changes to internal string representation (implementation as GSoC project)
- Unicode memory layout currently split in order to more easily support resizing and subclassing in C
- need to build and measure to see speed and memory impacts
- alternative idea may be to explore multiple implementation techniques (similar to PyPy)

Speed (again!)
- Unladen Swallow dormant. Major maintainers moved on to other things, fair bit of work in picking it up
- even trying to glean piecemeal upgrades (e.g. to cPickle) is a challenge
- interest in speeding up Python has really shifted to PyPy
- for CPython, gains would need to be really substantial to justify additional complexity
- really need to get the macro benchmarks available on 3.x
- Guido: pickle speedup experience is to be cautious, even when the speed gains are large.
- speed hack attempts on CPython are still of interest, especially educational ones
- speeding up overall is a very hard problem, but fixing specific bottlenecks is good
- stable ABI will help
- PyPy far more sensitive to refcounting bugs than CPython
- static analysis to pick up refcounting bugs could help a great deal
- "Here there be dragons": Unladen Swallow shows that overall speedups are not easy to come by

Regex engine upgrade
- new regex library proposed
- added many new features, including the Unicode categories needed to select out Python 3.x identifiers
- potentially big hassle for other implementations since re module includes a lot of C
- IronPython currently translates to .NET compatible regexes, but could rewrite more custom code

GUI Library
- Guido: GUI libraries are nearly as complicated as the rest of Python put together and just aren't a good fit with the release cycle of the standard lib
- Don't want to add another one, but don't want to remove Tcl/Tk support either

twisted.reactor/deferred style APIs in the standard library
- asyncore/aynchat still has users
- would like to have an alternative in the stdlib that offers a better migration path to Twisted
- deferred could be added, such that asyncore based apps can benefit from it
- reactor model separates transport/protocol concerns far more cleanly than asyncore
- protocol level API and transport level API for asyncore may be a better option
- would allow asyncore based applications to more easily migrate to other async loops
- defining in a PEP would allow this to be the "WSGI" for async frameworks ("asyncref", anyone?) (Jesse suggested concurrent.eventloop instead)
- still need someone to step up to write the PEP and integrate the feedback from the Twisted team and the other async frameworks
- plenty of async programming folks able to help and provide feedback (including glyph)
- having this standardised would help make event loop based programming more pluggable
- Guido still doesn't like the "deferred" name
- Glyph considers deferred to be less important than standardising the basic event loop interface

Python VM Summit - Somewhat Coherent Thoughts

2011-03-10T22:38:00.005+10:00

Yay, sleep :)

Last night I just dumped my relatively raw notes into a post. This review is more about distilling what was discussed over the day into a few key themes.

Speed Good

One major point was to do with "How do we make Python fast?". Dave Mandelin (Mozilla Javascript dev) was asking how open CPython was to people tinkering with JIT and other technologies to try and speed up execution, and it was acknowledged that python-dev's reaction to such proposals is rarely more than lukewarm. A large part of that resistance comes from the fact that CPython is generally portable to many more architectures than the real speed hacks (which are generally x86 + x86-64 + ARM at best, and sometimes not even all 3 of those). Unladen Swallow also lost a lot of steam, as so much of their effort was going into tasks not directly related to "make CPython faster" (e.g. fixing LLVM upstream bugs, getting benchmarks working on new versions).

Instead, we tend to push people more towards PyPy if they're really interested in that kind of thing. Armin decided years ago (when switching his efforts from psyco to PyPy) that "we can't get there from here", and it's hard to argue with him, especially given the recent results from the benchmarks executed by speed.pypy.org.

There was definitely interest in expanding the speed.pypy.org effort to cover more versions of more interpeters. We don't actually have any solid data in CPython regarding the performance differences between 2.x and 3.x (aside from an expectation that 3.x is slower for many workloads due to the loss of optimised 32 bit integers, additional encoding/decoding overhead when working with ASCII text, the new IO stack, etc). We aren't even sure of the performance changes within the 2.x series.

That last is the most amenable to resolution in the near term - the benchmarks run by speed.pypy.org are all 2.x applications, so the creation of a speed.python.org for the 2.x series could use the benchmarks as is. Covering 3.x as well would probably be possible with a subset of the benchmarks, but others would require a major porting effort (especially the ones that depend on twisted).

Champions and specific points of contact for this idea aren't particularly obvious at this stage. Jesse is definitely a fan of the idea, but has plenty on his plate already, so it isn't clear how that will work out from a time point of view. There'll likely need to be some self-organisation from folks that are both interested in the project and aren't already devoting their Python-relates energies to something else.

The Python Software Foundation, not the CPython Software Foundation

The second major key point was the PSF (as represented by Jesse Noller from the board, and several other PSF members, including me, from multiple VMs) wanting to do more to support and promote implementations other than CPython. We are definitely at the point where all 4 big implementations are an excellent choice depending on the target environment:

CPython: the venerable granddaddy, compatible with the most C extensions and target environments, most amenable to "stripping" (i.e. cutting it down to a minimal core), likely the easiest sell in a corporate environment (due to age and historically closest ties to the PSF)
Jython: the obvious choice when using Python as a glue language for Java components, or as a scripting language embedded in a Java environment
IronPython: ditto for .NET components and applications
PyPy: now at the point where deployments on standard server and desktop environments should seriously consider it as an alternative to CPython. It's not really appropriate for embedded environments, but when sufficient resources are available to let it shine, it will run most workloads significantly faster than CPython. It even has some support for C extensions, although big ticket items like full NumPy support are still a work in progress. However, if you're talking something like a Django-based web app, then "CPython or PyPy" is now becoming a question that should be asked.

It didn't actually come up yesterday, but Stackless probably deserves a prominent mention as well, given the benefits that folks such as CCP are able to glean from the microthreading architecture.

Currently, however, python.org is still very much the CPython website. It will require a lot of work it to get to a place where the other implementations are given appropriate recognition. It also isn't clear whether or not the existing pydotorg membership will go along with a plan to modernise the website design to something that employs more modern web technologies, and better provides information on the various Python implementation and the PSF. While the current site is better than what preceded it, a lot of pydotorg members are still gun shy due to the issues in managing that last transition (even the recent migration of the development process docs over to a developer-mainted system on docs.python.org encountered some resistance). However, when the broader Python community includes some of the best web developers on the planet, we can and should do better. (A personal suggestion that I didn't think of until this morning: perhaps a way forward on this would be to first build a new site as "beta.python.org", without making a firm commitment to switch until after the results are available for all to see. It's a pretty common way for organisations to experiment with major site revamps, after all, and would also give the pydotorg folks a chance to see what they think of the back-end architecture)

Standardising the Standard Library

Finally, with the hg transition now essentially done, efforts to better consolidate development effort on the standard library (especially the pure Python sections) and the associated documentation will start to gather steam again. As a preliminary step, commit rights (now more accurately called "push rights") to the main CPython repository are again being offered to maintainers from the other major interpreter implementations so they can push fixes upstream, rather than needing to maintain them as deltas in their own repositories and/or submit patches via the CPython tracker.

Python VM Summit - Rough Notes

2011-03-10T11:28:00.002+10:00

In parallel with the the 2 days of tutorials at Pycon, there are a couple of day long meetings for invited folks active in the evolution of the language itself. Today was the VM summit, which focuses on the major Python interpreter implementations (CPython, PyPy, Jython, IronPython), the current status of each, and where things are likely to head in the near- and long-term. (The Thursday session focuses more on the evolution of the language itself, as well as of the wider ecosystem).

CPython and PyPy both had multiple devs at the summit, IronPython and Jython devs were also there (although IronPython got to share their's with CPython). We also has some Parrot VM folks there, as well as one of the Mozilla Javascript devs - a bunch of issues with VM development for dynamic languages apply across languages, despite differences in the surface syntax.

The notes below are probably too cryptic to make sense out of context, bit will hopefully give the gist of what was discussed. These notes are my interpretation of what was said, and may or may not reflect what people actually meant. Names omitted to protect the guilty (and because I didn't write them down)

Commit rights for other VM core devs
- good idea
- did some of this last Pycon US
- will look into adding more this week

Splitting out the standard library and test suite (again)
- duplication of effort between CPython/IronPython/Jython/PyPy
- shared commit rights intended to make it easier near term to use CPython as master, allowing bugs to be fixed "upstream"
- hg transition should make sharing easier
- main CPython release will stay "batteries included"
- open to the idea of providng "CPython minimal" and "standard library" downloads (but much work to be done in defining a minimum set)
- longer term, may want to separate pure-Python stdlib development from "C skills required" hacking on the CPython interpreter core and C accelerated implementation modules for the stdlib

Speed benchmarking
- speed.pypy.org (very cool!)
- benchmarks originally chosen by Unladen Swallow team
- PSF may talk to OSU OSL about setting up speed.python.org
- benchmark multiple versions of CPython, as well as Jython and IronPython
- currently benchmarks are 2.x specific, may be a while before 3.x can be compared fully
- may be GSoC projects in:
- improving backend infrastructure to handle more interpreters
- porting benchmarks to Python 3
- can highlight key performance differences between the implementations (e.g slowspitfire vs spitfire-cstringio)

Python.org download pages
- should start recommending alternative interpreters more prominently
- PyPy likely to be faster for pure Python on major platforms
- IronPython/Jython/CPython still best at integration with their respective environments (Java libraries, .NET linraries, C extensions)

Cool hacks
- Maciel: pypy JIT viewer
- Dave Malcolm: CPython HEAP viewer in GDB 7

Parrot VM (and JIT for dynamic languages)
- target VM for dynamic languages (primarily Perl 6 and Tcl at the moment)
- loadable operations, loadable object types
- dynamic ops were original speed target, now moving towards dynamic types instead
- exploring reducing number of core ops to make JIT more practical
- looking into taking advantage of LLVM
- Unladen Swallow blazed this trail, so LLVM has better dynamic language support
- PyPy has tried and failed to use LLVM as an effective backend
- some issues may have been fixed due to Unladen Swallow's efforts, but others still exist (e.g. problems with tail recursion)
- SpiderMonkey similarly struggles with JIT and dynamic patching issues
- GNU Lightning and LiveJIT projects noted, but nobody really familiar with them
- any future Python-on-Parrot efforts likely to focus on using PyPy frontend with Parrot as a backend
- proof-of-concept written (for a thesis?) that used .NET as a backend target for PyPy
- original Python-on-Parrot ran into problems due to semantic mismatches between Perl 6 and Python - reached the limits of the degree of difference the Perl 6 toolchain was willing to tolerate)

Role of the PSF
- supports Python the Language, not just CPython the Reference Interpreter
- could use additional feedback on how to better fulfill that role
- getting the "boring stuff" done?
- project-based grants, not blanket personal funding
- project proposals requiring more funds than the PSF can provide are still valuable, as PSF can help facilitate co-sponsorships (however, still a novel concept - only been done once so far).

2.7 to 3.2
- PyPy just reaching feature parity with 2.7
- PyPy now becoming far more interesting for production usage
- treat PyPy Python 3 dialect like a major Python library (e.g. sponsored by PSF)

CPython warnings for reliance on implementation details
- ResourceWarning was a nice addition (detects reliance on refcounting for resource cleanup
- non-string keys in class namespaces would be another good candidate for a warning
- clarifying finalisation-at-shutdown semantics would be nice (but fixing those semantics in CPython first would help with that)

What is a Python script?

2011-03-07T00:19:00.000+10:00

This is an adaptation of a lightning talk I gave at PyconAU 2010, after realising a lot of the people there had no idea about the way CPython's concept of what could be executed had expanded over the years since version 2.4 was released. As of Python 2.7, there are actually 4 things that the reference interpreter will accept as a main module.

Ordinary scripts: the classic main module identified by filesystem path, available for as long as Python has been around. Can be executed without naming the interpreter through the use of file associations (Windows) or shebang lines (pretty much everywhere else).

Module name: By using the -m switch, a user can tell the interpreter to locate the main module based on its position in the module hierarchy rather than by its location on the filesystem. This has been supported for top level modules since Python 2.4, and for all modules since Python 2.5 (via PEP 338). Correctly handles explicit relative imports since Python 2.6 (via PEP 366 and the __package__ attribute). The classic example of this usage is the practice of invoking "python -m timeit 'snippet'" when discussing the relative performance of various Python expressions and statements.

Valid sys.path entry: If a valid sys.path entry (e.g. the name of a directory or a zipfile) is passed as the script argument, CPython will automatically insert that location at the beginning of sys.path, then use the module name execution mechanism to look for a __main__ module with the updated sys.path. Supported since Python 2.6, this system allows quick and easy bundling of a script with its dependencies for internal distribution within a company or organisation (external distribution should still use proper packaging and installer development practices). When using zipfiles, you can even add a shebang line to the zip header or use a file association for a custom extension like .pyz and the interpreter will still process the file correctly.

Package name: If a package name is passed as the value for the -m switch, the Python interpreter will reinterpret the command as referring to a __main__ submodule within that package. This version of the feature was added in Python 2.7, after some users objected to the removal in Python 2.6 of the original (broken) code that incorrectly allowed a package's __init__.py to be executed as the main module. Starting in Python 3.2, CPython's own test suite supports this feature, allowing it to be executed as "python -m test".

The above functionality is exposed via the runpy module, as runpy.run_module() and runpy.run_path().

If anyone ever sees me (metaphorically) jumping up and down about making sure things get mentioned in the What's New document for a new Python version, this is why. Python 2.6 was released in October 2008, but we didn't get the note about the zipfile and directory execution trick into the What's New until February 2010. It is described in the documentation, but really, who reads the command line documentation, or is likely to be casually browsing the runpy docs? This post turning up on Planet Python will probably do more to get the word out about the functionality than anything we've done before now :)

Status quo wins a stalemate

2011-02-28T01:07:00.001+10:00

Sometimes language design arguments can reach a point of stalemate. The status quo is only arguably flawed, and there are also perceived flaws in any or all of the proposed alternatives. An appropriate shared design principle can help identify when this point has been reached, and let the discussion die a natural death rather than endlessly rehashing the same points without anyone changing their opinion.

Every time we (python-dev) change anything significant, no matter how positive the end result, it can create a lot of churn in the community. Books need to be rewritten, other implementations modified, advice, recipes and examples updated, questions clarified as to which version they relate to, and version compatibility issues need to be monitored closely for projects that need to cope with older execution environments.

So, before any significant changes are made, we want to be fairly certain that the gain in clarity for future Python programs is worth the inevitable near term costs as the update ripples across the Python ecosystem. Sometimes newcomers have some interesting ideas, but still fail to clear this hurdle. The simple "it's not worth the hassle" response they're likely to receive may then come across as stodgy developers rejecting an outsider's ideas without adequate consideration.

This was something that came up fairly often during the Python 3000 mailing list discussions, to the point where I posted a message explaining why the principle of "Status quo wins a stalemate" is a very practical way to avoid meaningless churn in the language design and to cut short design discussions that obviously aren't going anywhere productive.

Python 3000 was already going to have a lot of major changes (most notably, finally improving the non-ASCII text handling story, in a way that means most Python 3 libraries and applications will be more likely to get it right). We needed to ride close herd on the design discussions to try to make sure that gratuitous changes with insufficient long term benefits were avoided.

So, lambda eventually stayed and map() and filter() were retained as builtins, while the attractive nuisance that is reduce() was merely banished to the functools module rather than getting dropped entirely as was originally proposed. PEP 348 was rejected to be replaced by the far less ambitious PEP 352. str.format() was still added, but as a complement to the legacy percent formatting mechanism rather than as a wholesale replacement.

Untold numbers of ideas on the mailing lists and the tracker were dropped with "too much pain for not enough benefit" as the rationale. More recently, PEP 3003 was instituted to enforce a moratorium on core language changes for Python 3.2 in order to give the rest of the community more time to catch up to Python 2.7 and the 3.x series, even though we knew it meant delaying good ideas like the improved generator refactoring capabilities provided by PEP 380.

The fact that Python 3 migration support tools like 2to3, 3to2 and the six module work as well as they do is probably due to this principle of language design as much as it is to any other factor (not to take anything away from the fine work that has gone into implementing them, of course!).

Posting code and syntax highlighting

2011-02-27T19:43:00.001+10:00

Before publishing the previous post, I looked into recommendations for syntax highlighting in coding-oriented blogs. In a quick search, syntaxhighlighter showed up repeatedly as the preferred choice, so that's what I went with.

It looks like I'm not the only one that isn't entirely happy with that solution (although by using the "pre" tags rather than "script", my code should at least appear in the RSS feed).

Working with ReST would certainly be easier than the semi-HTML I'm currently using. Still, I think I have plenty to learn about Blogger's formatting tools before I abandon them entirely in favour of preformatted posts (which have their own drawbacks).

Justifying Python language changes

2011-02-27T19:13:00.007+10:00

A few years back, I chipped in on python-dev with a review of syntax change proposals that had made it into the language over the years. With Python 3.3 development starting and the language moratorium being lifted, I thought it would be a good time to tidy that up and republish it as a blog post.

Generally speaking, syntactic sugar (or new builtins) need to take a construct in idiomatic Python that is fairly obvious to an experienced Python user and make it obvious to even new users, or else take an idiom that is easy to get wrong when writing (or miss when reading) and make it trivial to use correctly.

Providing significant performance improvements (usually in the form of reduced memory usage or increased speed) also counts heavily in favour of new constructs.

I strongly suggest browsing through past PEPs (both accepted and rejected ones) before proposing syntax changes, but here are some examples of syntactic sugar proposals that were accepted.

List/set/dict comprehensions
(and the reduction builtins any(), all(), min(), max(), sum())

target = [op(x) for x in source]

instead of:

target = []
for x in source:
    target.append(op(x))

The transformation (`op(x)`) is far more prominent in the comprehension version, as is the fact that all the loop does is produce a new list. I include the various reduction builtins here, since they serve exactly the same purpose of taking an idiomatic looping construct and turning it into a single expression.

Generator expressions

total = sum(x*x for x in source)

instead of:

def _g(source):
    for x in source:
        yield x*x
total = sum(_g(x))

or:

total = sum([x*x for x in source])

Here, the GE version has obvious readability gains over the generator function version (as with comprehensions, it brings the operation being applied to each element front and centre instead of burying it in the middle of the code, as well as allowing reduction operations like sum() to retain their prominence), but doesn't actually improve readability significantly over the second LC-based version. The gain over the latter, of course, is that the GE based version needs a lot less memory than the LC version, and, as it consumes the source data
incrementally, can work on source iterators of arbitrary (even infinite) length, and can also cope with source iterators with large time gaps between items (e.g. reading from a socket) as each item will be returned as it becomes available (obviously, the latter two features aren't useful when used in conjunction with reduction operations like sum, but they can be helpful in other contexts).

With statements

with lock:
    # perform synchronised operations

instead of:

lock.acquire()
try:
    # perform synchronised operations
finally:
    lock.release()

This change was a gain for both readability and writability - there were plenty of ways to get this kind of code wrong (e.g. leave out the try-finally altogether, acquire the resource inside the try block instead of before it, call the wrong method or spell the variable name wrong when attempting to release the resource in the finally block), and it wasn't easy to audit because the resource acquisition and release could be separated by an arbitrary number of lines of code. By combining all of that into a single line of code at the beginning of the block, the with statement eliminated a lot of those issues, making the code much easier to write correctly in the first place, and also easier to audit for correctness later (just make sure the code is using the correct context manager for the task at hand).

Function decorators

@classmethod
def f(cls):
    # Method body

instead of:

def f(cls):
    # Method body
f = classmethod(f)

Easier to write (function name only written once instead of three times), and easier to read (decorator names up top with the function signature instead of buried after the function body). Some folks still dislike the use of the @ symbol, but compared to the drawbacks of the old approach, the dedicated function decorator syntax is a huge improvement.

Conditional expressions

x = A if C else B

instead of:

x = C and A or B

The addition of conditional expressions arguably wasn't a particularly big win for readability, but it was a big win for correctness. The and/or based workaround for the lack of a true conditional expression was not only hard to read if you weren't already familiar with the construct, but using it was also a potential source of bugs if A could ever be False while C was True (in such cases, B would be returned from the expression instead of A).

Except clause

except Exception as ex:

instead of:

except Exception, ex:

Another example of changing the syntax to reduce the potential for non-obvious bugs (in this case, except clauses like `except TypeError, AttributeError:`, that would actually never catch AttributeError, and would locally do AttributeError=TypeError if a TypeError was caught).

Bye-bye Blogilo

2011-02-15T23:29:00.000+10:00

OK, when a blogging app can't figure out my blog identity automatically and crashes every time I submit a post (but after submitting the post to blogger), 'tis clearly not the app for me.

I'm just happy the first 3 posts didn't properly include the 'python' tag either, so at least Planet Python shouldn't have been spammed with any noise.

Back to the in-browser editor for now...

To Pycon and beyond...

2011-02-15T23:21:00.002+10:00

All these Planet Python posts about interesting talks and info at Pycon finally tipped me over the edge into making the trek across the Pacific to meet some of these people I've been working with online for the past half-dozen years or so.

With 3.3 still 18-24 months away, we should be able to get a pretty good road map thrashed out for ideas we want to explore for possible inclusion. Some face-to-face discussions will be especially handy for me, given the things I'd like to see sorted out: module aliasing to clean up __main__ handling once and for all, bringing back implicit context managers now we have more collective experience with explicit ones, an alternative to PEP 377 that will allow context managers to do some additional setup inside the scope of the try block, clarifying the semantic questions raised by discrepancies between the PEP 3118 buffer API spec and its implementation.

I still have some paperwork to sort out once my renewed passport arrives, but aside from that, the trip is good to go. I did stuff my travel dates up a bit and will have a day to kill in Atlanta on the 9th, but I'm sure I'll be able to figure out something interesting to do :)

Linking sites in blog posts

2011-02-05T00:30:00.000+10:00

Call me paranoid, but the idea of trusting a blogging app with my Google account details really doesn't appeal to me. So, "BlogThis!" on the links bar it is.

It would be nice if BlogThis! popped up the full Blogger editor instead of a partial one (missing features like editing the post tags), but using it to save pre-linked drafts should be more than adequate for those occasions when I'm commenting on a link rather than writing something from scratch.

Test: editing an existing post...

Comments update and site to-do list

2011-01-27T22:50:00.001+10:00

Having used DISQUS elsewhere as a commenter and not being a great fan of the default Blogger comment system, I've configured the blog to use DISQUS instead.

I've also asked to have the blog's python related posts added to the Planet Python feed, so we'll see how that pans out.

The main thing I'm still not entirely happy with is the site colour scheme - while I'm still a fan of the whole light-text-on-dark-background style, the contrast between clicked links and the current black background really isn't significant enough. I'll probably tinker with that a bit over the next few days.

Some goals for Python 3.3

2011-01-27T22:45:00.000+10:00

With 3.2 nearly out the door, it's time to think seriously about goals for Python 3.3 and anything else I'd like to get done on the Python front this year. This post will serve as a to-do list of sorts.

PEP 1 Update
When cleaning up PEP 0 to clear out some of the accumulated cruft in the lists of Meta and Informational PEPs, I ran into a problem where the API specification PEPs use the "Final" state to indicate when consensus has been reached and the API has been locked in. This conflicts with the normal use of the Final state to indicate that a PEP is over a done with and is only being kept around for historical reasons.

A brief discussion on python-dev suggested "Consensus" as a new end state for these PEPs. I like that solution, but adopting it requires an update to PEP 1. I'd like to get to that sometime this year.

PEP 343 and 377, redux
There are a couple of rough edges on the with statement and the associated context management protocol that still bother me.

Firstly, the fact that there is no way for a context manager to skip the body of a with statement means certain constructs simply can't be factored out properly. I previously tried to address this with PEP 377, but that solution was rightly rejected as having too great an impact on common cases which didn't need the extra complexity. I have since thought of an alternative approach that is both more flexible and has a much lower impact on ordinary cases, so it has a higher chance of acceptance.

Secondly, I'd like to revisit the idea of implicit context managers. These were dropped from PEP 343 due largely to terminology problems - we weren't sure whether the term "context manager" referred to objects with enter and exit methods, or to the objects that were able to create such an object on demand. With the meaning of context manager now well established, I believe it should be possible to implement and document this in a way that makes intuitive sense, while making it significantly easier to write context managers that are both stateful and reusable.

That's my __name__, don't wear it out
As per a recent python-ideas discussion, __name__ currently serves multiple masters, which leads to conflicts in certain situations (with __name__ set to a value that is correct for some purposes, but wrong for others). This is especially prevalent with the __main__ module, but can also apply to pseudo-packages, where something is documented as a single unified namespace, but is actually implemented as multiple files combined into a package.

For Python 3.3, I'd like to have a mechanism in place to start sorting this out without breaking every Python script on the planet that relies on the "if __name__ == '__main__':" idiom.

Other PEPs (e.g. PEP 380, 393)
There are a few other PEPs that will hopefully be landing for 3.3, including the subgenerator and memory efficient string PEPs. While I probably won't take much of an active hand in implementing those, there will still be plenty of related python-dev discussion and checkins to review.

And on a completely non-code-related front... with any luck I'll be able to find myself a more directly open source focused job this year. I have the luxury of being fussy in my choice of employment though, so I can happily sit back and relax while waiting to see how that pans out :)

Which matters more to you: being right or staying friends?

2011-01-11T03:21:00.001+10:00

Browsing through the Religious Tolerance site, I felt inspired to turn my Ethos post into a Visitor Essay submission. I have no idea if they will ever post it (as it turns out, they did so quite promptly, here), but decided to put the whole thing up here (after the jump) regardless.The role of faith in my life has been a varied one. My parents are Roman Catholics (albeit ones with more than a few issues with the way the Church hierarchy run the show), and so I grew up doing the traditional Roman Catholic thing - convent primary school (although there were no teaching nuns by the time I came through) followed by a Christian Brothers high school (although again, one with mostly lay teachers). Church on Sunday was a weekly family ritual, and after drifting away into the typical agnosticism of the Australian teenager, I rediscovered my faith through the National Evangelisation Teams program. Enthused about my faith again, I started actually studying it - reading the Bible, looking at church teachings, that kind of thing. But something strange happened - the more I studied, the more hollow it all seemed. I *didn't* agree with everything the Bible said, or everything the Church taught, and neither did my parents. Clearly I was using something else as the basis for my moral decisions in which Church teachings I chose to agree with and which I chose to reject, but what was it?A Star Wars fan website (of all things [1]) led me to encounter the phrase "secular humanism". Reading up on that, I found it agreed perfectly with the criteria I had been applying in picking and choosing amongst the teachings of the Catholic Church. I was happy to have found a term for my beliefs that focused on what I believed in, rather than simply highlighting one thing I had chosen to reject (i.e. "atheism"). One cognitive science degree and a whole lot of reading later (including the likes of Dawkins, Dennett, Hofstadter, Pinker and Sagan), and I was even happier with the idea of a world where we create our own meaning rather than having it passed down to us from on high.The journey from childhood faith, through teenage agnosticism to adult faith and my eventual deliberate choice of a naturalistic world view was something that involved a lot of thought and introspection, that I was fortunate enough to have the time for. However, it left me with a problem. Many of my family are still devout (or not-so-devout) Catholics and my friends embrace a variety of religious traditions. How could I align what I had discovered for myself with their different points of view, without coming off (to myself or to others) as a condescending twit making allowances for the "less enlightened"?The militant atheists (such as Dawkins) often come across as wanting to see the religious baby thrown out with the fundamentalist bathwater. Others (such as Dennett) are able to put forward a more nuanced view, that attempts to better understand the whole of religion, so that we can more effectively harness the positive aspects while reducing the negative effects. Having been a believer as an adult, and having suffered no significant negative experiences through my faith, I have a much easier time seeing the good side of religious belief than many atheists, so Dennett's line of rapprochement seemed far more appealing to me.Eventually, I was able to distil my personal ethos down into a belief that the pillars of ethical and moral living consist of just 5 basic concepts: honesty, compassion, justice, freedom and hope. As I see it, we have to believe in something, otherwise nihilism wins and there is no point in getting out of bed in the morning. For myself, and humanists like me, I see those five things as products of evolution. I see them as glorious, beautiful things in their own right, the most profound creations of the human race. It is up to each of us to honour and steward each of the principles as best we can. Others choose to see those 5 principles as something that was gifted to us by a higher power, rather than something we created collectively ourselves over the course of time. I don't agree with such beliefs, but that is only because I see them as unnecessary. In practice, if the outcome is a personal faith that accepts and promotes the same 5 principles I believe in, then I don't really care all that much as to why somebody agrees they're important.Where I start to have an issue is when groups and individuals either consciously or inadvertently start to disrespect those ideals. Whether the offenders are governments, organised religions, corporations, or specific individuals, malicious deceit, discounting of the feelings of others, inequitable treatment, unnecessary coercive restraints and excessive pessimism and fatalism are all concepts I have fundamental problems with. Of course, balancing all of those principles means that there are many situations encountered in the real world that have no right answers, only less bad wrong ones, which is why reasonable people may disagree on the correct courses of action. Simply "getting rid of religion" doesn't magically make the underlying issues go away.The dangerous religions are the ones that create a priestly caste and then say to everyone else "It's complicated, it's a mystery, don't worry your pretty little head about it". By convincing people to shut down their capacity for critical thinking, they are trampling over a number of those principles I value (especially honesty and freedom). Religious traditions that instead encourage their believers to adopt an attitude of constant questioning are far more compatible with the humanist point of view, as it allows members of that faith to make their own judgments based on universal ethical principles in addition to the teachings of that specific tradition.This separation of beliefs from consequences (similar to the distinction made between beliefs and actions here at OCRT) and individual believers from religious organisations was the insight I needed to realise that it was possible to be religiously tolerant without feeling (or being) condescending about it.[1] For the record, it was Stardestroyer.net

Joining the modern era of blogging

2010-12-12T23:29:00.000+10:00

Wow, that template was old. OK, brushed the dust out of the corners (i.e. switched to the modern Blogger template engine and away from the old crappy one) and correctly configured the domain delegation for the www.boredomandlaziness.org address.

Once the DNS updates filter through the intarwebs, I'll reconfigure Blogger to use that address instead of the default one.

Musings on the LotR movie adaptations

2010-12-12T20:54:00.014+10:00

A friend recently asked me how I thought the major changes Peter Jackson made when adapting Lord of the Rings to the screen were received by fans of the book. Even with a few years perspective, I can only speak for myself and the fans that I know personally in saying they were well received. However, they were all also well-justified from a movie crafting perspective, so that definitely helped with the way they were perceived.

In general, I believe each of the changes was greeted with grumbling in some places, but the whole of the work was so obviously respectful of the original, and such an excellent movie for fans of the genre, that all the changes were forgiven. I'm also ignoring things that were cut from the theatrical releases, but included in the extended DVD versions (e.g. Galadriel's gifts).

Be warned, there are plenty of spoilers for both the books and the movies ahead, so if you're at all sensitive about such things, stop reading now :)

(And yes, I was able to name all these discrepancies off the top of my head... I probably missed some, though, since I forced myself not to look anything up on the web while writing it). Details after the jump...

Show a bit of urgency, people
In the book, 17 years pass between Bilbo's birthday party (Bilbo 111, Frodo 33) and the time when Gandalf tells Frodo to get out of the shire. Such a lack of urgency really wouldn't have worked for the movie, and the loss of the One Ring adequately explains Bilbo's rapid aging in the time he was at Rivendell before Frodo arrives. Most people I know thought the original time frame in the book was kind of dumb, so speeding it up for the movie met with general approval.

Removal of Old Man Willow, Tom Bombadil and the Barrows
Some people love this part of the books, many (*ahem*) see it as a relatively pointless detour that needlessly delays the arrival at Bree and the introduction of Aragorn as a character. You can get away with that kind of slow introduction in a long novel, but there was no way it was going to fit into the movie version of Fellowship. There was still plenty of tension in the journeys to Bree and Rivendell with just the Dark Riders as the threat.

Arwen stealing Glorfindel's lines
In the books, Arwen is actually portrayed as a princess type that actually doesn't do a great deal. Giving her Glorfindel's part in addition to her own gave the character more screen time and set her up as a far more suitable match for Aragorn. If they hadn't done this, I think people that weren't familiar with the books would have been (justifiably) up in arms about Aragorn choosing her over Eowyn. (Heck, even in the books Aragorn's choice isn't well justified - while Arwen is established as a generally competent lass in the Tolkien's notes, she really isn't portrayed as such in the books).

Also, it meant one less character to introduce within the time constraints of a movie (particularly since Glorfindel basically disappears from the story after Rivendell).

Aren't those separate stories?
The Two Towers actually has a fairly questionable story telling structure. The first half is the tale of what happened to Merry, Pippin, Aragorn, Legolas and Gimli, while the second is the tale of Frodo and Sam. Interleaving them was an obvious change to make for movie adaptation, and I don't think there was even any significant grumbling about that one.

Hey, let's throw Aragorn off a cliff!
Given the number of things from the books that were cut, this random addition was actually the change I found most irritating (and I'm not alone). In the books, nothing much of significance happens on the journey to Helm's Deep that I can recall.

That said, this sequence did give them a chance to practice a bit of "show, don't tell" as far as Eowyn's feelings for Aragorn went, so it made a certain amount of sense from that point of view. On the commentaries, I believe Jackson said they were concerned about the lack of threat to Aragorn for most of the movie and put this in to spice things up a bit.

Elves in Rangers' clothing
In the book, there are no elves at Helm's Deep. Instead, a bunch of Aragorn's fellow Rangers (i.e. long-lived Men of Numenor, like Aragorn) turn up. However, using the Rangers in the movie would have required setting up the Rangers as a group that existed, what their relationship was with Aragorn, and why their assistance was to be valued so much more than a similarly sized group of ordinary men. By using elves instead, Jackson was able to achieve a similar effect without trying to cram in all that additional exposition to established just who these people were.

Most people I know were happy to accept the troop of elves as the spiritual representatives of the Rangers from the books, but there was definitely some online griping about this one.

The outcome of the Entmoot
In the books, the result of the Entmoot is a decision to go destroy Isengard. After-the-fact persuasion by Merry, a curious perceptive blindness by Treebeard as to the state of the forests near Isengard and a subsequent unilateral decision to attack on seeing them in person are nowhere to be found. At this stage of the books, Merry and Pippin are still really just along for the ride - their characters arcs really don't start properly until Return of the King, and culminate in the Scouring of the Shire.

So, while I don't particularly like what Jackson, Walsh and Boyens did here (mainly because I think it disagrees with Tolkien's lore on what Ents can perceive within their own forests), I can understand the desire to give Merry and Pippin more of an active role earlier in the story, since the end of their pre-established arcs was going to be left out (more on that later).

Hobbits in Osgiliath
Another controversial one - in the books, Frodo and Sam are never in Osgiliath, and Faramir doesn't get there until some time in Return of the King. Instead, Faramir encounters the two hobbits and takes them back to his base in Ithilien. They there convince him of their purpose and he releases them to continue on their journey.

I suspect the screenwriters had two motives in making Faramir a bit more hardnosed about things and relocating the Ithilien part of the story. Firstly, it allowed the two halves of the story to be better weaved together at the end of second movie. As noted above, the book treats the two storylines as completely separate, which didn't really fit with the interleaved structure of the movie.

Secondly, it saved them the hassle of creating the Ithilien base as a location. (at least, I don't recall the movie making much, if any, use of that base).

I actually didn't mind this change, but a lot of people didn't like what they felt it implied about Faramir's character (personally, I thought making him a bit more hard-nosed than the book version wasn't a bad thing).

Giant spiders can't read calendars
The encounter with Shelob is actually part of The Two Towers. The screenwriters postponed it to Return of the King in order to balance out the screen time in the final movie a little better. This change also fit with the change to make their release by Faramir the climax of this half of the storyline in the second movie.

This was a change that made a lot of sense and I don't believe there were many complaints about it. Without it, The Two Towers ends on a serious downer with Frodo trapped in the tower, and their storyline in Return of the King consists of escaping from the tower and an awful lot of trudging across the Plains of Gorgoroth.

He's dead, Jim
Saruman didn't die at Isengard in the books. Instead, they get the palantir because Wormtongue throws it at them, and then they leave him in the tower under the guard of the Ents. This is important, since he plays a vital role in the Scouring of the Shire.

When they dropped the Scouring from the movies, they decided to make sure people knew he was quite assuredly dead by not only dropping him from a great height, but also making sure he landed on some big spikes that not only went all the way through him, but also were conveniently mounted on a wheel that could put him underwater as well. Just in case you were thinking he might have survived falling off Orthanc. I was somewhat surprised nobody carved out his heart and ate it on screen just to further make the point "No, he's really, really dead. He's NOT coming back at the end of the movie."

Pippin the pyromaniac
I believe the scene with Pippin sneaking up to light the signal fire was a new addition for the movie (although I don't recall what happened in the book instead - while Gandalf certainly argued with Denethor a lot about calling for the Rohirrim, I seem to recall Denethor grudgingly authorising the signal). I didn't mind it, since it gave Pippin more of an active role in the story, a necessary change due to the final (and likely least popular) change, which we move on to now...

The (non-)Scouring of the Shire
Wow, this one upset a lot of people. People that haven't read the books probably don't realise just how much Jackson cut from the denouement following the destruction of the ring.

It was a genuine loss that this had to go, but it really did have to go. The Scouring is a great story - it was the real payoff for Pippin and Merry's character arcs as they become the first true hobbit warriors the world has ever seen and take back the Shire from the nefarious villain that has taken control of it (who turns out, *gasp*, to be a certain prisoner that has escaped from Orthanc and wants revenge on those darn hobbits that ruined his schemes).

The other part of the Scouring is that after Merry and Peppin do the actual scouring part, the start of the rebuilding is what truly closes off Sam's story arc, as he takes the seed he received as a gift from Galadriel and plants it as a replacement for the party tree that was destroyed in their absence.

However, the problem with the Scouring from a movie makers point of view is that it is also quite a long story that takes place after the climax of the main storyline (i.e. the destruction of the ring and Sauron). Return of the King had a good dozen or so endings as it was, there was absolutely no way for them to also try to cram the Scouring in there.
In addition, the Scouring is all about the indomitable spirit of the hobbits when given a little bit of hope in the form of the mature and battle hardened Merry and Pippin and the ever-resilient Sam, despite the physical destruction and dominant presence of "Sharkey". That kind of thing is quite hard to portray on screen - an-indomitable-spirit-despite-surrounding-destruction looks an awful lot like plain old destruction to the viewer's eye.

So, they took the simpler option of leaving the Scouring out of the story altogether, with a homage to it included in Fellowship when Frodo looks into Galadriel's mirror. As noted in the relevant sections above, this also forced them to bring the development of Merry and Pippin forward a bit, and give them more to do in other parts of the story. (The bit with Merry stabbing the Witch King in the leg so Eowyn could kill him, though? Yeah, that was from the book - you know, with the whole prophecy thing and all...)

As I said above, I actually agree with their decision to drop the Scouring of the Shire. As cool a story as it is, it just didn't fit into the limitations of the movie format. Better to leave it out completely than try to jam it in where it didn't fit and do a poor job of it as result.

An Ethos

2010-12-12T17:19:00.009+10:00

(Note: I expanded this post into a more comprehensive essay)
Somewhat inevitably, Howard Tayler's review of Voyage of the Dawn Treader over on the Schlock Mercenary blog got into something of a religious debate in the comments.

My contribution to the thread was an attempt at writing down the core concepts of a personal ethos I have been formulating over the last decade or so. I quite liked how it came out, so I figured I would capture it here as well.

For mine, I see the pillars of ethical and moral living as a belief in 5 basic concepts: honesty, compassion, justice, freedom and hope. We have to believe in something, otherwise nihilism wins and there is no point in getting out of bed in the morning.

For myself, and humanists like me, I see those five things as products of evolution. I see them as glorious, beautiful things in their own right, the most profound creations of the human race. It is up to each of us to honour and steward each of the principles as best we can.

Others choose to see those 5 principles as something that was gifted to us by a higher power, rather than something we created collectively ourselves over the course of time. I don't agree with such beliefs, but that is only because I see them as unnecessary. In practice, if the outcome is a personal faith that accepts and promotes the same 5 principles I believe in, then I don't really care all that much as to why somebody agrees they're important.

Where I start to have an issue is when groups and individuals either consciously or inadvertently start to disrespect those ideals. Whether the offenders are governments, organised religions, corporations, or specific individuals, malicious deceit, discounting of the feelings of others, inequitable treatment, unnecessary coercive restraints and excessive pessimism and fatalism are all concepts I have fundamental problems with.

Of course, balancing all of those principles means that there are many situations encountered in the real world that have no right answers, only less bad wrong ones, which is why reasonable people may disagree on the correct courses of action. But "getting rid of religion" doesn't magically make the underlying issues go away.

The dangerous religions are the ones that create a priestly caste and then say to everyone else "It's complicated, it's a mystery, don't worry your pretty little head about it". By convincing people to shut down their capacity for critical thinking, they are trampling over a number of those principles I value (especially honesty and freedom). Religious traditions that instead encourage their believers to adopt an attitude of constant questioning are far more compatible with the humanist point of view.

Status quo wins a stalemate

2010-01-29T22:05:00.003+10:00

Another old post for reference. This one's a fair bit shorter than the last one and just articulates why "do nothing" needs to be the default option when considering language changes where there are no particularly strong arguments either for or against the proposed change.

Justifying Python language changes

2010-01-29T21:59:00.002+10:00

Link to a mailing list post for my own benefit. Finding this again when I want to refer to it is usually a pain, so hopefully I'll remember it is linked here in the future. Either that or I'll tidy it up and turn it into a blog post in its own right.

Kubuntu dev packages to build Python

2010-01-17T22:46:00.003+10:00

List of all the additional packages needed for a full build of the optional Python modules in the 2.x standard library on Kubuntu 9.10:

libssl-dev (OpenSSL integration)
libreadline-dev (readline module integration)
libgdbm-dev (a DBM backend)
tk-dev (also brings in TCL via dependencies)
tk-tile (extra bit of tk needed for some of the TTK GUI functionality)
libsqlite3-dev (SQLite3 integration)
libdb4.7-dev (BSDDB integration)

(at least one of those brings in the necessary zlib build dependencies as well - probably SSL dev)

Mostly for my own reference, but if it proves useful to somebody else via a search engine, cool :)

What he said

2007-06-24T10:19:00.000+10:00

Occasionally you can come across an article which summarises your own thoughts so well, you don't really need to add anything. For me, the author's note on the linked webcomic is such an article.

Ego Surfing

2006-04-08T01:49:00.000+10:00

So, one of Techdirt's latest stories led me back to this article about the perils of putting too much personal info on line. Possibly even things like having the words "boredom" and "laziness" as part of a domain name you own, and then not updating the associated website for months on end, so people googling your name can wonder about your follow-through when it comes to keeping up with projects you start*.

Anyway, as a result I did a bit of googling of my own. It used to be that Google gave me far more prominence than one might expect, with buckets of references to python-list and python-dev posts. When I noticed those posts weren't on the first list of results anymore, and links for both the actor and the surfer were now included, I thought Google might have done something to give the non-mailing list results more importance, and give a result that better reflected mainstream celebrity.

A closer look though and I find that while the list of sites has changed, my original opinion still holds: even with a lot more mainstream media moving online, your Google footprint still says far more about your online presence than your fame (or lack thereof) in the offline world. And I still manage to find that fact both obvious and intriguing at the same time :)

*(FWIW, most of the stuff I write online is Python related, and ends up on the Python mailing lists rather than here)

Lance & Hannah

2005-07-20T23:04:00.000+10:00

Lance & Hannah
Originally uploaded by Nick Coghlan.

Lance & Hannah in Japan

Grand Column

2005-07-20T22:18:00.000+10:00

This marble column from the Oregon Caves looks pretty impressive.

Grand Column
Originally uploaded by Nick Coghlan.

That is, until you see it with a person in the picture for scale ;)

Or Maybe Not So Grand Column?
Originally uploaded by Nick Coghlan.

Self-defeating Advertisements

2005-04-23T03:15:00.000+10:00

My main web browser is Mozilla Firefox. It has this lovely little extension called AdBlock, which let's me eliminate irritating content from the web pages I view. So what does that have to do with the title of the article?

Simple: A lot of the time, I do not switch AdBlock on. Normal banner ads, that generally stay out of my way, just plain don't bother me that much. I'm happy to let them appear on the screen (even if I don't click on them). As soon as an ad starts intruding on my web experience, though, out comes AdBlock, so I can say "Leave me the hell alone, you rude, pushy mongrels". Some ads get my attention without triggering that reflex (Absolut and IBM come to mind), because they don't block the main text, and are genuinely entertaining. Other ads (especially ones that interfere with real content) discover that getting too much of my attention can be a bad thing - it raises the irritation level enough to get me to take action. And, thanks to AdBlock's address wildcards, that will mean far more than just the one ad that triggered the block will be eliminated from my web surfing.

Because James doesn't remember it

2005-02-05T23:59:00.000+10:00

TMGAIHAA

Email Blogging

2004-12-24T19:54:00.000+10:00

Just checking it works. Nothing to see here. Move along.

--
Nick Coghlan | ncoghlan@email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net

Type-checking in Python

2004-12-24T19:18:00.000+10:00

So, Guido has brought up the idea of optional static typing again, posting his current throughts on the idea, as well as noting what he sees as the problem areas.

His favoured syntax is:

def (a: sometype, b: sometype) -> sometype:
  pass

Bleh. The main reason I dislike the current version of anonymous functions is because they embed a colon in the middle of an expression so you can probably guess how I feel about Guido's reuse of the colon here. And I've always quite liked VB's approach of using 'as' to indicate the return type of a function. Anyway, syntax aside, there's a question of what the optional type declarations are actually for.

Now one point to make at the start is that optional typing (checked by the VM) and optional static typing (checked by the compiler) are different things, and it makes some sense to do the former before doing the latter. Once you have a syntax for optional typing, making it static is merely a question of figuring out how to get the compiler to do the type check, instead of the VM. This activity would then blend in with Python's general issue of "how can we move things to the compiler to save run-time activity, without losing too much dynamism?"

Having dropped the static idea for the moment, there's the fundamental question of what does a type declaration mean? Python has historically relied on an approach that says "if it defines the right methods, it's OK by me". This is great for flexibility and code reuse, but plays merry hell with type inferencing systems, and can lead to some exceedingly cryptic error messages when you pass a type that doesn't provide the correct methods (or provides methods with the right names, but the wrong signatures, etc, etc).

Stealing an example I like:

def int_divide(x as Integer, y as Integer) as Integer:
  return x / y

We don't really want x as Integer (or x : Integer in Guido's syntax) to mean isinstance(x, Integer) do we? After all, we'd like this function to work for builtin types, and Python's builtin types won't know anything about this interface we have created. It would be far nicer if the optional typing was just a way of formalising the 'duck typing' that Python currently relies on.

So let's consider something like the interfaces from PJE's PEAK, or Eiffel's idea of conformance (PEP 246, basically). In this case, we have a builtin method adapt() to adapt a given object to a given protocol. I'd suggest the meaning of the example should become:

def int_divide(x, y):
  x = adapt(x, Integer)
  y = adapt(y, Integer)
  return adapt(x / y, Integer)

Objects participate in this scheme as interfaces by definining __adapt__ special methods, and as adaptable objects by defining __conform__ special methods. That way, interfaces and types can be written in any order, and still play well together. For instance, the existing 'adaptation methods' understood by the builtin objects' constructors (i.e. __int__, __str__ and friends) could be incorporated into the system by having the __adapt__ methods of the relevant interfaces invoke the appropriate constructor - if the constructor throws an exception, then the adaptor method converts it to the appropriate adaptation exception.

As mentioned in PEP 246, it would also be possible to have an 'adaptation registry' which mapped from (type, interface) tuples to adaptation methods. While this doesn't really matter to the basic idea of adaptation, it's handy for people trying to integrate code which provides the right interface, but doesn't actually provide the relevant adaptation information (e.g. if it provides a read() method, and the function uses an interface which expects that method).

For containers, it would make sense to have the interfaces be parameterisable (e.g. List(Integer), List(Number), List(int, long) or List() - that last example meaning, "allow a List with any types", since a list which allowed no types wouldn't be very useful). This suggests the concept of interface factories - classes whose instances are themselves interfaces.

For example (assume AdaptationError is a subclass of TypeError that is thrown when an adaptation fails):

class AdaptedOk(Exception): pass

class List(object):
  def __init__(self, *args):
    self._allowed_interfaces = args

  def __adapt__(self, obj):
    try:
      lst = list(obj)
    except Exception, ex:
      raise AdaptationError(str(ex))
    interfaces = self._allowed_interfaces
    if interfaces:
      for i, x in enumerate(lst):
          try:
            for interface in interfaces:
              try:
                lst[i] = adapt(x, interface, None)
                raise AdaptedOk
              except AdaptationError:
                continue
            raise AdaptationError("List element %s does not "
                   "support any allowed interface" % str(x))
          except AdaptedOk:
            pass
    return lst

  def __eq__(self, other):
    return (isinstance(other, type(self)) and
     (self._allowed_interfaces == other._allowed_interfaces))

OK, so PEP 246 combined with syntactic support would give a cleaner mechanism for dynamic type checking. However, it would still be nice to have some sort of static checking for optimisation purposes (if the compiler knows the types at compilation time, it can do all the operator lookups and so forth then, instead of waiting to do the lookups at runtime).

Well, how about a slightly different pair of special methods: __adapt_strict__ and __conform_strict__. The result of strict adaptation guarantees that the result of adaptation is an actual instance of the interface. If an interface defines __adapt_strict__ without defining __adapt__, then Python can be certain that the results of adaptation to that interface will be an instance of that interface.

For example, the builtin types might provide __adapt_strict__ methods, allowing them to be used as interfaces which guaranteed that the result was an instance of the builtin type:

  isinstance(adapt(x, int), int) # Always true
  isinstance(adapt(x, Integer), Integer) # Likely false
  isinstance(adapt(x, Integer), int) # Only possibly true

This can give us static typing, as long as the compiler can check for the existence of __adapt__ and __adapt_strict__ on the supplied interface (e.g. by assuming the names of builtins actually refer to the builtins). Here's some hypothetical implementations of __adapt_strict__ for the builtins object and list:

def object:
  # The rest of object's definition is as normal
  # Naturally this would really be implemented in C. . .
  # Use a class method so any new-style class
  # can automatically be used for strict adaptation
  @classmethod
  def __adapt_strict__(cls, obj):
    if isinstance(obj, cls):
      return obj
    try:
      result = cls(obj)
    except Exception, ex:
      raise AdaptationError(str(ex))
    return result

class AdaptedOk(Exception): pass
def list(object):
  # The rest of list's definition is as normal
  # Naturally this would really be implemented in C. . .
  # Uses an instance method, so we use self
  # to store the list of allowed interfaces
  def __adapt_strict__(self, obj):
    try:
      lst = list(obj)
    except Exception, ex:
      raise AdaptationError(str(ex))
    if self:
      for i, x in enumerate(lst):
        try:
          for interface in self:
            try:
              lst[i] = adapt(x, interface, None)
              raise AdaptedOk
            except AdaptationError:
              continue
          raise AdaptationError("List element %s does not "
                 "support any allowed interface" % str(x))
        except AdaptedOk:
          pass
    return lst

With strict adaptation available, our earlier example of non-strict list adaptation would change to be:

def List(list):
  def __adapt__(self, obj):
    return self.__adapt_strict__(obj)

The rules for adaptation would change slightly from those suggested in the PEP:

If the object is an exact instance of the interface, return it
Try the object's __conform_strict__ method, if it has one. If that works, return the result.
If the interface allows non-strict adaptation (it defines __adapt__), then try the object's __conform__ method, if it has one. If that works, return the result.
Try the interface's __adapt_strict__ method, if it has one. If that works, return the result.
Try the interface's __adapt__ method, if it has one. If that works, return the result.

The new additions are steps 2 & 4, and step 3 has been modified so that it is only tried if the interface implements __adapt__. The idea of restricting step 1 to exact instances is taken from the PEP - it allows subclasses to say "I don't implement my parent's interface" by throwing an exception in its conformation method. The check for instances of subclasses has been moved to the implementation of object.__adapt_strict__. This allows interfaces to decide how they choose to deal with subclasses.

Google Footprint

2004-12-22T21:14:00.000+10:00

OK, so here's the thing. In the real world, most Australians talking about 'Nick Coghlan' are going to be talking about the guy on Secret Life of Us. If not him, then the surfer from somewhere down south. If we went to Canada, well, there's a diplomat by that name, so at least a few people would know who he is, or would have seen an article or two about him.

Enter the world of Google, though, and an awful lot of it is about me. On the first two pages of a search for "Nick Coghlan", there's just one entry halfway down the second page for the Canadian guy. The other two don't show up until the third page. The results are skewed massively in my favour because of the public mailing list web archives that Google indexes - and every one of my messages to those lists includes my name.

You can really see the effect of this by searching for "Nicholas Coghlan" instead. I disappear from the results, and the TV actor and the Canadian diplomat take control of the show. I don't show up until page 4, on an old announcement from UQ. As you might guess from that, I don't use my full first name very often.

Another interesting search is "nick coghlan -python -cygwin -software". The results are then quite similar to the "nicholas coghlan" results.

Anyway, I guess my only real observation is that, when searching just for names, Google gives extremely high weight to participation in public online communities. Obvious, one might say, but it's an interesting limitation when looking for information on more famous people that happen to share a name with someone like me.

Brain Dead Software #1: Yast Online Update

2004-12-06T19:50:00.000+10:00

Suppose a new employee came to work for you. They're asked to collect 20 items from the company warehouse. You come back at the end of the day to find they have retrieved only one of the items. When queried, they say, "Well, the warheouse didn't have the second item on the list, so I came back to see what you wanted to do about it." "OK, fine, where are the other 18 items, then?" "Oh, I didn't try to get those. I needed to ask you about the second item, so I came back here." You'd be justifiably pissed off, and they'd probably be well on their way to getting fired.

Yast Online Update is that employee. If it encounters a problem with any of the packages you ask it to install, it sits there with a freaking dialog box on the screen doing absolutely nothing until you come back and tell it, "Look, just get on with the rest of the downloads already".

To be fair, this problem isn't specific to YOU - YOU just happens to be the most recent example I've encountered. When a computer program is given a list of tasks to do, and encounters a problem with one of them, it should look at the list and continue on with as many of the remaining tasks as it can. At the end, it can present a report detailing any problems encountered, and asking what is to be done about each of them. With Brain Dead Software like YOU, I can't just leave it to run overnight - chances are it will only do useful work for a short while before some glitch causes it to twiddle its thumbs for the rest of the night, waiting for me to wake up and reassure it that everything is fine.

Anyway, that's Brain Dead Software - programs that do things that would get a human fired. I'm sure there'll be more entries in this category.

Mt St Helens

2004-11-27T00:47:00.000+10:00

When I was in the US, one of the most impressive things I saw was the area around Mt St Helens. It was one of those things that made me glad my digital camera makes it so easy to take panoramic shots (click the panoramics to see slightly larger versions. If you want actual prints of any photos I post, let me know, since the originals have much better resolution than the versions I post).

The river valley downstream from the mountain:

The view from the interpretative centre:

Heading around the back of the ridge:

Looking back towards the next ridge:

And here we have Leung making an appearance (Leung is the mascot I took with me on my trip. He's a little big to make a great photo mascot, so there are long stretches of photos where he doesn't appear). Anyway, the real point of this photo is the effect of the lateral blast from the eruption. Most of the grey stuff on the ridgeline is actually dead trees that were knocked over by the blast. You can see a bunch of trunks still standing where they were sheltered by the ridgeline.

Python Quirk

2004-11-27T00:32:00.000+10:00

Python's a very nice language to program in, but it does have a few quirks. Normally, if you put multiple strings in your code, Python will automatically combine them into a single string.

However, it *doesn't* do this if your string literals are positioned in the "docstring" location for a class or function. If they are then only the first string will be used as the docstring - the remainder will be ignored.

Update: A quick discussion on python-dev showed that the behaviour was fairly easily explained. The string literal concatenation magic only works inside a single expression (e.g. an assignment statement). Two string literals on separate lines look like two distinct statements to the interpreter. Escaping the newline after the first literal causes the string concatenation behaviour to be applied.

Firing it up

2004-11-25T22:03:00.000+10:00

So, I occasionally post stuff over at Talkinboutstuff. However, that's mostly for random crap I expect the rest of the guys to find interesting/annoying/whatever.

So, I created this extra blog for myself - somewhere to pontificate about things I don't expect them to find interesting, like software and Python and open source and what have you. And, anything of interest to them can be cross-linked quite happily :)

Maybe I'll even get around to attaching this thing to a real domain name!