The right tool for the job
July 26, 2004 – 7:33 pmIt is easier to bang in a nail with the back of a screwdriver than with a handleless hammer (believe me, I’ve tried and it wasn’t pretty). Even though a hammer is the right tool for that job, sometimes you have to switch tools if the one that you would like to use isn’t complete.
When we began Helium a few months ago, I’d hoped that we would be able to use Python for most of the project. Python is my favourite programming language, and I felt that it would help the project with productivity (by reducing the amount of time spent struggling with syntax) and longevity (since code that is easier to read is also easier to maintain). Since then, I’ve discovered several other more important reasons why Python was the right tool for the job. So why, then, is Helium written in Java?
The answer to this question can be found on the PythonInfo Wiki under WebProgramming. In order to list all of the web frameworks in Python, you have to scroll through several screenfuls of names. This, in and of itself, is enough to make me cringe. I can’t help but feel that this goes against part of Python’s spirit. In fact, one of the postulates of the Python philosophy states that “there should be one — and preferably only one — obvious way to do [something]”. It turns out that there isn’t just one obvious way to build a web application in Python: there are several pages worth of options to choose from.
If the problem only ran as deep as having to make a choice between them, I wouldn’t mind so much; I can get a coin (or, in this case, a 70-sided die) to make that decision for me. The real problem with this kind of rampant fragmentation is that you have no guarantee that the piece you choose will still be an active project in one, two, or five years from now. Furthermore, every piece will be far less mature than if there were very few options to choose from, since the developer effort will be spread thin across so many projects.
Beyond the choice of web framework, it was no more clear which Python Object/Relational mapping tool Helium should use. Every developer you ask has a different opinion of what the best way to do this is in Python. Again, it’s not their disagreement itself which is the problem, it is what their disagreement implies: there is not a single obvious way to do O/R in Python.
For these reasons, we chose to write Helium in Java. For the first few weeks our language choice made little difference. But as Helium grew in size and complexity, we kept finding ourselves stumbling over the language, rather than having the language make our lives easier. There are currently three independent parts of Helium which had to be written in a wrapper scripting language (we, of course, chose Python) because we couldn’t get Java to do what we needed it to. We’re also heavily interfacing with Subversion, and the Python-SVN bindings are far more mature than the Java bindings. Over and over, I felt constantly reminded that Python was the right tool for this job. Unfortunately, it was the hammer without a handle.
On my next web application project, I would like to use Python. But I’m not going to feel comfortable doing so until there’s a clear “winner” or two in the web framework field. Just a few minutes ago, Jon from the Memview team asked me, “What sort of web architecture do you like to use with python?”, and I was forced to shrug helplessly. I didn’t have a good answer. His response captured my feelings exactly: “At least with php you dont have to choose”.
14 Responses to “The right tool for the job”
I was hoping you would tell me what to use. In any case as I mentioned in chat, PSP is likey to remain around because:
The embedded python market is not flooded
There is a market because of php, asp, jsp, …
By Jon on Jul 26, 2004
FWIW, from following this trend the past few months, the clear winners in the field are, at the moment: Zope, its lighter brother Quixote, Twisted with Nevow, and Webware.
Zope (and its derivative products such as Plone) is the most polished of these products. However, like Nevow with Twisted, the learning curve is more a vertical wall with few handholds. Webware is a java servlet feelalike, the project is quite slow to release new versions though, and the advances don’t seem to be revolutionary. I haven’t used Quixote yet, but I’m thinking of giving it a go soon.
By Moof on Jul 29, 2004
I use Quixote. Like it a lot. Simple and effective. I don’t use PTL though. You might also want to look at CherryPy at http://www.cherrypy.org.
By ashish on Jul 29, 2004
Are you suggesting that it’s any better in Java?
I suppose that you could argue that Struts/JSPs are the popular choice (although I suspect that in reality, Model 1 JSPs actually the most popular). Howeverm that would ignore the substantial numbers of people who use one of Barracuda, Tapestry, Turbine, WebWork or Spring MVC, let alone Echo or Millstone, JPublish, Maverick or Niggle (and they’re just the one’s I’ve used or considered)!
Personally, and without having had a chance to use them (yet?), Python seems to be the one where the choices are easier to make!
By Gwyn Evans on Jul 29, 2004
I think some of these comments are missing the point. Yes, there are some useful web app frameworks in Python (and some object/relational mapping packages, etc.). The problem is that none of them are anywhere near as easy to pick up as Tapestry (which we’re using in Helium), Struts, Hibernate, and so on. Python’s user community is much smaller than Java’s, and will be for the foreseeable future; if Python tries to have as many XYZ’s as Java, none of them will be competitive.
And before anyone says, “But they work for me,” yeah, they probably do. Now take them into a room full of developers who aren’t language geeks, and who don’t spend their free time as well as their 40 hours a week programming, and see how well they fly…
I don’t think that dividing the Python web pages into “these ones are mature” vs. “these ones are weekend throwaways” will be enough to cure this problem. I think that at some point, the community needs to say, “Category XYZ is full—if you want to work in this area, please join one of the following N projects (for some very small value of N), rather than starting your own.” That’s already happened (more or less) with Numeric and PIL, and it doesn’t seem to have stifled innovation or bloated the core libraries.
I really, really wish we’d been able to use Python instead of Java for Helium. Unless the community is willing to be a bit more directive, though, I’ll probably find myself saying the same thing about my next project…
By Greg Wilson on Jul 29, 2004
Greg, I disagree — open source communities stay away from an area when there is a clear market leader with no major problem to solve. Witness even your two counter-examples. Of those, PIL is clearly a “market leader” which does what it claims to do (which is a task that a large numbers of people can agree on) well — it has no competitors. Numeric’s scope, however, spans people who want to do various kinds of numeric work, and for a significant fraction of those Numeric isn’t the best design. Hence numarray, which is as much of a choice as Zope vs. Quixote or PSP or …
Grassroots open source communities don’t ever not do “yet another X” because it would confuse the market. I was going to say that only clear vision and leadership results in people joining rather than starting their own, but then thought of Mono, which while it’s been admirably led by Miguel, has to deal with two other OSS implementations of .Net.
Michelle — Is your point that there are two many screwdrivers or that none of them is good enough? Looking at my toolbox, those are two very different problems…
By David Ascher on Jul 29, 2004
Just to mix everything up even worse, I’d say that Zope isn’t particularly polished. There are vast undocumented or unspecified aspects. The documentation remains very poor in areas, and half of the documentation leads you in directions that are no longer considered best practice for Zope (e.g., DTML).
It’s not just choice — right now there is no framework that is the Right Framework. They all have flaws, especially when you consider the Whole Picture, including deployment and maintenance, integration with other pre-existing code, facilitating multiple roles (e.g., programmer vs. graphic designer), etc. Nothing has it all (though most of them are missing some of the same pieces).
Now, it still might be worth it to use Python, since the language itself is great. But there’s more flaws than simply an excess of choice.
By Ian Bicking on Jul 29, 2004
This makes absolutely no sense. You mean, instead of conducting an evaluation of the available frameworks, joining a few mailing lists, asking a few questions, seeing which ones had been around for awhile and what they had to offer — instead you said, oh, man, there’s just so many it’s like a handle without a hammer, let’s just use JAVA?
I think the proliferation of frameworks, rather than being an indication of fragmentation within the community, shows how powerful, productive and relatively expressive the Python language is, despite its only-one-way philosophy. A lot of people are succeeding in creating new approaches to Web development. And had you taken a look around you would have found that there are really only a few really “mature” frameworks out there — depending on what you mean by maturity. The fact is, what you’ve really said is that you never came up with a list of criteria to judge the various options against. Had you done so, then conducting a relatively quick and effective evaluation would have been straightforward, quick, and rewarding. Instead it seems that you were thrown by the fact there was no community consensus — which frequently means “popularity contest” — which is by no means, in itself, an indication of weakness or fragmentation.
You might do better next time by trusting your own ability to evaluate options, rather than waiting for the combatants to duke it out and see who’s left standing. If the Python community is lucky, there will always be many options to choose from and we’ll avoid the problem of the last man being not the best or the strongest, but the one that just flailed around the longest. We’ll call him “Java”.
By Greg McClure on Jul 29, 2004
Greg Wilson pointed me to this blog over at java.net… As a Java developer I can assure you that we have eleven hundred and seventy two MVC frameworks to choose from, and you can find some reason to be disappointed in each one.
Unlike Gred McClure, I don’t think the proliferation of frameworks is an indication of anything good, or has anything to do with a specific language. I think programmers are really picky, and we’d rather build our own then compromise. How many artists do you know who finish a painting that somebody else painted?
I like what I know about Python, I just haven’t had the pleasure of using it for anything real yet.
Do Python folks think that efforts like Groovy will make Java more palatable? Is Jython any help to you, or are the Java-ism’s just too much to stomach?
By John Reynolds on Jul 29, 2004
As someone who’s had a similar experience to the original post, my sense in the end was python (and similar languages) tend to have less of a ‘packaged product’ emphasis, and much more of a loose, flexible, and hackerly approach. This is great if you love tinkering with internals, and heavily tweaking things - not so great if you want a ‘pre-packaged’ solution to build on. Of course there are some exceptions to this, but by and large this was how it looked. I also noticed some people tend to freak out when this is brought up. Apparently it approaches a religious issue for some.
I’ve always thought a good way to strengthen web programming in python would be to poach java developers, ideally by creating an easy lily pad to jump to — basically a servlet/container-like application. While Webware is clearly in that space, it has been since long before the state of current java servlet development, and further it’s been nearly dead for quite some time. Yes, perhaps it’s just ’stable’, but most developers seem to want something a bit more alive and evolving.
There are some servlet-esque solutions built on mod_python too, but they are very low-profile. Quixote, while very cool, is unabashedly developer-focused and a bit of a hurdle for Joe Programmer, similar to Twisted/Nevow/etc.
Anyways, stepping away from java/python one web-app framework that seems to put on a good public face and have a nicely organized/polished presence is Rails [http://www.rubyonrails.org/], which is written in ruby. It’s really young, but so far looks to do a nice job of putting together a full-stack framework using common design idioms, while leveraging the strengths of a dynamic language and avoiding some complexities often found in java, like endless XML files. (also used in a public app, basecamphq.com).
A good start would be a few people who are or were using extensive java at the dayjob, saying enough is enough and building a really nice python solution using the strengths of the language while maintaining concepts from java to help pull over current java developers… or something like that. Unfortunately it’s a bit over my head to do so, but I hope it happens someday…
By TG on Jul 30, 2004
Just to throw some more gasoline on the fire (and ramble a little bit), I’d say part of the problem is Sun Microsystems (and “Java Culture”).
People like to be comfortable. Using Java often leaves you with a feeling that using anything from java.lang.* or sun.* will be okay because, well, it’s the official “thing” for the job.
It’s about being canonical.
Eventually you find out that not EVERYTHING is there, so some people then pull in a few projects from Apache and IBM. In the end, the idea is that there is an “obvious” solution that’s pretty much canon. This is pretty much an illusion, but much of the Java community is built around the ability to not lose your job by choosing part of your planform (i.e. it’s ENTERPRISE).
In terms of a canonical platform, Python does that too, to a certain extent, since it has such a huge library of built-in functionality. Indeed, that’s its biggest strength in the face of Java (and Perl for that matter, let’s just say CPAN can leave a bit to be desired).
In the end, it doesn’t include the enterprise illusion in it’s canonical set of functions. The closest thing you get to a “web framework” is some bastard mixture of asyncore and the HTTP classes.
That said, enterprise Python is there, you just have to find it (most of it’s developers are too busy to find you, they’ve got work to do).
In the end, I’d say it depends on what you want to do. Zope/Plone if you want a rapid interface developed and a flexible object storage. If you want to do your own interface and security, just use ZODB. If you need to be network-heavy, use Twisted. For Helium, I would use Plone (and therefore Zope). No question, Plone (with Archetypes, which comes standard) is what you’d need. Heck, you can even use ArchGenXML to design all of your data structures in UML and GENERATE the code! How enterprise is that? CASE! It give me goosebumps.
In this case, I think the problem is that nobody has the time to really learn the system. I find your claim of fighting with the language to be less than compelling. Having developed large projects with both Java and Python, I’d say that I find Python to take just as much time to figure out, but be far more rewarding when you’ve done it. Your team already knows Java.
It think you recognize that Python has a lot to offer. Why else would you be considering Python (as well as for your next project)? This isn’t a problem of language or options, this is a problem of constraints. No one has the time to find the right pieces for this puzzle. I would also say that any impression that Java offers less options (or more canonical options) is illusory. You know what tools to use in Java from previous experience. Put another way, you shrug about the “right” web framework for Python because you haven’t explored Python. For someone new to Java, Tapestry, Hibernate, and Struts would not be immediately obvious (or really that canonical). That said, Struts is no easier to pick up than Zope’s TAL (or better documented), Hibernate is harder to use than ZODB, and Tapestry doesn’t have anything on Zope either. The crux of it is that you have someone who can give you the answers for Java, but having someone who knows the answers for Python is just not as common right now.
On another side note, I’d also remind you that “everything looks like a nail when all you have is a hammer”. You mention O/R bindings as a concern. Don’t. If you need objects, use an object database. Squeezing objects into relations only makes you spend most of the maintenance cycle fighting with the glue code. But, then again, SQL is enterprise…
ZODB is easy to use, mature, and has ZEO for replication (and Replication from Zope Corp for a very reasonable price, if you need to run a full scale sourceforge). ZODB is fiendishly simple yet still modular enough to be useful. Here’s some code:
>>> from ZODB.FileStorage import FileStorage
>>> from ZODB import DB
>>> fs = FileStorage(’filename’)
>>> db = DB(fs)
>>> conn = db.open()
>>> root = conn.root()
or
>>> import ZODB,ZODB.FileStorage
>>> root = ZODB.DB(ZODB.FileStorage.FileStorage(’filename2′)).open().root()
With this, you get pluggable backends, versioning, undo, transactions, and about the smoothest possible integration with Python.
If that doesn’t float your boat, try Berkley XMLdb. It’s mature, as solid as BerkleyDB and makes your object serialization layer VERY language independent (if that’s a concern, since it is probably the number one reason not to use ZODB). It’s Python bindings are (supposedly) top-notch, although I’ve never needed it.
By Jayson Vantuyl on Jul 30, 2004
More gasoline!:
It seems apparent by now that there’s something fundamentally off-putting about Zope/Plone to many developers, as on paper and by boosters it sounds great, but most people don’t get far with it (myself included — and many apparently go write their own framework). I can’t believe how slow 99% of Plone sites are. Maybe the only way to get a speedy Plone site is with Squid and so forth and many sites don’t have access to that(?). Nobody seems to mention it in Plone circles, I suspect it’s been said already and is being worked on.
Zope (and related) just seem to be more of a paradigm shift than most people are prepared to undertake. No RDBMS (I know one can be used, but that’s clearly not the “intended” way), the through-the-web orientation (even though I think that’s avoidable now), and so on are just not the way most people are used to working.
And whether or not it’s as easy or easier to do lots of things compared to java, the resources for learning java and it’s many frameworks WAY outdo those available for zope (and offspring). It’s almost like having to learn J2EE with no documentation or instruction.
OK, that’s enough, this has all been said before. I’ll go write my own framework now
By TG on Jul 30, 2004
The problem isn’t just that there are so many python web-frameworks. It’s that they the are all require a hefty investment of time to get a handle on and start to use. And more so to be ready for a production environment. I’ve spent most of 2004 installing, learning and producing test apps in them. And I was never happy.
Until I found this: http://www.rubyonrails.org/
It’s not Python, it’s Ruby. Which made me resist it for a while. But it’s freaky well designed. I’m in love with it.
I’m not necessarily telling you to ditch python (though I pretty-much have) but Rails deserves a serious study from anyone who it trying to build an MVC web-app framework.
By Xian on Aug 16, 2004
Ive just spent four months considering whether to use a Python web framework or Java. As a lone developer about to undertake a rather large project Python is attractive due to productivity gains but in the end Ive decided to go with Java - a combination of Hibernate & Tapestry and possibly Jython. It comes down to resources, for example, try find good tutorials and examples for Zope3.
TG summed it up nicely:
“the resources for learning java and it’s many frameworks WAY outdo those available for zope (and offspring). It’s almost like having to learn J2EE with no documentation or instruction.”
By Nico on Apr 19, 2005