gravityboy's Journal
Checked: 1 hour 46 min ago
Updated: 8 weeks 5 days ago
Update every: 2 hours
To The Point Of Collapse - LiveJournal.com
I Didn't Think I'd Ever Be Celebrating This Victory
gravityboy's Journal - Fri, 19/09/2008 - 4:58pm
This is a wonderful day. The various non-free bits of X donated by SGI have, thanks to the efforts of those at the FSF, been relicensed under the MIT/X11 license. Congratulations to the FSF for pushing this very difficult task through. I've been told numerous times whenever I tried to take a stab at the issue that it was "tilting at windmills," but the FSF persevered and made the impossible happen. This should be evidence for the naysayers that the FSF is out there doing the really hard, and all too often thankless, work that helps keep our software Free.
This is a wonderful day. The various non-free bits of X donated by SGI have, thanks to the efforts of those at the FSF, been relicensed under the MIT/X11 license. Congratulations to the FSF for pushing this very difficult task through. I've been told numerous times whenever I tried to take a stab at the issue that it was "tilting at windmills," but the FSF persevered and made the impossible happen. This should be evidence for the naysayers that the FSF is out there doing the really hard, and all too often thankless, work that helps keep our software Free.
Goings On
gravityboy's Journal - Wed, 16/07/2008 - 3:21pm
I've been rather disconnected lately, trying to finish my PhD, find a job, etc. I got permission from my committee to start writing my thesis a few weeks ago, so I've been trying to get that in gear, as well as finishing up data for publication. This should all be done by November, if all goes well, so I can get back to spending more time on the things that I love.
I've tried to stay somewhat current with what's going on, and there's been a noticeable change over the past couple weeks in the tone of discussion around the community. I've personally been fascinated by the appearance of two things: the Linux Hater's Blog and the debate about Gtk 3.0. What's striking about both of these things is that they focus very much on the more consumer-oriented side of Linux. It's all about pleasing the independent vendors and grandma, and not about doing cool things. This is a huge shift from a few years ago. When I (and I assume many of us) got started with Free Software it wasn't really about these things, but more about getting your own work done and less about pleasing other people. Pleasing others was good of course, but it wasn't really expected. Just getting the system up and running was cool at the time, but using it exclusively for all your work? Only if you were in the right line of work!
We've come to a point where we expect a hell of a lot more though. We've got very vocal community members who want to spread Linux far and wide, and they want to do it today. And arguably, Linux is ready for it. We have good software that works rather well, can be easily installed and set up, and will run most of what people need. Yay us. On the other hand, after spending the better part of the decade using Linux on the desktop I'm finding that I agree with almost everything that the Linux Hater's Blog says. It's hard to argue with the truth, and the truth is that things are still difficult for people. I've spent the last few years trying to make X in Debian easier for people to deal with, and I've barely made a dent in just this one problem. And there's plenty more to pick and choose from. Sure, you can talk about how Windows and OSX have problems too, but we can't just be as good as them. We have to be better if we want to spread Linux and Free Software far and wide.
But do we really want to do that? Well, to be honest, I don't think it matters. No matter whether or not you care about grandma using Linux, we all want to have systems that work well and are easy to manage. Currently we have a lot of things in modern distros that could be a hell of a lot better. And many of them are directly related to fundamental assumptions we've made that don't really hold up as well as they should. They lead to lots of extra work leading to a sub-par product. We can do better and we should do better. If we have the absolute best system, world domination would be a natural side effect.
That's why I think that things like this need to stop and that we need more things like this. Sure, one is a hell of a lot harder, but no one cares if you solve an easy problem. It's the hard ones that matter, and provide the real payoff in the end. We need a better system to stop the hemorrhage of developers to OSX. When Miguel talks about how the people pushing for gtk 3.0 are all using OSX, I get very worried. If we want to be in control of our own destiny then we need to face our problems head on, and solve them.
I've been rather disconnected lately, trying to finish my PhD, find a job, etc. I got permission from my committee to start writing my thesis a few weeks ago, so I've been trying to get that in gear, as well as finishing up data for publication. This should all be done by November, if all goes well, so I can get back to spending more time on the things that I love.
I've tried to stay somewhat current with what's going on, and there's been a noticeable change over the past couple weeks in the tone of discussion around the community. I've personally been fascinated by the appearance of two things: the Linux Hater's Blog and the debate about Gtk 3.0. What's striking about both of these things is that they focus very much on the more consumer-oriented side of Linux. It's all about pleasing the independent vendors and grandma, and not about doing cool things. This is a huge shift from a few years ago. When I (and I assume many of us) got started with Free Software it wasn't really about these things, but more about getting your own work done and less about pleasing other people. Pleasing others was good of course, but it wasn't really expected. Just getting the system up and running was cool at the time, but using it exclusively for all your work? Only if you were in the right line of work!
We've come to a point where we expect a hell of a lot more though. We've got very vocal community members who want to spread Linux far and wide, and they want to do it today. And arguably, Linux is ready for it. We have good software that works rather well, can be easily installed and set up, and will run most of what people need. Yay us. On the other hand, after spending the better part of the decade using Linux on the desktop I'm finding that I agree with almost everything that the Linux Hater's Blog says. It's hard to argue with the truth, and the truth is that things are still difficult for people. I've spent the last few years trying to make X in Debian easier for people to deal with, and I've barely made a dent in just this one problem. And there's plenty more to pick and choose from. Sure, you can talk about how Windows and OSX have problems too, but we can't just be as good as them. We have to be better if we want to spread Linux and Free Software far and wide.
But do we really want to do that? Well, to be honest, I don't think it matters. No matter whether or not you care about grandma using Linux, we all want to have systems that work well and are easy to manage. Currently we have a lot of things in modern distros that could be a hell of a lot better. And many of them are directly related to fundamental assumptions we've made that don't really hold up as well as they should. They lead to lots of extra work leading to a sub-par product. We can do better and we should do better. If we have the absolute best system, world domination would be a natural side effect.
That's why I think that things like this need to stop and that we need more things like this. Sure, one is a hell of a lot harder, but no one cares if you solve an easy problem. It's the hard ones that matter, and provide the real payoff in the end. We need a better system to stop the hemorrhage of developers to OSX. When Miguel talks about how the people pushing for gtk 3.0 are all using OSX, I get very worried. If we want to be in control of our own destiny then we need to face our problems head on, and solve them.
Things To Do When You're Busy Trying To Graduate
gravityboy's Journal - Thu, 26/06/2008 - 12:35am
I've been too busy to do any serious free software work lately, but here's my "low time commitment TODO list":
I've been too busy to do any serious free software work lately, but here's my "low time commitment TODO list":
- Give xconq some serious love. That includes bugfixes, updating to the new tk, and a total repackaging with more standardized methods.
- Write more docs for X. I think we badly need a simple well integrated user's guide, followed by better high and low level internals documentation.
The Cognitive Load Of Web Development
gravityboy's Journal - Sat, 07/06/2008 - 6:55pm
The most dynamic segment of the software industry that I'm aware of is web programming. Flamewars rage about frameworks and browsers and standards and whatnot, and the openness of the whole thing feels like some of the things that made me fall in love with computing as a whole and Free Software in particular. The problem with all that openness is that it's rather difficult to navigate. Aside from the obvious issues with browser incompatibilities, there's an enormous amount of software solutions on the server side, spanning languages, libraries, and frameworks. Each one is an ecosystem to itself, often being large and complicated.
This makes it rather difficult to do certain aspects of web programming well. You are constantly switching between tools and languages, as well as coding paradigms, in order to build a web app in full. You'll be writing HTML, CSS, and Javascript on the client side, each with its own peculiarities. On the server side you'll probably have settled on one language with its own very different varieties of coding style and patterns. Some, like Ruby on Rails, are their own unique brand of language that makes you work both in the DSL (Rails in this case) and the parent language (Ruby here). Template engines often will do the same thing, such as in the case of Django.
It's pretty obvious that having to juggle all these tools places a fairly large burden on the coder. Much to their credit, it's not difficult to work in any of these languages, but to work in them at a high level becomes more difficult the deeper and wider the software stack reaches. You get this a bit in systems programming, juggling C/C++ with a build system and a scripting language, but that's usually it. You can get that number of moving parts just on the client side of the web with HTML, CSS, and Javascript.
Something interesting that seems to be happening is that people are moving to decrease this cognitive burden. One good example is the Google Web Toolkit which lets you write all your AJAXy client-side stuff in Java, and let it get compiled in to Javascript. That way you cut the languages you have to work in. A similar concept is behind Microsoft allowing Ruby (and Python, if I've heard correctly) in Silverlight. Alternately, you can bring Javascript (back) to the server side, which is the motivation behind Helma and Rhino on Rails (the latter of which I hope sees the light of day).
Personally, I'm rooting for Javascript to not become an assembly language, but to take over the server side again. It's a capable and powerful language (much better than Java in my opinion), and we're collectively leveraging it very poorly. Rhino is interesting and has enormous potential, but it needs library binds in the worst way. Currently it's basically using Javascript to write Java, which is rather atrocious (COBOL in any language, etc) but with some Javascript-style wrappers around the Java classes it could be phenomenal. Alternately, there's Spidermonkey, which could be wrapped in a more capable shell with a good FFI, and we could easily have wrappers to our favorite Free Software libraries. This project seems far along here, but it's currently very windows-centric.
There's a lot of potential here though. Unifying the development language for both web and system apps would benefit the Free Software desktop, and give us the ability to better integrate our stuff with the web. One of the best things about Free Software is that it dropped the barrier to entry in software development. I think we can repeat that success again here.
The most dynamic segment of the software industry that I'm aware of is web programming. Flamewars rage about frameworks and browsers and standards and whatnot, and the openness of the whole thing feels like some of the things that made me fall in love with computing as a whole and Free Software in particular. The problem with all that openness is that it's rather difficult to navigate. Aside from the obvious issues with browser incompatibilities, there's an enormous amount of software solutions on the server side, spanning languages, libraries, and frameworks. Each one is an ecosystem to itself, often being large and complicated.
This makes it rather difficult to do certain aspects of web programming well. You are constantly switching between tools and languages, as well as coding paradigms, in order to build a web app in full. You'll be writing HTML, CSS, and Javascript on the client side, each with its own peculiarities. On the server side you'll probably have settled on one language with its own very different varieties of coding style and patterns. Some, like Ruby on Rails, are their own unique brand of language that makes you work both in the DSL (Rails in this case) and the parent language (Ruby here). Template engines often will do the same thing, such as in the case of Django.
It's pretty obvious that having to juggle all these tools places a fairly large burden on the coder. Much to their credit, it's not difficult to work in any of these languages, but to work in them at a high level becomes more difficult the deeper and wider the software stack reaches. You get this a bit in systems programming, juggling C/C++ with a build system and a scripting language, but that's usually it. You can get that number of moving parts just on the client side of the web with HTML, CSS, and Javascript.
Something interesting that seems to be happening is that people are moving to decrease this cognitive burden. One good example is the Google Web Toolkit which lets you write all your AJAXy client-side stuff in Java, and let it get compiled in to Javascript. That way you cut the languages you have to work in. A similar concept is behind Microsoft allowing Ruby (and Python, if I've heard correctly) in Silverlight. Alternately, you can bring Javascript (back) to the server side, which is the motivation behind Helma and Rhino on Rails (the latter of which I hope sees the light of day).
Personally, I'm rooting for Javascript to not become an assembly language, but to take over the server side again. It's a capable and powerful language (much better than Java in my opinion), and we're collectively leveraging it very poorly. Rhino is interesting and has enormous potential, but it needs library binds in the worst way. Currently it's basically using Javascript to write Java, which is rather atrocious (COBOL in any language, etc) but with some Javascript-style wrappers around the Java classes it could be phenomenal. Alternately, there's Spidermonkey, which could be wrapped in a more capable shell with a good FFI, and we could easily have wrappers to our favorite Free Software libraries. This project seems far along here, but it's currently very windows-centric.
There's a lot of potential here though. Unifying the development language for both web and system apps would benefit the Free Software desktop, and give us the ability to better integrate our stuff with the web. One of the best things about Free Software is that it dropped the barrier to entry in software development. I think we can repeat that success again here.
Ruby Stuff
gravityboy's Journal - Sat, 31/05/2008 - 3:09pm
So Avi Bryant finally showed off the work he's been doing with the Gemstone folks at Railsconf, and it's made quite a splash. With performance improvements like that it shouldn't be a surprise. The most interesting thing about it to me though is that it's the first time in a very long while that we've seen a proprietary implementation of a major tool absolutely destroy all the Free implementations. We've had things like Intel's C compiler outperforming gcc before, but nothing on this level, especially because the main ruby implementation is so notoriously slow. Just another feather in the cap of Smalltalk's long legacy.
What's troubled me for some time about the post-Rails Ruby community is that it has a distinct bent away from its Free Software roots. I understand Matz actually used to use (not sure about today) Debian Unstable, and Ruby traditionally displayed its roots quite strongly, with a Perl heritage and a community consisting largely of hardcore *NIX people. With the advent of Rails, the move has been towards things like TextMate and OSX. Software like Gems (no relation to Gemstone) fits in fine with one of these systems, but not so well with modern Free Software systems, and I think it's symptomatic of the change. Given this propensity in the Ruby community, and given the numbers Gemstone is posting, I'd be surprised if lots of Rubyists don't move that way as soon as it's available.
Given all this, I really have to wonder if the modern Ruby fits me any more. I generally think it's important for the Free Software community to support itself first and then try to grow out from there, and the Ruby community isn't really on this path right now. That's fine, there's nothing wrong with it, it's just not something that really interests me. It does make me wonder how something like this could happen, and it really comes down to the fact that a lot of smart people who might otherwise be really passionate about Linux systems are choosing OSX instead. Maybe it's hardware support (I hope the modernization of X will help here) and maybe it's just the whole package being nice and easy to use. Whatever it is, people are choosing it and we're the poorer for it.
So Avi Bryant finally showed off the work he's been doing with the Gemstone folks at Railsconf, and it's made quite a splash. With performance improvements like that it shouldn't be a surprise. The most interesting thing about it to me though is that it's the first time in a very long while that we've seen a proprietary implementation of a major tool absolutely destroy all the Free implementations. We've had things like Intel's C compiler outperforming gcc before, but nothing on this level, especially because the main ruby implementation is so notoriously slow. Just another feather in the cap of Smalltalk's long legacy.
What's troubled me for some time about the post-Rails Ruby community is that it has a distinct bent away from its Free Software roots. I understand Matz actually used to use (not sure about today) Debian Unstable, and Ruby traditionally displayed its roots quite strongly, with a Perl heritage and a community consisting largely of hardcore *NIX people. With the advent of Rails, the move has been towards things like TextMate and OSX. Software like Gems (no relation to Gemstone) fits in fine with one of these systems, but not so well with modern Free Software systems, and I think it's symptomatic of the change. Given this propensity in the Ruby community, and given the numbers Gemstone is posting, I'd be surprised if lots of Rubyists don't move that way as soon as it's available.
Given all this, I really have to wonder if the modern Ruby fits me any more. I generally think it's important for the Free Software community to support itself first and then try to grow out from there, and the Ruby community isn't really on this path right now. That's fine, there's nothing wrong with it, it's just not something that really interests me. It does make me wonder how something like this could happen, and it really comes down to the fact that a lot of smart people who might otherwise be really passionate about Linux systems are choosing OSX instead. Maybe it's hardware support (I hope the modernization of X will help here) and maybe it's just the whole package being nice and easy to use. Whatever it is, people are choosing it and we're the poorer for it.
Card Games
gravityboy's Journal - Mon, 26/05/2008 - 10:17pm
A few months back, keithp introduced me to Treehouse/Icehouse as a generic system for gaming, like playing cards. Recently I realized that I didn't actually know many card games, and most of those that I did know I'd forgotten long ago. So today I spent a good chunk of the afternoon learning a few new games and playing them. John McLeod's site was the first hit I found on Google, and it's a phenomenal resource full of games from all over the world. The games that I learned or re-learned today were Casino, 500 Rum (which is a Rummy/Gin Rummy variant), and German Whist.
It's stiking how many games are generally unknown these days. We have so many other forms of entertainment available that we've collectively forgotten how to play most of these games. It's fun to be reminded what you can do with a simple deck or two of cards though.
A few months back, keithp introduced me to Treehouse/Icehouse as a generic system for gaming, like playing cards. Recently I realized that I didn't actually know many card games, and most of those that I did know I'd forgotten long ago. So today I spent a good chunk of the afternoon learning a few new games and playing them. John McLeod's site was the first hit I found on Google, and it's a phenomenal resource full of games from all over the world. The games that I learned or re-learned today were Casino, 500 Rum (which is a Rummy/Gin Rummy variant), and German Whist.
It's stiking how many games are generally unknown these days. We have so many other forms of entertainment available that we've collectively forgotten how to play most of these games. It's fun to be reminded what you can do with a simple deck or two of cards though.
Upside Down
gravityboy's Journal - Sun, 04/05/2008 - 11:55pm
Things have been strange lately. I've been taking an official indefinite vacation from Debian due to real life priorities making it impossible to participate in the project at the level that I want, which just led to frustration. Mainly, I just haven't had the time to work on X properly, so I'm leaving it to the rest of the XSF for a while. This makes me sad, but they've grown in to a fantastic team so I know they'll keep doing a killer job while I'm away.
I have had some time though, just not enough to follow two major projects like X.org and Debian, so I've been trying to leverage it in productive ways. Perhaps the most notable is that I've been spending most of my time coding Smalltalk using Squeak now that it's in Debian. Squeak has its flaws, but it's very fun to work with and Smalltalk is probably my favorite language at this point, displacing Ruby. This shouldn't be too big a surprise, since Ruby consciously inherits a lot from Smalltalk. Squeak is a good environment, but I don't feel too compelled to write desktop apps in it because they feel so displaced from the rest of the system. GNU Smalltalk will hopefully fix this in time. As a result though, I've been doing my first bit of web coding in a very long time. The main reason for this is the incredible Seaside framework built on Smalltalk. It's been a lot of fun to play with, and although it's very underdocumented, there's enough out there (especially with the new book) that I've used to get going. Right now I'm trying to learn AJAX techniques as well, which is something I never thought I'd be doing.
As a result of learning Smalltalk, combined with past experiences, I've been trying to learn emacs, and switch a lot of my text editing over there. I came to realize that the reasons I chose vim over emacs many years ago aren't really holding up any more, and now that I've got a lot more of both UNIX and coding under my belt that it's time to reevaluate the decision in a more intelligent way. Vim has been great to me over the years, but I found it constraining when trying to work on very large and difficult trees like the X server. I'm hoping that emacs will do a little better, since I like the way it handles multiple buffers better. Additionally, the purity of Smalltalk "all the way down" has made me appreciate the emacs architecture of lisp (almost) "all the way down" and I'm looking forward to making use of it. It's been a painful transition so far, given the years of muscle memory I'm trying to change. I've been avoiding viper mode to really try and learn emacs, which has made it even harder. I don't know if I'll end up using emacs after all is said and done, but at least I'll have a better idea of how the two major editors really compare for my own use.
With all these changes things have been a little strange lately. Debian was the rock that I've clung to over the past few years, and not being totally entrenched in it has felt unnerving. Combining that with very new and different ways of working has been a rather large change. One thing is for certain: it's been very good for me to take a break for a while and work on small things at my own pace rather than try to keep up with large projects. People burn out all the time in the free software community, and I think that disconnecting and working on small fun things is a great way to heal.
Things have been strange lately. I've been taking an official indefinite vacation from Debian due to real life priorities making it impossible to participate in the project at the level that I want, which just led to frustration. Mainly, I just haven't had the time to work on X properly, so I'm leaving it to the rest of the XSF for a while. This makes me sad, but they've grown in to a fantastic team so I know they'll keep doing a killer job while I'm away.
I have had some time though, just not enough to follow two major projects like X.org and Debian, so I've been trying to leverage it in productive ways. Perhaps the most notable is that I've been spending most of my time coding Smalltalk using Squeak now that it's in Debian. Squeak has its flaws, but it's very fun to work with and Smalltalk is probably my favorite language at this point, displacing Ruby. This shouldn't be too big a surprise, since Ruby consciously inherits a lot from Smalltalk. Squeak is a good environment, but I don't feel too compelled to write desktop apps in it because they feel so displaced from the rest of the system. GNU Smalltalk will hopefully fix this in time. As a result though, I've been doing my first bit of web coding in a very long time. The main reason for this is the incredible Seaside framework built on Smalltalk. It's been a lot of fun to play with, and although it's very underdocumented, there's enough out there (especially with the new book) that I've used to get going. Right now I'm trying to learn AJAX techniques as well, which is something I never thought I'd be doing.
As a result of learning Smalltalk, combined with past experiences, I've been trying to learn emacs, and switch a lot of my text editing over there. I came to realize that the reasons I chose vim over emacs many years ago aren't really holding up any more, and now that I've got a lot more of both UNIX and coding under my belt that it's time to reevaluate the decision in a more intelligent way. Vim has been great to me over the years, but I found it constraining when trying to work on very large and difficult trees like the X server. I'm hoping that emacs will do a little better, since I like the way it handles multiple buffers better. Additionally, the purity of Smalltalk "all the way down" has made me appreciate the emacs architecture of lisp (almost) "all the way down" and I'm looking forward to making use of it. It's been a painful transition so far, given the years of muscle memory I'm trying to change. I've been avoiding viper mode to really try and learn emacs, which has made it even harder. I don't know if I'll end up using emacs after all is said and done, but at least I'll have a better idea of how the two major editors really compare for my own use.
With all these changes things have been a little strange lately. Debian was the rock that I've clung to over the past few years, and not being totally entrenched in it has felt unnerving. Combining that with very new and different ways of working has been a rather large change. One thing is for certain: it's been very good for me to take a break for a while and work on small things at my own pace rather than try to keep up with large projects. People burn out all the time in the free software community, and I think that disconnecting and working on small fun things is a great way to heal.
Little Things
gravityboy's Journal - Mon, 24/03/2008 - 11:50pm
Still putting most all of my brain and time in to trying to graduate. This has left me shockingly little time for things like working on Debian.
Part of the reason why I haven't been working on more visible things (X, Debian) is that I haven't had the solid blocks of time to devote to reading email. I think I need to unsubscribe from all but two or three lists so I don't end up paralyzed by the email onslaught the way I am now. I need to get back to slinging code.
My iPod completely crapped out on me a week and a half ago. I plug it in to my computer and it's totally dead. It was only a little over a year old. Massive fail. Fuck you Apple. I've purchased a 4GB Cowon U3 which plays oggs natively and explicitly "supports" linux. We'll see how that goes, but after one day of use I'm pretty happy.
The entry of Squeak in to Debian and my own realization that GNU Smalltalk exists for those of us who might not want to live inside the smalltalk environment has prompted me to look at the language for the first time. A half hour a night (about a third of the time it takes me to read my email on a very good night) has only made me more excited about this language. If you haven't taken a serious look at Smalltalk yet, it's worth your while. It's like all sorts of things that we can approximate and dream about not only are a reality, but they were invented back in the 70's and have been here all along. It's made me seriously question some of the directions that Linux and Debian have gone in.
Still putting most all of my brain and time in to trying to graduate. This has left me shockingly little time for things like working on Debian.
Part of the reason why I haven't been working on more visible things (X, Debian) is that I haven't had the solid blocks of time to devote to reading email. I think I need to unsubscribe from all but two or three lists so I don't end up paralyzed by the email onslaught the way I am now. I need to get back to slinging code.
My iPod completely crapped out on me a week and a half ago. I plug it in to my computer and it's totally dead. It was only a little over a year old. Massive fail. Fuck you Apple. I've purchased a 4GB Cowon U3 which plays oggs natively and explicitly "supports" linux. We'll see how that goes, but after one day of use I'm pretty happy.
The entry of Squeak in to Debian and my own realization that GNU Smalltalk exists for those of us who might not want to live inside the smalltalk environment has prompted me to look at the language for the first time. A half hour a night (about a third of the time it takes me to read my email on a very good night) has only made me more excited about this language. If you haven't taken a serious look at Smalltalk yet, it's worth your while. It's like all sorts of things that we can approximate and dream about not only are a reality, but they were invented back in the 70's and have been here all along. It's made me seriously question some of the directions that Linux and Debian have gone in.
Random Bits
gravityboy's Journal - Sat, 15/03/2008 - 6:01pm
I've been extremely busy with labwork lately, trying to get a second paper ready for submission in the coming months. It's taken an obvious toll on my free software work, but that's life I guess. The upside is that the other night I realized why a year and half's worth of experiments kept turning up with negative results that seemed to contradict our other results. It turns out that this experiment was conceptually flawed, and even though I wasted a year and a half on this I did learn something even from those negative results that should go in to the final paper. I'm actually relieved, because even though I don't have all the data yet I can really see how the paper will end up, so I can start writing soon. Oftentimes knowing why something is wrong so you can fix it is even more important than being right.
I set up an ikiwiki to help organize my lab data a few weeks ago, and it's been a huge win. Joey really did a fantastic job on the program and I can't imagine using a wiki that doesn't share a very similar design for any of my personal stuff. I showed my boss some of my data using it and now he wants me to set up a wiki for the lab. He actually wants me to show off my wiki to give my lab members a feel for the idea. Since ikiwiki doesn't seem to be appropriate for my lab members, I'm going to have to look in to the giant list of wikis to figure out the exact right one for my lab's requirements. This is something I never expected to be doing in grad school.
Because there's no way my brain can go without doing a little bit of programming, I finally sat down and wrote myself a small shell script that lets me grep through pdf's, which is something I couldn't find googling around. If you have a lot of pdf's like I do and don't something like beagle or tracker churning away in the background constantly just so you can search them, this is very helpful. It's obviously very crude, but it gets the job done nicely. It relies on the pdftotext program; hopefully it'll be of use to someone.
I've been digging in to my copy of UNIX Power Tools lately, trying to pick up new UNIX and vi arcana, which is something I haven't made a real serious effort to do in a long time. It's been surprisingly fun to try and replace old habits with things that I know are better but don't use. A good example is that I usually will fire off a new subshell in vim when I should really just suspend vim and use my original shell to save time. Re-examining the basics of how I use the system has been a lot of fun, and has made me a lot more comfortable with the simple day to day tasks.
I've been extremely busy with labwork lately, trying to get a second paper ready for submission in the coming months. It's taken an obvious toll on my free software work, but that's life I guess. The upside is that the other night I realized why a year and half's worth of experiments kept turning up with negative results that seemed to contradict our other results. It turns out that this experiment was conceptually flawed, and even though I wasted a year and a half on this I did learn something even from those negative results that should go in to the final paper. I'm actually relieved, because even though I don't have all the data yet I can really see how the paper will end up, so I can start writing soon. Oftentimes knowing why something is wrong so you can fix it is even more important than being right.
I set up an ikiwiki to help organize my lab data a few weeks ago, and it's been a huge win. Joey really did a fantastic job on the program and I can't imagine using a wiki that doesn't share a very similar design for any of my personal stuff. I showed my boss some of my data using it and now he wants me to set up a wiki for the lab. He actually wants me to show off my wiki to give my lab members a feel for the idea. Since ikiwiki doesn't seem to be appropriate for my lab members, I'm going to have to look in to the giant list of wikis to figure out the exact right one for my lab's requirements. This is something I never expected to be doing in grad school.
Because there's no way my brain can go without doing a little bit of programming, I finally sat down and wrote myself a small shell script that lets me grep through pdf's, which is something I couldn't find googling around. If you have a lot of pdf's like I do and don't something like beagle or tracker churning away in the background constantly just so you can search them, this is very helpful. It's obviously very crude, but it gets the job done nicely. It relies on the pdftotext program; hopefully it'll be of use to someone.
I've been digging in to my copy of UNIX Power Tools lately, trying to pick up new UNIX and vi arcana, which is something I haven't made a real serious effort to do in a long time. It's been surprisingly fun to try and replace old habits with things that I know are better but don't use. A good example is that I usually will fire off a new subshell in vim when I should really just suspend vim and use my original shell to save time. Re-examining the basics of how I use the system has been a lot of fun, and has made me a lot more comfortable with the simple day to day tasks.
Shell Scripting
gravityboy's Journal - Sat, 01/03/2008 - 2:58pm
I've never bothered to learn the gory details of shell scripting. This is somewhat embarrassing for me, although at the same time I don't feel like I've really suffered for the decision. I know enough of the basics to do some cool things (for loops, if tests and whatnot) but the the syntax is fraught with such problems that it feels like a waste of time to do more. Obviously, others have felt similarly, which is why perl was invented.
I've become more interested in it again recently though, mainly because I've been reading a lot of Kernighan. I don't want to deal with Bourne syntax though, which eliminates the only two shells I've ever spent serious time with in my life, bash and zsh, as well as ksh. I know csh scripting is considered harmful 'n shit, so that's right out.
Since I only want to shell script for local use, I don't need to worry about portability so I can try using more exotic shells. I'm also willing to script in a different language from the shell that I use, so the doors are really open. There's scsh if I want to write scheme, zoidberg if I want to write perl, rush if I want to write ruby.
What looked most appealing is rc though, which has a very nice Bourne-like syntax that's not nearly so warty. It was reimplemented before plan9 was released, and the reimplementation is in Debian, although I ran in to some undocumented differences with the plan9 version very fast. It may be worth trying out the plan9 version at this point to see how that goes.
Any other suggestions from the lazyweb are greatly appreciated. It'd be nice to have this aspect of my toolbox be more solid.
I've never bothered to learn the gory details of shell scripting. This is somewhat embarrassing for me, although at the same time I don't feel like I've really suffered for the decision. I know enough of the basics to do some cool things (for loops, if tests and whatnot) but the the syntax is fraught with such problems that it feels like a waste of time to do more. Obviously, others have felt similarly, which is why perl was invented.
I've become more interested in it again recently though, mainly because I've been reading a lot of Kernighan. I don't want to deal with Bourne syntax though, which eliminates the only two shells I've ever spent serious time with in my life, bash and zsh, as well as ksh. I know csh scripting is considered harmful 'n shit, so that's right out.
Since I only want to shell script for local use, I don't need to worry about portability so I can try using more exotic shells. I'm also willing to script in a different language from the shell that I use, so the doors are really open. There's scsh if I want to write scheme, zoidberg if I want to write perl, rush if I want to write ruby.
What looked most appealing is rc though, which has a very nice Bourne-like syntax that's not nearly so warty. It was reimplemented before plan9 was released, and the reimplementation is in Debian, although I ran in to some undocumented differences with the plan9 version very fast. It may be worth trying out the plan9 version at this point to see how that goes.
Any other suggestions from the lazyweb are greatly appreciated. It'd be nice to have this aspect of my toolbox be more solid.
Two Fun Things
gravityboy's Journal - Fri, 22/02/2008 - 2:56am
1. After several years of it sitting on my shelf, I finally read Kernighan and Pike's "The Practice of Programming". I was always a little intimidated by this slender little book. I felt like it was full of wisdom that I couldn't really absorb at the right pace. Now that I've spent some time working on more challenging code and trying to come to grips with at the least the basics of more advanced computing concepts it was the right time to read it. I can't recommend it highly enough, it really is packed with wisdom. I've got Kernighan's "Software Tools" on the way, and I'll post a proper review of this at some point since it's not as well known.
2. I decided to spend a little bit of time poking at the almost-but-not-quite-dead language Dylan. I'd remembered being curious about this language back from when I was a serious Mac child in the mid 90's, since Apple was hinting that it would be the future. While the Gwydion Dylan implementation was removed from Debian, there's binaries for both it and the OpenDylan implementation that can be downloaded and run on both Sid and Ubuntu.
What I can grasp of the language is interesting. It's sorta like what little I know of scheme, except that it's got an object system that is apparently like CLOS from Common LISP, which I also don't know. Most notably though, is that even though Dylan is considered a LISP it doesn't look like one at all. There's no parentheses except what you'd expect, and things are infix rather than prefix, so it looks more like algol-derived languages like C. It's strikingly easy to read, and while the object system is very different from C++-style, it's fairly easy to grasp because it relies on familiar concepts like multiple dispatch to work. It's optionally typed, which is something I've been after (give me safety when I want it, but don't force me!) and provides all sorts of cool things like closures. Additionally, it does allow hygienic macros, which are apparently LISP's claim to power and fame, although I don't understand such things yet even in Dylan. Despite all this, the language appears to be relatively simple, which seems nice. It seems to be a very well designed language that was allowed to languish in obscurity because Apple abandoned it*.
Unfortunately the implementations aren't really all there yet. I ran in to obscure compiler bugs in gwydion writing something as simple as quicksort, and gwydion doesn't really have much of a standard library yet, although it can apparently bind to C quite well and fairly easily (this is something I haven't tried). In addition, there's OpenDylan, which is even less there on Linux, because it was an opened commercial compiler (derived from the original Apple one I believe) and it's had fewer people working on it. It seems to have a very large library, but documentation is sparse on it and I haven't had any time to really go after it. I've really been using gwydion exclusively, although I'd love to see OpenDylan get up to speed. While I have no time to really work on anything outside of X right now, this seems like a good language that deserves some attention.
* This is a familiar story to all of us who lived through Apple's mid-90's
1. After several years of it sitting on my shelf, I finally read Kernighan and Pike's "The Practice of Programming". I was always a little intimidated by this slender little book. I felt like it was full of wisdom that I couldn't really absorb at the right pace. Now that I've spent some time working on more challenging code and trying to come to grips with at the least the basics of more advanced computing concepts it was the right time to read it. I can't recommend it highly enough, it really is packed with wisdom. I've got Kernighan's "Software Tools" on the way, and I'll post a proper review of this at some point since it's not as well known.
2. I decided to spend a little bit of time poking at the almost-but-not-quite-dead language Dylan. I'd remembered being curious about this language back from when I was a serious Mac child in the mid 90's, since Apple was hinting that it would be the future. While the Gwydion Dylan implementation was removed from Debian, there's binaries for both it and the OpenDylan implementation that can be downloaded and run on both Sid and Ubuntu.
What I can grasp of the language is interesting. It's sorta like what little I know of scheme, except that it's got an object system that is apparently like CLOS from Common LISP, which I also don't know. Most notably though, is that even though Dylan is considered a LISP it doesn't look like one at all. There's no parentheses except what you'd expect, and things are infix rather than prefix, so it looks more like algol-derived languages like C. It's strikingly easy to read, and while the object system is very different from C++-style, it's fairly easy to grasp because it relies on familiar concepts like multiple dispatch to work. It's optionally typed, which is something I've been after (give me safety when I want it, but don't force me!) and provides all sorts of cool things like closures. Additionally, it does allow hygienic macros, which are apparently LISP's claim to power and fame, although I don't understand such things yet even in Dylan. Despite all this, the language appears to be relatively simple, which seems nice. It seems to be a very well designed language that was allowed to languish in obscurity because Apple abandoned it*.
Unfortunately the implementations aren't really all there yet. I ran in to obscure compiler bugs in gwydion writing something as simple as quicksort, and gwydion doesn't really have much of a standard library yet, although it can apparently bind to C quite well and fairly easily (this is something I haven't tried). In addition, there's OpenDylan, which is even less there on Linux, because it was an opened commercial compiler (derived from the original Apple one I believe) and it's had fewer people working on it. It seems to have a very large library, but documentation is sparse on it and I haven't had any time to really go after it. I've really been using gwydion exclusively, although I'd love to see OpenDylan get up to speed. While I have no time to really work on anything outside of X right now, this seems like a good language that deserves some attention.
* This is a familiar story to all of us who lived through Apple's mid-90's
Where I've Been, Where I'm Going
gravityboy's Journal - Fri, 22/02/2008 - 2:36am
After about two weeks I'm finally able to come back to Debian, which feels good. After my last package upload was botched (and gracefully recovered by my teammates) I realized that I needed to step away for a little bit.
I'm currently in my fifth year of my PhD program. For those who don't know how such things work in the US, I have a small committee of people (my thesis committee) who act as advisors and gatekeepers on my work. They're the ones who gauge my progress, and ultimately give me permission to write my thesis and receive my doctorate. I'm supposed to meet with them every 6 months, and it had been over a year since my last meeting. Because I'd like to actually graduate one day, I was rather nervous about having the meeting. I was more nervous than I realized, and the botched upload reflected that.
Anyway, to make a long story short, the meeting went as well as I could have hoped for and my committee seems to be happy with my progress. I'm looking to get a paper out in the spring, and they'd like to meet with me again in three months time (which will absolutely fly by) to assess where that paper is at, and how things are going. At that point, I wouldn't be surprised if they tell me to write my thesis and graduate.
What I do after my dissertation work has been weighing on my mind for years and years, and it's obviously coming to a head. My boss will let me stay on as a post-doc in the lab for a while, which I may do depending on other job opportunities and progress on my projects. Ultimately though I need to make a decision that's familiar to a few others in my position. Do I stay in science or go try and work on free software for a living? I honestly don't have an answer yet, because I love them both.
I see tons of incredible opportunities in my field right now, and while I'm not the only coder interested in synthetic biology, I'm one who's positioned to get in on the ground floor of the science in a very real way and have a significant hand in shaping its future. On the other hand I spend my free time devouring computing literature, not biology literature, as well as producing software and things related to it. I do this because I enjoy it, and I know I enjoy it on a deeper level than the biology. Would I enjoy it this way if I did it for a living? I've got no idea, since I've never coded professionally.
I'm coming up to the fork quite fast now, and I'll have to choose soon. I know I'm lucky because I do have the chance to choose for myself, but that doesn't make it any easier.
After about two weeks I'm finally able to come back to Debian, which feels good. After my last package upload was botched (and gracefully recovered by my teammates) I realized that I needed to step away for a little bit.
I'm currently in my fifth year of my PhD program. For those who don't know how such things work in the US, I have a small committee of people (my thesis committee) who act as advisors and gatekeepers on my work. They're the ones who gauge my progress, and ultimately give me permission to write my thesis and receive my doctorate. I'm supposed to meet with them every 6 months, and it had been over a year since my last meeting. Because I'd like to actually graduate one day, I was rather nervous about having the meeting. I was more nervous than I realized, and the botched upload reflected that.
Anyway, to make a long story short, the meeting went as well as I could have hoped for and my committee seems to be happy with my progress. I'm looking to get a paper out in the spring, and they'd like to meet with me again in three months time (which will absolutely fly by) to assess where that paper is at, and how things are going. At that point, I wouldn't be surprised if they tell me to write my thesis and graduate.
What I do after my dissertation work has been weighing on my mind for years and years, and it's obviously coming to a head. My boss will let me stay on as a post-doc in the lab for a while, which I may do depending on other job opportunities and progress on my projects. Ultimately though I need to make a decision that's familiar to a few others in my position. Do I stay in science or go try and work on free software for a living? I honestly don't have an answer yet, because I love them both.
I see tons of incredible opportunities in my field right now, and while I'm not the only coder interested in synthetic biology, I'm one who's positioned to get in on the ground floor of the science in a very real way and have a significant hand in shaping its future. On the other hand I spend my free time devouring computing literature, not biology literature, as well as producing software and things related to it. I do this because I enjoy it, and I know I enjoy it on a deeper level than the biology. Would I enjoy it this way if I did it for a living? I've got no idea, since I've never coded professionally.
I'm coming up to the fork quite fast now, and I'll have to choose soon. I know I'm lucky because I do have the chance to choose for myself, but that doesn't make it any easier.
On Visualizing Biological Data
gravityboy's Journal - Sun, 20/01/2008 - 1:24am
The following is a brain dump of some of the things I've been thinking about lately.
One of the biggest changes over the past several years in biology has been the incredible deluge of information. In response to this there's been a rise in bioinformatics to cope with all this. While this has led to some major successes, where it's failed is in its ability to impart a greater understanding of the subject at hand. Biologists still learn primarily from reading papers, the same way we always have. There are massive databases full of wonderful information but most of it is encoded with minimal or no context so you're always forced to go back to the papers to understand what the database is actually telling you. In that sense, these databases are fantastic at indexing information, but very poor at organizing it in such a way as to teach people about the topic at hand. We're still forced to slog through papers for just about everything.
What's striking about this is that the most informative bits in any biological paper I've ever read are encapsulated in the figures. The images themselves, provided you have sufficient background knowledge, show the basic data and give you the most understanding for the smallest investment of time. You can skim an article's abstract, it's figures and figure legends and gain a fair understanding of the topic before deciding whether or not to go further.
Now, there's a contradiction of sorts here. Many of these figures are generated via computers, usually an excel-made graph. The rest are actual photographs of things, such as blots, gels, or stained tissues, eventually inserted and processed via the computer. The contradiction is that the computer is used intensively to organize this data for publication, but we have a hard time extracting the essence of that visualization for indexing in "big picture" sorts of ways. That almost always has to be done by hand (and brain) by the biologist. This, of course, is suboptimal when you have thousands of individual genes.
The fundamental reason for all of this is that biological information depends wholly on context. For example, you can sequence the whole genome, but it's totally unclear what genes will be expressed at any given time unless you have much more information about the cell type, developmental stage, pathologies, and so forth. As far as I can see, all our bioinformatics tools have failed completely at providing any sort of context for their information. A common thing to see is so-called "wiring diagrams" that display molecular interactions. These diagrams are full of nodes and edges and it looks almost impossible to understand such things. While there is a great deal of complexity, contextual information provides us with a framework to understand what's actually working. Looking at these diagrams, there's no sense of this though, it looks instead like complexity overrun.
So that presents us with the challenge for the future. What's required of bioinformatics is to not only index the raw data, but also the context, and then present the data to us in a context-dependent manner. I am convinced that the key to presenting data this way is to come up with novel visualization methods because it's the visualizations in the papers that we use today to get the most out of our time. I believe that this problem is tractable and that there is a solution. More than likely we'll need several solutions, and I look forward to seeing them develop.
The following is a brain dump of some of the things I've been thinking about lately.
One of the biggest changes over the past several years in biology has been the incredible deluge of information. In response to this there's been a rise in bioinformatics to cope with all this. While this has led to some major successes, where it's failed is in its ability to impart a greater understanding of the subject at hand. Biologists still learn primarily from reading papers, the same way we always have. There are massive databases full of wonderful information but most of it is encoded with minimal or no context so you're always forced to go back to the papers to understand what the database is actually telling you. In that sense, these databases are fantastic at indexing information, but very poor at organizing it in such a way as to teach people about the topic at hand. We're still forced to slog through papers for just about everything.
What's striking about this is that the most informative bits in any biological paper I've ever read are encapsulated in the figures. The images themselves, provided you have sufficient background knowledge, show the basic data and give you the most understanding for the smallest investment of time. You can skim an article's abstract, it's figures and figure legends and gain a fair understanding of the topic before deciding whether or not to go further.
Now, there's a contradiction of sorts here. Many of these figures are generated via computers, usually an excel-made graph. The rest are actual photographs of things, such as blots, gels, or stained tissues, eventually inserted and processed via the computer. The contradiction is that the computer is used intensively to organize this data for publication, but we have a hard time extracting the essence of that visualization for indexing in "big picture" sorts of ways. That almost always has to be done by hand (and brain) by the biologist. This, of course, is suboptimal when you have thousands of individual genes.
The fundamental reason for all of this is that biological information depends wholly on context. For example, you can sequence the whole genome, but it's totally unclear what genes will be expressed at any given time unless you have much more information about the cell type, developmental stage, pathologies, and so forth. As far as I can see, all our bioinformatics tools have failed completely at providing any sort of context for their information. A common thing to see is so-called "wiring diagrams" that display molecular interactions. These diagrams are full of nodes and edges and it looks almost impossible to understand such things. While there is a great deal of complexity, contextual information provides us with a framework to understand what's actually working. Looking at these diagrams, there's no sense of this though, it looks instead like complexity overrun.
So that presents us with the challenge for the future. What's required of bioinformatics is to not only index the raw data, but also the context, and then present the data to us in a context-dependent manner. I am convinced that the key to presenting data this way is to come up with novel visualization methods because it's the visualizations in the papers that we use today to get the most out of our time. I believe that this problem is tractable and that there is a solution. More than likely we'll need several solutions, and I look forward to seeing them develop.
Maths Break
gravityboy's Journal - Thu, 17/01/2008 - 12:10am
I've been mentoring a high school student in biology for the past few weeks. The other day she came to me with a math problem that she was having trouble with. Looking at this relatively simple logic problem I realized that when faced with such a thing these days that I'd rather write a program to solve it because that's way more fun. So later on I wrote a little python script to calculate all the possible values that were correct for this program up to any given value. I'm not sure if the realization that I'd rather use a computer to solve even simple problems is a good or bad thing, but I bet I would have liked math class a lot more if this is what I had to do for it.
I've been mentoring a high school student in biology for the past few weeks. The other day she came to me with a math problem that she was having trouble with. Looking at this relatively simple logic problem I realized that when faced with such a thing these days that I'd rather write a program to solve it because that's way more fun. So later on I wrote a little python script to calculate all the possible values that were correct for this program up to any given value. I'm not sure if the realization that I'd rather use a computer to solve even simple problems is a good or bad thing, but I bet I would have liked math class a lot more if this is what I had to do for it.
Finally
gravityboy's Journal - Tue, 08/01/2008 - 4:38am
A few years back I asked Branden Robinson for access to the X Strike Force SVN repository in order to improve the X server's autodetection, with the goal of stealing what I could from knoppix. At the time, users were constantly wandering in to #debian saying that they had used knoppix to create an XF86Config-4 and then they used that file on their Debian installations. They were also constantly whining about how Debian wasn't doing as good a job. So I decided to do something about it, and took a look at what knoppix was actually doing so much better than us, and I was surprised to find that they pretty much just wrote a skeleton configuration file and let the X server fill in the details. I had no idea the server was capable of such things. So Branden graciously gave me SVN access and I began comparing the knoppix method to the script called by "dpkg-reconfigure xserver-xfree86" and realized that we probably couldn't adopt the same method because we had tons of checks for portability. So I put the problem on the backburner for few weeks and worked on something else for a while.
Well, somewhere in between then and now I got sucked in to transitioning Debian to X.org (using the aforementioned SVN access) and then working on all the things that went along with maintaining X in Debian, some well and others less so. Between transitioning to Xorg, and then transitioning to a modular Xorg, even with the ability to steal Ubuntu's packages it still took about two full years with me being a rookie and all that. There were tons of people who worked on this with me, but it was just a damn big job. Eventually though, etch released with modular X packages, and we were running at a pretty good pace with upstream, so it was time to revisit the problem of configuration again after the two year detour.
The problem of configuration had been reframed for me by having looked at what knoppix was doing and discussions with upstream. Upstream was starting to come to the conclusion that we shouldn't have a config file at all, and that the server should be smart enough to do everything by itself. It was already partially there, it just needed a push in the right direction. This was a major shift in how I thought of it. Coming from a Debian background, where the answer is to always just regenerate or edit your config file, having the server work things out for you was a totally alien idea. But I have my deepest roots in the Macintosh world, so immediately fell in love with it. The problem was that the server had a whole body of code to use if you had no config file at all, and another, with far less automagic goodness, if you had a config file, even if it was a 0 byte file. The goal became to have the server work really well with a minimal config file, so you could override what you don't like in the defaults and let the server figure things out for itself at boot. The way forward was to translate as much of the logic in our configure script in to the X server itself as possible.
Ubuntu had put in a lot of work in to the configuration setup that we automatically benefitted from, so given that most users were happy with things as they were, I was able to carve away at the problem without disrupting anything. And there was a lot to carve out, as the script that runs the config is an absolute mess that was slated by Branden for a rewrite all those years ago. Early on I picked off the low hanging fruits like the font path and modules. Similarly, Redhat's Adam Jackson had also been working on this problem, and he killed off the ServerLayout section as well as putting in lots of critical fixes all over the place elsewhere. More recently, I've gone and cut explicit modesetting out of the configure script. Hardcoding this information is generally a bad idea in the randr 1.2 world, and most of the drivers will do as good or better job of figuring out the modes than our config script would. This let us jettison using the xresprobe program to ask the monitor for the settings to use. This cut out a lot of the code that we had to deal with, simplifying the configure script and letting us all benefit from upstream's work. This leaves a big gaping hole in user configuration, which is something I'm looking to address in a few weeks, but for now there are workarounds like editing your xorg.conf manually to make things work.
Finally, yesterday I was able to upload a version of the script that no longer uses the discover program to figure out what driver to load. I've patched the server to do this at runtime. If there's no driver listed it simply scans the PCI bus, picks out your primary video card, and loads the first driver that claims to support that PCI ID. This let us jettison the last external dependency that the configure script had, so now we have a relatively small chunk of shell script with no external C code and a simplified setup. At this point we're now shipping a more skeletal config file than knoppix ever did simply because it wasn't possible to ship something like this before and have the server work at all. This is a huge milestone for me because finally, after over three years, the problem I originally set out to work on with X is done. There's still bugs to uncover and fix with all of this, but I'm convinced that what we have now is superior to our old method. Eventually, with any luck, xorg.conf will just fade away for most people. There's a lot to work on before that happens though, but I'm happy to have finally gotten here.
A few years back I asked Branden Robinson for access to the X Strike Force SVN repository in order to improve the X server's autodetection, with the goal of stealing what I could from knoppix. At the time, users were constantly wandering in to #debian saying that they had used knoppix to create an XF86Config-4 and then they used that file on their Debian installations. They were also constantly whining about how Debian wasn't doing as good a job. So I decided to do something about it, and took a look at what knoppix was actually doing so much better than us, and I was surprised to find that they pretty much just wrote a skeleton configuration file and let the X server fill in the details. I had no idea the server was capable of such things. So Branden graciously gave me SVN access and I began comparing the knoppix method to the script called by "dpkg-reconfigure xserver-xfree86" and realized that we probably couldn't adopt the same method because we had tons of checks for portability. So I put the problem on the backburner for few weeks and worked on something else for a while.
Well, somewhere in between then and now I got sucked in to transitioning Debian to X.org (using the aforementioned SVN access) and then working on all the things that went along with maintaining X in Debian, some well and others less so. Between transitioning to Xorg, and then transitioning to a modular Xorg, even with the ability to steal Ubuntu's packages it still took about two full years with me being a rookie and all that. There were tons of people who worked on this with me, but it was just a damn big job. Eventually though, etch released with modular X packages, and we were running at a pretty good pace with upstream, so it was time to revisit the problem of configuration again after the two year detour.
The problem of configuration had been reframed for me by having looked at what knoppix was doing and discussions with upstream. Upstream was starting to come to the conclusion that we shouldn't have a config file at all, and that the server should be smart enough to do everything by itself. It was already partially there, it just needed a push in the right direction. This was a major shift in how I thought of it. Coming from a Debian background, where the answer is to always just regenerate or edit your config file, having the server work things out for you was a totally alien idea. But I have my deepest roots in the Macintosh world, so immediately fell in love with it. The problem was that the server had a whole body of code to use if you had no config file at all, and another, with far less automagic goodness, if you had a config file, even if it was a 0 byte file. The goal became to have the server work really well with a minimal config file, so you could override what you don't like in the defaults and let the server figure things out for itself at boot. The way forward was to translate as much of the logic in our configure script in to the X server itself as possible.
Ubuntu had put in a lot of work in to the configuration setup that we automatically benefitted from, so given that most users were happy with things as they were, I was able to carve away at the problem without disrupting anything. And there was a lot to carve out, as the script that runs the config is an absolute mess that was slated by Branden for a rewrite all those years ago. Early on I picked off the low hanging fruits like the font path and modules. Similarly, Redhat's Adam Jackson had also been working on this problem, and he killed off the ServerLayout section as well as putting in lots of critical fixes all over the place elsewhere. More recently, I've gone and cut explicit modesetting out of the configure script. Hardcoding this information is generally a bad idea in the randr 1.2 world, and most of the drivers will do as good or better job of figuring out the modes than our config script would. This let us jettison using the xresprobe program to ask the monitor for the settings to use. This cut out a lot of the code that we had to deal with, simplifying the configure script and letting us all benefit from upstream's work. This leaves a big gaping hole in user configuration, which is something I'm looking to address in a few weeks, but for now there are workarounds like editing your xorg.conf manually to make things work.
Finally, yesterday I was able to upload a version of the script that no longer uses the discover program to figure out what driver to load. I've patched the server to do this at runtime. If there's no driver listed it simply scans the PCI bus, picks out your primary video card, and loads the first driver that claims to support that PCI ID. This let us jettison the last external dependency that the configure script had, so now we have a relatively small chunk of shell script with no external C code and a simplified setup. At this point we're now shipping a more skeletal config file than knoppix ever did simply because it wasn't possible to ship something like this before and have the server work at all. This is a huge milestone for me because finally, after over three years, the problem I originally set out to work on with X is done. There's still bugs to uncover and fix with all of this, but I'm convinced that what we have now is superior to our old method. Eventually, with any luck, xorg.conf will just fade away for most people. There's a lot to work on before that happens though, but I'm happy to have finally gotten here.
First In A Totally Disconnected Series of Posts
gravityboy's Journal - Sun, 06/01/2008 - 4:56pm
I want to echo what Brice has said so well already. X is full of interesting problems and important things to work on. When you consider the free software that's out there, very little of it comes close to the importance of X in your day to day life. When we've done our job well, you don't even think about it because it just works, but when X fails it's like a minor catastrophe. Working on the kernel is largely considered to be something glorious and grand, but people tend to ignore X even though it plays very much in the same domain as the kernel. So working on X you not only get to play with interesting problems that are similar to the ones that the kernel folks get to play with, but you also get to work on software that's critically important and makes a significant difference in people's day to day lives.
Furthermore, X.org is a great organization to work in. The number of core contributers is very few (maybe 20 all told) so everyone knows who everyone else is and people are willing to help others out by answering questions about how things work. Reflecting this, the XSF has turned in to a fantastic team over the past few years, and it's one I couldn't be prouder to be a part of. Julien and Brice are doing insanely fantastic work, and we're constantly pushing to do a better job at just about everything, from collaborating with Ubuntu to using the best cutting edge tools out there for our work.
Despite all this though, as Brice said, we're overwhelmed by everything. We're still carrying hundreds of bugs in the Debian BTS alone, most of which we'll never have a chance to look at again. As mentioned elsewhere on planet upstream is totally swamped as well. There's just too many critical projects that need doing (new drivers, fixed drivers, adding capabilities to the server, and that's ignoring day to day needs like documentation and patch review) and too few people to do them effectively. We have a wonderful set of things to do and a great group of people to mentor motivated people to do them, and yet we're still lacking contributers to this critically important set of software.
So as Brice said, if you want to get involved in something that's essentially important to Linux and Free Software, if you want to get involved in something with great problems and really fantastic opportunities to make a difference, you can't do much better than getting involved in X. Brice's method of getting involved in processing bug reports is a perfect way to start. Another is to write much needed documentation. Another is to simply help the XSF maintain some small piece of the Xorg stack, be it a driver for some hardware that you own (the XSF only really runs two or three video drivers collectively so we badly need driver maintainers), or just start poking at something you're curious but mystified about like how mesa or the X server work. There's no magic to any of it, and while it can get complicated I promise that you will become a better coder or maintainer by working on something as challenging as X. You don't have to understand it all from the start, just picking a small thing to work on will be greatly appreciated. So drop us a note on the debian-x list or drop in to #debian-x on freenode and we'll help you get going.
I want to echo what Brice has said so well already. X is full of interesting problems and important things to work on. When you consider the free software that's out there, very little of it comes close to the importance of X in your day to day life. When we've done our job well, you don't even think about it because it just works, but when X fails it's like a minor catastrophe. Working on the kernel is largely considered to be something glorious and grand, but people tend to ignore X even though it plays very much in the same domain as the kernel. So working on X you not only get to play with interesting problems that are similar to the ones that the kernel folks get to play with, but you also get to work on software that's critically important and makes a significant difference in people's day to day lives.
Furthermore, X.org is a great organization to work in. The number of core contributers is very few (maybe 20 all told) so everyone knows who everyone else is and people are willing to help others out by answering questions about how things work. Reflecting this, the XSF has turned in to a fantastic team over the past few years, and it's one I couldn't be prouder to be a part of. Julien and Brice are doing insanely fantastic work, and we're constantly pushing to do a better job at just about everything, from collaborating with Ubuntu to using the best cutting edge tools out there for our work.
Despite all this though, as Brice said, we're overwhelmed by everything. We're still carrying hundreds of bugs in the Debian BTS alone, most of which we'll never have a chance to look at again. As mentioned elsewhere on planet upstream is totally swamped as well. There's just too many critical projects that need doing (new drivers, fixed drivers, adding capabilities to the server, and that's ignoring day to day needs like documentation and patch review) and too few people to do them effectively. We have a wonderful set of things to do and a great group of people to mentor motivated people to do them, and yet we're still lacking contributers to this critically important set of software.
So as Brice said, if you want to get involved in something that's essentially important to Linux and Free Software, if you want to get involved in something with great problems and really fantastic opportunities to make a difference, you can't do much better than getting involved in X. Brice's method of getting involved in processing bug reports is a perfect way to start. Another is to write much needed documentation. Another is to simply help the XSF maintain some small piece of the Xorg stack, be it a driver for some hardware that you own (the XSF only really runs two or three video drivers collectively so we badly need driver maintainers), or just start poking at something you're curious but mystified about like how mesa or the X server work. There's no magic to any of it, and while it can get complicated I promise that you will become a better coder or maintainer by working on something as challenging as X. You don't have to understand it all from the start, just picking a small thing to work on will be greatly appreciated. So drop us a note on the debian-x list or drop in to #debian-x on freenode and we'll help you get going.
Processing
gravityboy's Journal - Sat, 29/12/2007 - 11:30pm
I've been taking a little bit of time away from Debian and X to see what sort of fun stuff is out there that I've been putting off learning. Aside from finding what looks to be a good fitness training plan for the next year (and hopefully beyond), I've been looking at different programming languages. One that I found most recently is Processing.
This program is absurd amounts of fun to play with. I've long been interested in how to do display complicated data sets visually, and processing is really built to allow you to do that relatively easily. More and more often in my work lately I've been generating fairly large data sets, and I'm not really satisfied with the standard graphs that come with a spreadsheet. They work for basic data, but they completely fall down when you need something bigger. You can either add more graphs or just have a large table, both of which are suboptimal. Processing is centered around the concept of sketches, quickly where you quickly type some java code in to the IDE, hit the play button, and have it run. The API is very simple and should be intuitive to anyone who's done any coding before. This lets you play around with how to visualize your data at very low cost.
The software is GPL'ed, and while it relies on java, this shouldn't really be a problem in the near future. Unfortunately, java is java, and to be honest I'd rather be writing something like python or ruby. Options for those seem to be forthcoming (nodebox is apparently being ported to linux and scribble is out there in some form now) but processing has an impressive and easy setup as it is. I'll definitely be picking up one of the new books when it's available.
I've been taking a little bit of time away from Debian and X to see what sort of fun stuff is out there that I've been putting off learning. Aside from finding what looks to be a good fitness training plan for the next year (and hopefully beyond), I've been looking at different programming languages. One that I found most recently is Processing.
This program is absurd amounts of fun to play with. I've long been interested in how to do display complicated data sets visually, and processing is really built to allow you to do that relatively easily. More and more often in my work lately I've been generating fairly large data sets, and I'm not really satisfied with the standard graphs that come with a spreadsheet. They work for basic data, but they completely fall down when you need something bigger. You can either add more graphs or just have a large table, both of which are suboptimal. Processing is centered around the concept of sketches, quickly where you quickly type some java code in to the IDE, hit the play button, and have it run. The API is very simple and should be intuitive to anyone who's done any coding before. This lets you play around with how to visualize your data at very low cost.
The software is GPL'ed, and while it relies on java, this shouldn't really be a problem in the near future. Unfortunately, java is java, and to be honest I'd rather be writing something like python or ruby. Options for those seem to be forthcoming (nodebox is apparently being ported to linux and scribble is out there in some form now) but processing has an impressive and easy setup as it is. I'll definitely be picking up one of the new books when it's available.
PlzKThx
gravityboy's Journal - Sat, 01/12/2007 - 6:48pm
I'm really tired of people pushing the adoption of svn because it's a better CVS. The only people who that argument hasn't swayed are those who don't care what vcs they use, and that's because they don't care about the argument. Despite its name, svn is the status quo now, so do try to compare it to systems that it hasn't already made obsolete.
I'm really tired of people pushing the adoption of svn because it's a better CVS. The only people who that argument hasn't swayed are those who don't care what vcs they use, and that's because they don't care about the argument. Despite its name, svn is the status quo now, so do try to compare it to systems that it hasn't already made obsolete.
Update On The Shiny
gravityboy's Journal - Fri, 02/11/2007 - 6:43pm
If you've been wondering what's up with compiz in Debian, Sean Finney has been doing a ton of work on the packages. He's posted a major update here.
If you've been wondering what's up with compiz in Debian, Sean Finney has been doing a ton of work on the packages. He's posted a major update here.
Call For Testing
gravityboy's Journal - Thu, 01/11/2007 - 3:56am
I've just uploaded a new version of the xorg package to unstable. This package builds (somewhat confusingly, but trust me, there's a good reason) the xserver-xorg package that many people use to create their xorg.conf's. For this upload, we've been ripping out the logic in the xserver-xorg scripts that create the xorg.conf. The reason for this is that the X server can actually handle most of this configuration automatically at runtime, and I'm working on making it do even more. So rather than rely on a custom hacked together script we're moving towards a shared codebase that all linux and bsd distros should end up using to dynamically configure X at startup. This will be a significant win and it's really my primary release goal for Lenny.
So while this work is still ongoing, I'd like to ask that you all, our wonderful users of unstable, test these changes in the 7.3+3 package out. You see, we don't force an update to your xorg.conf, so your previous config should continue to work just like it always did. However, what we need to know if these slimmer xorg.confs work also. So if you have an xorg.conf that isn't customized, please copy and rename the copy somewhere (don't call the copy xorg.conf or else the server might load it even if it's not in the canonical /etc/X11 location) and run dpkg-reconfigure xserver-xorg to generate a new file. You'll see a lot less questions than in the past, but your old responses should be used by default. When you're done, restart the server and make sure it loaded like it used to. If not, please file a bug using reportbug against the xserver-xorg package.
This is still a work in progress, and I've still got a number of things I want to carefully extract from the scripts before the release, most notably any reliance on external programs like discover and xresprobe. But as things are, the configs should be slimmer and there's less time spent answering questions that aren't useful to getting a working X server running. I'm sure I'll send out another call for testing as I remove more major parts of the config, but this is an important step. Thanks everyone for helping make Lenny awesome!
I've just uploaded a new version of the xorg package to unstable. This package builds (somewhat confusingly, but trust me, there's a good reason) the xserver-xorg package that many people use to create their xorg.conf's. For this upload, we've been ripping out the logic in the xserver-xorg scripts that create the xorg.conf. The reason for this is that the X server can actually handle most of this configuration automatically at runtime, and I'm working on making it do even more. So rather than rely on a custom hacked together script we're moving towards a shared codebase that all linux and bsd distros should end up using to dynamically configure X at startup. This will be a significant win and it's really my primary release goal for Lenny.
So while this work is still ongoing, I'd like to ask that you all, our wonderful users of unstable, test these changes in the 7.3+3 package out. You see, we don't force an update to your xorg.conf, so your previous config should continue to work just like it always did. However, what we need to know if these slimmer xorg.confs work also. So if you have an xorg.conf that isn't customized, please copy and rename the copy somewhere (don't call the copy xorg.conf or else the server might load it even if it's not in the canonical /etc/X11 location) and run dpkg-reconfigure xserver-xorg to generate a new file. You'll see a lot less questions than in the past, but your old responses should be used by default. When you're done, restart the server and make sure it loaded like it used to. If not, please file a bug using reportbug against the xserver-xorg package.
This is still a work in progress, and I've still got a number of things I want to carefully extract from the scripts before the release, most notably any reliance on external programs like discover and xresprobe. But as things are, the configs should be slimmer and there's less time spent answering questions that aren't useful to getting a working X server running. I'm sure I'll send out another call for testing as I remove more major parts of the config, but this is an important step. Thanks everyone for helping make Lenny awesome!
They'd Better Not Spike My Beer With AJAX
gravityboy's Journal - Wed, 31/10/2007 - 3:45pm
Web 2.0 may be hate, but free drinks are love. Come help me spend Conde Nast's money tonight!
Web 2.0 may be hate, but free drinks are love. Come help me spend Conde Nast's money tonight!
The Land Of The Gnomes Is At The End Of The Green Line
gravityboy's Journal - Sun, 07/10/2007 - 6:36pm
I've been at the gnome summit for most of the past two days. Not actually working on gnome makes this a little strange, but it's kinda fun anyway. I saw demos of both the online desktop and hotwire. I'm not sold on the online desktop yet, but hotwire looks promising and fun. I chatted with ajax about all the cool stuff I missed at XDS this year, and also managed to write some code that lets automatically loading drivers override each other (say, have intel override i810). I'll be putting that code in to the xserver as soon as I get some time, probably this week, so I guess I managed to be productive.
I've been at the gnome summit for most of the past two days. Not actually working on gnome makes this a little strange, but it's kinda fun anyway. I saw demos of both the online desktop and hotwire. I'm not sold on the online desktop yet, but hotwire looks promising and fun. I chatted with ajax about all the cool stuff I missed at XDS this year, and also managed to write some code that lets automatically loading drivers override each other (say, have intel override i810). I'll be putting that code in to the xserver as soon as I get some time, probably this week, so I guess I managed to be productive.
More Code And Less User Support For A While
gravityboy's Journal - Tue, 25/09/2007 - 3:08pm
While it's cute that atoponce wants to blame #debian for his problems, what he neglects to mention is that I spent some time with him in #debian finding his bug and telling him that his best bet was to watch that bug report because that's the most up to date information available on the problem. Furthermore, I told him how to add a "me too" to the report so that he would be taken care of. I also googled him up a wiki page that explained what GLcore was better than I ever could. And I tried to do it as politely and respectfully as I could.
I'm not sure what more people want from the channel. The #debian policy is that "if you're running sid, you're on your own." The reason is that it's your machine and you're responsible for it. The fact that Aaron chooses to blame other people for his decision to run a development version of a package (and then to not downgrade it when it becomes too problematic) is the kind of frustrating attitude that makes me want to stay away from being active in user support.
Fortunately, #debian is actually a very good channel on the whole, especially in comparison to the past, where some individuals really were just plain mean spirited. It's full of knowledgeable users who are generally willing to help you find a solution to your Debian problem. Sure, they're wrong sometimes, everyone is. And no matter how knowledgeable you are, you're not going to be able to predict that the next upgrade will hose some random guy's system. If you want to run unstable, be prepared for what's in store and learn how to downgrade.
While it's cute that atoponce wants to blame #debian for his problems, what he neglects to mention is that I spent some time with him in #debian finding his bug and telling him that his best bet was to watch that bug report because that's the most up to date information available on the problem. Furthermore, I told him how to add a "me too" to the report so that he would be taken care of. I also googled him up a wiki page that explained what GLcore was better than I ever could. And I tried to do it as politely and respectfully as I could.
I'm not sure what more people want from the channel. The #debian policy is that "if you're running sid, you're on your own." The reason is that it's your machine and you're responsible for it. The fact that Aaron chooses to blame other people for his decision to run a development version of a package (and then to not downgrade it when it becomes too problematic) is the kind of frustrating attitude that makes me want to stay away from being active in user support.
Fortunately, #debian is actually a very good channel on the whole, especially in comparison to the past, where some individuals really were just plain mean spirited. It's full of knowledgeable users who are generally willing to help you find a solution to your Debian problem. Sure, they're wrong sometimes, everyone is. And no matter how knowledgeable you are, you're not going to be able to predict that the next upgrade will hose some random guy's system. If you want to run unstable, be prepared for what's in store and learn how to downgrade.
< MadCoder> wow gravity is spamming us
gravityboy's Journal - Sun, 16/09/2007 - 11:40pm
Brice already mentioned it, but I've uploaded Xorg 7.3 to unstable this afternoon. This includes Xserver version 1.4, which has improvements all over the place, perhaps most notably by including input hotplugging (via HAL) and many improvements in EXA, the next generation 2D driver acceleration architecture.
Now that this is uploaded, I'll be ripping out large parts of the xorg.conf generation script. All the various improvements over the past few releases have made it possible to do this, and for lenny, if all goes according to plan, we won't have to generate a xorg.conf for installs at all. You can still use one (and many people will want one to customize their settings in some way) but for the most part people shouldn't have to bother with them. I'll be spending a lot of my time over the next few months tightening this stuff up so that a lot of the little things really work the way they should, and now that we have the updated server available I'll be able to actively work on these things in Debian so you'll be able to see the results quickly.
On a side note, I went to Software Freedom Day yesterday. I heard ari give a talk on the Gimp. At the last minute I almost gave a lightning talk on where X.org is headed, but they ran out of time. It would have been kind of weird though, since the talks were on things like software patents, free culture, and miro (formerly democracy-player) rather than the nuts and bolts sorts of things that I'm more comfortable with.
Brice already mentioned it, but I've uploaded Xorg 7.3 to unstable this afternoon. This includes Xserver version 1.4, which has improvements all over the place, perhaps most notably by including input hotplugging (via HAL) and many improvements in EXA, the next generation 2D driver acceleration architecture.
Now that this is uploaded, I'll be ripping out large parts of the xorg.conf generation script. All the various improvements over the past few releases have made it possible to do this, and for lenny, if all goes according to plan, we won't have to generate a xorg.conf for installs at all. You can still use one (and many people will want one to customize their settings in some way) but for the most part people shouldn't have to bother with them. I'll be spending a lot of my time over the next few months tightening this stuff up so that a lot of the little things really work the way they should, and now that we have the updated server available I'll be able to actively work on these things in Debian so you'll be able to see the results quickly.
On a side note, I went to Software Freedom Day yesterday. I heard ari give a talk on the Gimp. At the last minute I almost gave a lightning talk on where X.org is headed, but they ran out of time. It would have been kind of weird though, since the talks were on things like software patents, free culture, and miro (formerly democracy-player) rather than the nuts and bolts sorts of things that I'm more comfortable with.
I Need More Sleep
gravityboy's Journal - Wed, 12/09/2007 - 5:00am
Yes, I'm totally jealous of Brice, but this is like his sweet revenge for missing the fun at Debconf. In other news, I'm 27 years old now, D looks like a really promising language for my needs, and I desperately need to find some time to do basic grunt work on the video drivers so that I can really scrap discover from the xserver postinstall scripts. I need to find the time to do a lot of things actually, but that one's so close I can taste it.
Brice mentions pci-rework, which promises to make my grunt work only a temporary Debian-only solution, so once I'm done with that I'll work out a more permanent solution that should go in upstream. Ultimately, the goal will be to have the server automatically load the correct driver given what the driver itself claims to do. There's a reasonable method of choosing a driver in the current server, but it's neither fully correct nor flexible, and my goal is to have something that'll work every time and not require the server itself to carry around any information to make it happen. It's just one more thing on the long list of stuff that the server really should be doing, but it isn't.
Yes, I'm totally jealous of Brice, but this is like his sweet revenge for missing the fun at Debconf. In other news, I'm 27 years old now, D looks like a really promising language for my needs, and I desperately need to find some time to do basic grunt work on the video drivers so that I can really scrap discover from the xserver postinstall scripts. I need to find the time to do a lot of things actually, but that one's so close I can taste it.
Brice mentions pci-rework, which promises to make my grunt work only a temporary Debian-only solution, so once I'm done with that I'll work out a more permanent solution that should go in upstream. Ultimately, the goal will be to have the server automatically load the correct driver given what the driver itself claims to do. There's a reasonable method of choosing a driver in the current server, but it's neither fully correct nor flexible, and my goal is to have something that'll work every time and not require the server itself to carry around any information to make it happen. It's just one more thing on the long list of stuff that the server really should be doing, but it isn't.
This Is My Good Free Software Experience
gravityboy's Journal - Sat, 11/08/2007 - 1:35pm
What do you know? I wake up this morning, check my mail, and find that Paul Wise contacted the Free Software sticker book guys and got them to change their license to MIT, which is very much Free. Thanks to Paul and the sticker creators for acting on this so fast! Apparently they were going to GPL it, but they thought that the license terms wouldn't make sense because it refers to "software", so they went with CC because it supposedly made more sense.
I've seen this sort of thing show up on debian-legal before. The opinion of most of the people on the list is that it doesn't matter because software is speech, and everything else derives from that principle. I agree with this, but it'd be really nice to have a more generic license to point people to instead for cases like these just to get rid of the doubts. Perhaps the CC-BY-SA license is a good choice. On the other hand, maybe we really do want to add one more to the license proliferation pile to solve this problem.
What do you know? I wake up this morning, check my mail, and find that Paul Wise contacted the Free Software sticker book guys and got them to change their license to MIT, which is very much Free. Thanks to Paul and the sticker creators for acting on this so fast! Apparently they were going to GPL it, but they thought that the license terms wouldn't make sense because it refers to "software", so they went with CC because it supposedly made more sense.
I've seen this sort of thing show up on debian-legal before. The opinion of most of the people on the list is that it doesn't matter because software is speech, and everything else derives from that principle. I agree with this, but it'd be really nice to have a more generic license to point people to instead for cases like these just to get rid of the doubts. Perhaps the CC-BY-SA license is a good choice. On the other hand, maybe we really do want to add one more to the license proliferation pile to solve this problem.
This Is Why Mako Is Right
gravityboy's Journal - Fri, 10/08/2007 - 9:17pm
It's mildly disconcerting that this otherwise nice collection of Free Software stickers is under a non-free license. You'd think that people would get it by now.
It's mildly disconcerting that this otherwise nice collection of Free Software stickers is under a non-free license. You'd think that people would get it by now.
Random Snippets From My Life
gravityboy's Journal - Fri, 03/08/2007 - 3:52am
The Bad: The kid in the lab next door had Boston's "More Than A Feeling" on repeat for about half an hour today. I don't even know where to begin.
The Good: The gym that I get to use for free at work is very small but fairly complete. Due to these two attributes, there's very little free space to use for jumping rope, which is something I like to do very much. Tuesday I saw a guy stake out a very small space and managed to do just that, so I tried it myself today and was able to get in jumping rope in to my workout for the first time in months. Much sweat did ensue.
The Bad: The kid in the lab next door had Boston's "More Than A Feeling" on repeat for about half an hour today. I don't even know where to begin.
The Good: The gym that I get to use for free at work is very small but fairly complete. Due to these two attributes, there's very little free space to use for jumping rope, which is something I like to do very much. Tuesday I saw a guy stake out a very small space and managed to do just that, so I tried it myself today and was able to get in jumping rope in to my workout for the first time in months. Much sweat did ensue.
I Want More Monads
gravityboy's Journal - Fri, 06/07/2007 - 4:09am
Over the past few days, I've returned to python for the first time in three or four years. My two excuses for not really liking python in the past were 1) a bad experience editing someone else's mixed tab-and-space indented code and 2) the language never really flowed out of me for some reason. I've gotten over the bad taste from the former after being assured by everyone I talked to that this sort of thing doesn't happen often, although I'm still wary of how easy it is for people to do this sort of thing.
The latter seems to have changed by a surprising avenue: learning haskell. I spent months trying to learn haskell, and I've gotten far enough along where I can sorta kinda construct useful programs with it. I'm not comfortable enough with it yet to do everything I want, but I like the language a lot because I always feel like I'm doing the right thing when I actually get a program to work. Strangely though, when I came back to python, all of a sudden how to write a lot of the code just made sense. Python lends itself well to a functional style of programming, especially given that it's started to consciously steal things like the new itertools and partial functions in version 2.5. I guess I shouldn't be too surprised that learning haskell suddenly made it a lot easier to understand other languages, but it's funny that haskell should be my gateway to python and not the other way around.
Over the past few days, I've returned to python for the first time in three or four years. My two excuses for not really liking python in the past were 1) a bad experience editing someone else's mixed tab-and-space indented code and 2) the language never really flowed out of me for some reason. I've gotten over the bad taste from the former after being assured by everyone I talked to that this sort of thing doesn't happen often, although I'm still wary of how easy it is for people to do this sort of thing.
The latter seems to have changed by a surprising avenue: learning haskell. I spent months trying to learn haskell, and I've gotten far enough along where I can sorta kinda construct useful programs with it. I'm not comfortable enough with it yet to do everything I want, but I like the language a lot because I always feel like I'm doing the right thing when I actually get a program to work. Strangely though, when I came back to python, all of a sudden how to write a lot of the code just made sense. Python lends itself well to a functional style of programming, especially given that it's started to consciously steal things like the new itertools and partial functions in version 2.5. I guess I shouldn't be too surprised that learning haskell suddenly made it a lot easier to understand other languages, but it's funny that haskell should be my gateway to python and not the other way around.
Post Debconf Spewage
gravityboy's Journal - Fri, 29/06/2007 - 3:15am
I've been back from Edinburgh for a few days, but this is the first real chance I've had to sit down and write a little bit about it. I'll preface this blog post by saying that my first Debconf was probably the best single week of my entire life. Yeah, it was that awesome.
I won't talk about all the stuff that happened, because that would just take too long. The most important thing was that I got to see a number of old friends again and spend more time with them in one run than ever before. That alone was really enormous for me. On top of that was the pleasure of finally meeting so many people in person. I met a few XSF members finally, including Julien Cristau, my partner in crime. There's no way I could possibly list everyone that I'm so thankful to have finally met. There was one very memorable evening in the night venue where it was largely the same crowd who's usually in #debian-devel when I'm on, and we all just couldn't stop cracking jokes and laughing (not to mention ITP'ing absurd programs). There was another night when Old World Cambridge collided with New World Cambridge, and much Pimm's was had by (almost) all. There was staying up until 5 in the morning and stumbling back to the hostel in the dawn to try and get some sleep before running back to the conference. The most delightful thing about all this was that so many people I already knew and loved were there, and everyone who I hadn't met in person before turned out to be even better in real life. It was like a week of the purest joy.
I also got the privilege of giving my talk and BoF to fairly full audiences, particularly the former. The BoF was on maintaining packages with git. I didn't expect more than 10 people to show, but more than 5 times that did express interest, so many that they moved it to the main lecture hall instead of the small discussion room that it was going to be in originally. I hadn't really planned out how to handle an audience like that, and while I managed to get other people in the audience talking (thanks in particular to Madduck, MadCoder, and keithp for adding so much to the discussion) I felt like I had to fumble through a lot of it. That's the nature of a BoF I guess. I also gave a talk on my plans for the XSF for the Lenny release. I'll talk about that stuff in future blog posts (some of which are overdue) but I will say that it was surprisingly well attended given that it was scheduled for 9:45 am after one of the usual nights of drinking and hacking until dawn.
I have to echo others and give a big thank you to the organizers. You guys did a great job, and persevered in the face of adversity (read: cowgate) and really made a great conference out of it. I learned a lot, laughed even more, and drank even more than that and I can't wait to do it again.
I've been back from Edinburgh for a few days, but this is the first real chance I've had to sit down and write a little bit about it. I'll preface this blog post by saying that my first Debconf was probably the best single week of my entire life. Yeah, it was that awesome.
I won't talk about all the stuff that happened, because that would just take too long. The most important thing was that I got to see a number of old friends again and spend more time with them in one run than ever before. That alone was really enormous for me. On top of that was the pleasure of finally meeting so many people in person. I met a few XSF members finally, including Julien Cristau, my partner in crime. There's no way I could possibly list everyone that I'm so thankful to have finally met. There was one very memorable evening in the night venue where it was largely the same crowd who's usually in #debian-devel when I'm on, and we all just couldn't stop cracking jokes and laughing (not to mention ITP'ing absurd programs). There was another night when Old World Cambridge collided with New World Cambridge, and much Pimm's was had by (almost) all. There was staying up until 5 in the morning and stumbling back to the hostel in the dawn to try and get some sleep before running back to the conference. The most delightful thing about all this was that so many people I already knew and loved were there, and everyone who I hadn't met in person before turned out to be even better in real life. It was like a week of the purest joy.
I also got the privilege of giving my talk and BoF to fairly full audiences, particularly the former. The BoF was on maintaining packages with git. I didn't expect more than 10 people to show, but more than 5 times that did express interest, so many that they moved it to the main lecture hall instead of the small discussion room that it was going to be in originally. I hadn't really planned out how to handle an audience like that, and while I managed to get other people in the audience talking (thanks in particular to Madduck, MadCoder, and keithp for adding so much to the discussion) I felt like I had to fumble through a lot of it. That's the nature of a BoF I guess. I also gave a talk on my plans for the XSF for the Lenny release. I'll talk about that stuff in future blog posts (some of which are overdue) but I will say that it was surprisingly well attended given that it was scheduled for 9:45 am after one of the usual nights of drinking and hacking until dawn.
I have to echo others and give a big thank you to the organizers. You guys did a great job, and persevered in the face of adversity (read: cowgate) and really made a great conference out of it. I learned a lot, laughed even more, and drank even more than that and I can't wait to do it again.
Am I Emo Or Just Really Pissed Off?
gravityboy's Journal - Fri, 15/06/2007 - 12:34am
In any graduate student career there are days where you just want to quit and do something else. Anything else. Like get a job in computers where you'll be making a lot more and potentially working a lot less. Today I've felt that urge more strongly than ever before. When I get back from Debconf I'm going to have to decide if it's worth sticking around for another year to finish my phd.
In any graduate student career there are days where you just want to quit and do something else. Anything else. Like get a job in computers where you'll be making a lot more and potentially working a lot less. Today I've felt that urge more strongly than ever before. When I get back from Debconf I'm going to have to decide if it's worth sticking around for another year to finish my phd.
Sobriety
gravityboy's Journal - Tue, 15/05/2007 - 11:57pm
For everyone who's celebrating the ATI announcement about fglrx, please read this. I can only imagine that it's wishful thinking that people reading Blizzard's blog entry and interpreting "committed to fixing the ATI problems with open source" as "we're going to open up fglrx". ATI hasn't really done anything yet, and they certainly aren't playing in the same class that Intel is playing. We as a community need to push Intel hardware because they're showing an incredible level of support that no other graphics vendor is willing to do right now. The Intel guys are even working on getting Real Documentation out there so that the community has an easier time helping hack on the driver. All we have from ATI right now are very old specs and a vague promise. Don't fall for ATI's marketing unless you like getting burned.
For everyone who's celebrating the ATI announcement about fglrx, please read this. I can only imagine that it's wishful thinking that people reading Blizzard's blog entry and interpreting "committed to fixing the ATI problems with open source" as "we're going to open up fglrx". ATI hasn't really done anything yet, and they certainly aren't playing in the same class that Intel is playing. We as a community need to push Intel hardware because they're showing an incredible level of support that no other graphics vendor is willing to do right now. The Intel guys are even working on getting Real Documentation out there so that the community has an easier time helping hack on the driver. All we have from ATI right now are very old specs and a vague promise. Don't fall for ATI's marketing unless you like getting burned.
Useful Informations On The Internets
gravityboy's Journal - Fri, 04/05/2007 - 4:36am
Did you ever want to know how a video card works? Here you go, courtesy of the fine folks at X.org and the DRI project: How Video Cards Work and Radeon Architecture.
Did you ever want to know how a video card works? Here you go, courtesy of the fine folks at X.org and the DRI project: How Video Cards Work and Radeon Architecture.
Why Xorg 7.2 Isn't In Unstable Yet
gravityboy's Journal - Mon, 16/04/2007 - 4:37pm
I think people are still a little euphoric over the etch release, but I'd like to let people know why Xorg 7.2 is still in experimental. Julien Cristau has done a ton of the grunt work, uploading all the usual library updates to unstable, and they're looking good so far, so at least that part is in place. The problem relates to the fact that the XSF wants to push forward with moving from Xlib to XCB. XCB represents a significant improvement over the traditional Xlib, and it's important that we work to nurture and standardize it. The good news about XCB is that most people transitioned over to it without any noticeable issues. Everything pretty much seems to work transparently, which is testament to the amount of care and time the XCB team has put in to creating it over the past several years.
But now for the bad news: there's some regressions, and they're almost all in non-free software. The biggest issue is this one: XCB will break java. You can work around it in older versions of java (as documented in that bug report) but in the newer ones you can't do much of anything. The XCB team and XSF (there's actually a fair bit of overlap there, because the XCB guys are Debian developers and I consider them XSF team members in that sense) were trying to hash out a solution to the problem. We're still working on it, but it represents the problems with depending heavily on non-free software in a free software context. Hopefully Sun will actually try to respond to the issue and release a java update that works with XCB. We can hack around the problem by disabling some checks in XCB, but the problem is really with buggy software, and we want to catch and fix those bugs. Essentially though, I'm not willing to break java in unstable so we're at something of a standstill.
In addition, there's a bug with a bit of proprietary Nvidia code. The XCB guys have a handle on that one, and think they can patch XCB itself to work around it, so hopefully that's a non-issue. Finally, some number of users have reported major performance degredation using XCB in some configurations. The XCB guys were actively going after that bug as well yesterday, so hopefully that'll be resolved soon too.
So that's where we're at. The server is looking good, and thanks to Julien and Drew Parsons we've got the latest X server release candidate and new drivers sitting in experimental, complete with the new (intel-only right now) randr 1.2 hotness, so if you want it, you know where to get it. I've finally begun my small contributions to the having a fully automagic configuring X server in experimental as well, but I'll blog about that when I reach my first milestone.
I think people are still a little euphoric over the etch release, but I'd like to let people know why Xorg 7.2 is still in experimental. Julien Cristau has done a ton of the grunt work, uploading all the usual library updates to unstable, and they're looking good so far, so at least that part is in place. The problem relates to the fact that the XSF wants to push forward with moving from Xlib to XCB. XCB represents a significant improvement over the traditional Xlib, and it's important that we work to nurture and standardize it. The good news about XCB is that most people transitioned over to it without any noticeable issues. Everything pretty much seems to work transparently, which is testament to the amount of care and time the XCB team has put in to creating it over the past several years.
But now for the bad news: there's some regressions, and they're almost all in non-free software. The biggest issue is this one: XCB will break java. You can work around it in older versions of java (as documented in that bug report) but in the newer ones you can't do much of anything. The XCB team and XSF (there's actually a fair bit of overlap there, because the XCB guys are Debian developers and I consider them XSF team members in that sense) were trying to hash out a solution to the problem. We're still working on it, but it represents the problems with depending heavily on non-free software in a free software context. Hopefully Sun will actually try to respond to the issue and release a java update that works with XCB. We can hack around the problem by disabling some checks in XCB, but the problem is really with buggy software, and we want to catch and fix those bugs. Essentially though, I'm not willing to break java in unstable so we're at something of a standstill.
In addition, there's a bug with a bit of proprietary Nvidia code. The XCB guys have a handle on that one, and think they can patch XCB itself to work around it, so hopefully that's a non-issue. Finally, some number of users have reported major performance degredation using XCB in some configurations. The XCB guys were actively going after that bug as well yesterday, so hopefully that'll be resolved soon too.
So that's where we're at. The server is looking good, and thanks to Julien and Drew Parsons we've got the latest X server release candidate and new drivers sitting in experimental, complete with the new (intel-only right now) randr 1.2 hotness, so if you want it, you know where to get it. I've finally begun my small contributions to the having a fully automagic configuring X server in experimental as well, but I'll blog about that when I reach my first milestone.
That's it? If you had a million dollars, you'd do two releases at the same time?
gravityboy's Journal - Sun, 08/04/2007 - 9:55pm
Today is such a wonderful day! Congratulations to everyone in the project who made the release possible, and a special thank you to the release team for the heroic effort!
Anyone in Boston/Cambridge who's considering coming out to the last-minute release party tonight should just go for it. It's a big day and we should celebrate it!
Today is such a wonderful day! Congratulations to everyone in the project who made the release possible, and a special thank you to the release team for the heroic effort!
Anyone in Boston/Cambridge who's considering coming out to the last-minute release party tonight should just go for it. It's a big day and we should celebrate it!
A (Dubious) Benefit To FizzBuzz
gravityboy's Journal - Sun, 25/03/2007 - 7:36pm
There's been a ton of blog spew about FizzBuzz. My two favorites are from the always excellent Raganwald and Giles Bowkett. I waited until after the furor was over (at least, I hope it's over now) to post this, because a lot of the criticism of the FizzBuzz madness is spot on.
I think one of the neat things to fall out of it all that people seem to have ignored is that FizzBuzz provides an incredibly simple framework with which to play with essentially any language feature you want. Sure, the problem is trivial to implement, but that's the beauty of it. You can wrap up your simple solution in absurdly complicated structures in order to play with the more advanced features of the language. In this way, the problem gets out of your way and lets you focus on the language or toolkit.
I used it to play with some of Ruby's more functional aspects, for example, because I didn't feel like coding a more complicated app just for that. Sure, people are doing things like showing off how much java they can code around this little problem, and I don't think there's any value in that at all. But FizzBuzz does provide a surprisingly useful set of training wheels for diving in to new aspects of whatever you're working on. Note that FizzBuzz doesn't provide any real advantage over implementing something like a quicksort here, but that doesn't really diminish it either.
There's been a ton of blog spew about FizzBuzz. My two favorites are from the always excellent Raganwald and Giles Bowkett. I waited until after the furor was over (at least, I hope it's over now) to post this, because a lot of the criticism of the FizzBuzz madness is spot on.
I think one of the neat things to fall out of it all that people seem to have ignored is that FizzBuzz provides an incredibly simple framework with which to play with essentially any language feature you want. Sure, the problem is trivial to implement, but that's the beauty of it. You can wrap up your simple solution in absurdly complicated structures in order to play with the more advanced features of the language. In this way, the problem gets out of your way and lets you focus on the language or toolkit.
I used it to play with some of Ruby's more functional aspects, for example, because I didn't feel like coding a more complicated app just for that. Sure, people are doing things like showing off how much java they can code around this little problem, and I don't think there's any value in that at all. But FizzBuzz does provide a surprisingly useful set of training wheels for diving in to new aspects of whatever you're working on. Note that FizzBuzz doesn't provide any real advantage over implementing something like a quicksort here, but that doesn't really diminish it either.
Much Like The Internet, Your Mom Is A Series Of Tubes
gravityboy's Journal - Mon, 19/03/2007 - 10:41pm
This article got me thinking about some new ways we could dramatically improve Debian. We had a lot of success in the past because good packaging helps people get over the fearful hump of installing and setting up new software in order to learn it. We're great at that, and the battle is pretty much over there.
Afterwards though, I don't think we're so good at helping our users through the scary parts. We include a lot of documentation, but that's not really cutting it any more. I set out to work on X because the scariest thing for a lot of users is getting X up and running, and I wanted to help them (and myself!) do it. We still don't have that nailed, but it's in the pipeline, so I think the next stage will be to help people do cool things with X that they wouldn't have done before. In the past, this would have involved doing things like setting xrdb settings, but these days such things aren't really useful. I'm going to continue brainstorming this, but to this end I've started a page on the wiki that's expressly for helping users get over some of the more difficult and interesting things that people might want to do with, or to, X.
Currently the only tip is on building packages using upstream's git HEAD and our own packaging, but I'm looking to put more stuff up in the nearish future. While we haven't been maintaining it so well, the XSF FAQ has a lot of these sorts of tips too.
I think we can push harder and be more clever about giving our users and ourselves cool things to do with Debian beyond packaging. The XSF move to git opened up the opportunity to easily track the X.org HEAD with the Debian packaging, and I think that sort of thing is indicative that we can be clever about enabling people at all stages of the development process, not just through the packaging itself. If you guys have any ideas for more X tips you'd like to see documented, please edit that wiki page or at least drop me or debian-x a mail so we can write it up for you. I want to see how far we can push this.
This article got me thinking about some new ways we could dramatically improve Debian. We had a lot of success in the past because good packaging helps people get over the fearful hump of installing and setting up new software in order to learn it. We're great at that, and the battle is pretty much over there.
Afterwards though, I don't think we're so good at helping our users through the scary parts. We include a lot of documentation, but that's not really cutting it any more. I set out to work on X because the scariest thing for a lot of users is getting X up and running, and I wanted to help them (and myself!) do it. We still don't have that nailed, but it's in the pipeline, so I think the next stage will be to help people do cool things with X that they wouldn't have done before. In the past, this would have involved doing things like setting xrdb settings, but these days such things aren't really useful. I'm going to continue brainstorming this, but to this end I've started a page on the wiki that's expressly for helping users get over some of the more difficult and interesting things that people might want to do with, or to, X.
Currently the only tip is on building packages using upstream's git HEAD and our own packaging, but I'm looking to put more stuff up in the nearish future. While we haven't been maintaining it so well, the XSF FAQ has a lot of these sorts of tips too.
I think we can push harder and be more clever about giving our users and ourselves cool things to do with Debian beyond packaging. The XSF move to git opened up the opportunity to easily track the X.org HEAD with the Debian packaging, and I think that sort of thing is indicative that we can be clever about enabling people at all stages of the development process, not just through the packaging itself. If you guys have any ideas for more X tips you'd like to see documented, please edit that wiki page or at least drop me or debian-x a mail so we can write it up for you. I want to see how far we can push this.
This May Be A Whiskey And Work Weekend
gravityboy's Journal - Thu, 15/03/2007 - 4:06pm
I was away in Philadelphia over the past week for the annual Drosophila meeting. My boss presented our work, and I shared a poster about it with the postdoc in the lab. I was constantly running from talk to talk and from poster to poster, trying to soak up as much as I could. I also got to meet a fair number of famous people, which was pretty awesome. It's not every day you get to meet a Nobel laureate, but that's why you go to these things. We got a lot of good feedback on our work, and we've basically gone in to overdrive to try and get the paper (which will be my first one) submitted.
Since I've been back, I haven't had any significant time to do Debian work, mainly because I've been in the lab or just taking care of life stuff almost non-stop. I managed to sneak in some time to tweak the XSF wiki pages, but the rest of the team has been handling things in my absence. I wish I could report on what they've been up to, but I haven't looked at any non-personal email for the past week.
Finally having a chance to check planet Debian today, I'm excited to see Martin's various posts about the release management for different free software projects. Obviously that's been the major thorn in our side for a while now, and while we're improving we can obviously do better. I'm looking forward to a post-mortem report and discussion from the release team about what the next steps we need to take are. I have my own ideas of course, but the release team has a much wider field of view than I've been able to achieve.
Off to see the Shins tonight, hopefully they're as good live as I've heard.
I was away in Philadelphia over the past week for the annual Drosophila meeting. My boss presented our work, and I shared a poster about it with the postdoc in the lab. I was constantly running from talk to talk and from poster to poster, trying to soak up as much as I could. I also got to meet a fair number of famous people, which was pretty awesome. It's not every day you get to meet a Nobel laureate, but that's why you go to these things. We got a lot of good feedback on our work, and we've basically gone in to overdrive to try and get the paper (which will be my first one) submitted.
Since I've been back, I haven't had any significant time to do Debian work, mainly because I've been in the lab or just taking care of life stuff almost non-stop. I managed to sneak in some time to tweak the XSF wiki pages, but the rest of the team has been handling things in my absence. I wish I could report on what they've been up to, but I haven't looked at any non-personal email for the past week.
Finally having a chance to check planet Debian today, I'm excited to see Martin's various posts about the release management for different free software projects. Obviously that's been the major thorn in our side for a while now, and while we're improving we can obviously do better. I'm looking forward to a post-mortem report and discussion from the release team about what the next steps we need to take are. I have my own ideas of course, but the release team has a much wider field of view than I've been able to achieve.
Off to see the Shins tonight, hopefully they're as good live as I've heard.
Because You Probably Don't Track Experimental
gravityboy's Journal - Mon, 05/03/2007 - 12:46am
Julien Cristau and I have finished putting Xorg 7.2 in to experimental, so have at it! Julien did most of the work on this release, including the server and libs. The transition to git appears to be paying off already, as we're all moving towards running the bleeding edge stuff mainly because it's so easy to do now. This should give us the ability to take a more active role in upstream development. I've already sorta taken on responsibility for some of the upstream documentation (don't ask me how that happened) but now that things are moving I'll be looking to do work in the code as well. I wouldn't be surprised if other XSF members start to do the same.
Julien Cristau and I have finished putting Xorg 7.2 in to experimental, so have at it! Julien did most of the work on this release, including the server and libs. The transition to git appears to be paying off already, as we're all moving towards running the bleeding edge stuff mainly because it's so easy to do now. This should give us the ability to take a more active role in upstream development. I've already sorta taken on responsibility for some of the upstream documentation (don't ask me how that happened) but now that things are moving I'll be looking to do work in the code as well. I wouldn't be surprised if other XSF members start to do the same.
ImageJ Rocks
gravityboy's Journal - Sat, 24/02/2007 - 8:54pm
ImageJ is probably the coolest piece of Free Software that the Free Software community isn't broadly aware of yet. It's a stable, mature piece of cross-platform code that supports both plugins and macros, and a large number of these have already been written by the scientific community. When contrasted to the photoshop/gimp debates, there simply is no competitor to ImageJ in the world of biological imaging: it's the standard for such work (except for specialized tasks like driving microscopes). If it wasn't in java (which I don't really know the tools for right now) I'd have uploaded it to Debian long ago.
A word of note though: if you want to use it seriously, I recommend building it yourself. I just spent two hours reading through the interpreter trying to speed up an extremely slow macro only to find that with a simple rebuild with the jdk in unstable's non-free I got an immediate and more than acceptable speedup, which should speed up my PhD work significantly. Still, the code is relatively nice and it was fun digging in to the interpreter, which was quite easy to understand. I may have to add a few more macro functions that I've been itching to have.
With the recent freeing of java, it'll be great to see software like imagej and jedit (which I'm also really enjoying) enter Debian finally.
ImageJ is probably the coolest piece of Free Software that the Free Software community isn't broadly aware of yet. It's a stable, mature piece of cross-platform code that supports both plugins and macros, and a large number of these have already been written by the scientific community. When contrasted to the photoshop/gimp debates, there simply is no competitor to ImageJ in the world of biological imaging: it's the standard for such work (except for specialized tasks like driving microscopes). If it wasn't in java (which I don't really know the tools for right now) I'd have uploaded it to Debian long ago.
A word of note though: if you want to use it seriously, I recommend building it yourself. I just spent two hours reading through the interpreter trying to speed up an extremely slow macro only to find that with a simple rebuild with the jdk in unstable's non-free I got an immediate and more than acceptable speedup, which should speed up my PhD work significantly. Still, the code is relatively nice and it was fun digging in to the interpreter, which was quite easy to understand. I may have to add a few more macro functions that I've been itching to have.
With the recent freeing of java, it'll be great to see software like imagej and jedit (which I'm also really enjoying) enter Debian finally.
We've Moved!
gravityboy's Journal - Wed, 07/02/2007 - 3:17am
For several months now, the XSF has been discussing, planning, and preparing to move from our old reliable svn repository to our new distributed git repositories on git.debian.org. As of a few minutes ago, that move is complete.
I'd like to thank the Alioth admins for setting up git.debian.org and letting us hammer it with the whole xorg codebase. Also deserving the lion's share of the credit for this move is Thierry Reding, who wrote the infrastructure and did a lot of debugging of the conversion script that came with git to let us make this move. It was a huge task, and I'm still amazed at how quickly he managed to pull it off.
I'm really excited that this is finally done. I've been wanting to have our codebase in some sort of distributed system ever since I started working on xorg a while back. Because I was deriving packages from Ubuntu, who was in turn deriving packages from us, it seemed like having the whole thing in a distributed system would have made my life a lot easier. It would also potentially make it easier on other derivatives who customize the X packages, like Knoppix. For a while I thought we were going to move to mercurial, but once X.org itself moved to git, it only made sense to follow suit. I'm excited at the benefits that this move will provide us, which will hopefully translate to faster updates and higher quality packages for Debian users. And this is only the beginning of what's going to come during the Lenny release cycle.
For several months now, the XSF has been discussing, planning, and preparing to move from our old reliable svn repository to our new distributed git repositories on git.debian.org. As of a few minutes ago, that move is complete.
I'd like to thank the Alioth admins for setting up git.debian.org and letting us hammer it with the whole xorg codebase. Also deserving the lion's share of the credit for this move is Thierry Reding, who wrote the infrastructure and did a lot of debugging of the conversion script that came with git to let us make this move. It was a huge task, and I'm still amazed at how quickly he managed to pull it off.
I'm really excited that this is finally done. I've been wanting to have our codebase in some sort of distributed system ever since I started working on xorg a while back. Because I was deriving packages from Ubuntu, who was in turn deriving packages from us, it seemed like having the whole thing in a distributed system would have made my life a lot easier. It would also potentially make it easier on other derivatives who customize the X packages, like Knoppix. For a while I thought we were going to move to mercurial, but once X.org itself moved to git, it only made sense to follow suit. I'm excited at the benefits that this move will provide us, which will hopefully translate to faster updates and higher quality packages for Debian users. And this is only the beginning of what's going to come during the Lenny release cycle.
Bits From The XSF
gravityboy's Journal - Thu, 01/02/2007 - 4:47am
This has been a little overdue, but here it is. Lots of less than glamorous stuff has been going on in the XSF, and a few more exciting things are in the pipeline.
Among the less glamorous stuff, our resident release junkie/alpha porter, Steve Langasek, fixed #392500 which was our major RC bug on the alpha arch. Aside from that bug, there's been a whole host of bug triage by our newest team member, Brice Goglin. Brice has taken on the unenviable task of going through the massive list of bugs owned by the XSF. I've been traditionally less than responsible about handling bug reports like I should, and Brice is dealing with the mess I've left behind. Before he started, we were at over 2000 outstanding bugs, and as of this writing we're down to 1646 bugs, which is a huge amount of work. Hopefully this will go a long way towards making our little corner of the BTS usable for mere mortals. Aside from that, there's been an enormous amount of small cleanups and bug fixes by the whole team, most notably Julien Cristau (who's been doing more actual release management than yours truly), Thierry Redding, and Drew Parsons. Perhaps most important among these are miscellaneous last minute driver fixes that will enable a fair number of people to actually run Etch without backports. Michel Dänzer has continued to be his usual awesome self, responding to the tough DRI bugs that the rest of us are terrified to approach.
Some of the more exciting stuff from a user perspective has been the various updates and new packaging going on. Thierry has taken on mesa, which is an incredibly daunting task, and he's done and incredible job of it so far. He and Julien cooperated on getting the newest mesa release, 6.5.2, in to experimental. That's a huge step on the way to getting the 7.2 release in to the archive. Thierry has also packaged up the newest compiz and some additional plugins ported from beryl, both of which are waiting in NEW right now for our overworked ftp masters to find time to have a look at them. Speaking of beryl, we also have that more or less complete and packaged. Our second-newest XSF member, Shawn Starr, has taken on beryl, and done a great job of getting the packages ready for experimental. They came back from review with a few minor comments, so he's busy getting those last issues resolved, so beryl should be winging its way over to NEW again for re-review soon. Finally, Josh Triplett and Jamie Sharp, who are both Debian and XCB developers, have put XCB in to experimental for you to test and play with. Expect to hear more about this when the Lenny development cycle starts.
The other huge thing that's been going on, and what I've been devoting most of my time to (aside from reading all the mail from Brice's bug triage), is transitioning the XSF over to git. Thierry has written major chunk of infrastructure to help move us over. I've done most of the conversion locally, and have been putting things up on git.debian.org as I go. As of now, we've got a very large chunk of X in git now, and using svn for those bits is officially closed. I'm hoping to complete the move in the next two weeks or so. If you want the above software that's not yet in the archive (beryl, compiz, compiz-extra, etc) you can clone it from the git repositories and build away. The other major thing that I've been doing is writing up a XSF git policy. This required a lot of input from the team, and we had to come up with a reasonable way of working the archive so that we could easily work with an upstream that was also using git, which as far as I know is a unique situation right now. It's in pretty good shape right now and we're just starting to put it in to real use. I've proposed a talk for Debconf on the XSF roadmap for Lenny, and I'll be sure to talk about our experiences using git if it gets accepted. In addition, we've all been learning the ins and outs of git, which is a whole other post in itself. I think that the XSF will end up being a really good resource for people in Debian who are looking to use git, as we'll have a lot of real world experience with a tool that relatively few people actually seem to know. Since I think git usage will only grow in the future, hopefully this will end up being valuable for Debian as a whole.
This has been a little overdue, but here it is. Lots of less than glamorous stuff has been going on in the XSF, and a few more exciting things are in the pipeline.
Among the less glamorous stuff, our resident release junkie/alpha porter, Steve Langasek, fixed #392500 which was our major RC bug on the alpha arch. Aside from that bug, there's been a whole host of bug triage by our newest team member, Brice Goglin. Brice has taken on the unenviable task of going through the massive list of bugs owned by the XSF. I've been traditionally less than responsible about handling bug reports like I should, and Brice is dealing with the mess I've left behind. Before he started, we were at over 2000 outstanding bugs, and as of this writing we're down to 1646 bugs, which is a huge amount of work. Hopefully this will go a long way towards making our little corner of the BTS usable for mere mortals. Aside from that, there's been an enormous amount of small cleanups and bug fixes by the whole team, most notably Julien Cristau (who's been doing more actual release management than yours truly), Thierry Redding, and Drew Parsons. Perhaps most important among these are miscellaneous last minute driver fixes that will enable a fair number of people to actually run Etch without backports. Michel Dänzer has continued to be his usual awesome self, responding to the tough DRI bugs that the rest of us are terrified to approach.
Some of the more exciting stuff from a user perspective has been the various updates and new packaging going on. Thierry has taken on mesa, which is an incredibly daunting task, and he's done and incredible job of it so far. He and Julien cooperated on getting the newest mesa release, 6.5.2, in to experimental. That's a huge step on the way to getting the 7.2 release in to the archive. Thierry has also packaged up the newest compiz and some additional plugins ported from beryl, both of which are waiting in NEW right now for our overworked ftp masters to find time to have a look at them. Speaking of beryl, we also have that more or less complete and packaged. Our second-newest XSF member, Shawn Starr, has taken on beryl, and done a great job of getting the packages ready for experimental. They came back from review with a few minor comments, so he's busy getting those last issues resolved, so beryl should be winging its way over to NEW again for re-review soon. Finally, Josh Triplett and Jamie Sharp, who are both Debian and XCB developers, have put XCB in to experimental for you to test and play with. Expect to hear more about this when the Lenny development cycle starts.
The other huge thing that's been going on, and what I've been devoting most of my time to (aside from reading all the mail from Brice's bug triage), is transitioning the XSF over to git. Thierry has written major chunk of infrastructure to help move us over. I've done most of the conversion locally, and have been putting things up on git.debian.org as I go. As of now, we've got a very large chunk of X in git now, and using svn for those bits is officially closed. I'm hoping to complete the move in the next two weeks or so. If you want the above software that's not yet in the archive (beryl, compiz, compiz-extra, etc) you can clone it from the git repositories and build away. The other major thing that I've been doing is writing up a XSF git policy. This required a lot of input from the team, and we had to come up with a reasonable way of working the archive so that we could easily work with an upstream that was also using git, which as far as I know is a unique situation right now. It's in pretty good shape right now and we're just starting to put it in to real use. I've proposed a talk for Debconf on the XSF roadmap for Lenny, and I'll be sure to talk about our experiences using git if it gets accepted. In addition, we've all been learning the ins and outs of git, which is a whole other post in itself. I think that the XSF will end up being a really good resource for people in Debian who are looking to use git, as we'll have a lot of real world experience with a tool that relatively few people actually seem to know. Since I think git usage will only grow in the future, hopefully this will end up being valuable for Debian as a whole.
Why All Our Apt Frontends Suck
gravityboy's Journal - Wed, 24/01/2007 - 1:43am
liw's latest blog post got me thinking again about our apt frontends. I've never been satisfied with them, and I've found synaptic particularly unwieldy. They get the job done just fine, but I haven't been able to figure out a better way to do it.
Looking back though, I think there's only one real answer: we want an apt frontend that's just like this. You just search for what you want, mark the packages that show up to be installed or removed, and then just run the thing. Our current setup isn't too different from ye olde web directories (remember when you found links on yahoo or dmoz by drilling down through hand-sorted categories?) which mercifully died in favor of a simple text box and submit button. I think our current apt frontends need to do the same.
liw's latest blog post got me thinking again about our apt frontends. I've never been satisfied with them, and I've found synaptic particularly unwieldy. They get the job done just fine, but I haven't been able to figure out a better way to do it.
Looking back though, I think there's only one real answer: we want an apt frontend that's just like this. You just search for what you want, mark the packages that show up to be installed or removed, and then just run the thing. Our current setup isn't too different from ye olde web directories (remember when you found links on yahoo or dmoz by drilling down through hand-sorted categories?) which mercifully died in favor of a simple text box and submit button. I think our current apt frontends need to do the same.
It's Not A Crisis Until The Java Guys Actually Take Over
gravityboy's Journal - Mon, 15/01/2007 - 10:45pm
Lately I keep seeing people talking about a "crisis" in the ruby community due to the fact that rubinius, jruby, and YARV are all being written. There seems to be some subtle hand-wringing under the covers about divergence between these implementations. The ruby language is in pretty solid form, with a few little problems that should be cleaned up for the 2.0 release, so I don't think it's very likely that any divergence will be particularly damaging.
More importantly though, I think it's a sign that ruby is alive and well, given that there's three different projects that are working to provide an improved implementation for a language that was largely ignored until two or three years ago. Plus the benchmarks for YARV (which was just merged to the ruby trunk) look pretty hot.
Unfortunately for the doomsayers, most of what I've heard and read from the actual ruby community about all three of these projects has been a giant ball of excitement. I think it's really sweet that we'll have ruby running on the JVM, on a sexy Smalltalk style VM, and also a significantly sped up classic interpreter. Now all I want for Christmas is more time to actually write the rails site that's been pounding away inside my head like Athena.
Lately I keep seeing people talking about a "crisis" in the ruby community due to the fact that rubinius, jruby, and YARV are all being written. There seems to be some subtle hand-wringing under the covers about divergence between these implementations. The ruby language is in pretty solid form, with a few little problems that should be cleaned up for the 2.0 release, so I don't think it's very likely that any divergence will be particularly damaging.
More importantly though, I think it's a sign that ruby is alive and well, given that there's three different projects that are working to provide an improved implementation for a language that was largely ignored until two or three years ago. Plus the benchmarks for YARV (which was just merged to the ruby trunk) look pretty hot.
Unfortunately for the doomsayers, most of what I've heard and read from the actual ruby community about all three of these projects has been a giant ball of excitement. I think it's really sweet that we'll have ruby running on the JVM, on a sexy Smalltalk style VM, and also a significantly sped up classic interpreter. Now all I want for Christmas is more time to actually write the rails site that's been pounding away inside my head like Athena.
Communities
gravityboy's Journal - Sat, 30/12/2006 - 10:48pm
I spent some time playing with haskell over the past month or so. One of the most incredible things about it is the wonderful community that has sprung up around the language. The haskell homepage is a treasure trove of information, excited users throw all sorts of new articles at reddit every day, and the #haskell channel is exactly what you want out of an irc channel. A nice bonus for Debian types is the incredibly close ties the haskell community has with our distro.
Comparing this to Debian is instructive. The haskell community seems to be having a ton of fun using their language. The sense in #debian, or from whatever Debian tutorials I read, is that people aren't using their system to have fun any more. The air of joy and occasional whimsy that traces its way through #haskell (and also very strongly through the ruby community) seems to be somewhat surpressed in Debian, as though we're so big and cool and technically competent that we don't have time for frivolity. It's hard to have fun and create a warm and inviting community with this kind of attitude. Turf wars, protocol, and politics only compound the problem.
Doing something fun for its own sake should be a part of the Debian experience the same way that it's inherently part of the free software experience. It's way too easy to lose that in the mountain of packages, emails, and bug reports. Using Debian makes me happy, and I want to try and let that spirit out to play a little more.
I spent some time playing with haskell over the past month or so. One of the most incredible things about it is the wonderful community that has sprung up around the language. The haskell homepage is a treasure trove of information, excited users throw all sorts of new articles at reddit every day, and the #haskell channel is exactly what you want out of an irc channel. A nice bonus for Debian types is the incredibly close ties the haskell community has with our distro.
Comparing this to Debian is instructive. The haskell community seems to be having a ton of fun using their language. The sense in #debian, or from whatever Debian tutorials I read, is that people aren't using their system to have fun any more. The air of joy and occasional whimsy that traces its way through #haskell (and also very strongly through the ruby community) seems to be somewhat surpressed in Debian, as though we're so big and cool and technically competent that we don't have time for frivolity. It's hard to have fun and create a warm and inviting community with this kind of attitude. Turf wars, protocol, and politics only compound the problem.
Doing something fun for its own sake should be a part of the Debian experience the same way that it's inherently part of the free software experience. It's way too easy to lose that in the mountain of packages, emails, and bug reports. Using Debian makes me happy, and I want to try and let that spirit out to play a little more.
Document Writing And Programmer Tools
gravityboy's Journal - Tue, 26/12/2006 - 6:08pm
I love that one of the benefits to working on free software is that it exposes you to tools that you wouldn't normally be aware of. A prime example is software that you can use for writing documentation, like LaTeX and revision control systems. At work I've been writing up a fairly long document. Normally at work I'm forced to use OO.o or word because I have to trade the file back and forth with my boss, but this time since I'm doing all the actual writing, I've been using LaTeX and git to manage the thing. In addition, I need a bibliography and references, so I basically needed something like LaTeX. I send my boss an rtf (made with latex2rtf) and pdf (made with pdflatex) when I want comments, and I can still use the highly structured style of LaTeX to make sure that things like my lists get formatted properly every single time. Trying to do this in Word was both frustrating and time consuming.
The best part about the experience has been using revision control though. Having atomic changesets with a log message is something we tend to take for granted in free software work, but compare it to the change tracking in OO.o and Word. It's amazing how much easier it makes it. I can actually see what I was thinking a few days ago, which was the last time I had been revising the document. Being able to visualize the changes and history with tools like qgit are even nicer. And tags... I don't think OO.o or Word lets you tag specific revisions, but I know that when I do this with Word, I end up with a ton of documents labeled "DavidRev1" and so forth, and I always mess up the revisions by using "Save" instead of "Save As".
And this doesn't even touch on the things I haven't done yet. I haven't done any branching, for example. I'm sure that when I write my thesis, I'll end up using that, but just knowing that it's available is exciting. The thing I wish I could do most though is actually put the repo up somewhere and have me and my boss actually use it the same way a normal shared source code repository is used.
The standard sorry ending to this story is that getting these things in broad use is basically a pipe dream. It continually amazes me that people will put up with Word or Powerpoint when it's so often the wrong tool for the job. The simplest things, like being able to look at a document's history and the diff and know in a minute exactly what you were thinking when you put down your work a week ago, help so much. It's sad because the technology problem has been largely solved (a small GUI frontend would be the last piece), but the evangelism problem remains. One thing is for sure, I'm not looking forward to writing my next paper in Word.
I love that one of the benefits to working on free software is that it exposes you to tools that you wouldn't normally be aware of. A prime example is software that you can use for writing documentation, like LaTeX and revision control systems. At work I've been writing up a fairly long document. Normally at work I'm forced to use OO.o or word because I have to trade the file back and forth with my boss, but this time since I'm doing all the actual writing, I've been using LaTeX and git to manage the thing. In addition, I need a bibliography and references, so I basically needed something like LaTeX. I send my boss an rtf (made with latex2rtf) and pdf (made with pdflatex) when I want comments, and I can still use the highly structured style of LaTeX to make sure that things like my lists get formatted properly every single time. Trying to do this in Word was both frustrating and time consuming.
The best part about the experience has been using revision control though. Having atomic changesets with a log message is something we tend to take for granted in free software work, but compare it to the change tracking in OO.o and Word. It's amazing how much easier it makes it. I can actually see what I was thinking a few days ago, which was the last time I had been revising the document. Being able to visualize the changes and history with tools like qgit are even nicer. And tags... I don't think OO.o or Word lets you tag specific revisions, but I know that when I do this with Word, I end up with a ton of documents labeled "DavidRev1" and so forth, and I always mess up the revisions by using "Save" instead of "Save As".
And this doesn't even touch on the things I haven't done yet. I haven't done any branching, for example. I'm sure that when I write my thesis, I'll end up using that, but just knowing that it's available is exciting. The thing I wish I could do most though is actually put the repo up somewhere and have me and my boss actually use it the same way a normal shared source code repository is used.
The standard sorry ending to this story is that getting these things in broad use is basically a pipe dream. It continually amazes me that people will put up with Word or Powerpoint when it's so often the wrong tool for the job. The simplest things, like being able to look at a document's history and the diff and know in a minute exactly what you were thinking when you put down your work a week ago, help so much. It's sad because the technology problem has been largely solved (a small GUI frontend would be the last piece), but the evangelism problem remains. One thing is for sure, I'm not looking forward to writing my next paper in Word.