My first intense hacking session: learning a few lessons
September 7th, 2007Word of caution: “Hacking” and “hacker” do not actually mean the way most uninformed media would like you to believe. Hacking is not about breaking and entering into protected or private computers and networks. That’s cracking. Hacking is about the joy of computer programming and working on computers. Think of it like a passion for computers and programming. Don’t be mislead, read http://en.wikipedia.org/wiki/Hacker.
Yesterday I experience my first ever intense hacking session. The experience, though probably shallow for more seasoned developers, was both exhilarating and exhausting. It started with an itch to scratch. Kubuntu was going to ship D3lphin as the new default file manager for Kubuntu 7.10 Gutsy Gibbon. Now D3lphin is based on Dolphin, which will be the new default file manager for KDE 4. However, work on the KDE 3.5 version of Dolphin stopped in order to focus more on KDE 4. Fortunately, Marchel Juhnke took up the code and added bugfixes and features to try to match the KDE 4 counterpart and it was named D3lphin. But it still lacked a few features that I consider to be semi-necessary for a decent, modern file manager. The thought that Kubuntu would have to ship without these bothered me enough to work on it myself.
The first missing feature was a simpler one. D3lphin didn’t have a service menu (a.k.a. right-click menu) to compress files into Gzipped or Bzip2-ed tarballs. It did have a feature to compress the whole current folder’s contents, but not individual files. It was a bit easy to make one since D3lphin practically used the same service menus as Konqueror. Unfortunately, there was one issue I couldn’t get around to. You can compress the whole current folder. You can multiple files and compress them. But you can’t select a mixture of files and folders and compress them. I’m guessing it’s a problem with Dolphin’s service menu implementation.
The other missing feature was Trash handling. You can move files to the trash, but you can’t restore them from the trash, or empty the trash folder through Dolphin. Now this was a wee bit more complicated for a beginner like me. I had to look at the KDE 4 Dolphin code and see how they implemented the features. Then I had to look for the corresponding KDE 3 code that would let it do the same. I was really intent and motivated to work on this, so much that I almost developed a new motto: “Scratch an itch… until it’s sore.” Staying up until 03:00 a.m., waking up less than 6 hours later and jumping in front of the computer to continue working on it. Waiting for the other developers on the other side of the world (to whom I’m very much indebted to in making this patch work). Learning about KDE code (like how Dolphin in KDE 4 is still dependent on libkonq) and different tools (lxr.kde.org and api.kde.org are great!). And then finally seeing it available as an update, sitting back and finally relaxing. It was quite an experience.
But aside from learning about coding stuff, I did come to some realizations, specially in working. It became apparent that I needed to learn packaging whether I liked it or not. I recently decided to put MOTU/packaging plans on the backburner to give focus to my coding studies. But in coding and developing for a project, like a distribution, requires you to comply with certain idiosyncrasies, like Debian packaging or automake in my case. Maybe that was obvious for some, but I thought that I could get away with just plain coding. Apparently not. The other lesson I learned is to go the extra mile. I guess it’s entirely my fault to think that coding was enough, that making the .diff patch was enough. It would probably be more than an average user could do, but I realized later that it was quite mediocre on my part. By not going further, I wasn’t really being part of the solution. I was being part of another problem. I’m supposed to be helping to solve a problem, not to create another problem along the way. So next time I try to help with something, I’ll have this little experience to remind me to go all the way.
But that next time will be quite a while from now. The downside of sudden intense spurts of activity like this is that your body has to play catch up. For every action, there is an equal and opposite reaction. So right now I’m currently experiencing some downtime, physically and psychologically… Until then I have to figure out how to get my patch upstream to D3lphin…
Just some small updates, while I'm trying to get off my ass and migrate the site completely to Textpattern.
September 7th, 2007 at 5:29 am
> “But aside from learning about coding stuff, I did come to some realizations, specially in working. It became apparent that I needed to learn packaging whether I liked it or not. [...] But in coding and developing for a project, like a distribution, requires you to comply with certain idiosyncrasies, like Debian packaging [...] I guess it’s entirely my fault to think that coding was enough, that making the .diff patch was enough.”
Frankly, I think that is partially a self-created problem, because you’ve fallen victim to a rather troublesome mindset which is not unique to the Kubuntu community, but one of it’s most prevalent ugly features: Treating upstream as an afterthought, and developing for the distribution project first. Which has created a fair share of problems for both Kubuntu users and the developers behind the applications it ships.
Distributions generally do not have the required talent pool to properly perform code review on feature-level modifications they make to applications they relay to end-users, simply because they are not the developer community behind those applications, and don’t have the same experience with and knowledge about the codebases they work on. In order to ensure proper quality control of such modifications, they must happen within the project producing the application.
Consequently it is absolutely legitimate for distribution-affiliated programers to contribute changes to application projects that they feel are important to their target audience, but they should not make the heavy-weighing mistake of essentially forking those projects and performing development on them in their own trees and only sending things upstream far too late in the game. One of the key strengths of the open source development model is that a development community composed diverse interests can, given successful self-management, generate a more balanced, useful piece of software than any single-minded entity - and isolated, downstream development undermines that. Not only does it result in the sort of quality control problems that Kubuntu has suffered from, but it can also severely damage the relationship between the distribution and the upstream developers.
Additionally, as a contributor to a development project, you generally can get away with trading a .diff with the other, established developers and leave packaging to others. Namely, distributions. That’s their job as organizations, and not development.
Don’t start off on the wrong foot.
September 7th, 2007 at 7:02 am
[...] leggere questo post di Juan Carlos Torres si direbbe che si tratta della “vera verità”, soprattutto anche perché esiste una [...]
September 7th, 2007 at 2:17 pm
First: great that you work on KDE 4 for Gusty and especially on “D{3,o}lphin”
“It did have a feature to compress the whole current folder’s contents, but not individual files.”
compressing only files that are not in a folder is bad and god should kill a kitten every time someone does this *scnr*
put the part with the kitten aside ;). If this feature is not implemented no one will use it. And please be honest: wasn’t you desktop never full of files, just because someone thought packing 400 files directly in a archive and not put them in a folder first is a good idea or not necessary?
September 9th, 2007 at 12:21 pm
@Eike:
I delayed replying in order to understand what you’re trying to say. Frankly, I’m a bit confused, possibly because I’m so new in the game.
Does it mean that downstream devs are incapable of creating “feature-level” modifications?
Does it mean that all feature development should happen/start from upstream only?
Does it mean that distributions are limited to just superficial and minor tweaks and leave “feature-level” improvements to upstream?
Again I’m sorry if I misunderstand the whole message and the situation. I have always thought, perhaps wrongly or too ideally, that unlike a real river, development in FOSS flows both downstream and upstream. Perhaps I was wrong?
September 9th, 2007 at 1:58 pm
Good work Jucato. IMHO, starting off on the wrong foot is better than not starting at all. I’m happy that you got to start with coding for real and hope you can keep the energy up for those restless insanely inhuman wee hours of the morning where good code magically start to appear
September 9th, 2007 at 9:17 pm
Regarding comment 4…
Incapable or limited is a matter of perspective. Any distro can indeed create feature-level modifications if they so choses to. The question is whether that is always appropriate given their resources. The resources may be time, the number of developers, or how well they understand the code they are adding modifications too. The upstream developers likely know the code a lot better, and may therefore produce a patch or modification of better quality.
And no, I don’t think all feature development should start from upstream. Their time is not always enough to do it, and they may suffer from blind spots that end users and/or distro makers do not suffer from. But there is no need to learn to package stuff to get a patch into the upstream source either. That is going about things the wrong way if you ask me, and this ties in into things going both up and downstream.
If you as an individual hacker modifies something, don’t package a new modified version. Create a patch-file and get it to the upstream developers. If it’s good enough and doesn’t break things you may not see, it could be incorporated into the next release. At that point, it would trickle down downstream again to the benefit of everyone and just the Kubuntu/Fedora/whatever users.
Packaging it would make sure that your contributions would (mostly) only be suitable for your distro of choice and its derivatives. Sending the source back upstream does not. Besides, if the patch is not incorporated in the main source you would most likely have to reapply the patch(es) whenever the next version is released upstream.
So yes, development goes both ways. At least ideally, but not always.
A while back there was a good deal of bad blood in the Debian community regarding Ubuntu, because they felt that Ubuuntu was free-loading on the Debian infrastructure and good-will while not sending their patches back upstream as much as they should have. Well, that was the Debian take on the story at least. I didn’t pay enough attention to know how true that point of view is. Still, doing ones utmost to avoid such sentiments to arise in the first place is always good whether it is true or not in that specific instance.
September 9th, 2007 at 11:15 pm
> Does it mean that all feature development should happen/start from upstream only?
> Does it mean that distributions are limited to just superficial and minor tweaks and leave “feature-level” improvements to upstream?
It means that development should happen within the organization created for that purpose, i.e. upstream, where parties with varied agendas convene to assemble a balanced, useful piece of software on neutral ground where they can help out each other and share their improvement to everyone’s collective benefit.
As I wrote in my earlier post, it’s perfectly legitimate for a distribution project to employ developers to work on functionality they feel is important for their target demographic and then contribute it to the upstream project. The obvious example is the Linux kernel, where the majority of the (non-driver) code is being written by developers who are on the payroll of a distribution, but it’s submitted to the Linux community for discussion and integration.
Whenever this hasn’t been done, however, i.e. when distributions have kept kernel features out of the mainline and away from the wider community, their quality has suffered massively from the lack of testing and review, and the distributions have had to put in extra work to keep their additions working with mainline, because they aren’t pulled along by the community. Kubuntu has suffered from similar problems as you know.
A distribution-affiliated developer has two simultaneously wear both hats: He’s a member both of the upstream project and the downstream distribution, and has to responsibly balance out the interests of both. The better distributions have an active interest in making sure that it is as easy as possible for the developers they sponsor to function as a member of the upstream projects, because it gets them better results (and thus people will think of themselves as Kernel developers and not Red Hat developers).
> Does it mean that downstream devs are incapable of creating “feature-level” modifications?
Incapable in so much as they do not have the means to ensure quality, and are likely to turn their development enterprise into an uphill battle as the wider community becomes less and less likely to want to help them out in a bind because they keep development to themselves rather than share it for the benefit of everyone (and no, “but anyone who wants it can grab it from somewhere on our website” does not work).
> I have always thought, perhaps wrongly or too ideally, that unlike a real river, development in FOSS flows both downstream and upstream. Perhaps I was wrong?
The point is that there is a need and an incentive for different parties who use and distribute an open source product to convene in a central space for development, to avoid fragmentation, to keep quality up, to make sure that improvements are broadly available to all end-users. Imagine a KDE world where every distribution kept to themselves rather than contribute to kde.org: The more fragmentation, the harder it gets to support end-users, the harder it gets for application developers to target the platform, the harder it gets to make any forward progress because of the lack of pooled manpower.
I have attended Kubuntu development meetings in the past for various reasons, and have encountered a worrying amount of project members who do not understand that dynamic, who treat upstream projects not as the shared spaces they are, but as wall-socket code supply that occassionally drops some more code into their lap via some mysterious process, and then the distribution gets to do all the important work. Who actively jump at including half-baked patches as differenciation factor from other distributions, and have no interest in bringing those modifications up to speed and actively sharing them with the wider community of the component they apply to.
If I seem unseemingly critical of you right now, it’s because I’m concerned that as a new open source contributor who is loyal to the first project he made a connection with you might be adversely influenced by a work ethic and mindset that does not reflect the real strengths of the open source model.
September 10th, 2007 at 8:47 pm
@nocti: Thanks for your words of encouragement
@Jonas: That issue with Debian is precisely something I want to avoid when working for KDE and Kubuntu.
The reason why I started the patch in Kubuntu instead of upstream is because of the circumstances at that time. First, Kubuntu will be using D3lphin by default in Kubuntu 7.10, which is about a month away. So time is of the essence. I needed to get this patch done, tested, and uploaded ASAP. Second reason would be a lower entry barrier. If I were to submit the patch to both Kubuntu developers and to Marchel Junke at the same time, I bet that the patch would get tested and incorporated quicker in Kubuntu. And again, the time factor played an important part in that. Given more time, I would most probably have communicated with D3lphin developers first. Who wouldn’t want their work (or name) to reach more people, anyway?
@Eike: thank you for your concern. Both KDE and Kubuntu are special to me. You know me and how committed I am to creating an amicable working relationship. I am committed to not cutting out either end of the development stream. I do want to get my patch upstream. But after this, I’m a bit wary of how to proceed. Will he/they eat me?
But I also wonder if all changes should always be pushed or started upstream. What if upstream refuses to accept those patches or new features? What if other downstreams (of the same upstream) do not want/need those changes? Perhaps it’s a case to case basis. Some patches would probably be of interest to upstream and the general whole, while some would most probably be of interest only to a specific downstream. Perhaps implementing some patches on the distro level first also has advantages. Patches would get used and tested first by more users, allowing upstream to observe the changes. Think of it as testing it on guinea pigs and weeding out the bad patches.
>> The point is that there is a need and an incentive for different parties who use and distribute an open source product to convene in a central space for development, to avoid fragmentation, to keep quality up, to make sure that improvements are broadly available to all end-users.
This one I absolutely agree on. But maybe saying feature-level development should only be in upstream isn’t really the solution. Isn’t this the sort of technical and social centralization that Linus is essentially trying to overcome (re: git discussion in core-devel)? Instead of a central space for development, how about a central place for communication? I do understand, though, that distros shouldn’t just keep to themselves. But I guess upstream shouldn’t always expect or demand that development should start or happen on their turf alone, or that things will always go their way. The road goes both ways IMHO. And I will try hard to wear both hats responsibly. Of course presuming I get to wear either hat in the first place.
If there’s anything that all these has made me realize, it’s that a common place of communication is needed, something I thought about in my other post about upstream and downstream. But perhaps I’m to idealistic. The utopian dreams of the young.
September 10th, 2007 at 10:17 pm
> Second reason would be a lower entry barrier. If I were to submit the patch to both Kubuntu developers and to Marchel Junke at the same time, I bet that the patch would get tested and incorporated quicker in Kubuntu.
Which is sort of the point: Applying a patch to a package within a distribution is quicker and easier than it should be, because more likely than not it’s not being tested and reviewed properly. I’m speaking from experience. I maintain several applications within the KDE project, and developers affiliated with distributions as varied as SuSE, Debian and Gentoo send us patches regularly (thanks!). As developers familiar with the code base, we can often quickly spot obvious bugs in those patches that the authors are oblivious to, get them sorted out, and integrate the patches. If the distribution goes straight ahead and applies the patches to their packages directly without getting them upstream first, however, those bugs stay in and it’s the users who bear the cost, and the developers it reflects poorly upon. It’s unfun to learn this the hard way for all involved.
> I do want to get my patch upstream. But after this, I’m a bit wary of how to proceed. Will he/they eat me?
If he’s any good, he’ll be happy about any patch that makes the app better, and do his part to get it in. Ideally an open source developer puts the project above himself and makes sure he’s not being a single point of failure for it. And he should try to make the experience pleasant enough that the contributor will want to come back.
> What if other downstreams (of the same upstream) do not want/need those changes? Perhaps it’s a case to case basis. Some patches would probably be of interest to upstream and the general whole, while some would most probably be of interest only to a specific downstream.
Generally speaking, it’s in the interest of a given project to make their product flexible enough to accomodate a wide range of downstream needs. An easy example is making it straight-forward to swap differently branded visuals by having proper theming facilities, but there are much more technical cases, too, of course. So yes, there is certainly a lot of opportunity for the upstream project to be managed so poorly as to make downstream’s life hard despite downstream having all the best intentions, and the burden of doing the right thing doesn’t solely rest on distributions. More often than not it’s whether it’s able to balance contending needs that makes the difference between a good and a bad open source project.
> Think of it as testing it on guinea pigs and weeding out the bad patches.
I don’t think it’s desirable to abuse users as beta testers if it’s avoidable.
> Isn’t this the sort of technical and social centralization that Linus is essentially trying to overcome (re: git discussion in core-devel)?
No, those are slightly different discussions. That vendor kernel trees are an increasing problem for the Linux industry and generally frowned upon should tell you as much, though. In fact, the kernel community is trying rather hard to make it easier and easier for people to stay close to the mainline in what they ship and deploy; that was one of the main reasons for changing the development model a few years back.
> Instead of a central space for development, how about a central place for communication?
Development -is- communication, to a very large degree. That’s why I’m emphasizing the upstream project as a shared space for all involved parties.
> But I guess upstream shouldn’t always expect or demand that development should start or happen on their turf alone, or that things will always go their way. The road goes both ways IMHO.
You’re still clinging to that “us vs. them” image that you shouldn’t be having in the first place, so it seems I’m not really getting through. Rather than seeing upstream as a foreign country, see upstream as the club house for every one, a shared facility. The mantra is “involve yourself”.
September 26th, 2007 at 2:03 am
[...] I have written some pieces (or rather just one) of documentation and tried to make some patches here and there. But I’m planning on expanding my territories. I’m currently learing C++ in [...]
October 15th, 2008 at 5:26 pm
just a novis can you help to make my first hack
jason