I’m a software guy; I like writing code. Over the last decade, my niche has been a modest amount of coding and a lot of writing about it. The size of my audience suggests that this is a reasonably useful combination. But I’m kind of discouraged and stuck at the moment. I’ll get over it, but the story’s worth sharing.
I have some sympathy with Ed Finkler’s The Developer's Dystopian Future (read the commentary from Marco Arment) and I hope to avoid going where Matt Gemmell has gone; see Confessions of an ex-developer. Matt has decided to become a novelist; I scratch my writing itch in smaller pieces, but I have to say he’s eloquent.
Where I’m stuck · I have a tab open to a page in the Gradle docs: Chapter 50. Dependency Management. It has 63 header-delimited sections organized into 10 top-level subsections, and it’s chapter 50 of 65 (plus five appendices).
Short story: I’m getting an incomprehensible Groovy error trying to do something that should be simple, and following the most-obvious documentation trail led me here.
65 chapters! Java in a Nutshell, which is sort of my canonical example of a ponderous-but-dense essential technical reference (the “Nutshell” part of the title has long since been a Java-community in-joke) has a mere 22 chapters, of which 14 are the API Quick Reference.
Here’s the thing. You’d think that to contribute to an Android app, you’d need a decent grasp of Java syntax/semantics, the Android App Framework, and the problem domain. So to continue making my very-minor contributions to OpenKeychain, my assumption was that I’d have to learn a lot about PGP keys and protocols.
In fact, for every hour I’ve put into actually fiddling with Java code,
I’ve invested another fighting git submodule subcommands, and
now I’m staring at what feels like a thousand-meter-high Gradle rock-face.
I should count myself fortunate that I’m not building a browser-based app; I’d have to budget an even higher proportion of the time staying on top of this week’s funkiest new JS libraries and scrambling to have Wave effects before everyone has them and they’re boring.
Pardon my whining · I’ll figure this thing out and make progress; I may not be a full-time programmer any more, but I’m still reasonably competent.
But there is a real cost to this continuous widening of the base of
knowledge a developer has to have to remain relevant. One of today’s
buzzwords is “full-stack developer”. Which sounds good, but there’s a little
guy in the back of my mind screaming “You mean I have to know Gradle internals
and ListView failure modes and NSManagedObject quirks
and Ember containers and the Actor model and what
interface{} means in Go and Docker support variation in
Cloud providers? Color me suspicious.
Contributions
Comment feed for ongoing:
From: Brandon (Jul 17 2014, at 14:35)
I'm still a youngish developer myself (CS degree in 2009), but development has *always* been like this for me. I'm in the embedded world where you don't get browser funkiness but you get a slightly-broken FPGA image instead, which is about as much fun as it sounds. Actually getting my actual code to work is usually the least of my problems.
I think this broadness does end up contributing more to the slightly mythical "10x" software developer than is realized, though. Sometimes it's not about solving the problem but just getting to the problem in the first place.
[link]
From: Phil (Jul 17 2014, at 15:06)
There used to be a lot of value with Java when it came to cross-platform compatibility. There were always library issues. I loved the stuff that started coming from the Eclipse community as it made building packages easy.
Now as an eclipse user I'm frustrated because even that system is incredibly complex for end-users.
I also wonder at whether Android's differences add to the confusion and the sheer number of permutations and combinations for Java developers.
More work is needed here, and I'd love it if the java and/or android communitues got serious about it.
[link]
From: Ian Ragsdale (Jul 17 2014, at 15:18)
I think this sort of thing is a big part of what's driving people to Go.
While the language itself is missing a good number of the newer niceties, it's nicely compact, and pretty much all you need to know to develop in Go is Go itself.
The build system, system for installing libraries, project structure, and all sorts of practical matters like that are built in and/or standardized, so everyone is using the same conventions and it's easy to jump into.
I think the concept of going pretty basic with the language but spending a bunch of time thinking about the practicalities of working with the language has really paid off for them. It just feels easier than many other languages once you know how to think in Go.
[link]
From: Loyd Blankenship (Jul 17 2014, at 15:55)
This is, sadly, the story of my life as well. For every hour of productive coding, it feels like there are 4 hours of trying to solve insane permission problems, or deal with module dependencies, or any of a number of other problems with the server or development tools or routing issues or... you get the idea. And the bigger your 'stack', the more time you will spend solving problems getting the pieces of the stack to behave.
[link]
From: Serge Masse (Jul 17 2014, at 15:58)
Simplicity seems to have lost its popularity, particularly for development tools lately. I suspect that contributing factors are the drive to show off and brag that you can use a complex tool, and also the drive to build tools with more functions, not realizing that a 'better' tool is often one with fewer functions, or at least easier to use for common tasks. I still think that Ant in an IDE is better for me than Maven or Gradel.
[link]
From: Jordan Zimmerman (Jul 17 2014, at 16:57)
This happened to me recently as well. I don't get all the hype behind Gradle. I don't know Groovy nor do I want to. Even if I did, I don't want to learn the Gradle DSL. Maven was hard enough to learn. I don't want to go through a new, different learning curve just to get my builds done.
[link]
From: Jordan Zimmerman (Jul 17 2014, at 17:19)
I don't know where Ian is coming from. I've just been trying Go and the dependency management system is horrid. There are 5 or 6 competing tools and no standardization at all. Google seriously dropped the ball on this.
[link]
From: Liam Quin (Jul 17 2014, at 18:59)
Full Stack seems to me a thinly veiled attempt to impose JavaScript on the back end (node.js etc.) rather than to have cooperating teams agreeing on protocols or APIs and use whichever tools best fit to the task. Maybe I'm cynical. But you can't really expect one person to be fluent in everything needed these days, from TLS and ssh certs through to accessible HTML forms and responsive design ads, it's making an insane learning curve.
[link]
From: Ian McKellar (Jul 17 2014, at 21:56)
The thing that's telling about your story is that it's gradle that's tripping you up. There's been a trend increasing over the couple of decades that I've been writing software to build increasingly complicated software to "make things easier" for developers - that don't tend to actually make things easier. The way we *used* to compile Java programs was:
javac *.java
Now the best way is to run a shell script to run gradle which runs a groovy script to process your tree of sort of declarative scripts. And this definitely seems better than ant. But it's so complicated that if anything goes wrong you have to learn a whole new programming language.
This is where I'd normally start ranting about how domain specific languages are a deep trap, but it's late and I already spent most of my day trying to work out why my debugger was seg-faulting...
[link]
From: Kragen Javier Sitaker (Jul 17 2014, at 22:50)
I understand where you're coming from; this is one of the things that has always driven the cycle of moving to smaller, simpler platforms — the old platforms become too complicated to understand fully, and so new developers pick new platforms. I think Plauger in one of the "Programming on Purpose" or Bentley in one of his Pearls columns tells this story about the moves to minicomputers and then microcomputers. And today you have Flash developers who don't know what a subdirectory is. You yourself have been a great contributor to the move to Dalvik on Android, which several of my officemates here are benefiting from already. (Hopefully Android is a smaller, simpler platform than regular Linux, although I'm not sure yet.)
I think, though, that a lot of what you're experiencing is a result of Java, not of programming in general. It's very reminiscent of what a lot of people have said about their experience before switching from Java to Python.
My own defense mechanism has been to try to build smaller, more comprehensible versions of things. This has so far been unsuccessful.
Best wishes.
[link]
From: ofr (Jul 18 2014, at 00:55)
Thanks for that, it struck a nerve. Being a full-stack developer means that you not only have to grasp Gradle/Maven but also the proliferation of build-tools for Javascript/CSS/SomeNewLanguageToBeCompiledToJS etc. plus an ever growing heap of frameworks.
This is getting worse with ever faster "innovation" cycles. One may say, just stick to the tools you know, but as you observed, this is just not possible when you are going to use certain libraries or tools that trip you up.
[link]
From: Scott (Jul 18 2014, at 03:07)
I've developed a few things for Android using Eclipse and not had too many problems. A couple days ago Udacity announced an Android developer class, so I went to check it out and see if I'd pick up any thing new. It requires Android Studio rather than Eclipse, OK, so I'll get that. After a few hours of downloading very large sets of device and system plugins and the program growing to over 8GB in size, I am ready to launch and create the default project. Of course in the 1 day since the program launched, Android Studio has been completely redesigned to have a different interface than what is shown in the class. In addition, just creating a default project in Android Studio results in hundreds of errors from something called Gradle, which I don't use. 10 hours later I was still trying to fix the problem by researching what Gradle is and what it should have to do with creating a default project in Android Studio before I can proceed. During this time all the Udacity forums give a message that they are down for service. I finally give up on this and move on. What a basket case this stuff is.
[link]
From: Ian Ragsdale (Jul 18 2014, at 08:37)
@Jordan Zimmerman: "go get <url to dependency>". Once you've got them, you vendor them. That's it.
A lot of extra tools are popping up, but I find them to be unnecessary and don't use them. If you've got a build that's working, check in your dependencies. If you want to update, "go get" again, run your tests, and commit the newer version.
[link]
From: Rui Carmo (Jul 18 2014, at 08:44)
That underlying complexity is why I was drawn to Clojure - besides the relative simplicity of the language, Leiningen is a simple, straightforward build tool that you hardly need to think about, and using Java libraries is usually a single project config line away (Leiningen will fetch the jars and handle everything for me).
Go is, well... OK, provided you use godep or something similar to vendor dependencies, but I'm not completely sold on it yet (I still prefer Python and an introspection-enabled REPL on the side). It's an acquired taste, I suppose.
As to Java, a recent experience in hammering out a custom Android app in a few evenings made me loathe Eclipse, Android Studio, etc. They're simply useless at providing you with any useful context that doesn't involve at least fifteen alternatives for refactoring code in ways that you really don't want to, or inane "helpers" that are utter failures at cutting through all the silly abstractions one has to put up with in modern Javaland - especially when trying to focus on your program's underlying logic.
After a brief stint into vim-land and Ant to regroup and get a sane fallback environment, I just went with NetBeans to finish that Android app. Although I've been using IntelliJ recently, NetBeans Android support just worked, including fetching dependencies from Maven, debugging, and (much to my surprise) profiling (although that was less integrated).
In general, there are simply too many layers involved in modern toolchains for them to be even remotely sane (I have a particular beef with the npm/flavor-of-the-month JS crowd, but let's keep it civil...)
So these days I tend to stick to languages/runtimes that don't impose too much ceremony (except for playing in C# inside Unity3D, but that's another thing altogether) and, when doing web development, prefer my client-side JavaScript plain and unfettered (even though I'm very partial to ClojureScript's reagent wrapper for React).
My litmus test these days is whether or not I can actually build something using nothing but vim, an hour or so of reading docs and a small (i.e., less than 6) set of simple commands.
[link]
From: Meower68 (Jul 18 2014, at 09:07)
Groovy is mostly just Perl for the JVM. If you've never wrapped your head around Perl, Groovy is kinda ... WTF?
I'm in my 40's. I don't have time to do deep dives into different platforms. I need it to work, so I can get this working and move onto the next 6 things in the pipeline.
Unfortunately, for every hour I spend coding I spend at least one more hour wrestling with everything else. Getting that working with this balky database that we have to use (given enough time, ALL databases are balky). Getting git to include the right stuff, on the right branch, so it can be merged into the main package. Getting the build right on the CI tool. Jumping through the hoops to get that deployed to the testing tier. Updating the wiki-based docs so the QA team knows what is and isn't within scope of testing on this project. Dealing with the latest change to the requirements or the latest change to the service/framework that this project depends on. Because one change to that results in a cascade of all of the above.
Software development, especially "enterprise" web development, is a part-time job. Wrestling with all of the interlocking pieces/parts which surround it is the OTHER part-time job.
[link]
From: Markus Kohler (Jul 18 2014, at 12:14)
Dependency management is probably *the* hard problem for larger software projects. Java and other static languages more or less enforce you to manage the dependencies properly. Unfortunately there is no standard way in Java and Java does not easily allow you to use the same class or "module" within one process.
Node.js can load modules with different versions into one process. It seems to me that this is mainly because of Javascript not being statically typed. I don't think that really solves the problem. I think it is rather more common to have smaller projects with Node.js/Javascript and therefore unmanaged dependencies are not such a big deal. Note that even with Node.js you better do not use too many versions of the same module because you would have to do security audits for all of them.
GO also "only" offers a different trade off,by making it easy to produce standalone executable. That also doesn't solve the problem in the long run. You will end up with potentially a lot of large binaries. I also recently started to work with Gradle and I think the problem with it is more that it supports everything e.g. a smooth migration path from ant, maven, ivy etc.
You might have to learn groovy also to understand the error messages, but I think that is still better than an external DSL, because you can debug groovy scripts. whereas debuggin ANT scripts for example is a real pain.
[link]
From: Hans Dockter (Jul 19 2014, at 01:30)
As the founder of Gradle, I'm replying to the Gradle part of your posting.
With Gradle we want to achieve two things. To make it scalable as well as to make it approachable.
Gradle is used a lot at organizations like LinkedIn. LinkedIn has a huge software stack with 1000+ developers working on it. This is a scalability challenge for a build system. Not just because of the size but also of the inherent complexity of this software stack. It comprises a variety of languages, it has very complex multi-platform dependency management requirements, the collaboration workflow is also complex. LinkedIn is nowadays capable of doing two releases per day for there whole stack! There is just no way to solve those challenges with a flexible but overly simplistic tool. They had an Ant build before they switched to Gradle. It was a dead end.
The reality of nowadays software stacks is that they are multi-platform and complex. Even just within Android you have the NDK. Any enterprise stack has today at least Java, Android and iOS. Quite likely also JavaScript, possibly some Scala, ... As a build tool we can't and won't challenge that reality. We simply embrace it. On top of being multi-platform, users expect nowadays much more frequent releases. All that puts scalability pressure on the build infrastructure. You don't have to be LinkedIn to feel it. I would say that by average any 50+ developer team that is serious about releasing high quality software frequently has this challenge. Those are also the environments where most professional software developers are working in. It is our mission to be the build platform of choice when it comes to the challenges of those teams. This is one of the areas where Gradle shines and we continue to invest heavily into enhancing its scalability capabilities.
Besides being scalabe, a build system also needs to be approachable. This means good documentation, error reporting and tooling as well as hiding unrequired complexity. You pointed out two areas where Gradle is lacking.
One is documentation. Java in a Nutshell was written by David Flanagan, an absolutely outstanding technical author. In addition it had an excellent O'Reilly editor with Mike Loukides. It is not quite fair to compare them with a bunch of open source devs trying to keep the documentation up to date while we are coding Gradle :). The Gradle user guide could be heavily revised. Every tiny plugin has its own chapter. We could document the plugins somewhere else, ect.. We will do something about it this year. In addition there are really good Gradle books out there like Gradle in Action by Manning. That would be a fairer comparison.
The other area you pointed out are error messages. I agree with you. Although Gradle error messages can be spot on, it is too often the case that they are not very helpful. Right now we are laying the foundation to enable Gradle to provide much better error messages. So expect this to start improving signigicantly over the course of the year.
A key feature of Gradle in regard to approachbility and hiding unrequired complexity is our extendable toolkit/DSL. Look at this Gradle based Android build script which describes a very complex workflow, although it is extremely simple and expressive: https://gist.github.com/tobrien/5376666 Writing this with Ant, Shell scripts or Maven would require 1000's line of code.
We are working on improving the tooling support. This includes the interaction with Android Studio, where I'm confident that this is going to evolve as the best truly integrated development environment on the planet when it comes to buidl and CI integration. It also includes providing better support for Eclipse. Tooling is an important part for hiding unrequired complexity. Our goal is to make Gradle approachable to people in a way that they don't even need to be aware that Gradle build script exists for standard scenarios.
Hans
--
Hans Dockter
Founder, Gradle
http://www.gradle.org, http://twitter.com/gradleware
[link]
From: Aaron (Jul 19 2014, at 18:36)
I just had to learn maven. Same experience, but not unusual. Learning new tech always has this kind of frustration. I suspect the deeper issue has to do with the dependencies people are willing to create for some simple functionality.
[link]
From: Kieran Simpson (Jul 21 2014, at 01:39)
I feel your pain with git submodules. This is another area where I think Git made it far harder than it needed to be. Mercurial's sub repo's are a lot easier and nicer to deal with (hint they just work ;) )
[link]
From: Chris E (Jul 21 2014, at 09:20)
I don't think it's much to do with growing old.
Somehow people got the idea that everyone could develop libraries and frameworks, and so everyone did. As they were as good at writing documentation as they were at writing frameworks other people found it easier to write their own competing library than what was there already - and then document it badly.
This occurred at every level - including things that should have been relatively simple to use. Including containers, SCMs and build systems.
As a poor benighted 'user', sooner or later you need to get library Foo that uses build system X to work with library Bar that uses build system Y. There is a mechanism to invoke Y from X, but it relies on either an old version of X or an experimental branch of Y. The mechanism is badly documented, and so it goes. Eventually you find a solution which involves yet another library/a monkey patch you don't understand, which you cut and paste to solve your local issue.
Bottom line: You could take down large parts of the IT industry by DDOSing stackoverflow.
[link]
From: J.D. (Jul 22 2014, at 12:38)
@Hans Docket - I was really encouraged by the very last statement of your post ("Our goal is to make Gradle approachable to people in a way that they don't even need to be aware that Gradle build script exists for standard scenarios"). That is exactly what we need.
I understand that it is hard for scenarios such as LinkedIn but I would guess that in the 80/20 picture they fall in the 20% range and the frustration being expressed is by the 80%. Small teams and Indies simply want to be able to build our apps in a simplistic manner.
I agree with Chris also who stated something along the lines of...
"sooner or later you need to get library Foo that uses build system X to work with library Bar that uses build system Y. There is a mechanism to invoke Y from X, but it relies on either an old version of X or an experimental branch of Y. The mechanism is badly documented, and so it goes. Eventually you find a solution which involves yet another library/a monkey patch you don't understand, which you cut and paste to solve your local issue."
That one statement kind of sums up todays development experience and its sad - very sad. I came across this article as I was searching out tutorials on Git. Why is it so hard for tool makers to understand that all I want to is set up a repo and check in and check out? That surmises 95% of the functionality I need on a day-to-day basis.
Why the @*$% should I have to drop into a cmd line and type out a commit command for every changed source file? Are you f'king kidding me? I'm sure there is some other command that can be used to commit all changes but again why is there no IDE integration that simply lets me right click on my project root and then select "Commit changes"?!?!
And speaking of command line, why is there such a fascination with the command line amongst today's script kiddies. I gather its because they came along in a time where the GUI and wizard interfaces were common place to even average users and working on the command line gives them a "superiority" complex over the average user.
I've done enough command line work from the early and mid 90's (when there were no or few GUI options) to where I don't need the cmd line to validate my self-worth as programmer.
[link]
From: Dan Davis (Jul 22 2014, at 19:27)
I've somewhat recently left my startup and systems software development for an architecture position in web software development. Where you are discouraged, I'm excited because it is all new. There is an awful lot other than coding, but that is because you are good. A junior programmer may never get as good as Tim Bray, and so he must ask questions like whether to use Gradle or Maven of someone. Of course I've had those moments where I look at Rakefile and think - it's a small project; I'll just use make. But when I have to tell someone what to use, I'll do the research.
[link]
From: Henry (Jul 23 2014, at 08:43)
The good thing is, though, while developing software gets more and more complicated, there are people out there who try and simplify it as much as possible. This results in the power that is brought by the increased complexity as well as the ease of use brought by simplifying tools. Take for example:
https://kite.io/
https://www.nitrous.io/
bowery.io
[link]
From: kirk (Jul 23 2014, at 10:18)
Gone are the simple days where make was your most complex tool.
Every time one pokes ones head into a new area there is hours of sifting thru crappy documentation (almost as bad as I'd do) .. I lost hours of my life trying to figure out what "exploding" an android screen file was all about. At then end I thought "open" would have been a better term to use !
[link]
From: Ken (Jul 23 2014, at 13:34)
Hans said: "Any enterprise stack has today at least Java, Android and iOS. Quite likely also JavaScript, possibly some Scala, ... As a build tool we can't and won't challenge that reality. We simply embrace it."
Our common build tools *choose* to embrace that. I've worked on enterprise build systems that needed to do all kinds of crazy things not supported by any build system. The answer was always to use the "run arbitrary command" build feature, and not clutter up the build tool with complex things that 0.01% of users will ever need.
I'm not sure the reason for modern build systems being so complex. Maybe people making build systems today don't seem to recognize the real cost that these features have for users. Maybe they all work at places like LinkedIn and Google with super high complexity already.
Every other case I can think of where we chose to "embrace reality" was merely as a stepping stone. My C++ compiler in 1998, for example, supported inline 8086 assembly code, and probably still does today, but nobody has to even think about that feature to write a C++ program today. Even the programs I worked on that needed that feature in 1998 had moved off it in a year or two.
[link]
From: WilliamDee (Jul 23 2014, at 15:29)
@J.D. - while this is only semi-relevant to the discussion as a whole, it does address your remarks about a "right-click -> git -> do something" in an IDE.
I use NetBeans IDE 8.0 and there is a right-click -> Git option. This doesn't cover everything, you do need to use the command-line for some stuff (initial setup for example).
Cheers,
WilliamDee
[link]
From: len (Aug 01 2014, at 07:28)
Churn: why I hate the GUI Migration where new replaces old without a single bit of real functional advancement. Churn is to software what the new kid in town is to music: not much innovation; just a fresh face. Or to paraphrase Hank Williams Jr sang, "I like to code stacks I've never had."
I spent a year solving the many problems of marrying two XML languages that spanned years of different technical approaches using an obsolete stylesheet language, an editor that locked up on the incompatible operating system, an obsolete VB/VS development environment that I had to bring myself and some decent code editors. With pluck and many many free hours of overtime, I succeeded. Then I was fired so the in charge could take credit and continue his piggy ways claiming it didn't work when demonstrably it does.
Welcome to an era when FOX News asserts it actually is and that style of accusing, disinforming and outright lieing has permeated the management suites where to be successful one must swear fealty to the animal farm. Here's the funny bit: the pig demanded copies of the editor I wrote to make all of that work but didn't know he needed the source code and didn't wait until someone else was trained. So for the next week after being marched to the door being cussed, his minions sent me emails asking how to use it. Oopsie.
I'm going back to playing open mic nights and giving songs to Google so they can charge companies for ads but pay me nothing. And you know what, I feel fine. Great days. I have 20k of running code. Worthless in the churn but... running code still talks. :)
[link]