In 2014, it’s not OK if it’s hard for a developer to run a simple program from the command line.
I wrote some code to connect Keybase and OpenKeychain, and plan to write more. Since it’s in an Android app the code was in Java and it occurred to me that since (so they say) other people use Java too, those people might be able to use it, so I’m working on that. But it shouldn’t be so hard.
I had the Java code already working and in production, so I copied it out of OpenKeychain and made a new project and wanted a smoke test which I thought I ought to be able to run from the command line.
Only I couldn’t. First of all, it took me forever to figure out the java command-line incantations to tell it that it needed my project’s class files and the json.org library (which I’d already downloaded so I could compile the sucker). Yeah, I used to know that stuff ten years ago, but there really shouldn’t be any complexity here.
After I’d figured it out, it refused to run because I wanted to, you know,
fetch an https:// URL, but no, Java 7 won’t do that because,
well, I don’t actually know, there’s some incomprehensible botch in some certs
table. This is Java 7 out of the box, OS X out of the box, everything out of
the box, and it doesn’t work; apparently you can fix it with
low-level system security rejiggering, but I just want to fetch a fucking
HTTPS link
and I shouldn’t have rejigger anything. By the way, curl can fetch that same
URL just fine from the command line, the one that Java can’t.
So the easier solution was to create a stub Android project called KeybaseTest with an empty TextView into which I could do the moral equivalent of:
printf("%d %s", https_status, https_message_body);
Of course, to try this out, I had to plug an actual physical Android device into my physical Mac that should have been able to run this simple straightforward code, then use adb incantations to debug. Remember, this is the easy way to run Java code.
Dear Java: I can run Ruby and Python and Go and JavaScript and C code from the command line on my Mac. If I can’t run you, that means you’re broken.
So for the moment, my Keybase Java client is structured as an Android project because near as I can tell, in 2014 that’s the only straightforward way to run a Java program that wants to use the Internet on a device that I own.
Contributions
Comment feed for ongoing:
From: Nelson (Jun 20 2014, at 08:47)
To be fair, some of the fault is with Apple's Java distribution. Oracle gets a lot of blame for packaging Java badly too.
[link]
From: Nick (Jun 20 2014, at 10:02)
@Nelson: Apple doesn't package Java anymore. When you install Java, it comes from Oracle.
[link]
From: Henry Story (Jun 20 2014, at 10:33)
For the command line consider using Scala, it's as concise as Ruby, Python, etc, but on top you also get type safety if you want, which is really helpful for documentation and writing code quickly, and even more for writing good asynchronous code.
With a bit more knowledge of Scala you can then use much more interesting URL libraries such as Spray.
http://spray.io/documentation/1.2.1/spray-http/
I wonder if Java 8 fixes your certificate problem.
[link]
From: Matt (Jun 20 2014, at 10:39)
Tim,
Java can be a little verbose but I just tested a simple program to fetch an HTTPS URL in Java. Feel free to give it a try.
Example:
mymac:zed matt$ cd ~/Downloads/zed
mymac:zed matt$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)
mymac:zed matt$ javac -version
javac 1.7.0_55
... paste java source (below) into UrlManager.java file...
mymac:zed matt$ javac UrlManager.java
mymac:zed matt$ java UrlManager https://www.google.com
Connecting to URL https://www.google.com
Response code: 200
HTTP response information:
Content-Type: text/html; charset=ISO-8859-1
Content-Disposition: null
Content-Length: -1
File Name: www.google.com
Reading from connection............done
Downloaded 14352 bytes of data.
Data cached to /Users/matt/Downloads/zed/www.google.com
matt
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.HttpURLConnection;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
public class UrlManager {
public static void main(String[] args) {
try {
if (args.length < 1) {
System.out.println("UrlManager <url> [output folder]");
System.err.println("Please provide a URL to fetch.");
System.exit(-1);
}
String url = args[0];
if (url.trim().isEmpty()) {
System.err.println("Please provide a URL to fetch.");
System.exit(-1);
}
if (!url.toLowerCase().startsWith("http://") && !url.toLowerCase().startsWith("https://")) {
System.err.println("URL must start with http:// or https://");
System.exit(-1);
}
File cacheFolder = null;
if (args.length >= 2) {
String outputFolder = args[1];
if (outputFolder.trim().isEmpty()) {
cacheFolder = new File(".");
System.out
.println("Output folder (argument 2) not specified. File will be cached to the current folder: "
+ cacheFolder.getAbsolutePath());
} else {
cacheFolder = new File(outputFolder);
if (!cacheFolder.mkdirs()) {
System.err.println("Failed to create output folder " + outputFolder);
System.exit(-1);
}
System.out.println("Files will be cached to folder: " + cacheFolder.getAbsolutePath());
}
}
fetchUrl(url, cacheFolder);
System.exit(0);
} catch (Throwable t) {
t.printStackTrace();
}
}
private static void fetchUrl(String url, File cacheFolder) throws IOException {
System.setProperty("https.protocols", "SSLv3");
URL u = new URL(url);
HttpURLConnection con = null;
try {
System.out.println("Connecting to URL " + u.toString() + "\n");
if (url.startsWith("https")) {
con = (HttpsURLConnection) u.openConnection();
} else {
con = (HttpURLConnection) u.openConnection();
}
int responseCode = con.getResponseCode();
System.out.println("Response code: " + responseCode + "\n");
if (responseCode == HttpURLConnection.HTTP_OK) {
String fileName = "";
String disposition = con.getHeaderField("Content-Disposition");
String contentType = con.getContentType();
int contentLength = con.getContentLength();
if (disposition != null) {
// extracts file name from header field
int index = disposition.indexOf("filename=");
if (index > 0) {
fileName = disposition.substring(index + 10, disposition.length() - 1);
}
} else {
// extracts file name from URL
if (url.contains("/")) {
fileName = url.substring(url.lastIndexOf("/") + 1, url.length());
} else {
fileName = "index.html";
}
}
StringBuilder b = new StringBuilder("HTTP response information:\n");
b.append("\tContent-Type: ").append(contentType).append("\n");
b.append("\tContent-Disposition: ").append(disposition).append("\n");
b.append("\tContent-Length: ").append(contentLength).append("\n");
b.append("\tFile Name: ").append(fileName).append("\n");
System.out.println(b + "\n");
// opens input stream from the HTTP connection
InputStream inputStream = con.getInputStream();
// opens an output stream to save into memory (for writing string)
// ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
// opens an output stream to save into file
File cacheFile = new File(cacheFolder, fileName);
FileOutputStream outputStream = new FileOutputStream(cacheFile);
int bytesRead = -1;
byte[] buffer = new byte[4096];
System.out.print("Reading from connection");
while ((bytesRead = inputStream.read(buffer)) != -1) {
outputStream.write(buffer, 0, bytesRead);
System.out.print(".");
}
System.out.println("done\n");
outputStream.close();
inputStream.close();
// print URL data as a string to the console
// System.out.println("Downloaded " + outputStream.size() + " bytes of data.\n");
// System.out.println("Downloaded data:\n" + new String(outputStream.toByteArray(), "UTF-8"));
System.out.println("Downloaded " + cacheFile.length() + " bytes of data.\n");
System.out.println("Data cached to " + cacheFile.getAbsolutePath());
} else {
System.out.println("Server response not " + HttpURLConnection.HTTP_OK + ". HTTP code: " + responseCode);
}
} finally {
if (con != null) {
con.disconnect();
}
}
}
}
[link]
From: Paul Houle (Jun 20 2014, at 10:39)
This is why I wrote this open source package
https://github.com/paulhoule/centipede
you can create a new project with from a maven archetype that has log4j and Spring set up for you. A centipede application contains a bunch of little command line applications that are defined simply by writing classes that implement CommandLineApplication; Spring automatically finds all of these and makes them available.
Centipede also defines a per-user configuration mechanism that means you have no excuse to hardwire database passwords into your version control.
[link]
From: Ned (Jun 20 2014, at 10:55)
Nelsen: What exactly is Apple's fault here? Is the stock Java from Oracle working, but Apple's own distribution hosed HTTPS somehow?
Or do the stock versions of Ruby, Python, or Go have this problem, too, and it's only through Apple's own packaging (even though Apple hasn't packaged Go) that they work for this at all?
[link]
From: Tom Purl (Jun 20 2014, at 11:08)
Maybe you could do this with Groovy in the future - it's very close to Java at a low level:
* http://www.infoq.com/news/2014/06/groovy-android
[link]
From: Rob (Jun 20 2014, at 11:09)
Actually Java 7 on a Mac is provided by oracle.
[link]
From: Kevin H (Jun 20 2014, at 11:37)
Somewhere among Bowery, Docker, Bitnami et al, I think there's gotta be a pre-configured stack available to reduce your pain on this.
Of course finding the right needle in that haystack can be a challenge, but I think you are in a great position to make a lazyweb request and get an answer in pretty short order.
[link]
From: Phillip (Jun 20 2014, at 11:45)
Just use Groovy instead... problem solved.
[link]
From: None (Jun 20 2014, at 11:46)
It's not a botch.
You have two choices to the following question: when you connect to a server over https, do you verify the certificate or not ?
If you verify the certificate, then you need the certificate (or one of its signers) in your certificate store. This certificate is not in your store (is it self signed ?).
If you do not verify the certificate, you are open to man in the middle attacks.
Java defaults to the more secure first behaviour. Either you add the certificate (or one of its signers) to your certificate store if it is not already present, but most of the well known ones should already be there, or you override the certificate verification and live with the dangers.
[link]
From: MVK (Jun 20 2014, at 12:33)
Java's not good for simple utilities. The dependency management (CLASSPATH stuff) is pretty unwieldy for simple tools. But it's rigidity is what makes it so much better for high traffic, high throughput applications requiring 100% reliability than the dynamic languages that auto-magically take care of all that. And you can make Java far more dynamic if needed.
As others have said, there are tools to make Java easier for simple stuff. But that's not Java's sweet spot. If your project isn;t big enough to account for time to make sure you are linking to exactly the libraries, versions, etc... that you need, it's not right for Java.
At the same time, there's a reason why most Java jobs I get called for these days are ports from Ruby. Ruby's easy but not robust.
[link]
From: Lmm (Jun 20 2014, at 14:03)
So don't run the java incantation by hand. Use something like mvn exec:java, or right-click the class in your IDE. The low-level "java" program is and should remain as simple and robust as possible; fancy logic can be handled by higher-level tools.
(Yes this assumes you have a bare minimum of development tools installed, but you're only in this situation because you're trying to run development code. If you're running a released tool it should already be packaged it up as an executable jar (using something like the maven shade plugin), and then it's as simple as java -jar program.jar arguments).
As for the certificate store... meh. Java is the rare language that is actually willing to sacrifice some ease-of-use in the interests of security, and I believe that's the right tradeoff. Yes, using HTTPS is hard, but that's better than it being easy and insecure.
[link]
From: peter (Jun 20 2014, at 21:31)
Nah, you're just shit for brains.
Stick to scripting languages noob.
[link]
From: Ivan Ristic (Jun 21 2014, at 01:29)
BTW, the reason you can't connect with Java 7 to https://keybase.io is because Keybase have (mis)configured their server so it doesn't offer any cipher suites Java 7 could use.
https://www.ssllabs.com/ssltest/analyze.html?d=keybase.io&s=54.84.133.185
You can see this in the SSL Labs simulator, in the bottom part of the report. Clicking "Java 7" will show the cipher suites available by default.
[link]
From: Dave (Jun 23 2014, at 13:06)
It's pretty easy to package your class and any dependencies in a runnable jar. Most IDEs including Eclipse will do this, or Maven will do it from the command line. In Eclipse it's "File => Export => Java => Runnable JAR file". Then execute with "java -jar yourfile.jar args".
[link]