Dogfooding, or eating your own dog food, is a practice that all product developers should implement all the time. According to wikipedia:
When we run the above program, we’re getting the following output:
The output generated from this is:
Instead, it was like this (which works the same way):
I thought to myself, do I really need all that string concatenation? I already have this
That would be cool, too. I could show that our DSL API and our parser can do the same things. But I didn’t want to show the DSL API in this blog post. It would only distract from my main point. I could, of course, parse the queries explicitly:
But why not just add convenience API that does this for me?
Where
That’s mere convenience. It is not needed. But it is very useful:
How many times have we written this kind of code? N times (and a big N, too)
How many times have we enjoyed writing this code? 1 time
How many times too many have we written this code? N + 1 times.
Dogfooding is a very good way to make sure this kind of convenience our users love us for will make it into products much earlier on. Because we, the vendors, are the first users, and we hate writing this silly code all the time. And who is a better first user to implement cool new features for, if not ourselves?
Dogfooding, occurs when an organization uses its own product. This can be a way for an organization to test its products in real-world usage. Hence dogfooding can act as quality control, and eventually a kind of testimonial advertising. Once in the market, dogfooding demonstrates confidence in the developers’ own productsI’ve recently started delivering this talk about API design at conferences, where I mentioned dogfooding as an excellent approach to make sure the user experience of your API is great: The more you use your own API, the better it gets from a UX and usability perspective.
Dogfooding via tests
We do a lot of dogfooding ourselves, inevitably, as we write tons and tons of tests for jOOQ, to make sure jOOQ works correctly on all the currently 26 RDBMS that we support. Writing a test for new API means we have to immediately use the new API for the first time. This helps discover the first usability problems. But there are even better ways:Dogfooding via significant new functionality
Another great way to dogfood is to create significant new features. For example, this new schema diff tool that jOOQ 3.13 will ship with (#9425). It is part of a bigger project that we call DDL interpretation, where we start implementing the DDL part of a database by maintaining an up to date database schema depending on a stream of DDL statements. This will be part of a variety of improvements and value propositions in the area of database change management / SQL migrations. Recently, we’ve published a post about an improved Liquibase integration, which goes in a similar direction. The schema diff tool is something many database products are offering. For us, it is relatively simple to implement as we:- Support a variety of schema meta representations
- Support a lot of DDL syntax
- Support exporting schema meta representations as DDL
org.jooq.Meta. That type represents your database. Historically, it just gave jOOQ-style access to JDBC’s java.sql.DatabaseMetaData. But over time, we also started supporting exposing generated jOOQ code, or XML files as org.jooq.Meta.
With jOOQ 3.13, we’ll support interpreting arbitrary DDL (parsed or manually created using the jOOQ API), or also Liquibase XML files to create such a org.jooq.Meta representation. Which can then be turned again into DDL using the new Meta.ddl() method:
System.out.println(
ctx.meta("")
.apply("create table t (i int)")
.apply("alter table t add j int")
.apply("alter table t alter i set not null")
.apply("alter table t add primary key (i)")
);
create table t( i int not null, j int null, primary key (i) );That’s already cool – we can create a snapshot of our schema at any point of a set of database migration scripts. Can we also do the inverse? We can, with the new diff tool:
System.out.println(
ctx.meta(
"create table t (i int)"
).migrateTo(ctx.meta(
"create table t ("
+ "i int not null, "
+ "j int null, "
+ "primary key (i))"
))
);
alter table t alter i set not null; alter table t add j integer null; alter table t add constraint primary key (i);This is part of an exciting new set of DDL / migration features, we can hardly wait to publish with jOOQ 3.13 (around Q1 2020), as we believe it will greatly help with your existing database change management solution – perhaps as a Flyway or Liquibase plugin. What does this have to do with dogfooding? While developing this feature, we’ve discovered numerous missing features, which we also implemented for jOOQ 3.13:
- #7752: Our current meta model for sequences does remember flags like
MAXVALUE - #9428:
Meta.toString()could just callMeta.ddl(), the DDL export. This is what I’ve shown above, very useful when debugging with small schemas! - #9433:
Meta.equals()andMeta.hashCode()was not yet implemented, as this didn’t make sense, historically. - #9434: Our current DDL export should support reordering objects in alphabetical order for the export. This is useful for equality checks, text-based diffs, etc.
- #9437: We don’t support
ALTER SEQUENCEstatements that allow for modifying sequence properties likeMAXVALUEyet. - #9438: The current emulation of
ALTER SEQUENCE .. RESTARThard codes restarting the sequence at 1, when in fact, it could be some otherSTART WITHvalue - #9440: We’ll need a synthetic syntax to drop unnamed foreign keys.
Meta.migrateTo(Meta) method. The migrateTo() method will be the one highlighted in the release notes of 3.13. But the little things above are what ultimately makes jOOQ so useful – the many little things that were discovered while dogfooding and that help everyone, not just the users that use the migrateTo() method.
Dogfooding via blogging and documentation
But, perhaps even better than when implementing new features is blogging or documenting API. When blogging or documenting, the vendor / maintainer has to put themselves into the position of the user, in particular, the first time user of a specific API. Imagine, after writing all this code that I as an author love, I have to reset my experience, and pretend I don’t know what to expect. Then, sit down, and write a really good and simple example using the API that I wrote. The first example I came up with was not like this (as that API didn’t exist yet):
System.out.println(
ctx.meta("")
.apply("create table t (i int)")
.apply("alter table t add j int")
.apply("alter table t alter i set not null")
.apply("alter table t add primary key (i)")
);
System.out.println(
ctx.meta("create table t (i int);\n"
+ "alter table t add j int;\n"
+ "alter table t alter i set not null;\n"
+ "alter table t add primary key (i);")
);
Meta.apply(Queries) method, which would add even more value. But then, my code would look like this:
System.out.println(
ctx.meta("")
.apply(queries(createTable("t").column("i", INTEGER)))
.apply(queries(alterTable("t").add("i", INTEGER)))
.apply(queries(alterTable("t").alter("i").setNotNull()))
.apply(queries(alterTable("t").add(primaryKey("i")))
);
System.out.println(
ctx.meta("")
.apply(ctx.parse("create table t (i int)"))
.apply(ctx.parse("alter table t add j int"))
.apply(ctx.parse("alter table t alter i set not null"))
.apply(ctx.parse("alter table t add primary key (i)"))
);
System.out.println(
ctx.meta("")
.apply("create table t (i int)")
.apply("alter table t add j int")
.apply("alter table t alter i set not null")
.apply("alter table t add primary key (i)")
);
Meta.apply(String) looks like this:
public final Meta apply(String diff) {
return apply(dsl().parser().parse(diff));
}
- For tests
- For blog posts
- For documentation
- For new users
- For simple applications
Queries (which wraps a bunch of parsed or hand-constructed jOOQ Query) objects is the real feature here. But convenience helps the user very much!
I would not have discovered this requirement without dogfooding.
Convenience
A much underrated tweet by Brian Goetz is this one: APIs need a ton of abstractions. The JDKCollector API is a very good example. In Java and in SQL, in order to write a custom aggregate function, you need these four operations:
- Supplier<A>: A supplier that provides an empty, intermediary data structure to aggregate into
- BiConsumer<A, T>: A accumulator that accumulates new values from the stream into our intermediary data structure.
- BinaryOperator<A>: A combiner that combines two intermediary data structures. This is used for parallel streams only.
- Function<A, R>: The finisher function that extracts the result from the intermediary data structure.
Collector from scratch is super annoying and “low level”. Using convenience API is much better, this is why the JDK has Collectors, or jOOλ has Agg, and there are other Collector libraries that provide some basic building blocks. The Stream API does not have to know about all of these building blocks, it just knows a single, abstract type. (serves the API). But we users, we don’t want only a single abstraction. We want those convenient building blocks (serves the user).
Convenience is never requirement, but always a big plus. In APIs like in languages.
Have you noticed this new method on InputStream, which has been added in Java 9?
public long transferTo(OutputStream out) throws IOException {
Objects.requireNonNull(out, "out");
long transferred = 0;
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
int read;
while ((read = this.read(buffer, 0, DEFAULT_BUFFER_SIZE)) >= 0) {
out.write(buffer, 0, read);
transferred += read;
}
return transferred;
}