August 3, 2010

Custom Fields and the Death of the Field Group

Where did we come from?

Before we can responsibly discuss the future of EE Custom Fields, we need to to establish how we got to where we are today.

pMachine Pro

In 2002, EllisLab made their first foray into web publishing with a lightweight blogging tool called pMachine Pro. You could set up multiple weblogs, and their entries had the usual assortment of entry submission fields: ‘Blurb’, ‘Body’, and ‘More’.

“At the time it was all about the blogging tools, and all of them (or at least the ones I looked at) seemed to be limited to ‘title’, ‘summary’, ‘body’, and ‘extended’ fields. To have 3 additional ones to play with was heady stuff.”

—Mike Boyink, Train-ee

Despite being more akin to traditional blogging software than general-purpose content management systems, there was a great diversity in the sites being built with it. EllisLab noticed, and decided to foster it. So pMachine 2.0 added three new “Custom Fields” to the submission page, which could be enabled on a per-weblog basis.

For its price point, this was a huge advancement in web publishing, because it enabled web developers to tailor their publish forms to fit their needs. They were no longer stuck with the exact recipe of fields the app’s creator had decided upon.

Sure the feature had its limits, but it gave developers a taste for what was to come.

ExpressionEngine

As users continued to stretch pMachine in new ways, Rick Ellis saw the need for a next-generation publishing system—something new from the ground up. So he began working on ExpressionEngine.

During the pMachine Pro days I was amazed at how far people we're pushing the app in ways I hadn't anticipated, so when I started developing ExpressionEngine I had a much better sense of where I needed to take the app.

ExpressionEngine 1.0 was released in April 2004. With it, EllisLab built upon pMachine Pro’s Custom Fields concept in two key ways:

1. Custom Fields now had a variety of new Field Types to choose from. One could be a drop-down menu, another could be a date picker, etc..
2. Custom Fields were no longer assigned directly to weblogs, but rather to Field Groups, which could contain any number of Custom Fields. Each weblog was then assigned one Field Group.

These made for an extremely powerful combination, defining ExpressionEngine as a truly malleable content management system.

Fieldtype Development

ExpressionEngine’s ability to select variant Field Types was nothing short of revolutionary, but it was missing a key ingredient: a supported way for add-on developers to inject custom Field Types into the system.

That’s not to say people didn’t find ways to do it, though. In September 2004, Arnold Jagt published a guide to hacking FCKeditor into ExpressionEngine 1.2. And in July 2005, Chris Jennings described how to hack TinyMCE into ExpressionEngine 1.3.

Things got a little easier in May 2006. EllisLab released ExpressionEngine 1.4.2, which introduced 7 new extension hooks, making it possible for add-on developers to write their own “fieldtype extensions”—extensions which added new Field Types to your arsenal, without requiring you to hack EE’s system files.

“When I first got into EE back in 2006, there was Mark Huot making fieldtypes. Then Leevi, and then Brandon. Then Brandon released FieldFrame, and things went ape shit!”

—Ryan Masuga, Devot:ee

Mark Huot is largely responsible for pioneering early fieldtype development. He made his debut with Multi Drop-down List, and went on to release several others, including the wildly popular File field.

Not long after, Leevi Graham got into the scene with LG TinyMCE, arguably the most popular WYSIWYG field ever released for ExpressionEngine 1. In early 2008, I took a stab at fieldtype extensions with Playa.

While the fieldtype hooks certainly made fieldtypes easier to install, they weren’t very easy to write. Developing extensions in EE tends to get a little messy, and fieldtype extensions were a glaring example of that. So there were only a small handful of other developers that ever joined in the fun. When I set out to write Playa 2, I couldn’t even muster up the stomach to deal with it again. (Roughly half of Playa 1’s code was a jumble of regular expressions that had no relevance on the actual app logic.) So I decided to first write FieldFrame. FieldFrame abstracted all the complexities of fieldtype extension development, lowering the barrier of entry to that of a plugin. And once it was out, fieldtype development finally took serious flight.

Outgrowing Field Groups

Field Groups in ExpressionEngine work well for a specific case: websites which have multiple, identical sections. A shining example of this is blogs.cisco.com, which holds several identically-modeled weblogs, differentiated by subject matter alone.

More often than not, though, different weblogs have different data models. So creating and filling up a Field Group is a process linked directly with creating a new weblog. Most of the time that’s perfectly acceptable, but it’s not uncommon for two weblogs to have almost identical data models. Unfortunately, ExpressionEngine provides no way for two Field Groups to share fields, so you’d be faced with two options: (1) create two near-identical Field Groups, or (2) lump all the fields (both shared and weblog-specific) together in one Field Group, and provide field instructions that remind the author which fields to ignore, depending on which weblog they’re posting in. Either way was frustrating.

In October 2008, EllisLab released a build of ExpressionEngine 1.6.5 that added a new extension hook, “publish_form_field_query”. Essentially, it gave extensions the power to override which fields would appear on the publish page. Two months later I put it to work, with Gypsy. Gypsy gave web developers the option of assigning their Custom Fields directly to one or more weblogs, ignoring their Field Group assignments.

Just days ago, Tim Kelty released Field General, another extension to take advantage of the hook.

It dawned on me that, like categories, there was no architectural reason why you couldn’t have multiple Field Groups assigned to a weblog. The hook was there for the Field Group query, so I ran with it.

I asked Tim what his thoughts were on Gypsy vs. Field General:

I could never get over the management issue. I want to see and edit which fields are assigned to weblogs, but with Gypsy the field is the boss, not the weblog, so it’s hard to see an overview and control display order.

I certainly like the idea of managing fields by weblog rather than weblogs by field. But as Tim pointed out to me, going through Field Groups does tend to require creating a lot of single-field Field Groups.

Where are we now?

On July 12, EllisLab officially released ExpressionEngine 2, bringing ExpressionEngine closer to a general-purpose CMS than ever before. Case in point, “weblogs” were renamed to “channels”, the most nondescript term they could think of.

Two of the most significant additions in the release revolved around Custom Fields.

Fieldtype API

Fieldtypes are finally first-class add-ons in ExpressionEngine 2. To add-on developers, that means writing fieldtypes in EE2 is just as easy as it was to write them with FieldFrame. (In fact, EllisLab intentionally architected the fieldtype API to look familiar to FieldFrame vets.) To end users, it means we should be seeing a lot more third party fieldtypes right off the bat. And according to Derek Jones, that’s exactly what’s happening:

It's still early, but the adoption rate easily surpasses the adoption rate of extensions by developers when we first added them to ExpressionEngine.

Beyond porting the fieldtypes that came with EE1 using this new API, EllisLab also wrote four new ones: Checkboxes, File, Multi Select, Radio Buttons.

Publish Layouts

Publish Layouts are one of ExpressionEngine 2’s most interesting new concepts. They give you complete control over the layout of your Publish page—for tabs as well as fields—on a per-channel basis. You can create, remove, reorder, and rename tabs, choose which fields go in which tabs, and even shrink adjacent fields’ widths a bit to get them to display side-by-side.

According to Derek Jones, this was just another step toward making ExpressionEngine as unassuming as possible:

This came about entirely in response to the demands being made on ExpressionEngine for publishing wider varieties of content. The HTML form that is well suited to publishing articles for a newspaper or online magazine are entirely different from the form that is well suited to entering hundreds of business addresses and phone numbers. Extending on EE’s principle of making no assumptions about your data, we decided to make no assumption about what your client's publish form needs to look like, and let you decide.

To be honest, when I first started using the EE2 Beta, I wasn’t so sure about the concept. It was a lot to take in, coming from a point where we were limited to customizing the display order of the fields alone. It wasn’t until I was building pixelandtonic.com that I realized what a powerful concept it was.

One thing that struck me was that the feature doubled as a way to hide fields from certain channels. For example, I have an ‘EE Add-ons’ channel set up on pixelandtonic.com. It’s assigned an ‘EE Add-ons’ Field Group with the following Custom Fields:

• Tagline
• Intro Paragraph
• Support URL
• Compatibility
• Devot:ee Rating
• Main Content
• Sidebar Content
• For Sale?

Each of those fields are relevant to my primary add-ons. On the other hand, entries in the Dive Bar’s channel only have a need for Compatibility and Main Content. Back in the EE1 days, I would have either created a new Field Group, duplicating those two fields, or used Gypsy to assign the six unneeded fields directly to the EE Add-ons channel alone. But since pixelandtonic.com runs on EE2, I had a new option: I could simply edit the Dive Bar channel’s Publish Layout, and just hide those unneeded fields.

Module Tabs

Modules learned a new trick in ExpressionEngine 2: they can now create their own tabs in the publish page, and fill them up with a default set of fields. I say “default” because Module Tabs/fields are subject to the same manipulation that all other tabs/fields are via Publish Layouts.

I had a chance to play around with Module Tabs when writing Juniper, my homegrown eCommerce module for Pixel & Tonic. Juniper adds a new tab to the publish page called ‘Juniper’, pre-set with two fields: ‘Price’ and ‘Release Notes’. In my EE Add-ons channel, I customized my Publish Layout, moving the ‘For Sale?’ field into that Juniper tab, and in all other channels I removed the tab entirely.

Where are we headed?

There are two trends that stand out to me in all of this:

1. Custom Fields have come a hell of a long way from their humble beginnings in pMachine Pro. They’ve benefitted from vibrant innovation by EllisLab and add-on developers alike.
2. Field Groups, on the other hand, have only become less relevant with time. Add-on developers have spent significant time exploring ways to get around their limitations, and with the additions of Publish Layouts and Module Tabs in ExpressionEngine 2, EllisLab themselves have been playing down their importance.

Looking forward, I think it’s safe to say that Custom Fields will continue to become more versatile, and Field Groups will continue to lose relevance.

There’s still a lot of frustration over Custom Fields in ExpressionEngine, primarily due to their Field Group constraint. I know it (I’m constantly getting requests to port Gypsy to EE2) and EllisLab knows it (they’re constantly getting requests to add the publish_form_field_query hook to EE2). But the hook isn’t coming back, and even if it were, I’m not porting Gypsy.

EllisLab would rather find a way to solve the problem for everyone, without requiring a third party extension, and I’m fully supportive of that.

So how do they solve the problem, exactly? Over the past couple weeks, I’ve been putting a lot of thought into it, and I think I’ve come up with an idea that should work for everyone.

My Proposal

First, channels get a new “Edit Publish Layout” link between “Edit Preferences” and “Edit Group Assignments” in the Channel Management page.

Clicking on a channel’s Edit Publish Layout link takes you to a new page where you can fully customize its Publish Layout. (Note that this would completely replace the right-hand toolbar on Publish pages.)

My comp’s a little rough around the edges, but you get the idea. Each visible tab is listed on the left, populated with the fields it contains, and unused fields are on the right. Dragging fields/tabs from the right to left selects them; dragging from left to right deselects them. You can also reorder tabs and move individual fields from tab to tab via drag-n-drops.

And here’s the kicker: all fields are available, from every field group. Field Groups still exist, but they’re used for organizational purposes only (like Upload Directories).

Finally, when you’re creating new channels, you’d have the option of duplicating an existing channel’s publish layout.

So in summary, fields are assigned directly to channels via their Publish Layouts, and field groups are reduced to organizational constructs.

Far From Perfect

My proposal is an evolutionary step that addresses current pain points. But new ones will surface, as web developers continue to test the limits of ExpressionEngine.

I asked Rick Ellis where he thinks custom fields are headed. He imagines a much more revolutionary future is in store for them, in time:

Bruce Lee said that the ultimate technique is no technique. What he meant is that the goal of a martial artist is to posses such well developed knowledge, awareness, fluidity, timing, and sensitivity, that the specific techniques don't matter anymore; the goal is to go beyond technique into a state of pure effectiveness.

So in a way I have this vague idea that Custom Fields should evolve into a vehicle that is so transparent, intuitive, flexible, and useful, that it allows a pure expression of information, without the constraints of boxes on a page. I don't have a design concept yet, but I do believe we’re far from having perfect tools.

Thanks!

Many thanks to all of you who helped me fill in the blanks and answer my questions while writing this: Mike Boyink, Rick Ellis, Derek Jones, Lisa Wess, Pascal Kriete, Mark Huot, Tim Kelty, and Ryan Masuga. I couldn’t have put all this together without you.