Drupal Planet

Syndicate content
Drupal.org - aggregated feeds in category Planet Drupal
Updated: 43 weeks 5 days ago

Acquia: Drupal 8 Configuration Workflows using Git

Thu, 2014-02-27 14:15

This blog post is a textual representation of the video shared yesterday. If you are visual learner, watch it. If you are in a hurry, read this blog :). Peter's video also shows how configuration_log module can be used to materialize all config changes in Prod so they may be easily integrated back into the codebase. That is not covered here.

Categories: Drupal

Freelock : Successful projects, part 4: It costs more if you do half the work

Thu, 2014-02-27 12:11

"My budget is really tight, can you get the project started and show me what to do to finish it?" -- Yet another request from several different prospective customers.

This sounds appealing, right? Drupal does put a huge amount of power and control in the hands of users. Aside from custom theming, many of our projects don't involve writing any code -- it's more a matter of mapping out the user stories, putting an appropriate set of modules together, and configuring them to deliver the desired result.

DrupalDrupal PlanetMistakesProject Management
Categories: Drupal

Phase2: Get Your FACLs Straight

Thu, 2014-02-27 10:25

Fiddling with file permissions is an all-too-common part of setting up a new service or website. Usually a daemon program needs to write files for user uploads, generate image derivatives for alternate sizes, or store other cache data.

If the daemon is running as a different user than those who deploy or edit CMS application code, file permissions must be carefully configured to allow shared write access. Typically the suggested solution is to change the owner, group, or permissions of the file or directory.

For example, the Drupal installation guide recommends setting the permissions on a specific settings file to allow all users to write during the installation procedure, and also setting the permissions on the “files” directory to allow all users to write to it.

Most file permission issues with Drupal are resolved by ensuring your settings file and files directory have permissions like this:

default user$ ls -l total 96 -rw-r--r--  1 user     group  23196 Jan 15 13:43 default.settings.php drwxrwxrwx  2 user     group     68 Feb 19 18:53 files -rw-rw-rw-  1 user     group  23196 Feb 19 18:53 settings.php default user$

However, this approach has limitations. It allows for permissions specific to a user, a group, and everyone else. More complex use cases may call for granting permissions to multiple individual users as well as more than one group. For example, a common use case would be to allow write access to a group of developers, a web server user, and a continuous integration system user.

These challenges could lead one to use “chmod 777,” or to make files readable, writable, and executable to everyone, and that is poor practice.

Beyond 777

Most modern operating systems support file system access control lists (ACLs), which allow defining specific permissions to read, write, or execute a file or directory to one or more users or groups. By using ACLs, one can define sophisticated permissions schemes that grant rights to many users and groups without allowing overly broad access or adjusting group membership. ACL entries can both expand and reduce the scope of permissions defined by the standard user-group-other permissions.

The first step in getting familiar with ACLs is identifying whether a file or directory has ACL permissions. Usually the presence of ACL permissions is signified by a + (plus sign) next to the user-group-other permissions in a directory listing in long format. For example:

~ user$ ls -l total 0 drwx------+ 12 user     group   408 Feb 13 19:16 Desktop drwx------+  6 user     group   204 Feb 13 19:29 Documents drwx------+ 37 user     group  1258 Feb 19 13:41 Downloads drwx------@ 48 user     group  1632 Feb 13 17:51 Library drwx------+  3 user     group   102 Dec 19 16:42 Movies drwx------+  3 user     group   102 Dec 19 16:42 Music drwx------+  5 user     group   170 Feb 11 19:01 Pictures drwxr-xr-x+  6 user     group   204 Jan 20 11:13 Public ~ user$

This shows some of the default directories in a user’s home directory on Mac OS X. In this case, the ACL permissions are used to prevent normal users from deleting the directories for the Desktop, Documents, Downloads, etc.

The methods for reading and writing file ACLs vary by operating system. The Linux/Unix world has the getfacl and setfacl commands. In OS X, file ACLs can be included in directory lists using the “ls -le” command. The OS X and Windows both provide configuration for file ACLs in the file management GUIs.

The following is an example of the output of getfacl for the “files” directory mentioned above:

default user$ getfacl files # file: files # owner: apache # group: apache user::rwx user:apache:rwx user:jenkins:rwx group::r-x group:devel:rwx mask::rwx other::r-x default:user::rwx default:user:apache:rwx default:user:jenkins:rwx default:group::r-x default:group:devel:rwx default:mask::rwx default:other::r-x default user$

In this example, the Jenkins user (in addition to Apache, the owner) and the devel group (in addition to the file’s assigned group) are granted write access to the directory. Since all users and groups that need write access are granted it through the ACL entries, it’s not necessary to allow other users write access.

This example illustrates two other features of ACLs: If a file or directory has non-owner users or groups, then a “mask” entry exists to set a ceiling on the permissiveness of ACL permissions. If the ACL mask entry is set to read-only, then ACL entries for specific non-owner users and groups cannot grant write access. Since this resource is a directory, it also has “default” ACL entries, which defines the default ACL entries to add to files created in the directory.

The setfacl command in Linux/Unix allows adding, changing, or removing an entry in a file’s or directory’s ACL. For example, the following command adds ACL entries to allow the Jenkins user and the devel group read, write, and execute access to the files directory:

setfacl -m user:jenkins:rwx,group:devel:rwx files

setfacl also provides an option for removing all ACL entries on a resource, which reverts the effective permissions to the basic owner-group-other permissions. This would remove all ACL entries from settings.php:

setfacl -b settings.php

Since granting permissions via ACLs does not require modifying the user or group that owns the file or directory, it’s possible to apply ACLs periodically and in batch to files to ensure that standard access permissions are enforced.

Keep It Simple

Filesystem ACLs offer a method for creating powerful permissions schemes, but there are some drawbacks.

The ACLs associated with a file or directory are not as clearly exposed nor as well known as the basic user-group-other permissions. Therefore, permissions issues may be obscured by the use of ACLs.

For instance, it’s possible that an ACL entry imposes a permission that contradicts the apparent user-group-other permission. A file can be set to writable for all users, but an ACL entry denies write access to specific users or groups. This may lead to a confusing access problem, as most documentation points only to the user-group-other permissions.

Another major issue is inconsistent application support. In some cases, applications use file system functions or metadata in a way that considers only the base user-group-other permissions and does not correctly apply ACLs. For example, there are some active issues related to ACL permissions for Drupal, where it reports erroneously that a file or directory is not writable even though access is granted by ACLs.

This limits the effectiveness of ACLs, but does not negate their value. The basic user-group-other scheme can be set to support legacy applications, while compatible programs and users are granted access via ACLs.

Try It Out

Take the next challenge with file permissions as an opportunity to give ACLs a try. The best documentation around filesystem ACLs is specific to the operating system, so look for it with the OS’s ACL tools: setfacl for Linux/Unix, chmod for OS X, and icacls for Windows.  In the meantime, check out Fredric Mitchell’s blogpost about keeping Drupal Simple (KDS).

Categories: Drupal

DrupalCon Austin News: Sustainability at DrupalCon - Trash!

Thu, 2014-02-27 10:24

Last week we posted about carbon offsets as a way to mitigate the impact of the CO2 produced when attendees travel to the event. What else are we doing at DrupalCon to make our events more sustainable? Today we’ll dive into our TRASH!

Here's how we are diverting items from the landfill and reducing the use of non-renewable/virgin resources.

Categories: Drupal

Palantir: D8FTW: Hacking Core Without Killing Kittens

Thu, 2014-02-27 09:44

The upcoming Drupal 8 includes many changes large and small that will improve the lives of site builders, site owners, and developers. In a series we're calling, "D8FTW," we look at some of these improvements in more detail, including and especially the non-obvious ones.

Many years ago — back when Drupal 4.7 was cutting edge, I was a Drupal neophyte, and giant reptiles such as Sue the T-Rex (a client of Palantir's) still had meat on their bones — someone inquired on the Drupal development mailing list about how to modify or replace a function in Drupal core that didn't quite do what he wanted. His inquiry was met with polite apology for the fact that our guest was asking something utterly impossible: In most cases it was impossible to change what files Drupal would load other than adding modules; removing code that shipped with Drupal was impossible without modifying Drupal directly, a process affectionately known as "hacking core" and less affectionately for resulting in harm to adorably cute kittens (if only in effigy).

The problem wasn't PHP itself per se; the problem was the nature of procedural code, upon which Drupal of the day was based. Fast forward to Drupal 8, however, and that problem has been neatly resolved in most cases. The key is services, those objects we introduced in our last episode. Drupal 8, for the first time, effectively enables developers to "hack core without hacking core", and provides the tools necessary to allow modules to enable the same superpower.

The basics

Let's define a few first-principles that help make this magic trick possible. As discussed previously, a service is an object that contains some useful logic, is stateless, and in practice will usually only instantiate once (although it's not the service's job to enforce that). A service may well make use of another service, that is, it depends on it. If a service calls another service directly, that is a "hard dependency."

Generally speaking, hard dependencies are a bad idea as it means we cannot make use of a service without also making use of all of its dependencies, even if we just want to test it. Instead, we can make use of an interface, a PHP language construct that defines what methods an object has, and what they do, but not how they do it. We then pass an object that implements that interface to a service's constructor to be saved. By design we can pass any object that implements that interface and everything still works. That process of "pass objects into the constructor to be used later" is known as "dependency injection" (where "injection" is simply the needlessly medical way of saying "pass stuff in").

Of course, passing all of those objects to other objects can get quite tedious, which is why many people recoil at the effort necessary. Most projects of notable size, therefore, use something called a "dependency injection container," which is again an overly-pedantic way of saying "one object to rule them all, one object to find them, one object to inject them all and in the bootstrap instantiate them." (Sorry, I just got back from Drupal South in New Zealand; the Tolkien runs deep.)

Put less poetically, a Dependency Injection Container (DIC) is simply an easier place to wire up what objects get passed to what objects. The container will then take care of creating the object on-demand, including creating its dependent objects (if they haven't been created already) and passing them in (injecting them).

Now we get to the key point: That "wiring up" concept is not, technically, code. It's configuration. And configuration can be changed without breaking the code itself, as long as the code's assumptions (the interfaces of its dependencies) don't change. Therein lies the power of dependency injection: It makes the way an application is built configuration, not code.

Get the syringe

Let's look back at the breadcrumb example from episode 1. We registered our breadcrumb builder like so, in mymodule.services.yml:

# mymodule.services.yml
    class: Drupal\mymodule\NewsBreadcrumbBuilder
      - { name: breadcrumb_builder, priority: 100 }

Those few lines of YAML are configuration instructions for the DIC. The particular syntax is a Symfony thing (there are other DIC implementations with their own syntax and quirks), but the concept is universal: The service named "mymodule.breadcrumb" is an instance of Drupal\mymodule\NewsBreadcrumbBuilder, built on demand. The Symfony DependencyInjection component allows for that configuration to exist in YAML, or in code. It's also possible to change it from code. Each system that uses the Symfony DependencyInjection component implements it a little differently, but in Drupal it comes down to two simple interfaces.

A key feature of Drupal's DIC implementation is the "provider class". Every module may have one specially-named class in its root namespace named $CamelizedModuleServiceProvider. That is, if our module is called "my_module" then the class will be named Drupal\my_module\MyModuleServiceProvider. That class may implement one or both of two interfaces: Drupal\Core\DependencyInjection\ServiceProviderInterface, which has a register() method, and Drupal\Core\DependencyInjection\ServiceModifierInterface, which has an alter() method.

If that class implements ServiceProviderInterface, then the register() method is passed the container definition and the module can register additional services using the container's API. See the Symfony documentation for the full details on what is available. In practice that's mostly only needed for registering compiler passes as the YAML file is much easier to work with. (More on compiler passes another time.) More useful is the ServiceModifierInterface, whose alter() method will also be passed the container definition.

If you've worked with Drupal before, you probably know how this works. The container definition is built in one pass, and then passed to any "alter objects," just as alter hooks have worked in the past. In the alter() method, we can add services based on other services or, more realistically, change or even remove existing services.

Let's say we want to completely remove the "book" breadcrumb logic from core. We simply don't want that code to run at all, period. First we look up the name of that service in the book.services.yml file, where we find this:

  class: Drupal\book\BookBreadcrumbBuilder
  arguments: ['@entity.manager', '@access_manager', '@current_user']
    - { name: breadcrumb_builder, priority: 701 }

Now we can add our own module with a service modifier and remove that book.breadcrumb service. All we need is the class below, placed in our module:

namespace Drupal\no_book_breadcrumb;


use Drupal\Core\DependencyInjection\ContainerBuilder;


NoBookBreadcrumbServiceProvider extends ServiceProviderBase {
  public function alter(ContainerBuilder $container) {

Wait, that's it? Really? Really! Give it a try. This class completely removes the book breadcrumb builder from the system; it's now just taking up space on disk but has no runtime impact on the system at all. Seriously, how cool is that?

We can do much more than that, of course. For instance, rather than just removing one breadcrumb builder, let's take over the entire breadcrumb system and declare that, for our site, we have absolute control over breadcrumbs and no other module has any say. (We're professionals; don't try this at home. Or do. It's kinda fun.) We can take over the entire breadcrumb manager, like so:

  public function alter(ContainerBuilder $container) {
    $breadcrumb = $container->getDefinition('breadcrumb');

Now, the system will ignore the core BreadcrumbManager entirely and use our class instead. Ideally every service has a separate interface, but BreadcrumManager doesn't as of this writing. (Someone file a patch!) Instead we'll just subclass it:

Categories: Drupal

Drupal Association News: Drupal Making News in BBC School Report

Thu, 2014-02-27 08:02

Open source software has long been a natural choice for schools and educators teaching children about technology: both share an emphasis on exploration and collaboration, and access to free, open source technology can be a welcome relief to schools and parents.

Categories: Drupal

Chris Hall on Drupal 8: Using Drupal 8 to decouple yourself from Drupal

Thu, 2014-02-27 07:52

When doing development loose coupling is good, generally I don't think there would be a lot of argument there. For this post I want to extend the metaphor to coupling between a developer and the tools they use.

I suggest that currently many (not all) Developers using Drupal are tightly coupled to Drupal and that this harms their potential career development, it harms their employers (and/or people that may wish to use their services) and ultimately it harms the entire Drupal community.

I believe that Drupal 8 will go a long way to removing the negative effects of this tight coupling, something that is good all around but that there may be a bumpy ride ahead for some. Ultimately I trust that mastery of Drupal 8 will break the tight coupling opening up many more opportunities and approaches.

My view is that if most of the knowledge you have about what you need to do your work is specific to a particular tool, you are tightly coupled to it, and prior to Drupal 8 there was no way to avoid this to a large degree. 

Site building versus development

For somebody who is mostly working through the administrative interface of Drupal and various contrib. modules, there will always be a high degree of coupling. Although some degree of generic CMS information and knowledge will be part of their work, switching to a completely different CMS will require a significant amount of re-learning, apart from the administrative interface they will have to get used to a new landscape of modules and plugins (or whatever the equivalent is in the new CMS).

If you are a developer it is worth installing and playing with Drupal 8 now, to get a feel for these changes even if you are not ready to poke around under the hood yet, it will not take you long to find your feet.

Fortunately for the Sitebuilder things have just got better, my experience of the Drupal 8 admin interface so far is that it is mostly familiar, but much improved (try doing serious work on the average Drupal 7 site with a 7 inch tablet for example). Assuming that there are enough contrib. modules available I don't foresee most SiteBuilders having problems with Drupal 8, in fact the small amount of re-learning required will come with many benefits, and the new configuration management implies more opportunities for site builders to take on management of multiple sites and environments without requiring development assistance.

Whilst there are many differences between CMSs there is a big problem if somebody can't learn to start using one 'relatively quickly' for this reason I don't think a fork like the one seen for Backdrop is going to be driven by Sitebuilders. Changes in Drupal 8 for SiteBuilders and Administrators are in the order of magnitude that you would expect for a new version of anything these days and come with the expected improvements in usability and functionality.

Drupal like all it's competitors is in the business of making more and more possible for SiteBuilders (it would be failing if it wasn't) development fits into the remaining 10% - 20% you can't do without coding PHP (much higher percentages for particularly bespoke or enterprise needs, migration, upgrades etc. these are the bits where I get involved and interested).

Developer versus 'Drupal Developer'

This is going to ruffle a few feathers, but if it doesn't apply to you then ...... it doesn't apply to you so relax, it doesn't apply to many people developing with Drupal, there is enough of a problem to really cause issues if you are hiring developers to do Drupal work however.

Many (not all) Drupal Developers are incapable of moving outside of Drupal and operating as modern PHP developers or quickly assimilating a new framework or even new programming languages. Stable geniuses in their own environment, they cannot operate effectively out of it, or at least would have to drop down a couple of pay scales. To these people everything starts looking like content (even when dealing with data and data processing tasks) and Drupal is the answer to everything (because it is the only answer they have experience of providing).

I have seen and been in places with 'Drupal Developers' and developers (who handle EVERYTHING not Drupal).

I have heard organisations refusing to use Drupal because in the current climate "it is too hard to find Drupal Developers".

I have been through a 1.5 hour telephone interview for a PHP position where at the end the interviewer apologised for the simplicity of some of the questions but stated "We have found we have to do this to weed out certain types of Wordpress and Drupal developers, who have years of experience but fundamental gaps".

Generally recruitment for Drupal demands two years or so commercial experience. whereas for many frameworks six months would be enough assuming it was associated with other development experience. i know that Drupal is first a CMS but it has to be considered a full framework now in my opinion simply becasue for many websites Drupal is used for everything the site does.

Accomplished non-drupal developers with lots of experience usually fail to produce acceptable sites quickly enough if they are dropped in to using Drupal, they can become productive much more quickly in other new frameworks.

I have been working on websites for more than fifteen years now, using ASP, ColdFusion, Perl, Java, PHP and Python and used a number of frameworks particularly in Java. I found the Drupal learning curve (started seriously for me in the early days of 6) to be the most unpleasant and bizarre, eventually it paid off but it would have been easy to give up and stick with Java. Everything above is based on my own experience and from looking around the agencies, companies and devs. in the places I work.

All the above points indicate the strong coupling between the Drupal Developer and the tool.

Community versus tribe

The Drupal community has some terrific aspects but there are times when it verges towards tribalism, this is seen for example in one area which is a particular 'pet hate' of mine, framing all things opensource in the context of Drupal. Some (not all) Drupal developers are so tightly coupled to Drupal that they show disdain (and often disrespect) to developers who use Drupal but have not contributed. The usual warm welcome given to perceived newcomers eventually turns to a feeling of  "you have been using Drupal for years now yet YOU HAVE NOT CONTRIBUTED", 'taker' vs 'maker' arguments are raised.

This individual tool-centric world-view is not the way opensource is supposed to work, you have to turn it around and frame Drupal in the context of all things opensource. A Drupal site sits in a sea of opensource, probably the OS and supporting software of the server, various Javascript libraries used, the IDE you develop in? do I need to go on? Anybody who has not contributed to Drupal may have contributed to any of these, they may even give up free time to run a Drupal site for a community project or charity (which imho. totally absolves them from the sin of being a 'taker'). 

In my experience the communities where everybody has/should be a contribute are the same ones where there is a very tight exclusive coupling between the developer and tool.

I still foresee a strong Drupal community ahead, but with more blurring at the edges and more transitioning amongst the members.

Drupal 8 loosens the ties

The obvious nature of the Symfony components in Drupal 8 tend to help dissolve tribalism. It is amusing to see one or two instances in issues now where attitude changes noticeably when people suddenly realise that new guy with the Drupal profile a few weeks old is actually an accomplished and skilled Symfony developer (who knows more about the current problem being discussed than most).

Now it is possible to learn some aspects of Drupal 8 development by reading Symfony documentation, reading more about Symfony teaches a lot about  frameworks in general.

Now it is possible to bring people already comfortable with Twig into a project (or move the other way), you could have frontend developers moving between Django (a Python framework) and Drupal in the same company. 

Some of the knowledge that is useful for Drupal 8 is directly applicable in many other technologies. An example from many: Skulpin a PHP static site generator, which can use Twig templates, and employs the Symfony HTTP kernal.

My journey

I have never regretted my decision to focus on using Drupal, but I have been nervous over the last year or so about the amount of knowledge I am using that is Drupal or Drupal 7 specific. Fourteen years ago ColdFusion looked like a good bet, I was delighted however when ColdFusion switched to running on Java, I could diversify and eventually moved into Java development (and now most people snigger when I mention ColdFusion). Drupal should be more resilient due to it's opensource background but there are no certainties, this industry is characterised by sweeping change.

Since I have started looking at Drupal 8 it is like the lifting of a great weight from my mind. Finally a chance to do some serious OO work in PHP (although a lot is familar from Java I need  time on task to internalise the differences, some subtle some not so subtle). I have happily taken on more non-Drupal work than ever before (Drupal still figures heavily in my day to day work though). I fixed up and released an ExpressionEngine site, have worked extensively on a bespoke OO PHP framework I inherited. All the time I can keep my eyes open and learn/hone skills that will help with my Drupal 8 development but that are also universally applicable elsewhere.

If Drupal continues to rise, Drupal 8 onwards in particular  (which is what I am betting on) , it is all win for me, if not I have options "never bet more than you can afford to lose".

An industry of change

Change is a given in the industry I work in, specific knowledge of configuration params and arcane hooks used in a version of Drupal, whilst useful for a while, has little value over time. General knowledge and practice of common design patterns, advanced techniques and widely used components are much more bankable.

Categories: Drupal

KatteKrab: DrupalSouth Wellington 2014

Wed, 2014-02-26 17:57
Thursday, February 27, 2014 - 11:57

Well it happened! DrupalSouth was pretty damn magnificent. Learning was done, fun was had, and the community gathered and did what it does best.

There's been a couple of nice write ups by people who attended.

Did I miss yours? Let me know? http://twitter.com/kattekrab

DrupalSouth was so good, we're even running away with the name and going to run it in Melbourne next year!


Categories: Drupal

more onion - devblog: Using Drupal behind a reverse-proxy

Wed, 2014-02-26 14:06

As the title suggests we have Drupal hosted behind a reverse proxy. Usually this doesn't matter a lot, but there are few things like IP-blocking or oauth that need to know some data about original request. Namely that's: The client-IP, whether the request was issued via HTTPS and the server port. Here is how to achieve that.

Categories: Drupal

DrupalCon Austin News: Call for Frontend Sessions

Wed, 2014-02-26 13:22

With only 9 days left until the call for content deadline, the Frontend track team warmly invites all themers and Frontend developers (or even those working closely with Frontend devs) to submit a Frontend session proposal for DruaplCon Austin 2014. The deadline is March 7, 2014 at 11:59pm CET.

We are looking for proposals geared towards Drupal 8, with an emphasis on how the life of a Frontend developer will change and your expert advice on how to best prepare for the transition.

Categories: Drupal

Mike Crittenden: I wrote a book for O'Reilly: "Responsive Theming for Drupal"

Wed, 2014-02-26 12:52
78 pages of theming, what!

My book "Responsive Theming for Drupal" has been released!

In it, you'll find:

  • An introduction to responsive web design
  • A simple process for adding responsiveness a simple Drupal theme
  • Workflows for extending Aurora, Omega, and Zen in Drupal 7 for responsive theming
  • Pros, cons, and discussion about some of the most common base themes
  • Some common gotchas and requests with responsive theming and solutions
  • Next steps for continuing your responsive education

Check it out!

Categories: Drupal

Drupal Association News: DrupalCon Wants YOU (As A Sponsor!)

Wed, 2014-02-26 12:16

DrupalCon Austin is right around the corner, and opportunities abound for enterprising companies wanting to get their brand out in front of some of the best Drupal companies and talent in the world.

Categories: Drupal

Acquia: Moving Your Drupal 8 Configuration from Local to Server and Back

Wed, 2014-02-26 10:08

Two weeks ago I had a great opportunity to spend a few days working with Moshe Weitzman (moshe weitzman), Justin Randell (beejeebus), Alex Bronstein (effulgentsia), and Stéphane Corlosquet (scor) to look at the challenges and best practices for using the new Drupal 8 configuration system (a.k.a. CMI) to move changes between a local development environment and one or more server environments.

Categories: Drupal

Metal Toad: DrupalCon 2014 Call for Sessions!

Wed, 2014-02-26 09:44

It's that time again! DrupalCon is right around the corner and it's time to submit your talk. The theme this year is Drupal 8 and that means that we'll be covering things like Symphony, Twig, and the other new components that make up Drupal 8. Do you have a talk about how to use the new templating system? Want to share how OOP works in Drupal 8? Submit your session now! See you in Austin!

The call for content ends March 7th at 11:59PM Austin local time (UTC -6).

Categories: Drupal

LevelTen Interactive: Adding Bootstrap Badges to Drupal Views Lists

Wed, 2014-02-26 09:23

There is a neat little trick in Views to create lists with counts. For example, if I created a list of categories for blog posts, I can have Views add the number of posts in each category with a little extra configuration.

 ... Read more

Categories: Drupal

Dcycle: Eight tips to remember on your path to automated testing

Wed, 2014-02-26 08:56

Many Drupal projects now under maintenance suffer from technical debt: a lot of the functionality is in the database and outside of git, and the code lacks automated testing. Furthermore, the functionality is often brittle: a change to one feature breaks something seemingly unrelated.

As our community and our industry mature, teams are increasingly interested in automated testing. Having worked on several Drupal projects with and without automated testing, I've come to the conclusion that any line of code which is not subject to automated tested is legacy code; and I agree with Michael Feathers who stated in his book Working Effectively with Legacy Code[1] that a site with zero automated tests is a legacy site from the moment you deliver it.

But the road to automatic testing for Drupal is, as I've learned the hard way, strewn with obstacles, and first-time implementations of automated testing tend to fail. Here are a few tips to keep in mind if your team is ready to implement automated testing.

Tip #1: Use a continuous integration server

Tests are only useful if someone actually runs them. If you don't automate running the test suite on each push to your git repo, no one will run your tests, however good their intentions are.

The absolute first thing you need to do is set up a continuous integration (CI) server which runs a script every time your git repo changes. To make this easier I've set up a project on GitHub which uses Vagrant and Puppet to set up a quick Jenkins server tailored for use with Drupal.

Even before starting to write tests, make sure your continuous integration job actually runs on your master branch. When your project passes tests (it should pass if there are no tests), your project will be marked as stable.

Notice that I mentioned the master branch: although git has advanced branching features, the only branch you should track in your CI server is your stable branch (often master, although for projects with more than one stable release, like Drupal itself, you may have two or three stable branches).

It is important at this point to get the team (including the client) used to seeing the continuous integration dashboard, ideally by having a monitor in a visible place. If your code is flagged as failed by your CI server, you want it to be known as soon as possible, and you want the entire team to have responsibility for fixing it immediately. Your main enemy here is failure fatigue: if your master branch is broken, and no one is working at fixing it, you will fail at implementing automated testing.

Eventually, you will want to add value to your continuous integration job by running Code Review tests, and other code analysis tools like Pdepend. With these kinds of tools, you can get a historical perspective on metrics like adherance to Drupal coding standards, the number of lines of code per function, code abstraction, and the like. I even like to have my Jenkins job take a screenshot of my site on every push (using PhantomJS), and comparing the latest screenshot to the previous one ImageMagick's compare utility.

Basically, any testing and analysis you can do on the command line should be done within your continuous integration job.

If done right, and if you have high confidence in your test suite, you can eventually use your CI server to deploy continuously to preproduction.

Tip #2: Test your code, not the database

Most Drupal developers I've talked to create their local development environment by bringing their git repo up to date, and cloning the production database.

They also tend to clone the production or preproduction database back to Jenkins in their continuous integration.

For me, this is wrong approach, as I've documented in this blog post.

Basically, any tests you write should reside in your git repo and be limited to testing what's in the git repo. If you try to test the production database, here is a typical scenario:

  • Someone will do something to your database which will break a test.

  • Your Jenkins job will clone the database, run the test, and fail.

  • Another person will make another change to the database, and your test will now pass.

You will now see a history of failures which will indicate problems outside of your code. These will be very hard to reproduce and fix.

Keep in mind that the tests you write should depend on a known good starting point: you should be able to consistently reproduce an environment leading to a success or a failure. Drupal's Simpletests completely ignore the current host database and create a new database from scratch just for testing, then destroy that database.

How to do this? First, I alway use a site deployment module whose job it is to populate the database with everything that makes your site unique: enabling the site deployment module should enable all modules used by your site, deploy all views, content types, and the like, set all variables and set the default theme. The site deployment module can then be used by new developers on your team, and also by the CI server, all without cloning the database. If you need dummy content for development, you can use Devel's devel_generate utility, along with this trick to make your generated content more realistic.

When a bug is reported on your production site, you should reproduce it consistently in your dummy content, and then run your test against the simulation, not the real data. An example of this is the use of Wysiwyg: often, lorem ipsum works fine, but once the client starts copy-pasting from Word, all kinds of problems arise. Simulated word-generated markup is the kind of thing your test should set up, and then test against.

If you are involved in a highly-critical project, you might eventually want to run certain tests on a clone of your production database, but this, in my opinion, should not be attempted until you have proper test coverage and metrics for your code itself. If you do test a clone of your production database and a bug is found, fix it in the code and redeploy; don't fix it on production directly.

Tip #3: Understand the effort involved

Testing is time-consuming. If your client or employer asks for it, you need to have the resources available to set it. Near the beginning of a project, you can easily double all time estimates, and the payoff will come later on.

Stakeholders cannot expect the same velocity for a project with and without automated testing: if you are implementing testing correctly, your end-of-sprint demos will contain less features. On the other hand, once you have reached your sweet spot (see chart, above), the more manageable number of bugs will mean you can continue working on features.

Tip #4: Start gradually

Don't try to test everything at once. If your team is called upon to "implement automated testing" on a project, you are very likely to succumb to test paralysis if you try to implement it all at once.

When working with legacy sites, or even new sites for which there is pressure to delivery, I have seen many teams never deliver a single test, instead delivering excuses such as "it's really simple, we don't need to test it", or "we absolutely had to deliver it this week". In reality, we tend to see "automated testing" as insurmountable and try to weasel our way of it.

To overcome this, I often start a project with a single test: find a function in your code which you can run against a unit test (no database required), and write your first test. In Drupal, you can use a Simpletest Unit test (as in this example and then run it straight from the browser.

Once you're satisfied, add this line to your CI job so the test is run on every push:

drush test-run mytestgroup

Once that is done, it becomes easier for developers to write their own tests by adding it to the test file already present.

Tip #5: Don't overestimate how good a developer you are

We all think we're good developers, and really we can't imagine anything ever going wrong with our code, I mean, it's so elegant! Well, we're wrong.

I've seen really intelligent people write code which looks really elegant, but still breaks.

I've seen developers never write tests for the simple stuff because it's too simple, and never write tests for the more complex stuff because they never practiced with the simple stuff.

Even though you're positive your code is so robust it will never break, just test it.

Tip #6: Start with the low-hanging fruit

This is an error I made myself and which proved very painful. Consider a system with three possible use cases for the end user. Each use case using the same underlying calls to the database, and the same underlying pure functions.

Now, let's say you are using a high-level testing framework like Behat and Selenium to test the rich user interface and you write three tests, one for each use case. You don't need unit tests, because whatever it is you want to test with your unit tests is already tested by your high-level rich user interface tests.

Don't forget, your specs also call for you to support IE8, IE9, Webkit (Safari) and Firefox. You can set up Jenkins to run the rich GUI tests via Selenium Grid on a Windows VM, and other fancy stuff.

This approach is wrong, because when you start having 5, 8, 10, 20 use cases, you will be tempted to continue just testing everything through the rich GUI, and your tests will end up taking hours.

In my experience, if your entire test suite takes more than two hours to run, developers will start resenting the process and ignoring the test results, and you are back to square one.

In his book Succeeding with Agile, Mike Cohn came up with the idea of a test pyramid, as shown in the diagram below (you can learn more about the concept in this blog post).

Based on this concept, we quickly realize that:

  • Several steps are redundant among the GUI use cases.
  • The exact same underlying functionality is tested several times over.

Thinking of this from a different angle, we can start by testing our pure functions using unit tests. This will make for lightning-fast tests, and will get the team into the habit of not mixing UI functions, database functions and pure functions (for an example of what not to do, see Drupal's own block_admin_display_form_submit).

Once you have built up a suite of unit tests which actually has value, move on to the next step: tests which require the database. This requires some variation of a site deployment module or another technique to bring the database to a known-good starting point before you run the test; it is harder to grasp and setting up a CI job for these types of tests is difficult too. However, your team will more likely be willing to work hard to overcome these obstacles because of the success they achieved with unit tests.

All of the above can be done with Drupal's core simpletest.

Finally, when you are satisfied with your unit test suites and your database tests, you can move outside of Drupal and on to Behat, Mink, Selenium, Windows/IE VMs. If you start with the fancy stuff, the risk of failure is much greater.

Tip #7: Don't underestimate developers' ability to avoid writing tests

If you implement all the tips you've seen until now in this article, something curious will happen: no one will write any tests, not even you.

Here's the psychology behind not writing tests:

  • You really have the intention of writing tests, you just want to get your feature working first.
  • You work hard at getting your feature ready for the end-of-sprint demo.
  • You show off your feature to the team and they like it.
  • You don't write any tests.

The above will happen to you. And keep in mind, you're actually very interesting in automated testing (enough to have read this article until now!). Now imagine your teammates, who are less interested in automated testing. They don't stand a chance.

These are some techniques to get people to write tests:

The first is used by the Drupal project itself and is based on peer review of patches. If you submit a patch to core and it does not contain tests, it will not make it in. This requires that all code be reviewed before making it into your git repo's stable branch. There are tools for this, like Phabricator, but I've never successfully implemented this approach (if you have, let me know!).

The second approach is to write your tests before writing a new feature or fixing a bug. This is known as test-driven development (TDD) and it requires everyone to change their approach. Here is a typical scenario of TDD:

  • A bug comes in for project xyz, and you are assigned to it.

  • You write a test for it. If you don't know something (no function exists yet, so you don't know what it's called; no field exists yet, so you don't know how to target it), just put something feasible. If you're dealing with the body field in your test, just use body. Try to think of everything here, the happy path and the sad path.

  • Now switch modes: your goal is to make the test pass. This is an iterative process which entails writing code and changing your test as well (your test is code too, don't forget!). For example, perhaps the body field's machine name is not body but something like field_body[und][0]. If such is the case, change the test, as long as the spirit of the test remains.

The above techniques, and code coverage tools like code_coverage or the experimental cover, which I like, will help you write tests, but changing a team's approach can only be achieved through hard work, evangelizing, presentations.

Tip #8: Don't subvert your process

When it becomes challenging to write tests, you might figure that, just this once, you'll not test something. A typical example I've seen of this, in project after project, is communication with outside systems and outside APIs. Because we're not controlling the outside system, it's hard to test it, right? True, but not impossible. If you've set aside enough time in your estimates to do things right, you will be able to implement mock objects, making sure you test everything.

In this blog post, I demonstrate how I used the Mockable module to define mock objects to test integration between Drupal and content deployment system.

You will come across situations where implementing testing seems very hard, but however much effort I put into implementing automated testing for something, I have never regretted it.


Automated testing is about much more than tools (often the tools are quite simple to set up). The human aspect and the methodology are much more important to get your automated testing project off the ground.

[1] See Jez Humble and David Farley's Continuous Delivery, Addison Wesley.

Tags: blogplanet
Categories: Drupal

Drupal.org Featured Case Studies: Printer's Devil Review

Wed, 2014-02-26 08:23
Completed Drupal site or project URL: http://pdrjournal.org

Printer’s Devil Review is an independent, open access journal of literary and visual art. We provide emerging writers and artists with access to publication and inquisitive readers with new voices and visions. We sell print editions at cost, but evey issue of the magazine is available for free download as a PDF (some are also available in e-book formats). We launched the site for the magazine, built in Drupal 7, in November 2011. More recently, we redesigned the site to allow users of mobile devices to more easily access our content.

Key modules/theme/distribution used: OmegaOmega ToolsDeltaContextFeaturesBlock ClassFE Block2Views SlideshowWebformDrushDevel@font-your-faceTypogrifyBackup and MigrateThemeKeyFacebook OAuth (FBOAuth)IMCEIMCE Wysiwyg bridgeWysiwygJournal Crunch
Categories: Drupal

Microserve: New Module: Block and Unpublish

Wed, 2014-02-26 07:53

As an agency we get to work on a wide range of projects, each with its own requirements for solutions and features. On occasion we find ourselves working on something that could work as a standalone module and solve common Drupal issues. Even with our often busy schedules we like  to stay on the lookout for these kinds of opportunities for us to contribute back to the Drupal community.

So today we’re proud to announce our latest contributed module - Block and Unpublish. This module adds a new option to the Drupal user bulk-operations; the ability to block users and unpublish their content and comments in a single action. The functionality is pretty basic but it does save a lot of time where needed and conserves user data while administrators decide their next action.

Use Case

This module comes in handy on sites that contain user-generated content. When dealing with such privileged users in some cases it’s necessary for the administrator to, temporarily or permanently, hide content posted by a particular user. Block and Unpublish allows you to do this easily, without having to sift through heaps of content and edit nodes/comments individually.

How to use

It’s quite straight forward. Simply navigate to the “People” page, tick the users whom you want to block and unpublish their content, then choose the “Block user and unpublish content & comments” option and submit.

  You can also revert this action by following the same instructions but selecting “Undo: Block user and unpublish content & comments” instead.
Categories: Drupal

Open Source Training: Understanding The New Drupal Release Cycle

Wed, 2014-02-26 07:46

At the end of 2013, big changes were made to the Drupal release cycle.

You can read full details of the changes at https://drupal.org/node/2135189.

The changes will start with Drupal 8, but are going to impact almost all current and future versions.

In this blog post, I'll give you a short, plain-English overview of the changes.

I'll show how the changes will impact users of Drupal 6, 7, 8 and 9.

Categories: Drupal

TimOnWeb.com: Fixing Views' Scroll to Top When You Have a Fixed Header

Wed, 2014-02-26 07:41

Drupal Views offers us a cool feature: ajaxified pagers. When you click on a pager, it changes the page without reloading the main page itself and then scrolls to the top of the view. It works great, but sometimes you may encounter a problem: if you have a fixed header on your page (the one that stays on top when you scroll the page) it will overlap with the top of your view container thus scroll to top won't work preciselly correct and the header will cover the top part of your view.

Drupal Tags  drupal 7, drupal planet Read on about Fixing Views' Scroll to Top When You Have a Fixed Header
Categories: Drupal

Powered By