In Part 1, I talked about using Google Docs + Migrate to populate your site. Now we’re going to do that with the Migrate Google Sheets module. Below, I’ll provide the steps to get your own migration up and running, but if you prefer to experiment with a working example, check out a demo of the Migrate Google Sheets Example module (provided as a submodule within Migrate Google Sheets). All content on that site was built using this example Google Sheet.Setup: Install the Module
If you’ve already got a Drupal 8 site up and running, you can install the module in any of the normal ways. I’m assuming here that you have access to the site using Drush, as it’s not possible to run migrations through anything but the command line at this time. At ThinkShout, we use composer to build our site distributions, and have a repo for building the demo site here.Step 1: Creating Your Custom Migration Module
The easiest way to get started on your own set of migrations is to copy the migrate_google_sheets_example submodule and rename it something of your own. Let’s say we rename it “my_migration.” Follow these steps:
- Rename your .install file to “my_migration.install”, and change the function migrate_google_sheets_example_uninstall to “my_migration_uninstall”.
- Delete the helper submodule “migrate_google_sheets_example_setup” entirely – that is just necessary to build the content types required for the example module, but you shouldn’t need it for your migration module.
- Rename your migrate_google_sheets_example.info.yml as “my_migration.info.yml” and open it up. At the very least, you’ll want to change the name of the migration to “name: my_migration” but you’ll also likely wish to remove the migrate_google_sheets:migrate_google_sheets_example_setup dependency. Mine ended up looking like this:
When completed, your module structure should look like this:
You are now ready to enable your My Migrations module. (Make sure you disable the migrate_google_sheets_example module first, if you’d previously enabled that!)Step 2: Create Your Spreadsheet
Assuming you have the Game and Landing page content types, you could now run the migrations in your “My Migrations” module and it will pull the data from the Google Sheet.
But since you don’t have permissions to edit that sheet, you’re going to need to copy the existing sheet and create your own to do any customizations.
When this is done, you’ll get a url like this:
https://docs.google.com/spreadsheets/d/YourLongHashIDHere where YourLongHashIDHere is your feed key.
Now you’ll need to publish your new spreadsheet. This is an option under “File” -> “Publish to the web”
To verify that your migration module will be able to see the Google sheet, try opening an anonymous browser window and visiting the Feed version of the url, whose format is this:
If visiting that URL throws out a bunch of json, you’re ready to start migrating!
But of course, your current set of migration files still point to the old feed. In the my_migrations/config/install folder, you’ll need to find all instances of our feed string (1spS1BeUIzxR1KrGK2kKzAoiFZii6vBHyLx_SA0Sb89M) and replace them with your feed string.Step 3: Decide Which Migrations to Keep
The Migrate Google Sheets Example module provides one Migration Group (games_example) and 6 Migrations. Depending on your site configuration, some of these might be useful, like the menu_links and the blocks migrations, and some of them will not be so useful (like the node_game migration, likely). This is a good time to trim or modify any migrations that aren’t going to be useful for your Drupal site. That being said, here are a few things that the sample migrations demonstrate:
- Block UUIDs: When you place a block using the Block Layout screen, the block’s UUID is saved in config. If you’re running a migration over and over, your block’s ID will iterate on its own, but the UUID can remain constant if you add it to the migration. In the demo site, this allows us to create a persistent CTA block in the header.
- Menu Links parents: You can specify that a menu link item has a parent from within the current migration. This lets us say /bohnanza and /hanabi are children of /games
- Page and Game redirects: These sheets demonstrate how to add the redirect from the url of content on an old site to the new home right in the content sheet. Try going to https://live-mgs-demo.pantheonsite.io/that-fireworks-game and see where you end up.
- Related content as strings or ids: On the Page sheet, we have a reference to the “Related games” for the given page. This is an entity reference which we could fill with a couple of things. We could refer to the ID of the related games, as they are stored in the Games sheet, or we could do what we’ve done here and use the migrate_plus plugin “entity_lookup” to search for the related game node by name. As long as there is a Game node called Bohnanza, we’ll always link to the right one. This is particularly useful with Term references, where the name of the item ideally remains constant.
- Game downloadable file: Games have associated images, which are files hosted externally to the spreadsheet. In order to relate my game content to its image, I need to download the image, get it into the file_managed database table (creating a file entity) and THEN relate that entity to the current node. This is done with the following lines in the “node_games” migration:
You can keep as many or as few of the migration files as you’d like. You can also create new ones.Step 4: Tell Drupal About Your Changes
Drupal 8 only sees the changes you’ve made to your migration yml files when you first install the module. That means that you need to uninstall and reinstall the module to make new changes show up. ThinkShout has a Robo script that does this, but the same thing can be done in Drush:drush mr --all # Rolls back all migrations drush pmu my_migration -y # Disables my migration module drush en my_migration -y # Enable my migration module drush ms # Displays my current migration status
You can also string these together as one line:drush mr --all && drush pmu my_migration -y && drush pmu my_migration -y && drush ms Step 5: Run your migrations
This part is simple. To run all migrations, it’s a quick drush command:
drush mi --all
If you’d like to find out about all the possible options for the migrate-import command, you can run
drush help mi
You can also see your list of migration groups at /admin/structure/migrate, and you can review your migrations by clicking “List Migrations.” The resulting page will give you the same output, more or less, that you get from running a drush ms.
These pages are helpful to know about, as they give you an easy place to find errors logged during the migration process. However, you can’t currently run a migration from the UI (although there is an issue for this).Gotchas
But before we close, I do want to acknowledge some challenges we’ve seen in this approach.
Sad fact #1: HTML in a spreadsheet is ugly.
Google Spreadsheets don’t let you make your rows smaller than the number of line breaks in a cell. So if you have pretty html with a bunch of line breaks, your row might be too tall to fit on your screen. People have some clever workarounds for this, but so far we’ve not implemented any.
Sad fact #2: Sheet order matters (right now)
Maintaining the order of sheets isn’t top on everyone’s minds as they’re making changes to a spreadsheet, especially when duplicating tabs. Migrate Google Sheets asks for each sheet based on tab order. If I make a copy of the Page tab, the Game tab is now the fourth tab instead of the third tab.
As it stands now, the module will happily request columns that don’t exist on the third tab and then fail in puzzling ways.
There is currently only one issue in the issue queue for the Migrate Google Sheets module, and it’s to fix this.
Sad fact #3: Google sheets must be publicly viewable to work (again, right now)
As the module exists right now, there’s no authentication involved, so any migrated content must be publicly viewable. Google authorization is possible with Oauth2, but that is not currently implemented.Conclusion
In 2016, our team worked on a sponsored contrib module, GatherContent. The goal of the project was to recreate the module for Drupal 7 using best practices, and to create a brand new module for Drupal 8.
This video tutorial was made for our customer in order to demonstrate how to build a custom view to extract the data they need.
Here we are building a view to extract sales data per project where each project is classified into a category. We link invoice table with project table and add filter to be able to view data by year and category of project.
The tables and data sources used in this view are custom tables from our back office management solution built on Drupal 8. However, the principles of building a view are applicable to any other data source and this tutorial can be used to learn simple view building with tables relationships and filter.
Your browser doesn't support HTML5 video tag.
After migrating an older Drupal 6 site with 20,000 media items to Drupal 8, I found a strange problem with image uploads. On the Drupal site, using Image FUpload and Adobe Flash, I could upload up to 99 images in one go. On the new Drupal 8 site, I was only able to upload 20 images, even though I didn't see an error message or any other indication that the rest of the images I had selected through the Media Image upload form were not successfully added.
I could choose 21, 40, or 500 images, but only 20 were ever added to an album at any time.
There were no apparent warnings on the screen, so I just assumed there was some random bug in the Media Image Entity or Media module suite that limited uploads to 20 files at a time.
But due to an unrelated error, I glanced at the PHP logs one day, and noticed the following error message:
I was starting my weekly work for the Drupal GraphQL module by the customary composer update --prefer-source -vvv command, ready to watch composer spit out some hundred lines sipping my coffee, but this time something turned out to be wrong:[...snip...]
The "https://packages.drupal.org/8/drupal/provider-2016-4%24a30289dd8394e5271bd77777bb14b361c5938656f1cddad7fae1c00d5d6ba9c6.json" file could not be downloaded (HTTP/1.1 404 Not Found)
OK, so packages.drupal.org is down for now. How can we work around this ?
This comprehensive guide is intended to help get you started when developing a governance plan for your institution’s digital communications.In this guide we will cover...
- What digital governance is
- Why digital governance is important
- What topics you need to think about when building out a digital governance plan
We want to make your project a success.Let's Chat.
Download the guide here.
Stay connected with the latest news on web strategy, design, and development.Sign up for our newsletter.
Drupal 8.3 is still a few months away, coming April 5, 2017, but there are already some changes we can look at, most notably in the experimental modules. In December, 2 new experimental modules were added to core, and BigPipe was officially changed from a beta module to stable. The 2 new modules you'll find in 8.3 are Workflows and Layout. Let's take a peek at what these are all about.
In this five-part series, every Monday in January we’ll explore a New Year’s resolution and how it can apply to your web project.
We want to make your project a success.Let's Chat.
Unless you’re Marie Kondo of the KonMari Method and make your living organizing, you likely have a few messes somewhere. Photo albums? That pile of books? Year-end taxes and receipts? (‘Tis almost the season for that too.)
When it comes to your online space, here are a few tips we can offer on how to get organized for 2017:
- Strategy Matters: How to Start a Successful Project
- Project Kickoffs That Work
- Writing an RFP to Attract the Right Response
- Addressing Accessibility for the Web
- Planning for Long-Term Success on the Web
Lastly, if you want a comprehensive look at how to get your internal governance in order, check out our detailed guide on digital governance (37 page PDF). The ownership, management, and sustainability plan for an organization’s various digital communications platforms seems simple at a distance, but the devil is in the details.
Next week’s resolution: stay fit and healthy.
Let Palantir help you get organized.
We'd love to help you keep your 2017 resolution.Let's chat.
When I was a kid, my siblings and I used to play with Mad Libs. If you aren’t familiar with them, Mad Libs are basically one-page stories or vignettes, with various words missing. Where those words should be are blanks, and each blank is labeled “noun” or “verb” or whatever type of word should be filled in.
One project I'm working on needed a Behat test added to test whether a particular redirection works properly. Basically, I wanted to test for the following:
- An anonymous user accesses a file at a URL like http://www.example.com/pictures/test.jpg
- The anonymous user is redirected to the path http://www.example.com/sites/default/files/pictures/test.jpg
Since the site uses Apache, I added the actual redirect to the site's .htaccess file in the docroot, using the following Rewrite configuration:<IfModule mod_rewrite.c>
# Rewrite requests for /profile_images to public files directory location.
RewriteRule ^ pictures/(.*)$ /sites/default/files/pictures/$1 [L,NC,R=301]
Testing with curl --head, I could see that the proper headers were set—Location was set to the correct redirected URL, and the response gave a 301. So now I had to add the Behat test.
I'm releasing a new series today!!!!!
Over the last year, I've given a talk at DrupalCon, DrupalCorn Camp, and Drupal Camp Colorado all about using Composer and Configuration Management in Drupal 8. Those sessions were around 45 minutes, which is much too short to go in depth, and explain everything thoroughly.
There is just over 1hr 15min of content in 26 videos that fall pretty well into seven parts. Here's the outline:Part 1: Intro
- Creating a New Drupal 8 Project Using the Composer Template
- Setting Up MAMP to Serve Your Site Locally
- Using xip.io for Local Device Testing
- Creating a Drush Alias
- Installing Drupal with Console
- Configuring settings.php and settings.local.php
- Committing Your Project to Git
- Installing and Uninstalling Modules with Composer
- Installing the Dev Version of Modules
- Updating and Downgrading Modules
- Skipping Specific Module Versions
- Specifying Acceptable Version Ranges
- Enabling Modules with Drush and Deciding What Version Pattern to Use
- Setting the Config Directory in settings.php
- Exporting Config Locally
- Using the Configuration Installer Install Profile
- Installing the Site on a Production Server with Composer
- Setting up settings.local.php
- Changing the Site Name and Disabling CSS Aggregation in settings.local.php
- How to Enable Theme Debugging on Development Sites
- Overriding Module Configuration (Like Google Analytics) in settings.local.php
- Configuring a Local Site and Exporting it's Configuration with Git
- Pulling Changes to a Remote Site (And some gotchas)
- Using Drush Shell Aliases to Make Development Easier
- Verifying the Changes Made on Local and Reflected on Live
I'm pretty excited to finally have this series out, and hope you enjoy it! Oh... did I mention that it's ABSOLUTELY FREE?!?! Well, it is! So, check it out now, and let me know what you think.Tags: Drupal 8ComposerConfiguration ManagementLocal DevelopmentDrushGitShell-Aliasesplanet-drupal
OK - I'll hold my hands up. The title of this post is misleading. I'm not going to give you an ABC on how to secure a Drupal site (maybe another day). I'm responding to a post on the Reseller Club blog entitled How to Secure Your Client's Drupal Website.
There is some good advice in that article, but it's mixed in with some bad advice, and in other parts it's just plain confused. In the hope that it helps people, I'm going to try and untangle things.Blog Category: Drupal Planet
Drupal 8 has been released more than one year ago, but Drupal 7 is still widely used: it's totally stable, feature-rich, actively maintained and has tons of available modules for functionality extension. In this article I would love to draw your attention to Drupal 7 performance only. You can find a lot of information on Drupal 7 performance over the Internet since Drupal 7 is available for a long time, but anyway things change and new options appear. Website performance is very important because it can lead to higher conversion rates, higher search ranking and hosting costs reduction. I'll focus on Drupal 7 performance on the server side, but there are other areas where website performance can and should be improved including front-end, database, etc. These options are highlighted here.
SQL databases are really fast when you need all the information stored together in a record row, but they are a bad fit when you need to search for relationship patterns that are not already stored together in your database. A significant performance penalty is incurred for every additional table that needs to be joined for a query. That is why SQL databases are notoriously bad at deducting relationships from datasets. Graph databases however are really good at this task.
I like things that work. I think most technicians do, but as a web developer I have a very serious problem. My most effective environment for doing web development is the one that exists on my own personal box. It can also be a rather impractical place to develop because most of my customers (current and historic) are on rather customized server stacks. Typically, the host has customized the environment to their own specifications. It's not uncommon to find additional services like solr or maybe a memcache server in the mix.
The monthly security release window for Drupal 8 and 7 core will take place on Wednesday, January 18.
This does not mean that a Drupal core security release will necessarily take place on that date for any of the Drupal 8 or 7 branches, only that you should watch for one (and be ready to update your Drupal sites in the event that the Drupal security team decides to make a release).
There will be no bug fix or feature release on this date. The next window for a Drupal core patch (bug fix) release for all branches is Wednesday, February 01. The next scheduled minor (feature) release for Drupal 8 will be on Wednesday, April 5.
Drupal 6 is end-of-life and will not receive further security releases.
Within weeks of introducing the contribution credit system on Drupal.org we realized we had created something powerful. Like all open source projects, Drupal has a behind-the-scenes economy of contribution in which individuals, organizations, and end users work together to maintain the software as a public good. That behind-the-scenes economy was brought to the fore when we chose to rank the Drupal Marketplace by issue credits. For the first time, Drupal.org gave businesses a direct financial incentive to contribute code.
Being good stewards of these incentives is a sobering responsibility, but also a great opportunity. We can use this system to recognize the selfless effort of our community volunteers, to reward the organizations that sponsor their employees' time to give back to the project, and to connect end-users with the organizations that are the biggest contributors.
But as we often say in this community—contribution is more than code. It is the time provided by dedicated volunteers; the talent of community organizers, documentation maintainers, and developers; and the treasure provided by organizations that sponsor Drupal events and fund the operations and infrastructure that maintain the project.What are we changing?
We’re updating the ranking algorithm for Drupal.org’s Marketplace of service providers and list of all organizations in the Drupal ecosystem. We've expanded on the issue credit system to create a more generic contribution credit system which lets us recognize more types of contribution. Each type of contribution is now weighted to give the organization an overall amount of contribution credit. We've built this system so that we can continuously evolve the incentives it creates by adjusting the weight given to each type of contribution as the project's needs change. To prevent gaming, we will not be publishing the exact weights or total contribution score, but those weights have been reviewed by the Association Board and Community Working Group.
We've carefully chosen a few new types of contribution to factor into the ranking. These were selected because they create incentives to reach specific goals: encouraging organizations to sponsor development of Drupal, gathering more Drupal 8 success stories that can be used to promote Drupal adoption, and recognizing the financial contributions that promote the fiscal health of the Drupal association.
We now calculate the following 4 types of contribution into overall contribution credit:
- Issue credits — helping build the Drupal software happens in the issue queues. Issue credits remain the primary factor in ranking, and continue to be shown prominently. Issue credits on more widely used projects, like Drupal Core, will also receive greater weight in the ranking. Learn how to help in the issue queue
- Drupal 8 case studies — success stories show how Drupal is used across industries and the world, helping effectively introduce Drupal to more people. Learn how to write a case study
- Drupal Association Supporter Programs and Organization Membership — our partners and members help us build and maintain Drupal.org. Learn about supporter programs and organization membership
- Projects supported — the work to maintain a project sometimes happens outside of issues. Project maintainers can credit organizations which help provide time and sponsorship. Learn more about crediting project contributions
Of course, these new factors still don't include all types of contribution. This iteration aims to add measurable factors that reward the behavior of organizations that are good Drupal citizens, and incentivize some of the most important contributions that have a big impact in moving the project forward. But there are other factors we'd like to include in the future! We're keeping track of these additional kinds of contribution, such as sponsoring local user groups, organizing training days, writing documentation, and more, in this issue: #2649100: Improve contribution statistics on user and organization profiles.
There are two factors in particular that we are not yet including that we'd like to address.
The first is project application reviews. These reviews are a critical part of the lifecycle of a new project on Drupal.org, but because we are making the Project Application Revamp a key priority for the first part of 2017, this was not our focus in this initial update. We may revisit this factor as the Project Application Revamp initiative gets underway.
The second is camp organization. We know that there are many individuals and organizations who invest heavily in Drupal Camps, and this has been a critical part of the project's success. However, at this time our data about the individuals and organizations who participate in camp organization is purely self-reported, and therefore too vulnerable to manipulation to include in the algorithm at this time. In the future we hope we can find a responsible way to measure and credit this kind of contribution.
We’ll continue to look for other good factors to add, and do our best to weigh them fairly.How often will the algorithm change? Who governs these changes?
As this is our first major change to the marketplace ranking system since the launch of issue credits, we may need to make some small adjustments in the first weeks following the launch. However, we know that too frequent changes to the incentive structure will be frustrating for the individuals and organizations who are contributing to the project. Therefore, after the initial tuning we intend to update the marketplace ranking system on a roughly 6 month cycle.
While the primary responsibility to manage the contribution credit system is ours, we have committed to vetting these and future changes with members of the Drupal Association Board and Community Working Group.
Following the previous blog post about our software engineering team culture that I wrote with my colleague Andrew Harmel-Law, I spoke about the subject at the January Drupal Show & Tell last night.
I've been meaning to speak at a meetup for a long time, and if I hadn't done it last night, I'd probably be putting it as one of my objectives for the year. The trouble was, I could never think of what to say. But conversations turned to tweets turned to blog posts, and it felt right to talk about this subject, particularly given that one of the themes of the blog post is the importance of communication between people.
I'd been to the Drupal Show & Tell meetup a couple of times before, and it's a friendly group with some familiar faces, so when I saw the call for speakers, it seemed the ideal opportunity for my first venture into public speaking.
As I rode my bike through the snow to the meetup, I was a little worried that the attendance might be a little sparse, and my blocked nose wasn't helping my confidence. After a few anxious moments where we thought there might be more speakers than people in the audience, more people arrived, and we got started, with interesting and thought-provoking talks from Anthony Seale and Nigel Milligan.
Finally, it was my turn, and despite losing my thread once or twice, I think it went fairly well for a first attempt. As I mentioned in the talk, one of the key points is about improving through iteration - I'll be tweaking the talk and delivering a new version of it at one of our internal lightning talks sessions soon.
Your browser does not support iframes. Please visit https://malcomio.github.io/presentations/how-we-work/#/ to view the presentation.Tags: Drupal Presentation development All tags