Migrating a Drupal 7 site with “Migrate Drupal”

Speed up migration development

One of the things that Drupal developers do for clients is content migration. This process uses hours of development time and often has one developer dedicated to it for the first half of the project. In the end, the completeness of the migration depends partly on how much time your client is willing to spend on building out migration for each piece of their content and settings. If you’ve come here, you probably want to learn how to speed up your migration development so you can move on to more fun aspects of a project.

The challenge

Our client, the NYU Wagner Graduate School of Public Service was no exception when they decided to move to Drupal 8. Since our client had 65 content types and 84 vocabularies to weed through, our challenge was to build all those migrations into their budget and schedule.

The proposed solution

Since this was one of our first Drupal 8 sites, I was the first to dig my hands into the migration system. I was particularly stoked in the fact that everything in Drupal 8 is considered to be an entity. This opened up a bunch of possibilities. Also, the new automated migration system—Migrate Drupal—that came with core was particularly intriguing. In fact, our client had started down the path of using Migrate Drupal to upgrade their site to D8. Given they had field collections, entity references, and the fact that the Migrate Drupal module was still very much experimental for Drupal 7 upgrades, this didn’t pan out with a complete migration of their data.

The proposed solution was to use the --configure-only method on the drush tool migrate-upgrade. Doing so would build out templated upgrade configurations that would move all data from Drupal 7 or Drupal 6 to Drupal 8. The added bonus is that you can use that as a starting point and modify them from there.

Migration in Drupal 7 vs Drupal 8

Since we have the 100 mile high view of what the end game is, lets talk a little about why and how this works. In Drupal 7 Migrations are strictly class-based. You can see an example of a Drupal 7 migration in the Migrate Example module. The structure of the migration tends to be one big blob of logic (broken up by class functions of course) around a single migration. Here are the parts:

• Class Constructor: where you define your source, destination, and field mapping
• Prepare: a function where you do all your data processing

In Drupal 8, the concept of a migration has been abstracted out into the various parts that makes them reusable and feel more like “building with blocks” approach. You can find an example inside the Migrate Plus module. Here are the parts:

• Source Plugins: a class defining the query, initial data alteration, keys, and fields provided by the source
• Destination Plugins: a class defining how to store the data received in Drupal 8
• Process Plugins: a class defining how to transform data from the source to something that can be used by the destination or other process plugins; you can find a full list of what comes with core in Migrate’s documenation
• Migration Configuration: a configuration file that brings the configuration of all the source, destination, and process plugins to make a migration

Now yall might have noticed I left out hook_prepare_row. Granted, this is still available. It was also a key way many people used to manipulate data across several source fields that behaved the same. With the ideal of process plugins, you can now abstract out that functionality and use it in your field mapping.

How “Migrate Drupal” makes the process better

There are tons of reasons to use Migrate Drupal to start your migration.

It builds a migration from your Drupal site

You might have seen above that I mentioned that Migrate Drupal provides a templated set of configurations. This is a product of some very elaborate migration detection classes. This means you will get all the configurations for:

• content types
• field setup
• field configuration
• various site settings
• taxonomy vocabularies
• custom blocks
• content and their revisions
• etc…

These will be built specifically for the site you are migrating from. This results in tons of configuration files—my first attempt created over 140 migration YAML files.

It’s hookable

Hookable means that it’s not just a part of core thing and that it’s expandable. That means that contributed modules can provide their own templates for their entities and field types, allowing Migrate Drupal to move over that data too. For example, it is completely possible (and in progress) for the Field Collection module to build in migration templates so that the migration will know how to import a field collection field. Not only that, the plugins provided by the contributed modules can be used in your custom migrations as well.

No initial need for redirection of content

Here’s an interesting one, everything comes over pretty much verbatim. Node IDs, term IDs, etc. are exactly the same. URL aliases come over, too, by default. Theoretically, you could have the same exact site from D7 on D8 if you ported over the theme.

More time to do the alterations the client wants

Since you aren’t spending your time building all the initial source plugins, process plugins, destination plugins, and configurations, you now have more time to alter the migrations to fit the new content model, or work with the new spiffy module like paragraphs.

How-To: Start a Migration with “Migrate Drupal”

Ok so here is the technical part. From here on is a quick How-To that gets you up and going. Things you will need are:

• a Drupal 6 or 7 site
• your brand new Drupal 8 site
• a text editor
• drush

1. Do a little research and install contrib modules

We first need to find out if our contrib modules that are installed on our Drupal 6/7 site are available and have a migration component to them in Drupal 8. Once we identify the ones that we can use, go ahead and install them in Drupal 8 so they can help you do the migration. Here are a couple of thoughts:

Is the field structure the same as in Drupal 6/7? The entity destination plugin is a glorified way to say $entity->create($data); $entity->save();. Given this, if you know that on Drupal 6/7 that the field value was, for example…

[
  'value' => 'This is my value.',
  'format' => 'this is my format'
]

…and that it’s the same on Drupal 8, then you can rest easy. The custom field will be migrated over perfectly.

Is there a cckfield process plugin in the Drupal 8 Module for the custom field type? When porting fields, there is an automated process of detecting field types. If the field type you are pulling from equates to a known set of field types by the cckfield migration plugin, it will be used. You can find these in src/Plugin/migrate/cckfield of any given module. The Text core module has an example.

Is there a migration template for your entity or field in the Drupal 8 module? A migration template tells the Drupal Migrate module that there are other migrations that need to be created. In the case of the Text module. you will see one for the teaser length configuration. There can be multiple and look like migrations themselves, but are appended to in such a way to make them special for your site. You can find these in
migration_templates in the module.

Are there source, process, or destination plugins in the Drupal 8 module? These all help you (or the Migrate Drupal module) move content from your old site to your new one. It’s very possible that there are plugins not wired up to be used in an automated way yet, but that doesn’t keep you from using them! Look for them in src/plugin/migrate.

2. Install the contrib migrate modules

First you must install all the various contributed modules that help you build these configurations and test your migrations. Using your favorite build method, add the following modules to your project:

• Migrate Plus
• Migrate Tools
• Migrate Upgrade

NOTE: Keep in mind that you will need to be mindful of the version that goes with what version of Drupal Core. Example 8.x-1.x goes with Drupal 8.0.*, 8.x-2.x goes with Drupal 8.1.*, and 8.x-3.x goes with Drupal 8.2.*.

3. Set up the upgrade/migrate databases

Be sure to give your database an key. The default is ‘upgrade’ for drush migrate-upgrade and ‘migrate’ for drush migrate-import. I personally stick with ‘migrate’ and just be sure to give the custom settings to migrate-upgrade. I use drush migrate-import a ton more than drush migrate-upgrade.

$databases = [
  'default' => [
    'default' => [
      'database' => 'drupal',
      'username' => 'user',
      'password' => 'pass',
      'host' => 'localhost',
      'port' => '',
      'driver' => 'mysql',
      'prefix' => '',
    ],
  ],
  'migrate' => [
    'default' => [
      'database' => 'migrate',
      'username' => 'user',
      'password' => 'pass',
      'host' => 'localhost',
      'port' => '',
      'driver' => 'mysql',
      'prefix' => '',
    ],
  ],
];

4. Export the migration configuration

First I want to give credit to Mike Ryan for originally documenting this process. Without it, or his help in IRC, you wouldn’t have gotten this article today.

Go ahead and import your Drupal 6/7 database if you aren’t connecting to a live instance in your database settings with your preferred method. Take your pick:

drush sql-sync

drush sql-drop --database=migrate; gunzip -c /path/to/migrate.sql.gz | drush sqlc --database=migrate

Next run Migrate Upgrade to get your configuration built and stored in the Drupal 8 site.

drush migrate-upgrade --legacy-db-key=migrate --configure-only

Finally store your configuration. I prefer just to stick it in the sync directory created by Drupal 8 (or in my case configure for checking into Git).

drush config-export sync -y

I’m verbose about the directory because we usually have one for local development stored in the local key also. You can leave off the word sync if you only have a single sync directory.

5. Update your migration group with the info for the migration

This is a quick and simple step. Find migrate_plus.migration_group.migrate_drupal_7.yml or migrate_plus.migration_group.migrate_drupal_6.yml and set the shared configuration. I usually make mine look like this:

langcode: en
status: true
dependencies: {  }
id: migrate_drupal_7
label: 'Import from Drupal 7'
description: 'Migrations originally generated from drush migrate-upgrade --configure-only'
source_type: 'Drupal 7'
module: null
shared_configuration:
  source:
    key: migrate

5. Alter the configuration

Ok here comes the fun part. You should now have all the configurations to import everything. You could in fact now run drush mi --all and in theory get a complete migration of your old site to your new site in the data sense.

With that said, you will most likely need to make alterations. For example, in my migration we didn’t want all of the filters migrated over. Instead, we wanted to define the filters first, and then use a map to map filters from one type to another. So I did a global find across all the migration files for:

    plugin: migration
    migration: upgrade_d7_filter_format
    source: format

And replaced it with the following:

    plugin: static_map
    source: format
    map:
      php_code: filter_null
      filtered_html: basic_html

Another example of a change you can make is the change of the source plugin. This allows you to change the data you wanted. For example, I extended the node source plugin to add a where-clause so that I could only get data created after a certain time.

namespace Drupalwg_drupal7_migratePluginmigratesource;

use DrupalnodePluginmigratesourced7Node as MigrateD7Node;
use DrupalmigrateRow;

/**
 * Drupal 7 nodes source from database.
 *
 * @MigrateSource(
 *   id = "wg_d7_node",
 *   source_provider = "node"
 * )
 */
class Node extends MigrateD7Node {

  /**
   * {@inheritdoc}
   */
  public function query() {
    $query = parent::query();
    // If we pass in a timestamp... only get things created since then.
    if (isset($this->configuration['highwater'])) {
      $query->condition('n.created', $this->configuration['highwater'], '>=');
    }
    return $query;
  }
}

Lastly, you may want to change the destination configuration. By default, the configuration of the migration will go to a content type with the same name. It may be possible that you changed the name of the content type or are merging several content types together. Simply altering…

destination:
  plugin: 'entity:node'
  default_bundle: page

…to be…

destination:
  plugin: 'entity:node'
  default_bundle: landing_page

…may be something you need to do.

Once you are done altering the migration save the configuration files. You can use the sync directory or if you plan on distributing it in a module, you can use the
config/install folder of you module.

Rebuild your site with the new configuration via your preferred method, or simply run drush config-import sync -y.

6. Migrate the data

This is the last step. When you are ready, migrate the data either by running each of the migrations individually using --force, run the migration even though other pieces haven’t, use the --execute-dependencies, or just go ahead and go for the gold drush migrate-import --all

Caveats

So finally after you go through all the good news, there are a few valid points that need to be made about the limitations of this method.

IDs are verbatim due to the complexity of dependencies

So this means that the migrations are currently expecting all the nids, tids, fids, and other IDs, to be exactly what they were on Drupal 6 or 7. This causes issues when your client is building new staged data. You have three options in this case:

1. Alter the node, node_revision, file_managed, taxonomy_term_data, users, and probably some others I’m missing here that house the main entities that entity reference fields will need, so that their keys are something your client will not reach on their current production site while you are developing.
2. Do not start adding or altering content on Drupal 8 until all migrations are done.
3. Go through all the migrations and add migration process plugins where an entity is referenced, and then remove the main id from the migration of that entity.

In my case, I went with the first solution because this realization hit me kinda late. Our plan was to migrate now for data so our client would have something to show their stakeholders, and then migrate again later to get the latest data before going live.

There are superfluous migrations

You will always find out that you don’t want to keep the settings verbatim to the Drupal 6 or 7 site. This means you will have to remove that migration and remove it’s dependency from all the other migrations that depend on it. Afterwords, you will need to make sure that that case is covered. I shared an example in this article where we decided to go ahead and configure new filter formats. Another example may be that you don’t even give a crap about the dblog settings from your old Drupal site.

Final thoughts

For NYU Wagner, we were able to save a ton of time having the migrations built out for us to start with. Just the hours spent on building the field configurations for the majority of the content types that were to stay the same was worth it. It was also a great bridge into “How Do Migrations Work?” We now have a more complete custom migration for our client in a fraction of the time once our feature set was nailed down, than if we were to go build out the migrations one at a time. Happy migrating.