• Published:
  • Under: Development

Recently I was given the task of importing about 150 users into a new Drupal 8 site. This amount of data usually requires an automated migration and in Drupal 7 the Feeds module would have been perfect. It required minimal effort from developers to configure, so content entry could run concurrently with site development. Unfortunately in Drupal 8, the Feeds module isn’t ready for production sites, so developers have to create database migrations which means their time may be split between content entry and site development.

After doing some research I found the Migrate Source CSV module, which is an extension for Drupal 8’s Core Migrate module. While this module requires more configuration than Feeds, it can free up a developer from digging through databases to map content from one site to another. To test the module, I used a simple view from the old Drupal 7 site to create a file with all the relevant user information I needed and then cleaned up the content using Google Spreadsheets. The following custom module has been made specifically for User profiles, but could easily be modified to work with any type of data.

The User

In this example we are expanding the User entity with a couple more fields. Along with the standard fields (username, password, status and user picture) we will add a first name, last name and department. The first and last name will be simple strings, while the department will be a entity reference field, specifically a taxonomy reference. 

You will need to update your User fields with the following:

  • First Name - plain text field (field_first_name)
  • Last Name - plain text field (field_last_name)
  • Departments - taxonomy reference field, with at least 8 random items (field_department)

The actual Department labels don’t matter, since we will be using their tids to connect the user to the department.

The Modules

Before starting, let’s download and install all the contrib modules we will need. Migrate is in core, so it will just need to be installed.

All the code for a migration needs to be contained in a custom module. This way the module could be installed on any site and with the new fields and contrib modules installed. I’ve developed this module so it can run even if there is already data on the site. The folder and file naming is important so the contrib modules know where to look for files.

Custom module structure:

teacher_migrate
 teacher_migrate.info.yml
 teacher_migrate.module
 /config/user.csv
 /config/install/migrate_plus.migration.teacher_image.yml
 /config/install/migrate_plus.migration.teacher_user.yml

The info file can be kept very simple, just naming the module and setting the required names and descriptions.

<teacher_migrate.info.yml>
name: Teacher Migrate
type: module
description: Custom Migrate of Teachers using a CSV file
core: 8.x
package: Migration
dependencies:
    - migrate_source_csv
    - migrate_plus
    - migrate_tools

 

Most the work is being done in the YAML files, so the module file only has to set the location of module using hook_migration_plugins_alter and drupal_get_path so we don’t have to include direct paths to the files when mapping the content. I’ve also added the basic hook_help if you need to add anything special for other developers.

<teacher_migrate.module>
<?php

/**
 * @file
 * Contains teacher_migrate.module.
 */

use Drupal\Core\Routing\RouteMatchInterface;

/**
 * Implements hook_help().
 */
function teacher_migrate_help($route_name, RouteMatchInterface $route_match) {
  switch ($route_name) {
    // Main module help for the custom_user_migrate module.
    case 'help.page.teacher_migrate':
      $output = '';
      $output .= '<h3>' . t('About') . '</h3>';
      $output .= '<p>' . t('Teacher Migrate') . '</p>';
      return $output;

  }
}

/**
 * Implements hook_migration_plugins_alter().
 */
function teacher_migrate_migration_plugins_alter(&$definitions) {
  $definitions['teacher_user']['source']['path'] =
    drupal_get_path('module', 'teacher_migrate') . $definitions['teacher_user']['source']['path'];
  $definitions['teacher_image']['source']['path'] =
    drupal_get_path('module', 'teacher_migrate') . $definitions['teacher_image']['source']['path'];
}

The CSV file is replacing our need for a database in the migration, but we will have to make sure it has all the content we need. If there columns or tables you know will have default values, they can be left out and added programmatically with Drupal Migrate’s default_value plugin in the YAML file or added to the CSV and mapped with the rest of the data. I have included examples of both in this module. In the CSV you may notice I don’t have any of the files locally stored, I’m using an external path and Drupal’s file_copy plugin to copy the file to my Drupal 8 default/files folder.

Now that we have our info, module and CSV created we can start mapping our content with the migration YAML files. To make sure I accounted for the required columns, I did most of the data entry in the CSV file instead of during the mapping process. As you get more comfortable with migration YAML files, a lot of the data manipulation could be done during migration mapping to cut down on repeated information in the CSV.

Mapping the Teacher Pictures

The User entity requires two migrations to complete, because of an issue I found with User pictures. The User picture is not a traditional file upload field, but actually a reference to a file fid, which means we need to migrate all the images first and then reference them when migrating the users.

In the first part of the file, we define the migration and set dependencies.

id: teacher_image
status: true
label: Teacher Image Migration
dependencies:
    enforced:
        module:
            - teacher_migrate

 

Next we set the source for the migration. Here we tell Drupal to use the Migrate Source CSV module by setting plugin to CSV. We are also setting the relative path to the CSV thanks to the hook_migrations_plugin_alter in our module file. Unique to the CSV migration, we need to set how many header rows are in the CSV and I’ve added a key for the unique_id in our CSV to make sure we can tie this migration to the next one.

source:
  plugin: csv
  track_changes: true
  path: /config/user.csv
  header_row_count: 1
  keys:
    - unique_id

 

The destination for our migration uses the entity:file plugin so Drupal knows we are adding files and where to save them. The source_path_property and destination_path_property are both mapped to columns in our CSV.

destination:
  plugin: entity:file
  source_path_property: filepath
  urlencode: true
  destination_path_property: uri

 

The final part of this migration is mapping the actual file data for Drupal to migrate. The first process is for the filename. The get plugin is always used if no plugin is defined, but I have to define it here since I also setup the migration to skip the entire row if there is no filename.

process:
  filename:
    -
      plugin: get
      source: filename
    -
      plugin: skip_on_empty
      method: row

 

Next we migrate the file itself using the file_copy plugin. This will download the file using the uri path we set in the CSV. Without this step, the file information may get migrated in but the file won’t exist in default/files on the new server. If you have already copied the files over, you may be able to skip this file copy step and just map out the information. We finish the migration by mapping the filemime, status, and setting the uid to the admin account with the default_value plugin.  

  uri:
    plugin: file_copy
    source:
     - filepath
     - uri
  filemime: filemime
  status: status
  uid:
    plugin: default_value
    default_value: 1

Your final file should look like this

<migrate_plus.migration.teacher_image.yml>
id: teacher_image
status: true
label: Teacher Image Migration
dependencies:
    enforced:
        module:
            - teacher_migrate
source:
  plugin: csv
  track_changes: true
  path: /config/user.csv
  header_row_count: 1
  keys:
    - unique_id
destination:
  plugin: entity:file
  source_path_property: filepath
  urlencode: true
  destination_path_property: uri
process:
  filename:
    -
      plugin: get
      source: filename
    -
      plugin: skip_on_empty
      method: row
  uri:
    plugin: file_copy
    source:
     - filepath
     - uri
  filemime: filemime
  status: status
  uid:
    plugin: default_value
    default_value: 1

 

Mapping the Users

Now that we have all the images mapped we have to map the actual users. This YAML file is very similar to the previous one, so I am only going to call out the major differences.

Our destination now uses the entity:user plugin.

destination:
    plugin: entity:user

 

After mapping each field to its column in the CSV we have to map the user_pictures. This starts out like our image mapping above, if the filename is empty, instead of skipping the entire row the field is skipped with the skip_on_empty plugin. If there is a filename the plugin migration will allow us to map this field using the mappings of another migration. Since only the fid is needed, but this could be different each time if there are files already uploaded to the new site, the unique_id will be our key. The migration plugin will look at the teacher_migration mapping that is saved in your database and find the file with the same unique_id and set the reference for us. I also included no_stub so a placeholder isn’t created if nothing is found.  

user_picture:
        -
            plugin: skip_on_empty
            method: process
            source: filename
        -
            plugin: migration
            migration: teacher_image
            source: unique_id
            no_stub: true

The complete teacher_user file. For a full explanation on all plugins used and more that I didn’t list, Drupal has them listed here.

<migrate_plus.migration.teacher_user.yml>
id: teacher_user
langcode: en
status: true
dependencies:
    enforced:
        module:
            - teacher_migrate
label: 'Teacher migration'
source:
    plugin: csv
    track_changes: true
    path: /config/user.csv
    header_row_count: 1
    keys:
        - unique_id
destination:
    plugin: entity:user
process:
    name: username
    field_first_name: field_first_name
    field_last_name: field_last_name
    pass: password
    status: status
    mail: email
    init: init
    field_department:
        plugin: skip_on_empty
        method: process
        source: tid
    user_picture:
        -
            plugin: skip_on_empty
            method: process
            source: filename
        -
            plugin: migration
            migration: teacher_image
            source: unique_id
            no_stub: true
    

 

Let’s Migrate

Migrate Source CSV doesn’t contain a UI so you will need to run the migrations using Drush. Luckily all those modules we enabled give us a couple of commands that make this easy. All the commands I am using can be found here, as it’s always good to look up any commands before running them in terminal.

First we need to enable our new Custom Module. This will install the module and add our YAML files to the site’s config manager. I did not set UUIDs for these YAML files, which means to update them you have to remove them from config and re-upload them each time. I did this as I didn’t want to possibly have any issues when giving this module to another developer to test or install on the database of record to run.

Next in terminal, navigate to the Drupal site and run the following commands:

To see all migrations available and their status:

drush migrate-status

To import the images first:

drush migrate-import teacher_image

And now for the users:

drush migrate-import teacher_user

After both of these finish you should have your users imported and ready for use. If you run into any errors or want to make some changes, you can always run the migrate rollback command on the page above to undo the migrations.

There you have it! If you have any questions or comments please reach on Twitter to get the conversation started.