How to send real time notifications to your Drupal site users

Often you would want to notify logged in users of an urgent deployment that is going to take the site down. For these unavoidable deployments, real-time notifications to end users would be God-send as it would help editors save their work (if they are adding content) in adequate time.

Here is a simple attempt at building one such system. HTML5 Server Sent Events is a one way communication technology which we can leverage for this. The caveat is that editors need to use HTML5 browsers or resort to XHR polling.

In this example, we are using Jenkins for deployment and want to notify logged-in site editors when a deployment is about to happen. We'll build a separate notification server for this purpose (on node.js) that will register browser connections and will also listen for notifications from Jenkins. Once it receives a notification regarding a deployment, it will broadcast that to the registered browsers. We'd like this server to support different sites and their notifications (multi-site scenario in Drupal). Jenkins can send build info notifications via the Notifications plugin. This server has been built and hosted on GitHub along with usage instructions.

When each browser gets loaded, it fires off an EventSource() to the server, registering its connection. The server keeps a list of active connections per site. The Notifications plugin on Jenkins posts JSON build information to our server as it happens, with the site information as parameter. The server in turn broadcasts this to the registered browsers for that site, who then get a Growl/Ubuntu like notification. The GitHub repo contains both the server side code (node.js) and client side javascript with the required polyfill, which can be used for any web application.

To get the client-side working in Drupal, read on.

Drupal

Create a module that adds the client JS and CSS on all pages. It would help to create and use a permission to receive notifications for editors and other roles. Some of the crucial hooks are listed below. The JS assets are loaded on hook_init() based on 'receive sse notifications' permission. For the Growl-like notifications, add in some required bit of html in hook_preprocess_page() which invokes a theme function so as to make it extensible/over-writeable.

/** * Implements hook_init(). */ function mymodule_init() { if (user_access('receive sse notifications')) { drupal_add_js(SSECLIENTPOLYFILLPATH, 'external'); drupal_add_js(variable_get('mymodule_eventsource_url', SSECLIENTJSPATH), 'external'); // Jquery notify additions // See http://www.erichynds.com/blog/a-jquery-ui-growl-ubuntu-notification-widget drupal_add_css(drupal_get_path('module', 'mymodule') . '/assets/css/ui.notify.css'); drupal_add_js(drupal_get_path('module', 'mymodule') . '/assets/js/jquery.notify.js'); // Add the notifications server URL for the SSE client JS to use. drupal_add_js(array( 'mymodule' => array( 'server' => variable_get('mymodule_server_address', SSESERVERPATH) ) ), 'setting'); } }
/** * Implements hook_permission. */ function mymodule_permission() { return array( 'receive sse notifications' => array( 'title' => t('Receive HTML5 SSE notifications'), 'description' => t('Receive notifications regarding site builds in real-time.'), ), ); } /** * Implements hook_theme. */ function mymodule_theme() { return array( 'mymodule_jquery_notify' => array( 'variables' => array(), 'template' => 'jquery_notify', 'path' => drupal_get_path('module', 'mymodule') . '/templates', ) ); } /** * Implements hook_preprocess_page(). */ function mymodule_preprocess_page(&$variables) { if (!user_access('receive sse notifications')) { return; } // Append notify container to footer or content. if (!empty($variables['page']['footer'])) { $variables['page']['footer']['jquery_notify']['#markup'] = theme('mymodule_jquery_notify'); } else { $variables['page']['content']['jquery_notify']['#markup'] = theme('mymodule_jquery_notify'); } }

Improve performance: Organize your files into subfolders - Drupal 7

Having too many files in a single folder can be detrimental to your site's performance. It is advisable to segregate files into multiple folders for various benefits. Your sysadmin would surely thank you later :-)

This Acquia post introduces the problem of file organization and suggests two token based modules. Personally, having tried both, File Entity Paths is my choice as it allows you to create folder structures based on properties of the file per-se and not the parent entity. This is useful if you would not like files from changing their location every time they get reused in another entity and need to be in folders based on that entity's properties. By ensuring a folder structure based on file id (fid) for example, you can also guarantee a more even spread and can limit the number of folders created.

For example, to store up to 1,000,000 files with not more than 1000 in a folder, it means we do not need more than 1000 folders. For an even distribution of files, this folder structure could be: [0-9]/[00-99] where the [0-9] is the third digit from the end of the fid, and the [00-99] digits are based on the last two digits of the fid (left pad fids with '0' for fids < 99). So a file 'image.jpg' having fid 123456 would be in files/4/56/image.jpg.

To implement the above logic, we can modify the helpful Drupal 7 drush script to relocate files based on their fid, instead of the timestamp.

Here's the script (Drupal 7):

	<?php
	/**
	* Implements of hook_drush_command().
	*/
	function migration_filepath_drush_command() {
	$items = array();
	$items['migration_filepath'] = array(
	'description' => "Move files to subdirectories",
	'callback' => 'drush_migration_filepath',
	'aliases' => array('mf'),
	// We are using options rather than arguments so the source can be omitted
	// if needed (e.g. files are just directly under sites/default/files).
	'options' => array(
	'source' => array(
	'description' => 'The directory under files where the to be relocated files are.',
	'example_value' => 'field/image',
	),
	'target' => array(
	'description' => 'The target directory where the files will be moved to and restructured.',
	'example_value' => 'pictures',
	),
	'limit' => array(
	'description' => 'Use this option to explicitly limit the number of files to move. Do not use this option if you want to move all files.',
	'example_value' => '15',
	),
	'list-info' => 'Print total number of files remaining to be moved and other helpful info and exit.',
	),
	'examples' => array(
	'drush migration_filepath' => 'Moves all files located in the public file directory to fid based directory structure.',
	),
	'bootstrap' => DRUSH_BOOTSTRAP_DRUPAL_FULL
	);
	return $items;
	}
	/**
	* Command callback to perform the file migration.
	*
	* The two optional options are the source and target directories. By default
	* assumes that these are under the default path, e.g. sites/default/files.
	*/
	function drush_migration_filepath() {
	// Source directory to be cleaned up. All images in this directory will be relocated.
	$source_directory = rtrim(drush_get_option('source', ''), '/');
	// Directory to place the new structure under. If does not exist will be created.
	$target_directory = rtrim(drush_get_option('target', ''), '/');
	// Regular expression to find files in the source directory.
	// For now assume public files only.
	// public://field/image/imagefield_hENvtS.png
	$extensions = array('jpeg', 'jpg', 'gif', 'png');
	// Construct a expression to find images located in the source directory.
	$file_pattern = "[^\/]*"; // Finds anything that does not contain "/", should be fine.
	// Append the trailing slash for the regular expression.
	// Note, in some instances drupal places under public:/// (three slashes)
	// even when no folder specified. Reason for this is not known yet.
	$source_pattern = $source_directory ? $source_directory . "\/" : '';
	$regex = "^public:\/\/" . $source_pattern . "(" . $file_pattern . ")\.(" . implode($extensions, '|') . ")$";
	// Query the database for files that match this pattern.
	$filetypes = array('image/jpeg', 'image/jpg', 'image/gif', 'image/png');
	$query = db_select('file_managed', 'f')
	->condition('filemime', $filetypes , 'IN')
	->condition('uri', $regex, 'REGEXP')
	// Select the files to be moved.
	->fields('f', array('fid', 'filename', 'uri', 'timestamp'))
	// Start from the oldest so that new additions in between the file move
	// and the site using the new fe_paths path won't be tarnished.
	->orderBy('timestamp', 'asc');
	// Open a file in the public files/ directory to write redirects into.
	$public_path = variable_get('file_public_path', conf_path() . '/files');
	$redirects_filename = $public_path . "/image-redirects.map";
	// Print helpful information on pending file moves and exit.
	if (drush_get_option('list-info')) {
	print $query;
	print_r($query->arguments());
	// Print total number of files that need moving
	$num_files = $query->countQuery()->execute()->fetchField();
	drush_print(dt('There are ' . $num_files . ' files in total to be moved.'));
	drush_print(dt('The redirects file will be created at: ' . $redirects_filename));
	return TRUE;
	}
	// Limit the number of files.
	if ($limit = drush_get_option('limit')) {
	$files = $query->range(0, $limit)->execute()->fetchAll();
	}
	else {
	$files = $query->execute()->fetchAll();
	}
	$total_files = count($files);
	drush_print(dt($total_files . ' files are set to be moved.'));
	$count = 1;
	// Keep a list of files that have not been moved.
	$files_not_moved = array();
	// Create or open the redirects file for writing.
	$redirects_file = fopen($redirects_filename, 'a') or die("Unable to open redirects file for writing.");
	foreach ($files as $file) {
	preg_match_all("/$regex/i", $file->uri, $matches); // Note, $file->filename can be the SAME for different uri-s!
	$filename = $matches[1][0] . "." . $matches[2][0];
	$old_file_wrapper = file_stream_wrapper_get_instance_by_uri($file->uri);
	// If the file has already been moved, or does not exist in the filesystem, move on.
	if (FALSE === ($status = $old_file_wrapper->url_stat($file->uri, STREAM_URL_STAT_QUIET))) {
	$files_not_moved[] = $file->uri;
	drush_log("File entry in the database does not exist on the filesystem for " . $file->uri, 'notice');
	continue;
	}
	// Each file should go to the directory based on its fid.
	$target_directory_for_file = $target_directory . '/' . generate_directory_name($file);
	// Construct a dummy URI for it so we can use the stream wrappers.
	$new_directory = file_build_uri($target_directory_for_file);
	$wrapper = file_stream_wrapper_get_instance_by_uri($new_directory);
	// Make sure that the new directory exists.
	$wrapper->mkdir($new_directory, 0777, TRUE);
	// Construct the new directory.
	$wrapper = file_stream_wrapper_get_instance_by_uri($file->uri);
	$new_uri = file_build_uri($target_directory_for_file . '/' . $filename);
	if (!$wrapper->rename($file->uri, $new_uri)) {
	$files_not_moved[] = $file->uri;
	drush_log("There was a problem moving " . $file->uri, 'notice');
	continue;
	}
	$progress = round(($count / $total_files) * 100);
	drush_print($progress . '%');
	$query = db_update('file_managed')
	->fields(array('uri' => $new_uri))
	->condition('fid', $file->fid)
	->execute();
	// Append to the image redirects list.
	$rel_old_url = parse_url(file_create_url($file->uri), PHP_URL_PATH);
	$rel_new_url = parse_url(file_create_url($new_uri), PHP_URL_PATH);
	fwrite($redirects_file, '~^' . $rel_old_url . ' ' . $rel_new_url . ';' . PHP_EOL);
	$count++;
	}
	fclose($redirects_file);
	$not_moved = count($files_not_moved) ? count($files_not_moved) : 0;
	drush_print(dt('(' . ($total_files - $not_moved) . '/' . $total_files . ') ' .
	'files moved.'));
	if ($not_moved) {
	drush_print(dt($not_moved . ' files do not exist in the filesystem.'));
	}
	drush_print(dt('Redirects file: ' . $redirects_filename));
	}
	/**
	* Generate the new file structure based on the timestamp.
	*/
	function generate_directory_name($file) {
	$fid = str_pad($file->fid, 10, '0', STR_PAD_LEFT);
	$len = strlen($fid);
	$top_dir = $fid[$len - 3];
	$sub_dir = $fid[$len - 2] . $fid[$len - 1];
	$value = $top_dir . '/' . $sub_dir;
	return $value;
	}
	?>

view raw d7_fid_migration_filepath.drush.inc hosted with ❤ by GitHub

For new files uploaded via the UI, it's easy to create a file token based on the above logic, to be used in File Entity Paths settings.

/** * Implements hook_token_info(). */ function mymodule_token_info() { $info['tokens']['file']['fid_split'] = array( 'name' => t('Split based on File id'), 'description' => t('Creates two directories based on fid'), ); return $info; }
/** * Implements hook_tokens(). */ function mymodule_tokens($type, $tokens, $data = array(), $options = array()) { $replacements = array(); if (($type == 'file') && !empty($data['file'])) { $fid = str_pad($data['file']->fid, 10, '0', STR_PAD_LEFT); $len = strlen($fid); // Top level dir to be 0 - 9 $top_dir = $fid[$len - 3]; // Sub level dir to be 00 - 99 (100 per top level dir) $sub_dir = $fid[$len - 2] . $fid[$len - 1]; $value = $top_dir . '/' . $sub_dir; $replacements[$tokens['fid_split']] = $value; } return $replacements; }

At the time of this writing, you need to use the latest dev version of fe_paths module (commit dated 16 Oct 2012).

Alter path alias programmatically for an existing taxonomy term

Here's a method to alter the path alias (if you're using the pathauto module) of an existing taxonomy term programmatically. Here we shall be doing this in a hook_update_N(). It is quite un-intuitive given the lack of proper documentation as to how to alter the alias of an existing term. Here is my trial & error method:

/**
 * Re-alias 'Trailer' term & create a redirect from its old path to the new alias.
 */
function xxxxx_update_7007() {
  // 'Trailer' term.
  $term = taxonomy_term_load(20);
  // Set 'Generate automatic URL alias' to FALSE to allow custom alias setting later.
  $term->path['pathauto'] = FALSE;
  taxonomy_term_save($term);

  // Create a new alias
  $new_path = path_load('taxonomy/term/20');
  $new_path['alias'] = 'trailers';
  path_save($new_path);

  // Setup redirect from the old url 'trailer' to 'trailers'
  $redirect = new stdClass();

  module_invoke(
    'redirect',
    'object_prepare',
    $redirect,
    array(
      'source' => 'trailer',
      'source_options' => array(),
      'redirect' => 'trailers',
      'redirect_options' => array(),
      'language' => LANGUAGE_NONE,
    )
  );

  module_invoke('redirect', 'save', $redirect);
}

In the above example, we are changing the path of a term from '/trailer' to '/trailers' and setting up a redirect from the old path so that users don't get lost (plus it's better for SEO).

We first have to unset the 'Generate automatic URL alias' option and save the term. Then create the new alias, followed by adding the redirect from the old alias.