Data Exchange Framework and Json

I had the privilege of taking part in putting together a leg of a POC that included the task of parsing a Json data set representing a group of users and doing two things with them. The first, adding them as Sitecore items in what could represent a profile page. The second, creating Sitecore user accounts based on that same json formatted data set.

Fellow Velirian, Nick Dorrough is presenting a series of posts on this POC. Be sure to check out his current post and the soon to be upcoming posts.

Along with the current DXF documentation being a resource, Nick's posts will dive into the details of the POC and working with DXF. So I certainly won’t repeat that information here. What I will focus on is the ability to add functionality of parsing Json and working with Sitecore not only in content creation, but to work in other Sitecore aspects, such as adding Sitecore User accounts.

In initial investigations, it was really evident that my work was not going to be done for me. No Json provider was noticed nor within the Sitecore provider a way of creating users. Curses!

Actually – I kind of thought that might be the case – so into the sandbox to play…..

First I looked into the File System Reader pipeline step provided in the documentation example (http://integrationsdn.sitecore.net/DataExchangeFramework/v1.1/implementing-a-provider/implement-pipeline-step/implement-pipeline-step-processor.html). The process works on the concept of file location and parsing by delimiter. I knew getting the json parser to work was likely more important, so opted to use a local json file. I could always change the way the json was delivered after the proof of concept and likely would need to, based on using the patterns for future solution implementations. I also needed to generate some data representing user accounts. I pulled user data from our GitHub company account and noticed the results needed a little data scrubbing, so at that point there was no turning back from using a local file.

So here is the result of the code for the new converter and pipeline step.

The converter I did was in fact no different than the one in the File System reader example. Except for the template Guid.

namespace UserImport.Json.Converters.PipelineSteps
{
	public class ReadJsonFileStepConverter : BasePipelineStepConverter<ItemModel>
	{
		private static readonly Guid TemplateId = Guid.Parse("{D5591FEF-8964-471B-B785-AC183C2F614B}");
		public ReadJsonFileStepConverter(IItemModelRepository repository) : base(repository)
		{
		  SupportedTemplateIds.Add(TemplateId);
		}
		protected override void AddPlugins(ItemModel source, PipelineStep pipelineStep)
		{
		  AddEndpointSettings(source, pipelineStep);
		}
		private void AddEndpointSettings(ItemModel source, PipelineStep pipelineStep)
		{
		  var settings = new EndpointSettings();
		  var endpointFrom = ConvertReferenceToModel<Endpoint>(source, ReadJsonFileStepItemModel.EndpointFrom);
		  if (endpointFrom != null)
		  {
		    settings.EndpointFrom = endpointFrom;
		  }
		  pipelineStep.Plugins.Add(settings);
		}
	}
}

For the processor, I noticed the file system example loading “record” elements in a pipeline plugin that contained on object of IterableDataSettings. I knew that likely downstream I would create more work for myself if I changed this approach so, doings as the locals do, I deserialized the json into a List of POCO elements.

//
//read data from file
IEnumerable<GitMemberJsonModel> lines = LoadJson(settings);
			
//
//add the data that was read from the file to a plugin
var dataSettings = new IterableDataSettings(lines);
private IEnumerable<GitMemberJsonModel> LoadJson(JsonFileSettings settings)
{
	List<GitMemberJsonModel> returnObjects = new List<GitMemberJsonModel>();
	string json;
	using (var r = new StreamReader(settings.Path))
	{
		json = r.ReadToEnd();
	}

	returnObjects = JsonConvert.DeserializeObject<List<GitMemberJsonModel>>(json);
			
	return returnObjects;
}

And then getting to what the pipeline looks like…

So as you can see it is similar to the example. The major difference between the two is obviously the Read Json User File pipeline step.

So when the pipeline batch is executed – the json file is read, json “parsed”, field values are mapped, and like magic we have our Sitecore items.

1st goal of POC completed. No sweat. Also no rest for the wicked, I have another goal to accomplish…

So now instead of writing to Sitecore items I need to create users through the membership provider. So simple…well it took a little bit to find the right fit. I had a few options, some kinda gross and one not gross.

I could have just extended the UpdateSitecoreItemStepProcessor to add Membership.CreateUser after the item update. This is gross because I’ve just couple item create and user creation, which goes against the pipeline concept. Not happening.

I could add a pipeline step within the User Info from File to User Item Pipeline, but then again my entire pipeline has both item creation and user account creation and when calling the pipeline I have to content with having both.

I honestly did do this for a bit for debugging purposes to make it easier to figure out where my data iteration lived and what it looked like for the plugin. But it wasn’t staying that way because sometimes I don’t want a Reeses Peanut Butter cup, sometime I just want the peanut butter.

So a new pipeline was created.

Adding to run two pipelines...

Also needed a new converter and processor.

namespace UserImport.Json.Converters.PipelineSteps
{
	public class UpdateSitecoreUserStepConverter : BasePipelineStepConverter<ItemModel>
	{
		private static readonly Guid TemplateId = Guid.Parse("{DC2EA70F-52E8-42FC-B348-336013E23295}");
		public UpdateSitecoreUserStepConverter(IItemModelRepository repository) : base(repository)
		{
			SupportedTemplateIds.Add(TemplateId);
		}

		protected override void AddPlugins(ItemModel source, PipelineStep pipelineStep)
		{
			AddUpdateSitecoreUserSettings(source, pipelineStep);
		}

		private void AddUpdateSitecoreUserSettings(ItemModel source, PipelineStep pipelineStep)
		{
			var settings = new UpdateSitecoreUserItemSettings();
			var emailfield = source[UpdateSitecoreUserItemModel.EmailField].ToString();
			var defaultpassword = source[UpdateSitecoreUserItemModel.DefaultPassword].ToString();
			var usernamefield = source[UpdateSitecoreUserItemModel.UsernameField].ToString();
			if (!string.IsNullOrEmpty(emailfield))
			{
				settings.EmailField = emailfield;
			}
			if (!string.IsNullOrEmpty(defaultpassword))
			{
				settings.DefaultPassword = defaultpassword;
			}
			if (!string.IsNullOrEmpty(usernamefield))
			{
				settings.UsernameField = usernamefield;
			}
			
			pipelineStep.Plugins.Add(settings);
		}
	}
}

The converter now stores the Sitecore pipeline step item’s field values as a plugin for retrieval in the processor. Knowing the json record were typical fields in the itemModel – I just grabbed them directly for storage.

Now the processor I based off of the decompilation of the UpdateSitecroeItemStepProcessor. I removed pieces around endpoint retrieval and modified the fixing of the Sitecore item to actually do this:

protected virtual void PrepareUserAccount(GitMemberJsonModel itemModel, PipelineContext pipelineContext)
{
  var logger = pipelineContext.PipelineBatchContext.Logger;
  var usernameValue = string.Empty;
  var emailValue = string.Empty;

  try
  {
    UpdateSitecoreUserItemSettings updateUserItemSettings = pipelineContext.CurrentPipelineStep.GetPlugin<UpdateSitecoreUserItemSettings>();
    if (updateUserItemSettings != null)
    {
	if ((string.IsNullOrEmpty(updateUserItemSettings.EmailField)) ||
	(string.IsNullOrEmpty(updateUserItemSettings.UsernameField)) ||
	(string.IsNullOrEmpty(updateUserItemSettings.DefaultPassword)))
	{
	  return;
	}

	emailValue = (string)itemModel.GetType().GetProperty(updateUserItemSettings.EmailField).GetValue(itemModel, null);
	if (string.IsNullOrEmpty(emailValue))
	{
	  return;
	}

	usernameValue = (string)itemModel.GetType().GetProperty(updateUserItemSettings.UsernameField).GetValue(itemModel, null);
	if (string.IsNullOrEmpty(usernameValue))
	{
	  return;
	}

	usernameValue = this.SitecoreScrub(usernameValue);

	var user = Membership.GetUser(usernameValue);
	if (user == null)
	{
	  Membership.CreateUser(usernameValue, updateUserItemSettings.DefaultPassword, emailValue);
	  logger.Info("CREATED: Username {0} in Sitecore.", usernameValue);
	}
	else
	{
	  logger.Info("NOT CREATED: Username {0} already exists in Sitecore.", usernameValue);
	}
    }
  }
  catch (Exception ex)
  {
    logger.Error(
	"Sitecore User Accont not created. " +
	"(username: {0}, error message: {1})",
	(!string.IsNullOrEmpty(usernameValue)) ? usernameValue : "Unknown", ex.Message);
  }

}

Now if this wasn’t a POC, I would do more validation, likely add profile field updates, and so on. But it’s a POC and the goal is to create a user with

Membership.CreateUser(usernameValue, updateUserItemSettings.DefaultPassword, emailValue);

Note: Because Stephen Pope pays well for referencing him in blog posts, this has been added to make my wallet a bit more full for the holidays. (http://stackoverflow.com/questions/11840886/how-to-create-user-profiles-in-sitecore-programatically)

And with that goal #2 complete.

Hopefully this post, along with Nick’s current and upcoming posts, will provide you a bit of insight on how to stretch a bit with utilization of the Data Exchange Framework.