Friday, January 23, 2009

Value Assistance Management without DAB Pattern

Providing non-technical end user ability to modify value assistance list of a custom attribute without giving them access to Application Builder.

Due to the complex workflows and nature of the broadcast business, Documentum needs to be customized for television broadcast engineering in order to use it as a media management system. Part of the customization includes representing different episodes as objects in the repository, and storing cuesheet information of each episode as an attribute. One of the required attributes is closed captioning information, which can only be one of a pre-defined list of values. These values may require modification from time to time by the broadcast engineering staff. This issue can be resolved using a vanilla Documentum install by adding the predefined list of values as a value assistance list to a custom type’s attribute, and allowing the end users access to Documentum Application Builder (DAB) to update the value assistance list. However, DAB encompasses a lot of functionality that is not desirable in a typical end user’s hands. The only other vanilla approach would be to provide the end users with Documentum Query Language (DQL) queries they can enter via Documentum Administrator (DA). This approach also is risky as it requires system administrators to expose DA to non-technical staff and requires training the end users in DQL. Therefore, the only approach is through customization. So how should this customization be implemented?

There are several options that can be taken, but most can be summarized into the 2 below:
1) Providing Lookups for Data Fields (this is a pattern already in development at the EMC developer community)
2) Provide a service that encapsulates either the Documentum Foundation Services (DFS) or Documentum Foundation Classes (DFC). The service must also provide a way for clients to execute Documentum Query Language (DQL) required to update the value assistance data for a custom attribute. Then provide a client to consume the service, which can be anything from a thick client to a mobile client.

Obviously, since Providing Lookups for Data Fields is already a pattern in progress, I will focus on the second option.

Manage Value Assistance Pattern

The primary business users for this pattern would be operations staff in the television broadcasting industry, though other industry verticals may find use for this pattern as well.

As a precursor, it is assumed that the current IT environment contains a Documentum repository and a box running either DFS or DFC. Since there is already DFS/DFC in the environment, why does this pattern include a service to encapsulate this functionality? This is to account for the relatively high costs of a Documentum implementation; there are licensing costs associated with every server hosting DFS/DFC.

Using a design based on services oriented architecture, this pattern will expose the functionality to update the value assistance data for any custom type in a Documentum repository to multiple non-technical users. This will avoid licensing and training costs associated with deploying DAB to multiple desktops and training non-IT staff on how to use DAB.

Functional How-To:
The Manage Value Assistance pattern is based on the layered services pattern. First, create a method that accepts the name of a custom type and a custom attribute as parameters, and returns a collection of objects with members for the value, display string and description of each value assistance item. Then, create a method that accepts the name of a custom type, the name of a custom attribute of the custom type, and a collection of value assistance items as described above. This method basically overwrites the existing value assistance data for the custom attribute of the custom type and replaces them with the values contained in the collection passed through as a parameter. Essentially the collection of value assistance data is used to build a string object that will be the DQL query required to run against the repository. Once the DQL query is constructed, the method will run the query against the repository with the PUBLISH keyword to ensure other users have access to the new value assistance data.

Since this second method will be exposed to multiple users, it must be made re-entrant or thread-safe to prevent multiple users from editing the same custom type and attribute at the same time. As a security precaution, there should be authorization checking implemented here so only users who are supposed to be modifying the value assistance are allowed to. Therefore, this service must never allow anonymous access.

Next, expose these methods via service consumable by clients hosted on other machines. The favourite choice would be a XML web service but WCF is also a good option. This will enable many users access to this functionality, so it can be shared across the enterprise.

When to use it:
Obviously there are multiple ways to allow non-technical users to gain access to the value assistance data. The database table these values reside in can be updated directly with a stored procedure, saving the trouble of learning the Documentum APIs. Although this is the more efficient way, I do not recommend it as it bypasses a lot of the built in plumbing that Documentum provides. It’s always safer to use the Documentum provided facilities to manipulate it as they naturally invoke required background processes to ensure everything goes smoothly.

I implemented an example on

The requirements to make the example work are:
Content Server 5.3 SP6.
DFC 5.3 SP6.
Microsoft.NET 3.5 (WCF and WPF).

Friday, January 16, 2009

Improving Performance of LINQ Queries

Part of the challenge facing television broadcasters is reconciling asrun logs against the playlists they are supposed to represent. Every once in a while content is changed on the fly so what is actually broadcasted is different from what the original playlist stated. Part of this reconciliation involves the generation of recon keys, which involve comparing event information of an asrun log against the playlist it was generated for. In order to compare the original playlists and the resulting asrun logs, I turned to LINQ to get the information.

One of the LINQ queries I used for generating the recon keys was:

var rOriginalEvent = from orgEvents in db.BIP_Playlist_Events
where orgEvents.eventid.Equals(searchStr)
orderby orgEvents.Last_Updated descending
select orgEvents;

if (rOriginalEvent.Count() > 0)
hasMatchingEvents = true;
dlstartDateTime = rOriginalEvent.First().dlstart;
orgDlhnumber = rOriginalEvent.First().dlhnumber;
orgEventid = rOriginalEvent.First().eventid;

The purpose of this query was to search archives of the original playlists from which the D-class asrun log was generated to compare the original event id, start time and dlhnumber against the asrun log’s information. These playlists were archived in a SQL Server 2005 database table, so this LINQ query was using it as the source. I timed the query and it ran a total of 0s according to my calculations. However, when I timed the following rOriginalEvent.First() calls, they took a total of .45 seconds. This does not seem to be a significant amount of time, but when repeated 1500 times for a single transaction, it meant the transaction would last almost 20 minutes.

Since LINQ automatically generates the most efficient query in the background, I looked for ways to extract the record I wanted directly from the var query. I then turned to the following LINQ query:

var rOriginalEvent = (from orgEvents in db.BIP_Playlist_Events
where orgEvents.eventid.Equals(searchStr)
orderby orgEvents.Last_Updated descending
select orgEvents).FirstOrDefault();

This would move all the processing into the LINQ SQL connector and extract either the record I wanted or a null object. The important thing is that I no longer needed the First() call when getting the information:

dlstartDateTime = rOriginalEvent.dlstart;
orgDlhnumber = rOriginalEvent.dlhnumber;
orgEventid = rOriginalEvent.eventid;

By moving the costly First() call into the LINQ query itself I reduced the time of the total operation from .45 seconds to .1 seconds, thus a performance improvement of 4.5 times faster than before.

In conclusion, I recommend that if you are after a single record for a specific set of information, use the above LINQ expression. The FirstOrDefault() method returns the first element in the collection or the default value if the collection is empty, so you don’t have to worry about dealing with a null object.

Saturday, January 10, 2009

What is Software Engineering?

Software engineering is defined as the application of engineering principles and practices to software development with the end-goal of making software development a predictable and controlled discipline.

CMMI attempts to bring this discipline to software development projects by offering suggestions for roles, responsibilities and processes for software projects.

However, the goal of this blog is not to discuss process; I'll leave that for my software engineering management blog. This blog will focus on commonly experienced software development problems, and their respective solutions. The reason for this is to reduce the amount of unknowns in software projects to make them more predictable and controlled. Thus this blog will contribute to the software engineering discipline.