Photo from Chile

How I Use Transfer - Part IX - My Abstract Transfer Decorator Object - The Populate Method

In the previous post in this series about Transfer (an ORM for ColdFusion) I introduced my AbstractTransferDecorator and discussed some of its simpler methods. In this post I want to go through the populate() method in detail. I have posted bits and pieces of this method in the past, but I don't think I've ever documented the whole thing, as it stands today. I'll break it into pieces to make it a bit more manageable.

view plain print about
1<cffunction name="populate" access="public" output="false" returntype="void" hint="Populates the object with values from the argumemnts">
2    <cfargument name="args" type="any" required="yes" />
3    <cfargument name="FieldList" type="any" required="no" default="" />
4    
5    <cfset var theFieldList = "" />
6    <cfset var TransferMetadata = getTransfer().getTransferMetaData(getClassName()) />
7    <cfset var Properties = TransferMetadata.getPropertyIterator() />
8    <cfset var theProperty = 0 />
9    <cfset var varName = 0 />
10    <cfset var varType = 0 />
11    <cfset var varValue = 0 />
12    <cfset var CompType = 0 />
13    <cfset var hasIterator = false />
14    <cfset var theIterator = 0 />
15    <cfset var theComposition = 0 />
16    <cfset var ChildClass = 0 />
17    <cfset var ChildPKName = 0 />
18    <cfset var theChild = 0 />

This method accepts just two arguments:

  • args, which is a structure containing data with which to populate the object. In my apps, args is generally passed the attributes scope, which is where Fusebox puts all user input.
  • FieldList, which is a list of fieldnames which can be used to limit the properties that are populated by this method.

To start, a bunch of local variables are declared. This includes requesting the TransferMetaData for the object's class from Transfer, as well as getting the PropertyIterator from that metadata.

Next, I loop through all of the object's properties:

view plain print about
1<cfloop condition="#Properties.hasnext()#">
2    <cfset theProperty = Properties.next() />
3    <cfset varName = theProperty.getName() />
4    <cfset varType = theProperty.getType() />
5    <cfif NOT ListLen(arguments.FieldList) OR ListFindNoCase(arguments.FieldList,varName)>
6        <cfif varName EQ "LastUpdateTimestamp" AND varType EQ "Date">
7            <cfset setLastUpdateTimestamp(Now()) />
8        <cfelseif Right(varName,4) EQ "Flag" AND varType EQ "Numeric">
9            <cfif StructKeyExists(arguments.args,varName)>
10                <cfset varValue = Val(arguments.args[varName]) />
11            <cfelse>
12                <cfset varValue = 0 />
13            </cfif>
14            <cfinvoke component="#this#" method="set#varName#">
15                <cfinvokeargument name="#varName#" value="#varValue#" />
16            </cfinvoke>
17        <cfelseif StructKeyExists(arguments.args,varName)>
18            <cfset varValue = arguments.args[varName] />
19            <cfif variables.myInstance.CleanseInput>
20                <cfset varValue = HTMLEditFormat(varValue) />
21            </cfif>
22            <cfif IsValid(varType,varValue)>
23                <cfinvoke component="#this#" method="set#varName#">
24                    <cfinvokeargument name="#varName#" value="#varValue#" />
25                </cfinvoke>
26            <cfelseif theProperty.getIsNullable() AND NOT Len(varValue)>
27                <cfinvoke component="#this#" method="set#varName#Null" />
28            <cfelse>
29                <cfinvoke component="#this#" method="setInvalid_#varName#">
30                    <cfinvokeargument name="1" value="#varValue#" />
31                </cfinvoke>
32                <cfset ArrayAppend(arguments.args.Errors,"The contents of the " & varName & " field must be a valid " & varType & " value.") />
33            </cfif>
34        </cfif>
35    </cfif>
36</cfloop>

I start by extracting the name and datatype of the property into local variables. I then use the FieldList to filter which properties get populated. This allows me to use this routine to populate only part of an object. An empty FieldList means that all properties get populated.

Most of my Business Objects have a LastUpdateTimestamp property, which should be populated with the current timestamp, so I do that next.

To deal with the age old checkbox problem, most of my boolean fields in my database are named xxxFlag. I also define these to Transfer as numeric, rather than boolean. So, if I'm at one of those properties in my loop, I check to see whether I've been passed a corresponding argument, and if so I use that value, converting blanks to zeros via Val(). If I haven't been passed anything I assume that a checkbox has gone unchecked, so I default the value to 0. Then I load that value into the object via its setter using cfinvoke.

Now that I've dealt with the special cases, I check to see whether a corresponding argument for the property has been passed in, and if so I save the value to a local variable. If my object has been configured to CleanseInput, which was discussed in the previous post, I use HTMLEditFormat() as part of a scheme to protect the site against Cross Site Scripting (XSS). I found that it is very rare that an object actually needs to allow true HTML to be loaded into it, so I find that this is a convenient way to approach this. If a particular object should allow HTML then I just override the value of CleanseInput in that object's configure() method.

I then validate the value passed in against the property's datatype. If it's valid, I set the property, if not I check to see if the property is nullable and whether the invalid value is in fact an empty string, in which case I set the property to null. If neither of those cases are true, then the value is in fact invalid, so I want to store it somewhere so I can display it back to the user. I use a "mock method" to store the invalid data in a private variable. This is made possible through the use of onMissingMethod(), which I'll describe at the end of this post. I also add an error message to the array of errors to be displayed to the user.

Now that all of the object's properties have been addressed, I turn my attention to the object's compositions. The hurdle I had to overcome here is that the values being submitted by the user for these compositions are usually just the primary key (or id, in Transfer terminology) of the associated object, but in order to load these into the object we need to first retrieve the actual object that corresponds to that key. What follows is my attempt to overcome that hurdle for ManyToOne and ParentOneToMany compositions:

view plain print about
1<cfloop list="ManyToOne,ParentOneToMany" index="CompType">
2    <cfinvoke component="#TransferMetadata#" method="has#CompType#" returnvariable="hasIterator" />
3    <cfif hasIterator>
4        <cfinvoke component="#TransferMetadata#" method="get#CompType#Iterator" returnvariable="theIterator" />
5        <cfloop condition="#theIterator.hasnext()#">
6            <cfset theComposition = theIterator.next() />
7            <cfset varName = theComposition.getName() />
8            <cfset ChildClass = theComposition.getLink().getTo() />
9            <cfset ChildPKName = theComposition.getLink().getToObject().getPrimaryKey().getName() />
10            <cfif StructKeyExists(arguments.args,ChildPKName)>
11                <cfset varValue = arguments.args[ChildPKName] />
12                <cfset theChild = getTransfer().get(ChildClass,varValue) />
13                <cfif theChild.getIsPersisted()>
14                    <cfif CompType CONTAINS "Parent">
15                        <cfset varName = "Parent" & theComposition.getLink().getToObject().getObjectName() />
16                    </cfif>
17                    <cfinvoke component="#this#" method="set#varName#">
18                        <cfinvokeargument name="transfer" value="#theChild#" />
19                    </cfinvoke>
20                </cfif>
21            </cfif>
22        </cfloop>
23    </cfif>
24</cfloop>
25    
26</cffunction>

The code for each composition type (ManyToOne and ParentOneToMany) is pretty much identical, so I loop over my list of composition types, effectively executing the code block once for ManyToOne and once for ParentOneToMany.

I first check to see if an Iterator for the composition type exists, and if so I get a copy of it from the TransferMetaData. I then use the Iterator to loop through the compositions defined for the object, extracting the name of the composition, the Transfer class of the child, and the name of the primary key of that child.

If an argument has been passed in that corresponds with the primary key that I just found, e.g., The primary key is ProductId and an argument of ProductId was passed in, I do the following:

  • extract the value of the argument
  • get the child object from Transfer
  • check to see whether I've been handed back an object from the database, using getIsPersisted()
  • if I do find a valid Transfer Object, I want to load that object into the current object

The wrinkle is that if I'm working with a ParentOneToMany composition the name of the set() method needs to be determined differently than if I'm working with a ManyToOne, so I address that with an if statement.

When that loop is done my object is fully loaded, so I end the method.

I mentioned my use of onMissingMethod() above, so here's the code for that:

view plain print about
1<cffunction name="onMissingMethod" access="public" output="false" returntype="Any" hint="Very useful!">
2    <cfargument name="missingMethodName" type="any" required="true" />
3    <cfargument name="missingMethodArguments" type="any" required="true" />
4    
5    <cfset var varName = 0 />
6    <cfset var ReturnValue = "" />
7
8    <cfif Left(arguments.missingMethodName,Len("setInvalid_")) EQ "setInvalid_" AND StructKeyExists(arguments.missingMethodArguments,"1")>
9        <cfset varName = ReplaceNoCase(arguments.missingMethodName,"setInvalid_","") & "_Invalid" />
10        <cfset variables.myInstance[varName] = arguments.missingMethodArguments.1 />
11    <cfelseif Left(arguments.missingMethodName,Len("getInvalid_")) EQ "getInvalid_">
12        <cfset varName = ReplaceNoCase(arguments.missingMethodName,"getInvalid_","") & "_Invalid" />
13        <cfif StructKeyExists(variables.myInstance,varName)>
14            <cfset ReturnValue = variables.myInstance[varName] />
15        </cfif>
16    </cfif>
17    <cfreturn ReturnValue />
18</cffunction>

Basically what I'm doing here is allowing for an unlimited number of Invalid private properties, which can be accessed via standard getters and setters. I actually go into a lot more detail about this approach and why I'm doing this in this blog post, so I won't repeat it all here. I know it's not perfect, but I have yet to come up with a better method that would allow me to use my Business Object to keep track of the invalid data that a user has entered. Feedback in this area would be greatly appreciated.

I do recall someone indicating that there was a problem with the logic of my populate() method, that would make it unreliable, but it seems to work consistently for me. If anyone gives it a try and encounters any strange results please let me know.

The only area that I see changing dramatically at this point is the validations. My original thought was that I was already documenting the datatypes of the Business Objects in my transfer.xml file, so I should just use that data to perform these validations. I still see the logic in that statement, but I'd rather move all of my validation logic into external validation objects, so I'll probably do away with that part of the code. I'll still have to address the issue of storing invalid values, and dealing with nullable properties, but I'm sure that there are other solutions to those problems.

Speaking of validations, that is the last part of the AbstractTransferDecorator left to describe, so I'll address that in my next post.

TweetBacks
Comments
Superb!

The last two days I spent trying to come up with a generic populate method that
could account for invalid datatypes. Man, that wasn't easy...

In utter despair I googled 'transfer orm populate datatype' and there your
blog showed up with all the code for a generic populate().

On top of that, the getInvalid_X methods in the decorator, returning invalid
datatypes to the view, solve the problem of an inconsistent
UI treatment of non valid input (wrong datatype input cannot be set into the TO,
whereas other invalidated input can). Maybe a generic get() in the
AbstractTransferDecorator checking for invalid values wouldn't be such a bad idea.
Though a bit of overkill, as you suggest, it saves you the work from writing these
methods. But that's a minor detail I suppose (one could always use Illudium PU-36
to generate them all and throw away the ones you don't need).

Thanks a lot, saves me days of coding (well, I like coding don't I? Yes, I do, honestly)
# Posted By Richard | 7/22/08 5:54 PM
Thanks Richard, I'm glad you find my approach to your liking. I can't tell you how many hours and iterations I've gone through to get to this point, and I'm sure I'm not done yet.

Yes, if one is going to go this route (when dealing with invalid data) it probably does make sense to go with a generic get(), but I just don't like the smell of that. For now, because I actually have very few properties that will need this special treatment, I think I'll stick with the custom getters in the decorator.

I'd love to hear how other people are tackling this specific issue, as it's probably one of the most difficult that I've had to deal with in my move to OO.
# Posted By Bob Silverberg | 7/22/08 6:23 PM
Indeed this is a mindbreaker, as long as you want to populate formfields directly
from the bean.

This issue has been surprisingly few addressed on the web as far as I know.

Another approach to re-displaying invalid data is to leave
the display of invalid data to the view after all. Then your view takes it's values from
the bean unless there's an error result for the corresponding
field. Instead of value="#UserRecord.getAge()#" you
call some function, for instance value="#getValue(UserRecord,'Age')#"
that would first check to see if there were a validation result sent back to the
view for that particular field or otherwise return UserRecord.getAge().

I used to set both validation error type together with the
invalid input data in the validation result so to be able to
force back invalid data into the formfield.

BTW, my beans do not address the user directly when stumbling
upon some invalid data, since I believe directions as to what
caused the invalidation or how to solve it belong to the view.
So the validation result contains something like 'INVALID_DATA_TYPE',
leaving it up to the view to translate that to something human-readable.
Only a minor adaptation to your code.

So instead of ArrayAppend(arguments.args.Errors, '...') I have
getValidator().addError(varName,'INVALID_DATA_TYPE',varValue)

(getValidator returns a Validator object for the bean that can have additional
methods for non datatype related validations.)

Anyway, whether or not I'll be using the mock methods a lot, I sure will
use that populate method you wrote. Thanks again!
# Posted By Richard | 7/23/08 4:20 AM
Hmm, I like the approach that you're taking with putting the invalid values in the validation result, and then using the getValue() function in your view. That would certainly remove the requirement of creating all of the extra getters in the decorators. Where does the getValue() function live? How does the view have access to it?

I know that the biggest problem with my current approach is the potential unfriendliness of the error messages upon datatype validation failure. I am hoping that when Mark adds annotations to transfer.xml I'll be able to address this. I see your point about leaving the actual error messages to the view, but I was trying to remove the need to define each of those error messages individually, as I was hoping that I could generate them from all of the metadata available to me.

As I mentioned in the article, I'll probably be changing the way I'm doing these validations in the future, so we'll have to see how this evolves.

Thanks again for the feedback.
# Posted By Bob Silverberg | 7/24/08 5:14 AM
Maybe I should group all view functions neatly in a component but for now I've just cfincluded it in the view.

Here's what I do

<code>
<cffunction name="getValue" access="public" output="false" returntype="string">
   
   <cfargument name="TO" type="any" required="true" />
   <cfargument name="field" type="string" required="true" />
   
   <cfif isStruct(validation) AND structKeyExists(validation,field)>
      <!--- show nasty data to be corrected --->
      <cfset ret = validation[field].value />
   <cfelse>
      <!--- show the value rightly set into the TO --->
      <cfinvoke component="#TO#" returnVariable="ret" method="get#field#" />
   </cfif>
   
   <cfreturn ret />
   
</cffunction>   
</code>

Still have to be careful with formatting functions like DateFormat, because if someone entered
birthday: 'can't remember', forcing that string back into the input would cause the DateFormat function
to break.

<cfset val = getValue(InvoiceRecord,'RETO_DAT')/>
<cfif isDate(val)><cfset val = dateFormat(val,'yyyy-mm-dd') /></cfif>
<input type="text" name="RETO_DAT" value="#val#" />

So that's a bit of a pain, but these are all view issues.
# Posted By Richard | 7/24/08 9:01 AM
That's cool. I have a component called ViewUtils which I instantiate as a singleton and then just put in the variables scope for the view at the beginning of the request. It seems to do the trick for things like this. I include a myDateFormat function which does what you describe, but it's all encapsulated within the function so I don't need that code over and over in my view.
# Posted By Bob Silverberg | 7/24/08 9:31 AM
Bob this is great stuff. You have done what I am looking to do with Transfer.
Your posts have been invaluable in helping me get my head around Transfer. Thank you for your contributions.
# Posted By Gerald Guido | 11/17/08 2:06 PM
Bob - what is the performance penalty of having an abstract decorator that is extended? I thought I had read once on the MG or Transfer list that this additional level of extension would exacerbate issues when generating lots of Transfer objects. Have you seen any issues or done any tests?
# Posted By Brian | 11/24/08 2:13 PM
I have not seen any issues, nor have I done any tests. I have never heard before that having a class that extends another class introduces any performance penalties. I would hope that that is not the case, as being able to extend classes is pretty fundamental to doing object oriented programming. I understand that object creation is expensive in CF, but if one is to avoid extending classes to create objects, that would introduce huge limitations to one's ability to write decent object oriented code.

If you are able to find the original source that led you to believe this I'd be very interested to hear more about it.
# Posted By Bob Silverberg | 11/24/08 2:26 PM
Well I had not heard that before although I could appreciate why it might happen (effectively concatenating multiple expensive-to-create classes together might increase the problem). I went back to the Transfer group and did some searching and can't find the post I had in my mind - maybe I dreamt the whole thing up! I hope so... I'd like to abstract out the populate() function I have in my ~hundred decorators or so. And I dig your approach here; very robust!
# Posted By Brian | 11/24/08 9:09 PM
I wouldn't consider using Transfer without an Abstract Decorator, and I'm pretty sure that it's a common approach.

Give it a shot and do some simple load testing to see if it does impact performance.

And blog your results, of course ;-)
# Posted By Bob Silverberg | 11/24/08 9:25 PM
Bob, I found that your populate() method doesn't take into account setting the primary key of a simple table. E.g., if I have &lt;id name="foo" /&gt; and I pass in a value of "foo" to your populate, it won't set that field.

I've added this code to the bottom of your populate; I'd be interested in your thoughts on if this is good/bad or if there is another way of going about it?

<pre><code>
<cfset varName = TransferKey.getName() />
<cfif NOT TransferKey.getIsComposite() AND structKeyExists(arguments, varName)>
   <cfset varValue = arguments[varName] />
   <cfset varType = TransferKey.getType() />
   <cfif IsValid(varType, varValue)>
    <cfinvoke component="#this#" method="set#varName#">
    <cfinvokeargument name="#varName#" value="#varValue#" />
    </cfinvoke>
   <cfelseif theProperty.getIsNullable() AND NOT Len(varValue)>
    <cfinvoke component="#this#" method="set#varName#Null" />
   </cfif>
</cfif>
</code></pre>
# Posted By Brian | 1/26/09 1:17 AM
@Bob - one other question; in your ManyToOne/OneToMany section, you check to see if the passed in argument is persisted (theChild = getTransfer().get(ChildClass, varValue). This prevents you from doing like populate(unpersistedManyToOneObject). Is there a reason you elected to prevent this? It seems like it would be useful to allow since you could use cascadeSave() to address any linked, but unsaved, relationships.

Thanks!
# Posted By Brian | 1/26/09 1:10 PM
Brian, you are correct, my method does not take setting the primary key manually into account, as I haven't needed to do it. Your code looks like a reasonable approach to the problem. I'd probably add it at the top of the method, before the properties loop, as that's where it would make the most sense to me.

Generally (perhaps even always) primary keys are not allowed to be null, so I don't think you need the extra code that checks to see if nulls are allowed. I guess it doesn't hurt to add it, but I think it may be unnecessary.
# Posted By Bob Silverberg | 1/26/09 1:56 PM
Hey Guys,

Just as a footnote to people who might read this article in the future. Regarding your idea of the getValue() UDF which you have, MG:G has a new feature known as 'helpers' which would work perfectly for this, no need to include it into the view, simply place the UDF in the /helpers directory and you'd then have access to it in the view like #helpers.utils.getValue()#.

The helpers scope is available in both the view and the controller.

Rob
# Posted By Robert Rawlins | 4/21/09 7:55 AM
Looking back at this post and the comments (because of a new comment), I realize that I have developed a far better way of dealing with this. I just haven't taken the time to post about it yet. It is similar, but does away with the need for creating any custom getters. When I find the time I'll write up a post about it.
# Posted By Bob Silverberg | 4/21/09 4:57 PM
@Bob:

Was wondering if you posted your new solution to the Populate() method anywhere? Also would like to thank you for all your contributions to the community, it has helped me TONS!

Keep up the good work.
# Posted By Alex Sante | 7/23/09 7:23 PM
@Alex: I don't think that I've specifically written about my new approach, but you can find an example of it in action in the demo that I created for ValidateThis!, my validation framework for ColdFusion objects. You can download the code from http://validatethis.riaforge.org/, and when you unzip it look at /BODemo/model/service/AbstractService.cfc and /BODemo/model/AbstractTransferDecorator.cfc. That should point you in the right direction.

I'll try to get a post up about it sometime in the not-too-distant future as well.
# Posted By Bob Silverberg | 7/24/09 9:42 AM