ColdFusion Intermittently Cannot Find CFC
- March 3, 2009 1:21 PM
- ColdFusion, Mystery Error Message
- Comments (14)
Ok, so a new one on me... ColdFusion has started to intermittently fail to find a CFC when trying to instantiate it. I started getting emails that the system was erroring out because it failed to find a component for part of a process in our system, but when I went back to try it myself, things worked fine. I dismissed it as odd until it started to happen more often. Click "more" to see how I replicated the problem and help me figure out what is going on.
The Setup
Ok, so basically the system gets a "grading engine" to do some calculations based on given inputs. Not that complex. Each time one of these inputs is submitted the system grabs the engine and causes the system to recalculate information based on what was entered. Pretty standard stuff there. Now all of a sudden the system sometimes can't find what it needs. I set up the following test and run it a half dozen times without issue:
<cfset errors = ArrayNew(1) />
<cfloop from="1" to="2000" index="x">
<cftry>
<cfset objGradeEngine = CreateObject("component", "myapp.cfc.util.gradingEngine.district7Engine") />
<cfcatch>
<cfset count = count + 1 />
<cfset ArrayAppend(errors, cfcatch)>
</cfcatch>
</cftry>
</cfloop>
Count: <cfdump var="#count#"><br />
Iterations:<cfdump var="#x#">
<cfif ArrayLen(errors) gt 0>
<cfdump var="#errors[1]#" >
</cfif>
The Problem
And then it hits:
Suddenly I do actually have a problem. Despite creating like 6 to 8 thousand copies fine during the first times I ran the test, suddenly 231 of 2001 tries failed to find the component. That means that ColdFusion could find the component the other 1,770 times.
And before you ask, I did try the following:
<cfset errors = ArrayNew(1) />
<cfloop from="1" to="2000" index="x">
<cftry>
<cflock name="getEngine" timeout="2">
<cfset objGradeEngine = CreateObject("component", "myapp.cfc.util.gradingEngine.district7Engine") />
</cflock>
<cfcatch>
<cfset count = count + 1 />
<cfset ArrayAppend(errors, cfcatch)>
</cfcatch>
</cftry>
</cfloop>
Count: <cfdump var="#count#"><br />
Iterations:<cfdump var="#x#">
<cfif ArrayLen(errors) gt 0>
<cfdump var="#errors[1]#" >
</cfif>
And the <cflock> didn't make a difference: lots of tries with nothing, then one with a couple hundred fails. My next thought is that I'm exceeding resources somehow, so I jumped into CF Admin and currently things are set like this:
Problem is that I really don't have an idea how these would interact with the system and the rate of errors.
Any ideas?
Updates
Update: Not sure if it matters, but I'll point out that my CFC "District7Engine" extends "DefaultEngine", so there is an additional layer of dependence there. I copied the files to my local test machine and have thus far been unable to cause this error on my local host despite spamming and even holding the refresh to try to gum up the number of requests.I wonder... is it possible something on the server is actually locking CF out of reading the file on some kind of interval? Would CF return that it couldn't find the file if the file was being used by another application?
Update 2 Updated my test to look like this:
<cfset errors = ArrayNew(1) />
<cfset times = ArrayNew(1) />
<cfloop from="1" to="2000" index="x">
<cftry>
<cflock name="getEngine" timeout="2">
<cfset objGradeEngine = CreateObject("component", "livegrades.cfc.util.gradingEngine.district7Engine") />
</cflock>
<cfcatch>
<cfset count = count + 1 />
<cfset ArrayAppend(errors, cfcatch)>
<cfset ArrayAppend(times, now())>
</cfcatch>
</cftry>
</cfloop>
Count: <cfdump var="#count#"><br />
Iterations:<cfdump var="#x#">
<cfif ArrayLen(errors) gt 0>
<cfdump var="#errors[1]#" >
</cfif>
<cfif ArrayLen(times) gt 0>
<cfdump var="#times[1]#" ><br />
<cfdump var="#times[ArrayLen(times)]#" >
</cfif>
And got the following:
Basically shows that the errors all happened withing a 5 second spread. Thats a long time, practically the life span of the request. I'll try to also find out which iterations failed, so I can see if they are contiguous or random within the test.
Update 3 At Joshua Cyr's suggestion, I've updated my test suite to look like this:
<cfset count = 0 />
<cfset filenotfound = 0 />
<cfset errors = ArrayNew(1) />
<cfset times = ArrayNew(1) />
<cfloop from="1" to="2000" index="x">
<cftry>
<cflock name="getEngine" timeout="2">
<cfset objGradeEngine = CreateObject("component", "myapp.cfc.util.gradingEngine.district7Engine") />
</cflock>
<cfcatch>
<cfset count = count + 1 />
<cfset ArrayAppend(errors, cfcatch)>
<cfset ArrayAppend(times, now())>
</cfcatch>
</cftry>
<cfif NOT FileExists("d:\websites\ myapp\cfc\util\gradingEngine\district7Engine.cfc")>
<cfset filenotfound = filenotfound + 1 />
</cfif>
</cfloop>
File not Found: <cfdump var="#filenotfound#"><br />
CreateObject error Count: <cfdump var="#count#"><br />
Iterations:<cfdump var="#x#"><br />
<cfif ArrayLen(errors) gt 0>
<cfdump var="#errors[1]#" >
</cfif>
<cfif ArrayLen(times) gt 0>
First: <cfdump var="#times[1]#" ><br />
Last: <cfdump var="#times[ArrayLen(times)]#" ><br />
</cfif>
End: <cfdump var="#now()#">
I hope that this will help me catch if its that something is keeping CF from accessing the file or something else. My problem is that thus far I've gotten only one error, and that was a lock error after flooding the system with requests. I'm thinking that this is tied to usage and or other interactions that aren't occurring right now. I'll have to try again in the morning.
Comments
@Raul; I'm not sure thats the case here, since its not shared hosting or anything; I know what else is on the box. I'm also aware of name conflicts/interactions as I've used them to my advantage in an application or two. I will go back and check though, just in case thats a problem. I'm not sure how shared application variables would affect my example though.
Also about to update with some additional information about the setup and testing.
You had mentioned above that moving the file to your local server made the problem go away. You've not said where this template is, but is it pointed to by a CF mapping, using a UNC or mapped drive perhaps? If it's not, perhaps the info may help someone else who's using such a remote drive and hitting this problem. Some may pick up other useful info regardless.
If you were pointing to a remote drive for that CFC (or indeed if your whole site docroot is remote), I would argue that the problem could be due to a temporary inability of CF to find the file at the time of attempting to load the file. Now you may wonder, doesn't it load it once and leave it at that?
Well, no. Not if CF's "trusted cache" setting is not turned on (and it can bite you even if it is, as I'll explain). You mentioned in the other entry that you do not have it enabled, and you admitted that you're not too clear about it. This isn't the place to belabor it, but if it's NOT enabled, then CF checks to see if a file that's requested (a CFM or CFC) has been updated. (This is how CF magically runs a newly updated page. But if you're in a production or test environment, where the code is not changing, it's generally best to disable it.)
I'd bet the problem happened when CF couldn't temporarily access the remote drive, when it tried to create the new instance, which your createobject is doing. (One may ask separately if you wouldn't benefit from loading the CFC once and reusing it instead, but that too is a subject too big to discuss here in a comment.)
I would bet that if you turned trusted cache on, the problem would go away, or at least diminish. It's still possible that CF may reload the template (or in your case, CFC), even with trusted cache on, if the template was flushed from the template cache, which can happen when the template cache is too small and many more pages are being put into it than there's room, which causes CF to force out the least recently used one, so that the next request for it will require a reload, which will check for the file. If you don't use the "save classes" option, then CF will need to recompile the file and therefore need to access the source again.
But at least the odds of it hitting a network burp are dramatically reduced, compared to when trusted cache is NOT enabled.
Of course, if you do enable it and leave it on, you then need to understand how the template cache and trusted cache work, so that if you change source files they get picked up by CF. Again, too much to explain here.
BTW, one may wonder if this was related to an issue described here: http://www.talkingtree.com/blog/index.cfm/2006/2/2.... It's close, but not really the same. In this case, when CF requested a template that could that not be found (such as on a mapped drive could due to a network burp), and CF would cache (in the template cache) the fact that it got a 404. You were noticing that the problem seemed to clear itself up. And indeed, Steven's entry says this bug was fixed in 7.02. So prior to that, you may have found that once this happened, the CFC would NEVER be reloaded, and there the trick offered to clear the template cache would have been your resort--at least until the network burp happened again.
That's not quite your issue. I'd assert that your "5 seconds" where the file was not available was just 5 seconds that there was network burp where your CFC couldn't be found, and it was being sought because the trusted cache was not on. CF was looking for the file to see if it had changed. Try turning on the trusted cache to see if that makes it go away in your testing.
If it does, then let's see how the app-level mapping may be related. That's interesting. I don't yet see a ready connection, other than perhaps the application.cfc itself may not be found in this instance, and then maybe CF isn't showing that error for some reason. That's a stretch.
While we're here, note that your comment above with the link to the other entry has a broken link. It somehow has 2 urls in the one href.
Finally, your server has been very erratic in its response today, sometimes not responding, sometimes getting a "503 Request timed out waiting to execute". Are you aware a known issue?
Or might this be related to your Feb 5 entry about instability with your CF instance? It was that entry which brought me to your site today, and I wanted to offer to help. I was afraid that trying to offer a comment there may be too much (indeed, consider the above!), instead I was going to offer that I do this sort of CF server troubleshooting. There are just lots of little dots to connect that even experienced people often miss. But I know experienced folks often won't think to pay for help, if they feel they can solve it themselves.
Still, once in a while I pick someone out of the community who blogs about having a problem and I offer a free consultation to help solve the problem. I now make that offer to you. If you'd be interested, drop me a note at charlie at carehart dot org. I can help you find and understand the root cause for the problems, and hopefully resolve it. I have a great track record doing so. It's all I do now (for work).
I can of course also help anyone who needs to better understand all that template cache/trusted cache stuff mentioned above, or solve other CF problems. See more at carehart.org/consulting. Sorry if that last sentence feels like a commercial. Hey, I hope it's a reasonable trade for all the info above and the offer of free help for John's problem. :-) Just trying to help.
Thanks for the detailed response. I've responded to Sean Corfield in <a href="http://www.jonhartmann.com/index.cfm/2009/3/4/Cold... other blog post</a> that the files for the mapping at local, so if my app was located at D:\websites\www.myapp.com\ then the mapping is to D:\websites\www.myapp.com\ as well. I'd not do this except that my applications are usually developed at addresses like test.company.com/myapp/ and then deployed to myapp.company.com. Usage of the mapping meant that I could just us the mapping location and not worry about if CFCs were in /cfc or /myapp/cfc/. I was setting the mapping using <cfset this.mappings["/myapp"] = GetDirectoryFromPath(GetCurrentTemplatePath()) /> in my application.cfc file.
I'd also considered the possibility of CF loosing the ability to see the location (as per Joshua Cyr above), so I expanded my test, which you can see in the second posting. The results showed that even when throwing errors, CF could see that the file existed. Given that this is a local location and that the issue seemed to have hinged upon adding a server level mapping, I'm not sure that network connectivity figures in at this point.
As for the aside, in this particular case, I do need to get new instances of the CFC. The grading process is highly database intensive (not long queries, but lots and lots of small ones), so the engine caches a lot of information to cut down on re-querying data at each step of the process. I don't use one object for everything because I worry that one object might build up too much data in is cache. Also, since the grades are very sensitive, limiting the engine to grading one thing at a time means that there is 0 chance of cross mingling of information (not that there would be, but it makes my superiors feel better).
You're right about the URL... I must have gotten hasty and messed it up. Hopefully the link above will work better. As for the server issues... yes I'm aware of it. I get free hosting from work, and one of our boxes was tanking under user load (same app we are talking about actually). Since my blog and it are on the same server (don't ask, I'm not in charge of stuff like that -_- ), I was getting taken down too.
The server instability problem is a separate but related issue, because although its the same application (a future version), it sits on another set of servers entirely. I would be more the happy to take you up on the offer, but currently the issue was "resolved" by removing Symantec Endpoint Protection from the servers. It was the same problem that was taking out the box that runs my blog on Friday, with the same solution (removal of Endpoint), but I'm not sure why CF and Endpoint would be reacting badly. I wish I had better answers about it, but I'm not the one that keeps up with the servers themselves, so I often don't get the whole picture. In the other case, I thought it might be because the servers were setup with DFS to replicate changes, but this server isn't setup that way.
http://www.gold4power.com
I've been having a problem with a cfc getting "stuck in memory", or at least that's how it seems to me. Do you think in might be related to any of the stuff in your original problem? Or do you have any ideas about how to go about dealing with this one? Here's the problem:
I am porting an application from CF7 to CF8.
In my Application.cfc onApplicationStart() method I initialise a component thus:
application.stuObjs.Utilities = createObject("component", "path.Utilities").init(dsn=application.stuConfig.dsn);
In the onRequestStart() method I make a call to onApplicationStart() so that my objects re-initialise on every request.
I add a new method to my Utilities.cfc.
When I cfdump application.stuObjs.Utilities the new method is not present.
If I RENAME the cfc to UtilitiesX.cfc (and adjust the code accordingly), application.stuObjs.Utilities contains the new method.
If I then rename the cfc back to what it was, the new method is again missing in application.stuObjs.Utilities.
So, what is CF8 doing? It appears to be storing an old instance of Utilities.cfc somewhere in memory, yet calling createObject() on it to re-initialise it seems to have no effect.
This should be a cakewalk but I just can't figure out why the object is stuck in memory. Looking through CFADMIN I can't see any setting that looks odd and clearing the template cache seems to have no effect either.
Can anybody shed any light on this?