Learning to hate CFMX

A few months back, we upgraded all our CFMX 6 servers to CFMX 6.1. Since that time, we’ve been fighting an ongoing battle with our custom CFMX/Spectra OS-based CMS over what we’ve termed "dodgy objects" in our CODB. These dodgy objects cause the CF web servers which deliver some of our sites to be amazingly unstable. Frankly, it’s driving us nucking futs…

We’ve had Robin Hilliard from RocketBoots to assess our issues and work with us on them. We found several issues, which we have progressively addressed, including:

  • a buggy Oracle driver in CFMX 6.1
  • previously undiscovered bugs in the Spectra code base (not really news)
  • several points of unfriendliness between the Spectra OS code and CFMX 6.1
  • a few others less interesting

We don’t have the option to stop using Spectra at this point, so we are having to wrestle with all these issues. The upshot of the whole thing is that management and staff confidence in our system is at a very low ebb and it’s very disheartening for me and the good (and very skilled) people who work for me. Frankly, we need a couple of wins.

What we’re seeing more often than not is a combination of increasing thread count and memory usage on the JRun executable on our servers. At some point there’s a sour spot (i.e. the opposite of sweet spot) where memory usage by the JRun executable and the ability of the executable to get more threads to do work reaches a critical point and JRun craps itself.

As far as we can tell, this critical point is approached progressively over time as the site tries to do work with dodgy data that has been entered by one of our (approximately 500) authors – usually by doing some sort of copy-paste from Word. When things such as Verity or our in-memory caching model try to access data with bad content, the memory usage and thread count creep up until the critical point is met and JRun, as stated, goes off into Lalaland.

We’re doing a bunch of things to try to cover our collective backsides until we can get all our potential solutions in, but it’s a progressive and slightly slow process, especially considering our already overworked team.

Here’s what we’re doing:

  • writing a Java class to replace all the Spectra db transaction code (cfa_contentobject, cfa_contentobjectget, etc.)
  • making sure our authoring system doesn’t allow dodgy content, especially high-ASCII characters, in the content object data/WDDX
  • putting some serious logging in our scheduled tasks (indexing, caching) to see where problems might be occurring
  • monitoring just about everything going on on our servers in real time so we can see problems occur
  • nursing our servers along in the meantime

We really need a break and some fresh ideas. Anything anyone might have, no matter how out there, is welcome.

9 Replies to “Learning to hate CFMX”

  1. For scrubbing out bad MS Word characters, you might want to look at DeMoronize at cflib.org; it’s a great UDF that cleans up all of that high ASCII garbage that pasting from Word gives you.

  2. As for garbage collection, did you tried to increase the cached templates option in the CFAdministrator? I usually get better CPU (not memory) results increasing it to the total number of cfm templates on my app.

  3. We are a spectra shop also and ended up reverting back to CF5 due to many issues in CFMX 6.1. Now we have decided to switch to the Open Source FarCry. It is very spectra like. Many spectra consultants help build in and they have done many migrations. We feel for you, good luck. FarCry also has some code for spectra that may help like a caching replacement. http://farcry.daemon.com.au/
    check out cachitron. http://farcry.daemon.com.au/go/spectra/cachitron-2.0

  4. Spectra was a great idea, done completely wrong. The gui was too hard to learn, and the database structure was horrible.

    That’s why it should be dropped and something new adopted…

    But it’s going to be a painful process…

    You might need to make sure you have the right jdbc driver for oracle…

  5. Now the biggest suggestion .. what about exchanging Spectra for another system? Spectra is so old, and hasn’t been build for the CFMX platform.

    I know this is a very hard decision on a point where management lost its trust in the solution, and you have to come up with more expenses resolving them, but compare them to the hours you invest in fixing, patching and keeping the system running.

  6. Firstly, I thought Spectras UI was just an example and typically you were supposed to come up with your own.

    I know that other Spectra houses have had problems similiar to what you outlined, and they supposedly re-wrote a lot of the framework to suite their needs. This sounds like a dawnting possibility, in that you may need to litterally go over code for code and asses why/how each piece works and refine it that way.

    My personal experience with Spectra isn’t as vast as yours, (sounds like you guys would live/breathe it) – so my only input would be to simply direct you to the Daemon guys or if you need a consultant, see Spike (www.spike.org.uk) – he used to be a MM Spectra Team person or whatever and he’s switched on.

    They’d be my first point of calls, but Robin is just as switched on, so it may be a dead-end solution. Can’t hurt to include the other two, as they may have something Robin & Co left out? doubt it, but you never know.

  7. Man, feeling your pain. I wish I had something sensible to say. FWIW, some thoughts, which I’m sure you’re already had:

    Short term – intercept authored data as close to the authoring point as possible, strip out everything that’s not absolutely white-bread. Become progressively more accepting.

    Also short-term: identify known-good data styles, force users to go through a gatekeeper/editor team that conform all data manually. Keeps you out of trouble while you solve the problems. Requires some smart monkeys.

    Staged publication: make your public-serving server independent of the authoring servers. Monitor the authoring servers after content is added, and only allow data across to the public server after it’s known-good. Trouble is how to apply the test: try it on a canary-in-coalmine system which will either flake out or serve successfully, get a human to check the content, then send to public server. Authors will experience lag, but that’s preferable to a downed server.

    Lastly: take axe to servers. Doesn’t solve immediate problem, but will improve team morale.

  8. I can say only : EXACTLY the same over here…
    still in contact with the mm gold support (or should I say “plate” support) …

    we think about this “axe idea”, so far the best solution we think

  9. We??ll still fighting and wrote some Spectra Tags new.
    We become more and more stability in our system, but the JVM hangs somtimes without a reason.

    “plate support” *lol*

Leave a Reply