Kevin Yeandel’s Random Access Memory

“There’s data that’s been backed up and data that’s not been lost……. yet.”

I was eating porridge this morning which eminded me of my 9th birthday ‘day out with friends’. My mother was driving. Now picture this: we passed by a motorcyclist led on the pavement with the top of his head missing. He managed to slice the top clean off. I’m no more a pathologist today as I was at 9 but felt sure the fella was dead and I knew that’s what the connection with porridge is – anyone who has seen a brain go splat has seen porridge.

If a safety helmet costs £25 and you don’t fall off your bike was it money well spent or was it wasted? If you ride a bike every day can you honestly say you will never fall or be pushed and never wreck your head and spend a week or more in a medically induced coma? You might not intend to fall off or expect to be pushed but accidents happen. There are only 4 reasons I can think of not to have object level recovery installed in your docbase:

1. Your docbase is enabled with some sort of freaky application that prevents it.
2. You spent all your money on Documentum products and your department is flat broke.
3. You completely underestimate or don’t know how hard it is for a dba to ‘partially’ restore a docbase or bits of it.
4. You have low value data that doesn’t change much and very generous downtime allowances.

A continually growing number of companies have Documentum Repositories and probably their DR plan is a tried and tested feature of that system. Except of course when a bit drops off, i.e. a disk set fails carrying a chunk of docbase or a user deletes a folder full of folders and files from a docbase (that’s what we used to call a repository – and that was a joke). If a bit of docbase drops off then it’s probably quicker to roll back to yesterdays backup loosing a days work and having a downtime that leaves people with time on their hands to maintain their facebook entries while you rebuild the system.

During my time as a Documentum dba someone would come up with a complaint that they lost a file out of Documentum.

Recovering deleted documents from a repository can be 1) difficult or 2) near to (if not) impossible. Getting back a deleted folder structure will drive you insane and probably require the restoration of the repository database to a separate place for analysis (or worse, roll back the repository…. not usually an option). If there’s a chance of recovering a document’s content (because dm_clean and dm_filescan jobs have not run yet) then you have to know where to find it on the file system managed by the content server. As Documentum supports multiple content storage areas it’s usually not clean-cut. If the system has distributed content then there is an additional action to perform to get to that content back.

Basically, restoring content and metadata after an accident without the right tools can ruin the whole week for one or more people and be more costly that first thought.

It’s hard to measure the value of a tool until you REALLY, REALLY need it.

CYA have a tool-suite that aids recovery from disaster or simple deletions. It brings back the content, the metadata and all the dependencies (workflow, relations etc).

Today they launch a CYA Recycle bin.

Where a layer of complexity is a bonus

Malfeasance is a word I learned today in a Webinar and I looked on Wikipedia for an explanation. The analogy was made that if a catering company accepts a bribe to undercook food then this is malfeasance. My analogy is that if you work for the police and some offers a bribe to have their criminal record erased then that amounts to the same thing. CYA makes it harder for evidence to be covered up or lost – moreso with the SmartRecovery product but the recycle bin can lend itself to this area also. This is always good news for pharmaceuticals with submissions that require FDA approval. CYA is compliant in the pharma vertical – many pharma’s have CYA and my experience with the product is positive. (CYA also provide excellent technical support).
I think probably many of us Documentum dudes have written such a tool in the past but to make a fully operational recycle bin isn’t just a day’s coding. There are a lot of dependencies when deleting/restoring stuff – not just the document, consideration has to be given to workflows, users, acls, relations, renditions and so forth. Well, CYA have the experience from SmartRecovery and SmartReplicator to deliver such a product and, as a big CYA fan, I’m personally looking forward to seeing it in action.

About a month ago I was asked to have a look at the soon to be released D6. I’m about to embark on a detailed examination and develop a web front end on this platform.

In recent years I have been a Unix Documentum platform ’supporter’ but you can’t always choose what the end client prefers.
In this case the distribution I had was Windows and it actually brought back some fond memories…
Firstly the version of the server is Windows was XP Professional version 2002 SP2. It only had 2GB ram which is half of that required.
I have to say I am totally impressed with the D6 install. The others were fairly ok also but considering my lack of prep, I had virtually no problems getting this thing up and running. I guess really it shouldn’t be a surprise, afterall, it is a vanilla install and we know the devil is in the detail when it comes to configuring larger systems. I hope time allows me to expand my findings into a more detailed area.

So my plan was to make an install and NOT follow the instructions. I wanted to break it, fix it and learn from it. Historically there are 3 things that prevent a seemless installation when it comes to Documentum:

1. You didn’t follow the installation guides and met the system software and hardware requirements.

2. You DID follow the installation guides and met the system software and hardware requirements (we all know Documentum would never do that right?!)

3. There is a bug.

I was not to be too disappointed because I had a problem with the Tomcat and DA installation. That was soon fixed and I will discuss it later.

So I took Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 – Production and installed it out of the box. Nothing else. I tested the install with SQLPlus. It was fine. At this point I should just mention something about that version of Oracle and the versions that follow up to 10.2.0.3 – there is a bug in it and on heavily loaded systems which have multiple docbases on the same database host could cause the schemas to swap data and your content server will fail intermittently.

I progressed the installation of the content server. Even though I’ve installed Documentum products must be nearly a hundred times over the past 10 years. Pilots use checklists even if they fly the same plane every day and I use a checklist of sorts (each time I install something – even if I’ve done it 5 times that week), read the release notes or in the case of a new version it is sometimes necessary to even read the install guide! However, this install was a breeze, a few extra questions during the installation (more that the 5x version, it seemed) but it was on.

The installation automatically created DM_HOME and DOCUMENTUM in the environment variables (do this by hand on a Unix System) and added the two port entries to the c:\windows\drivers\etc\system file (Again, a manual process on a Unix server).

Now to test it I went to the DOS prompt. D6 has emulated the iapi tester too. It is still called iapi32 (on Windows and iapi on Unix). As far as I know at this time, a big change to D6 is the loss of the dmcl due to javarization but it seems the API is emulated – this is a great releaf to System Administrators who do not code beyond scripting languages.
You are probably expecting that to be in the Documentum Administrator tool also – it certainly is!

Probably one of the most well known things about D6 prior to it’s launch is the lack of a dmcl.ini. Oddly enough an empty one was created in the c:\windows directory so maybe that’s a hangover and will be purged from the code in the D6 release version.

The installation worked. I was also surprised how similar the server log is to the 5.3 and previous versions…. when running a docbase (first time, ever) it’s always good to check the server log, found in $DOCUMENTUM/dba/log. In Unix you would start the system and follow up with % tail -f <docbasename>.log but in Windows, using the Documentum Server manager you can “view log” as the system starts an tails live data from the server log as it is written.

Configurations with connection brokers and so forth are kept in dfc.properties files. For my installation of the content server it was located in:

C:\Documentum\config (aka $DOCUMENTUM/config or %DOCUMENTUM%\config – depending on your platform). There is at least one other dfc.properties file you will need to remember and that lives in the WEB-INF/classes folder of your application server that hosts your dfc/wdk apps.

for Tomcat 5.5.25 users it is the web.xml in the ./conf folder

That comes now:

So the next was to install DA. While another common information byte is that Documentum is using BEA Weblogic, let’s clear something up: The method server is indeed a 9.1 BEA App server. You may wish to make notes of the ports for your system build documentation. The applications (WDK and DFC) work fine on Tomcat and it is documented. You can also do a netstat -oa to determine what ports are in use. Even in Windows from the Service manager you can add a column to show process ID’s so between netstat and the service manager you can determine quite a bit of information.

Here’s the only problem I had:

After deploying the da.war file to the Tomcat servlet container, there were a few errors thrown into the log when a browser was launched with the url to the da. I was told

com.documentum.web.form.control.TagPoolingEnabledException: JSP tag pooling is not supported. Please refer to the product deployment guide for instructions on turning off tag pooling for the application server.

The verion of tomcat I had was not supported. A function of the new D6 WDK apps is the requirement

CATALINA_HOME=C:\TC\apache-tomcat-5.5.25

C:\TC\apache-tomcat-5.5.25\bin\catalina.bat start

You need at least firefox 2.0.0.3 or IE7 but should refer to the Webtop Release Notes document for the specifics about the versions and JRE’s for browser and java plugins.

The initial delight of being able to install the pre-release Documentum 6 in a Windows environment is fading.
The basic install was fairly trouble free but there seems to be bits missing from the deployment.

Haven’t an earlier version of Documentum to compare with but dmqdocbroker.awk is calling uname which doesn’t exist in my windows build. uname is generally a well known Unix command but Windows equivs can easily be written. Documentum did this with ‘tail’ (dmtail).

To get round this problem I modified the respective file with two commented out lines and one additional line as follows.

if (current_host==”")
{
if (!silent_mode)
printf(“Targeting current host\n”);
// system(“uname -n > host.current”);
//getline current_host < “host.current”
current_host=”localhost”
}

This is from the dmqdocbroker.awk file in the server/bin dir.

Now it works (of course a ‘bodge’ which is not recommended outside of a test/dev build but as I will tear it down later) but I think dctm need to fix this before the official release.

I noticed the Documentum Server Manager is very dated also. e.g. The button “Edit DMCL.INI” – will indeed edit a dmcl.ini but why? It is my understanding and personal experience that we are using dfc.properties files for storing docbroker and other relevant config info. I noticed the appearance of a dmcl.ini in the c:\windows dir.