Introduction

(For Oracle/Documentum audittrail lockdown please see here)

I would love to sit all day writing shell scripts for things like this and there would be no end of work too.

So I took a couple of days and wrote this script based on experience working with customers of Documentum that have separate Unix and Oracle teams and limited exposure to Documentum itself. The audience here is more operations-related than Documentum and of less value to those running Documentum on Windows (hello!). Hopefully it will help a little in crossing the bridge into the world of Documentum running on the servers which they are managing.

The point of this is really two-fold:
In the first case it gives some further insight into what is already known about the newly hosted Documentum Content Server – essentially a Unix/LinuxWindows application that writes files to a file system and makes connections to a database and to other parts within an infrastructure.

The script uses SQL instead of DQL (Documentum Query Language) – although very similar SQL is more familiar to an Oracle Admin that has not used tools like Documentum Administrator or the idql tool.

Secondly, it’s about locking down the installation from a security/operations point of view and looking for vulnerabilities.

Validity and prerequisites

This script is ‘bash’ and should be run as root. It assumes root password had been properly contained and not been made available to other administrators. If not run as root then full checks won’t be possible and it will persistently challenge for the Documentum installation owner password.

Below are highlights of the code and not the main body. I will upload a fully working script in the coming weeks.

Highlights of the code

Note the variable:

SIGNATURE=”273c7ea8ffdf840138b2c6ab86095de3″

This is the standard signature of dm_check_password which runs through many versions of Documentum (maybe as back as 5.4 unchanged).
You can deem this fingerprint to be reliable but should be properly obtained from an approved source – such as via the official EMC/Documentum download site.
As part of your upgrade/patching process it is advisable to build into your checklists a new check against dm_check_password using
md5sum dm_check_password
and place the value into the SIGNATURE variable shown above.
If there are different versions of Documentum on the same host you made need multiple copies of dm_sectest.sh – you can obviously rename the script as required.
You may wish to consider any dependencies this file has (such as libraries).

The script declares the variable SANITY
SANITY="/srv/ecm/$2/dba/config/$1/server.ini.default"

In this case a verified copy of $DOCUMENTUM/dba/config/docbase-name/server.ini to be made to a location declared by SANITY. It should be owned by root with read access to the installation owner of Documentum (not write access).

I have actually tested re-owning the actual server.ini as root with read access to the installation owner without encountering problems but you must ensure you change the permissions back before doing an upgrade. You may well find the EMC never thought to test this so probable they would report this activity as unsupported.

Example:
Locked server.ini
-rw-r–r– 1 root dmadmin 1585 May 4 2010 server.ini

Unlocked server.ini
-rw-rw-r– 1 dmadmin dmadmin 1585 May 4 2010 server.ini

While a root owned server.ini could lend itself to managing and controlling change, it is a massive hindrance to the Documentum Administrator in that a small but vital emergency increasing the number of sessions – is going to add delay at a busy (inconvenient) time but worked around by a change to the docbase start script to point the repository at a writeable initialisation file – such an unmanaged or unauthorised change will be identified by this script and written to the log.

2 final notes on the subject of server.ini: (1) The script expects a space before and after the ‘=’ with regards to variable declaration. If your server.ini doesn’t conform to this then tidy it up or rewrite this script. (2) server.ini can be a different name – e.g. if host repository installation was created using the content server cfs tool from a running repository on a different host.

And a final note in regards to configuration files in general: You could use this means (with copy and paste) to add further tests for other configuration files which are part of your installation – such as $DOCUMENTUM_SHARED/config/dfc.properties for changes to trace levels etc.

Limitation and testing

The script goes some way to reducing risk but does not exhaust all the possibilities. It is very focussed on vulnerabilities “around” Documentum and access to that core system. For example, if you upload confidential documents to a Documentum system and apply a security model which makes them public then that’s the type problem dealt with elsewhere (not this script – yet) and more to do with your system design and implementation.

I tested the script on RedHat Release 5.3 and 5.4
If you need to know which version of RedHat you are using at the prompt you can enter
cat /etc/redhat-release

The Oracle Version used for the client and server was 11.1.0.7.0
If you need to determine which version of Oracle client you are using, at the prompt type
sqlplus -version

The Documentum version was 6.5.0.342 SP3 Linux.Oracle.
If you want to know what version of Documentum you are running type
documentum -version
at the prompt but log in as the installation owner and source the variables to do this – especially if there are multiple versions of Documentum on the same host)

Usage

The script should be installed as root user with rwx permissions. The user running the script must be able to execute sqlplus.

dm_sectest.sh REPOSITORY installation_owner
Example:
./dm_sectest.sh ECMREPO dmadmin

This will set up two variables which will be used throughout the script.
DOCBASE=$1
USER=$2

Output

The output from executing the script is a log file the location and name of which is stored in
LOG=”./dm_sec_check.log”
The log should be written with the same permissions as the person executing it.
Since the script uses the same convention for renaming the docbase server log that Documentum does it backs up the old log and creates a new one.
This will fail if the user tries to run the script as non-root after previously running it as root. The name of the old log is the time at which it was saved as a new name – not the time it was executed. This can be found at the top of the report.

The code user to rename the log is

RTIME=`date +%m.%d.%Y.%H.%M.%S`
if [ -f $LOG ] ; then
mv $LOG $LOG.save.$RTIME
fi


How it works

The script goes through a number of checks which are described under the following headings.

Get the initialisation file

Analyse a process running as documentum and determine the name of the init file

INITFILE=`ps -fel | grep "\./[d]ocumentum -docbase_name $DOCBASE" | awk {'print $21'} | awk 'NR>0 && NR<1'`

This will be the initialisation file passed in by the dm_start_DOCBASE script. The process is documentum and is entered [d]ocumentum to prevent the INITFILE being blank because the ps returned the result of the grep function instead of the process.

Check the script is being run by root


if [ "$(id -u)" != "0" ]; then
echo "WARNING - Script should be run as root" >>$LOG
fi

If the script is not being run as root user you will need the passwords to the Documentum installation owner account and will be challenged for this password. It’s OK to run this as an installation owner as it will produce some very useful output but that output will also be written as the installation owner and subject to alteration.

Check the repository is running


if [[ -z "$INITFILE" ]] ; then
echo "Docbase not running"
exit
fi


Check the root_secure_validator

Since we know exactly which password check program was passed to the content server we should test it against a known signature.
The known signature is hardcoded into the script in the variable SIGNATURE and discussed previously.
This signature should be checked against a standard. You should also note that the program relies on other binary libraries which may have started life as an open source project on some operating systems.
These dependencies can be identified by examining the source code that was definitely used in the compilation of dm_check_password or using the Unix commands lsof (tricky since this examines running processes) or strace (which could be prefixed and dm_check_password and run at the prompt to grab the system calls). The dependencies may be checked for timestamps against known files.


RSVAL=$(echo `grep root_secure_validator $INITFILE | awk '{print $3}'`)
RSVSUM=$(echo `md5sum $RSVAL | awk {'print $1'}`)
if [[ "$RSVSUM" == "$SIGNATURE" ]] ; then
echo "Signature Match ($RSVSUM)" >>$LOG ;
else
echo "ALERT: Signature Mismatch. Found ($RSVSUM) expected ($SIGNATURE)" >>$LOG
fi

As can be seen in the code above, the password check program is extracted from a copy of a file which is supposed to be running in memory.

It may be possible to start Documentum then change the server.ini file on the filesystem so it would not be picked up by a security audit but changes to this file would be detected by this script in the defences are in place. But the security manager may also want to make a note of the time and date stamps on configuration files in general.

Database Checks – check the root_secure_validator

I’m keen to discover why Documentum keeps the dm_check_password program in both a database table AND a configuration file but I certainly know it totally depends on server.ini. However, as part of out checks we’ll determine where Documentum expects to see the root_secure_validator from and RDBMS perspective.


#extract a couple of values for the following functions
DOMAIN=$(echo `grep database_conn $INITFILE | awk '{print $3}'`)
DBP=$(echo `grep database_password_file $INITFILE | awk '{print $3}'`)
function db_check_root_secure_validator(){
ENC=$(cat $DBP)
X=`su - $USER -c "iapi $DOCBASE -U$USER -P -e << EOF
decrypttext,c,$ENC
exit
EOF
status=$?
" `
PASS=$(echo $X | awk -F$ENC'|~' '{print $2}' | awk {'print $2'})
if [[ -z "$PASS" ]] ; then
echo "Password failure"
exit 1
fi
X=`su $USER -c "sqlplus -s -l $DOCBASE/$PASS@$DOMAIN <<EOF
SELECT COUNT(*) FROM DM_LOCATION_S WHERE FILE_SYSTEM_PATH='\$RSVAL';
exit
EOF
"`
echo "ROWS = $X"
ROWS=$(echo $X | awk -F'COUNT\\(\\*\\) ---------- |~' '{print $2}')
if [[ "$ROWS" -eq "0" ]] ; then
echo "ALERT: Mismatch between DM_LOCATION_FILE_SYSTEM_PATH and server config" >>$LOG;
else
echo "Database and initialisation file correspond (check_password program)" >>$LOG;
fi
}


What we are doing in the above code is switching user to the installation owner to get some database connectivity details then connecting as the Oracle user and executing a query against the database. If there is an inconsistency then the report will be appended. The output is parsed as text. We could have made life easier in the SQL by including,
SET HEADING OFF
SET VERIFY OFF
SET FEEDBACK OFF
Which would cause sqlplus to return the output free of chaff and make it easier to store in a Unix variable.

Checking the LDAP Active Directory Configuration

The function below checks to see the installation is using Active Directory.
AD is a means of authenticating users into a repository. If it is setup without using network encryption then it could offer an opportunist the chance to obtain authentication details of any user using easy to install network packet sniffers.

function ldap_check_ssl(){
echo "" >>$LOG
echo "Checking LDAP configuration" >>$LOG
X=`su $USER -c "sqlplus -s -l $DOCBASE/$PASS@$DOMAIN <<$LOG
SET VERIFY OFF
SELECT ALL r_object_id,ssl_mode,ssl_port,TRIM(ldap_host),TRIM(certdb_location) FROM dm_ldap_config_s where r_object_id like '08%';
exit;
else
echo "LDAP CONFIGURATION EXISTS" >>$LOG
fi
SSLMODE=$(echo $X | awk '{print $2}')
SSLPORT=$(echo $X | awk '{print $3}')
LDAPHOST=$(echo $X | awk '{print $4}')
echo "SSL MODE = $SSLMODE" >>$LOG
echo "SSL PORT = $SSLPORT" >>$LOG
echo "LDAP HOST = $LDAPHOST" >>$LOG
if [[ "$SSLMODE" -eq "0" ]] ; then
echo "Secure Sockets Layer not available. It is recommended you change your configuration or implement alternative security" >>$LOG
fi
}

System Admin and Superuser Check

It’s very easy in Documentum as an Administrator for one Superuser to create another user with God like privileges (PLUS the ability to view, edit and purge the audit).

Since you can give that account “inline” passwords it means the Administrator doesn’t have to go through a standard process within the company to formalise a request of of a new user via, say, the AD team and thus reduce the chances of the request be challenged and sanctioned.

The block below adds a table to the report of users having System Admin or Superuser privileges. From this table you can also deduce those that have inline passwords. (I used decode in the SQL query because “inline password” is two words and I actually have far more sophisticated checks which I am not publishing and was reading the values into Unix variables)


function check_sysadmin_and_superusers(){
echo "" >>$LOG
echo "Checking User Permissions" >>$LOG
USERP=`su $USER -c "sqlplus -s -l $DOCBASE/$PASS@$DOMAIN <=8;
SPOOL $LOG APPEND
select user_name, decode(user_source,'inline password','inline',' ','NULL'),user_privileges from dm_user_s where user_privileges >=8;
SPOOL OFF
exit
EOF
"`
echo $USERP
}


Auditing privileged users

In the context of this document, an “Administrator” is one who has obtained a password to the Documentum system which gives them Administrator access to tools for which they could administrate the system. It doesn’t mean they have been working with Documentum for 1+ years and done the certifications.

This function looks for sysadmins and superusers which can execute a getlogin if they have access to a suitable tool – such as iapi (Documentum’s Interactive Application Progamming Interface) or Documentum Administrator.

A getlogin api command returns a long “token” which can be used in place of a password, so, if an Administrator wanted to connect as another user in Documentum Webtop or TaskSpace then they could do this without that user being aware.
The unaware user could be a business user who could find themselves in a situation where something was approved and published for which they could be blamed.
Although a Documentum Administrator already has access to most of that system, in most cases having a sneaky look at a document of interest would most likely not set off any alarm bells – unless that person was under corporate surveillance.

function dm_check_audit_events(){
echo “”
echo "PRIVILEGED USERS WITH NO GETLOGIN EVENT" >>$LOG
USERP=`su $USER -c "sqlplus -s -l $DOCBASE/$PASS@$DOMAIN
SET HEADING OFF
SET LINESIZE 200
SPOOL $LOG APPEND
select user_name from dm_user_s where user_privileges>=8 and user_name not in (select user_name from dmi_registry_s where event in ('dm_getlogin'));
SPOOL OFF
exit
EOF
"`
echo $USERP
echo "" >>$LOG
echo "PRIVILEGED USERS WITH GETLOGIN EVENT" >>$LOG
SPOOL $LOG APPEND
select user_name from dm_user_s where user_privileges>=8 and user_name in (select user_name from dmi_registry_s where event in ('dm_getlogin'));
SPOOL OFF
exit
EOF
"`
echo $USERP
}


Executing the code

Some tests are made when the script is executed – such as determining if the repository is running. It really begins in the function start() which is below.
It would be an idea to automate the execution of this script and add an alert handler which would email the report to a security official.

A final check done by the script is to determine if there is an alternate server.ini file on the system and compare one with unmodified parameters against one that root owns and add that to the report.

function start(){
db_check_root_secure_validator;#must be run first to get oracle password
ldap_check_ssl;
check_sysadmin_and_superusers;
dm_check_audit_events;
if [ -f $SANITY ]; then
echo "Sanity checking server init file against root secured init file" >>$LOG
D=$(echo `diff --brief $INITFILE $SANITY`)
if [[ -z "$D" ]] ; then
echo "Root secured init files compares with live" >>$LOG;
else
echo "Live init file and root secured file differ" >>$LOG
fi
else
echo "Can't sanity check $INITFILE against $SANITY because $SANITY is missing"
fi
};
#start. This is the entry point
start
echo "Finished"
#[conditionally] mutt a mail but better to make the log accessible to monitoring such as the nagios system.


Conclusion

This is a paranoid approach to adding some security or having reason to monitor situations/people – especially since Documentum is used to secure a companies most valuable assets.