First make sure that openser is configured properly, that means that openser is listen on TLS ports and the certificate settings are correct. For this please read the TLS guide for the openser version you use: http://www.kamailio.org/docs/
Further, if the following hints doe not solve your problems, please also read the mailing list archives of openser, e.g: http://www.nabble.com/forum/Search.jtp?forum=26014&local=y&query=tls
For testing purposes you can use the TLS certificates provided by openser. For real production usage either buy TLS certificates from a well known CA or make your own certificates (make a CA certificate which is self-signed and make a certificate for each proxy which is signed by your CA certificate - there are thousands of tutorials available on the net which tells you how to do this using the openssl tools. Note: There is no difference when making TLS certificates between SIP, http or email. Hence you can use any tutorial which describes making certificates for e.g. Apache.).
Further make sure that the CA certificate is imported into the certificate store of your SIP client - the SIP client needs it to validate the certificate presented by the SIP proxy.
Further, usually SIP clients do not have a TLS certificate. Thus, if you use TLS between SIP client and SIP proxy make sure that the TLS connection is setup by the TLS client. (In this case TLS offers authentication of the SIP proxy against the SIP client)
Some SIP clients do not support TLS (e.g. Windows Messenger) but support SSL2/3.
Mostly TLS problems are caused by failing TLS handshake due to non-matching certficate chains in the TLS client and the SIP proxy. To debug TLS problems: * increase debug level (debug=4 in openser.cfg) and watch the log messages during TLS handshake. Usually log messages are logged using syslog which per default logs to /var/log/syslog on Debian based Linux and /var/log/messages on RedHat based Linux. * use a packet sniffer to debug the handshake. Use wireshark or a specialized too like “ssldump”. Find out which party drops the TLS connection and then inspect the log file of this party to find out why the TLS connection was dropped. *
* Eyebeam Eyebeam does not have a separate certificate store but uses the certificate store of the OS (at least under Windows. Does somebody know how this is handled under MacOS or Wine?). Thus, import the CA certificate into Windows certificate store (e.g. using Internet Exporer).
* Minisip
* SNOM phones If the phone does not allow you configure TLS transport manually try the transport setting “automatic” and configure a NAPTR record (see RFC 3263) for your SIP domain with TLS as preferred protocol.
* SJphone
* pjsua This is a command line client using the pjsip SIP stack. http://www.pjsip.org/pjsua.htm#opt_tls