When adding a new Service Provider(sp) in wso2 Identity Server, to add your security certificates, there is only a drop down box with existing certificate alias names. So in order to insert a custom certificate to this list, you can go through following.
1.Create a .pem file that has the content of your certificate. It should be in following format.
..Add content of xf09certificate...
2. Go to <IS_HOME>/respository/resources/security folder and take a backup of wso2carbon.js.
3. From your terminal, go to above folder location and add below command, which will import your certificate to the keystore. In this example, I'm trying to add sample.pem file with the alias sampleCertificate.
keytool -import -alias sampleCertificate -file sample.pem -keystore wso2carbon.jks -storepass wso2carbon
3. Once this command is executed, it will ask to Trust the store, enter yes for that question, and you should get a message like below.
'new file added to truststore'
4. Now once you restart IS server, and go to add new Service Provider page, your new file will be listed from it's alias name.
note: If your .pem file is corrupted/ invalid you will get errors at step 3 so make sure your certificate is valid.