master-datasources.xml content always revert back to initial configuration when I start wso2server 5.9 - postgresql

Question

I am beginner to WSO2, and I'm trying to configure Identity server data-source to PostgreSQL, using the documentation.
JDBC driver used
My latest master-datasources.xml is
<datasources-configuration xmlns:svns="http://org.wso2.securevault/configuration">
<providers>
<provider>org.wso2.carbon.ndatasource.rdbms.RDBMSDataSourceReader</provider>
</providers>
<datasources>
<datasource>
<name>WSO2_CARBON_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2CarbonDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:postgresql://localhost:5432/wso2_db</url>
<username>postgres</username>
<password>root</password>
<driverClassName>org.postgresql.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1; COMMIT</validationQuery>
<validationInterval>30000</validationInterval>
<defaultAutoCommit>true</defaultAutoCommit>
<commitOnReturn>true</commitOnReturn>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_SHARED_DB</name>
<description>Shared Database for user and registry data</description>
<jndiConfig>
<name>jdbc/SHARED_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:postgresql://localhost:5432/wso2_db</url>
<username>postgres</username>
<password>root</password>
<driverClassName>org.postgresql.Driver</driverClassName>
<testOnBorrow>true</testOnBorrow>
<maxWait>60000</maxWait>
<defaultAutoCommit>true</defaultAutoCommit>
<validationInterval>30000</validationInterval>
<maxActive>50</maxActive>
<jmxEnabled>false</jmxEnabled>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2_IDENTITY_DB</name>
<description>Shared database for identity data</description>
<jndiConfig>
<name>jdbc/WSO2IdentityDB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:postgresql://localhost:5432/wso2_db</url>
<username>postgres</username>
<password>root</password>
<driverClassName>org.postgresql.Driver</driverClassName>
</configuration>
</definition>
</datasource>
</datasources>
</datasources-configuration>
When I start running WSO2 server , master-datasources.xml revertback to initial H2 configuration.
I modified deployment.toml based on the suggestion from #Piraveena Paralogarajah.
[server]
hostname = "localhost"
node_ip = "127.0.0.1"
base_path = "https://$ref{server.hostname}:${carbon.management.port}"
[super_admin]
username = "admin"
password = "admin"
create_admin_account = true
[user_store]
type = "read_write_ldap"
connection_url = "ldap://localhost:${Ports.EmbeddedLDAP.LDAPServerPort}"
connection_name = "uid=admin,ou=system"
connection_password = "admin"
base_dn = "dc=wso2,dc=org" #refers the base dn on which the user and group search bases will be generated
[database.identity_db]
type = "postgre"
hostname = "localhost"
name = "wso2_db"
username = "postgres"
password = "root"
port = "5432"
[database.shared_db]
type = "postgre"
hostname = "localhost"
name = "wso2_db"
username = "postgres"
password = "root"
port = "5432"
[keystore.primary]
name = "wso2carbon.jks"
password = "wso2carbon"
executed Query
<IS-HOME>/dbscripts/identity/postgresql.sql
<IS-HOME>/dbscripts/identity/uma/postgresql.sql
<IS-HOME>/dbscripts/consent/postgresql.sql
this time master-datasources.xml updated for postgress. But got exception while running server.
2020-02-19 16:44:35,247] [] ERROR {org.wso2.carbon.user.core.common.DefaultRealm} - nullType class java.lang.reflect.InvocationTargetException org.wso2.carbon.user.core.UserStoreException: nullType class java.lang.reflect.InvocationTargetException
at org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:397)
at org.wso2.carbon.user.core.common.DefaultRealm.initializeObjects(DefaultRealm.java:224)
at org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:129)
at org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:264)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:102)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:115)
at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:72)
at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61)
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:842)
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:834)
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:791)
at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1013)
at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
at org.eclipse.osgi.container.Module.doStart(Module.java:598)
at org.eclipse.osgi.container.Module.start(Module.java:462)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$1.run(ModuleContainer.java:1820)
at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$2$1.execute(EquinoxContainerAdaptor.java:150)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1813)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1770)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1735)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1661)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:351)
... 25 more
Caused by: org.wso2.carbon.user.core.UserStoreException: Error occurred while checking is existing domain : PRIMARY for tenant : -1234
at org.wso2.carbon.user.core.util.UserCoreUtil.persistDomain(UserCoreUtil.java:860)
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.persistDomain(AbstractUserStoreManager.java:6190)
at org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager.<init>(ReadOnlyLDAPUserStoreManager.java:240)
at org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager.<init>(ReadWriteLDAPUserStoreManager.java:120)
... 30 more
Caused by: org.wso2.carbon.user.core.UserStoreException: DB error occurred while checking is existing domain : PRIMARY & tenant id : -1234
at org.wso2.carbon.user.core.util.UserCoreUtil.isExistingDomain(UserCoreUtil.java:1009)
at org.wso2.carbon.user.core.util.UserCoreUtil.persistDomain(UserCoreUtil.java:849)
... 33 more
Caused by: org.postgresql.util.PSQLException: ERROR: relation "um_domain" does not exist
Position: 26
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2510)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2245)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:311)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:159)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy53.executeQuery(Unknown Source)
at org.wso2.carbon.user.core.util.UserCoreUtil.isExistingDomain(UserCoreUtil.java:998)
... 34 more
[2020-02-19 16:44:35,275] [] ERROR {org.wso2.carbon.user.core.internal.Activator} - Cannot start User Manager Core bundle org.wso2.carbon.user.core.UserStoreException: Cannot initialize the realm.
at org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:274)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:102)
at org.wso2.carbon.user.core.common.DefaultRealmService.<init>(DefaultRealmService.java:115)
at org.wso2.carbon.user.core.internal.Activator.startDeploy(Activator.java:72)
at org.wso2.carbon.user.core.internal.BundleCheckActivator.start(BundleCheckActivator.java:61)
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:842)
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:834)
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:791)
at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1013)
at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:365)
at org.eclipse.osgi.container.Module.doStart(Module.java:598)
at org.eclipse.osgi.container.Module.start(Module.java:462)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$1.run(ModuleContainer.java:1820)
at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$2$1.execute(EquinoxContainerAdaptor.java:150)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1813)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1770)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1735)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1661)
at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
Caused by: org.wso2.carbon.user.core.UserStoreException: nullType class java.lang.reflect.InvocationTargetException
at org.wso2.carbon.user.core.common.DefaultRealm.initializeObjects(DefaultRealm.java:318)
at org.wso2.carbon.user.core.common.DefaultRealm.init(DefaultRealm.java:129)
at org.wso2.carbon.user.core.common.DefaultRealmService.initializeRealm(DefaultRealmService.java:264)
... 22 more
Caused by: org.wso2.carbon.user.core.UserStoreException: nullType class java.lang.reflect.InvocationTargetException
at org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:397)
at org.wso2.carbon.user.core.common.DefaultRealm.initializeObjects(DefaultRealm.java:224)
... 24 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.wso2.carbon.user.core.common.DefaultRealm.createObjectWithOptions(DefaultRealm.java:351)
... 25 more
Caused by: org.wso2.carbon.user.core.UserStoreException: Error occurred while checking is existing domain : PRIMARY for tenant : -1234
at org.wso2.carbon.user.core.util.UserCoreUtil.persistDomain(UserCoreUtil.java:860)
at org.wso2.carbon.user.core.common.AbstractUserStoreManager.persistDomain(AbstractUserStoreManager.java:6190)
at org.wso2.carbon.user.core.ldap.ReadOnlyLDAPUserStoreManager.<init>(ReadOnlyLDAPUserStoreManager.java:240)
at org.wso2.carbon.user.core.ldap.ReadWriteLDAPUserStoreManager.<init>(ReadWriteLDAPUserStoreManager.java:120)
... 30 more
Caused by: org.wso2.carbon.user.core.UserStoreException: DB error occurred while checking is existing domain : PRIMARY & tenant id : -1234
at org.wso2.carbon.user.core.util.UserCoreUtil.isExistingDomain(UserCoreUtil.java:1009)
at org.wso2.carbon.user.core.util.UserCoreUtil.persistDomain(UserCoreUtil.java:849)
... 33 more
Caused by: org.postgresql.util.PSQLException: ERROR: relation "um_domain" does not exist
Position: 26
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2510)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2245)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:311)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:447)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:368)
at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:159)
at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy53.executeQuery(Unknown Source)
at org.wso2.carbon.user.core.util.UserCoreUtil.isExistingDomain(UserCoreUtil.java:998)
... 34 more
I tried this but it is not working.

Answer 1

With the 4.5.0 carbon-kernel release, all WSO2 products such as APIM 3.0.0, IS 5.9.0 introduced a new config model. According to the new config model, there is a centralized configuration file (deployment.toml) where users add the configurations, then those configurations will be added to the respective .xml files.
So if you want to do some changes in the master-datasources.xml file, you have to add the relevant configs in deployment.toml file according to the new config model. With the new config model, all the changes made by you in the xml config files will be overridden by the toml configs during the server startup.
Please follow this documentation to refer further information on this new config model
Related documents:
https://wso2.com/blogs/thesource/2019/10/simplifying-configuration-with-WSO2-identity-server
Please follow this documentation if you are using trying to configure WSO2 Identity server with postgres db.
https://is.docs.wso2.com/en/next/setup/changing-to-postgresql/
[updated according to the new issue]
Please execute this script also
/dbscripts/postgresql.sql
. From the error logs it says "um_domain" does not exist. That table creation happens from this script and you haven't executed this particular script.

Answer 2

Caused by: org.postgresql.util.PSQLException: ERROR: relation "um_domain" does not exist
Position: 26
It seems you are missing some tables. Maybe your DB schema is not compliant with wso2 DB schema
To fix that you need to run WSO2 DB scripts on PostgresDB. You can find the scripts inside the product in the following paths {is-home}/dbscripts and {is-home}/dbscripts/identity. Postgres scripts are under the name of "postgres.sql".

Answer 3

Make sure, the deployment.toml configuration has worked like as a publisher file, So, it's rollback to H2 database because the LDAP was configured on localhost.
please follow the below.
open the developement.toml find in this path C:[Program Files]WSO2\Identity Server\5.11.0\repository\conf
Remove the LDAP ~ AD configuration and add that
[user_store]
type = "database_unique_id"
Change the user database configuration
[database.user]
url = "jdbc:postgresql://localhost:5432/wso2"
username = "postgres"
password = "MohsenPass"
driver = "org.postgresql.Driver"
Change the identity_db database configuration
[database.identity_db]
type = "postgre"
hostname = "localhost"
name = "wso2"
username = "postgres"
password = "PassMohsen"
port = "5432"
Change the shared_db database configuration
type = "postgre"
hostname = "localhost"
name = "wso2"
username = "postgres"
password = "MohsenPass"
port = "5432"
Now Start-up the server, that process will do initialization of new configuration and new destination as well,
I hope do well to fix your issues.
Any questions regarding in wso2 identity server to set up and development ask me on twitter #MohsenEnazi.