Kumar,
I don't have a solution now. But you can try a few things.
Since this is a LDAP user source repository, you can try change the filter to read LDAP groups only or local JAVA groups only.
You can also try to read 1 JAVA groups only to see if this is an authorization issue or performance issue.
Cheers
Chenyang