Classloader cache is not refresh properly

Impact

After Jira restart, some groups were removed from application access license giving groups impacting thousands of users. It was the fist time that we saw this behavior we though it may have been someone manually removing them unintentionally and added those group back and the app users were able to function normally.

Finding 

I want to admit that class cast exception in question is caused by Scriptrunner plugin spring import misconfiguration and we will address it.

Also, I performed investigation and my side and found some clues on why it looks like groups are removed from application access.

At the beginning we confirmed that group application access isn't actually modified because contents of DB table "licenserolesgroup" isn't changed after issue happens.

Then I dig into the code and found the following.
Whenever that class cast exception happens it's caught and wrapped into

DirectoryInstantiationException
in
com.atlassian.crowd.directory.loader.RemoteDirectoryInstanceFactoryUtil#newRemoteDirectory(java.lang.Class<T>, com.atlassian.crowd.util.InstanceFactory, java.lang.Long, java.lang.String, java.util.Map<java.lang.String,java.lang.String>)
which is in crowd-persistence lib.

Then it comes to ApplicationServiceGeneric from crowd-core (decompiled code below)

    public Group findGroupByName(Application application, String name) throws GroupNotFoundException {
        Iterator var3 = this.getActiveDirectories(application).iterator();

        while(var3.hasNext()) {
            Directory directory = (Directory)var3.next();

            try {
                return this.directoryManager.findGroupByName(directory.getId(), name);
            } catch (GroupNotFoundException var6) {
            } catch (DirectoryNotFoundException var7) {
                throw concurrentModificationExceptionForDirectoryIteration(var7);
            } catch (com.atlassian.crowd.exception.OperationFailedException var8) {
                logger.error(var8.getMessage(), var8);
            }
        }

        throw new GroupNotFoundException(name);
    }

DirectoryInstantiationException extends OperationFailedException, method above logs and ignores it. And then throws GroupNotFoundException.

Then CrowdServiceImpl handles this exception and transforms it to null response.

    public Group getGroup(String name) {
        Application application = this.getApplication();
        if (application == null) {
            return null;
        } else {
            try {
                return ConversionUtils.toEmbeddedGroup(this.applicationService.findGroupByName(application, name));
            } catch (GroupNotFoundException var4) {
                return null;
            }
        }
    }

Then it comes to DefaultApplicationRoleManager.RoleLoader class, which loads group names that have application access and maps them to group object filtering out nulls.
And this result is cached and sits there until cache is flushed.
Which in turn looks like some groups from remote directory where removed from the application access.

Workaround

Clearing internal Jira caches which can be done using Built-in script in Scriptrunner

Affected version

Jira Software 7.13.12 and SR 5.7.x