PCJ-The custom field 'XXXX ' requires option 'YYYY' for the import but it does not exist in the current JIRA instance


Bug fixed in recent versions of Jira

Please note, that according to JRASERVER-65734, the bug causing this issue has been fixed in recent versions of Jira. For example, those importing into Jira 7.11.0 or newer should not be affected by this issue.

Problem

Sometimes you will see this error:

The custom field 'XXXX ' requires option 'YYYY' for the import but it does not exist in the current JIRA instance

You check the configuration for custom field  'XXXX' at the destination instance and option 'YYYY' is there! This may be caused by JRASERVER-63226, which is explained with greater technical detail in issue JRASERVER-65734. We will explain here a workaround that does not require changing the source database, but only the exported zip file, so it should be more convenient to use in most cases.

How to confirm you are affected by this problem

Run these queries at the source database:

select * from fieldconfiguration where id in (select fieldconfiguration from fieldconfigschemeissuetype where fieldconfigscheme!=fieldconfiguration)
select * from customfieldoption where customfieldconfig in (select fieldconfiguration from fieldconfigschemeissuetype where fieldconfigscheme!=fieldconfiguration)

The first shows you the impacted custom field configuration (whose name usually contains a reference to the custom field name). The second query shows impacted options. If both custom field names and impacted options coincide with the ones shown in the error messages, then you have confirmed this issue.

How to Fix It

  1. First, you will need to uncompress the exported zip file and copy the file "entities.xml" into the tmp directory (or replace the ENTITIES_PATH and RESULT_PATH values in the following scripts with the path of the entities.xml file and where you want the result to be saved).
  2. Run one of the scripts below, depending on if you want to use ScriptRunner or not.
  3. After running the script, a file called "entitiesFixed.xml" will appear in the tmp directory. Copy this file to where you extracted the original "entities.xml" file, remove the old one and rename "entitiesFixed.xml" to "entities.xml".
  4. Finally, recompress the ZIP file and run the import again. Ensure the ZIP file has the following structure:


projects.zip
├── config.xml
└── data
    ├── attachments
    │   └── PROJECT_KEYS...
    └── data.zip
        ├── activeobjects.xml
        └── entities.xml

Scripts

For ScriptRunner

Paste the following code into the Script Console and click Run

CustomFieldOptionEntitiesFixer
import java.util.regex.Matcher
import java.util.regex.Pattern
import org.apache.log4j.Logger
import org.apache.log4j.Level

// Enable logging for the script
def log = Logger.getLogger("com.adaptavist.CustomFieldOptionEntitiesFixer")
log.setLevel(Level.DEBUG)

// Edit these constants if needed
final int MAX_ITERATIONS = 1
// Path for the entities.xml that caused custom field options errors
final String ENTITIES_PATH = "/tmp/entities.xml"
// Path for the new entities.xml file with those errors fixed
final String RESULT_PATH = "/tmp/entitiesFixed.xml"
// Path used for multiple iterations of this script (if necessary)
final String AUX_PATH = "/tmp/entitiesFixedAux.xml"

def currentIteration = 0
def lastPassWasAux = false
def resultIsFixed = false
def finalResultPath = RESULT_PATH
Map<String, String> blockingFieldConfigIdsToCorrectIds = [:]
// Regex pattern to detect FieldConfigSchemeIssueType lines and capture its fieldconfigscheme and fieldconfiguration ids
def fieldConfigSchemeIssueTypePattern =
        Pattern.compile($/.*<FieldConfigSchemeIssueType.*fieldconfigscheme="(\d*)".*fieldconfiguration="(\d*)".*/>.*/$)

// Initializes blockingFieldConfigIdsToCorrectIds map
def initBlockingFieldConfigIdsToCorrectIds = {
    def entitiesStream = new FileInputStream(ENTITIES_PATH)
    BufferedReader original = new BufferedReader(new InputStreamReader(entitiesStream))
    // Read the original file and find the field config ids that block the import
    // This is equivalent to running the query:
    // select * from fieldconfigschemeissuetype where fieldconfigscheme!=fieldconfiguration
    log.info "Finding blocking field config ids..."
    original.lines().each {
        Matcher match = fieldConfigSchemeIssueTypePattern.matcher(it)
        if (match.matches()) {
            def fieldConfigSchemeId = match.group(2)
            def fieldConfigurationId = match.group(1)
            if (fieldConfigSchemeId != fieldConfigurationId) {
                log.debug "  * " + match.group()
                blockingFieldConfigIdsToCorrectIds.put(
                        fieldConfigSchemeId,
                        fieldConfigurationId)
            }
        }
    }
    log.info "blockingFieldConfigIdsToCorrectIds: $blockingFieldConfigIdsToCorrectIds"
    original.close()
}

def executeOneIteration = { String entryFilePath, String resultFilePath ->
    log.info("Starting iteration #$currentIteration with entry file $entryFilePath and result file $resultFilePath")

    // Objects to read the entities.xml file
    def entitiesStream = new FileInputStream(entryFilePath)
    // Object to write into the new entities.xml file
    BufferedWriter entitiesFixed = new BufferedWriter(new FileWriter(resultFilePath))

    // Regex Pattern to detect CustomFieldOption lines
    def customFieldOptionPattern =
            Pattern.compile($/.*<CustomFieldOption.*customfieldconfig="(${String.join("|", blockingFieldConfigIdsToCorrectIds.keySet() as Collection)})".*/>.*/$)
    def customFieldValuePattern =
            Pattern.compile($/.*<CustomFieldOption.*value="([^"]*)".*/>.*/$)
    def fixedCustomFieldOptionValues = []

    log.info "Replacing blocking ids with correct ones..."

    // Reset the entities.xml reader to the start of the file
    entitiesStream.getChannel().position(0)
    BufferedReader original = new BufferedReader(new InputStreamReader(entitiesStream))

    // Read the original file again and copy it into the result file
    // This time, when we find a CustomFieldOption that points to a wrong customfieldconfig, we fix it before copying it
    original.lines().eachWithIndex{String line, int index ->
        Matcher match = customFieldOptionPattern.matcher(line)
        if(match.matches()){
            // Found a custom field value entry with wrong custom field config ids
            // Fix it and write the fixed entry
            def blockingId = match.group(1)
            def correctId = blockingFieldConfigIdsToCorrectIds.get(blockingId)
            def fixedLine = line.replace("customfieldconfig=\"$blockingId\"", "customfieldconfig=\"$correctId\"")

            log.debug "  * Found incorrect CustomFieldOption in line ${index + 1}" // +1 since file lines start at 1
            log.debug "        Original line: $line"
            log.debug "        Replacing customfieldconfig id $blockingId with $correctId"

            def valueMatch = customFieldValuePattern.matcher(line)
            if(valueMatch.matches()){
                fixedCustomFieldOptionValues.add(valueMatch.group(1))
            }

            entitiesFixed.write(fixedLine)
        } else {
            // Write every other line as-is
            entitiesFixed.write(line)
        }
        // Keep line separators between lines
        entitiesFixed.write("\n")
    }

    // Check if no more Custom Field Values need fixing
    if(fixedCustomFieldOptionValues.isEmpty()){
        resultIsFixed = true
        log.info("No more Custom Field Values needed to be fixed!")
        finalResultPath = lastPassWasAux ? AUX_PATH : RESULT_PATH
        return resultIsFixed
    }

    // Show the Custom Field Values fixed identified by their values
    log.info("Custom Field Values fixed: $fixedCustomFieldOptionValues")

    // Close both file readers to prevent memory errors
    log.debug "Closing files..."
    original.close()
    entitiesFixed.close()

    currentIteration++
    return resultIsFixed
}

// First iteration, this will always happen
initBlockingFieldConfigIdsToCorrectIds()
resultIsFixed = executeOneIteration(ENTITIES_PATH, RESULT_PATH)

// Additional iterations
while(currentIteration < MAX_ITERATIONS && !resultIsFixed){
    if(lastPassWasAux){
        resultIsFixed = executeOneIteration(AUX_PATH, RESULT_PATH)
    } else {
        resultIsFixed = executeOneIteration(RESULT_PATH, AUX_PATH)
    }
    // invert lastPassWasAux value
    lastPassWasAux = !lastPassWasAux
}

// Show where the resulting file was saved
log.info "Final fixed file is '$finalResultPath'"
return "Final fixed file is '$finalResultPath' - Please check the logs tab for the full output"

Without ScriptRunner

Install Groovy in your machine.

Create a new text file called "CustomFieldOptionEntitiesFixer.groovy" and paste the following script into it:

CustomFieldOptionEntitiesFixer.groovy
import java.util.regex.Matcher
import java.util.regex.Pattern

// Edit these constants if needed
final int MAX_ITERATIONS = 1
// Path for the entities.xml that caused custom field options errors
final String ENTITIES_PATH = "/tmp/entities.xml"
// Path for the new entities.xml file with those errors fixed
final String RESULT_PATH = "/tmp/entitiesFixed.xml"
// Path used for multiple iterations of this script (if necessary)
final String AUX_PATH = "/tmp/entitiesFixedAux.xml"

def currentIteration = 0
def lastPassWasAux = false
def resultIsFixed = false
def finalResultPath = RESULT_PATH
Map<String, String> blockingFieldConfigIdsToCorrectIds = [:]
// Regex pattern to detect FieldConfigSchemeIssueType lines and capture its fieldconfigscheme and fieldconfiguration ids
def fieldConfigSchemeIssueTypePattern =
        Pattern.compile($/.*<FieldConfigSchemeIssueType.*fieldconfigscheme="(\d*)".*fieldconfiguration="(\d*)".*/>.*/$)

// Initializes blockingFieldConfigIdsToCorrectIds map
def initBlockingFieldConfigIdsToCorrectIds = {
    def entitiesStream = new FileInputStream(ENTITIES_PATH)
    BufferedReader original = new BufferedReader(new InputStreamReader(entitiesStream))
    // Read the original file and find the field config ids that block the import
    // This is equivalent to running the query:
    // select * from fieldconfigschemeissuetype where fieldconfigscheme!=fieldconfiguration
    println "Finding blocking field config ids..."
    original.lines().each {
        Matcher match = fieldConfigSchemeIssueTypePattern.matcher(it)
        if (match.matches()) {
            def fieldConfigSchemeId = match.group(2)
            def fieldConfigurationId = match.group(1)
            if (fieldConfigSchemeId != fieldConfigurationId) {
                println "  * " + match.group()
                blockingFieldConfigIdsToCorrectIds.put(
                        fieldConfigSchemeId,
                        fieldConfigurationId)
            }
        }
    }
    println "blockingFieldConfigIdsToCorrectIds: $blockingFieldConfigIdsToCorrectIds"
    original.close()
}

def executeOneIteration = { String entryFilePath, String resultFilePath ->
    println("Starting iteration #$currentIteration with entry file $entryFilePath and result file $resultFilePath")

    // Objects to read the entities.xml file
    def entitiesStream = new FileInputStream(entryFilePath)
    // Object to write into the new entities.xml file
    BufferedWriter entitiesFixed = new BufferedWriter(new FileWriter(resultFilePath))

    // Regex Pattern to detect CustomFieldOption lines
    def customFieldOptionPattern =
            Pattern.compile($/.*<CustomFieldOption.*customfieldconfig="(${String.join("|", blockingFieldConfigIdsToCorrectIds.keySet() as Collection)})".*/>.*/$)
    def customFieldValuePattern =
            Pattern.compile($/.*<CustomFieldOption.*value="([^"]*)".*/>.*/$)
    def fixedCustomFieldOptionValues = []

    println "Replacing blocking ids with correct ones..."

    // Reset the entities.xml reader to the start of the file
    entitiesStream.getChannel().position(0)
    BufferedReader original = new BufferedReader(new InputStreamReader(entitiesStream))

    // Read the original file again and copy it into the result file
    // This time, when we find a CustomFieldOption that points to a wrong customfieldconfig, we fix it before copying it
    original.lines().eachWithIndex{String line, int index ->
        Matcher match = customFieldOptionPattern.matcher(line)
        if(match.matches()){
            // Found a custom field value entry with wrong custom field config ids
            // Fix it and write the fixed entry
            def blockingId = match.group(1)
            def correctId = blockingFieldConfigIdsToCorrectIds.get(blockingId)
            def fixedLine = line.replace("customfieldconfig=\"$blockingId\"", "customfieldconfig=\"$correctId\"")

            println "  * Found incorrect CustomFieldOption in line ${index + 1}" // +1 since file lines start at 1
            println "        Original line: $line"
            println "        Replacing customfieldconfig id $blockingId with $correctId"

            def valueMatch = customFieldValuePattern.matcher(line)
            if(valueMatch.matches()){
                fixedCustomFieldOptionValues.add(valueMatch.group(1))
            }

            entitiesFixed.write(fixedLine)
        } else {
            // Write every other line as-is
            entitiesFixed.write(line)
        }
        // Keep line separators between lines
        entitiesFixed.write("\n")
    }

    // Check if no more Custom Field Values need fixing
    if(fixedCustomFieldOptionValues.isEmpty()){
        resultIsFixed = true
        println("No more Custom Field Values needed to be fixed!")
        finalResultPath = lastPassWasAux ? AUX_PATH : RESULT_PATH
        return resultIsFixed
    }

    // Show the Custom Field Values fixed identified by their values
    println("Custom Field Values fixed: $fixedCustomFieldOptionValues")

    // Close both file readers to prevent memory errors
    println "Closing files..."
    original.close()
    entitiesFixed.close()

    currentIteration++
    return resultIsFixed
}

// First iteration, this will always happen
initBlockingFieldConfigIdsToCorrectIds()
resultIsFixed = executeOneIteration(ENTITIES_PATH, RESULT_PATH)

// Additional iterations
while(currentIteration < MAX_ITERATIONS && !resultIsFixed){
    if(lastPassWasAux){
        resultIsFixed = executeOneIteration(AUX_PATH, RESULT_PATH)
    } else {
        resultIsFixed = executeOneIteration(RESULT_PATH, AUX_PATH)
    }
    // invert lastPassWasAux value
    lastPassWasAux = !lastPassWasAux
}

// Show where the resulting file was saved
println "Final fixed file is '$finalResultPath'"
return "Final fixed file is '$finalResultPath' - Please check the logs tab for the full output"

Open a terminal in the same folder where the script file is located and execute the following command:

groovy CustomFieldOptionEntitiesFixer.groovy