Update All Jira Usernames

Recently I’d been asked to integrate Jira with Identity Provider (IdP) so that we can have Single Sign On (SSO) in place. We’re using out of box integration available in Jira, but it was not working as expected!

It turns out that IdP was providing email as SAML userid but Jira was expecting plain username. And since this both are different in Jira world, end result was Jira couldn’t able to log in given user.

To fix this problem, Atlassian suggested we need to update all usernames as emails. This in turn forced me to write a script using Jira REST APIs so that I could all usernames to their emails.

Also I was able to use TQDM library to show progress of username retrieval, something I learned new along the way.

Here is the link to source code if you like to give a try. Let me know what do you think

import logging
import requests
from dotenv import load_dotenv
import os
import sys
from pprint import pformat
import json
from datetime import datetime

# https://towardsdatascience.com/progress-bars-in-python-4b44e8a4c482?gi=6a0158a5a16e
from tqdm.auto import tqdm

logging_level_dict = {
    'DEBUG': logging.DEBUG,
    'INFO': logging.INFO,
    'ERROR': logging.ERROR,
    'WARNING': logging.WARNING,
    'CRITICAL': logging.CRITICAL
}

def get_jira_users(jira_logger, jira_base_url, auth, group_name, include_inactive_users, start_at):

    users = []
    user_count = 0
    pbar = tqdm(total=100)

    resp = requests.get(f'{jira_base_url}rest/api/2/group/member?groupname={group_name}&includeInactiveUsers={include_inactive_users}&startAt={start_at}', auth=auth)    
    if resp.status_code == 200:
        total = resp.json()['total']
        start_at = resp.json()['startAt']
        current_result_count = len(resp.json()['values'])

        while (current_result_count > 0):
            jira_logger.debug(f'Total - {total}, Starts At - {start_at}, Current Result Count - {current_result_count}')
            current_user_set = resp.json()['values']
            for i in range(current_result_count):
                jira_logger.debug(f"{current_user_set[i]['name']},{current_user_set[i]['emailAddress']},{current_user_set[i]['displayName']},{current_user_set[i]['active']}")
                users.append(f"{current_user_set[i]['name']},{current_user_set[i]['emailAddress']},{current_user_set[i]['displayName']},{current_user_set[i]['active']}")
                user_count += 1
            
            percent = (current_result_count/total)*100
            pbar.update(percent)

            # Get next set of results if available.
            start_at = start_at + current_result_count            
            resp = requests.get(f'{jira_base_url}rest/api/2/group/member?groupname={group_name}&includeInactiveUsers={include_inactive_users}&startAt={start_at}', auth=auth)    
            current_result_count = len(resp.json()['values'])
    else:
        jira_logger.error(f"Response Code: {resp.status_code}, Response Message: {resp.text}")

    jira_logger.info(f"Total {user_count} users found in group - '{group_name}'")    
    pbar.close()

    return users

def update_jira_username(jira_logger, jira_base_url, auth, username, new_username_value):

    is_update_successful = True
    update_status_code = 200
    update_status_message = ""
    
    headers = {'Content-type': 'application/json'}
    
    # name is username attribute in Jira Internal Directory
    json_body = {
        'name' : new_username_value
    }
    
    resp = requests.put(f'{jira_base_url}rest/api/2/user?username={username}', data=json.dumps(json_body), auth=auth, headers=headers)
    if resp.status_code != 200:
        is_update_successful = False
        update_status_code = resp.status_code
        update_status_message = resp.json()["errors"]["active"]
        jira_logger.error(f"Response Code: {update_status_code}, Response Message: {update_status_message}")
    
    jira_logger.info(f"{username} - update status: {is_update_successful}")
    return is_update_successful, update_status_code, update_status_message

# update user name with value from emailAddress
def update_jira_usernames(jira_logger, jira_base_url, auth, group_name, user_dict):
    update_operation_status_list = []
    update_operation_status_list.append("username,Is Update Successful?, Error Details")

    with open(f"{group_name}.group_users_update_execution.csv", "a") as output_csvfile:

        output_csvfile.writelines("\n\n")
        output_csvfile.writelines(datetime.now().strftime("%d/%b/%Y %H:%M:%S") + "\n\n")
        
        for username in user_dict:
            #if username is already updated to email, skip this user!
            if username != user_dict[username]:
                jira_logger.info(f"Update username from {username} to {user_dict[username]}")
                update_status_info_tuple = update_jira_username(jira_logger, jira_base_url, auth, username, user_dict[username])
                jira_logger.debug(f"{username} - {update_status_info_tuple[0]}")
                update_operation_status_list.append(username + "," + str(update_status_info_tuple[0]) + "," + update_status_info_tuple[2])
                output_csvfile.writelines(username + "," + str(update_status_info_tuple[0]) + "," + update_status_info_tuple[2]+ "\n")
            
def main():

    load_dotenv(override=True)

    jira_base_url = os.getenv('JIRA_ENV_BASE_URL')
    jira_env = os.getenv('JIRA_ENV')

    jira_logger = logging.getLogger(__name__)
    # Following check is necessary otherwise everytime we run this in Jupyter lab Cell, new handler is getting added resulting in duplicate logs printed!
    if not jira_logger.handlers:
        jira_logger.setLevel(logging_level_dict[os.getenv('LOG_LEVEL')])

        file_handler = logging.FileHandler(os.getenv('LOG_FILE'))
        file_handler.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - line %(lineno)d - %(message)s'))

        out_hdlr = logging.StreamHandler(sys.stdout)
        out_hdlr.setFormatter(logging.Formatter('line %(lineno)d - %(message)s'))

        jira_logger.addHandler(out_hdlr)
        jira_logger.addHandler(file_handler)

    auth = (os.getenv('USERID'), os.getenv('PASSWORD'))

    group_name = 'jira-software-users'
    include_inactive_users = 'true'
    start_at = 0
    
    jira_logger.info("Reteriving current usernames to store in backup file!")
    users = get_jira_users(jira_logger, jira_base_url, auth, group_name, include_inactive_users, start_at)
    
    if users:
        jira_logger.info("Backup complete!")
        # First write current usernames into backup file in case we need them.
        with open(f"{group_name}.{jira_env}.txt", 'w') as filehandle:
            filehandle.writelines("username,email,display_name,is_user_active\n")
            filehandle.writelines("%s\n" % user for user in users)

        username_email_dict = {}
        for user in users:
            #Also we will skip username as which we will be running this script, so that subsequent calls will not fail!
            username = user.split(',')[0].strip()
            email = user.split(',')[1].strip()
            # We don't want to update logged in User's username otherwise script will fail.
            # we will need to update username to email to make it working agian!
            if username not in [os.getenv('USERID')]:
                username_email_dict[username] = email

        jira_logger.info("Updting usernames now...")
        update_jira_usernames(jira_logger, jira_base_url, auth, group_name, username_email_dict)
        jira_logger.info("Update complete!")

if __name__ == '__main__':
    main()

How to create *Date* type ScriptRunner Script Field in Jira

This post describes how one can create a calculated Script Field of type Date based on existing Jira custom field of type Date / Date Time.

Tested on Jira Server, Data Center. You will need awesome ScriptRunner App to start with.

Why one need Calculated Field?

I’m glad you asked! Here is use case that I was tasked to solve. Our users fill Proposed Purchase Date as part of new Jira ticket creation process. Over the lifetime of this Jira ticket, this field is updated as we get more up-to-date information. But recently our Business Team came back saying they would like to know very first value that was set for this field for a given Jira Ticket. Since this value is already recorded into Change History for field Proposed Purchase Date, I wanted to find out how can I retrieve this value and present it every time Ticket is viewed. And this is where ScriptRunner comes into picture. ScriptRunner provides ability to create *calculated fields* based on available data within in Jira.

But very soon I found out that creation of this field is not easy as it sounds also there was not much documentation available either. So by joining forces with Atlassian Community finally I was able to successfully create calculated field which we can call as First Proposed Purchase Date to display initial value of Proposed Purchase Date.

Steps for creation of Script Field of type Date

  • Create a regular customer field of type Script Field
    • Navigate to Administration >> Issues >> Custom Fields >> Add Custom Fields
    • On Select a Field Type screen, select Scripted Field
    • After completion of the new custom field, go to Edit Custom Field Details screen for this new field.
      • Make sure you set Search Template as a Date Time Range Picker
      • Also as a good practice, limit the application context of this field to specific project that you are interested in.
    • Edit this Custom field to set Date Time Range picker as a Search Template
  • ScriptRunner logic setup for this Script Field
    • Go to Manage Apps >> Script Fields >> Script Fields section
    • You should see field First Proposed Purchase Date which we created as part of our first step. Click on Gear Icon >> Edit
    • Select Date Time as a template
    • And in script section enter following code
import com.atlassian.jira.component.ComponentAccessor
import java.text.SimpleDateFormat
import org.apache.log4j.Logger
import org.apache.log4j.Level
import java.util.Date

def log = Logger.getLogger("jira.script_field.first_proposed_purchase_date")
log.setLevel(Level.DEBUG);
log.debug("Inside 'First Proposed Purchase Date' Script Field Calculation, Issue Key - "+ issue.key);

def proposed_purchase_date_changes = ComponentAccessor.getChangeHistoryManager().getChangeItemsForField(issue, "Proposed Purchase Date")

//Make sure last_deployed_date has been set at least once. Else this Scripted field should return nothing!
if (proposed_purchase_date_changes && proposed_purchase_date_changes.size() > 0){
    def formatter = new SimpleDateFormat("yyyy-MM-dd")
	formatter.setTimeZone(TimeZone.getTimeZone("PST"));

    def first_change_in_proposed_purchase_date = proposed_purchase_date_changes.first()
    def from_date = first_change_in_proposed_purchase_date?.from;
    def to_date = first_change_in_proposed_purchase_date?.to;
    
    Date first_proposed_purchase_date = null;
    //If "Proposed Purchase Date" has been set during issue creation then we should use "From Date" from change History
    //Else "To Date" should be used from Change History.
    if (from_date){
    	first_proposed_purchase_date = formatter.parse(from_date)
    }
    else{
        first_proposed_purchase_date = formatter.parse(to_date)
    }

    return first_proposed_purchase_date;
}

More about Groovy Code

Code listed above is one that working successfully and helped me to resolved following bugs during testing.

  1. Calculated Field was not displayed if source field was set during Ticket Creation process itself!
    • If Proposed Purchase Date was set as part of Ticket creation process, then Jira doesn’t produce ChangeHistory for it. And hence in such cases First Proposed Purchase Date was not displayed though it should be.
    • Fix was make sure we retrieve 1st change from ChangeHistory and check if From is available or not. If yes, use this value!
  2. Calculated field was not displaying correct value for some Tickets.
    • This was due to first fix! In situations where field has been updated as part of Ticket Editing, first change is captured in *To* value.
    • So correct logic should be – In 1st change, we should always verify if *From* is available or not. If available, use that value else use *To* value
  3. Calculated field was showing incorrect value (one day offset) for some Tickets.
    • This was a tricky one. And after few tries I realized problem arising because our Jira Server time is based on UTC and hence Date Object is created it was in UTC object. Later Jira converts it into user selected time zone and that was reason it was getting it wrong sometimes!
    • To get around this problem, I’ve to set the timezone on DateFormatter to PST/PDT (our user base is primarily here in West Coast – I still think there should be better way to get this done.) And then we have a date as user set it for first time!
  4. There was also one weird problem that I faced was if we haven’t set Search Template to *Date Time Range picker*, Jira couldn’t able to re-index this field and it was not allowing creation of any new tickets across Jira. So please be careful and do this!

Want to say big thank you to Dev & Prasad for helping me to create this field successfully.

Application link creation for Jira OAuth

In order to access Jira using OAuth token, we also need to create Jira Application link using public key that you have generated as a first step.

Steps for Application Link creation in JIRA

  1. Click on Add Application Link
    1. Give any application URL that you like (and which doesn’t exists) such as https://jira-oauth-rest-api-access or http://example.com etc.
    2. You will see error No response was received from the URL you entered… Ignore it and click on Continue
    3. Application Name: JIRA OAuth REST API Access
    4. Application Type: Generic Application
    5. Click on Create
    6. Click on Edit for newly created Application link with name JIRA OAuth REST API Access.
    7. Click on Configure -> Incoming Authentication
      1. Consumer Key: jira-oauth-rest-api-access
      2. Consumer Name: jira-oauth-rest-api-access
      3. Description: <Some description if you like>
      4. Public Key: <copy it from jira_oauth.pub>
      5. Rest of the fields leave empty
    8. Now Incoming Authentication status should be shown as Configured.

 

OAuth with Jira

Jira provides rich set of REST APIs for user interaction through automation. One can authenticate with these REST APIs in three different ways: 1) Using Basic Auth 2) Cookie based Auth and 3) Using OAuth Token

It’s recommended one should use last two options specially with OAuth Token. In this tutorial we will learn how to generate OAuth token that you can use within your code.

Pr-requisite:
* Availability of OpenSSL command line
* Python 3
* Get Jira OAuth Generator library from Github

Steps

  • Clone Jira OAuth Generator library from GitHub
  • Python 3 Setup
    • Create Python Virtual Environment
      • mkvirtualenv -p python3 jira_oauth1_py3_env
    • Activate this environment to work on
      • workon jira_oauth1_py3_en
    • Install all required libraries
      • pip install -r requirements.txt
  • Generate RSA public and private keys
    openssl genrsa -out oauth.pem 1024
    openssl rsa -in oauth.pem -pubout -out oauth.pub
  • Performing OAuth Dance
    • Through Web browser login to Jira as a user for which you want to generate OAuth token.
    • From command line, run python script
      python jira_oauth_token_generator.py config/starter_oauth.config
    • Copy and paste link in browser as suggested. You will be asked to *Authorize / Decline* Upon authorization, you will get final access token information.
      • This oauth_token and oauth_token_secret data points you will need later to authenticate yourself while using OAuth with Jira.

How to use OAuth to access Jira

Couple of ways you can Access using Python.

1. Using Python Jira library (recommended):
Take a look at access_using_jira_library.py to find out how you can use OAuth with this library.
(jira_oauth1_py3_env) ➜ jira-oauth-generator git:(master) ✗ python access_using_jira_library.py config/final_oauth_token.config

Retrieving Issue: EXJIRA-123
Issue: EXJIRA-123,  Summary: Deploy new JIRA Version

Retrieving 1st 3 Jira Projects available to you
First 3 Projects are [‘ABC’, ‘EDF’, ‘EXJIRA’]

2. Using bare bones Jira REST API along with Requests Library:
Take a look at access_using_requests_package.py to find out how you can use OAuth to access Jira with out box REST APIs.
(jira_oauth1_py3_env) ➜ jira-oauth-generator git:(master) ✗ python access_using_requests_package.py config/final_oauth_token.config

Retrieving 1st Jira Project available to you
(ABC) American Born Car

Adding comment to issue ABC-123
Comment successfully added. Please verify through browser!