Adding new survey question and improving logic for capturing answers

Author: mageroni

README.md

@@ -6,9 +6,9 @@
 
 As more companies adopt GitHub Copilot, it becomes increasingly important to measure the benefits it brings to the organization. This survey is an effort to combine both quantitative and qualitative data. To improve validity of the quantitative responses, Developers are asked to document their rationale for the time-savings percentage they choose. 
 
-Quantitative feedback from the Developer at the time of creating a PR provides valuable insights on the time savings experienced by the Developer. Time savings is needed first before other downstream impacts (like velocity increases, or other improvements can happen. The level of granularity provides multiple feedback opportunities for Developers and can capture a variety of PRs so we can understand adoption challenges and improvement opportunities. If helpful, the Survey results may also be combined with Key Performance Indicators (KPIs) that the product provides to further contextualize the survey responses.
+Quantitative feedback from the Developer at the time of creating a PR provides valuable insights on the time savings experienced by the Developer. Time savings is needed first before other downstream impacts like velocity increases, or other improvements can happen. The level of granularity provides multiple feedback opportunities for Developers and can capture a variety of PRs so we can understand adoption challenges and improvement opportunities. If helpful, the Survey results may also be combined with Key Performance Indicators (KPIs) that the product provides to further contextualize the survey responses.
 
-The survey responses are stored in your private Azure SQL database to provide insights into how developers are using the tool, the value they report, and the challenges they encounter.
+The survey responses are stored in a file called results.csv in a new branch "copilot-survey-engine-results" to provide insights into how developers are using the tool, the value they report, and the challenges they encounter.
 
 We hope that this project provides value to your organization, and we encourage you to contribute and build upon it. Your contributions can help further enhance the survey capabilities and provide even greater insights into the developer experience with Copilot.
 
@@ -55,6 +55,13 @@ Note: *If the env file does not contain a Language API Key or Endpoint, the anal
 
  - [ replace this line with your answer. ]
 
+6. ***Where did you invest your Copilot Time Savings?***
+- [ ] Resolve vulnerabilites
+- [ ] Experiment, Learn and Wellness
+- [ ] Technical debt and refactorization
+- [ ] Work on other items in the backlog
+- [ ] Other. Please explain in the comment
+
 ### Where does the app store surveys?
 
 As we receive edits on the issue, the App will validate the responses received (options selected) and once all questions have been answered, the issue will be closed automatically and the responses will be saved into a results.csv file in the same repo in which the issue was created.

index.js

@@ -65,10 +65,6 @@ module.exports = (app) => {
 
   async function GetSurveyData(context) {
     let issue_body = context.payload.issue.body;
-    let pctSelected = false;
-    let pctValue = new Array();
-    let freqSelected = false;
-    let freqValue = new Array();
 
     // save comment body if present
     let comment = null;
@@ -79,56 +75,49 @@ module.exports = (app) => {
     // find regex [0-9]\+ in issue_body and get first result
     let pr_number = issue_body.match(/[0-9]+/)[0];
 
-    // find regex \[x\] in issue_body and get complete line in an array
-    let checkboxes = issue_body.match(/\[x\].*/g);
-
-    // find if checkboxes array contains Sim o Si or Yes
-    let isCopilotUsed = checkboxes.some((checkbox) => {
+    // Get answers to first question and find if they contain affirmative answer
+    let firstQuestionResponse = await getQuestionResponse(1, 2, issue_body);
+    let isCopilotUsed = firstQuestionResponse.some((response) => {
       return (
-        checkbox.includes("Sim") ||
-        checkbox.includes("Si") ||
-        checkbox.includes("Yes") ||
-        checkbox.includes("Oui")
+        response.includes("Sim") ||
+        response.includes("Si") ||
+        response.includes("Yes") ||
+        response.includes("Oui")
       );
     });
 
-    if (isCopilotUsed) {
-      // loop through checkboxes and find the one that contains %
-      
-      for (const checkbox of checkboxes) {
-        if (checkbox.includes("%")) {
-          pctSelected = true;
-          copilotPercentage = checkbox;
-          copilotPercentage = copilotPercentage.replace(/\[x\] /g, "");
-          pctValue.push(copilotPercentage);
-          app.log.info(copilotPercentage);
-        }
-      }
+    // Get answers to second question and store in pctValue
+    let pctValue = await getQuestionResponse(2, 3, issue_body);
+    let freqValue = await getQuestionResponse(4, 5, issue_body);
+    let savingsInvestedValue = await getQuestionResponse(6, '', issue_body);
 
-      // loop through checkboxes and find the ones that do not contain % and are not Yes or No
-      for (const checkbox of checkboxes) {
-        if (
-          !checkbox.includes("%") &&
-          !checkbox.includes("Sim") &&
-          !checkbox.includes("Si") &&
-          !checkbox.includes("Yes") &&
-          !checkbox.includes("Oui") &&
-          !checkbox.includes("Não") &&
-          !checkbox.includes("No") &&
-          !checkbox.includes("Non") || 
-          checkbox.includes("Not very much")
-        ) {
-          freqSelected = true;
-          frequencyValue = checkbox;
-          frequencyValue = frequencyValue.replace(/\[x\] /g, "");
-          freqValue.push(frequencyValue);
-          app.log.info(frequencyValue);
-        }
+    if( isCopilotUsed && pctValue && freqValue && savingsInvestedValue){
+      // All questions have been answered and we can close the issue
+      app.log.info("Closing the issue");
+      try {
+        await context.octokit.issues.update({
+          owner: context.payload.repository.owner.login,
+          repo: context.payload.repository.name,
+          issue_number: context.payload.issue.number,
+          state: "closed",
+        });
+      } catch (err) {
+        app.log.error(err);
       }
-
-      if( pctSelected && freqSelected ){
-        // close the issue
+    }
+    
+    if (
+      firstQuestionResponse.some((response) => {
+        return (
+          response.includes("Não") ||
+          response.includes("No") ||
+          response.includes("Non")
+        );
+      })
+    ){
+      if (comment) {
         try {
+          // close the issue
           await context.octokit.issues.update({
             owner: context.payload.repository.owner.login,
             repo: context.payload.repository.name,
@@ -139,33 +128,8 @@ module.exports = (app) => {
           app.log.error(err);
         }
       }
-    } else {
-      if (
-        checkboxes.some((checkbox) => {
-          return (
-            checkbox.includes("Não") ||
-            checkbox.includes("No") ||
-            checkbox.includes("Non")
-          );
-        })
-      ) {
-
-        if (comment) {
-          try {
-            // close the issue
-            await context.octokit.issues.update({
-              owner: context.payload.repository.owner.login,
-              repo: context.payload.repository.name,
-              issue_number: context.payload.issue.number,
-              state: "closed",
-            });
-          } catch (err) {
-            app.log.error(err);
-          }
-        }
-      }
     }
-
+    
     let data = {
       enterprise_name: context.payload.enterprise ? context.payload.enterprise.name : '',
       organization_name: context.payload.organization ? context.payload.organization.login : '',
@@ -175,8 +139,9 @@ module.exports = (app) => {
       PR_number: pr_number || '',
       assignee_name: context.payload.issue.assignee ? context.payload.issue.assignee.login : '',
       is_copilot_used: isCopilotUsed ? 1 : 0,
-      saving_percentage: pctValue || '',
-      usage_frequency: freqValue || '',
+      saving_percentage: pctValue ? pctValue.join(" || ") : '',
+      frequency: freqValue ? freqValue.join(" || ") : '',
+      savings_invested: savingsInvestedValue ? savingsInvestedValue.join(" || ") : '',
       comment: comment || '',
       created_at: context.payload.issue ? context.payload.issue.created_at : '',
       completed_at: context.payload.issue ? context.payload.issue.updated_at : ''
@@ -187,13 +152,13 @@ module.exports = (app) => {
   }
 
   async function insertIntoFile(context) {
-      let fileContent = "";
-      let results = [];
+      let newContent = "";
       let resultString = "";
+      let results = [];
 
       try {
 
-        fileContent = await GetSurveyData(context);
+        newContent = await GetSurveyData(context);
 
         // Try to get the file
         let file = await context.octokit.repos.getContent({
@@ -207,6 +172,7 @@ module.exports = (app) => {
         let fileContents = Buffer.from(file.data.content, "base64").toString();
         // If the file contents are not empty, parse the CSV
         if (fileContents.length > 0) {
+          app.log.info("Starting to parse the CSV file...");
           // create a readable stream
           let readableStream = new stream.Readable();
           readableStream.push(fileContents);
@@ -232,14 +198,14 @@ module.exports = (app) => {
           if(issue_id_index != -1){
             // save previous comments
             if (results[issue_id_index].comment) {
-              fileContent.comment = results[issue_id_index].comment + ' || ' + fileContent.comment;
+              newContent.comment = results[issue_id_index].comment + ' || ' + newContent.comment;
             }
 
             // if the issue_id exists, update the row in the array results
-            results[issue_id_index] = fileContent;
+            results[issue_id_index] = newContent;
           }else{
             // if the issue_id does not exist, push the row into the array
-            results.push(fileContent);
+            results.push(newContent);
           }
 
           resultString = Object.keys(results[0]).join(',') + '\n'; 
@@ -267,8 +233,8 @@ module.exports = (app) => {
       } catch (error) {
         // If the file does not exist, create it
         if (error.status === 404) {
-          let completeData = 'enterprise_name,organization_name,repository_name,issue_id,issue_number,PR_number,assignee_name,is_copilot_used,saving_percentage,usage_frequency,comment,created_at,completed_at\n' 
-                              + Object.values(fileContent).join(',');
+          let completeData = 'enterprise_name,organization_name,repository_name,issue_id,issue_number,PR_number,assignee_name,is_copilot_used,saving_percentage,usage_frequency,savings_invested,comment,created_at,completed_at\n' 
+                              + Object.values(newContent).join(',');
           await createBranch(context);
           await context.octokit.repos.createOrUpdateFileContents({
             owner: context.payload.repository.owner.login,
@@ -282,49 +248,64 @@ module.exports = (app) => {
           app.log.error(error);
         }
       }
-      
-      async function createBranch(context) {
-        // Step 1: Get reference to the default branch
-        let ref;
-        try {
-          // Try to get the 'main' branch
-          const { data: mainRef } = await context.octokit.git.getRef({
-            owner: context.payload.repository.owner.login,
-            repo: context.payload.repository.name,
-            ref: 'heads/main',
-          });
-          ref = mainRef;
-        } catch (error) {
-          // If 'main' branch does not exist, try to get the 'master' branch
-          if (error.status === 404) {
-          const { data: masterRef } = await context.octokit.git.getRef({
-            owner: context.payload.repository.owner.login,
-            repo: context.payload.repository.name,
-            ref: 'heads/master',
-          });
-          ref = masterRef;
-          } else {
-            app.log.error(error);
-          }
-        }
 
-        // Step 2: Create a new branch from the default branch
-        try {
-          await context.octokit.git.createRef({
-            owner: context.payload.repository.owner.login,
-            repo: context.payload.repository.name,
-            ref: `refs/heads/${BranchName}`,
-            sha: ref.object.sha,
-          });
-        } catch (error) {
-          if (error.status === 422) {
-            app.log.info(`Branch ${BranchName} already exists`);
-          } else {
-            app.log.error(error);
-          }
-        }
+  }
+
+  async function createBranch(context) {
+    // Step 1: Get reference to the default branch
+    let RefBranch = null;
+    try {
+      // Try to get the 'main' branch
+      RefBranch = await context.octokit.git.getRef({
+        owner: context.payload.repository.owner.login,
+        repo: context.payload.repository.name,
+        ref: 'heads/main',
+      });
+    } catch (error) {
+      // If 'main' branch does not exist, try to get the 'master' branch
+      if (error.status === 404) {
+        RefBranch = await context.octokit.git.getRef({
+          owner: context.payload.repository.owner.login,
+          repo: context.payload.repository.name,
+          ref: 'heads/master',
+        });
+      } else {
+        app.log.error(error);
+      }
+    }
+
+    // Step 2: Create a new branch from the default branch
+    try {
+      await context.octokit.git.createRef({
+        owner: context.payload.repository.owner.login,
+        repo: context.payload.repository.name,
+        ref: `refs/heads/${BranchName}`,
+        sha: RefBranch.data.object.sha,
+      });
+    } catch (error) {
+      if (error.status === 422) {
+        app.log.info(`Branch ${BranchName} already exists`);
+      } else {
+        app.log.error(error);
       }
+    }
+  }
 
+  async function getQuestionResponse(start, end, issue_body) {
+    app.log.info("Getting answers for question " + start + " to " + end);
+    let AnswerSelected = false;
+    let Answers = new Array();
+    let Expression = end ? new RegExp(start + "\\. (.*" + end + "\\." + ")?", "s") : new RegExp(start + "\\. (.*" + ")?", "s");
+    let QuestionOptions = issue_body.match(Expression)[0].match(/\[x\].*/g);
+    if(QuestionOptions){
+      AnswerSelected = true;
+      QuestionOptions.forEach((option) => {
+        let cleanAnswer = option;
+        cleanAnswer = cleanAnswer.replace(/\[x\] /g, "");
+        Answers.push(cleanAnswer);
+      });
+    }
+    return AnswerSelected ? Answers : null;
   }
 
   // For more information on building apps:

issue_template/copilot-usage-en.md

@@ -28,4 +28,11 @@ For Pull Request XXX:
 
 5. ***What other information can you share about Copilot's ability to save you time coding?*** 
 
-- (Please tell us in a comment)
+- (Please tell us in a comment)
+
+6. ***Where did you invest your Copilot Time Savings?***
+- [ ] Resolve vulnerabilites
+- [ ] Experiment + Learn and Wellness
+- [ ] Technical debt and refactorization
+- [ ] Work on other items in the backlog
+- [ ] Other. Please explain in the comment

issue_template/copilot-usage-es.md

@@ -28,4 +28,11 @@ Para el Pull Request XXX:
 
 5. ***¿Qué otra información puedes compartir sobre la capacidad de Copilot para ahorrarte tiempo codificando?***
 
-- (Por favor díganos en un comentario)
+- (Por favor díganos en un comentario)
+
+6. ***¿En qué has invertido el tiempo ganado con Copilot?***
+- [ ] Resolver vulnerabilidades
+- [ ] Experimentar + Aprender y Bienestar
+- [ ] Reducir deuda técnica y/o refactorización
+- [ ] Trabajar en otras tareas en el backlog
+- [ ] Otros. Por favor explica en el comentario

issue_template/copilot-usage-fr.md

@@ -28,4 +28,11 @@ Pour le Pull Request XXX :
 
 5. ***Quelles autres informations pouvez-vous partager sur la capacité de Copilot à vous faire gagner du temps en matière de codage ?***
 
-- (Merci de nous le dire dans un commentaire)
+- (Merci de nous le dire dans un commentaire)
+
+6. ***Qu'avez-vous investi dans le temps gagné avec Copilot ?***
+- [ ] Résoudre les vulnérabilités
+- [ ] Expérience + Apprendre et/ou Bien-être
+- [ ] Réduire la dette technique et/ou refactoring
+- [ ] Travailler sur d'autres tâches dans le backlog
+- [ ] Autres. Veuillez expliquer dans le commentaire