# rpl_datacollectiondescription

further_description:
  semi-structured, face-to-face

  "To design and test the semi-structured interview guideline for the GT study, Dillman’s [30] three-stage pretesting process was followed." (Section 3)

  "The participants were asked which IDEs they had experience with and in which areas of software development they had worked in the past or were still working in. Then, the participants were shown different types of security warnings, which are introduced in the following section. Follow-up questions were asked by the interviewer to clarify different statements the interviewee expressed." (Section 3)


further_description:
  online (Zoom)

  "(2) a 45min semi-structured interview covering perceptions and prior experiences with CSP" 
  CSP = Content Security Policy


further_description:
  coding / programming task: 
  "In the coding task, we ask each participant to create a CSP that mitigates the effect of XSS attacks for a small Web application. We inform the participants that they can use any resources throughout the task to make sure that we are simulating their usual programming behavior as closely as possible. During the procedure, the participants are asked to share the screen such that we can observe their coding behavior during the experiment." (Section 3.4)


further_description:
  "Those questions are supplemented by a drawing task in which participants draw and explain a XSS attack of their choice in detail. For this purpose, we provide the participants with a graphic containing four stakeholders: The attacker, the victim, the vulnerable server, and an attacker-controlled server (see Figure 1)." 
  "The drawing is done either via Zoom’s Annotate feature or alternatively with enabled screen sharing and diagrams.net [14]."
  (Section 3.3)


further_description:
  "In addition to the interviews, we conducted one focus group with seven academic researchers using the same guideline."
  "The participants were asked which IDEs they had experience with and in which areas of software development they had worked in the past or were still working in. Then, the participants were shown different types of security warnings, which are introduced in the following section. Follow-up questions were asked by the interviewer to clarify different statements the interviewee expressed." (Section 3)


further_description:
  online
  "To gather further insights and test our hypothesis, we designed and ran an online survey. The full survey can be found in the supplemental material." (Section 7.1)


further_description:
  1 Phd student with industry experience embedded in medium size software company for 8 months.

  "The participant observer in this research was a computer science PhD student with prior training in qualitative research methods as well as ample industry experience. This experience allowed the researcher to integrate quickly into the daily work. The researcher spent three days per week at the company for a duration of eight months. Although the researcher’s background was in security, the researcher was not limited to security-related tasks and participated in all activities a regular SWE at the company would, such as sprint planning, scrum meetings, bug fixes, feature design/implementation/testing, and code reviews." (Section 2.2)

  "we believe the development team is representative of one in a mid-sized software development company," (Section 7)


further_description:
  "This paper presents Build-it, Break-it, Fix-it (BIBIFI), a new security contest with a focus on building secure systems. A BIBIFI contest has three phases. The first phase, Build-it, asks small development teams to build software according to a provided specification that includes security goals. The software is scored for being correct, efficient, and feature-ful. The second phase, Break-it, asks teams to find defects in other teams’ build-it submissions. Reported defects, proved via test cases vetted by an oracle implementation, benefit a break-it team’s score and penalize the build-it team’s score; more points are assigned to security-relevant problems. (A team’s break-it and build-it scores are independent, with prizes for top scorers in each category.) The final phase, Fix-it, asks builders to fix bugs and thereby get points back if the process discovers that distinct break-it test cases identify the same defect." (Section 1), for more info, see Section 2.2


further_description:
  "Contestants sign up for the contest through our web application frontend, and fill out a survey when doing so, to gather demographic and other data potentially relevant to the contest outcome (e.g., programming experience and security training)." (Section 2.4)


further_description:
  Multiple types of surveys, often repeatedly administered. 
  "Roughly each hour, we stopped work and asked the subjects to complete a survey (Appendix C)." (Section 4.3)
  "Each day ended with an end-of-day survey (Appendix D), and the final day included an end-of-experiment survey (Appendix E)." (Section 4.3)


further_description:
  We collected " [...] GitLab commits, comments, and work products. We also collected data using execsnoop, which logged programs started by the subjects, and x11monitor, which monitored the subjects’ X11 cursor focus." (Section 4.3)


further_description:
  "series of technical skill assessment tests." (Section 4.2)

  "All three skill assessments followed the same form, consisting each time of a new set of five binaries taken from a pool of fifteen. We took these binaries from three popular public corpora of fuzzing targets: the Trail of Bits adaptations of the Cyber Grand Challenge binaries [38], the MIT Lincoln Laboratory Rode0day bug-injection challenges [12], and Google’s OSS-Fuzz project [32]. In some cases, we provided source code." (Section 4.2)

  "Subjects were given exactly one hour to make progress on these targets; clearly not enough time for a deep-dive into any of them. Their instructions emphasized two goals: (1) find bugs and (2) create fuzzing harnesses." (Section 4.2)


further_description:
  programming task with Android

  "In the ‘study project’, participants were provided a skeleton Android application and asked to solve three different security related programming tasks. Each participant received the same three programming tasks, although the task ordering was randomized. For each task, a corresponding unit test was provided that participants could run to check if their solution is functional. Note that the test cases do not check if the solution is secure." (Section 6.2)


further_description:
  exit survey

  "While a participant was still completing the tasks, FixDroid showed an "Open Survey" button, encouraging participation in our the survey. When a participant had written code that passed all of the three test cases, FixDroid prompted explicitly for participation in the survey. FixDroid also asked for survey participation when a participant closed or quitted Android Studio without completing all the tasks. 
  For the group with FixDroid enabled the survey included questions about the participant’s interaction with FixDroid together with demographic questions. For the group without FixDroid, the survey asked only demographic questions. Appendix Section A has details of the questions." (Section 6.4)


further_description:
  Was called experience sampling

  "In our pilot study we found from the exit survey that some participants could not recall the causes of the issues detected, nor distinguish Fixdroid functionality from existing Android Studio features. So we added functionality to sample programmers’ experience as they interact with different components of FixDroid. This functionality includes: 
  Copy & Paste event: When FixDroid detects a pasted insecure code snippet, it asks the programmer for the source of that snippet, preferably as a URL. 
  Quick-Fix applied event: When the programmer applies a suggested quick-fix, FixDroid asks how useful they found the quick-fix, on a five point Likert scale." (Section 4.3.3)


further_description:
  "Security bad practice events, including the time, the type of bad practice (pitfall), and whether the code in question was copied/pasted. 

  Security good practice events, code that avoids a pitfall, to help measure if participants’ security programming skills improve from using FixDroid. 

  Security tooltip events record whether a particular warning message has been read by developers, how long developers spend to read it. 

  Quickfix events, indicating whether an offered quickfix was used by developers, when it was used, and whether the developer used the default shortcut of Android Studio or clicked on security indicator."

  (Section 4.3.2)


further_description:
  post-study questionnaire

  "After final code submission freelancers were informed about the real purpose of the project and were invited to answer a questionnaire. The questionnaire can be found in the supplemental material and is an extended version of the one used in [21, 22]" (Section 4)


further_description:
  "As a starting point we adopted the methodology and study design frame from [21, 22], which was described in the Introduction." (Section 4)

  "In 2017 and 2018, Naiakshina et al. conducted two studies in which 40 computer science students were asked to implement the registration process for a university social network [21, 22]." (Section 1) - task was to implement password storage

  "We changed the task from a university social networking platform to a sports photo sharing social network. To make it more believable we created a web presence for the company. Screen-shots from the different pages and the task description can be found in the supplemental material." (Section 4)


further_description:
  warning message design, e.g. "(8) What do you think a helpful security warning message in the console looks like? Please sketch an example message on a piece of paper. You are allowed to draw multiple messages. Present and discuss your drawings. (9) Please look at the following security warning proposal (cf. Figure 4) for a few minutes. What do you (a) like and (b) dislike about it compared to your design? Discuss." (Section Study Protocol)


further_description:
  "We conduct four FGs with the goal to improve CAPI feedback for developers"

  FG protocol was presented via a slide deck

  CAPI = Crypographic API


further_description:
  online survey, 

  "We collected a random sample of 50,000 email addresses of Android application developers listed in Google Play (the official Android application market). We emailed these developers, introducing ourselves and asking them to take our online survey." (Section III)


further_description:
  "After a brief introduction to the study and the skeleton app, participants were given four programming tasks in random order (detailed below), with approximately 20-30 minutes to complete each."


further_description:
  structured interviews (in-person)

  "After completing each task (or running out of time), participants were given a short exit interview about their experience. Using a five-point Likert scale, we asked whether each task was fun, difficult, and whether the participant was confident they got the right answer. [...]" (Section IV, D).


further_description:
  "In addition to each participant’s code, think-aloud responses, and exit interview responses, we collected browser activity during the session (for participants in all but the book condition) using the History Export5 extension for Chrome, which stores all visited URLs in a JSON file." (Section IV E)


further_description:
  semi-structured interviews in German, 1 in person, six via telephone

  "The interview was structured into (1) general questions about the daily work routine of the participant, (2) general experiences with updates, (3) a more detailed assessment of specific aspects, and (4) additional comments. The guidelines are in Appendix B." (Section 3.1)


further_description:
  online-survey

  "Following the interviews, we performed a quantitative online survey. We created statements based on our observations in the interview study and developed an online questionnaire to quantify and enrich them." (Section 4)


further_description:
  online survey

  "In the survey, we showed the participants one of three code snippets, each of which contained a password storage implementation. The participants were asked to write code reviews for the snippets. We also asked them to list criteria on which they based their reviews and if they would release the code as it is, with minor adjustments or not at all. After the participants finished their code review, they were requested to explain the concepts of hashing and salting. Further, we asked them about their code reviewing experience and how much of a priority security is to them."

  "Since we did not ask participants to write but only review programming code, we included a small programming test to the survey, which was also used by Danilova et al. in [11]."

  "Finally, the participants had to answer demographic questions."


further_description:
  code reviewing tasks in online survey

  "For the code review task, we chose code snippets submitted by freelancers from the study conducted by Naiakshina et al. [23]" 

  "Third, we added two distraction tasks to all of the snippets: 1) Exception swallowing: An empty catch block is considered to be bad practice as possible exceptions would be ignored. 2) Logical mistake: Within an if-loop we used only one “=” in the condition"

  (Section 3.2)

  "Apart from prompting the prompted groups of participants to ensure the password is stored securely, we did not give any criteria to complete the review. We wanted to find out which criteria the participants chose and which issues they found without further requests." (Section 3)


further_description:
  semi-structured interviews (multiple participants; 1-3 per interview)

  "Face-to-face interviews were conducted if feasible. Otherwise, participants were given the choice of a phone or video conference interview. Five interviews were conducted face-to-face, 10 by phone, and six via video." (Section 3.2)

  "The interviews began with demographic questions about the organization (e.g., size, products) and the individual participants (e.g., role within the organization, professional background). Subsequently, participants were asked to describe their organizations’ development and testing practices and associated challenges for their cryptographic products. Questions then transitioned into exploring cryptographic resources used by the organizations and how the participants thought those resources might be improved, if at all. The complete interview protocol is included in Appendix A." (Section 3.2)


further_description:
  Build it, break it, fix it competitions

  "In the first (build it) phase, teams are given just under two weeks to build a project that (securely) meets a given specification. During this phase, a team’s build-it score is determined by the correctness and efficiency of their project, assessed by test cases provided by the contest organizers. [...] 
  In the second (break it) phase, teams are given access to the source code of their fellow competitors’ projects in order to look for vulnerabilities. Once a team identifies a vulnerability, they create a test case (a break) that provides evidence of exploitation. [...] Successful breaks add to a team’s break-it score, and reduce the target project’s team’s build-it score. 
  The final (fix it) phase of the contest affords teams the opportunity to fix bugs in their implementation related to submitted breaks. Doing so has the potential benefit that breaks which are superficially different may be unified by a fix, preventing them from being double counted when scoring." (Section 2.1)


further_description:
  "we perform a laboratory user study. Thereby, participants had to solve programming tasks with the help from Stack Overflow." (Section 8)


  "Participants were randomly assigned to one of two treatment conditions. [...] Both class templates provided code skeletons that were intended to reduce the participants’ workload and simplify the programming tasks. [...] All tasks had to be solved within one hour." (Section 8.1)


further_description:
  "participants were asked to complete a short exit survey. We asked specific questions addressing the effectiveness of the security nudges and whether they were noticed by the participants. Also, we only asked demographic questions at this point to avoid any bias during the study." (Section 8.3)


further_description:
  Section 3: 
  online survey through Qualtrics

  "We asked our respondents to answer questions about their Android application development behavior and context relevant for application security and privacy, and a set of demographic questions.  [...] Appendix B contains the full list of questions." (Section 3.1)


further_description:
  Binaries of developers published apps were analysed

  "We downloaded the application binaries for at least one application by each of the survey respondents, wherever possible; we ran the full set of scanning tools on each, and counted the issues (reports of possible vulnerabilities) generated. Appendix A lists the versions of the tools we used."


further_description:
  "After finishing the tasks, participants completed an exit survey about the code they wrote during the study, as well as their educational background and programming experience." (Section 3)

  "Once all tasks had been completed or abandoned, the participants were directed to a short exit survey (cf. Appendix A). We asked for their opinions about the tasks they had completed: Did they think they had solved them? How did they perceive the tasks’ difficulty? Did they think their solution was secure? We also were interested in whether they thought about security or privacy when working on the tasks. Finally, we wanted to know whether our participants had worked on similar programming problems in the past." (section 3.4)


further_description:
  "Participants were assigned to complete a set of three short programming tasks using Python: an encryption task, a task to store login credentials in an SQLite database, and a task to write a routine for a URL shortener service. Each participant was assigned the tasks in a random order (no task depended on completing a prior task)." (Section 3)


further_description:
  online survey using Qualtrics (Section 3)

  "Our survey, which can be found in Appendix A, was structured into three sections. The first part asked whether the company uses a companywide account per user that is centrally managed, e.g., for logging onto workstations, email, communications platforms, and the like. If this was the case, we asked for the authentication method(s) with which this account is secured, e.g., passwords, biometrics, or tokens. For those companies that did not have such central accounts, we opted to study the authentication methods for the company email accounts as we were fairly certain that most companies would have such a service. This way, we were able to include these companies and observe possible differences in these application areas. The second section included questions regarding the authentication method(s) details, e.g., if a password composition policy is used, who created the PCP, and asked for the PCP itself. We encouraged participants to copy and paste their policy if they were allowed. Further, we asked for our participants’ opinions on the authentication methods’ security and usability. The last section contained general information and demographic questions. As noted above, we collected only very minimal demographic information." (Section 3.1)


further_description:
  "We designed a semi-structured interview study and received IRB clearance. The interviews targeted 5 main topics: general development activities, attitude towards security, security knowledge, security processes, and software testing activities (see Appendix A for interview script)." (Section 3)

  "one-on-one interview" (Section 3)

  "Interviews were conducted in person (n = 3) or through VOIP/video-conferencing (n = 10)." (Section 3)


further_description:
  "We disclosed the respective vulnerability, offered further assistance and asked whether the developers were willing to discuss the details of the security bug either via telephone or email. We promised the developers to anonymize all information they provided and to make neither their names nor the apps’ names publicly available. 14 developers agreed to an interview"
  "The interviews were conducted in German or English, depending on the developer’s origin. However, not all developers were native speakers."


further_description:
  The authors performed an online study where participants had to perform several queries using Google Search. They were introduced into a scenario setting where they had to solve hypothetical programming tasks on the topic of AES encryption. 
  Participants were instructed to type a search query into the provided search bar for each task in order to find help online. Next, they had to select the result that appeared to be the most relevant to solve the task. To decide relevance, participants could visit the webpage of each result and make a final decision. Thereby, we stored the search queries, the links from the results including their rank, and a click log


further_description:
  The authors performed an online study where participants were presented the same tasks from Study 1, i.e., CIPHER, IV, and KEY. This time they not only had to search for potential solutions online but also had to write small Java programs in order to solve them.
  A control group used the standard google search engine, the treatment group used a version with a security-based-reranked version.


problem_description: Unsure if this is mainly interview? Or should just go with the drawing task?
further_description:
  Semi-structured interviews including a three-part drawing task. Interviews took place via Skype or in a quiet room inside the researchers labs.

   the main part of the study–namely the interview with the drawing tasks–was conducted. In order to elicit articulations and visualization of user mental models, the participants were guided through three drawing tasks based on different scenarios and asked to verbalize their thought process as they drew, consistent with traditional think aloud protocols [39].-2


further_description:
  online survey
  "We first conducted an online survey to understand developers’ attitudes and behaviors regarding ad networks (see Appendix A). We asked about participants’ experience developing apps and working with advertising networks. To gain more insights about particular experiences, we next asked participants to focus on one app for which they were involved in choosing and/or integrating ad network code. We asked what resources were used to choose an ad network, and had them rate which factors they valued when choosing an ad network. The survey concluded with demographic questions."
  (Section 4.1)


further_description:
  semi-structured interviews
  "We conducted semi-structured interviews with some survey participants to gain deeper insights into mobile developers’ views, behaviors, and attitudes towards ad networks (see Appendix B). We first asked participants about their background and experience with developing apps. Second, we asked which ad networks they had used and their respective experiences. Third, we asked how an ad network was chosen and how ads were configured in their app. Fourth, we asked about issues, problems, and consumer risks they had seen, heard of, or experienced with ad networks. Lastly, we asked broader questions to elicit their general thoughts regarding ad networks."
  (Section 4.2)


further_description:
  online survey

  "(Q1−3) Same as Section 3.1(Q1) – (Q3) 
  (Q4) Same as Section 3.1(Q4), but using percentage of development tasks instead of percentage of projects to avoid corner situations where the participant only worked on one project. Answering never here ends the survey. 
  (Q5) What are the most common cryptography-related tasks you need in your applications? Rank the tasks ... Tasks identified from S2: Store/ authenticate user login, Encrypt files, Secure connections and communications, and Transfer files securely. Additional tasks can be added through “Other” fields. Note that the timeline of S3 and S4 overlapped, which is why S3 tasks are not included here. 
  (Q6) Did you use Java Cryptography APIs before? Yes/No. Answering No ends the survey. 
  (Q7) Please rank the Java cryptographic libraries/APIs below according to your frequency of use where 1 is most used. This question examines if most developers actually use the JCA APIs. Choices include Java Cryptography Architecture (JCA) APIs (irrespective of provider) and Lightweight Bouncy Castle APIs [2]. Additional APIs can be added through “Other”. 
  (Q8) Thinking of your most-used API, <(Q7) rank#1 choice>, how would you rate its ease of use in terms of accomplishing your tasks correctly and securely? very hard to use, hard to use, easy to use, and very easy to use.
  (Q9) What [obstacles make it hard for you to learn and use|features make it easy to use] <(Q7) rank#1 choice>? Free-text. 
  (Q10) Thinking of when you have a new cryptography-related task to implement in your software (e.g., ...) do you ever have difficulties with the following? Short-form of obstacle choices is shown in Fig. 3, each with choices never, rarely, occasionally, frequently, and don’t know. Note that we divide the obstacle previously used in S2, “Identifying which concepts and algorithms to use”, into two obstacles “Identify the correct algorithm (e.g., AES vs DES) to use” and “Identify which concepts (e.g., encryption vs hashing) to use”. 
  (Q11) Same as Section 3.1(Q10)"
  (Section 5.1)


further_description:
  Used snowball sampling approach. Semi-structured interviews took about 30 minutes. In-person or online based on participant location. Researchers followed the “grand tour interviews” principle.

  "We adopted the semi-structured interview type for our research and framed our questions to allow developers to define the flow of the discussion, i.e., followed the “grand tour interviews” principle [17]. Still, we made sure all interviews included the following parts6 (not necessarily in that order): 
  • Introduction - interviewer describes the context and motivation for the study; 
  • Developer’s self-presentation - developer (D) presents her professional experience and the context of her current activities; 
  • Selection of new dependencies - D describes selection and inclusion of new dependencies into her software projects; 
  • Updating dependencies - D explains the motivations and insights of updating dependencies in her projects, i.e., when it is the right time to update, how often she updates dependencies, and if there is any routine or regulation regarding the dependency update process in her company; 
  • Usage of some automated tool for dependency analysis - D describes an automatic tool (if used) that facilitates dependency analysis process in her projects, and provides some general details about the integration of this tool; 
  • Mitigation of dependency issues - D describes how she addresses issues in dependencies (e.g., bugs or vulnerabilities); 
  • Other general comments regarding dependency management this includes some general perceptions, comments, or recommendations that D may give on the process of dependency management and, in particular, about the security issues introduced by software dependencies. 

  There were two interviewers at each interview session. Each interviewer had a list of the interview parts mentioned above and crossed off a part if she subjectively counted it as discussed. An interview finished as soon as all the parts became crossed."


further_description:
  "We designed an online survey with 16 questions, which took 5–10 minutes to answer. Out of the 16 questions, 14 were closed and respondents had to choose an answer from a list of options. The survey also had two open-ended questions."
  "For each language, there were two slightly different versions based on whether the respondents had experience in software development or not. The difference in the versions was only in the phrasing of the questions in order to reduce confusion. For example, developers were asked: “Would users be willing to use your system if they are worried about privacy issues?” whereas users were asked: “Would you be willing to use the system if you are worried about privacy issues?”" 
  (Section 2.2.)


further_description:
  "After completing the above for both the financial and gaming apps, they were sent to an exit survey with the questions about: how they would go about asking for user consent for the personalized ads, how the choice of ad type would affect an app’s revenue or number of users, what role does user privacy play in their daily development routines, and how much users and developers have control over data collected by ad networks." (Section 3)


further_description:
  semi-structured interview via video teleconference which took between 40 and 75 minutes. All interviews were conducted by a single interviewer.
  three sources to find participants: software testing and vulnerability discovery organizations, public bug bounty data, and personal contacts.

  "We performed semi-structured, video teleconference interviews, which took between 40 and 75 minutes. All interviews were conducted by a single interviewer. Using a semi-structured protocol, the interviewer focused primarily on the set of questions given in Appendix B, with the option to ask followups or skip questions that were already answered [89]. Each interview was divided along three lines of questioning: general experience, task analysis, and skill development." (III B)


further_description:
  The authors employed a semi-structured, observation-based interview protocol, designed to yield detailed insights into RE experts’ processes. Interviews lasted 70 minutes on average. Audio and video were recorded during each interview. All interviews were led by the first author who also used to be a reverse engineer.

  full-protocol in Appendix A


further_description:
  Semi-structured interviews with respect to app security techniques and with respect to the contrasts among experts.
  Interviewees were chosen opportunistically, mainly through introductions from former colleagues of the researchers.


further_description:
  Within-subjects design experiment where participants had to analyze the code snippets decompiled by the three different decompilers. 
  Participants were allowed to use the Internet during the study since this mirrors how analysts actually work. 
  The tutorial was followed by a training phase to ensure that the participants were familiar enough with the system to avoid system-related mistakes. We provided a sample code snippet and instructed the participants to rename variables in it and look for information about a library function online.

  conducted online

  (Section VII, B and C)


further_description:
  "Our study is implemented as an online experiment that can be reached through a public website." (Section III B)


  Participants had to review code changes with IIV (Improper Input Validation), fill out questions on security/vulnerability knowledge, then they got a warning about vuln. in the change and got the possibility to review it again. The experiment was concluded with IIV disclosure and questions on demographics.
  (see Figure 1)


further_description:
  "We built our threat model using a survey of 49 instructors at U.S. universities to learn what remote learning platforms they use, the features they value in a platform, and their concerns (and those of their students), with particular emphasis on privacy and security." 
  "The surveys provide a qualitative framing for our platform analyses and recommendations later in the paper. Additional details regarding survey materials and responses are provided in Appendix A."
  (Section 2.2)


further_description:
  "Separately, we surveyed 14 U.S. university administrators about their schools’ procurement processes for new learning platforms. As administrators with influence and understanding of the procurement process are harder to reach, our sample size was limited to 14 participants" 
  "The surveys provide a qualitative framing for our platform analyses and recommendations later in the paper. Additional details regarding survey materials and responses are provided in Appendix A." 
  (Section 2.2)


further_description:
  "We conducted an online survey with Android application developers who already published at least one application on Google Play." (Section 2)

  questionnaire in Appendix A 

  "In the first part of the survey we seek to answer how app developers choose and integrate libraries into their apps." (Section 2.3)

  "In the second part of the survey, we asked the participants about app release frequency, whether they update their dependencies, and about their main motivation to perform app and library updates" (Section 2.4)

  "The last part of the survey asked questions about problems that might be a reason for not updating libraries. We also requested a self-reporting on reasons for outdated libraries and asked the developers for their opinion on possible solutions." (Section 2.5)

  + questions on demographics and background


further_description:
  "From February through June 2020, we conducted 16 semi-structured interviews via videoconferencing software. Each interview included two phases. Phase one asked the participants how they discovered and learned Rust, as well as about instances when they or their company decided to use Rust for projects (or not) and why. In the second phase, we asked more technical questions about programming with Rust, including what features of the language/ecosystem they like/dislike compared to other familiar languages, as well as opinions about features relating to Rust’s security, such as ownership and unsafe blocks. Each session lasted about an hour, giving participants a chance to share detailed experiences. The full interview protocol is given in Appendix A." (Section 3.1)


further_description:
  "Survey. The survey was designed to mirror the interviews, with closed-item answer choices inspired by answers from the open-ended interview questions. The survey was broken into seven sections; Table 1 tabulates the sections and provides some example questions. The full survey is given in Appendix B. It was active from July to September 2020." (Section 3.1)


further_description:
  online, using Developer Observatory
  "We used an online, between-subjects study to compare how effectively developers could write correct, secure code using either PyCrypto as a control, or our patched version of PyCrypto with the security intervention." (Section 6)

  "Our online developer study uses our publicly available framework (cf. [37]). The framework allows participants to write and test cryptographic code in their browser, is based on a Jupyter Notebook environment [26] and was hosted on our server." (Section 6.3)


further_description:
  "a semi-structured interview." (Section 3)
  "After participants had completed the exit survey, they were interviewed about their implemented solutions and enduser password storage security mechanisms. They were also asked what they would do if they were working for a company and were given exactly the same task with the same description and time constraints." (Section 3.4)

  face-to-face interviews (lab)


further_description:
  in person laboratory environment

  "We conducted an experiment with 40 computer science students in order to explore whether task framing and different levels of framework support for password storage affect the security of software. Participants were asked to implement a registration process for a web application in a social network context, as described in [42]." (Section 3.1)

  "The experiment was performed in an in-person laboratory, which allowed us to control the study environment and the participants’ behavior." (Section 3.3)


further_description:
  "We emailed a random sample of 500 developers with an introductory email and the plea to provide us feedback on their experience with pinning, or, if they were not the developer, to please forward the email to their app’s developer." (Section 7.1)

  "Since we emailed developers who our classification framework had identified as good candidates for pinning, but offered them no incentive for taking part in our survey, we could only work with the developers who responded, leaving us with an opt-in bias." (Section 8)


further_description:
  "ethnographic study of secure software development processes in a software company using the anthropological research method of participant observation. Two PhD students in computer science, trained in qualitative methods were embedded in a software company for 1.5 years of total research time"


further_description:
  A task-based study incorporating six distinct tasks for which this developer demographic must take responsibility: setting up a development environment, reviewing code, seeking help, seeking testers, selecting an advertisement SDK, and software licensing

  "Tasks 1 and 2 were selection tasks (see Fig. 2), in which participants had to make a single choice while viewing source-code (Task 2), or choose a maximum of 3 from a number of possible options (task 1) whose order was randomized. We assessed whether participants prioritizes security by checking if they selected security-supporting options.
  Tasks 3-6 were card sorting tasks, where participants were given a number of choices presented to them in random order (see Fig. 2) an were asked to sort them in the order they would take them. We assessed whether participants sorted options that supported security as top choice (prioritizing them), and sorting options that undermined security as bottom choice (deprioritizing them)." (Section 3)


further_description:
  "we adapted its tasks into a survey with Likert scale questions eliciting the perceived impact of the tasks on security." [tasks from study 1] (Section 5)


further_description:
  "We conducted hour-long, semi-structured interviews with each participant via online video conference from May through June 2020. All interviews were conducted by the first author for consistency."

  "The interviews were divided into the following sections. 
  Introduction and Experience.
  Malware Sources
  Analysis Workflow
  Dynamic Analysis System Configuration
  Malware Analysis Evolution."
  (Section 2.2)


further_description:
  "We designed an online, between-subjects study to compare how effectively developers could quickly write correct, secure code using different cryptographic libraries." 

  "Participants were assigned to complete a short set of programming tasks using either symmetric- or asymmetrickey cryptography, using one of five Python cryptographic libraries. Assignment to one of the resulting 10 conditions was initially random, with counterbalancing to ensure roughly equivalent participant counts starting each condition. As the study progressed, however, it became clear that dropout rates varied widely by condition (see Section IV-C for details), so we weighted the random assignment to favor conditions with higher dropout rates." 

  (Section III)


further_description:
  "In-person sessions were completed in the lab. Participants could also complete the study remotely through video-conferencing (e.g., Skype or Google Hangouts). The study session consisted of an interview and a questionnaire. During the interview, participants watched 6 different video clips and evaluated their perceived accuracy."

  "The interview protocol mirrors the study by Fulton et al. [14], and is as follows: Mental models: Participants answered questions to help assess their existing mental models of cybersecurity, hacking, and encryption. Personal experience: Participants described incidents where they, or someone they knew, were being hacked. Prior media exposure: Participants recalled whether they have seen any, or knew of, fictional TV/movies that contained content related to cybersecurity, hacking, and encryption. Video clips: Participants watched 6 video clips. At the end of each clip, they answered: (i) whether they could identify the media or the scene, (ii) with a summary of the video clip, (iii) what they found realistic, and (iv) what they found unrealistic. Realism in media: Participants described their general perceptions of (i) how realistic they find media in portraying these topics, (ii) media that portray these topics realistically, and (iii) media that portray these topics unrealistically."


further_description:
  Replication of  Naiakshina et al. (2019) with the main difference is that hired freelancers knew that they were participating in an experiment.

  "The task was to complete a Java registration functionality that facilitated the storage of user properties (including user passwords) from a web form in a database. We used the task description given in [28] and hired participants from Freelancer.com. Similar to the previous studies, prompting was one variable in our study: half of the participants were tasked to securely store the passwords while the other half was not explicitly asked to do so." (Section 3)


further_description:
  online via Internet Relay Chat (IRC), semi-structured (Section 3.3)
  "To better assess the respondents’ perceptions and opinions on security misconfigurations, we opted for semi-structured interviews with specific, yet open-ended questions. Our goal is to get a broader overview of systemic influences by letting participants digress when answering, which might happen because operators tend to be enthusiastic about their work (judging from our initial experience)."

  "Furthermore, we encouraged participants to provide technical and in-depth explanations of operations related topics"

  Topics: 
  "(1) Which security-related misconfigurations have you encountered?
  (2) How do you think misconfigurations occurred?
  (3) How did misconfigurations affect you and the way your company approaches and handles security?"
  (Section 3.1)


further_description:
  "We implemented our questionnaire using Google Forms [35]." (Section 5)
  "We specifically design our questionnaire so that it allows investigating the previously stated observations.1 To address multiple categories of subjects gracefully, there are multiple paths through our questionnaire, primarily based on the subjects’ current state of employment (Figure 1). See the full questionnaire in Appendix A." 
  "We also inform the participants that completing the survey will take between 10 and 20 minutes, depending on how many of the qualitative questions they will answer. Participants were not compensated for participating in our survey. 
  Throughout our survey and wherever an estimation within a certain range was needed, we use unipolar and bipolar five-step Likert-type scales with balanced options that would be perceived as equally far apart from each other [36]. 
  We focus our survey on: Occupation, Job Environment, Daily Business, Past Misconfiguration Experience, Misconfiguration Facilitating Factors, Consequences, Opinions, and Demographics." (Section 5.1)


further_description:
  "To test the effect specification writing had on implementation we designed a study where developers would implement the part of an app’s code for storing passwords. We chose password storage as a task as it is security relevant, implementable within a relatively short space of time and is a common task with plenty of guidance available that most developers would have encountered in their work. Our study was implemented as a set of online tasks and questions (to capture rationale). Developers were randomly assigned a grouping (either specification or no-specification)" (Section III A)


further_description:
  "The overall study design is presented in Table 4. The participants were randomly assigned to one of the four experimental groups. Each participant first worked on the clear code of one program, and then on the code of the other program obfuscated with NO or OP. For each program, the participant had to solve two tasks" (Section 4.3.2)

  "Each session started with a short presentation by the same researcher using the standardized set of slides. First, the purpose of software obfuscation was introduced, then the procedure was explained. The screenshots of the two programs were included, to make the participants familiar with the programs. One or two additional researchers (depending on the number of the session participants) were in the lab to ensure the smooth execution of the experiment.
  [...]
  After answering the pre-study questionnaire, they were shown a password that they entered to unzip the zip-file with the program code. By entering the password, Eclipse was automatically set up with the corresponding source code (unobfuscated for the first program) according to the group the participants belonged to. Also, the logging of all events and timings in Eclipse started. 
  Back in the online survey, a description of the the first program was shown. On the next page of the survey the first task was presented and the solution had to be filled in. When the first task was successfully completed, the survey asked the post-task questions. Next, the second task was presented in the survey. After finishing this task, participants where asked to close Eclipse. By doing so, a log-file with all events in Eclipse was sent to our server. Participants then filled out post-task questions again. Furthermore, the post-program questions were asked. Then the password for the second program was shown and the same procedure was repeated for the second (obfuscated) program."
  (Section 4.3.3)


further_description:
  "Our experiments proceeded as follows: After the recruitment phase, the participants were invited to the lab where they were shortly briefed about the purpose of our study. After signing a consent form, they received the study assignment as presented in Appendix A. In the given scenario, they assumed the role of an administrator of an SME who is in charge of securing the communication to an Apache web server with HTTPS in order to pass a security audit. The server system to secure was based on Raspian, a Debian-based Linux distribution. The Apache version in use was 2.4.11. We prepared and implemented a fictive Certificate Authority (CA) in order to facilitate the process of getting a valid certificate and to remove any bias introduced by the procedures from a certain CA. The fictive CA was available through a simple web interface and required the submission of a valid CSR (certificate signing request) for issuing a valid certificate. The user interface was very simplistic and the browser on the local machine already trusted our CA."


further_description:
  "The requirements for participation currently work as a security consultant or auditor and to have at least 2 years of prior experience in auditing web services. The expert interviews were conducted as semi-structured interviews with 7 security experts from well-respected security consulting firms in the German-speaking region."


further_description:
  "Our study contains two independent variables, each with two levels: analyzer (Clang Static vs. libFuzzer) and difficulty (easy vs. hard)." 
  "To reduce the time needed per participant, we opted to study the difficulty level between-subjects, which then gave us a mixed factorial study design. So each participant either did the easy task with both the Clang Static Analyzer and libFuzzer or did the hard task with both analyzers. We randomly assigned the participants to the hard or easy tasks and randomized the order in which participants used the analyzers to counter learning and fatigue effects. Due to the length of the tasks and the fact that fuzzers need to run for a while to find bugs, we conducted the study online. Participants had ten days per condition and were instructed to work ten hours." (Section 3.3)

  "The second study conducted with CTF players was designed and run after the results of the first study with CS students had been evaluated. While the studies are very similar, we did make three changes which we will highlight here. Firstly, based on the results of the CS study and the expected skill level of the CTF players, we dropped the easy tasks since we did not expect to learn much there and focused on the hard task." (Section 3.4)


further_description:
  "we opted for a lab study to monitor and control the participants’ behavior. In contrast to Krombholz et al.’s study, we looked at two independent variables. We conducted an A/B test to compare the usability of Certbot with a traditional CA. Thus, we had two treatment conditions: “CA-Certbot” (CA-Cbot) for the Certbot with Let’s Encrypt condition and “CA-Traditional” (CA-Trad.) for the traditional manual CA approach." (Section 4.1)

  "To determine the time needed, we conducted several pre-studies, and settled on a maximum editing time of three hours for the CA-Traditional task and a maximum of two hours for the CA-Certbot task. After the time limit was exceeded, the participant was asked to continue with the next condition." (Section 4.2)


further_description:
  The experiment has been executed with 90 students in their first year of a two-year master's in computer science at KU Leuven. They were enrolled in an obligatory course on software architecture and the work done by the students in the context of the experiment was an integral part of the course syllabus.
  Participants worked in teams of 2 to solve 5 tasks (involving
  the hardening of a software architecture via security patterns) and were allowed to choose their teammate. The resulting 45 teams have been randomly (stratified randomization) assigned to one of two treatment conditions. 23 to the annotated condition (used security patterns with extra annotations to be included in the pattern documentation) and 22 to the plain condition (without annotations). The experiment ran over a period of two weeks and
  was mainly executed during two supervised lab sessions.


further_description:
  "The study is performed with the architecture of a fictitious bank as object. Throughout the entire course and prior to entering this study, the students had to develop a project using the banking system. Therefore, the banking domain is well-known to the participants by the time this study starts. Most importantly, the banking application offers sufficient architectural complexity to be non-trivial. Functionality-wise, the architecture of the banking application supports a customer to perform monetary transactions via a bank clerk, the web, or a mobile app. Also, the banking system needs to interact with other banks for executing wire transfers, and allow for real-time debit card payment processing. 
  The architecture is modeled using UML and documented in a 32-page PDF file. The documentation includes a context diagram, a main component-and-connector diagram, decompositions of the most important components, a deployment diagram, and documentation of the interfaces. The complete document can be found online [16]."

  "In order to reduce variability and to make results comparable, the participants are provided with an initial design of the banking system, as described earlier in section III-B. The design does not contain any specific security measure. The participants work in teams and have to extend and harden the design by means of seven tasks, which are labeled A through G."


further_description:
  "In sum, we collected the following measures from the developers: (1) responses to puzzles; (2) developer-reported perceptions of puzzle difficulty, clarity, familiarity with puzzle functions, and confidence in solving the puzzle; (3) demographic information; (4) programming experience and skills, (5) personality traits, and (6) cognitive functioning scores."


further_description:
  online

  "We collected demographic data about our study participants using an exit survey"
  "Participants were asked about general information including country of origin, age, and gender. In order to control our samples for programming expertise, we asked for developer-specific information such as years of experience in Java and whether they were professionals. Lastly, we collected security-related information such as security experience and background."


further_description:
  "asked them to recreate their process while sharing their screen (RQ1)1. We chose to have participants demonstrate their process, asking them to open all tools they used and perform all original steps, so we could observe automatic and subconscious behaviors"
  "Demonstration provides a reasonable compromise, and is a standard practice in NDM studies [44]. In practice, we believe the effect of demonstration was small, especially because the interviewer asked probing questions to reveal subconscious actions."
  (Section 3.1)


further_description:
  "We conducted a pre-test study in which novice developers added SSL to their apps and used pinning and self-signed certificates. Since our framework made all these tasks trivial, we decided not to deploy this study on a large scale, since it would not have led to any valuable insights." (Section 6.1)


further_description:
  Method description unclear - probably interview
  "We presented our approach to the 14 developers from Section 4.2 and queried them if these features would fulfill their requirements and remove the need to customize the way their app uses SSL. We also asked if they would feel comfortable using configuration instead of coding to add advanced features such as SSL pinning to their apps." (Section 6.1)


further_description:
  "a short questionnaire with closed-ended questions covering demographics and questions on the participants’ online communication behavior." (Section III A)


further_description:
  "In order to elicit articulations and visualization of user mental models, the participants were guided through three drawing tasks based on different scenarios and asked to verbalize their thought process as they drew, consistent with traditional think aloud protocols [39]. The scenarios were (1) a general scenario of sending an encrypted message to a communication partner, (2) online shopping via HTTPS, and (3) online banking." (Section III A)


further_description:
  "we conducted an online survey-based between-subject experiment"
  "Each participant was presented with two hypothetical scenarios in a random order: one was about a gaming app, another one was about a financial app for personal finance management."
  "Participants were asked to imagine that they were a shareholder in a software development company, and together with a small team, they created a (financial or gaming) app, which will be published in Europe and the United States and is mainly targeted towards adults above the age of 18. Then, we asked them to answer questions posed by the “Acme Assistant”, a tool for an imaginary ad network that helps with integrating the ad network into the app." (Section 3)


further_description:
  "After finishing each task, participants are shown a brief questionnaire, where they can score the quality of the code produced by the decompiler, and a text field for additional feedback. Here, the participants are able to see the code again. We asked a total of 8 questions, 6 on readability properties and 2 on trust issues. Similar to the System Usability Score (SUS) [8], the questions are counterbalanced (positive/negative) to minimize the response bias, e.g., "This code was easily readable" and "It was strenuous to understand what this code did". The full question set can be found in Appendix B."

  (Section VII C)


further_description:
  "In addition to the questionnaire after each task, at the end of the study, we asked the participants about an overall rating on a scale from 1 (worst) to 10 (best). During this step they were shown the code snippets for every task and every decompiler to facilitate the direct comparison. To avoid biasing the participants the decompilers were named M, P and R." (Section VII C)


further_description:
  "Before the one-on-one interview, participants filled out a demographics questionnaire." (Section 3)


further_description:
  "For the survey we applied the suggested approach of [21] to use seven-point rating scales instead of five-point scales. We asked participants for their demographics and programming experience. They also had to indicate their experience with the APIs employed in the study."

  "we asked participants a Single Ease Question (SEQ) [60, 61] after they had read the study task description, but before they had started to work on the implementation task. Additionally, they had to answer the SEQ after they had finished the implementation task."
  (Section 3.4)


further_description:
  "an implementation task which was logged in its entirety," (Section 3)

  "Our goal was to design a Web development task that was short enough to be solved in one working day but also long enough to hide the fact that secure password storage was the only thing of interest to us."

  "The application which the participants were supposed to extend was designed with the standard Model-View-Controller pattern. The front end was provided and the database back end fully implemented and documented. To ease the integration, students were given an illustration of the layers for the whole application and documentation of the existing Java classes with implementation hints (see Appendix F, G). To facilitate a natural process, we explicitly allowed participants to use any kind of information source that could be found on the Internet."
  (Section 3.3)


further_description:
  "an implementation task which was logged in its entirety," 

  "In our usability lab, we provided the setting, including software and hardware, and thus were able to track all actions of the participants. Second, we had full control of time constraints and could avoid any kind of unnecessary disturbance during the implementation task. Finally, we could ensure that participants did not get outside help without such assistance being logged and the record available for analysis."
  (Section 3)

  "We decided to allow students to go back into the code after reading the questions in the survey but built in logging mechanisms to track when this happened." (Section 3.3)


further_description:
  "Before working on the task, participants filled out a short entry survey regarding their expectations for task difficulty. They also completed a self-assessment of their programming skills (see Appendix B). After finishing the implementation task, participants were required to complete an exit survey (see Appendix C). We asked participants for their demographics, security background knowledge, programming experience, and experience with the task and APIs. Furthermore, we asked open questions that could be answered with free text."
  (Section 3.4)


further_description:
  "We created an instrumented Ubuntu distribution designed for developer studies that included code-specific tracking features. Thus, we were able to collect all data produced by the participants within the 8 h sessions (e.g., the web history and program code history). Every code snippet that was compiled was secured in a history folder. In addition to a video recording of the participants’ desktops, the setup also allowed us to take frequent snapshots of their progress. In order to capture copy/paste events, we used Glipper [2], a clipboard manager for GNOME, which we modified slightly to meet our requirements (e.g., adding a time stamp to the events in a log file). In this manner, the study environment allowed us to identify all participants who copied and pasted code for password storage and the websites from which they received the code." (Section 3.3)


further_description:
  See Figure 1 (survey questions intersperse, e.g. in step 3)
  "the exact questions are available in the accompanying material [36])."


further_description:
  "Once both tasks had been completed or abandoned, the participants were directed to a short exit survey. We asked for their opinions about the tasks they had completed and the PyCrypto API, including the Acar et al. usability questionnaire for security APIs [1]. We also collected their demographics and programming experience. The participant’s code for each task was displayed (imported from our database) for their reference with each question about that task. We were also interested in whether participants perceived the security warning at all, if it was helpful and if participants could recall the security warning’s content. The Exit survey can be found in the Appendix D." (Section 6.5)


further_description:
  pre-study (screening) survey

  "As such, potential participants completed a 20 question survey (see Appendix A) that collected basic demographic information as well as information regarding participants’ area of expertise, technical skills, and job tasks."


further_description:
  "With an anonymous survey conducted before the start of the study, we have collected additional background information from the individual participants."


further_description:
  "At the end of the experiment, the teams were asked to fill in an online questionnaire. The purpose of the questionnaire is to validate the experimental assumptions and to gather feedback from the participants." (Section VI)


further_description:
  "After completing the programming task, participants were asked to fill out an online survey to obtain their opinions on the used frameworks, their demographics, and their feedback about the task." (Section 3)


further_description:
  "Participants were asked to keep a diary while working on the task detailing what they spent time on and what problems they encountered."


further_description:
  "After completing both tasks, participants took part in a 30-minute semi-structured interview." (Section 3.3)


further_description:
  "Since the tasks require C/C++ and Linux skills, we used a pre-questionnaire as a filter. We selected a self-reported skill level for Linux and C/C++ of four or higher on a scale of one to seven." (Section 3.3) for CS-Students

  "is pool contains roughly 80 people, of which 16 filled out the pre-screening survey. We removed participants who did not have at least one year of CTF experience and had taken part in at least one online and one in-person CTF challenge" (Section 3.4) for CTF-Players


further_description:
  "To counter this issue, we introduced an in-scenario support channel, similar to the study pilot used by Garfinkel et al. to interact with participants [22]. We used the Mattermost chat client6, an open source web chat platform, and a playbook (see Appendix D) to implement the support channel." (Section 4.5)

  playbook used for support interactions


further_description:
  "Qualitative data was collected from analyzing the discussions of the communication channel," 
  "Quantitative data was gathered from the analysis of the screen recordings of each participant in combination with the collected bash log files and the Apache2 configuration files." (Section 5)


further_description:
  "After being exposed to a condition, participants were asked to fill out a survey concerning the task they had just completed. At the end of the survey for the second task, they were additionally asked to complete a final survey on which comparative questions were asked." (Section 5.5)


further_description:
  "In order to collect data, we used a think-aloud protocol. While the participants were working on the task, they articulated their thoughts while an experimenter seated next to them observed their work and took notes." (Section 3.1)


further_description:
  "In addition to the notes from the observation, we captured the bash and browser history and the final configuration files." (Section 3.1)


further_description:
  "After completing the task, the participants were asked to fill out a short questionnaire with closedand open-ended questions which covered basic demographics, previous security experience in industry and reflections on the experiment. The complete assignment and questionnaire can be found in the Appendix of this paper." (Section 3.1)


further_description:
  "Our methodology follows that of the Fulton et al. study [14], with extra questions in the post-test questionnaire." (Section 3)

  "The study involved completing an online screener questionnaire, followed by either an in-person or remote study session for those who qualified. All questionnaires used in the study were hosted on Qualtrics 2. The questionnaires had a “prefer not to answer” option available for all questions." (Section 3.2)


further_description:
  "To record actions performed by participants during code analysis, we use the Eclipse plugin Fluorite [24] that creates an XML-log of all commands and events with the corresponding timestamps." (Section 3.3.2)


further_description:
  "All questionnaires and the code comprehension tasks were combined into one online questionnaire that was developed with LimeSurvey" (Section 4.1.2)

  "After answering the pre-study questionnaire, they were shown a password that they entered to unzip the zip-file with the program code. [...] When the first task was successfully completed, the survey asked the post-task questions. [...] Participants then filled out post-task questions again. Furthermore, the post-program questions were asked. " (Section 4.3.3)


further_description:
  "During the procedure, the participants are asked to share the screen such that we can observe their coding behavior during the experiment. Furthermore, we ask them to think aloud during the task, such that we not only see what they are doing but also understand the decisions made during development." (Section 3.4)


further_description:
  "Finally, participants were asked whether they had any formal qualifications in software engineering, or computer science; and, to rate their knowledge of security and cryptography on a 5-point Likert scale,and briefly describe their security and development experience." 

  "Our study was implemented as a set of online tasks and questions (to capture rationale)."

problem_description:
  somewhat unclear whether these are questions in survey form or in interview form
  "online questions" does point towards "survey" though.


further_description:
  exit survey

  "Once all tasks had been completed or abandoned, the participant was directed to a short exit survey. We asked for their opinions about the tasks they had completed and the library they used, including the standard System Usability Scale (SUS) [63] score for the library. We also collected their demographics and programming experience. The participant’s code for each task was displayed (imported from our database) for their reference with each question about that task." (Section III G)


further_description:
  "Our manual analysis of 94 BIBIFI projects" (consist of data from build-it phase ("Submitted projects may be written in any programming language and are free to use open-source libraries, so long as they can be built on a standard Ubuntu Linux VM."), break-it phase ("Once a team identifies a vulnerability, they create a test case (a break) that provides evidence of exploitation."), and fix-it phase ("The final (fix it) phase of the contest affords teams the opportunity to fix bugs in their implementation related to submitted breaks.")


# rpl_factorinfo

problem_description:
  21 of 52 tasks (40.4%) were solved functionally correctly

  ? probably 52, but not directly in that text


problem_description:
  "Because all functional solutions to the storage task were graded as secure regardless of condition, the regression coefficient for that task approaches infinity, and the results of the model estimates for that task are not interpretable. We omit it from the table." (Section V C)


# rpl_paper

doi: 10.1145/3377811.3380387
title:
  One Size Does Not Fit All: A Grounded Theory and Online Survey Study of Developer Preferences for Security Warning Types

bibtex:
  @INPROCEEDINGS{danilova2020OneSize,  
  author={Danilova, Anastasia and Naiakshina, Alena and Smith, Matthew},  
  booktitle={2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)},   
  title={One Size Does Not Fit All: A Grounded Theory and Online Survey Study of Developer Preferences for Security Warning Types},   
  year={2020},  
  volume={},  
  number={},  
  pages={136-148},  
  publisher={ACM}, 
  doi={10.1145/3377811.3380387}
  }


title: 12 Angry Developers A Qualitative Study on Developers’ Struggles with CSP
doi: 10.1145/3460120.3484780
bibtex:
  @inproceedings{10.1145/3460120.3484780,
  author = {Roth, Sebastian and Gr\"{o}ber, Lea and Backes, Michael and Krombholz, Katharina and Stock, Ben},
  title = {12 Angry Developers - A Qualitative Study on Developers' Struggles with CSP},
  year = {2021},
  isbn = {9781450384544},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3460120.3484780},
  doi = {10.1145/3460120.3484780},
  abstract = {The Web has improved our ways of communicating, collaborating, teaching, and entertaining us and our fellow human beings. However, this cornerstone of our modern society is also one of the main targets of attacks, most prominently Cross-Site Scripting (XSS). A correctly crafted Content Security Policy (CSP) is capable of effectively mitigating the effect of those Cross-Site Scripting attacks. However, research has shown that the vast majority of all policies in the wild are trivially bypassable.To uncover the root causes behind the omnipresent misconfiguration of CSP, we conducted a qualitative study involving 12 real-world Web developers. By combining a semi-structured interview, a drawing task, and a programming task, we were able to identify the participant's misconceptions regarding the attacker model covered by CSP as well as roadblocks for secure deployment or strategies used to create a CSP.},
  booktitle = {Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {3085–3103},
  numpages = {19},
  keywords = {content security policy, roadblocks, usable security, web security},
  location = {Virtual Event, Republic of Korea},
  series = {CCS '21}
  }


doi: 10.1145/3313831.3376791
title:
  On Conducting Security Developer Studies with CS Students: Examining a Password-Storage Study with CS Students, Freelancers, and Company Developers

bibtex:
  @inbook{10.1145/3313831.3376791,
  author = {Naiakshina, Alena and Danilova, Anastasia and Gerlitz, Eva and Smith, Matthew},
  title = {On Conducting Security Developer Studies with CS Students: Examining a Password-Storage Study with CS Students, Freelancers, and Company Developers},
  year = {2020},
  isbn = {9781450367080},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3313831.3376791},
  abstract = {Ecological validity is a major concern in usable security studies with developers. Many studies are conducted with computer science (CS) students out of convenience, since recruiting professional software developers in sufficient numbers is very challenging. In a password-storage study, Naiakshina et al. (CHI'19) showed that CS students behave similarly to freelance developers recruited online. While this is a promising result for conducting developer studies with students, an open question remains: Do professional developers employed in companies behave similarly as well? To provide more insight into the ecological validity of recruiting students for security developer studies, we replicated the study of Naiakshina et al. with developers from diverse companies in Germany. We found that developers employed in companies performed better than students and freelancers in a direct comparison. However, treatment effects were found to be significant in all groups; the treatment effects on CS students also held for company developers.},
  booktitle = {Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems},
  pages = {1–13},
  numpages = {13}
  }


title: An Analysis of the Role of Situated Learning in Starting a Security Culture in a Software Company
bibtex:
  @inproceedings {Tuladhar2021AnalysisRole,
  author = {Anwesh Tuladhar and Daniel Lende and Jay Ligatti and Xinming Ou},
  title = {An Analysis of the Role of Situated Learning in Starting a Security Culture in a Software Company},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {617--632},
  url = {https://www.usenix.org/conference/soups2021/presentation/tuladhar},
  publisher = {USENIX Association},
  month = aug,
  }


title: Build It, Break It, Fix It: Contesting Secure Development
doi: 10.1145/2976749.2978382
bibtex:
  @inproceedings{Ruef2016BuildIt,
  author = {Ruef, Andrew and Hicks, Michael and Parker, James and Levin, Dave and Mazurek, Michelle L. and Mardziel, Piotr},
  title = {Build It, Break It, Fix It: Contesting Secure Development},
  year = {2016},
  isbn = {9781450341394},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/2976749.2978382},
  doi = {10.1145/2976749.2978382},
  abstract = {Typical security contests focus on breaking or mitigating the impact of buggy systems. We present the Build-it, Break-it, Fix-it (BIBIFI) contest, which aims to assess the ability to securely build software, not just break it. In BIBIFI, teams build specified software with the goal of maximizing correctness, performance, and security. The latter is tested when teams attempt to break other teams' submissions. Winners are chosen from among the best builders and the best breakers. BIBIFI was designed to be open-ended-teams can use any language, tool, process, etc. that they like. As such, contest outcomes shed light on factors that correlate with successfully building secure software and breaking insecure software. During 2015, we ran three contests involving a total of 116 teams and two different programming problems. Quantitative analysis from these contests found that the most efficient build-it submissions used C/C++, but submissions coded in other statically-typed languages were less likely to have a security flaw; build-it teams with diverse programming-language knowledge also produced more secure code. Shorter programs correlated with better scores. Break-it teams that were also successful build-it teams were significantly better at finding security bugs.},
  booktitle = {Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {690–703},
  numpages = {14},
  keywords = {software engineering, security},
  location = {Vienna, Austria},
  series = {CCS '16}
  }


title: The Industrial Age of Hacking
bibtex:
  @inproceedings {Nosco2020IndustrialAge,
  author = {Timothy Nosco and Jared Ziegler and Zechariah Clark and Davy Marrero and Todd Finkler and Andrew Barbarello and W. Michael Petullo},
  title = {The Industrial Age of Hacking},
  booktitle = {29th USENIX Security Symposium (USENIX Security 20)},
  year = {2020},
  isbn = {978-1-939133-17-5},
  pages = {1129--1146},
  url = {https://www.usenix.org/conference/usenixsecurity20/presentation/nosco},
  publisher = {USENIX Association},
  month = aug,
  }


title: A Stitch in Time: Supporting Android Developers in Writing Secure Code
doi: 10.1145/3133956.3133977
bibtex:
  @inproceedings{Nguyen2017StitchTime,
  author = {Nguyen, Duc Cuong and Wermke, Dominik and Acar, Yasemin and Backes, Michael and Weir, Charles and Fahl, Sascha},
  title = {A Stitch in Time: Supporting Android Developers in WritingSecure Code},
  year = {2017},
  isbn = {9781450349468},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3133956.3133977},
  doi = {10.1145/3133956.3133977},
  abstract = {Despite security advice in the official documentation and an extensive body of security research about vulnerabilities and exploits, many developers still fail to write secure Android applications. Frequently, Android developers fail to adhere to security best practices, leaving applications vulnerable to a multitude of attacks. We point out the advantage of a low-time-cost tool both to teach better secure coding and to improve app security. Using the FixDroid IDE plug-in, we show that professional and hobby app developers can work with and learn from an in-environment tool without it impacting their normal work; and by performing studies with both students and professional developers, we identify key UI requirements and demonstrate that code delivered with such a tool by developers previously inexperienced in security contains significantly less security problems. Perfecting and adding such tools to the Android development environment is an essential step in getting both security and privacy for the next generation of apps.},
  booktitle = {Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {1065–1077},
  numpages = {13},
  keywords = {support developers, android security, usable security, cryptographic api},
  location = {Dallas, Texas, USA},
  series = {CCS '17}
  }


doi: 10.1145/3290605.3300370
title:
  If you want, I can store the encrypted password." A Password-Storage Field Study with Freelance Developers

bibtex:
  @inproceedings{Naiakshina2019IfYouWant,
  author = {Naiakshina, Alena and Danilova, Anastasia and Gerlitz, Eva and von Zezschwitz, Emanuel and Smith, Matthew},
  title = {"If You Want, I Can Store the Encrypted Password": A Password-Storage Field Study with Freelance Developers},
  year = {2019},
  isbn = {9781450359702},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3290605.3300370},
  doi = {10.1145/3290605.3300370},
  abstract = {In 2017 and 2018, Naiakshina et al. (CCS'17, SOUPS'18) studied in a lab setting whether computer science students need to be told to write code that stores passwords securely. The authors' results showed that, without explicit prompting, none of the students implemented secure password storage. When asked about this oversight, a common answer was that they would have implemented secure storage - if they were creating code for a company. To shed light on this possible confusion, we conducted a mixed-methods field study with developers. We hired freelance developers online and gave them a similar password storage task followed by a questionnaire to gain additional insights into their work. From our research, we offer two contributions. First of all, we reveal that, similar to the students, freelancers do not store passwords securely unless prompted, they have misconceptions about secure password storage, and they use outdated methods. Secondly, we discuss the methodological implications of using freelancers and students in developer studies.},
  booktitle = {Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems},
  pages = {1–12},
  numpages = {12},
  keywords = {usable security and privacy, security developer study, field study, developer password study},
  location = {Glasgow, Scotland Uk},
  series = {CHI '19}
  }


title: Listen to Developers! A Participatory Design Study on Security Warnings for Cryptographic APIs
doi: 10.1145/3313831.3376142
bibtex:
  @inbook{Gorski2020ListenDevelopers,
  author = {Gorski, Peter Leo and Acar, Yasemin and Lo Iacono, Luigi and Fahl, Sascha},
  title = {Listen to Developers! A Participatory Design Study on Security Warnings for Cryptographic APIs},
  year = {2020},
  isbn = {9781450367080},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3313831.3376142},
  abstract = {The positive effect of security information communicated to developers through API warnings has been established. However, current prototypical designs are based on security warnings for end-users. To improve security feedback for developers, we conducted a participatory design study with 25 professional software developers in focus groups. We identify which security information is considered helpful in avoiding insecure cryptographic API use during development. Concerning console messages, participants suggested five core elements, namely message classification, title message, code location, link to detailed external resources, and color. Design guidelines for end-user warnings are only partially suitable in this context. Participants emphasized the importance of tailoring the detail and content of security information to the context. Console warnings call for concise communication; further information needs to be linked externally. Therefore, security feedback should transcend tools and should be adjustable by software developers across development tools, considering the work context and developer needs.},
  booktitle = {Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems},
  pages = {1–13},
  numpages = {13}
  }


title: You Get Where You’re Looking For: The Impact of Information Sources on Code Security
doi: 10.1109/SP.2016.25
bibtex:
  @INPROCEEDINGS{Acar2016YouGet,
    author={Acar, Yasemin and Backes, Michael and Fahl, Sascha and Kim, Doowon and Mazurek, Michelle L. and Stransky, Christian},
    booktitle={2016 IEEE Symposium on Security and Privacy (SP)}, 
    title={You Get Where You're Looking for: The Impact of Information Sources on Code Security}, 
    year={2016},
    volume={},
    number={},
    pages={289-305},
    doi={10.1109/SP.2016.25}
  }


title:
  Security, Availability, and Multiple Information Sources: Exploring Update Behavior of System Administrators

bibtex:
  @inproceedings {Tiefenau2020SecurityAvailability,
  author = {Christian Tiefenau and Maximilian H{\"a}ring and Katharina Krombholz and Emanuel von Zezschwitz},
  title = {Security, Availability, and Multiple Information Sources: Exploring Update Behavior of System Administrators},
  booktitle = {Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020)},
  year = {2020},
  isbn = {978-1-939133-16-8},
  pages = {239--258},
  url = {https://www.usenix.org/conference/soups2020/presentation/tiefenau},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  Code Reviewing as Methodology for Online Security Studies with Developers - A Case Study with Freelancers on Password Storage

bibtex:
  @inproceedings {Danilova2021CodeReviewing,
  author = {Anastasia Danilova and Alena Naiakshina and Anna Rasgauski and Matthew Smith},
  title = {Code Reviewing as Methodology for Online Security Studies with Developers - A Case Study with Freelancers on Password Storage},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {397--416},
  url = {https://www.usenix.org/conference/soups2021/presentation/danilova},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  “We make it a big deal in the company”: Security Mindsets in Organizations that Develop Cryptographic Products

bibtex:
  @inproceedings {Haney2018WeMake,
  author = {Julie M. Haney and Mary Theofanos and Yasemin Acar and Sandra Spickard Prettyman},
  title = {"We make it a big deal in the company": Security Mindsets in Organizations that Develop Cryptographic Products},
  booktitle = {Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018)},
  year = {2018},
  isbn = {978-1-939133-10-6},
  address = {Baltimore, MD},
  pages = {357--373},
  url = {https://www.usenix.org/conference/soups2018/presentation/haney-mindsets},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  Understanding security mistakes developers make: Qualitative analysis from Build It, Break It, Fix It

bibtex:
  @inproceedings {Votipka2020UnderstandingSecurity,
  author = {Daniel Votipka and Kelsey R. Fulton and James Parker and Matthew Hou and Michelle L. Mazurek and Michael Hicks},
  title = {Understanding security mistakes developers make: Qualitative analysis from Build It, Break It, Fix It},
  booktitle = {29th USENIX Security Symposium (USENIX Security 20)},
  year = {2020},
  isbn = {978-1-939133-17-5},
  pages = {109--126},
  url = {https://www.usenix.org/conference/usenixsecurity20/presentation/votipka-understanding},
  publisher = {USENIX Association},
  month = aug,
  }


title: Stack Overflow Considered Helpful! Deep Learning Security Nudges Towards Stronger Cryptography
bibtex:
  @inproceedings {Fischer2019StackOverflow,
  author = {Felix Fischer and Huang Xiao and Ching-Yu Kao and Yannick Stachelscheid and Benjamin Johnson and Danial Razar and Paul Fawkesley and Nat Buckley and Konstantin B{\"o}ttinger and Paul Muntean and Jens Grossklags},
  title = {Stack Overflow Considered Helpful! Deep Learning Security Nudges Towards Stronger Cryptography},
  booktitle = {28th USENIX Security Symposium (USENIX Security 19)},
  year = {2019},
  isbn = {978-1-939133-06-9},
  address = {Santa Clara, CA},
  pages = {339--356},
  url = {https://www.usenix.org/conference/usenixsecurity19/presentation/fischer},
  publisher = {USENIX Association},
  month = aug,
  }

problem_description:
  The 5 (post-hoc) Chi-Squared Tests reported in this paper should probably have been Fisher's exact tests, since with only 27 participants, expected probabilities for some cells were probably below 5.


title:
  From Needs to Actions to Secure Apps? The Effect of Requirements and Developer Practices on App Security

bibtex:
  @inproceedings {Weir2020FromNeeds,
  author = {Charles Weir and Ben Hermann and Sascha Fahl},
  title = {From Needs to Actions to Secure Apps? The Effect of Requirements and Developer Practices on App Security},
  booktitle = {29th USENIX Security Symposium (USENIX Security 20)},
  year = {2020},
  isbn = {978-1-939133-17-5},
  pages = {289--305},
  url = {https://www.usenix.org/conference/usenixsecurity20/presentation/weir},
  publisher = {USENIX Association},
  month = aug,
  }


title: Security Developer Studies with GitHub Users: Exploring a Convenience Sample
bibtex:
  @inproceedings {Acar2017SecurityDeveloper,
  author = {Yasemin Acar and Christian Stransky and Dominik Wermke and Michelle L. Mazurek and Sascha Fahl},
  title = {Security Developer Studies with {GitHub} Users: Exploring a Convenience Sample},
  booktitle = {Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017)},
  year = {2017},
  isbn = {978-1-931971-39-3},
  address = {Santa Clara, CA},
  pages = {81--95},
  url = {https://www.usenix.org/conference/soups2017/technical-sessions/presentation/acar},
  publisher = {USENIX Association},
  month = jul,
  }


title: Please do not use !?_ or your License Plate Number: Analyzing Password Policies in German Companies
bibtex:
  @inproceedings {Gerlitz2021PleaseDo,
  author = {Eva Gerlitz and Maximilian H{\"a}ring and Matthew Smith},
  title = {Please do not use !?_ or your License Plate Number: Analyzing Password Policies in German Companies},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {17--36},
  url = {https://www.usenix.org/conference/soups2021/presentation/gerlitz},
  publisher = {USENIX Association},
  month = aug,
  }


title: Security in the Software Development Lifecycle
bibtex:
  @inproceedings {219394,
  author = {Hala Assal and Sonia Chiasson},
  title = {Security in the Software Development Lifecycle},
  booktitle = {Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018)},
  year = {2018},
  isbn = {978-1-939133-10-6},
  address = {Baltimore, MD},
  pages = {281--296},
  url = {https://www.usenix.org/conference/soups2018/presentation/assal},
  publisher = {USENIX Association},
  month = aug,
  }


title: Rethinking SSL Development in an Appified World
doi: 10.1145/2508859.2516655
bibtex:
  @inproceedings{10.1145/2508859.2516655,
  author = {Fahl, Sascha and Harbach, Marian and Perl, Henning and Koetter, Markus and Smith, Matthew},
  title = {Rethinking SSL Development in an Appified World},
  year = {2013},
  isbn = {9781450324779},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/2508859.2516655},
  doi = {10.1145/2508859.2516655},
  booktitle = {Proceedings of the 2013 ACM SIGSAC Conference on Computer &amp; Communications Security},
  pages = {49–60},
  numpages = {12},
  keywords = {ios, SSL, mitma, android, security, apps},
  location = {Berlin, Germany},
  series = {CCS '13}
  }


doi: 10.1145/3460120.3484763
title:
  The Effect of Google Search on Software Security:
  Unobtrusive Security Interventions via Content Re-ranking

bibtex:
  @inproceedings{10.1145/3460120.3484763,
  author = {Fischer, Felix and Stachelscheid, Yannick and Grossklags, Jens},
  title = {The Effect of Google Search on Software Security: Unobtrusive Security Interventions via Content Re-Ranking},
  year = {2021},
  isbn = {9781450384544},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3460120.3484763},
  doi = {10.1145/3460120.3484763},
  booktitle = {Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {3070–3084},
  numpages = {15},
  keywords = {content ranking, usable security, software development, web search},
  location = {Virtual Event, Republic of Korea},
  series = {CCS '21}
  }


title: "If HTTPS Were Secure, I Wouldn't Need 2FA" - End User and Administrator Mental Models of HTTPS
doi: 10.1109/SP.2019.00060
bibtex:
  @INPROCEEDINGS{8835228,  author={Krombholz, Katharina and Busse, Karoline and Pfeffer, Katharina and Smith, Matthew and von Zezschwitz, Emanuel},  booktitle={2019 IEEE Symposium on Security and Privacy (SP)},   title={"If HTTPS Were Secure, I Wouldn't Need 2FA" - End User and Administrator Mental Models of HTTPS},   year={2019},  volume={},  number={},  pages={246-263},  doi={10.1109/SP.2019.00060}}


title:
  “We Can’t Live Without Them!” App Developers’ Adoption of Ad Networks and Their Considerations of Consumer Risks

bibtex:
  @inproceedings {238289,
  author = {Abraham H. Mhaidli and Yixin Zou and Florian Schaub},
  title = {"We Can{\textquoteright}t Live Without {Them!}" App Developers{\textquoteright} Adoption of Ad Networks and Their Considerations of Consumer Risks},
  booktitle = {Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019)},
  year = {2019},
  isbn = {978-1-939133-05-2},
  address = {Santa Clara, CA},
  pages = {225--244},
  url = {https://www.usenix.org/conference/soups2019/presentation/mhaidli},
  publisher = {USENIX Association},
  month = aug,
  }


title: “Jumping Through Hoops”: Why do Java Developers Struggle With Cryptography APIs?
doi: 10.1145/2884781.2884790
bibtex:
  @inproceedings{10.1145/2884781.2884790,
  author = {Nadi, Sarah and Kr\"{u}ger, Stefan and Mezini, Mira and Bodden, Eric},
  title = {Jumping through Hoops: Why Do Java Developers Struggle with Cryptography APIs?},
  year = {2016},
  isbn = {9781450339001},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/2884781.2884790},
  doi = {10.1145/2884781.2884790},
  booktitle = {Proceedings of the 38th International Conference on Software Engineering},
  pages = {935–946},
  numpages = {12},
  keywords = {empirical software engineering, cryptography, API misuse},
  location = {Austin, Texas},
  series = {ICSE '16}
  }


title: A Qualitative Study of Dependency Management and Its Security Implications
doi: 10.1145/3372297.3417232
bibtex:
  @inbook{10.1145/3372297.3417232,
  author = {Pashchenko, Ivan and Vu, Duc-Ly and Massacci, Fabio},
  title = {A Qualitative Study of Dependency Management and Its Security Implications},
  year = {2020},
  isbn = {9781450370899},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3372297.3417232},
  booktitle = {Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {1513–1531},
  numpages = {19}
  }


title:
  Us and Them: A Study of Privacy Requirements Across
  North America, Asia, and Europe

doi: 10.1145/2568225.2568244
bibtex:
  @inproceedings{10.1145/2568225.2568244,
  author = {Sheth, Swapneel and Kaiser, Gail and Maalej, Walid},
  title = {Us and Them: A Study of Privacy Requirements across North America, Asia, and Europe},
  year = {2014},
  isbn = {9781450327565},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/2568225.2568244},
  doi = {10.1145/2568225.2568244},
  booktitle = {Proceedings of the 36th International Conference on Software Engineering},
  pages = {859–870},
  numpages = {12},
  keywords = {requirements engineering, user developer collaboration, interaction data, privacy, Human factors in software engineering, empirical studies},
  location = {Hyderabad, India},
  series = {ICSE 2014}
  }


title: Deciding on Personalized Ads: Nudging Developers About User Privacy
bibtex:
  @inproceedings {274471,
  author = {Mohammad Tahaei and Alisa Frik and Kami Vaniea},
  title = {Deciding on Personalized Ads: Nudging Developers About User Privacy},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {573--596},
  url = {https://www.usenix.org/conference/soups2021/presentation/tahaei},
  publisher = {USENIX Association},
  month = aug,
  }


title: Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes
doi: 10.1109/SP.2018.00003
bibtex:
  @INPROCEEDINGS{8418614,
    author={Votipka, Daniel and Stevens, Rock and Redmiles, Elissa and Hu, Jeremy and Mazurek, Michelle},
    booktitle={2018 IEEE Symposium on Security and Privacy (SP)}, 
    title={Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes}, 
    year={2018},
    volume={},
    number={},
    pages={374-391},
    doi={10.1109/SP.2018.00003}}


title: An Observational Investigation of Reverse Engineers’ Processes
bibtex:
  @inproceedings {247696,
  author = {Daniel Votipka and Seth Rabin and Kristopher Micinski and Jeffrey S. Foster and Michelle L. Mazurek},
  title = {An Observational Investigation of Reverse {Engineers{\textquoteright}} Processes},
  booktitle = {29th USENIX Security Symposium (USENIX Security 20)},
  year = {2020},
  isbn = {978-1-939133-17-5},
  pages = {1875--1892},
  url = {https://www.usenix.org/conference/usenixsecurity20/presentation/votipka-observational},
  publisher = {USENIX Association},
  month = aug,
  }


title: How to improve the security skills of mobile app developers? Comparing and contrasting expert views
bibtex:
  @inproceedings {197846,
  author = {Charles Weir and Awais Rashid and James Noble},
  title = {How to Improve the Security Skills of Mobile App Developers? Comparing and Contrasting Expert Views},
  booktitle = {Twelfth Symposium on Usable Privacy and Security (SOUPS 2016)},
  year = {2016},
  address = {Denver, CO},
  url = {https://www.usenix.org/conference/soups2016/workshop-program/wsiw16/presentation/weir},
  publisher = {USENIX Association},
  month = jun,
  }


title: Helping Johnny to Analyze Malware: A Usability-Optimized Decompiler and Malware Analysis User Study
doi: 10.1109/SP.2016.18
bibtex:
  @INPROCEEDINGS{7546501,
    author={Yakdan, Khaled and Dechand, Sergej and Gerhards-Padilla, Elmar and Smith, Matthew},
    booktitle={2016 IEEE Symposium on Security and Privacy (SP)}, 
    title={Helping Johnny to Analyze Malware: A Usability-Optimized Decompiler and Malware Analysis User Study}, 
    year={2016},
    volume={},
    number={},
    pages={158-177},
    doi={10.1109/SP.2016.18}}


title: Why Don’t Developers Detect Improper Input Validation? ’; DROP TABLE Papers; –
doi: 10.1109/ICSE43902.2021.00054
bibtex:
  @INPROCEEDINGS{9402102,  author={Braz, Larissa and Fregnan, Enrico and Çalikli, Gül and Bacchelli, Alberto},  booktitle={2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)},   title={Why Don’t Developers Detect Improper Input Validation? '; DROP TABLE Papers; --},   year={2021},  volume={},  number={},  pages={499-511},  doi={10.1109/ICSE43902.2021.00054}}


title: Virtual Classrooms and Real Harms: Remote Learning at U.S. Universities
bibtex:
  @inproceedings {274479,
  author = {Shaanan Cohney and Ross Teixeira and Anne Kohlbrenner and Arvind Narayanan and Mihir Kshirsagar and Yan Shvartzshnaider and Madelyn Sanfilippo},
  title = {Virtual Classrooms and Real Harms: Remote Learning at {U.S}. Universities},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {653--674},
  url = {https://www.usenix.org/conference/soups2021/presentation/cohney},
  publisher = {USENIX Association},
  month = aug,
  }


title: Keep Me Updated: An Empirical Study of Third-Party Library Updatability on Android
doi: 10.1145/3133956.3134059
bibtex:
  @inproceedings{10.1145/3133956.3134059,
  author = {Derr, Erik and Bugiel, Sven and Fahl, Sascha and Acar, Yasemin and Backes, Michael},
  title = {Keep Me Updated: An Empirical Study of Third-Party Library Updatability on Android},
  year = {2017},
  isbn = {9781450349468},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3133956.3134059},
  doi = {10.1145/3133956.3134059},
  booktitle = {Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {2187–2200},
  numpages = {14},
  keywords = {updatability, android, api, app security, third-party library},
  location = {Dallas, Texas, USA},
  series = {CCS '17}
  }


title: Benefits and Drawbacks of Adopting a Secure Programming Language: Rust as a Case Study
bibtex:
  @inproceedings {274473,
  author = {Kelsey R. Fulton and Anna Chan and Daniel Votipka and Michael Hicks and Michelle L. Mazurek},
  title = {Benefits and Drawbacks of Adopting a Secure Programming Language: Rust as a Case Study},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {597--616},
  url = {https://www.usenix.org/conference/soups2021/presentation/fulton},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  Developers Deserve Security Warnings, Too: On the Effect of Integrated Security Advice  on Cryptographic API Misuse

bibtex:
  @inproceedings {219441,
  author = {Peter Leo Gorski and Luigi Lo Iacono and Dominik Wermke and Christian Stransky and Sebastian M{\"o}ller and Yasemin Acar and Sascha Fahl},
  title = {Developers Deserve Security Warnings, Too: On the Effect of Integrated Security Advice on Cryptographic {API} Misuse},
  booktitle = {Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018)},
  year = {2018},
  isbn = {978-1-939133-10-6},
  address = {Baltimore, MD},
  pages = {265--281},
  url = {https://www.usenix.org/conference/soups2018/presentation/gorski},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  Why Do Developers Get Password Storage Wrong?
  A Qualitative Usability Study

doi: 10.1145/3133956.3134082
bibtex:
  @inproceedings{10.1145/3133956.3134082,
  author = {Naiakshina, Alena and Danilova, Anastasia and Tiefenau, Christian and Herzog, Marco and Dechand, Sergej and Smith, Matthew},
  title = {Why Do Developers Get Password Storage Wrong? A Qualitative Usability Study},
  year = {2017},
  isbn = {9781450349468},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3133956.3134082},
  doi = {10.1145/3133956.3134082},
  pages = {311–328},
  numpages = {18},
  keywords = {studies, priming, developer, password storage, methodology},
  location = {Dallas, Texas, USA},
  series = {CCS '17}
  }


title:
  Deception Task Design in Developer Password Studies:
  Exploring a Student Sample

bibtex:
  @inproceedings {219406,
  author = {Alena Naiakshina and Anastasia Danilova and Christian Tiefenau and Matthew Smith},
  title = {Deception Task Design in Developer Password Studies: Exploring a Student Sample},
  booktitle = {Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018)},
  year = {2018},
  isbn = {978-1-939133-10-6},
  address = {Baltimore, MD},
  pages = {297--313},
  url = {https://www.usenix.org/conference/soups2018/presentation/naiakshina},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  To Pin or Not to Pin—Helping App Developers
  Bullet Proof Their TLS Connections

bibtex:
  @inproceedings {190898,
  author = {Marten Oltrogge and Yasemin Acar and Sergej Dechand and Matthew Smith and Sascha Fahl},
  title = {To Pin or Not to {Pin{\textemdash}Helping} App Developers Bullet Proof Their {TLS} Connections},
  booktitle = {24th USENIX Security Symposium (USENIX Security 15)},
  year = {2015},
  isbn = {978-1-939133-11-3},
  address = {Washington, D.C.},
  pages = {239--254},
  url = {https://www.usenix.org/conference/usenixsecurity15/technical-sessions/presentation/oltrogge},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  An Ethnographic Understanding of Software  (In)Security and a Co-Creation Model to Improve Secure Software Development

bibtex:
  @inproceedings {255680,
  author = {Hernan Palombo and Armin Ziaie Tabari and Daniel Lende and Jay Ligatti and Xinming Ou},
  title = {An Ethnographic Understanding of Software ({In)Security} and a {Co-Creation} Model to Improve Secure Software Development},
  booktitle = {Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020)},
  year = {2020},
  isbn = {978-1-939133-16-8},
  pages = {205--220},
  url = {https://www.usenix.org/conference/soups2020/presentation/palombo},
  publisher = {USENIX Association},
  month = aug,
  }


title: Schrödinger’s Security: Opening the Box on App Developers’ Security Rationale
bibtex:
  @INPROCEEDINGS{9283987,  author={van der Linden, Dirk and Anthonysamy, Pauline and Nuseibeh, Bashar and Tun, Thein Than and Petre, Marian and Levine, Mark and Towse, John and Rashid, Awais},  booktitle={2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE)},   title={Schrödinger's Security: Opening the Box on App Developers' Security Rationale},   year={2020},  volume={},  number={},  pages={149-160},  doi={}}


title: An Inside Look into the Practice of Malware Analysis
doi: 10.1145/3460120.3484759
bibtex:
  @inproceedings{10.1145/3460120.3484759,
  author = {Yong Wong, Miuyin and Landen, Matthew and Antonakakis, Manos and Blough, Douglas M. and Redmiles, Elissa M. and Ahamad, Mustaque},
  title = {An Inside Look into the Practice of Malware Analysis},
  year = {2021},
  isbn = {9781450384544},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3460120.3484759},
  doi = {10.1145/3460120.3484759},
  booktitle = {Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {3053–3069},
  numpages = {17},
  keywords = {malware analysis, usable security},
  location = {Virtual Event, Republic of Korea},
  series = {CCS '21}
  }


title: Comparing the Usability of Cryptographic APIs
doi: 10.1109/SP.2017.52
bibtex:
  @INPROCEEDINGS{7958576,  author={Acar, Yasemin and Backes, Michael and Fahl, Sascha and Garfinkel, Simson and Kim, Doowon and Mazurek, Michelle L. and Stransky, Christian},  booktitle={2017 IEEE Symposium on Security and Privacy (SP)},   title={Comparing the Usability of Cryptographic APIs},   year={2017},  volume={},  number={},  pages={154-171},  doi={10.1109/SP.2017.52}}


title: Replication: Effects of Media on the Mental Models of Technical Users
bibtex:
  @inproceedings {274425,
  author = {Khadija Baig and Elisa Kazan and Kalpana Hundlani and Sana Maqsood and Sonia Chiasson},
  title = {Replication: Effects of Media on the Mental Models of Technical Users},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {119--138},
  url = {https://www.usenix.org/conference/soups2021/presentation/baig},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  Replication: On the Ecological Validity
  of Online Security Developer Studies: 
  Exploring Deception in a Password-Storage 
  Study with Freelancers

bibtex:
  @inproceedings {255692,
  author = {Anastasia Danilova and Alena Naiakshina and Johanna Deuter and Matthew Smith},
  title = {Replication: On the Ecological Validity of Online Security Developer Studies: Exploring Deception in a {Password-Storage} Study with Freelancers},
  booktitle = {Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020)},
  year = {2020},
  isbn = {978-1-939133-16-8},
  pages = {165--183},
  url = {https://www.usenix.org/conference/soups2020/presentation/danilova},
  publisher = {USENIX Association},
  month = aug,
  }


title: Investigating System Operators’ Perspective on Security Misconfigurations
doi: 10.1145/3243734.3243794
bibtex:
  @inproceedings{10.1145/3243734.3243794,
  author = {Dietrich, Constanze and Krombholz, Katharina and Borgolte, Kevin and Fiebig, Tobias},
  title = {Investigating System Operators' Perspective on Security Misconfigurations},
  year = {2018},
  isbn = {9781450356930},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3243734.3243794},
  doi = {10.1145/3243734.3243794},
  booktitle = {Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {1272–1289},
  numpages = {18},
  keywords = {security, system operations, misconfigurations, vulnerabilities, administrators, operators, human factors, computer systems},
  location = {Toronto, Canada},
  series = {CCS '18}
  }


title: “Do this! Do that!, And nothing will happen” Do specifications lead to securely stored passwords?
doi: 10.1109/ICSE43902.2021.00053
bibtex:
  @INPROCEEDINGS{9402024,  author={Hallett, Joseph and Patnaik, Nikhil and Shreeve, Benjamin and Rashid, Awais},  booktitle={2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)},   title={“Do this! Do that!, and Nothing will Happen” Do Specifications Lead to Securely Stored Passwords?},   year={2021},  volume={},  number={},  pages={486-498},  doi={10.1109/ICSE43902.2021.00053}}


title: Programming Experience Might Not Help in Comprehending Obfuscated Source Code Efficiently
bibtex:
  @inproceedings {219439,
  author = {Norman H{\"a}nsch and Andrea Schankin and Mykolai Protsenko and Felix Freiling and Zinaida Benenson},
  title = {Programming Experience Might Not Help in Comprehending Obfuscated Source Code Efficiently},
  booktitle = {Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018)},
  year = {2018},
  isbn = {978-1-939133-10-6},
  address = {Baltimore, MD},
  pages = {341--356},
  url = {https://www.usenix.org/conference/soups2018/presentation/hansch},
  publisher = {USENIX Association},
  month = aug,
  }


title: “I Have No Idea What I’m Doing” - On the Usability of Deploying HTTPS
bibtex:
  @inproceedings {203690,
  author = {Katharina Krombholz and Wilfried Mayer and Martin Schmiedecker and Edgar Weippl},
  title = {"I Have No Idea What I{\textquoteright}m Doing" - On the Usability of Deploying {HTTPS}},
  booktitle = {26th USENIX Security Symposium (USENIX Security 17)},
  year = {2017},
  isbn = {978-1-931971-40-9},
  address = {Vancouver, BC},
  pages = {1339--1356},
  url = {https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/krombholz},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  A Qualitative Usability Evaluation of the Clang Static Analyzer and libFuzzer with CS Students and CTF Players

bibtex:
  @inproceedings {274469,
  author = {Stephan Pl{\"o}ger and Mischa Meier and Matthew Smith},
  title = {A Qualitative Usability Evaluation of the Clang Static Analyzer and {libFuzzer} with {CS} Students and {CTF} Players},
  booktitle = {Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021)},
  year = {2021},
  isbn = {978-1-939133-25-0},
  pages = {553--572},
  url = {https://www.usenix.org/conference/soups2021/presentation/ploger},
  publisher = {USENIX Association},
  month = aug,
  }


title:
  A Usability Evaluation of Let’s Encrypt and Certbot:
  Usable Security Done Right

doi: 10.1145/3319535.3363220
bibtex:
  @inproceedings{10.1145/3319535.3363220,
  author = {Tiefenau, Christian and von Zezschwitz, Emanuel and H\"{a}ring, Maximilian and Krombholz, Katharina and Smith, Matthew},
  title = {A Usability Evaluation of Let's Encrypt and Certbot: Usable Security Done Right},
  year = {2019},
  isbn = {9781450367479},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3319535.3363220},
  doi = {10.1145/3319535.3363220},
  booktitle = {Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security},
  pages = {1971–1988},
  numpages = {18},
  keywords = {HTTPs, usable security, let's encrypt},
  location = {London, United Kingdom},
  series = {CCS '19}
  }


title: Does Organizing Security Patterns Focus Architectural Choices?
doi: 10.1109/ICSE.2012.6227155
bibtex:
  @INPROCEEDINGS{6227155,  author={Yskout, Koen and Scandariato, Riccardo and Joosen, Wouter},  booktitle={2012 34th International Conference on Software Engineering (ICSE)},   title={Does organizing security patterns focus architectural choices?},   year={2012},  volume={},  number={},  pages={617-627},  doi={10.1109/ICSE.2012.6227155}}


title: Do Security Patterns Really Help Designers?
doi: 10.1109/ICSE.2015.49
bibtex:
  @INPROCEEDINGS{7194582,  author={Yskout, Koen and Scandariato, Riccardo and Joosen, Wouter},  booktitle={2015 IEEE/ACM 37th IEEE International Conference on Software Engineering},   title={Do Security Patterns Really Help Designers?},   year={2015},  volume={1},  number={},  pages={292-302},  doi={10.1109/ICSE.2015.49}}


title: API Blindspots: Why Experienced Developers Write Vulnerable Code
bibtex:
  @inproceedings {219422,
  author = {Daniela Seabra Oliveira and Tian Lin and Muhammad Sajidur Rahman and Rad Akefirad and Donovan Ellis and Eliany Perez and Rahul Bobhate and Lois A. DeLong and Justin Cappos and Yuriy Brun},
  title = {{API} Blindspots: Why Experienced Developers Write Vulnerable Code},
  booktitle = {Fourteenth Symposium on Usable Privacy and Security (SOUPS 2018)},
  year = {2018},
  isbn = {978-1-939133-10-6},
  address = {Baltimore, MD},
  pages = {315--328},
  url = {https://www.usenix.org/conference/soups2018/presentation/oliveira},
  publisher = {USENIX Association},
  month = aug,
  }


# rpl_participantsampletype

name: security experts
problem_description:
  Details on participants: employees of organizations that develop cryptographic products (in Haney 2018)


name: End-users
problem_description:
  I am supposed to only consider experts. However, I included end-users since they were compared to administrators in the respective study...


# rpl_pvalue

associated_symbol: =
problem_description:
  unsure whether this is corrected, because authors just generally state that they correct where necessary


# rpl_qualitativeanalysismethod

analysis_method: Constructivist grounded theory method of Charmaz
analysis_method_further_description:
  "The interview and focus group transcriptions were evaluated independently and iteratively by two researchers using the constructivist grounded theory method of Charmaz [16, 19, 20, 22]." (Section 3.3)

  "The full GT process including themes and the codebook with examples from the interviews can be found in the supplementary material.
  In order to obtain the inter-coder agreement, we followed the approach of [18]. The coding schemes were compared, and inter-coder agreement was calculated using Cohen’s kappa coefficient (κ) [24]. The agreement measured 0.81" (Section 3.3)


analysis_method: open coding, thematic analysis, axial coding
problem_description: multiple measures of krippendorffs alpha, reported maximum of 0.92
analysis_method_further_description:
  "We unitized [ 10] the transcript and conducted open coding according to Strauss and Corbin [51] to analyze the data. For the analysis of the drawing and coding task, we additionally used the screen recordings to ensure no information is missed. In total two coders (the core authors) were involved in the coding process and construction of the codebook. The first coder constructed an initial version of the codebook, taking into account two interview transcripts. Based on the initial codebook, both coders coded all interviews, resolving issues and adjusting the codebook accordingly after each iteration. We continued with the coding procedure until both coders agreed that saturation was reached. In our case, this meant no new concepts emerged from the newest two interviews. We calculated the intercoder reliability of the different codebook versions before resolving issues. With the saturated version of the codebook, we re-coded all interviews to ensure that no information was missed during the initial coding." (Section 3.6)

  The inter-coder reliability Krippendorff’s alpha [ 29 ] was between 0.71 and 0.92 for each version of the code-book.


analysis_method: inductive coding
analysis_method_further_description:
  "We analyzed the qualitative data from the open-ended ques­tions in the survey through inductive coding [45]. Two re­searchers independently searched for codes and categories emerging from the raw data. After the coding process was completed, the codes were compared and inter-coder agree­ment by using Cohen’s kappa coefficient (K) [14] was calcu­lated. The agreement was 0.83."


analysis_method: general inductive approach
analysis_method_further_description:
  "Our data analysis utilized the general inductive approach [30], augmented by specific techniques for qualitative data analysis such as analytic notes and comparative analysis [6]. The researcher maintained descriptive field notes on daily activities and interactions. After three months of data collection, the research team met weekly to reflect on the observations made so far. The team went over the events of the past week and discussed the events concerning software development practices, security practices, and the relevant interactions."  

  "The researcher coded the raw field data based on the patterns and themes that emerged during these discussions."

  "The weekly iteration of data collections followed by in-depth discussions led to the refinement of the emerging categories used for coding [...]. Then, as broader themes were conceptualized, the researcher started to write analytic notes summarizing each theme and documenting ideas and analysis of each along with supporting data from exploration of code, tickets, and other relevant sources in addition to the raw fieldwork data." 

  "After the end of the fieldwork, the research team continued further analysis of data through extensive discussions to draw out the major implications of our observations to secure software development practices." (Section 2.2)


analysis_method: not specified
analysis_method_further_description:
  Not very much detail: 
  "As part of the data gathered, we also have the entire program produced during the build-it phase as well as the programs patched during the fix-it phase. We can then perform a qualitative analysis of the programs which is guided by knowing the security outcome of a given program." (Section 5)


analysis_method: inductive coding
analysis_method_further_description:
  "We used inductive coding [32] to analyze our qualitative data from the open questions in the survey as well as the chat interactions during development. Two researchers independently searched for themes and categories emerging in the raw data. After coding open questions of a new participant, both researchers went back to their codes, analyzing them again for the prior participants as well. After the coding process was completed, the codes were compared and the inter-coder agreement by using the Cohen’s kappa coefficient [9] was calculated." (Section 4 - Qualitative Analysis)


analysis_method: Not named, includes open coding
analysis_method_further_description:
  "The codebook was created by multiple researchers, reflecting different experiences and backgrounds which include formal education in media technology, mathematics and computer science; all authors have research experience in security, privacy and human factors research. We used the coding for grouping the FGs’ contents. Beginning with open coding, the first author analyzed and coded each contribution to the discussions. In this inductive process, he considered (1) exhaustiveness to ensure codings will reflect all main topics when assigned to higher-level semantic groups. He also adhered to (2) mutual exclusiveness of semantic groups to keep codes unambiguous. Paying attention to completeness, a total number of 210 subcodes of 21 higher level code groups were operationalized following a group discussion between the authors. Two additional coders with computer science background and minor HCI knowledge were introduced to the codebook in a one hour briefing, following which they independently coded FG number one (about 25% of all transcripts, exceeding recommendations for double-coding in qualitative research [19]) applying the codebook. The intention was to use inter-rater agreement scores to identify ambiguous codes or codings that were forgotten by the main coder. While the granular codebook was difficult for the additional coders to apply, mainly because they were inexperienced in the topic and in qualitative coding and due to the high number of codes, the following comparison and discussion resulted in a more precise operationalization of the codebook and confirmed the main coder’s coding decisions."


analysis_method: inductive coding
problem_description: multiple agreement stats reported, entered the final one
analysis_method_further_description:
  "We coded open answers inductively following the approach of Wertz, Charmaz et al. [66]. The two researchers categorized the data according to the research questions presented in Section 3. The first three interviews were coded in a batch to establish the first codebook. Each of the following four interviews was coded separately. Then, the conflicts were discussed, and new codes were added to the codebook. We calculated the combined Krippendorff’s alpha [36] before (0.61) and after (0.98) the discussion phase for each interview." (Section 3.3)

  "Our goal was to use the qualitative analysis solely as a first step and foundation for the following quantitative study. Therefore, we refrained from continuing with interviews until theoretical saturation [23] was reached." (Section 3.3)


analysis_method: inductive content analysis, independent parallel coding
problem_description: not totally clear that there were two coders, but there were two sets of categories
analysis_method_further_description:
  [14] in paper
  "We decided upon an inductive coding method, as opposed to a deductive coding method, as we did not want to assume what our results would be. The evaluation process included open coding and creating categories. Since we used the “independent parallel coding” approach of David R. Thomas [35], we compared two sets of categories and report the inter-coder agreement for them. The two sets of categories were subsequently merged into a combined set." (Section 3.4.2)


analysis_method: deductive and inductive coding practices (no defined method)
analysis_method_further_description:
  "We utilized both deductive and inductive coding practices. Initially we constructed an a priori code list based on our research questions and literature in the field to provide direction in the analysis. As we performed multiple rounds of coding, we also identified emergent codes in the data. This iterative, recursive process helped us identify additional codes and categories as we worked with the data until we reached saturation [26, 69]. 
  Five interviews (almost 24%) were first coded individually, then discussed as a group to develop a codebook. Although there is debate on the amount of text to collectively sample in qualitative research, the amount of text we group coded far exceeds the minimum of 10% often cited as standard practice [33].
  [...]
  After coding of the initial subset of data, the remaining 16 interviews were coded by two coders each. Once each pair completed their coding, they had a discussion about the data to address areas of divergence about their use and application of the codes. This discussion resulted in the coders being able to understand each other’s perspectives and come to a final coding determination. New codes that were identified during these discussions were added to the codebook, with previously coded interviews then re-examined to account for additions.
  [...]
  During the coding phase, we also engaged in writing analytic memos to capture thoughts about emerging themes [13]. For example, one memo captured thoughts on cryptography complexity. Once coding was complete, we reorganized and reassembled the data, created coding arrays, discussed patterns and categories, drew models, discussed relationships in codes and data, and began to move from codes to themes" (Section 3.3)

  Coder background described in Section 3

problem_description:
  Multiple agreement values for different interviews, average: Krippendorfs alpha = 0.7, maximum Krippendorfs alpha = 0.78 (maximum agreement below)

  Unclear number of coders

  based on "For example, two of our coders more often identified high-level, nuanced codes about emotions and personal values, which may be due to their many years of working in human-focused contexts. These interpretations were often missed by the other coders" (Section 3.3), I think all four authors were involved in the coding, so this is what I entered below


analysis_method: iterative open coding
problem_description: Agreement reported per variable, between 0.80 and 1, maximum agreement entered for Agreement value
analysis_method_further_description:
  "To develop our codebooks, two researchers first cooperatively examined 11 projects. For each, they reviewed associated breaks and independently audited the project for vulnerabilities. They met and discussed their reviews (totaling 42 vulnerabilities) to establish the initial codebook. 
  At this point, one of the two original researchers and a third researcher independently coded breaks in rounds of approximately 30 each, and again independently audited projects’ unidentified vulnerabilities. After each round, the researchers met, discussed cases where their codes differed, reached a consensus, and updated the codebook. This process continued until a reasonable level of inter-rater reliability was reached for each variable. [...] 
  Once a reliable codebook was established, the remaining 34 projects (with 166 associated breaks) were divided evenly among the two researchers and coded separately. " 

  "The final Krippendorff’s alpha for each variable is given in Table 1."
  (Section 3.2.2)


analysis_method: Open coding
analysis_method_further_description:
  "To analyze the PCPs, we used open coding as described by Corbin et al. [19]. Even though the policies mainly were enumerations of several elements and did not allow much room for interpretation (with few exceptions like “no easy passwords”), two researchers independently coded the policies to reduce errors. As suggested by Campbell et al. [16], we developed a code book by separately coding a small set of policies (n = 10) and comparing the codes. This then served as a base for future codes. After coding all the remaining policies independently, the codes were compared, and the inter-coder agreement was calculated using Cohen’s kappa coefficient (k) [18]. Our agreement was 81.48. A value above 0.75 is considered a good level of coding agreement [20]. We were able to resolve all conflicts." (Section 3.5)


analysis_method: Qualitative Content Analysis
analysis_method_further_description:
  One researcher coded all interviews. 30% of the data were also coded by independent researchers to test reliability. 
  Deductive and inductive methods of content analysis. The deductive method was used to structure the analysis according to the different development stages. An initial analysis matrix of the main SDLC stages was refined after a first analysis of the data.

  "Data was analyzed using the Qualitative Content Analysis methodology [9,23]. It can be deductive, inductive, or a combination thereof. For the deductive approach, the researcher uses her knowledge of the subject to build an analysis matrix and codes data using this matrix [9]. The inductive method, used when there is no existing knowledge of the topic, includes open coding, identifying categories, and abstraction [9]. We employed both the deductive and inductive methods of content analysis. The deductive method was used to structure our analysis according to the different development stages. We built an initial analysis matrix of the main SDLC stages [49]. After a preliminary stage of categorizing interview data and discussions between the researchers, the matrix was refined."

  "We then followed an inductive analysis method to explore practices and behaviours within each category (development stage) as recommended by the content analysis methodology. We performed open coding of the excerpts where we looked for interesting themes and common patterns in the data. This resulted in 96 codes. Next, data and concepts that belonged together were grouped, forming sub-categories. Further abstraction of the data was performed by grouping sub-categories into generic categories, and those into main categories. The abstraction process was repeated for each stage of development." (Section 3)

  "While all coding was done by a single researcher, two researchers met regularly to thoroughly and collaboratively review and edit codes, and group and interpret the data. To verify the reliability of our coding, we followed best practices by inviting a researcher who has not been involved with the project to act as a second coder, individually coding 30% of the data. We calculated Krippendorff’s alpha [33] to assess inter-rater reliability, and α = 0.89 (percentage of agreement = 91%). According to Krippendorff [34], alpha ≥ 0.80 indicates that coding is highly reliable and that data is “similarly interpretable by researchers”. In case of disagreements, we had a discussion and came to an agreement on the codes."


analysis_method: Not clearly specified
analysis_method_further_description:
  Results: Purely descriptive. Statements were cited and percentages were calculated, e.g. percentage of participants who recognise the problem. Or the number of participants who fixed the problem after the interview.


analysis_method: inductive coding, open coding, descriptive axial coding, selective coding
analysis_method_further_description:
  inductive coding

  "We applied two rounds of open coding to detect observable patterns. We then performed Strauss and Corbin’s descriptive axial coding [45] and selective coding to group our data into categories and models. We also used selective coding to relate the categories to our research questions. Throughout the coding process, we used analytic memos to keep track of thoughts about emerging themes."

  "As a first step, three researchers independently coded all questions and drawings of mental models. Subsequently, the resulting codes were discussed and refined to agree on a final code book. As a second step, two coders independently coded the data and again conflicts were resolved in discussions." (Section III E)


analysis_method: Descriptive coding
analysis_method_further_description:
  "Interviews were transcribed and then analyzed with descriptive coding [53]. Two of the authors developed an initial codebook by jointly reading the transcripts and identifying emergent themes. They then iteratively refined the codebook by separately coding an interview, determining inter-rater reliability, revising the codebook as needed, and repeating with a separate interview. This procedure was repeated for 5 interviews until high inter-rater reliability was reached (Cohen’s κ=.75) [53]. One of the authors then re-coded all interviews with the final codebook." (Section 4.2)


analysis_method: not clearly specified
analysis_method_further_description:
  "We grouped similar free-text answers provided by participants, resulting in nine categories" (Section 3.2)


analysis_method: Applied thematic analysis
analysis_method_further_description:
  Started with open coding, then selective coding and code groups. Iterating this process with new interviewees until saturation is accomplished.

  To validate observations and implications, the authors shared a one-page summary of this study, along with the full version of the paper with the interviewees. They asked them to validate if the results correspond to their expectations.

  "To analyse the interviews, we have adopted the applied thematic analysis [14]. Figure 1 summarizes our approach. It follows the principle of emergence [13], according to which data gain their relevance in the analysis through a systematic generation and iterative conceptualization of codes, concepts, and code groups. Data is analyzed, broken into manageable pieces (codes), and compared for similarities and differences. Similar concepts are grouped under the same conceptual heading (code group). Code groups are composed in terms of their properties and dimensions, and finally, they provide the structure of the analysis [40]. 

  The first phase of analysis (open coding) consists of collecting the critical point statements from each interviewee transcript; a code summarizing the key points in a few words is assigned to each key point statement. The interviewees are numbered #1 to #25. Two researchers independently followed the “iterative process” described by Saldaña [37] to code the transcribed interviews. Then they looked together at the resulting codes and agreed on the common code structure, which was reviewed by a third researcher not involved in the preliminary coding process. So after each iteration, we had a complete agreement on the codes and code groups by the three researchers. Each time we reviewed the resulted codes, we have also performed a check whether we have achieved a saturation of the reported observations [30], i.e., if the interviewees discuss the same concepts. After concluding that saturation is achieved, we interviewed additional developers to control the stability of our observations (Additional confirmation step in Figure 1).

  We started the coding process as soon as we had ten interviews. At first, we created 345 quotations and assigned 138 codes to them. During the first six iterations, we were consolidating both quotations and codes by looking at quotations and merging codes on close topics. This resulted in 151 quotations with 28 codes assigned to them. On the next stages, we have added 15 more interviews, which significantly enlarged the number of quotations (533 quotations on the 11th iteration). While adding them, we realized that there was one irrelevant code (Scala), so we deleted it. Hence, there were 27 codes on the 10th iteration. Then we have added quotations and codes for the developer roles (SME, LE, FOSS, or UG developer), which resulted in 31 codes and 574 quotations on the 11th iteration. On the last step of the coding process, we have added the codes corresponding to the interview process parts. Hence, we have ended up with four codes that correspond to developer roles, six codes for interview process parts, and 27 codes for developer answers assigned to 829 interview quotes. 

  To validate our observations and implications, we shared the one-page summary of this study, along with the full version of the paper with the interviewees. We asked them to validate if the results correspond to their expectations (last step in Figure 1)." 
  (Section 4.3)


analysis_method: iterative open coding, axial coding
problem_description: multiple alpha values reported, final value here
analysis_method_further_description:
  "The interviewer and another researcher independently coded each interview, building the codebook incrementally and re-coding previously coded interviews. This process was repeated until all interviews were coded. The codes of the two interviewers were then compared to determine intercoder reliability using the ReCal2 software package [92]. We use Krippendorff’s Alpha (α) to measure inter-coder reliability as it accounts for chance agreements [93]."

  "To achieve more conclusive results, we recoded the 16 of our 85 codes with an α less than .70. For each code, the coders discussed a subset of the disagreements, adjusted code definitions as necessary to clarify inclusion/exclusion conditions, and re-coded all the interviews with the updated codebook. After re-coding, the α for the study was .85. Additionally, all individual codes’ αs were above .70."

  "Next, we grouped the identified codes into related categories."
  "We then performed an axial coding to find connections between categories and between codes within categories [91, pg. 123-142]. Based on the categories and connections between them, we derive a theory"
  (Section III C)


analysis_method: iterative open and axial coding
analysis_method_further_description:
  iterative coding was repeated four times until alpha of 0.8 was achieved. axial coding was done subsequently to determine relationships

  "We applied iterative open coding to identify interview themes [66, pg. 101-122]. After completing each interview, the audio was sent to an external transcription service. The interviewer and another researcher first collaboratively coded three interviews—reviewing both the text and video—to create an initial codebook2. Then, the two coders independently coded 13 interviews, comparing codes after every three interviews to determine inter-coder reliability. To measure intercoder reliability, we used Krippendorff’s Alpha (α), as it accounts for chance agreements [67].3 After each round, the coders resolved any differences, updated the codebook as necessary, and re-coded previously coded interviews. The coders repeated this process four times until they achieved an α of 0.8, which is above the recommended level for exploratory studies [67, 69]. 
  Next, we sought to develop our theoretical model by extracting themes from the coded data. First, we grouped identified codes into related categories. Specifically, we discovered three categories associated with the phases of analyses performed by REs (i.e., Overview, Sub-component Scanning, and Focused Experimentation)."
  "Then, we performed an axial coding to determine relationships between and within each phase and trends across the three phases [66, pg. 123-142]. "
  (Section


analysis_method: Grounded Theory and Appreciative Inquiry
analysis_method_further_description:
  no more details than 
  "and analyzed them using Grounded Theory [3, 9]" and "Thus the nature of the questions and the thrust of the analysis were guided by a further research method, Appreciative Inquiry [5]." (Section 3)


analysis_method: iterative open coding
analysis_method_further_description:
  "Once the interviews were complete, two team members transcribed the audio recordings and then analyzed them using iterative open coding [8]. The interviewer and the other team member independently coded the interviews one at a time, developing the codebook incrementally and resolving disagreements after every transcript. This process continued until a reasonable level of inter-rater reliability was reached measured with the Krippendorff’s α statistic [22]. After seven interviews, the two researchers achieved a Krippendorff’s α = 0.80, calculated using ReCal2 [11]. This level of agreement is above the commonly recommended thresholds of 0.667 [18] or 0.70 [23] for exploratory research and meets the more general minimum threshold recommended by Krippendorff of 0.8 [22]. Once a reliable codebook was established, the remaining nine interviews were evenly divided among the two researchers and coded separately." (Section 3.2)


analysis_method: deductive
analysis_method_further_description:
  Open questions coded using the results form Study 1.

  "Free response questions from the survey were analyzed by one researcher using the same codebook from the interview. When new codes were added to the codebook, they were back-applied to the interviews."


analysis_method: security scoring evaluation
analysis_method_further_description:
  Security judgement is used for quantitative analysis.

  "We assigned security scores only to those solutions which were graded as functional. To determine a security score, we considered several different security parameters. Our scoring followed the relevant parts of the security scoring in [1]. Still we give a brief summary of the security scoring we applied." (Section 6.6)

  "We calculated Krippendorff’ alpha [28] for the initial coding by two coders across all security codes; α = 0.764, which is within reasonable bounds for agreement [14]. Conflicts were resolved afterwards." (Section 6.6)


analysis_method: Grounded Theory
analysis_method_further_description:
  "Two coders independently coded the participants’ answers by using Grounded-Theory and compared their final code books using the inter-coder agreement. The Cohen’s kappa coefficient (κ) [18] for all themes was 0.78."


analysis_method: not clearly specified
analysis_method_further_description:
  Description and informal categorization of responses to report shares of opinions

  "After we analyzed the 45 responses, we had gained insight into the major problem areas, a saturation of new input was reached and more responses would probably not have provided more valuable insights."


analysis_method: general inductive approach, techniques from grounded theory
analysis_method_further_description:
  two embedded researchers coded, but each their own data

  "Coding – including the development of specific codes and of a codebook – proceeded in an iterative fashion. Weekly meetings facilitated the discussion of emerging research themes and specific examples. Codes emerged from these discussions, where the researchers built consensus on specific analyses. The researchers relied on a general inductive approach [16], as well as techniques derived from grounded theory and related approaches for doing qualitative analysis [17]. Overall, the development of codes initially focused on the Silently Allow example, then on emerging results from penetration testing, the development of a specific coding system for each researcher for their field notes, and a final collaborative phase to find commonalities in codes for both the specific examples and overall corpus of field notes." 

  "Subsequent research focused on developing a full set of codes by each of the student researchers. Because the two embedded researchers often worked on different projects and at different times, each wrote their own field notes and then subsequently engaged in coding of their own notes." 
  (Appendix)


analysis_method: open coding
problem_description: multiple Agreement values reported for different tasks
analysis_method_further_description:
  "Rationale analysis. To assess whether the tasks were performed with a security rationale, and whether this correlated to their solution (RQ2), we analyzed the rationale elicited from participants after each task using open coding1. Two authors independently coded each result to determine whether the rationale indicated a focus on security. Inter-rater reliability measure (Cohen’s Kappa [23]) showed strong to perfect inter-rater agreement and reliability across all tasks (ide k = 0.91, code k = 0.91, help k = 1, testing k = 0.85, ads k = 0.87, legal k = 1)."


analysis_method: iterative open coding
analysis_method_further_description:
  "The interviews were recorded and then manually transcribed by the interviewer and analyzed using an iterative open coding methodology [72]. The interviewer, along with another co-author, developed a codebook (Appendix D) based on an independent review of four interviews. Subsequently, the two co-authors independently coded each interview using this codebook. The coders achieved "excellent" Krippendorff’s alpha intercoder reliability of 0.823 [57]. Reliability was used to ensure consistency in the interpretation of conceptual themes and workflows across the research team."
  (Section 2.3)


analysis_method: inductive thematic analysis
analysis_method_further_description:
  "Interview data was analysed using inductive thematic analysis [11]. We iteratively analyzed and created the codebook. Each transcript was reviewed multiple times, ensuring that every transcript was coded with the final codebook. 
  The first author conducted the interviews and was involved in all stages of the coding process. A second researcher helped code part of the data, and then left the project. A third researcher coded the remaining data and re-coded some prior data. The detailed process unfolded as follows (see Appendix B). While editing the transcripts, the first author noted any initial themes occurring in the data. These 9 initial themes were then used as the basis for the first codebook. The first author coded two transcripts with this set of codes, then iteratively revised the themes until no new codes emerged, resulting in a second codebook consisting of 16 high-level themes and 87 sub-codes. This second codebook was used in the initial thematic analysis of the first ten transcripts by the first author. 
  The first author and a second researcher then coded three new transcripts together, and revised the codebook, resulting in a third codebook. The researchers then re-coded the 3 transcripts individually, as well as 3 additional transcripts, upon which a fourth and final codebook with 17 high-level codes and 95 sub-codes emerged. The percentage agreement between the two researchers for the codes on the overlapping transcripts was 98.67% overall. 
  A third researcher then continued the analysis process alongside the first author. The first author and the third researcher coded 3 transcripts individually using the fourth codebook, and met to clarify any misconceptions. No further revisions to the codebook were needed. Agreement between the two coders on the overlapping transcripts was 99%. All remaining transcripts were analysed or revised using the fourth codebook: the third researcher independently coded an additional 7 transcripts, and revised the codes for 10 transcripts. The first author also coded a further 3 transcripts." (Section 3.5)


analysis_method: inductive coding
analysis_method_further_description:
  "The open-ended questions from the follow-up survey were analyzed by two researchers using inductive coding [41]. The two researchers independently searched for codes categories and themes emerging in the raw data. The codes were compared and the inter-coder agreement was calculated by using the Cohen’s kappa coefficient (κ) [14]. The agreement was 0.83."


analysis_method: inductive coding, Strauss and Corbin’s descriptive, axial coding
problem_description: unclear number of coders: two? one?
analysis_method_further_description:
  "In a first round, we analyze the anonymized interviews with QDA Miner [18], and gradually generate categories while keeping the underlying codes rather specific to not generalize them prematurely. Subsequently, we perform inductive coding [1924], which is commonly used to construct models and theories based on qualitative data in social sciences and usable security [9, 25]. 
  We then use Strauss and Corbin’s descriptive, axial coding [23] and selective coding to group our data into categories and models. We first code, then iteratively refine our research questions, with an emphasis on the different stages of coding. 
  We use one coder to construct the code book, which is deemed acceptable in social sciences, especially when the analysis is interpretative and exploratory [26]. An additional researcher, who had not been involved in the data collection, then uses the codebook to assess the frequency of codes in the interviews."
  (Section 3.2)


analysis_method: Scoring implementations according to given criteria
analysis_method_further_description:
  "Scoring implementations according to Naiakshina’s criteria could introduce subjectivity. To mitigate this, one author scored and then another author independently rescored all the implementations and calculated Cohen’s Kappa (a measure of inter-rater reliability [48]). The kappa-value indicates almost perfect agreement (κ = 0.94) [48]." (Section III D)


analysis_method: grounded theory approach
analysis_method_further_description:
  topic: explanations

  "Similarly, our codebooks, whilst grounded in data, were likely influenced by the coder’s background and experiences. Using our codebooks a separate coder independently re-coded the entire dataset. We found substantial agreement (κ = 0.72) with our coding for developers’ explanations for their implementations (Table VI)" (Section III D)

  "After implementing their solutions we asked developers why they had used a particular approach. Two of the authors used a grounded theory approach [44] to analyze the responses. Two passes were required to reach the point of theoretical saturation [45] when no new codes were identified. The resulting codebook, and illustrative examples of each code, is shown in Table VI." (Section V)


analysis_method: grounded theory approach
analysis_method_further_description:
  topic: threats

  "Similarly, our codebooks, whilst grounded in data, were likely influenced by the coder’s background and experiences. Using our codebooks a separate coder independently re-coded the entire dataset. We found [...] almost perfect agreement (κ = 0.84) with our coding for the threats developers considered." (Section III D)

  "After implementing their solutions we asked developers why they had used a particular approach. Two of the authors used a grounded theory approach [44] to analyze the responses. Two passes were required to reach the point of theoretical saturation [45] when no new codes were identified. The resulting codebook, and illustrative examples of each code, is shown in Table VI." (Section V)


analysis_method: iterative open coding, axial coding, selective coding
analysis_method_further_description:
  "For a qualitative analysis of the observation protocols we performed a series of iterative coding which is often used in usable security research to develop models and theories from qualitative data [27, 34, 39]. Our approach involved several steps in the analysis process and was implemented as follows: At first, two researchers traversed all data segments independently point-by-point and assigned descriptive codes. This process is referred to as open coding. The two researchers performed the initial coding independently from each other to minimize the susceptibility of biased interpretation. We evaluated the quality of our initial codes and agreed on a final set of codes which was then used to code the protocols. Our analysis showed a good inter-rater agreement between the two coders (Cohen’s κ=0.78). On the resulting initial set of coded data we performed axial coding to look for explanations and relationships among the codes and topics to uncover structures in the data. Then we performed selective coding to put the results together and derive a theory from the data."

  "n order to structure the data from the open-ended questions collected through the questionnaire we used an iterative coding process. Hence we went through the collected data and produced an initial set of codes. Then we revised the retrieved codes and discussed recurring themes, patterns and interconnections. After agreeing on a final set of codes, we coded the entire data." 
  (Section 3.3)


analysis_method: inductive coding
analysis_method_further_description:
  "To analyze the interviews and diaries, we used inductive coding [75] with two researchers. The two researchers started with coding the same four randomly chosen interviews independently and in parallel. They compared, analyzed and discussed the two resulting coding sets. It turned out that due to the open approach, the code sets of both researchers were substantially different. Through a discussion of the codes a common coding set was agreed upon. The four interviews were then recoded and discussed again. This procedure was repeated in steps of three interviews. The diaries of the participants were coded with the resulting coding-set from the coding of the interviews. During the coding of the diaries, the coding-set was again supplemented by codes that emerged from the data."


analysis_method: not clearly specified
analysis_method_further_description:
  "Twenty-five participants used the channel and asked 52 questions. Because the categorization of these messages was critical for all other results, we followed a two-stage coding procedure: First, three coders independently coded all support interactions using the categories. We calculated an initial Fleiss’ kappa (0.5) and Krippendorff’s alpha (0.5) [29]. With three coders and eight categories, values in this range are to be expected. All codes with disagreement were discussed, and full agreement was reached in the second round of coding. For coding categories, see Table 8." (Section 5.4)


analysis_method: Unnamed
analysis_method_further_description:
  "We analyzed these comments manually in three steps. First, we read each comment and annotated it with a few keywords. Thereby, we tried to reuse the keywords whenever possible. Second, we unified and grouped the keywords into topics, making sure that no important comments are lost. Finally, we read the comments again and assigned each of them to the identified topics."


analysis_method: affinity diagrams
analysis_method_further_description:
  "We used the virtual collaboration platform Miro [72] to create separate boards for each open-ended question and posted virtual sticky notes with participants’ responses. During a half-day virtual session with five security and privacy researchers with a minimum Master’s degree in computer science, and one senior Android developer, we identified the common themes through group affinity diagram building."
  (Section 3.3.2)


analysis_method: thematic analysis
analysis_method_further_description:
  "We used thematic analysis [13] to further understand participants’ rationales (RQs 1–3), especially those that were not security related. Two authors independently coded the rationale data and compared codebooks, agreeing on dominant themes emerging from the analysis." (Section 3.3)


# rpl_resultstatistic

name: Amount
description:
  condition: payment: 100, non-prompted

  , exclude one participant for recognizing that it's a study


# rpl_study

research_type: Quantitative
problem_description:
  unsure if this is really quant. Coding was used, but reports are mostly descriptive with some inductive stats, so not typically for qual


# rpl_variable

name: Preference Score
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  were asked to rate the following 6
  statements on a 7-point Likert-Scale [37] for each type of warning:
  S1 I would like to be informed about security issues in my code
  with this type of warning.
  S2 I would be quickly annoyed by this type of warning.
  S3 It would be easy to overlook this type of warning.
  S4 IDEs already have too many warnings of this type.
  S5 I am familiar with this type of warning.
  S6 This type of warning would fit into my workflow.

  we calculated a warning preference score for each type of warning.
  We included 4 of the 6 statements to the score, which could be
  evaluated as a positive (S1 and S6) or a negative (S2 and S3) attitude
  towards a security warning. S4 and S5 were not included in the
  preference score since they cannot be assessed as either positive
  or negative statements. For instance, not being familiar with one
  type does not make the type automatically more or less liked. We
  calculated the score of at most 24 points as follows: ((S1 + (7-S2) +
  (7-S3) + S6) - 2).


name: maximum preference score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  We selected the preference score of the favorite warning type of each participant [...] We used the maximum preference score instead of a combined score since it could be that one participant favors one type of warning in particular while disliking all other types of warnings.


name: security focus
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  We asked our participants to rate their personal focus between security and functionality using a slider with the mid-point indicating equal focus. We coded a security leaning focus as a positive value for security focus. We found that the self-reported security focus followed a normal distribution over our participants.

  Unclear what minimum and maximum security focus were


name: preference
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  participants stating that they would like to be warned with this type of warning (S1,Section 7.1)

  rate the following 6 statements on a 7-point Likert-Scale [37] for each type of warning: 
  S1 I would like to be informed about security issues in my code with this type of warning.


name: workflow suitability
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "we found that the statement that a warning type suits a developers workflow (S6, Section 7.1)"

  "[...] were asked to rate the following 6 statements on a 7-point Likert-Scale [37] for each type of warning
  S6 This type of warning would fit into my workflow."


name: security score
measurement_type: between
objectivity_type: objective
variable_type: Continuous
measurement_description:
  ordinal variable score to score how well they did. The score could range from 0 to 7; the following factors were considered: whether and what hash algorithm was used (0-3 points), the iteration count for key stretching (0-1 points), and whether and how the salt was generated (0-3 points). 
  extended version of security score from Naiakshina et al [28]


name: initial security score
measurement_type: between
objectivity_type: objective
variable_type: Continuous
measurement_description:
  coded by researchers 
  min 0- max 7

  "ordinal variable score3 to score how well they did. The score could range from 0 to 7; the following factors were considered: whether and what hash algorithm was used (0-3 points), the iteration count for key stretching (0-1 points), and whether and how the salt was gen­erated (0-3 points). The detailed security scale can be found in the supplemental materials"

  score assigned to the initial submissions before any security requests


name: Ship Score
measurement_type: between
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "The ship score is composed of points gained for correctness tests and performance tests. Each mandatory correctness test is worth M points, for some constant M , while each optional correctness test is worth M/2 points. Each performance test has a numeric measure depending on the specific nature of the programming project—e.g., latency, space consumed, files left unprocessed—where lower measures are better. A test’s worth is M · (worst − v)/(worst − best), where v is the measured result, best is the measure for the best-performing submission, and worst is the worst performing. As such, each performance test’s value ranges from 0 to M ." (Section 2.2.1)

  "We set the constant multiplier M to be 50 for both contests, which effectively normalizes the scores."  (Section 4.4)

problem_description:
  Ship score is made up of correctness and performance tests. Am unsure if performance tests measure efficiency


name: Contest
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "Whether the team’s submission was for the Spring 2015 contest or the Fall 2015 contest." (Section 4.4)

  Spring 2015 was the secure log problem, and only included MOOC participants.
  Fall 2015 was the ATM problem and included both MOOC and open participants.


name: Knowledge of C
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "The fraction of team members who listed C or C++ as a programming language they know. We included this variable as a proxy for comfort with low-level implementation details" (Section 4.4)


name: programming language category
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  the used programming language

  "We manually identified each team’s submission as having one “primary” language. These languages were then assigned to one of three categories: C/C++, statically-typed (e.g., Java, Go, but not C/C++) and dynamically-typed (e.g., Perl, Python)" (Section 4.4)


name: lines of code
measurement_type: between
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "The SLOC (11) count of lines of code for the team’s final submission at qualification time." (Section 4.4.) 11: https://dwheeler.com/sloccount/


name: security bug found in submission
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "a useful alternative is the likelihood that a build-it submission contains a security-relevant bug; by this we mean any submission against which at least one crash, privacy, or integrity defect is demonstrated." (Section 4.5)


name: Break-it score
measurement_type: between
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  "For each unique bug found against a team’s submission, we subtract P points from its resilience score; as such, it is non-positive, and the best possible resilience score is 0. For correctness bugs, we set P to M/2; for crashes that violate memory safety we set P to M , and for exploits and other security property failures we set P to 2M ." (Section 2.2.1)

  "After the break-it phase, a break-it team’s score is the summed value of all defects they have found, using the above P valuations. After the fix-it phase, this score is reduced. In particular, each of the N break-it teams’ scores that identified the same defect are adjusted to receive P/N points for that defect, splitting the P points among them." (Section 2.2.2)


name: usage of advanced techniques:
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Whether the breaking team reported using software analysis or fuzzing to aid in bug finding. Teams that only used manual inspection and testing are categorized as false. 26 break-it teams (24%) reported using advanced techniques." (Section 4.6)


name: count of security bugs found
measurement_type: between
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "how many total security bugs a break-it team found. Doing this disregards a break-it team’s effort at finding correctness bugs." (Section 4.6.)


name: functional correctness
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  For each programming task, a participant received score 1 for their functionality if the code passed the test, otherwise 0 is given. (Section 6.5)


name: security correctness
measurement_type: unsure
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  "Only functional solutions received a security score. Depending on the task, we evaluated different security considerations, coding each as secure (1) or not (0) as follows." (Section 6.5)

  Details given per task in Section 6.6.


name: surprised (hourly measurement)
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  measured through an hourly survey, "comparing between subjects from both teams during the first week" (Section 5.1)

problem_description:
  unsure whether continuous or categorical, question wording seems categorical, used test points towards continuous


name: hacking approach
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  compared both between and within groups, depending on the test. All participants tried out both approaches in randomized order, but tests were sometimes conducted on just one group of participants.

problem_description:
  compared both between and within groups, depending on the test. All participants tried out both approaches in randomized order, but tests were sometimes conducted on just one group of participants.


name: frustrated (hourly measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
problem_description:
  unsure whether continuous or categorical, question wording seems categorical, used test points towards continuous

  measurement type not obvious: "When comparing between subjects from both teams during the first week," (Section 5.1), but multiple measurements (hourly) for each subject


name: time interacting with tools (hourly measure)
measurement_type: unsure
measurement_description: self-reported in hourly survey in minutes
objectivity_type: subjective
variable_type: Continuous
problem_description:
  Not obvious between and within: "When comparing between subjects from both teams during the first week," (Section 5.1), but was measured multiple times per participant


name: time harnessing (hourly measurement)
measurement_type: unsure
measurement_description: self-reported in hourly survey, measurement in minutes, harnessing is writing fuzz targets
objectivity_type: subjective
variable_type: Continuous
problem_description:
  measurement type not obvious: "When comparing between subjects from both teams during the first week," (Section 5.1), but multiple measurements (hourly) for each subject


name: prepared (end of experiment measurement)
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
problem_description: probably continuous but unclear
measurement_description:
  comparing multiple questions all asked at the end of experiment survey (appendix E), probably continuous but unclear


name: interested (end of experiment measurement)
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
problem_description: probably continous but unclear
measurement_description:
  end of experiment survey (Appendix E), probably continous (likert or likewise) but unclear, comparing multiple questions measured at the same time


name: effective use of team's skills (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  Asked as preference question at the end of experiment survey: "Which vulnerability-discovery method do you think made the best use of your team’s skill?"


name: preference for hacking approach (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  end of experiment survey, question: If you were asked to lead a vulnerability-discovery project, which method would you choose?


name: feeling part of a team (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  end of experiment survey: question: Which vulnerability-discovery method made you feel like you were part of a team?


name: frustrated (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  preference question at the end of the survey: Which vulnerability-discovery method frustrated you the most?


name: ease of getting started with (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  Preference question in end of experiment survey: Which vulnerability-discovery method did you think was easier to get started with?


name: ease of contributing (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  Preference question in end of experiment survey: Which vulnerability-discovery method do you think is easier for a novice to contribute to?


name: learning during experiment (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  Preference question in end of experiment survey: Did you learn any valuable skills during the experiment?


name: bugfinding (end of experiment measurement)
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  question in end of experiment survey: How many unique bugs did you find during the experiment? 
  0 
  1–5 
  5–10 
  10–20 
  20+

  evaluated as: "finding at least one bug"


name: hacking skill score
measurement_type: within
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "After each assessment, we collected the subjects’ work products and notes and graded them with the goal of determining three objective measures: (1) number of working harnesses, (2) number of bugs found, and (3) number of bugs reproduced." (Section 5.3)

  "Given the large scope of this study, we chose to weight each category equally. A participant’s score, then, is the sum of all measures: h+b+r." (Section 5.3)


name: secure
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "For scoring the security of the freelancers’ code, we adopted the security scale of Naiakshina et al. [21, 22] (see supplemental material)." (Section 4 - Scoring Code Security)

  "binary variable secure indicating whether participants used any kind of security in their code" (Section 4 - Scoring Code Security)


name: security score
measurement_type: within
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  "For scoring the security of the freelancers’ code, we adopted the security scale of Naiakshina et al. [21, 22] (see supplemental material)." (Section 4 - Scoring Code Security)

  "ordinal variable security to score how well they did. The score is based on a range of factors such as what hash algorithm was used, the iteration count, and whether and how the salt was generated. The value of security could range from 0 to 7," (Section 4 - Scoring Code Security)


name: Payment
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  condition assignment; amount of money offered to participants initially (at the very end of the study, all participants were paid equally)


name: task description
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  Whether participants were told to consider password-storage security or not at the beginning of their work on the task


name: security request
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "If a participant delivered a solution where passwords were stored in plain text in the database we replied 'I saw that the password is stored in clear text. Could you also store it securely?' These participants are marked as having received the security request. We deliberately set the bar for this extra request low, to emulate what a security-unaware requester could do; i.e., if it looked like something hashed we accepted it." (Section 4 - Final Study)


name: functional correctness
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "functionality score of 1 (if the participant’s code compiled and completed the assigned task) or 0 (if not)." (Section IV E)


name: security score
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "To provide a security score for each task, we considered only those participants who had functional solutions to that task. We manually coded each participant’s code to one of several possible strategies for solving the task, each of which was then labeled secure or insecure. Based on these categories, each participant who completed a task was assigned a security score of 0 (insecure approach) or 1 (secure approach) for that task." (Section IV E)


name: participant type
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "For purpose of this analysis, we categorize 14 participants as professionals, including those who are currently or recently had been professional developers. The non-professional participants are primarily university students." (Section V C)


name: participant random effect
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "As is standard, we include random effects to group each participant’s tasks together." (Section IV E)


name: confidence in task correctness
measurement_type: within
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "we examined whether participants’ self-reported confidence in their tasks’ correctness (binned as strongly agree/agree and strongly disagree/disagree/neutral) matched with our functional correctness result."


name: amount of concern over problems after the installation of an update
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Post-installation problems in a live system are only a minor concern because they don’t happen frequently." (Table 4)
  measured on 7-point likert scale from 1 - strongly disagree to 7 strongly agree


name: feeling sufficiently trained
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  5-point likert scale from 1-Never to 5 -Always
  "I feel sufficiently trained as an administrator." Table 3


name: found password storage issue
measurement_type: between
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  "binary variable found password storage issue with two values: 1: participants stated in their reviews, that they found some issues with password storage security; 0: participants did not state in their reviews, that they found some issues with password storage security." (Section 3.4.1)


name: security score
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "to identify how accurate our participants’ code reviews were with regard to the secure password-storage parameters, we used the security score of Naiakshina et al. in [23]. Participants received 2 points for hashing and salting the user passwords, another 2 points if the salt was randomly generated and at least 32 bits in length, and another 3 points for iterations, a memory-hard hashing function and if the hash’s derived length was at least 160 bits long [24]:" (Section 3.4.1)


name: Number of found issues
measurement_type: between
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "We counted how many of our 3 issues the participants were able to find (password storage, 2 distraction tasks)." (Section 4.2.3)

  max: 3, min:0


name: activity type
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
problem_description:
  Don't know which variable category would fit, maybe system type? or task related? But it's not really a tested system, but just a question, and also not related to the actual task in the study, but to tasks developers encounter in their routine


name: vulnerability type
measurement_type: within
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  assigned by researchers in open coding process

  "We coded a vulnerability type as No Implementation when a team failed to even attempt to implement a necessary security mechanism. Presumably, they did not realize it was needed." (section 4.1)

  "A vulnerability type was coded as Misunderstanding when a team attempted to implement a security mechanism, but failed due to a conceptual misunderstanding." (Section 4.2)

  "Finally, some teams attempted to implement the solution correctly, but made a mistake that led to a vulnerability." (Section 4.3)

problem_description:
  Unsure about measurement type, think within because multiple vulnerabilities could be identified per submission/break...

  Unsure about behavior, since it's more traces from behavior - added a new variable category

  Each of these variable levels should actually be a separate variable with "present" and "absent" variable levels.


name: vulnerability no implementation subtype
measurement_type: within
objectivity_type: subjective
variable_type: Categorical
problem_description: see vulnerability type
measurement_description:
  assigned by researchers in open coding process

  "This type is further divided into the sub-type All Intuitive, Some Intuitive, and Unintuitive. In the first two sub-types teams did not implement all or some, respectively, of the requirements that were either directly mentioned in the problem specification or were intuitive (e.g., the need for encryption to provide confidentiality). The Unintuitive sub-type was used if the security requirement was not directly stated or was otherwise unintuitive (e.g., using MAC to provide integrity [1])." (Section 4.1)


name: vulnerability misunderstanding subtype
measurement_type: within
objectivity_type: subjective
variable_type: Categorical
problem_description: see vulnerability type
measurement_description:
  assigned by researchers in open coding process

  Bad choice: "categorizes algorithmic choices that are inherently insecure." (Section 4.2.1)
  "Teams that chose a secure design often introduced a vulnerability in their implementation due to a conceptual misunderstanding (rather than a simple mistake)." (Section 4.2.2)


name: language popularity
measurement_type: between
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "the IEEE programming-language rankings [41] as an estimate of language maturity" (Section 3.1.2) for the used programming language


name: library type
measurement_type: unsure
measurement_description: type of programming library used
objectivity_type: objective
variable_type: Categorical
problem_description:
  not obvious measurement type because unclear how it was rated when multiple libraries of different types were used within the project


name: minimal trusted code
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "Minimal Trusted Code was measured by checking whether the security-relevant functionality was implemented in multiple locations. Projects passed if they created a single function for each security requirement (e.g., encryption, access control checks, etc.) and called it throughout. The alternative - copying and pasting code wherever security functionality was needed—is likely to lead to mistakes if each code segment is not updated whenever changes are necessary." (Appendix B)


name: Economy of Mechanism
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "When coding Economy of Mechanism, if the reviewer judged the project only included necessary steps to provide the intended security properties, then the project’s security was economical. For example, one project submitted to the secure log problem added a constant string to the end of each access log event before encrypting. In addition to using a message authentication code to ensure integrity, they checked that this hardcoded string was unchanged as part of their integrity check. Because removing this unnecessary step would not sacrifice security, we coded this project as not economical." (Appendix B)


name: programming professionality status
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  self-reported, 
  "participants who had to program Java as primary activity for their work [...] or for whom writing Java code was part of their primary job in the last 5 years" Appendix A

problem_description:
  Not totally sure which question was used in the regression models, this one (professional/not professional, as reported in Tables 4 and 5), or programming experience (in years), as reported in text. Am using the former, since I get the data from the Tables


name: expertise support score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
problem_description: unclear whether used as categorical (-> ordinal) or continuous variable here
measurement_description:
  Expert, Champion in Team? self-reported

  0 no, 1 champion, 2 expert, 3 both

  "Expertise Support Score is based on an assumption that direct expert involvement is more effective than ‘security champions’" (3.6)


name: Requirements Score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  self-reported
  3 different questions, Coded: 0 not at all, to 4 extremely, summed up (Figure 2)

  "Requirements Score reflects the security need as the arithmetic sum of the three Likert-style responses encoded as integers;" (3.6)

  "The Requirements Score assumes that, for example, occasionally using two techniques is as effective as regularly using one;" (3.6)

  Questions: 
  "Requirements Score: sum of LSS encodings for Q12 (Secure against malicious attackers), Q12 (Protects users' privacy) and Q13 How important is security for sales?" (Appendix B + C)


name: Developer Knowledge Score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  self-reported

  Coded: 0 not at all, to 4 extremely (5-point likert)

  question: How knowledgeable do you consider yourself about information security?


name: Assurance Technique Score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Each of 5 Assurance Technique used

  coded 0 none to 4 every build

  and then summed up

  Assurance Techniques: 
  Producing a threat assessment for the app 
  Scanning code with an automatic code review tool 
  Using a tool to scan for libraries with known vulnerabilities 
  Code review by someone other than the developer 
  Penetration testing


name: Security Update Frequency Score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "This required an Update Frequency Estimate of Q10 encoded as shown multiplied by Q11 (Security bug fixes) and divided by 100. The score was Log (this value plus 1)." (Appendix C)

  Q10: "How often did you release a new version of your app over the past two years? Please give your best estimate; if you have more than one app, please answer for that app that was most frequently updated. [SS] Never (0), Annually (1), Quarterly (4), Monthly (12), More frequently (24)" 

  Q11: "Over the last one to two years, what content has been in your app updates (%)?
  Security bug fixes [0-100]"

  (Appendix B)


name: task type
measurement_type: within
measurement_description: the performed python programming tasks
objectivity_type: objective
variable_type: Categorical
problem_description:
  This variable is described as having three levels (three tasks), but in some analyses, the credential storage task is split up into password storage and SQL input


name: functionality
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  researcher-coded:
  "For each programming task, we assigned a participant a functionality score of 1 if the code ran without errors, passed the unit tests and completed the assigned task, or 0 if not." (Section 3.6)


name: security
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  researcher-coded: 
  "We assigned security scores only to those solutions which were graded as functional. To determine a security score, we considered several different security parameters. A participant’s solution was marked secure (1) only if their solution was acceptable for every parameter; an error in any parameter resulted in a security score of 0." (Section 3.6.)


name: is dangerous situation
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "We asked participants, for each task, whether they believed their result was secure. In this section, we analyze the incidence of what we call dangerous solutions: solutions that are functionally correct and where the participant believes the result is secure, but our analysis indicates that it is not. " (Section 4.4.1)


name: security impact of password policy
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "How do the password policies impact the security of the authentication system? 
  1: Very negative – 5: Very positive" (Likert with given end-points)
  (Appendix)


name: frequency of password problems
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "How often do passwords cause problems in your company (e.g., forgotten passwords, etc.)?
  1: Very rarely – 5: Very often" (Likert with given end points)
  (Appendix)


name: usability impact of password policy
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "How do the password policies impact the userfriendliness of the authentication system?
  1: Very negative – 5: Very positive" (Likert item with given end points)
  (Appendix)


name: satisfaction with authentication system
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "How satisfied are you with your authentication system?
  1: [unhappy smiley] – 5: [happy smiley] " (Likert item with given end points)
  (Appendix)


name: is password composition policy creator
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  self-reported

  "Who created the specifications (password policies) for the passwords?
  Myself / My predecessor / Somebody else: [Free text]/ I do not know / I do not wish to make a statement / There are no specifications"
  (Appendix)


name: Security
measurement_type: between
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  Used TUM Crypto an open-source dataset which contains security labels for the complete list of Java code examples on Stack Overflow that provide potential solutions for our study tasks CIPHER, IV, and KEY


name: Condition: Treatment
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  Randomly assigned in two groups. One who were supposed to use google to find help with their programming tasks or a group which was provided with the link to security-reranked google search.


name: Condition × Votes
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  The interaction variable Condition × Votes
  measures whether higher voting scores in search results under
  treatment lead to better security or functionality outcomes


name: Condition × Searches
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  interaction between the average number of searches per task and the condition (treatment/no treatment)


name: most-used library
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  ranking by survey participants, who ranked "their most-used Java APIs" (gave top 3 APIs),
  this variable only refers to the top1 most used Java API

  concrete levels unclear

  "Please rank the Java cryptographic libraries/APIs below according to your frequency of use where 1 is most used. This question examines if most developers actually use the JCA APIs. Choices include Java Cryptography Architecture (JCA) APIs (irrespective of provider) and Lightweight Bouncy Castle APIs [2]. Additional APIs can be added through “Other”."

  "Apart from SpongyCastle that was mentioned by two participants in the “Other” field (rank #1 and rank #2), additional APIs listed by participants include J2ME Crypto APIs (ranked #1), FlexiProvider (ranked #3), Keyczar (ranked #3), BCMail (ranked #2), and BCOpenPGP (ranked #3) – each mentioned only once."


name: Ease of use rating for Java APIs
measurement_type: between
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  ranking by survey participants,
  "Thinking of your most-used API, <(Q7) rank#1 choice>, how would you rate its ease of use in terms of accomplishing your tasks correctly and securely? very hard to use, hard to use, easy to use, and very easy to use."

  Was aggregated for analysis: "we aggregated very hard and hard as hard and very easy and easy as easy."


name: frequency of difficulties with identifying the API to use
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Thinking of when you have a new cryptography-related task to implement in your software (e.g., ...) do you ever have difficulties with the following? Short-form of obstacle choices is shown in Fig. 3, each with choices never, rarely, occasionally, frequently, and don’t know."

  obstacle: identifying the API to use


name: frequency of difficulties with setting up the environment
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Thinking of when you have a new cryptography-related task to implement in your software (e.g., ...) do you ever have difficulties with the following? Short-form of obstacle choices is shown in Fig. 3, each with choices never, rarely, occasionally, frequently, and don’t know."

  obstacle: setting up the environment


name: frequency of difficulties with identifying the provider to use
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Thinking of when you have a new cryptography-related task to implement in your software (e.g., ...) do you ever have difficulties with the following? Short-form of obstacle choices is shown in Fig. 3, each with choices never, rarely, occasionally, frequently, and don’t know."

  obstacle: identifying the provider to use


name: frequency of difficulties with understanding the underlying implementation
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Thinking of when you have a new cryptography-related task to implement in your software (e.g., ...) do you ever have difficulties with the following? Short-form of obstacle choices is shown in Fig. 3, each with choices never, rarely, occasionally, frequently, and don’t know."

  obstacle: understanding the underlying implementation


name: frequency of difficulties with identifying sequence of method calls
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Thinking of when you have a new cryptography-related task to implement in your software (e.g., ...) do you ever have difficulties with the following? Short-form of obstacle choices is shown in Fig. 3, each with choices never, rarely, occasionally, frequently, and don’t know."

  obstacle: identifying sequence of method calls


name: frequency of difficulties with identifying parameters
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Thinking of when you have a new cryptography-related task to implement in your software (e.g., ...) do you ever have difficulties with the following? Short-form of obstacle choices is shown in Fig. 3, each with choices never, rarely, occasionally, frequently, and don’t know."

  obstacle: identifying parameters


name: increase of privacy concerns through data sharing
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Data Sharing: The collected data might be given to third parties for purposes like advertising."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”"
  (Section 3.1)


name: increase of privacy concerns through data breaches
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Data Breaches: Malicious users might get access to sensitive data about other users."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”."
  (Section 3.1)


name: increase of privacy concerns through data aggregation
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Data Aggregation: The system discovers additional information about the user by aggregating data over a long period of time."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”"
  (Section 3.1)


name: increase of privacy concerns through data distortion
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Data Distortion: The system might misrepresent the data or user intent."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”."
  (Section 3.1)


name: decrease of privacy concerns through technical details
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Technical Details: Describing the algorithms/source code of the system in order to achieve higher trust (e.g., encryption of data)."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”."
  (Section 3.1)


name: decrease of privacy concerns through privacy policy
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Privacy Policy, License Agreements, etc.: Describing what the system will/won’t do with the data."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”."
  (Section 3.1)


name: decrease of privacy concerns through anonymization
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Anonymizing all data: Ensuring that none of the data has any personal identifiers."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”."
  (Section 3.1)


name: decrease of privacy concerns through details on usage
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Details on usage: Describe, e.g., in a table how different data are used."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”."
  (Section 3.1)


name: decrease of privacy concerns through privacy laws
measurement_type: between
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "Privacy Laws: Describing which national law the system is compliant with (e.g., HIPAA in the US, European privacy laws)."
  "For each concern, the respondents could answer using a 3-point semantic scale with the options: “Yes”, “Uncertain”, and “No”."
  (Section 3.1)


name: Participant location
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  self-reported: 
  "We asked respondents to specify with which region of the world they identify themselves. The options were North America, South America, Europe, Asia/Pacific, Africa, and other. Since we have only seven responses from South America and Africa combined, we focus on the differences between the others."


measurement_type: unsure
objectivity_type: unsure
variable_type: Categorical
name:
  prefer providing details on data usage for mitigating privacy concerns compared to privacy policies


name: Condition
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  Researcher assigned; Which type of description about different ad choices was shown to participants in the scenarios


name: Usability related questions
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "This code was easily readable
  It was strenuous to understand what this code did
  This code looks similar to the way I would write code
  The code is badly structured
  I am sure that I correctly understood what this code does
  It was hard to understand what the variables mean"
  each measured on a scale from 
  strongly disagree, disagree, neutral, agree, strongly agree

  (See Appendix, Table III)


name: Trust related questions
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  I trust that the decompiled code is correct

  I would rather analyze the assembly code

  Both rated on scale: strongly disagree, disagree, neutral, agree, strongly agree
  aggregated for analysis


name: decompiling success
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "had to answer questions about the functionality of the code." (Section VII B)
  "We assigned a weight to each question according to its importance in understanding the task and scored all answers. We conducted a two-pass scoring approach for calibration purposes. Since the answers contained variable names and referred to loop structures, it was not possible to blind this part of the evaluation." (Section VIII B)

  100% correctly answered (so min=0, may=100)


name: Fail
measurement_type: within
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "we additionally marked each task as a pass if participants scored over 70%. This level was chosen based on our judgement that these tasks had been answered sufficiently, meaning that their results would be useful in a malware analysis team." (Section VIII B)


name: type of control
measurement_type: within
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "How much control do users generally have over the amount of data collected by ad networks? • No control at all • Very little control • Moderate control • A lot of control • Full control"
  "As a developer, how much control do you generally have over the amount of data collected by ad networks? • No control at all • Very little control • Moderate control • A lot of control • Full control"

  Users vs. developers


name: criticality
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "we asked respondents to rate the following types of data on a 5-point semantic scale ranging from “Very critical” to “Uncritical”."
  (Section 3.3)

  Very critical ; critical ; neutral ; somewhat critical, uncritical


name: task functionality
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "For each programming task, we assigned a participant a functionality score of 1 if the code ran without errors, passed the tests and completed the assigned task, or 0 if not." (Section 6.6)


name: Task
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  performed task

  "For both tasks, participants were provided with stub code and some commented instructions. These stubs were designed to make the task clear and ensure the results could be easily evaluated. We also provided a main method pre-filled with code to test the provided stubs." (Section 6.4)


name: Warning displayed
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  whether a warning was displayed (could only happen in PyCrypto patch condition)
  "PyCrypto control condition, or the PyCrypto patch condition, where we tested our security warning" (Section 6)


name: Secure
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "In addition, we used a binary variable called secure which was given if participants used at least a hash function in their final solutions and thus did not store the passwords in plain text." (Section 3.5)


name: API usability
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "API usability as interpreted by answers to questionnaire by Acar et al. [1] based on the Cognitive Dimensions framework [10]" (Section 8.4)

  "Acar et al. usability questionnaire for security APIs [1]" (Section 6.5)


name: perceived usability measured by W1
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Agreement to: The security warnings displayed in the console helped to solve this task

  measured using likert scale: (strongly agree; agree; neutral; disagree; strongly disagree; does not apply)

  coded as strongly agree = 2, neutral = 0, strongly disagree=-2


name: perceived usability measured by W2
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Agreement to: "When I made a mistake, I got a meaningful error message/exception"


  measured using likert scale: (strongly agree; agree; neutral; disagree; strongly disagree; does not apply)
  coded as strongly agree = 2, neutral = 0, strongly disagree=-2


name: perceived usability measured by W3
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Agreement to:  
  "Using the information from the error message/exception, it was easy to fix my mistake"

  measured using likert scale: (strongly agree; agree; neutral; disagree; strongly disagree; does not apply)
  coded as strongly agree = 2, neutral = 0, strongly disagree=-2


name: Security score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "Every solution was rated on a scale from 0 to 7, according to the security score introduced in [42] (see Appendix D). This value is referred to as the security score." (Section 3.5)


name: Attempted security
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "We were also interested in participants who attempted to store user passwords securely, but struggled and then deleted their attempts from their solutions (this was coded as attempted but failed, or ABF). For this, we collected and analyzed participants’ code history. In order to identify security attempts, we used the Unix grep utility. With grep, we searched for security-relevant terms based on the frameworks and best practices (see Appendix F). When a term was found, we analyzed the code snippets manually." (Section 3.5)


name: Security rationales
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  broadly, whether developers have a security focus

  "Rationale analysis. To assess whether the tasks were performed with a security rationale, and whether this correlated to their solution (RQ2), we analyzed the rationale elicited from participants after each task using open coding1. Two authors independently coded each result to determine whether the rationale indicated a focus on security." (Section 3.3.)


name: perceived security impact of selecting external SDK
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Likert-scale 1-5 (strongly agree, agree, neutral, disagree, strongly disagree)

  The security of apps I develop is affected by decision I make on...
  the external SDKs I use


name: perceived security impact of writing code
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Likert-scale 1-5 (strongly agree, agree, neutral, disagree, strongly disagree)

  The security of apps I develop is affected by decision I make on...
  the code I write


name: perceived security impact of choice of license under what software is released
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Likert-scale 1-5 (strongly agree, agree, neutral, disagree, strongly disagree)

  The security of apps I develop is affected by decision I make on...
  the license my software is released


name: perceived security impact of testing
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Likert-scale 1-5 (strongly agree, agree, neutral, disagree, strongly disagree)

  The security of apps I develop is affected by decision I make on...
  whom I involve in testing


name: perceived security impact of help sources
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Likert-scale 1-5 (strongly agree, agree, neutral, disagree, strongly disagree)

  The security of apps I develop is affected by decision I make on...
  what sources I go to for help


name: perceived security impact of choosing development environment
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Likert-scale 1-5 (strongly agree, agree, neutral, disagree, strongly disagree)

  The security of apps I develop is affected by decision I make on...
  setting up my development


name: library experience
measurement_type: unsure
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  Whether the participant has used the library before, seen code that used it but not used it themselves; or neither. Self-reported


name: task functionality
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  does code run without errors? yes/no

  "For each programming task, we assigned a participant a functionality score of 1 if the code ran without errors, passed the tests and completed the assigned task, or 0 if not."


name: task security
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  coded by researchers

  "We assigned security scores only to those solutions which were graded as functional. To determine a security score, we considered several different security parameters. A participant’s solution was marked secure (1) only if their solution was acceptable for every parameter; an error in any parameter resulted in a security score of 0."

  "Not all security parameters applied to all libraries, as some libraries do not allow users to make certain potentially insecure choices. Details of how the different security parameters applied to each library can be found in Table III."

  "For key generation, we checked key size and proper source of randomness for the key material. We selected an appropriate key size for a particular algorithm (e.g., for RSA we required at least 2 048-bit keys [64]). For key storage we checked if encryption keys were actually encrypted and if a proper encryption key was derived from the password we provided. Depending on the library and task type, encrypting cryptographic key material requires the application of a key derivation function such as PBDKF2 [65]. For libraries in which developers had to pick parameters for PBKDF2 manually (cf. Table III), we scored use of a static or empty salt, HMAC-SHA1 or below as the pseudorandom function, and less than 10 000 iterations as insecure [66]."


  "For symmetric encryption, we scored ARC2, ARC4, Blowfish, (3)DES, and XOR as insecure, and AES as secure. We scored the ECB as an insecure mode of operation and scored Cipher Block Chaining (CBC), Counter Mode (CTR) and Cipher Feedback (CFB) as secure. Static, zero or empty initialization vectors were scored insecure. For asymmetric encryption we scored the use of OAEP/PKCS1 for padding as secure."


name: API usability score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  scores on our cognitive-dimension-based scale

  "We combined the cognitive dimensions framework [24] with usability suggestions from Nielsen and from Smith and Green [23], [27], and pulled out the factors that could most easily be evaluated via selfreporting from developers using the API. We transformed these factors into an 11-question scale (given in Appendix A) that focuses on the learnability of the API, the helpfulness of its documentation, the clarity of observed error messages, and other features. Our scale can be used to produce an overall score, as well as to target specific characteristics that impede the usability of each API." (Section III G)


name: security prompting
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "prompting was one variable in our study: half of the participants were tasked to securely store the passwords while the other half was not explicitly asked to do so." (Section 3)


name: stored passwords before
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  self reported
  "In addition to that, we asked our participants whether they had stored user passwords in a database before." (Section 6.3)


name: Study Scenario
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  Sample from Naiakshina (2019) with company deception or Danilova (2020) sample where participants knew they were participating in an experiment


name: Secure
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  a binary variable secure that indicated whether a participant included at least some kind of security


name: Security score
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  security score was used to assess the security of the solution according to the password security scale Naiakshina (2017) from 0-7

  "An ordinal variable security score was used to assess the security of the solution according to the password security scale [29] from 0-7; the security score considered the hash algorithm, iteration count for key stretching and salt generation." (Section 3.4)

  Appendix B:
  "1. The end-user password is salted (+1) and hashed (+1). 2. The derived length of the hash is at least 160 bits long (+1). 3. The iteration count for key stretching is at least 1 000 (+0.5) or 10 000 (+1) for PBKDF2 and at least 210 = 1 024 for bcrypt (+1). 4. A memory-hard hashing function is used (+1). 5. The salt value is generated randomly (+1). 6. The salt is at least 32 bits in length (+1)."


name: API usability score
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "API usability scores suggested by Acar et al. (2017) [2]"

  "The score could range from 0-100 with 100 being the highest usability score that can be achieved."


name: Implementation time initial submission
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "We compared the implementation time to submit the initial solution" (Section 6.6.1)

  (measured in days)


name: Implementation time time for SecRequest-P
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  measured in hours

  "Further, we compared the time spent on the first security request." 

  On the first security request:
  If the submissions included plain text password storage, subjects were asked to revise their submissions and to securely store user passwords.

  wording: "SecRequest-P(laintext): If the submissions included plain text password storage, subjects were asked to revise their submissions and to securely store user passwords."


name: organization type
measurement_type: unsure
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  We also find a reasonable variety in terms of industries that the participants work in: Spanning from IT enterprises to ISPs to government organizations and organizations that do not operate in the IT sector, but who rely on IT to support their core business operations.


name: superiors know what operators are doing
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  5point likert-scale with 0=neutral (I strongly agree, I agree, I neither agree nor disagree, I disagree, I strongly disagree, I don't know)

  "My direct supervisor understands what I’m actually doing."


name: current state of employment
measurement_type: unsure
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  "(1) Are you employed, self-employed or out of work? If you’re a student, please choose what you do besides studying. 
  # I’m employed part-time. 
  # I’m employed full-time. 
  # I’m my own boss. 
  # I do not work at the moment." (the last option was filtered out)


name: trust in used tools and equipment
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  Agreement on a 5-point likert-scale with 0=neutral (I strongly agree. I agree. I neither agree nor disagree. I disagree. I strongly disagree. I don’t know.)

  "I trust all the tools and equipment we’re using."


name: work experience
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  in years in the field, self-reported

  "How long have you worked as an operator overall? 
  # Less than 1 year. 
  # 1 up to 3 years. 
  # More than 3 up to 5 years. 
  # More than 5 up to 10 years. 
  # More than 10 up to 20 years. 
  # More than 20 years."


name: importance of blameless postmortems
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  agreement on 5-point likert-scale with 0=neutral (I strongly agree. I agree. I neither agree nor disagree. I disagree. I strongly disagree. I don’t know.)

  "Blameless postmortems help to detect essential issues in corporate procedures."


name: Prompted (to write specification)
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  treatment randomly assigned to 1/2 of the population

  "Developers in the specification group were then asked to write a specification for how the password should be stored."


name: type of specification
measurement_type: unsure
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  did participant write a specification before starting the task (and for what reason prompting/individually)


name: use of standard
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "developers reported referring to some kind of standard or guide when writing their password storage method" self-reported


name: security score (Naiakshina’s metric)
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  7 security criteria, each fulfilled on gives one point (min=0, max=7)

  "• The end-user password is salted (+1) and hashed (+1).
   • The derived length of the hash is at least 160 bits long (+1). 
  • The iteration count for key stretching is at least 1,000 (+0.5) or 10,000(+1) for PBKDF2 and at least 210 for bcrypt (+1). 
  • A memory-hard hashing function is used (+1). 
  • The salt value is generated randomly (+1). 
  • The salt is at least 32 bits in length (+1)." (Figure 1)


name: Correctness
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  Measured per program. Equals the number of correctly solved tasks (0-2)

  "Correctness (measured per program) is the number of correctly solved tasks: 0 if no task is solved correctly, 1 if precisely one task is solved correctly and 2 if both tasks are correct."


name: Total time
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  time per program in minutes independent of correctness

  "Total time (measured per program in minutes) shows how long a participant worked on the program, independently on the correctness of solutions. Although this is not a code comprehension variable by itself, we use it to derive the notion of efficiency below"


name: Programming experience
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  On a scale from 1-4

  1= Few and small programs (e.g., course exercises)
  2= Many small programs
  3= Small programs and 1 or 2 big programs (e.g., thesis
  and projects)
  4= Big programs

  "quality and type of code written so far"


name: Study-relevant Experience
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  refers to the experience and knowledge in code obfuscation, Java, the usage of Eclipse for software development, debugging software, the usage of Eclipse for debugging software.
  Values from 1=very low to 5=very high based on likert-scale questions in the pre study questionnaire


name: Comprehension Skills
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  "Comprehension Skills are measured by considering the efficiency of a participant when working with Clear code."


name: Obfuscation method
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  obfuscated with name overloading (NO) or with opaque predicates (OP), or control
  researcher assigned and prepared


name: Obfuscation Experience
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  experience with obfuscation and debugging
  "What is your experience/knowledge in... (5-point Likert scale from “very low” to “very high”) Q9. code obfuscation?"


name: Java Experience
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  experience with Java and using Eclipse


  "What is your experience/knowledge in... (5-point Likert scale from “very low” to “very high”) 
  Q10. Java? 
  Q11. the usage of Eclipse for software development?
  Q13. the usage of Eclipse for debugging software?"


name: Time correct (strict measurement)
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  Strict correctness as measured by Ceccato et al.

  "Time correct (measured per program in minutes) is the time spent on average for correctly solving tasks for a program. It is computed as the sum of times spent on correctly solved tasks divided by the number of correctly solved tasks. If no answer was given correctly, the participant was taken out of the calculations."


name: Efficiency
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  correctness divided by total time
  Correctness (measured per program) is the number of correctly solved tasks: 0 if no task is solved correctly, 1 if precisely one task is solved correctly and 2 if both tasks are correct.
  Total time (measured per program in minutes) shows how long a participant worked on the program, independently on the correctness of solutions. Although this is not a code comprehension variable by itself, we use it to derive the notion of efficiency below.


name: Number of file open commands
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "The number of the file open operations, which correspond to either opening a new file or switching the focus to the already opened one."


name: Number of advanced commands
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "The number of executed advanced commands such as automatic identifier renaming, construction of call graphs and type hierarchies."
  "Advanced commands have the command ids starting with org.eclipse.jdt.ui.edit.text.java."


name: Time spent on code reading
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  in minutes
  "The total time of code reading, which is defined as the overall processing duration for the given program minus the execution and debugging time."


name: quality of configuration
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  based our evaluation criteria on Qualy’s SSL Test

  "The rating of the evaluation criteria is expressed with a grade from A to F and composed out of three independent values: (1) protocol support (30%), (2) key exchange (30%) and (3) cipher strength (40%). Some properties, e.g., support for the RC4 cipher cap the overall grade as shown in Table 3." 

  "The full set of evaluation criteria based on the metrics used in Qualy’s SSL Test is listed in Appendix A." 
  (Section 4.1)


name: task completion
measurement_type: unsure
objectivity_type: unsure
variable_type: Categorical
measurement_description:
  "successfully deployed the website documents in the CA-Certbot task and 29 (94%) in the CA-Traditional task."


name: Chosen valid domains
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  valid for (only TLD or WWW) or both

  "domains the participants chose to include in the certificate separated by the CA condition. “TLD only” and “WWW only” mean that they entered only “tld.com” or “www.tld.com” as a valid domain."


name: learning in cryptography
measurement_type: between
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  participants were asked to rate the cryptography skills learned during the cybersecurity competition:
  on a scale of 1 (“None”)  to 4 (“A lot”)


name: selection time (T)
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "First, we measure the selection time (T ) as the time (in seconds) that is necessary for a team to arrive at their final pattern selection for each task."


name: selection efficiency (E)
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  The number of selected security patterns is divided by the number of viewed security patterns. The values are in [0,1] since every pattern has to be viewed to be selected.
  E=1 means that every viewed pattern was selected, maximizing efficiency. 
  Measured per team and task
  "Second, we measure the selection efficiency (E) as follows. Nview is the total number of patterns that the team looked at (i.e., opened in the catalog browser) when performing a task, ignoring whether or not the pattern was retained or discarded for instantiation. Nselect is the number of patterns that the team finally selected for instantiation in the architecture for that task. The selection efficiency then is E = Nselect/Nview. As every selected pattern must have been viewed, Nview ≥ Nselect and thus 0 ≤ E ≤ 1. Note that E = 1 means that every viewed pattern was selected, maximizing efficiency"


name: Task
measurement_type: within
objectivity_type: objective
variable_type: Categorical
measurement_description:
  Each team performed five tasks (A-E) involving the hardening of a software architecture via security patterns.


name: Task x Condition
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  First letter (A-E) indicates task, second letter (P/A) indicates treatment condition

  "Each task consists of elaborating the software architecture in order to support a security requirement. The requirement is documented via a description of the threats to protect against, in the form of two misuse cases [19], and a description of how the system is expected to react, in the form of a quality attribute scenario [18]."

  "Note also that, in tasks A and B, the correct solution requires the selection of a system of collaborating patterns, while in tasks C–E a single pattern can be found that solves the problem entirely. Further, task E is expected to be the easiest one to solve."

  "All teams were given a process to follow when hardening the architecture. This provides the students with the necessary guidance so that they do not get “stuck” with the tasks, as they never used security patterns before. The process is lightweight and very intuitive, as it resembles any problem solving approach: a list of potential solutions is scouted first and then the most adequate is chosen."

problem_description:
  variable not in the paper. Created it because no subsample selection was possible in Wilcoxon rank sum test.


name: covered misuse cases
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  first self-reported by students, then assessed by security expert

  "Measuring security. The most important measure that we collect is the number of covered misuse cases per task, which ranges from 0 to 5. 
  After the teams have completed all tasks and have submitted their solutions, they are provided with the misuse cases, which the experimenters use to evaluate the designs. The teams have to prepare a written report in which, per task, they discuss how their submitted solution eliminates (some of) the misuse cases or, alternatively, have to mark (some of) the misuse cases as uncovered. It is made clear to the participants that, because they are not security experts, their grade is not based on the number of covered misuse cases, but rather on the quality of their reasoning (i.e., why the misuse case is not covered, or how it is prevented by their solution).
  Three security experts (one not involved with the execution of this study) independently assess the solutions. Each expert evaluates the tasks performed by about one third of the teams. The expert makes his own assessments, but takes into account the provided reasoning of the team, for example to discover undocumented but plausible assumptions made by the team. In rare cases, a discussion during the exam is used to improve the expert’s understanding of the solution."


name: correctness score
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "As a complementary device, and before analyzing the misuse cases, the experts make an overall assessment of the correctness of each task’s solution. The correctness score is an ordinal variable, with 3 possible values: wrong (0), some errors (1), and correct (2)."


name: Productivity
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  "Measuring productivity. We also collect the time that each team spends executing each task. This time is seamlessly and reliably measured by the tool that the teams use (see Section IV-D). The measure starts when a task is made available to the team via the tool, and ends when the solution for that task is submitted (again, via the tool). Hence, the time includes the time needed to read and analyze the task description, choose a solution, and integrate the solution into the design that is given as a starting point."

  "We measure the time t it takes each of the teams to complete a security design task."

  time measured in hours


name: Task x With Patterns
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
problem_description: variable not in the paper. But it enables me to report the statistical results for subsamples
measurement_description:
  The first letter denotes the task and the second (Y/N) indicates whether the team was in the with or without patterns treatment.

  tasks: "In order to reduce variability and to make results comparable, the participants are provided with an initial design of the banking system, as described earlier in section III-B. The design does not contain any specific security measure. The participants work in teams and have to extend and harden the design by means of seven tasks, which are labeled A through G. A summary of the tasks is given in Table II. The complete task descriptions (as provided to the participants) are available on the companion website [16]."


name: presence of a blindspot
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  misunderstanding or misusing APIs introducing vulnerabilities

  "An API security blindspot is a misconception, misunderstanding, or oversight [9] on the part of the developer when using an API function, which leads to a violation of the recommended API usage protocol with possible introduction of security vulnerabilities. Blindspots can be caused by API functions whose invocations have security implications that are not readily apparent to the developer."


name: cyclomatic complexity
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "Puzzles varied in their cyclomatic complexity, defined as a quantitative measure of the number of linearly independent paths in the source code. We classified the cyclomatic complexity of each puzzle into one of three levels: an integer value of low (cyclomatic complexity of 1–2), medium (cyclomatic complexity of 3–4), or high (cyclomatic complexity > 4) complexity."


name: perceived difficulty
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  using a Likert scale (1 = not at all to 10 = very) 

  Difficulty (How difficult was this scenario?);


name: perceived clarity
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  developers responded to the about their puzzle perceptions using a Likert scale (1 = not at all to 10 = very)

  Clarity (How clear was this scenario?)


name: perceived familiarity
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  developers responded to the about their puzzle perceptions using a Likert scale (1 = not at all to 10 = very)

  "Familiarity with the API functions presented in the code snippet (How familiar were you with the functions in this scenario?)"


name: solution confidence
measurement_type: within
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  developers responded to the about their puzzle perceptions using a Likert scale (1 = not at all to 10 = very)

  "Confidence (How confident were you that you solved the scenario correctly?)"


name: blindspot puzzle accuracy
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  "the number of correctly solved blindspot puzzles across the four blindspot puzzles constituted the ordinal outcome variable blindspot puzzle accuracy, with a range from 0 to 4."


name: Reasoning
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  cognitive function

  "The BTACT is a battery of cognitive processing tasks for adults of different ages and takes approximately 20 minutes to complete. The BTACT sub-tests refer to episodic verbal memory, working memory, verbal fluency, inductive reasoning, and processing speed."

  "For the BTACT, we aggregated the total number of correct responses in each of the cognitive subset tasks."


name: Working Memory
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
measurement_description:
  cognitive function

  "For the Oral Symbol Digit task we computed a sum score of correct responses. For the BTACT, we aggregated the total number of correct responses in each of the cognitive subset tasks."

  "Oral Symbol Digit task [...] is a brief measure of processing speed and working memory. In this task, developers were presented with a coding key containing nine abstract symbols, each paired with a number between 1 and 9. They were then given 120 seconds to call out as many numbers that went with the corresponding symbols, in the order presented and without skipping any. The BTACT is a battery of cognitive processing tasks for adults of different ages and takes approximately 20 minutes to complete. The BTACT sub-tests refer to episodic verbal memory, working memory, verbal fluency, inductive reasoning, and processing speed."


name: Technical expertise
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  self reported

  Technical score (unclear how this is calculated)

  "The third part (Professional Experience and Expertise) included questions about the developers’ technical proficiency and years of programming experience in six commonly used programming languages (i.e., Java, Python, C/C++, PHP, Visual Basic.Net, and JavaScript). A free-text response field was provided for developers to record their preferred programming language, if it was not listed. Developers also indicated their level of knowledge in and experience with 17 programming concepts and technologies identified from the literature and via job postings for software developers [6, 30] (e.g., SQL/MySQL, Cryptography, File compression, Networking, HTTP/HTTPS, I/O operations)."


name: Agreeableness
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  Big Five Inventory (BFI) questionnaire to measure aspects of personality [29]. This questionnaire contains 44 items to assess five personality dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. 

  Developers rated the extent to which they endorsed each personality statement on a Likert scale (1 = disagree strongly; 5 = agree strongly). We computed the sum score across all items for each of the five personality dimensions.


name: Conscientiousness
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  Big Five Inventory (BFI) questionnaire was used to measure aspects of personality. This questionnaire contains 44 items to assess five personality dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism. Developers rated the extent to which they endorsed each personality statement on a Likert scale (1 = disagree strongly; 5 = agree strongly). We computed the sum score across all items for each of the five personality dimensions.


name: Openness
measurement_type: unsure
objectivity_type: unsure
variable_type: Continuous
measurement_description:
  Big Five Inventory (BFI) questionnaire was used to measure aspects of personality. This questionnaire contains 44 items to assess five personality dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.
  Developers rated the extent to which they endorsed each personality statement on a Likert scale (1 = disagree strongly; 5 = agree strongly). We computed the sum score across all items for each of the five personality dimensions.


name: acceptance of less privacy for monetary discounts
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "We asked respondents if they would accept less privacy for the following:"
  "For each option, the respondents could answer using a 3-point semantic scale having options: “Yes”, “Uncertain”, and “No”."
  "Monetary discounts (e.g., 10% discount on the next purchase)"
  (Section 3.4)


name: acceptance of less privacy for fewer advertisements
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "We asked respondents if they would accept less privacy for the following:"
  "fewer advertisements"
  "For each option, the respondents could answer using a 3-point semantic scale having options: “Yes”, “Uncertain”, and “No”."


name: acceptance of less privacy for "intelligent" or added functionality
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "We asked respondents if they would accept less privacy for the following:"
  "“Intelligent” or added functionality of the system (such as the Amazon recommendations)""
  "For each option, the respondents could answer using a 3-point semantic scale having options: “Yes”, “Uncertain”, and “No”."


name: participant origin
measurement_type: between
objectivity_type: objective
variable_type: Categorical
measurement_description:
  self-reported as country "In which country do you currently reside?" [List of countries provided]

  categorized by continent by researchers


name: (perceived) control
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "As a developer, how much control do you generally have over the amount of data collected by ad networks? • No control at all • Very little control • Moderate control • A lot of control • Full control"
  "How much control do users generally have over the amount of data collected by ad networks? • No control at all • Very little control • Moderate control • A lot of control • Full control" (Appendix)


name: Difficulty
measurement_type: within
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "A within-subjects factor that represents the difficulty of the task. Based on the results from our prestudy (§VII-B), we grouped the tasks according to their difficulty into two groups (medium and hard), each containing three tasks." (Section VII C)


name: part-time professional
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "We asked our participants whether they had a part-time job in computer science. In prior research, Acar et al. counted students who had part-time jobs as professionals [11]." (Section 7.1.2)


name: prior experience with password storage
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  binary
  "We hypothesized that participants who had previous experience storing user passwords in a database backend would be more likely to add security in the study. Therefore, we wanted to test whether participants who reported having stored passwords before performed differently regarding security compared to participants who had never stored passwords before." (Section 7.2.2)


name: API usability
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "We used the usability score from Acar et al. [9] (see Appendix C.1) to evaluate how participants perceived the usability of the two frameworks." (Section 7.2.4)


name: amount of attended security classes
measurement_type: unsure
objectivity_type: objective
variable_type: Continuous
problem_description: Maybe categorical (just 4 different values - variable used in Fishers Exact test)
measurement_description:
  "In prior work, Acar et al. found that security courses had a significant effect on security [11]; therefore, we asked our participants which courses they had attended at our university in the past. We gave one point per security-relevant course. Since not all Masters students had completed their undergraduate studies at the same university, we also asked for other courses. None of the participants added a security-relevant class in the open question space. Participants reported they had attended between 0 and 4 security classes (mean = 0.8, median = 1, sd = 0.99)." (Section 7.2.6)


name: StaticAnalysis
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  How often the participant found static analysis tools helpful in finding vulnerabilities (self-reported)


name: DynamicAnalysis
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  "How often the participant found dynamic analysis tools helpful in finding vulnerabilities" (self-reported)


name: Security
measurement_type: unsure
objectivity_type: subjective
variable_type: Categorical
measurement_description:
  "We assigned security scores only to those solutions which were graded as functional. To determine a security score, we considered several different security parameters. Our scoring followed the relevant parts of the security scoring in [1]. Still we give a brief summary of the security scoring we applied. 
  For key generation, we checked key size and randomness. For key storage we checked if encryption keys were actually encrypted and if a proper encryption key was derived from the password we provided. For key derivation, we scored use of a static or empty salt, HMAC-SHA1 or below as the pseudorandom function, and less than 10,000 iterations as insecure. For the symmetric encryption task, participants had to select encryption parameters. Therefore, we scored the security of the chosen encryption algorithm, mode of operation, and initialization vector. We scored ARC2, ARC4, Blowfish, (3)DES, and XOR as insecure, and AES as secure. We scored the ECB as an insecure mode of operation and scored CBC, CTR and CFB as secure. Static, zero or empty initialization vectors were scored insecure."


name: amount of payment
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "To allow researchers to conduct scientific studies in an economical manner, we began recruiting participants with a lower compensation amount of E120 for the project, including the programming task and the survey. Naiakshina et al. did not find a significant effect between the two payment levels on security. Therefore, after facing difficulties to recruit over 18 participants with €120, we started offering €220 " (Section 3.1)


name: participant random intercept
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "we used a mixed model that adds a random intercept to account for multiple tasks from the same participant."


name: cryptography experience
measurement_type: unsure
objectivity_type: subjective
variable_type: Continuous
measurement_description:
  self-reported on a 5-point Likert scale
  No experience, little experience, moderate experience, very experienced, extremely experienced

  "Participants reported their familiarity with cryptography on a 5-point Likert scale."


name: random intercept
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "When we consider results on a per-task rather than a perparticipant basis (for security and functionality results, as well as perceived security), we use a mixed model that adds a random intercept to account for multiple tasks from the same participant." (Section IV B)


name: participant random effect
measurement_type: unsure
objectivity_type: objective
variable_type: Categorical
measurement_description:
  "accommodating for (1) the hierarchical data structure in which each set of six puzzles (level-1) was nested within each developer (level-2) and (2) the dichotomous outcome variable puzzle accuracy (1 = correct answer, 0 = incorrect answer). The independent variable was the presence of a blindspot (0 = no blindspot; 1 = blindspot). In this model, we also considered the random effect of the intercept to accommodate for inter-individual differences in overall puzzle accuracy."


