# rpl_variablelevel

name: connection task
description:
  "Network Connection. This provides a code snippet to be completed to establish a connection to a server. Participants were given the domain and query path of the URL, then requested to add a protocol to make this URL valid and establish the connection and get a return code status (i.e 200, 400 or 500). The goal of this task was to check if participants used a secure connection (HTTPS) to connect to the given host. The host supported both HTTP and HTTPS protocols. A corresponding test case was provided, which passes when the method connect returns the value of 200. " (Section 6.2)


name: encryption task
description:
  "Data Encryption. In this task, participants were asked to encrypt a given plain-text. Participants were expected to encrypt a string and return an array of bytes. The goal of this task is to see how knowledgeable developers are about cryptographic APIs. A corresponding test case was provided, which passes when the returned array of encryption method is not null and has a length greater than 16." (Section 6.2)


name: sqlite task
description:
  "SQLite Query. In this task, participants were asked to build a SQL query to retrieve the age of a given user name. In the skeleton app, we have already created a SQLite database with a predefined table named "users". Table "users" has 4 columns: id, name, age, password. The goal of this task is to see if participants are aware of SQL injection attacks. A corresponding test case was provided, which passes when this test case returns the correct age of a predefined user which has been inserted into the attached SQLite database." (Section 6.2)


name: student
description:
  recruited from 5 universities in Germany, subject not specified, but assuming computer science or similar


name: true (experienced)
description:
  True if participant has submitted more than 5 apps to Google Play Store or has more than 3 years of experience. Self-reported.


name: false (not experienced)
description:
  If participant has neither submitted more than 5 apps to the Play Store, nor has more than 3 years of experience.


name: yes (secure = 1)
description:
  "Only functional solutions received a security score. Depending on the task, we evaluated different security considerations" (Section 6.5)


name: no (secure = 0)
description:
  "Only functional solutions received a security score. Depending on the task, we evaluated different security considerations," (Section 6.5)


name: depth-first search (SD)
description:
  "select a piece of software and invest significant human resources into finding bugs in that software without any prior indication of success." (Abstract)


name: breadth-first search (SB)
description:
  "In breadthfirst search, humans perform minimal work to enable automated analysis on a range of targets before committing additional time and effort to research any particular one." (Abstract)


name: secure networking task
description:
  "This task addressed correct usage of HTTPS and TLS in the presence of X.509 certificate errors." (Section IV C)


name: secure storage task
description:
  "This task focused on secure storage of the user’s login ID and password for the remote server." (Section IV C)


name: least permissions task
description:
  "In this task, participants were asked to add functionality to dial a hard-coded customer support telephone number." (Section IV C)


name: Official only (official)
description:
  "Participants were only allowed to access websites within the official Android documentation" (Section IV B)


name: Stack Overflow Only (SO)
description:
  "Participants were only allowed to access questions and answers within Stack Overflow" (Section IV B)


name: Book only (book)
description:
  "Participants were only allowed to use two books: Pro Android 4 [23] and Android Security Internals [12]." (Section IV B)


name: free choice (free)
description:
  "Participants were allowed to use any web resources of their choice, and were also offered access to the two books used in condition book." (Section IV B)


name: 1 (functional)
description:
  "functionality score of 1 (if the participant’s code compiled and completed the assigned task)" (Section IV E)


name: 0 (non-functional)
description:
  "functionality score of 1 (if the participant’s code compiled and completed the assigned task) or 0 (if not)" (Section IV E)


name: 1 (secure)
description:
  "Based on these categories, each participant who completed a task was assigned a security score of 0 (insecure approach) or 1 (secure approach) for that task." (Section IV E)


name: professional developer
description:
  "For purpose of this analysis, we categorize 14 participants as professionals, including those who are currently or recently had been professional developers." (Section V C)


name: SME (small and medium sized enterprises)
description:
  "companies with 250 employees or fewer were tagged as small and medium-sized enterprises" (Section 4.3)


name: Secure Log (SL)
description:
  "This problem asks teams to implement two programs: one to securely append records to a log, and one to query the log’s contents." (Section 2.2)


name: Secure communication (SC)
description:
  "This problem asks teams to build a pair of client/server programs. These represent a bank and an ATM, which initiates account transactions (e.g., account creation, deposits, withdrawals, etc.)." (Section 2.2)


name: Multiuser database (MD)
description:
  "This problem asks teams to create a server that maintains a secure key-value store. Clients submit scripts written in a domain-specific language. A script authenticates with the server and then submits a series of commands to read/write data stored there. Data is protected by role-based access control policies customizable by the data owner, who may (transitively) delegate access control decisions to other principals." (Section 2.2)


name: task cipher
description:
  from Symmetric Encryption
  "a symmetric cipher had to be initialized by setting the algorithm, block mode and padding." Section 8.2


name: task IV (initialization vector)
description:
  from Symmetric Encryption
  "Task IV: an initialization vector had to be instantiated." (Section 8.2)


name: task TLS
description:
  From Certificate Pinning: 
  "A solution for Task TLS would have been to select a secure TLS version to initialize the context." Section 8.2


name: task TM (trust manager)
description:
  From Certificate Pinning: 
  "Task TM: the server’s certificate had to be added to an empty custom trust manager replacing the default manager." Section 8.2


name: control
description:
  "In the control treatment, all security nudges on Stack Overflow were disabled. [...] 
  In all treatments, we restricted Stack Overflow search results to posts that contain a code example from the set of 10,558 code examples we extracted from Stack Overflow. Further, we applied a whitelist filter to restrict access to Stack Overflow in the Chrome browser. Any requests to different domains were redirected to the Stack Overflow search page." (Section 8.1)


name: nudge
description:
  "For the nudge treatment, we provided security warnings (Figure 2a) for insecure code examples, recommendations for secure snippets (Figure 2b) and recommendations lists attached to each warning (Figure 8). Further, security reminders were enabled. [...] 
  In all treatments, we restricted Stack Overflow search results to posts that contain a code example from the set of 10,558 code examples we extracted from Stack Overflow7. Further, we applied a whitelist filter to restrict access to Stack Overflow in the Chrome browser. Any requests to different domains were redirected to the Stack Overflow search page." Section 8.1


name: URL shortener
description:
  "For the URL shortening task, we checked how participants generated a short URL for a given long URL. We were mainly interested in whether participants relied on well established mechanisms such as message digest algorithms (e.g. the SHA1 or SHA2 family) or random number generators, or if they implemented their own algorithms. [...] We also considered the reversibility of the short URL as a security parameter (reversible was considered insecure). We did not incorporate whether solutions were likely to produce collisions (i. e. produce the same short URL for different input URLs) or the space of the URL-shortening algorithm (i. e. how many long URLs the solution could deal with) as security parameters: we felt that given the limited time frame, asking for an optimal solution here was asking too much" (Section 3.6)


name: credential storage - subtask password storage
description:
  "One score (password storage) considered how participants stored users’ passwords. Here, we were mainly interested whether our participants followed security best practices for storing passwords. Hence, we scored the plain text storage of a password as insecure. Additionally, applying a simple hash algorithm such as MD5, SHA1 or SHA2 was considered insecure, since those solutions are vulnerable to rainbow table attacks. Secure solutions were expected to use a salt in combination with a hash function; however, the salt needed to be random (but not necessarily secret) for each password to withstand rainbow table attacks. Therefore, using the same salt for every password was considered insecure. We also considered the correct use of HMACs [20] and PBKDF [18] as secure." (Section 3.6)


name: credential storage - subtask SQL input
description:
  "The second security score (SQL input) considered how participants interacted with the SQLite database we provided. For this evaluation, we were mainly interested whether the code was vulnerable to SQL injection attacks. We scored code that used raw SQL queries without further input sanitization as insecure, while we considered using prepared statements secure." (Section 3.6)


name: string encryption
description:
  "For string encryption, we checked the selected algorithm, key size and proper source of randomness for the key material, initialization vector and, if applicable, mode of operation. For symmetric encryption, we considered ARC2, ARC4, Blowfish, (3)DES and XOR as insecure and AES as secure. We considered ECB as an insecure mode of operation and scored Cipher Block Chaining (CBC), Counter Mode (CTR) and Cipher Feedback (CFB) as secure. For symmetric key size, we considered 128 and 256 bits as secure, while 64 or 32 bits were considered insecure. Static, zero or empty initialization vectors were considered insecure. For asymmetric encryption, we considered the use of OAEP/PKCS1 for padding as secure. For asymmetric encryption using RSA, we scored keys larger than or equal to 2048 bits as secure." (Section 3.6)


name: not dangerous misperception
description:
  Three other possibilities: 
  harmless misperception: perception insecure and scoring secure
  true positive: perception secure and scoring secure
  true negative: perception insecure and scoring insecure


name: Control–Minimal Information
description:
  "(1) Personalized ads: Acme can show personalized ads to your users. (2) Non-personalized ads: Acme will show only non-personalized ads to your users." (Section 3.1)


name: Data Processing Restrictions
description:
  "(1) Ads with unrestricted data processing: Acme can show personalized ads to your users based on a user’s past behavior, such as previous visits to sites or apps or where the user has been. (2) Ads with restricted data processing: Acme will show only non-personalized ads to your users based on contextual information, such as the content of your site or app, restricting the use of certain unique identifiers and other data." (Section 3.1)


name: User-Facing Descriptions
description:
  "(1) Ads with ‘Personalized Ads’ tag displayed to users: Acme can show personalized ads to your users. Users will see the ‘Personalized Ads’ tag next to the ‘Install’ button and the following text in your app description in the App Store or Google play “This app shows ads personalized based on your personal information.” (2) Ads with ‘Non-personalized Ads’ tag displayed to users: Acme will show only non-personalized ads to your users. Users will see the ‘Non-personalized Ads’ tag next to the ‘Install’ button and the following text in your app description in the App Store or Google play “This app shows ads not personalized based on your personal information.”" (Section 3.1)


name: Privacy Focused
description:
  "(1) Ads with lower user privacy: Acme can show personalized ads to your users based on their past behavior, such as previous visits to sites or apps or where the user has been. (2) Ads with higher user privacy: Acme will show only non-personalized ads to your users based on contextual information, such as the content of your site or app." (Section 3.1)


name: Revenue Focused
description:
  "(1) Ads with higher revenue: Acme can show personalized ads to your users, which may yield higher revenue than non-personalized ads. (2) Ads with lower revenue: Acme will show only non-personalized ads to your users, which may yield lower revenue than personalized ads." (Section 3.1)


name: Privacy vs. Revenue
description:
  "(1) Ads with higher revenue: Acme can show personalized ads to your users, which may yield higher revenue than non-personalized ads. (2) Ads with higher user privacy: Acme will show only nonpersonalized ads to your users which may increase your users’ privacy." (Section 3.1)


name: Encryption
description:
  "we re-used a subset of tasks from the Acar et al. study on the usability of cryptographic APIs [1]." 

  "We chose two symmetric encryption tasks: [...] using the [encryption] key to encrypt some plain text."
  (Section 6.4)


name: Storage
description:
  "we re-used a subset of tasks from the Acar et al. study on the usability of cryptographic APIs [1]." 

  "We chose two symmetric encryption tasks: generating an encryption key and storing it securely in a password-protected file"
  (Section 6.4)


name: priming
description:
  "Priming - Participants were explicitly told to store the user passwords securely in the Introductory Text and in the Task Description." (Section 3.1)


name: non-priming
description:
  "Non-priming - Participants were told the study is about API usability, but were not explicitly asked for secure password storage." (Section 3.1)


name: company deception
description:
  "In the previous study, the researchers presented themselves as a start-up company and revealed their academic aims only at the end of the programming task. To make the project as realistic as possible, the original task presented in [29, 30], needed to be changed from a university setting to a company setting. The authors created a fake company profile on Freelancer.com and a web presence for that company and shared that with the subjects." (Section 3.1)

  previous study was Naiakshina et al (2019)


name: "honest" study scenario
description:
  "One major change from the previous work was the public posting of the project on the freelancer platform as part of a scientific study." (Section 3.1)

  current study is Danilova et al (2020)


name: No specification
description:
  participant wrote no specification before the coding (was in the unprompted group, because prompted participants all had to write a specification)


name: NO (Name overloading)
description:
  "Name overloading [19] extends identifier renaming by using the same names for multiple different entities. We use name overloading as the first obfuscation technique in our study and abbreviate it by NO"


name: OP (opaque predicates)
description:
  "The obfuscation technique of opaque predicates (abbreviated as OP) can be used to alter the program’s execution flow. A predicate is called opaque if its outcome is known at obfuscation time but is hard to deduce by the reverse engineer [19]"


name: low
description:
  lowest number of websites visits was 20
  5 participants with lowest number of website visits ranged from 20 - 50


name: CA-Certbot
description:
  "Let’s Encrypt and Electronic Frontier Foundation’s Certbot aim to improve the TLS ecosystem by offering free trusted certificates (Let’s Encrypt) and by providing user-friendly support to configure and harden TLS (Certbot)." (Abstract)


name: CA-Traditional
description:
  "“CA-Traditional” (CA-Trad.) for the traditional manual CA approach. Although it would have been nice if we could have used the same web CA as used by Krombholz et al. to enable a more direct comparison with their work, we opted to use a more complex one that resembles the realistic workflow of acquiring a certificate from an existing CA. In particular, the method included ownership verification. There, a server owner has to prove that they are in possession of the server and domain by placing a specific file in the web folder or by responding with defined content to a request made by the CA" (Section 4.1)


name: A
description:
  AVAILABILITY. The sensor reading database is replicated, possibly using different database technologies. The sensor reading database host is shielded from the external network. Difficulty: hard

  For warm-up


name: B
description:
  ACCOUNTABILITY. Only authenticated users can perform
  actions, and they are logged in a tamper-proof (append-only) audit log. Difficulty: hard


name: C
description:
  CONFIDENTIAL DATA TRANSMISSION. Data in transit between the patient’s gateway and the PMS system is encrypted, for example using SSL. Difficulty: average


name: D
description:
  CONFIDENTIALITY. All actions that are performed by
  members of the community are checked against a policy. All necessary information to make an authorization decision is available to the component that makes the decision. Difficulty: average


name: E
description:
  DATA INTEGRITY. The system is designed so that external input is validated before it is used. Difficulty: easy


name: A x P
description:
  Task A (warm-up)

  scenario to prevent: A malicious person attempts to bring down the sensor reading database. This leads to the loss of received readings, and results in the unavailability of up-to-date information for the cardiologist.

  security objective and expected impact: AVAILABILITY. The sensor reading database is replicated, possibly using different database technologies. The sensor reading database host is shielded from the external network. Difficulty: hard

  Possible Pattern(s): Firewall; Load Balancer; Replicated System"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: B x P
description:
  Task B 

  "Scenario to prevent: A user of the system tries to avoid responsibility for his action, by tampering with the log files. 

  Security objective and expected impact ACCOUNTABILITY. Only authenticated users can perform actions, and they are logged in a tamper-proof (append-only) audit log. Difficulty: hard 

  Possible pattern(s): Authentication Enforcer; Audit Interceptor; Secure Logger"



  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: C x P
description:
  Task C:
  "Scenario to prevent: Unauthorized entities attempt to access sensor data about a patient (which is sensitive medical data). 

  Security objective and expected impact: CONFIDENTIAL DATA TRANSMISSION. Data in transit between the patient’s gateway and the PMS system is encrypted, for example using SSL. Difficulty: average 

  Possible pattern(s): Secure Pipe"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: D x P
description:
  Task D:

  "Scenario to prevent: A member of a patient’s personal health community tries to use the web interface to access information about a patient to which she is not related. 

  Security objective and expected impact: CONFIDENTIALITY. All actions that are performed by members of the community are checked against a policy. All necessary information to make an authorization decision is available to the component that makes the decision. Difficulty: average 

  Possible pattern(s): Authorization Enforcer"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: E x P
description:
  Task E:
  "Scenario to prevent: The web interface of the personal health community is targeted by a malicious entity, by providing malicious input in order to cause system malfunctioning or unauthorized access to patient information. 

  Security objective and expected impact: DATA INTEGRITY. The system is designed so that external input is validated before it is used. Difficulty: easy 

  Possible pattern(s): Input Guard"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: A x A
description:
  Task A (warm-up):
  scenario to prevent: A malicious person attempts to bring down the sensor reading database. This leads to the loss of received readings, and results in the unavailability of up-to-date information for the cardiologist.

  security objective and expected impact: AVAILABILITY. The sensor reading database is replicated, possibly using different database technologies. The sensor reading database host is shielded from the external network. Difficulty: hard

  Possible Pattern(s): Firewall; Load Balancer; Replicated System"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: B x A
description:
  Task B 

  "Scenario to prevent: A user of the system tries to avoid responsibility for his action, by tampering with the log files. 

  Security objective and expected impact ACCOUNTABILITY. Only authenticated users can perform actions, and they are logged in a tamper-proof (append-only) audit log. Difficulty: hard 

  Possible pattern(s): Authentication Enforcer; Audit Interceptor; Secure Logger"



  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: C x A
description:
  Task C:
  "Scenario to prevent: Unauthorized entities attempt to access sensor data about a patient (which is sensitive medical data). 

  Security objective and expected impact: CONFIDENTIAL DATA TRANSMISSION. Data in transit between the patient’s gateway and the PMS system is encrypted, for example using SSL. Difficulty: average 

  Possible pattern(s): Secure Pipe"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: D x A
description:
  Task D:
  "Scenario to prevent: A member of a patient’s personal health community tries to use the web interface to access information about a patient to which she is not related. 

  Security objective and expected impact: CONFIDENTIALITY. All actions that are performed by members of the community are checked against a policy. All necessary information to make an authorization decision is available to the component that makes the decision. Difficulty: average 

  Possible pattern(s): Authorization Enforcer"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: E x A
description:
  Task E:
  "Scenario to prevent: The web interface of the personal health community is targeted by a malicious entity, by providing malicious input in order to cause system malfunctioning or unauthorized access to patient information. 

  Security objective and expected impact: DATA INTEGRITY. The system is designed so that external input is validated before it is used. Difficulty: easy 

  Possible pattern(s): Input Guard"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: B
description:
  Protect sensitive information stored by the banking app on a mobile device. Expected pattern: Encrypted storage


name: C
description:
  Prevent read access to the transaction store from a publicly available system. Expected patterns: Demilitarized zone, Firewall


name: D
description:
  Make sure customers do not have to re-authenticate for every request. Expected patterns: Session, Session Timeout, Session Failover


name: E
description:
  An employee should be accountable for all actions performed on a customer account. Expected patterns: Secure Logger, Audit Interceptor


name: F
description:
  Ensure customers can only perform operations on their own accounts. Expected patterns: Input Guard, Authorization Enforcer


name: G
description:
  Protect the data that is transmitted between a mobile device and the bank. Expected pattern: Secure Pipe


name: A x Y
description:
  Task A: "(warm-up) Reduce the number of passwords that an employee has to remember (single sign on)."

  Y = yes (with patterns)


name: B x Y
description:
  Task B: "Protect sensitive information stored by the banking app on a mobile device. Expected pattern: Encrypted storage"

  Y = yes (with patterns)


name: C x Y
description:
  Task C:  "Prevent read access to the transaction store from a publicly available system. Expected patterns: Demilitarized zone, Firewall"

  Y = yes (with patterns)


name: D x Y
description:
  Task D: "Make sure customers do not have to re-authenticate for every request. Expected patterns: Session, Session Timeout, Session Failover"

  Y = yes (with patterns)


name: E x Y
description:
  Task E: "An employee should be accountable for all actions performed on a customer account. Expected patterns: Secure Logger, Audit Interceptor"  

  Y = yes (with patterns)


name: F x Y
description:
  Task F: "Ensure customers can only perform operations on their own accounts. Expected patterns: Input Guard, Authorization Enforcer"

  Y = yes (with patterns)


name: G x Y
description:
  Task G: "Protect the data that is transmitted between a mobile device and the bank. Expected pattern: Secure Pipe"

  Y = yes (with patterns)


name: A x N
description:
  Task A: "(warm-up) Reduce the number of passwords that an employee has to remember (single sign on)."

  N = no (without patterns)


name: B x N
description:
  Task B: "Protect sensitive information stored by the banking app on a mobile device. Expected pattern: Encrypted storage"

  N = no (without patterns)


name: C x N
description:
  Task C:  "Prevent read access to the transaction store from a publicly available system. Expected patterns: Demilitarized zone, Firewall"

  N = no (without patterns)


name: D x N
description:
  Task D: "Make sure customers do not have to re-authenticate for every request. Expected patterns: Session, Session Timeout, Session Failover"

  N = no (without patterns)


name: E x N
description:
  Task E: "An employee should be accountable for all actions performed on a customer account. Expected patterns: Secure Logger, Audit Interceptor"  


  N = no (without patterns)


name: F x N
description:
  Task F: "Ensure customers can only perform operations on their own accounts. Expected patterns: Input Guard, Authorization Enforcer"


  N = no (without patterns)


name: G x N
description:
  Task G: "Protect the data that is transmitted between a mobile device and the bank. Expected pattern: Secure Pipe" 

  N = no (without patterns)


name: end user
description:
  "For the non-expert users, our goal was to recruit a diverse sample of participants. Hence, we used three separate recruiting mechanisms to build our sample: mailing lists, online forums, and personal contacts for recruitment. We especially limited the number of students in our sample and refrained from recruiting computer science students or IT professionals."


name: administrator
description:
  "In contrast, the recruitment criteria for administrators was that they had to be in charge of administering systems and regularly-used services. We allowed both paid and voluntary work."


name: "intelligent" or added functionality
description:
  "“Intelligent” or added functionality of the system (such as the Amazon recommendations)" (Section 3.4)


name: secure
description:
  "For key generation, we checked key size and randomness. For key storage we checked if encryption keys were actually encrypted and if a proper encryption key was derived from the password we provided. For key derivation, we scored use of a static or empty salt, HMAC-SHA1 or below as the pseudorandom function, and less than 10,000 iterations as insecure. For the symmetric encryption task, participants had to select encryption parameters. Therefore, we scored the security of the chosen encryption algorithm, mode of operation, and initialization vector. We scored [...] AES as secure. We [...] scored CBC, CTR and CFB as secure. " (Section 6.6)


name: insecure
description:
  "For key generation, we checked key size and randomness. For key derivation, we scored use of a static or empty salt, HMAC-SHA1 or below as the pseudorandom function, and less than 10,000 iterations as insecure." 

  "Therefore, we scored the security of the chosen encryption algorithm, mode of operation, and initialization vector. We scored ARC2, ARC4, Blowfish, (3)DES, and XOR as insecure,"

  "We scored the ECB as an insecure mode of operation"

  "Static, zero or empty initialization vectors were scored insecure"


name: I/O
description:
  "(1) I/O, involving operations, such as reading and writing from/to streams and files, internal memory buffers, and networking activity;"


name: Crypto
description:
  "2) Crypto, involving functions handling cryptographic operations, such as encryption, decryption, and key agreement"


name: String
description:
  "(3) String, involving functions that perform string processing or manipulation, or queries and user input."


name: employed part-time
description:
  Selection from "Are you employed, self-employed or out of work? If you’re a student, please choose what you do besides studying."


