# rpl_variablelevel

name: connection task
description:
  "Network Connection. This provides a code snippet to be completed to establish a connection to a server. Participants were given the domain and query path of the URL, then requested to add a protocol to make this URL valid and establish the connection and get a return code status (i.e 200, 400 or 500). The goal of this task was to check if participants used a secure connection (HTTPS) to connect to the given host. The host supported both HTTP and HTTPS protocols. A corresponding test case was provided, which passes when the method connect returns the value of 200. " (Section 6.2)


name: encryption task
description:
  "Data Encryption. In this task, participants were asked to encrypt a given plain-text. Participants were expected to encrypt a string and return an array of bytes. The goal of this task is to see how knowledgeable developers are about cryptographic APIs. A corresponding test case was provided, which passes when the returned array of encryption method is not null and has a length greater than 16." (Section 6.2)


name: sqlite task
description:
  "SQLite Query. In this task, participants were asked to build a SQL query to retrieve the age of a given user name. In the skeleton app, we have already created a SQLite database with a predefined table named "users". Table "users" has 4 columns: id, name, age, password. The goal of this task is to see if participants are aware of SQL injection attacks. A corresponding test case was provided, which passes when this test case returns the correct age of a predefined user which has been inserted into the attached SQLite database." (Section 6.2)


name: student
description:
  recruited from 5 universities in Germany, subject not specified, but assuming computer science or similar


name: true (experienced)
description:
  True if participant has submitted more than 5 apps to Google Play Store or has more than 3 years of experience. Self-reported.


name: false (not experienced)
description:
  If participant has neither submitted more than 5 apps to the Play Store, nor has more than 3 years of experience.


name: yes (secure = 1)
description:
  "Only functional solutions received a security score. Depending on the task, we evaluated different security considerations" (Section 6.5)


name: no (secure = 0)
description:
  "Only functional solutions received a security score. Depending on the task, we evaluated different security considerations," (Section 6.5)


name: depth-first search (SD)
description:
  "select a piece of software and invest significant human resources into finding bugs in that software without any prior indication of success." (Abstract)


name: breadth-first search (SB)
description:
  "In breadthfirst search, humans perform minimal work to enable automated analysis on a range of targets before committing additional time and effort to research any particular one." (Abstract)


name: secure networking task
description:
  "This task addressed correct usage of HTTPS and TLS in the presence of X.509 certificate errors." (Section IV C)


name: secure storage task
description:
  "This task focused on secure storage of the user’s login ID and password for the remote server." (Section IV C)


name: least permissions task
description:
  "In this task, participants were asked to add functionality to dial a hard-coded customer support telephone number." (Section IV C)


name: Official only (official)
description:
  "Participants were only allowed to access websites within the official Android documentation" (Section IV B)


name: Stack Overflow Only (SO)
description:
  "Participants were only allowed to access questions and answers within Stack Overflow" (Section IV B)


name: Book only (book)
description:
  "Participants were only allowed to use two books: Pro Android 4 [23] and Android Security Internals [12]." (Section IV B)


name: free choice (free)
description:
  "Participants were allowed to use any web resources of their choice, and were also offered access to the two books used in condition book." (Section IV B)


name: 1 (functional)
description:
  "functionality score of 1 (if the participant’s code compiled and completed the assigned task)" (Section IV E)


name: 0 (non-functional)
description:
  "functionality score of 1 (if the participant’s code compiled and completed the assigned task) or 0 (if not)" (Section IV E)


name: 1 (secure)
description:
  "Based on these categories, each participant who completed a task was assigned a security score of 0 (insecure approach) or 1 (secure approach) for that task." (Section IV E)


name: professional developer
description:
  "For purpose of this analysis, we categorize 14 participants as professionals, including those who are currently or recently had been professional developers." (Section V C)


name: SME (small and medium sized enterprises)
description:
  "companies with 250 employees or fewer were tagged as small and medium-sized enterprises" (Section 4.3)


name: Secure Log (SL)
description:
  "This problem asks teams to implement two programs: one to securely append records to a log, and one to query the log’s contents." (Section 2.2)


name: Secure communication (SC)
description:
  "This problem asks teams to build a pair of client/server programs. These represent a bank and an ATM, which initiates account transactions (e.g., account creation, deposits, withdrawals, etc.)." (Section 2.2)


name: Multiuser database (MD)
description:
  "This problem asks teams to create a server that maintains a secure key-value store. Clients submit scripts written in a domain-specific language. A script authenticates with the server and then submits a series of commands to read/write data stored there. Data is protected by role-based access control policies customizable by the data owner, who may (transitively) delegate access control decisions to other principals." (Section 2.2)


name: task cipher
description:
  from Symmetric Encryption
  "a symmetric cipher had to be initialized by setting the algorithm, block mode and padding." Section 8.2


name: task IV (initialization vector)
description:
  from Symmetric Encryption
  "Task IV: an initialization vector had to be instantiated." (Section 8.2)


name: task TLS
description:
  From Certificate Pinning: 
  "A solution for Task TLS would have been to select a secure TLS version to initialize the context." Section 8.2


name: task TM (trust manager)
description:
  From Certificate Pinning: 
  "Task TM: the server’s certificate had to be added to an empty custom trust manager replacing the default manager." Section 8.2


name: control
description:
  "In the control treatment, all security nudges on Stack Overflow were disabled. [...] 
  In all treatments, we restricted Stack Overflow search results to posts that contain a code example from the set of 10,558 code examples we extracted from Stack Overflow. Further, we applied a whitelist filter to restrict access to Stack Overflow in the Chrome browser. Any requests to different domains were redirected to the Stack Overflow search page." (Section 8.1)


name: nudge
description:
  "For the nudge treatment, we provided security warnings (Figure 2a) for insecure code examples, recommendations for secure snippets (Figure 2b) and recommendations lists attached to each warning (Figure 8). Further, security reminders were enabled. [...] 
  In all treatments, we restricted Stack Overflow search results to posts that contain a code example from the set of 10,558 code examples we extracted from Stack Overflow7. Further, we applied a whitelist filter to restrict access to Stack Overflow in the Chrome browser. Any requests to different domains were redirected to the Stack Overflow search page." Section 8.1


name: URL shortener
description:
  "For the URL shortening task, we checked how participants generated a short URL for a given long URL. We were mainly interested in whether participants relied on well established mechanisms such as message digest algorithms (e.g. the SHA1 or SHA2 family) or random number generators, or if they implemented their own algorithms. [...] We also considered the reversibility of the short URL as a security parameter (reversible was considered insecure). We did not incorporate whether solutions were likely to produce collisions (i. e. produce the same short URL for different input URLs) or the space of the URL-shortening algorithm (i. e. how many long URLs the solution could deal with) as security parameters: we felt that given the limited time frame, asking for an optimal solution here was asking too much" (Section 3.6)


name: credential storage - subtask password storage
description:
  "One score (password storage) considered how participants stored users’ passwords. Here, we were mainly interested whether our participants followed security best practices for storing passwords. Hence, we scored the plain text storage of a password as insecure. Additionally, applying a simple hash algorithm such as MD5, SHA1 or SHA2 was considered insecure, since those solutions are vulnerable to rainbow table attacks. Secure solutions were expected to use a salt in combination with a hash function; however, the salt needed to be random (but not necessarily secret) for each password to withstand rainbow table attacks. Therefore, using the same salt for every password was considered insecure. We also considered the correct use of HMACs [20] and PBKDF [18] as secure." (Section 3.6)


name: credential storage - subtask SQL input
description:
  "The second security score (SQL input) considered how participants interacted with the SQLite database we provided. For this evaluation, we were mainly interested whether the code was vulnerable to SQL injection attacks. We scored code that used raw SQL queries without further input sanitization as insecure, while we considered using prepared statements secure." (Section 3.6)


name: string encryption
description:
  "For string encryption, we checked the selected algorithm, key size and proper source of randomness for the key material, initialization vector and, if applicable, mode of operation. For symmetric encryption, we considered ARC2, ARC4, Blowfish, (3)DES and XOR as insecure and AES as secure. We considered ECB as an insecure mode of operation and scored Cipher Block Chaining (CBC), Counter Mode (CTR) and Cipher Feedback (CFB) as secure. For symmetric key size, we considered 128 and 256 bits as secure, while 64 or 32 bits were considered insecure. Static, zero or empty initialization vectors were considered insecure. For asymmetric encryption, we considered the use of OAEP/PKCS1 for padding as secure. For asymmetric encryption using RSA, we scored keys larger than or equal to 2048 bits as secure." (Section 3.6)


name: not dangerous misperception
description:
  Three other possibilities: 
  harmless misperception: perception insecure and scoring secure
  true positive: perception secure and scoring secure
  true negative: perception insecure and scoring insecure


name: Control–Minimal Information
description:
  "(1) Personalized ads: Acme can show personalized ads to your users. (2) Non-personalized ads: Acme will show only non-personalized ads to your users." (Section 3.1)


name: Data Processing Restrictions
description:
  "(1) Ads with unrestricted data processing: Acme can show personalized ads to your users based on a user’s past behavior, such as previous visits to sites or apps or where the user has been. (2) Ads with restricted data processing: Acme will show only non-personalized ads to your users based on contextual information, such as the content of your site or app, restricting the use of certain unique identifiers and other data." (Section 3.1)


name: User-Facing Descriptions
description:
  "(1) Ads with ‘Personalized Ads’ tag displayed to users: Acme can show personalized ads to your users. Users will see the ‘Personalized Ads’ tag next to the ‘Install’ button and the following text in your app description in the App Store or Google play “This app shows ads personalized based on your personal information.” (2) Ads with ‘Non-personalized Ads’ tag displayed to users: Acme will show only non-personalized ads to your users. Users will see the ‘Non-personalized Ads’ tag next to the ‘Install’ button and the following text in your app description in the App Store or Google play “This app shows ads not personalized based on your personal information.”" (Section 3.1)


name: Privacy Focused
description:
  "(1) Ads with lower user privacy: Acme can show personalized ads to your users based on their past behavior, such as previous visits to sites or apps or where the user has been. (2) Ads with higher user privacy: Acme will show only non-personalized ads to your users based on contextual information, such as the content of your site or app." (Section 3.1)


name: Revenue Focused
description:
  "(1) Ads with higher revenue: Acme can show personalized ads to your users, which may yield higher revenue than non-personalized ads. (2) Ads with lower revenue: Acme will show only non-personalized ads to your users, which may yield lower revenue than personalized ads." (Section 3.1)


name: Privacy vs. Revenue
description:
  "(1) Ads with higher revenue: Acme can show personalized ads to your users, which may yield higher revenue than non-personalized ads. (2) Ads with higher user privacy: Acme will show only nonpersonalized ads to your users which may increase your users’ privacy." (Section 3.1)


name: Encryption
description:
  "we re-used a subset of tasks from the Acar et al. study on the usability of cryptographic APIs [1]." 

  "We chose two symmetric encryption tasks: [...] using the [encryption] key to encrypt some plain text."
  (Section 6.4)


name: Storage
description:
  "we re-used a subset of tasks from the Acar et al. study on the usability of cryptographic APIs [1]." 

  "We chose two symmetric encryption tasks: generating an encryption key and storing it securely in a password-protected file"
  (Section 6.4)


name: priming
description:
  "Priming - Participants were explicitly told to store the user passwords securely in the Introductory Text and in the Task Description." (Section 3.1)


name: non-priming
description:
  "Non-priming - Participants were told the study is about API usability, but were not explicitly asked for secure password storage." (Section 3.1)


name: company deception
description:
  "In the previous study, the researchers presented themselves as a start-up company and revealed their academic aims only at the end of the programming task. To make the project as realistic as possible, the original task presented in [29, 30], needed to be changed from a university setting to a company setting. The authors created a fake company profile on Freelancer.com and a web presence for that company and shared that with the subjects." (Section 3.1)

  previous study was Naiakshina et al (2019)


name: "honest" study scenario
description:
  "One major change from the previous work was the public posting of the project on the freelancer platform as part of a scientific study." (Section 3.1)

  current study is Danilova et al (2020)


name: No specification
description:
  participant wrote no specification before the coding (was in the unprompted group, because prompted participants all had to write a specification)


name: NO (Name overloading)
description:
  "Name overloading [19] extends identifier renaming by using the same names for multiple different entities. We use name overloading as the first obfuscation technique in our study and abbreviate it by NO"


name: OP (opaque predicates)
description:
  "The obfuscation technique of opaque predicates (abbreviated as OP) can be used to alter the program’s execution flow. A predicate is called opaque if its outcome is known at obfuscation time but is hard to deduce by the reverse engineer [19]"


name: low
description:
  lowest number of websites visits was 20
  5 participants with lowest number of website visits ranged from 20 - 50


name: CA-Certbot
description:
  "Let’s Encrypt and Electronic Frontier Foundation’s Certbot aim to improve the TLS ecosystem by offering free trusted certificates (Let’s Encrypt) and by providing user-friendly support to configure and harden TLS (Certbot)." (Abstract)


name: CA-Traditional
description:
  "“CA-Traditional” (CA-Trad.) for the traditional manual CA approach. Although it would have been nice if we could have used the same web CA as used by Krombholz et al. to enable a more direct comparison with their work, we opted to use a more complex one that resembles the realistic workflow of acquiring a certificate from an existing CA. In particular, the method included ownership verification. There, a server owner has to prove that they are in possession of the server and domain by placing a specific file in the web folder or by responding with defined content to a request made by the CA" (Section 4.1)


name: A
description:
  AVAILABILITY. The sensor reading database is replicated, possibly using different database technologies. The sensor reading database host is shielded from the external network. Difficulty: hard

  For warm-up


name: B
description:
  ACCOUNTABILITY. Only authenticated users can perform
  actions, and they are logged in a tamper-proof (append-only) audit log. Difficulty: hard


name: C
description:
  CONFIDENTIAL DATA TRANSMISSION. Data in transit between the patient’s gateway and the PMS system is encrypted, for example using SSL. Difficulty: average


name: D
description:
  CONFIDENTIALITY. All actions that are performed by
  members of the community are checked against a policy. All necessary information to make an authorization decision is available to the component that makes the decision. Difficulty: average


name: E
description:
  DATA INTEGRITY. The system is designed so that external input is validated before it is used. Difficulty: easy


name: A x P
description:
  Task A (warm-up)

  scenario to prevent: A malicious person attempts to bring down the sensor reading database. This leads to the loss of received readings, and results in the unavailability of up-to-date information for the cardiologist.

  security objective and expected impact: AVAILABILITY. The sensor reading database is replicated, possibly using different database technologies. The sensor reading database host is shielded from the external network. Difficulty: hard

  Possible Pattern(s): Firewall; Load Balancer; Replicated System"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: B x P
description:
  Task B 

  "Scenario to prevent: A user of the system tries to avoid responsibility for his action, by tampering with the log files. 

  Security objective and expected impact ACCOUNTABILITY. Only authenticated users can perform actions, and they are logged in a tamper-proof (append-only) audit log. Difficulty: hard 

  Possible pattern(s): Authentication Enforcer; Audit Interceptor; Secure Logger"



  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: C x P
description:
  Task C:
  "Scenario to prevent: Unauthorized entities attempt to access sensor data about a patient (which is sensitive medical data). 

  Security objective and expected impact: CONFIDENTIAL DATA TRANSMISSION. Data in transit between the patient’s gateway and the PMS system is encrypted, for example using SSL. Difficulty: average 

  Possible pattern(s): Secure Pipe"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: D x P
description:
  Task D:

  "Scenario to prevent: A member of a patient’s personal health community tries to use the web interface to access information about a patient to which she is not related. 

  Security objective and expected impact: CONFIDENTIALITY. All actions that are performed by members of the community are checked against a policy. All necessary information to make an authorization decision is available to the component that makes the decision. Difficulty: average 

  Possible pattern(s): Authorization Enforcer"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: E x P
description:
  Task E:
  "Scenario to prevent: The web interface of the personal health community is targeted by a malicious entity, by providing malicious input in order to cause system malfunctioning or unauthorized access to patient information. 

  Security objective and expected impact: DATA INTEGRITY. The system is designed so that external input is validated before it is used. Difficulty: easy 

  Possible pattern(s): Input Guard"


  P=PLAIN = "The PLAIN group had access to the same catalog, but without annotations and filtering options."


name: A x A
description:
  Task A (warm-up):
  scenario to prevent: A malicious person attempts to bring down the sensor reading database. This leads to the loss of received readings, and results in the unavailability of up-to-date information for the cardiologist.

  security objective and expected impact: AVAILABILITY. The sensor reading database is replicated, possibly using different database technologies. The sensor reading database host is shielded from the external network. Difficulty: hard

  Possible Pattern(s): Firewall; Load Balancer; Replicated System"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: B x A
description:
  Task B 

  "Scenario to prevent: A user of the system tries to avoid responsibility for his action, by tampering with the log files. 

  Security objective and expected impact ACCOUNTABILITY. Only authenticated users can perform actions, and they are logged in a tamper-proof (append-only) audit log. Difficulty: hard 

  Possible pattern(s): Authentication Enforcer; Audit Interceptor; Secure Logger"



  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: C x A
description:
  Task C:
  "Scenario to prevent: Unauthorized entities attempt to access sensor data about a patient (which is sensitive medical data). 

  Security objective and expected impact: CONFIDENTIAL DATA TRANSMISSION. Data in transit between the patient’s gateway and the PMS system is encrypted, for example using SSL. Difficulty: average 

  Possible pattern(s): Secure Pipe"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: D x A
description:
  Task D:
  "Scenario to prevent: A member of a patient’s personal health community tries to use the web interface to access information about a patient to which she is not related. 

  Security objective and expected impact: CONFIDENTIALITY. All actions that are performed by members of the community are checked against a policy. All necessary information to make an authorization decision is available to the component that makes the decision. Difficulty: average 

  Possible pattern(s): Authorization Enforcer"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: E x A
description:
  Task E:
  "Scenario to prevent: The web interface of the personal health community is targeted by a malicious entity, by providing malicious input in order to cause system malfunctioning or unauthorized access to patient information. 

  Security objective and expected impact: DATA INTEGRITY. The system is designed so that external input is validated before it is used. Difficulty: easy 

  Possible pattern(s): Input Guard"


  A= Annotated = "The ANNOTATED group teams had access to an annotated catalog, together with filtering functionality based on these annotations."


name: B
description:
  Protect sensitive information stored by the banking app on a mobile device. Expected pattern: Encrypted storage


name: C
description:
  Prevent read access to the transaction store from a publicly available system. Expected patterns: Demilitarized zone, Firewall


name: D
description:
  Make sure customers do not have to re-authenticate for every request. Expected patterns: Session, Session Timeout, Session Failover


name: E
description:
  An employee should be accountable for all actions performed on a customer account. Expected patterns: Secure Logger, Audit Interceptor


name: F
description:
  Ensure customers can only perform operations on their own accounts. Expected patterns: Input Guard, Authorization Enforcer


name: G
description:
  Protect the data that is transmitted between a mobile device and the bank. Expected pattern: Secure Pipe


name: A x Y
description:
  Task A: "(warm-up) Reduce the number of passwords that an employee has to remember (single sign on)."

  Y = yes (with patterns)


name: B x Y
description:
  Task B: "Protect sensitive information stored by the banking app on a mobile device. Expected pattern: Encrypted storage"

  Y = yes (with patterns)


name: C x Y
description:
  Task C:  "Prevent read access to the transaction store from a publicly available system. Expected patterns: Demilitarized zone, Firewall"

  Y = yes (with patterns)


name: D x Y
description:
  Task D: "Make sure customers do not have to re-authenticate for every request. Expected patterns: Session, Session Timeout, Session Failover"

  Y = yes (with patterns)


name: E x Y
description:
  Task E: "An employee should be accountable for all actions performed on a customer account. Expected patterns: Secure Logger, Audit Interceptor"  

  Y = yes (with patterns)


name: F x Y
description:
  Task F: "Ensure customers can only perform operations on their own accounts. Expected patterns: Input Guard, Authorization Enforcer"

  Y = yes (with patterns)


name: G x Y
description:
  Task G: "Protect the data that is transmitted between a mobile device and the bank. Expected pattern: Secure Pipe"

  Y = yes (with patterns)


name: A x N
description:
  Task A: "(warm-up) Reduce the number of passwords that an employee has to remember (single sign on)."

  N = no (without patterns)


name: B x N
description:
  Task B: "Protect sensitive information stored by the banking app on a mobile device. Expected pattern: Encrypted storage"

  N = no (without patterns)


name: C x N
description:
  Task C:  "Prevent read access to the transaction store from a publicly available system. Expected patterns: Demilitarized zone, Firewall"

  N = no (without patterns)


name: D x N
description:
  Task D: "Make sure customers do not have to re-authenticate for every request. Expected patterns: Session, Session Timeout, Session Failover"

  N = no (without patterns)


name: E x N
description:
  Task E: "An employee should be accountable for all actions performed on a customer account. Expected patterns: Secure Logger, Audit Interceptor"  


  N = no (without patterns)


name: F x N
description:
  Task F: "Ensure customers can only perform operations on their own accounts. Expected patterns: Input Guard, Authorization Enforcer"


  N = no (without patterns)


name: G x N
description:
  Task G: "Protect the data that is transmitted between a mobile device and the bank. Expected pattern: Secure Pipe" 

  N = no (without patterns)


name: end user
description:
  "For the non-expert users, our goal was to recruit a diverse sample of participants. Hence, we used three separate recruiting mechanisms to build our sample: mailing lists, online forums, and personal contacts for recruitment. We especially limited the number of students in our sample and refrained from recruiting computer science students or IT professionals."


name: administrator
description:
  "In contrast, the recruitment criteria for administrators was that they had to be in charge of administering systems and regularly-used services. We allowed both paid and voluntary work."


name: "intelligent" or added functionality
description:
  "“Intelligent” or added functionality of the system (such as the Amazon recommendations)" (Section 3.4)


name: secure
description:
  "For key generation, we checked key size and randomness. For key storage we checked if encryption keys were actually encrypted and if a proper encryption key was derived from the password we provided. For key derivation, we scored use of a static or empty salt, HMAC-SHA1 or below as the pseudorandom function, and less than 10,000 iterations as insecure. For the symmetric encryption task, participants had to select encryption parameters. Therefore, we scored the security of the chosen encryption algorithm, mode of operation, and initialization vector. We scored [...] AES as secure. We [...] scored CBC, CTR and CFB as secure. " (Section 6.6)


name: insecure
description:
  "For key generation, we checked key size and randomness. For key derivation, we scored use of a static or empty salt, HMAC-SHA1 or below as the pseudorandom function, and less than 10,000 iterations as insecure." 

  "Therefore, we scored the security of the chosen encryption algorithm, mode of operation, and initialization vector. We scored ARC2, ARC4, Blowfish, (3)DES, and XOR as insecure,"

  "We scored the ECB as an insecure mode of operation"

  "Static, zero or empty initialization vectors were scored insecure"


name: I/O
description:
  "(1) I/O, involving operations, such as reading and writing from/to streams and files, internal memory buffers, and networking activity;"


name: Crypto
description:
  "2) Crypto, involving functions handling cryptographic operations, such as encryption, decryption, and key agreement"


name: String
description:
  "(3) String, involving functions that perform string processing or manipulation, or queries and user input."


name: employed part-time
description:
  Selection from "Are you employed, self-employed or out of work? If you’re a student, please choose what you do besides studying."


# rpl_participantsample

problem_description:
  Very unclear about how many participants go into the calculations:
  "In all, we sent invitation emails to 210,854 developers and got 16 participants who volunteered to participate in our study and finished both writing code and filling in our exit survey."
  "We stopped recruiting when we had 24 students, due to budget constraints." -> 40 participants

  Table 4 (survey completion) : full functionality completed: 22, only telemetry completed: 35 -> 57 participants
  Table 5 (task completion): full functionality 3 tasks completed: 19, only telemetry 3 tasks completed: 19 -> 38 participants


problem_description:
  Sample size is unclear, because not the participants themselves were analysed here, but the code they produced, and the breaks they submitted. 
  There were 94 BIBIFI projects which were analyzed in this paper, 866 breaks, 42 additional vulnerabilities


problem_description:
  83 valid answers, but only 77 policies - numbers apply to policies?

  notes: participants came from German companies


# django_session

session_key: 40lobh0dk2rt28cmie7pxsk300yqzp6z
expire_date: 2022-09-01 12:10:44.676794
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oOeMG:VANutbOX9_lHyzhibTdTylqm-U2GRnjzE89j_2sLX8o


session_key: k1394sv07n72imyy0in8ler3xjwur1jq
expire_date: 2022-09-02 13:14:40.508564
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oP1pg:497J7HN0VUlA5-4ZT00MFzvJL_1kDNYdeNY1VHwtN6o


session_key: 99i14y0w7yj9xyiy8oepc5wbmvf85jx0
expire_date: 2022-09-05 09:05:20.238118
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oQ3N2:QCkH6c-I6ZJMaDLO0V4DpEqJgV3OG_g_NSvxtaMuF-Y


session_key: evdsocpaviwft80a8c4qm2k3pqhr54k4
expire_date: 2022-09-06 07:58:19.723189
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oQOnj:W7yEPJYSt1fCvyVag8uPX18U7fm51i6x6vqLUmmKOs8


session_key: w867g0idchps1a95lud5rlepr179yw1a
expire_date: 2022-09-07 09:13:20.249873
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oQmRs:gJhJj0smm4U5Eu2pcTVSB7bbeUOXVADYejgHNHyMOw8


session_key: i1h54h8orilimn0octq7nr6j75cch506
expire_date: 2022-09-08 09:40:20.393227
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oR9LY:RwWyjRXRZO8qRf3TzPWjt42ZaBmf1JeOuHisbphh3VA


session_key: 59ip7loqltd5glvbu1r6zay63q1n1ye0
expire_date: 2022-09-09 14:27:27.796440
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oRaIx:bgmQrGRAgtTsW8q0wajkruMsxvalowHL5vm20XBMLLo


session_key: ck2so3h9u54d9vz1fq13n9buppunii6n
expire_date: 2022-09-10 13:35:04.056891
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oRvxo:nwQfmuUw_hDxI6tVgyCDVJCVqhOyTw1MnvEROoSxlUI


session_key: l9c1dcrzoivh6gukqixs6jjz3udet0m6
expire_date: 2022-09-10 16:34:00.371853
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oRyky:NRqd7yV_JmQaGhg8nFhC5mjKGmef2V4LJZ9PES4N0zM


session_key: bo0sxsd7r3r4gu33ukgo5x4cwj9glxap
expire_date: 2022-09-12 07:39:44.665095
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oSZN2:ujG_q3YYON2n2y8xgOcDqk6ZVe5b8YPjAdQwYQ7kflc


session_key: 22ee1zmhiyw578ko7byssig7d6vx2zul
expire_date: 2022-09-12 09:34:09.780246
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oSb9l:duQADcfP7CZLbpz3Zqf1vXBoEu0qbNSpUCdJWKTqO1E


session_key: 4goj604ns9b75ey501f9r6d3c3r0vmsd
expire_date: 2022-09-13 07:29:14.359640
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oSvgQ:2-skHcYYqYH-Ge4UUvi3Gq8NpWyqZiIy0zv1T_tXdEE


session_key: 0m1hcv6646te9x10uj8frwsq1sb1ak3t
expire_date: 2022-09-14 07:53:56.890228
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oTIXs:1cvv2ntWlUGUaCnLlPwiAs9rzsTivdDJQAqnxFaLzrw


session_key: ibzmw4urr52gsw4a3mm9ku353xtoivk4
expire_date: 2022-09-22 08:51:01.366761
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oWDFV:QDttDSViCA_-VoVTzg2aeR4fZypPP20xp5A4LMlvrbI


session_key: pl170g0wmzmcgsdwdela6sy2n8vpm14n
expire_date: 2022-10-21 14:10:36.528430
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1ogo3g:nBwFaWFjHtAfnPxi5iUhp2ao4ZyyI3xSnDAar1sVr6g


session_key: 1q2dck30f3zfbvd7kxss7n9hxbm5h7o9
expire_date: 2022-10-28 12:28:12.981944
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1ojJnQ:9Dl0tRp14uH11_-WOPvGEJ8tFlyvkXFKoc7o-QWMcfc


session_key: hju08f5recpfp1qtd9y8wjpe1zgz1jmh
expire_date: 2022-11-04 09:27:46.338509
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oloJe:djGEKbPKQHLqxFETVWlGFYSmyVqae2zJ-THsXC0EpxA


session_key: 1erdpftisyss267faac9rj3mtcd8q6wd
expire_date: 2022-11-10 18:32:23.500408
session_data:
  .eJxVjEEOwiAQRe_C2pBxwlBw6d4zkIEBqRqalHbVeHdt0oVu_3vvbyrwutSw9jyHUdRFoTr9bpHTM7cdyIPbfdJpass8Rr0r-qBd3ybJr-vh_h1U7vVbs1jh5BMW8DaSt56sM4bEJSS0gGycJIe5DNZBIYLBgCMiNFHwDOr9AdYFNsI:1oo7fz:4NBVzg5riZfxHHgp3-_P_6tWGyjXmBNe3cDpmeGGFco


session_key: wtmar89tf5uw1r44t3c48hjde8oqlu7p
expire_date: 2022-11-13 21:03:12.259344
session_data:
  .eJxVjEEOwiAQRe_C2pBxwlBw6d4zkIEBqRqalHbVeHdt0oVu_3vvbyrwutSw9jyHUdRFoTr9bpHTM7cdyIPbfdJpass8Rr0r-qBd3ybJr-vh_h1U7vVbs1jh5BMW8DaSt56sM4bEJSS0gGycJIe5DNZBIYLBgCMiNFHwDOr9AdYFNsI:1opFSa:Em-CWwe_SzphsfMlwlfxZSMfEpKHjkpB_rBGFC7FjME


session_key: o1zradgasmgc1kzwjunvwpdzt3lkpll6
expire_date: 2022-11-22 14:10:49.807736
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1osPJR:4lHCwBulJ8XCkKQNhHkwk4M1QmrFB_dibril9CyXhcI


session_key: bh3ibifb68ty29xan9yvra0anw09j2ff
expire_date: 2022-11-28 09:50:51.701461
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1ouW79:dHGows72M17qmgcllv6pER0-a2fmxqZ-Yc5tDO9eALs


session_key: oajpg5gx7lwkw0v9jgj7kpdudiihobev
expire_date: 2022-11-30 12:28:38.985724
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1ovHWw:Zb8ZxW-GH0bn3y4IblI8e7BSCfCSY0pCmMNC5oIPORM


session_key: 83ssuahr29uqdwi6kyfw5hgo7bwund9a
expire_date: 2022-12-01 15:30:56.265094
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1ovgqu:Dr_S4DWwUluwlbKjOvaOsdJ74S1juvOGDZh8vmaR_H8


session_key: mwxjzdfhhhneba6i0b3mojpgoguqyots
expire_date: 2022-12-06 13:30:26.857924
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1oxTM2:dAm4w_EwWuhR5ZQda4eMzvsIZtOq-F14tIgIzjLKHvk


session_key: z386gk2hyipohzfy7o158x1a9s39b33x
expire_date: 2022-12-20 16:15:31.627160
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1p2abT:SEGuwUdkAoCalJg1ux3uUUkSJhw9y0J87fiLtWhWfaY


session_key: zvx9ng597dh4ka4wwej5y5givpccsi5z
expire_date: 2022-12-27 16:08:41.939294
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1p57ph:l-NajJUq3zauw8GSCi_8uQm2_BaBH50uqydRkIIbu7c


session_key: bm2ro8w2fe5lav02r93menuwc73jr9bi
expire_date: 2022-12-29 13:36:01.519656
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1p5oP3:vJE_J71sbb0Ken20DAfFCj4xCFQtrcpDiAcIPBpMsSw


session_key: kjfk88ohjid91mfc7l9sb8e0fzdt0soy
expire_date: 2023-02-07 15:47:33.964067
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pKLWH:x1fgbBseB3b00VkvyrE1pSIwydtm8VyKGhj3DLQSb2w


session_key: 066jsy5o05dx4szq7oqmf0oot57lr9fm
expire_date: 2023-02-08 09:24:06.586478
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pKc0k:yK08iR14Vg8GmzkKzQ0oYHyeG918Nm6EtAy9ODH5TpQ


session_key: r9fjk93bkjk0q4xmdtrvsxjnbtqy5fm3
expire_date: 2023-02-13 10:42:09.982829
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pMRc1:HXDEHSf6lOM814UGPUpVYHlVCkdfs8NP0tQ9b49OZ9I


session_key: r2ek1lumfnb7jq31s64f41bn5wbfk0q8
expire_date: 2023-02-14 14:07:06.544544
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pMrHu:dVi9xXAOR47sfsQwHTr7TFWArSC3YLycahiauKj9uTU


session_key: ji9iicjwes0d3fea1n8x8w1muj3724kh
expire_date: 2023-02-15 13:18:46.370057
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pND0g:dg3cGWJ28VuFvoZvyicOATzKf5DfNwqjLn3LJoI7pGs


session_key: khx6frzmk17pf1smtan5xg6hx55g80u6
expire_date: 2023-02-16 15:24:37.120260
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pNbS1:rFqbgUVQnWsliEa4L0ziTKWyvjX3F-y7tsSJOzoe9x0


session_key: 9onfwff7ekf72wq4s818afwfh6tsnyw4
expire_date: 2023-02-17 16:36:46.099785
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pNz3O:Vwi1LlxSGdkZOq2qQOmXp1zJM91z0ITff_ks87KVGdg


session_key: 9nibb5bunwap5i5v1y29ru0icoiuii2v
expire_date: 2023-02-21 10:10:00.555239
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pPKvI:2Aaa5evhq15fKQgmmLt5Kps1rmB6l9DDGLQjHqU2pjk


session_key: nfavbgjaejrxa7q0be1b8damxk7lgyqa
expire_date: 2023-02-22 14:07:43.809802
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pPl6t:GkVlfY0YNArkN8IJBR1YUgYPkulUbTPgU3mIQ0QC36M


session_key: ahsp6g2x2kk0qq3a2e24bjxbqf8b5mmz
expire_date: 2023-03-01 10:54:04.214232
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pSFQK:FzPwdykY-9-5eYsT66vtD8NH7gQV8QFsZZinaRUyhYU


session_key: m6y26bagk7t9xv5ggpdx0cykf2r0k50t
expire_date: 2023-03-02 08:09:05.559437
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pSZKD:4j4ZKFHzVLisx89HsXaZmR4xH9zhO0i0vAR4fn7CBpk


session_key: n9e9hlmxzucy5lco7vkgyjkv525lajw0
expire_date: 2023-03-10 15:36:21.744157
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pVa7R:ICZ5jYOS49SpV4LtPHq6f34CrNc2iq4p8H5-ICx1q74


session_key: u2c5ckr6knf1jrd66nrdie7m1bljxvhf
expire_date: 2023-03-17 11:08:05.658740
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1pY3Gf:0rEyL8RaTYvgFv0HhqrQdQGyTlYLb0BVIACh3lGlO-4


session_key: x6xyw4risc4s23dho9o1ywdwrbps1lco
expire_date: 2023-08-10 11:23:15.599093
session_data:
  .eJxVjDsOwyAQBe9CHSHAfFOm9xnQLgvBSYQlY1dR7h5bcpG0b2bem0XY1hq3npc4EbsyyS6_G0J65nYAekC7zzzNbV0m5IfCT9r5OFN-3U7376BCr3sdgoXisIgMtmgTpHNIgCis1LpkRYrQUEoggy8medIeBJDNgxzUHrDPFwkqOMI:1qOz5P:m-aEfoBDGdziHTiG4d3SDOmOPfda2vss4Pxqns-3dRI


# rpl_quantitativeanalysismethod

name: mixed logistic regression
problem_description:
  "When we considered results on a per-task rather than a per-participant basis, we used a mixed model that adds a random intercept to account for multiple tasks from the same participant." (Section 7.3) 

  Only final model is reported, but they "considered candidate models consisting of the required factors mode and task , the participant random intercept, plus every possible combination of the optional variables." (Section 7.3)


name: Unclear analysis method
problem_description:
  Unclear analysis method, although based on the model for security correctness, this is probably also logistic regression.

  Unsure whether only the mandatory factors/IVs (=mode, task type) were considered for functionality, or also the others (group, experienced, security background). Based on reporting, the former seems to be the case and I have entered data under this assumption.

  "We observed no statistically significant difference in the number of functional solutions between each task, and between groups (cf. Figure 16). Developers and students with FixDroid’s support do not perform significantly better compared to participants with only the telemetry features, in terms of functional correctness." (Section 7.4)


name: Mann-Whitney U test
original_effect_value: error
problem_description:
  "We use this test to compare the means of survey responses and conform to the necessary assumptions [24, §1.2] except that each entry is an independent trial. This is violated because we sample each subject multiple times over the course of each method. We expect there is variation within a single subject’s responses and thus we conducted multiple samplings. [...] We choose Mann-Whitney mainly because it is a non-parametric test with minimal assumptions about the data’s distribution and allows us to test the signed difference of means between two groups: SD and SB." (Section 5.1)

  wrong test


name: Mann-Whitney U test
original_effect_value: error
problem_description:
  "After the second week, we compared within-subjects on the team that transitioned from SD to SB." (Section 5.1) - here even more repeated measures because members had tried both methods (and multiple measurements per method), should probably be Wilcoxon Signed Rank tests

  "We use this test to compare the means of survey responses and conform to the necessary assumptions [24, §1.2] except that each entry is an independent trial. This is violated because we sample each subject multiple times over the course of each method. We expect there is variation within a single subject’s responses and thus we conducted multiple samplings. [...] We choose Mann-Whitney mainly because it is a non-parametric test with minimal assumptions about the data’s distribution and allows us to test the signed difference of means between two groups: SD and SB." (Section 5.1)

  wrong test


name: Mann-Whitney U test
original_effect_value: error
problem_description:
  should probably be wilcoxon signed rank test (Repeated measures), authors report MWU-test

  wrong test


name: Mann-Whitney U test
original_effect_value: error
problem_description:
  "subsample": both team's findings (Team A and Team B combined)

  wrong test - should take repeated measures into account


name: Bernoulli trial
problem_description:
  unclear if B is test statistic or p-value, based on rest of tests in paper, am assuming p-value and entering as such (equal problem for remaining Bernoulli trials from this paper)


name: Bernoulli trial
problem_description:
  unclear if B is test statistic or p-value, based on rest of tests in paper, am assuming p-value and entering as such


name: Fishers exact test (omnibus)
original_effect_value: error
problem_description:
  Fishers exact tests with multiple IVs - not possible to enter all data. Adding result stats to compensate.


name: fishers exact test
original_effect_value: error
problem_description:
  Unsure if the test is just about the use of update management tools, or about the main sources of information (7 groups)


name: fishers exact test
original_effect_value: error
problem_description:
  subsample: non-prompted group

  2x3 test

  more groups than can be entered: 
  non-prompted:
  plain text: 6 (2 found, 4 not found)
  MD5: 7 (0 found, 7 not found)
  Base64: 8 (0 found, 8 not found)


name: fishers exact test
original_effect_value: error
problem_description:
  subsample: prompted group

  2x3 test, in prompted group

  more groups than can be entered: 
  prompted:
  plain text: 8 (4 found, 4 not found)
  MD5: 7 (5 found, 2 not found)
  Base64: 8 (3 found, 6 not found)


name: chi square test
original_effect_value: phi
problem_description:
  difficult to define variables, variables was whether a project had vulnerabilities of this type or not

  ?n participants = n teams? -> 94
  two-by-two chisq? -> df=1?


name: chi square test
original_effect_value: phi
problem_description:
  difficult to define variables, variables was whether a project had vulnerabilities of this type or not

  ?n participants = n teams? -> 94
  two-by-two chisq? -> df=2?


name: chi square test
original_effect_value: phi
problem_description:
  Dv desired outcome nicht eindeutig. "Hard to find" für angreifer ist auch schwieriger zu beheben.

  ?n participants = n teams? -> 94
  two-by-two chisq? -> df=1?


name: chi square test
original_effect_value: none
problem_description:
  subsample: TM task
  "Participants from the nudge group provided 76.9% secure solutions for TM, while 66.7% of the respective solutions submitted by the control group were insecure."


name: chi square test
original_effect_value: none
problem_description:
  Subsample Key task:
  "Participants from the nudge group provided 84.6% secure solutions for Key, while 60.0% of the respective solutions submitted by the control group were insecure."


name: chi square test
original_effect_value: none
problem_description:
  Subsample: Cipher Task:
  " For Cipher, the nudge treatment performs very well, but only slightly better than the control treatment, as both achieved 86.7% and 81.8% secure solutions, respectively." (Section 8.4)


name: chi square test
original_effect_value: error
problem_description:
  Subsample: Task IV:
  "For IV the nudge results are less desirable with 46.2% secure solutions, while performing better but not significantly (p < 0.077)" 
  info secure in control not given


name: chi square test
original_effect_value: error
problem_description:
  Subsample TLS: 
  "For TLS, the nudge results are less desirable with 38.5% secure solutions, while performing better but not significantly (p < 0.53)" Section 8.4

  info secure in control not given


name: logistic regression
problem_description:
  used general credentials storage variable level, not the the split up scores (so actually 3 levels here)


name: logistic regression
problem_description:
  unsure about the positive outcome: 
  "Odds ratios (O.R.) indicate relative likelihood of being insecure." Table 5 caption


name: Mann-Whitney U test
original_effect_value: none
problem_description:
  group sizes not totally clear; assuming both types of accounts (company wide vs e-mail) count together:

  company wide:
  "From the 68 participants who use company-wide accounts, we were able to extract 64 password composition policies. [...] Twenty-nine (45%) participants were part of the password policy creation process and 15 (23%) stated that the PCP was created by their predecessor."

  e-mail: 
  "We received 15 (18% of all responses) answers and were able to extract 13 password composition policies. Though we did not ask whether the PCPs are given by an email provider, six (46%) indicated that they were part of the creation process."


name: Mann-Whitney U test
original_effect_value: error
problem_description:
  "17 copied and pasted code. Of these, 12 created a secure solution. The surprising aspect is that all secure solutions come from participants who copied and pasted security code. Not a single "non-copy/paste" participant achieved security."


name: multiple linear regression
problem_description:
  unsure about number of factors (how many factors does an interaction add - not all are reported in the table, because coefficients can be estimated from reported ones)


name: chi square test
original_effect_value: error
problem_description:
  unclear how many organization types are involved in the test
  descriptive stats:
  mean: direct supervisor understands what operators do
  organization without IT background: mean: 0.534
  government sector: mean:0.474
  IT service provider: 0.832
  IT enterprise: 0.810


name: chi square test
original_effect_value: error
problem_description:
  unclear, whether employed are lumped together in 2x2 test

  descriptive statistics:
  employed full-time: 0.784
  employed part-time: 0.864
  freelance: 0.278


name: chi square test
original_effect_value: error
problem_description:
  unclear test type (non-parametric correlation seems to fit better?)

  descriptive stats:
  mean trust in tools etc. per group:
  < 1 year experience: 0.400
  1-3 years experience: 0.05
  3-5 years experience: -0462
  >20 years experience: -1.032


name: chi square test
original_effect_value: error
problem_description:
  unclear whether the comparison is employed (lumping full-time and part-time together) vs. freelancing 

  descriptive stats: 
  mean importance judgments:
  freelancers: 0.867
  full-time employed: 1.325
  part-time employed: 1.400


name: fishers exact test
original_effect_value: none
problem_description:
  group sizes not given exactly (half / half for 31 participants), used 16 as group size for calculations

  subsample: CA-Certbot


name: Mann-Whitney U test
original_effect_value: error
problem_description:
  test used doesn't account for dependent samples (within groups design)

  subsample: computer science students


name: mc nemars test
original_effect_value: error
problem_description:
  subsample: all vulnerabilities

  "We considered the code review experiment output for the test. As output, we found that the probability of success is 0.25" (Section IV C)

  in total: found 20
  not found: 60


name: mc nemars test
original_effect_value: error
problem_description:
  subsample: IVQI

  "the probabilities of success for IVQI are  0.13"

  in additional review
  found 7
  not found 45


name: multilevel logistic regression
problem_description:
  Table 2 and text results don't fit together (possible mix-up between API usage model and cyclomatic complexity model) - data entered here correspond to the table


  "To control for family-wise type-I error inflation due to multiple dependent models (i.e., models that share the same dependent variable), we applied Bonferroni correction for the threshold of the p-value to determine statistical significance in these exploratory analyses (p < 0.025)."


name: multilevel logistic regression
problem_description:
  Table 2 and text results don't fit together (possible mix-up between API usage model and cyclomatic complexity model) - data entered here correspond to the table

  "To control for family-wise type-I error inflation due to multiple dependent models (i.e., models that share the same dependent variable), we applied Bonferroni correction for the threshold of the p-value to determine statistical significance in these exploratory analyses (p < 0.025)."


name: ordinal logistic regression
problem_description:
  we applied Bonferroni correction on the threshold of the p-values (p < 0.008 for H3)

  (corrected p-values are currently not possible for factor infos)


name: ordinal logistic regression
problem_description:
  data on signficance taken from table (text partially gives more accurate p-values, but is not always clear to which test they apply)


