We conducted a human subject study to evaluate the effectiveness of the refinement patterns in designing an STS. Our study was declared exempt by our university’s Institutional Review Board (IRB). We selected 32 graduate computer science students as study participants, and created two groups (Control and NoReST) balanced in terms of familiarity with conceptual modeling and software engineering industry experience. Each participant provided informed consent; upon completion, a participant received a payment of 20 USD.

We follow a one-factor two-alternatives design. Our study has three phases in which participants work as follows.

• Phase 1: Learn. Learn and define the requirements and norm specification for a healthcare privacy scenario.
• Phase 2: Design. Define the requirements and norm specification for a healthcare security scenario.
• Phase 3: Maintain. Comprehend and maintain the requirements and norm specification for an academic security access control scenario.

In each phase, participants in the NoReST group are guided with refinement patterns, whereas participants in the Control group are given a basic definition of refinement. Participants record the completion times for each phase, and complete a post-study survey at the end of the study regarding the helpfulness of patterns in their design. The first two authors designed an oracle solution for each phase, and marked the participants' designs accordingly.

• Coverage of design: Fraction of norms in the oracle that are stated by the participants in each phase. Higher is better.
• Correctness of design: Fraction of participant-stated norms that occur in the oracle for each phase. Higher is better.
• Time to design: Time in minutes recorded by participants to complete each phase. Lower is better.
• Ease of design: Subjective ratings provided by the participants via the post-study survey on a Likert scale (1–5, where 1 corresponds to very hard, and 5 corresponds to very easy) for each phase.

• H1: NoReST produces specifications with greater coverage than Control.
• H2: NoReST produces specifications with greater correctness than Control.
• H3: NoReST produces specifications faster than Control.
• H4: NoReST helps produce specifications more easily than Control.

We analyze the specifications designed by the participants in the Control and NoReST groups, and conduct 2-tailed t test to compare the means.

• H1 and H2 are rejected for the general subject population as there are no significant differences in coverage or accuracy between NoReST and Control. We perform further analysis by grouping the subjects by their prior experience in modeling and norms. H1 and H2 hold for participants with low experience in conceptual modeling or no prior knowledge of norms.
• H3 is rejected as there is no significant gain in time for the NoReST group. Further analysis suggests that for participants with no prior knowledge of norms, NoReST takes more time in the learning phase, but less time in designing and maintenance phases.
• H4 is rejected as perceived difficulty is found to be similar. However, participants in the pattern group strongly felt that patterns were helpful in their design (μ=4.4/5), and stated that an automated tool to generate alternative refinements would improve accuracy (μ=4.27/5). Moreover, both study groups employed a normative approach to capture requirements, and achieve good coverage and accuracy results (excluding minor errors regarding logic representation), which suggests that norms provide a promising way of capturing requirements. We did not employ an automated tool for NoReST, because we wanted to evaluate the usefulness of patterns without coupling

• Pre-participation survey
• Tutorial on Norms
• Tutorial on Refinement Patterns
• Phase 1 Scenario
• Phase 2 Scenario
• Phase 3 Scenario
• Post-participation survey for NoReST group
• Post-participation survey for Control group
• Study website for NoReST group
• Study website for control group
• Analysis