1naresh
Array
(
    [urn:ac.highwire.org:guest:identity] => Array
        (
            [runtime-id] => urn:ac.highwire.org:guest:identity
            [type] => guest
            [service-id] => ajnr-ac.highwire.org
            [access-type] => Controlled
            [privilege] => Array
                (
                    [urn:ac.highwire.org:guest:privilege] => Array
                        (
                            [runtime-id] => urn:ac.highwire.org:guest:privilege
                            [type] => privilege-set
                            [privilege-set] => GUEST
                        )

                )

            [credentials] => Array
                (
                    [method] => guest
                )

        )

)
1nareshArray
(
    [urn:ac.highwire.org:guest:identity] => Array
        (
            [runtime-id] => urn:ac.highwire.org:guest:identity
            [type] => guest
            [service-id] => ajnr-ac.highwire.org
            [access-type] => Controlled
            [privilege] => Array
                (
                    [urn:ac.highwire.org:guest:privilege] => Array
                        (
                            [runtime-id] => urn:ac.highwire.org:guest:privilege
                            [type] => privilege-set
                            [privilege-set] => GUEST
                        )

                )

            [credentials] => Array
                (
                    [method] => guest
                )

        )

)

Table 3: Interrater (between reviewers) κ statistic by top choice groups
Reviewer 1 2 3
Overall 
 1 1.0 0.35 0.40
 2 0.35 1.0 0.16
 3 0.40 0.16 1.0
Top choice
 1 1.0 0.47 0.21
 2 0.47 1.0 0.028
 3 0.21 0.028 1.0
Top 2 choices
 1 1.0 0.37 0.37
 2 0.37 1.0 0.27
 3 0.37 0.27 1.0
Top 3 choices
 1 1.0 0.31 0.42
 2 0.31 1.0 0.12
 3 0.42 0.12 1.0
Reviewer	1	2	3
Overall
1	1.0	0.35	0.40
2	0.35	1.0	0.16
3	0.40	0.16	1.0
Top choice
1	1.0	0.47	0.21
2	0.47	1.0	0.028
3	0.21	0.028	1.0
Top 2 choices
1	1.0	0.37	0.37
2	0.37	1.0	0.27
3	0.37	0.27	1.0
Top 3 choices
1	1.0	0.31	0.42
2	0.31	1.0	0.12
3	0.42	0.12	1.0