Abstract
There is growing interest in using data analytics to identify patterns or trends in the work or behaviour of a court or judge. As more analytics-generated information about courts and judges is pumped into public discourse, questions arise about what it takes for stakeholders — both those producing or generating judicial analytics outputs, and those relying on and attempting to make sense of analytics outputs — to properly engage with that data. In this paper, we identify two related but distinct kinds of literacy required for meaningful and responsible engagement with judicial analytics outputs: (1) data and statistical literacy, that is, familiarity and facility with the basic concepts and methods required to effectively engage in (and with) quantitative research; and (2) domain literacy, which requires users to understand the unique features of the domain or area that an analytics output reflects in order to properly interpret that output with full regard for the nuances of context. Using data from and about the Supreme Court as a case study, we consider the ways that both types of literacy are essential to ensure that judicial analytics outputs further the public’s understanding of the judiciary rather than mislead, confuse or distract.
Citation Information
McGill, Jena and Salyzyn, Amy.
"Beyond the Numbers: Statistical and Data Literacy, Domain Literacy and Supreme Court of Canada Data Analytics."
The Supreme Court Law Review: Osgoode’s Annual Constitutional Cases Conference
115.
(2024).
DOI: https://doi.org/10.60082/2563-8505.1457
https://digitalcommons.osgoode.yorku.ca/sclr/vol115/iss1/15
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
References
1 For academic articles that apply data analytics to SCC data, see, for example, Wolfgang Alschner & Isabelle St-Hilaire, "Using Network Citation Analysis to Reveal Precedential Archetypes at the Supreme Court of Canada", Ottawa Faculty of Law Working Paper No. 2023-10 (March 1, 2023), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4374976; Sean Rehaag, "Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal", Refugee Law Lab Working Paper (January 11, 2023), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4322881; Jon Khan, "If I Had More Time, Would I Have Written a Shorter and Faster Decision? An Empirical Examination of the Evolution of Trial Court Decisions" (2022) 45:2 Dal. L.J. 427; Mike Madden, "Stating it Simply: A Comparative Study of the Quantitative Readability of Apex Court Decisions from Australia, Canada, South Africa, the United Kingdom, and the United States" (2021) 23 N.C. J.L. & Tech. 270; Sean Rehaag, "Judicial Review of Refugee Determinations (II): Revisiting the Luck of the Draw" (2019) 45 Queen's L.J. 1. For online articles, see, for example, Simon Wallace, "Revisiting (computationally) an earlier study regarding SCC citation practices" Obiter.ai (April 19, 2023), online: https://obiter.ai/blog/posts/2023-04-19-scc-citations/; Paul-Erik Veel, "Factors Influencing the Likelihood of Winning an Appeal at the Supreme Court of Canada" Litigate.com (June 28, 2022), online: https://www.litigate.com/data-driven-decisions/blog#/factors-influencing-the-likelihood-of-winning-an-appeal-atthe-supreme-court-of-canada; Nicolas Kasting, "The Supreme Court of Canada in 4 Graphs" Data Science For Lawyers (March 17, 2021), online: https://www.datascienceforlawyers.org/2021/03/the-supreme-court-of-canada-in-4-graphs/. For social media posts, see, for example, https://twitter.com/SimonWallace/status/1662123207514636288?s=20.
2 Named Person v. Vancouver Sun, [2007] S.C.J. No. 43 at para. 85, 2007 SCC 43 (S.C.C.).
3 Jena McGill & Amy Salyzyn, "The Supreme Court of Canada and Mainstreamed Judicial Analytics" in Carissima Mathen, Wolfgang Alschner & Vanessa MacDonnell, eds., (De)Coding the Court: Legal Data Insights into Canada's Supreme Court (New York: Routledge, forthcoming), online: at 2. In this definition, we intentionally take a broad approach as to what technology and/or technological techniques fall within the term "judicial analytics". As observed by Ryan Whalen, "The emergence of computational legal studies: an introduction" in Ryan Whalen, ed., Computational Legal Studies (London: Edward Elgar Publishing, 2020) in his discussion of the term "computational legal studies": "Some projects may use computational techniques to gather data - for instance, by writing a custom web scraper or by downloading and parsing data from open data sources - others might use machine learning or natural language processing techniques to analyze legal data. What unites computational legal research is the fact that it would not be possible without the power of computation - that is, the power to make easy otherwise difficult calculations, or to make trivial otherwise burdensome tasks. In discussing judicial analytics here, our concern is not with which technical techniques are applied to court data but rather the consequences and questions arising when technology makes it much easier to analyze court data."
4 For example, one of the first legal analytics tools to launch commercially was Lex Machina in 2010 (Justin Smith, "10 Years of Legal Analytics" CIO Review (February 20, 2020). For more discussion of commercially available tools, see Jena McGill & Amy Salyzyn, "The Supreme Court of Canada and Mainstreamed Judicial Analytics" in Carissima Mathen, Wolfgang Alschner & Vanessa MacDonnell, eds., (De)Coding the Court: Legal Data Insights into Canada's Supreme Court (New York: Routledge, forthcoming), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4546074 at 2.
5 See, for example, Sean Rehaag, "Judicial Review of Refugee Determinations (II): Revisiting the Luck of the Draw" (2019) 45 Queen's L.J. 1 and Sean Rehaag, "Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal", Refugee Law Lab Working Paper (January 11, 2023), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4322881.
6 Jena McGill & Amy Salyzyn, "Judging by the Numbers: Judicial Analytics, the Justice System and its Stakeholders" (2021) 44:1 Dal. L.J. 249 at 252. See also Sarah Sutherland, Legal Data and Information in Practice: How Data and the Law Interact (New York: Routledge, 2022) at 23.
7 Jena McGill & Amy Salyzyn, "Judging by the Numbers: Judicial Analytics, the Justice System and its Stakeholders" (2021) 44:1 Dal. L.J. 249 at 252.
8 Jena McGill & Amy Salyzyn, "Judging by the Numbers: Judicial Analytics, the Justice System and its Stakeholders" (2021) 44:1 Dal. L J. 249 at 252. See also Sarah Sutherland, Legal Data and Information in Practice: How Data and the Law Interact (New York: Routledge, 2022) at 85.
9 The Supreme Court of Canada, online: https://www.scc-csc.ca/home-accueil/index-eng.aspx". While comprehensive, the SCC data set is not entirely complete. For example, audio recordings can only be downloaded by the public for hearings since October 2017: see, "Webcasts - Frequently Asked Questions", online: https://www.scc-csc.ca/terms-avis/faqvideo-eng.aspx#q7.
10 The Supreme Court of Canada, online: . Although we note that memorandums of argument on applications for leave to appeal are only posted on the SCC website for a limited period of time, i.e., "30 days after leave to appeal has been granted unless they contain personal information, information that is subject to a publication ban, or any other information that is not part of the public record": online: https://www.scc-csc.ca/case-dossier/info/mal-mdaa-eng.aspx?cas=40614.
11 For example, a member of the public may have to submit a formal application and/or pay to access an audio recording of a court proceeding. In some circumstances, a court order may be required. See, for example, Ontario's "Request Form/Undertaking to the Court for Access to Digital Court Recordings", online: https://www.ontariocourts.ca/en/forms/Undertaking-Court-Access-DRD.pdf.
12 Lenczner Slaght LLP, "The Supreme Court of Canada Decisions Project", online: https://supremecourtdatabase.com/.
13 The Refugee Law Lab, "Supreme Court of Canada Bulk Decisions Dataset", online: https://refugeelab.ca/bulk-data/scc/. Increased accessibility of court data is not limited to the Supreme Court. For example, the Refugee Law Lab also hosts bulk legal data sets for the Federal Court, the Federal Court of Appeal and the Tax Court of Canada; see online: https://refugeelab.ca/bulk-data/.
14 For further details, see discussion in Sean Rehaag, "Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal", Refugee Law Lab Working Paper (January 11, 2023), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4322881.
15 Sean Rehaag, "Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal", Refugee Law Lab Working Paper (January 11, 2023), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4322881. https://doi.org/10.2139/ssrn.4322881
16 Simon Wallace, "Supreme (AI) Transcripts: Leveraging state-of-the-art artificial intelligence models to transcribe Supreme Court of Canada hearings" Obiter.ai (December 5, 2022), online: https://obiter.ai/blog/posts/2022-12-05-scc-transcripts.
17 Simon Wallace, "Supreme (AI) Transcripts: Leveraging state-of-the-art artificial intelligence models to transcribe Supreme Court of Canada hearings" Obiter.ai (December 5, 2022), online: https://obiter.ai/blog/posts/2022-12-05-scc-transcripts.
18 Simon Wallace, "Supreme (AI) Transcripts: Leveraging state-of-the-art artificial intelligence models to transcribe Supreme Court of Canada hearings" Obiter.ai (December 5, 2022), online: https://obiter.ai/blog/posts/2022-12-05-scc-transcripts (observing that "for cents on the dollar, an AI transcriptionist can produce a high fidelity, multi-lingual transcript").
19 uOttawa Public Law Centre, 2021/2022 Annual Report, online: https://publiclawdroitpublic. ca/wp-content/uploads/2023/03/UOFD007_Rapport-annuel_2021-2022_EN_Web.pdf.
20 BenchBabble, online: https://www.benchbabble.ca/.
21 Lenczner Slaght LLP, "Data Driven Decisions", online: https://litigate.com/datadriven-decisions.
22 See, for example, Sean Fine, "Canada's Supreme Court is off-balance as 'large and liberal' consensus on the Charter falls apart" The Globe and Mail (January 15, 2022), online: https://www.theglobeandmail.com/canada/article-canadas-supreme-court-is-off-balance-aslarge-and-liberal-consensus-on/.
23 See, for example, Simon Wallace, "Revisiting (computationally) an earlier study regarding SCC citation practices" Obiter.ai (April 19, 2023), online:
Nicolas Kasting, "The Supreme Court of Canada in 4Graphs" Data Science for Lawyers (March 17, 2021), online: . For social media posts, see, for example, https://twitter.com/SimonWallace/status/1662123207514636288?s=20. Although not SCC data per se, see also this LinkedIn post which reports on an analysis of the contents of O'Bonsawin J.'s prior judicial decisions, online: https://www.linkedin.com/posts/jurisage_filac-classification-of-obonsawin-judgments-activity-6966414340023320577-hbu8/?utm_source=share&utm_medium=member_ios.
24 A more detailed analysis of this point can be found in Jena McGill & Amy Salyzyn, "The Supreme Court of Canada and Mainstreamed Judicial Analytics" in Carissima Mathen, Wolfgang Alschner & Vanessa MacDonnell, eds., (De)Coding the Court: Legal Data Insights into Canada's Supreme Court (New York: Routledge, forthcoming), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4546074.
25 For further details, see discussion in Sean Rehaag, "Luck of the Draw III: Using AI to Examine Decision-Making in Federal Court Stays of Removal", Refugee Law Lab Working Paper (January 11, 2023), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4322881.
26 For examples discussing or engaging these sorts of analysis, see, e.g., Maxime C. Cohen, Samuel Dahan, Warut Khern-am-nuai, Hajime Shimao & Jonathan Touboul, "The use of AI in legal systems: determining independent contractor vs. employee status" Artificial Intelligence and the Law (2023) online: https://link.springer.com/article/10.1007/s10506-023-09353-y and Painworth, online: https://www.painworth.com/.
27 See, for example, Yan Campagnolo & Camille Andrzejewski, "The Most-Cited Law Review Articles of All Time by the Supreme Court of Canada" (2022) 60 Alta. L. Rev. 129. https://doi.org/10.29173/alr2713
28 For an example of such statistics, see "Scotus Statistics" Harvard Law Review, online: https://harvardlawreview.org/supreme-court-statistics.
29 Canadian Charter of Rights and Freedoms, Part I of the Constitution Act, 1982, being Schedule B to the Canada Act 1982 (U.K.), 1982, c. 11 [hereinafter "Charter"].
30 See, for example, Sean Fine, "Canada's Supreme Court is off-balance as 'large and liberal' consensus on the Charter falls apart" The Globe and Mail (January 15, 2022), online: https://www.theglobeandmail.com/canada/article-canadas-supreme-court-is-off-balance-aslarge-and-liberal-consensus-on/.
31 See, for example, https://twitter.com/SimonWallace/status/1662123207514636288?s=20.
32 For an example of this sort of analysis using analytics tools, see Mike Madden, "Stating it Simply: A Comparative Study of the Quantitative Readability of Apex Court Decisions from Australia, Canada, South Africa, the United Kingdom, and the United States" (2021) 23 N.C. J.L. & Tech. 270. See also Nina Varsava, "Computational Legal Studies, Digital Humanities, and Textual Analysis" in Ryan Whalen, ed., Computational Legal Studie (London: Edward Elgar Publishing, 2020).
33 Jena McGill & Amy Salyzyn, "Judging by the Numbers: Judicial Analytics, the Justice System and its Stakeholders" (2021) 44:1 Dal. L.J. 249.
34 Jena McGill & Amy Salyzyn, "Judging by the Numbers: Judicial Analytics, the Justice System and its Stakeholders" (2021) 44:1 Dal. L.J. 249.
35 Mike Madden, "Stating It Simply: A Comparative Study of the Quantitative Readability of Apex Court Decisions from Australia, Canada, South Africa, the United Kingdom, and the United States" (2021) 23 N.C.J. L. & Tech. 270. https://doi.org/10.2139/ssrn.4160956
36 On interruptions at the Supreme Court of Canada, see, e.g., Simon Wallace, "Accessing Law in Audio with Artificial Intelligence" in Constitutional Cases (Pt 8) Scrutinizing the Supreme Court with Digital Technology (2023) Osgoode's Annual Constitutional Cases Conference. 13 at 17:05, online: https://digitalcommons.osgoode.yorku.ca/constitutional_cases/13.
37 For elaboration of this point, see Rohan Alexander, Telling Stories with Data with Applications in R (London: Chapman & Hall: 2023) at 164, stating: There is no clear measure of many fundamental concepts. Fried, Flake, and Robinaugh (2022) review the measurement of depression and find many concerns including a lack of validity and reliability. This is not to say that we should not try to measure such things, but we should ensure transparency about measurement decisions. For instance, Flake and Fried (2020) recommend answering various clarifying questions whenever measurements have to be constructed. These include questioning the underlying construct of interest, the decision process that led to the measure, what alternatives were considered, the quantification process, and the scale. These questions are especially important when the measure is being constructed for a particular purpose, rather than being adopted from elsewhere.
38 The caution in respect of "pre-ordained outcomes" is from Rohan Alexander, Telling Stories with Data with Applications in R (London: Chapman & Hall: 2023) at 164.
39 Not all measurement difficulties are the result of amorphous concepts (such as the "just" and fair" examples given). Sarah Sutherland explains, "even experts find it surprisingly difficult to do something as apparently simple as counting how many times something happened when collecting statistical data" (Sarah Sutherland, Legal Data and Information in Practice: How Data and the Law Interact (New York: Routledge, 2022) at 66, drawing on an example from David Spiegelhalter, The Art of Statistics: Learning from Data (New York: Viking, 2019) at 19-21 about the difficulties of counting something like "how many children die after heart surgery").
40 Adam Liptak, "When the Justices Ask Questions, Be Prepared to Lose the Case" The New York Times (May 25, 2009), online: https://www.nytimes.com/2009/05/26/us/26bar.html
41 Yan Campagnolo & Camille Andrzejewski, "The Most-Cited Law Review Articles of All Time by the Supreme Court of Canada" (2022) 60 Alta. L. Rev. 129 (finding that, with respect to the 39 most-cited law review articles in Supreme Court of Canada judgments from 1875 to 2021, 85 per cent were written in English). https://doi.org/10.29173/alr2713
42 J.B. Asendorpf, "Bias due to controlling a collider: A potentially important issue for personality research" (2012) 26 Eur. J Pers. 391 at 391. https://doi.org/10.1002/per.1865
43 Shai Danziger, Jonathan Levav & Liora Avnaim-Pesso, "Extraneous Factors in Judicial Decisions" (2011) 108 Proc. Nat'l Acad. Scis. 6889. https://doi.org/10.1073/pnas.1018033108
44 Konstantin Chatziathanasiou, "Beware the Lure of Narratives: 'Hungry Judges' Should Not Motivate the Use of 'Artificial Intelligence' in Law" (2022) 23:4 Ger. Law J. 452. https://doi.org/10.1017/glj.2022.32
45 Konstantin Chatziathanasiou, "Beware the Lure of Narratives: 'Hungry Judges' Should Not Motivate the Use of 'Artificial Intelligence' in Law" (2022) 23:4 Ger. Law J. 452 at 460. https://doi.org/10.1017/glj.2022.32
46 Konstantin Chatziathanasiou, "Beware the Lure of Narratives: 'Hungry Judges' Should Not Motivate the Use of 'Artificial Intelligence' in Law" (2022) 23:4 Ger. Law J. 452 at 458 (e.g., cases of prisoners who did not have lawyers were generally heard at the end of sessions). https://doi.org/10.1017/glj.2022.32
47 Konstantin Chatziathanasiou, "Beware the Lure of Narratives: 'Hungry Judges' Should Not Motivate the Use of 'Artificial Intelligence' in Law" (2022) 23:4 Ger. Law J. 452 at 459 (e.g., it could be possible that, right before lunch, judges attempt to avoid cases in which they were granting early release because these cases can be more complicated and could lead to "the risk of exceeding their time limit and take their break later or not at all"). https://doi.org/10.1017/glj.2022.32
48 Konstantin Chatziathanasiou, "Beware the Lure of Narratives: 'Hungry Judges' Should Not Motivate the Use of 'Artificial Intelligence' in Law" (2022) 23:4 Ger. Law J. 452 at 459 (e.g., "[b]ecause negative cases can be dealt with in a shorter timeframe, the proportion of cases that can still be dealt with in the remaining time before the break is greater among the negative cases"). https://doi.org/10.1017/glj.2022.32
49 Konstantin Chatziathanasiou, "Beware the Lure of Narratives: 'Hungry Judges' Should Not Motivate the Use of 'Artificial Intelligence' in Law" (2022) 23:4 Ger. Law J. 452 at 460. https://doi.org/10.1017/glj.2022.32
50 Criminal Code, R.S.C. 1985, c. C-46, ss. 691 and 693(1)(a).
51 Note that this is subject to some exceptions; see Supreme Court Act, R.S.C. 1985, c. S-26, s. 35.1 regarding appeals from certain inter-governmental disputes.
52 Supreme Court Act, R.S.C. 1985, c. S-26, s. 43(1)(a). https://doi.org/10.1086/scr.1985.3109496
53 To be clear, the 80 per cent statistic used here is entirely hypothetical and used only for the purposes of illustration.
54 For details on the required analysis under s. 24(2), see R. v. Grant, [2009] S.C.J. No. 32 at paras. 67-71, 2009 SCC 32 (S.C.C.).
55 Wolfgang Alschner & Isabelle St-Hilaire, "Using Network Citation Analysis to Reveal Precedential Archetypes at the Supreme Court of Canada" SSRN Ottawa Faculty of Law Working Paper No. 2023-10 (March 1, 2023), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4374976. https://doi.org/10.2139/ssrn.4374976
56 See, for example, Jon Khan, "The Life of a Reserve: How Might We Improve the Structure, Content, Accessibility, Length & Timeliness of Judicial Decisions?", Master of Laws (LL.M.) thesis, University of Toronto, Faculty of Law, 2019 [unpublished], online: https://tspace.library.utoronto. ca/bitstream/1807/98120/1/Khan_Jon_%20_201911_LLM_thesis.pdf.
57 Jena McGill & Amy Salyzyn, "The Supreme Court of Canada and Mainstreamed Judicial Analytics" in Carissima Mathen, Wolfgang Alschner & Vanessa MacDonnell, eds., (De)Coding the Court: Legal Data Insights into Canada's Supreme Court (New York: Routledge, forthcoming), online: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4546074.
