R for the Rest of Us Podcast Episode 12: Chris Knox

In this episode, I speak with Chris Knox, who is currently the Head of Data Journalism at the New Zealand Herald. Prior to that, he worked at the New Zealand ministry of health, where he led an analytics team focused on New Zealand's COVID response.

During our conversation, Chris highlights why he considers R as the optimal tool for data analysis and reporting, especially when dealing with frequently changing data sources and parameters. He also emphasizes the benefits of using R in a collaborative environment, where junior analysts can be quickly integrated into the data analytics and reporting process and assume significant responsibilities, thanks to the reproducibility of R code.

Connect with Chris on Twitter (@vizowl)

Learn More

If you want to receive emails when we publish new podcast episodes, sign up for the R for the Rest of Us newsletter. And if you're ready to learn R, check out our courses.

Audio Version

Transcript

[00:00:00] David: Hi, I'm David Keyes and I run R for the Rest of Us. You may think of R as a tool for complex statistical analysis, but it's much more than that. From data visualization to efficient reporting, to improving your workflow - R can do it all. On this podcast, I talk with people about how they use R in unique and creative ways.

[00:00:18] Join me and learn how R can help you.

[00:00:22] David Keyes: Well, I am joined today by Chris Knox. Chris is head of data journalism at the New Zealand Herald. Prior to that, he worked at the New Zealand ministry of health, where he led an analytics team focused on that. Country's COVID response, Chris, thanks for joining me today.

[00:00:39] I appreciate it.

[00:00:41] Chris Knox: Oh, you're welcome. Uh, happy to be here.

[00:00:44] David Keyes: Great. Um, well, let's talk a bit, um, about, we we've talked previously, but, um, you know, I'd like to have you kind of give a, a overview for folks who are listening to this. You have told me that. Our was I really instrumental in helping to stand up the, the COVID response of the government of New Zealand, um, to help that, to help stand that up really quickly.

[00:01:10] Can you kind of talk and give an overview of, of what the role our played was in, in the initial COVID response?

[00:01:18] Chris Knox: Yeah, absolutely. Um, so just so I, just, for a little bit of background, I joined the New Zealand COVID response, uh, in, uh, January, 2021. So it had already been going for a while by then. Um, so I was, was working at, in a different role at the Herald. And then moved over to, to help out with response moved when, um, so yeah, so the very early stages I'm I'm

[00:01:53] David Keyes: Sure.

[00:01:54] Chris Knox: um, So, yeah. So when I arrived in January, the entire way that we did all, all of our reporting was basically based on a big set of code. Uh, and, and that was up. Uh, so obviously at the, you know, 20, when new went into its first lockdown, uh, we, we didn't have anything in place, uh, for, for COVID reporting, uh, or anything like that.

[00:02:21] Uh, and. I think that like a lot of countries, our general infectious disease reporting was not designed to be kind day to day. Uh, so the real, you know, there were systems in place for reporting, not notifiable diseases, but you at the, of the, after everything had been tied it up and, and the big kinda change that.

[00:02:46] That COVID reporting brought into kind, the health system was the need to report on kinda in essentially incomplete data. Um, so under, under a under sort of normal health reporting circumstances, you wouldn't talk about something where, you know, potentially the person you're talking about, hasn't even been notified that they've got COVID yet.

[00:03:08] Um, because you're basically, you're reporting about positive tests that have through the lab. Um, certainly you reporting case interviews completed, or just maybe for of background new. Zealand's very lucky to have a bit of a, slow start we saw what happened with COVID around the rest of the world. Uh, and, and when, um, Eventually in New Zealand, we went into very strict lockdown and eliminated time. Where was transmit there?

[00:03:53] Went through detailed case interviews and contact tracing and community outbreaks. That type of response. And so we, we moved into situation more like the rest of the world with, you know, thousands of cases a day. Um, so when that happened, you know, so most just, yeah, most countries didn't have this detailed. Interview on for, and, and so we were reporting from basically information coming in from and case interviews,

[00:04:40] changing. The information was being collected was being loaded into database. But the way that basically the way that we were having to report things each day, at some points, it was essentially the reporting reporting requirements changed every day. And so we were producing daily reports in the morning for, for ministers.

[00:05:07] Uh, and then, um, their daily announcement went out to the public at 1:00 PM every. And so that there was a huge, a big set of R analysis sitting behind that code behind those announcements and was essentially the fact that we able to, to run that RLY, but also keep changing. It is what, uh, enabled us to do that.

[00:05:35] So, um, and, and obviously as, as things settled down at any particular type of, um, Information that's being collected as it settled down. Or it would be some things were being captured in operational databases and then moved into reporting databases, uh, some things via phone down on spreadsheets. So very. Test results being faxed from one of the DBS. Um, so there was, you know, there was bit of technological that to

[00:06:18] David Keyes: Hmm. Gotcha.

[00:06:22] Chris Knox: managed. Um, and so, um, Yeah, we, it basically, I guess the summary is, is, is yeah, lots of constantly changing sources of data. Um, but the need to consistently report on kind of changing parameters. And so I think something like a, um, I can't see sort of any workflow other than something like a basically R pipeline, do that work where you've ability, reproducibly run, most of what you're doing and the effort changing things.

[00:07:05] David Keyes: Can I ask you more about that? You said you can't imagine another workflow other than what you had with R why is that? Like, what, what would be the issue with a different type of workflow?

[00:07:15] Chris Knox: I mean, I guess so I guess the other types of work, you know, some of these types of systems work on, um, you know, more classic BI type system where you have everything going into a database and then something like Tableau, um, building a dashboard that sits on top of that. And. My experience is, is that those types of systems, um, I mean, I think so building something like a dashboard to do this

[00:07:44] David Keyes: I.

[00:07:45] Chris Knox: inevitably results in kind of are big development requirement whereby um, changes in the back end require quite a lot of things to change in the front end.

[00:07:56] And so you got this, so it becomes quite difficult, I think, to keep, um, Like the, you know, the, the scheme is at the back end are changing all the time, you know? So it's quite a lot of effort to keep a dashboard or something like that up to date. Um, I think anything that, the big advantage of. Sort of a text based, um, workflow is, is that everything's written down and you just run, you know, run those functions. And so, whereas more analytics, the amount that we would cause the other thing is this is running seven days a. With constantly changing team members. Um, we were burning people out recently quickly.

[00:08:46] Um, and so there was, you know, there was, there was turnover quite particularly at kind at key moments. And, and so. Having for someone come in who had understanding of you down and that sure. There's no errors. Um, you know, people can pick that quickly, whereas if you were having would've a harder. So I, I think that kinda. The, the most things written down in code and being RUNC with, with small changes where it needed and lots of error checks, um, is, is what I mean by the, yeah. Think that, that, just that flexibility. Um, and then over, over time we add, we added the whole workflow, moved into git So we had a GitLab instance. So everything was checked in, which actually meant that we could, um, we actually kept the code. Each day's code was kept. So you could go back, you know, and be like, oh, what, what, what analysis did we actually run? You know, on the 2nd of July or whatever, which was quite useful.

[00:10:05] Um, and again, trying, trying to what you did in a point analysis environment don't possible

[00:10:14] David Keyes: right.

[00:10:15] Chris Knox: in the same way. Uh, and then also we introduced, um, targets, which is a, a library that, um, is, is part of the. Ecosystem, uh, and it, it, that was fantastic. um, so that made a huge difference. That revolutionized the way we were doing things.

[00:10:35] Um, mostly in, in that, that the, um, it allowed us to, so the previous way that things been set up, which you know, was, um, worked extremely well. Became quite time consuming. We basically sort of standard you've and then rerun the whole thing. And so we, we got into some, there were some days when the 1:00 PM update didn't happen at 1:00 PM.

[00:11:16] David Keyes: Gotcha.

[00:11:18] Chris Knox: Um, and that's not ideal. Uh, and so, um, to kinda to spec what the targets workflow does is, is it basically. Allows you to define kind little steps of your workflow and if the input code doesn't then workflow, um,

[00:11:51] David Keyes: Can you, can you give me an overall sense, uh, you know, at a very high level of kind of what were the steps that your code was doing? Um, you know, I imagine it was grabbing data from a database, maybe doing some analysis, but I'd love to have you just kind of, again, at a very high level, walk me through what it, what were the B steps.

[00:12:11] Chris Knox: Yeah, so kind of at, at the highest level we had a database of cases. Uh, and then, the first thing we had the code had to do was work out whether or not we'd reported a case previously. Uh, and so there was a bit of sort of work around, um, particularly case there's the kind of the key step for, for most of, um, most of the time was identifying whether or not cases would order cases or community. Uh, and so we had, um, usually that would be recorded, uh, as part of the case entry, but we generally well also, so from a lot of the time, uh, international arrivals went into managed isolation. Uh, and so we'd, we'd want, if a case was identified, we'd wanna, uh, check and see that. facility they were in, whether there'd been other cases reported in that facility, um, check and see, you know, like if a case was, um, so we'd want we'd report on how many cases had at the border that were detected sort of on the first day, essentially weren't concerned.

[00:13:37] We expect cases to arrive at the border and be detected on the first day. Um, but if a case is being detected on sort of day seven or eight, Uh, then we, we would flag that to, to, up to people that would then go and investigate. Um, and so sort of basically it's that, I guess it's a, yeah. So it's merging up all of those databases to then summarize it. So it's like, you know, there were this many cases today and we don't need to worry about any of them with sort of one level of messaging, um, or, or it'd be like, here are two cases of concern and then people would, and information would come from that were cases of actually of concern. Cause they were that sort of thing.

[00:14:23] So.

[00:14:24] David Keyes: Okay. And, and you talked, so you talked about, you know, the first step being kind of some deduplication and then the piece you were just talking about trying to figure out if it was a community spread or, or a case from, from outside of New Zealand and the border cases, um, after you did that kind of analysis, what did the reporting look like?

[00:14:45] Like how did you do reporting? I mean, I know you said there was a 1:00 PM, um,

[00:14:50] Chris Knox: Yes. Right.

[00:14:51] David Keyes: Thing that went

[00:14:51] Chris Knox: the.

[00:14:52] David Keyes: What does, yeah. What did that look like?

[00:14:55] Chris Knox: So the fir so our reporting was, um, ironically I joined so as cause it was come from a data journalism background, um, A lot of my previous work had sort of been in fancy interactives that, so actually I, the primary output of code was documents, so a of down, um, to generate. And so basically the primary kind of reporting was, um, well actually, so at 9:00 AM, we would, first piece of reporting was actually a text. Uh, sent to high ministers

[00:15:41] distribution.

[00:15:45] David Keyes: And, uh, sorry, I'm sorry to interrupt you. How did, how did it get from R two sending a text? What, what was that process?

[00:15:53] Chris Knox: So that, that was unfortunate. It was a signal text. And that was, that was the most manual part of the job that one of the analysts running had to do it. So basically run everything, get a summary and like, okay, we're gonna send a,

[00:16:08] David Keyes: I see. Okay. So that was, that was manual. Okay.

[00:16:12] Chris Knox: Yeah. I mean, it might have been nice to connect R up to signal, but we never quite had the right bits and pieces to do do that, but yes was manual. Uh, and then, um, then there was a situation report that was sent each day at 11, uh, and that, um, that was a PDF, but, uh, started off as a word document. Um, and we would generate a whole bunch of charts and tables. That went into that. And the key kind, a lot of work went into ensuring consistency across all of those reporting, which got was route was tricky because data would status of cases and things between 9:00 AM. And

[00:17:02] David Keyes: Oh,

[00:17:04] Chris Knox: um, so a lot, so most we, we cut off the data at 9:00 AM, but. So most of it would stay fixed, but some of it would, would change and then have to take that into account.

[00:17:16] And then the 1:00 PM. Basically, there was a media statement we'd provide, which was sort of pretty pretty standard format, which we provide input into in which was a summary of numbers that was from that situation report. And then also there there's a that we update, um, at 1:00 PM

[00:17:43] David Keyes: And both, am I correct that the 11:00 AM and the 1:00 PM summaries, were those word documents that you knitted from our markdown or was, was it some other format?

[00:17:55] Chris Knox: Uh, no, they, they were so what, what, what actually, the way that it, so there was that 11:00 AM one, um, Input from other sources as well as just what came out of our kinda analytics pipeline. So the way that we would do it is we'd actually we'd have a, kinda a, a data sit, sit. Which we would knit. And then if, if there were issues or things needed to be changed, we'd knitted again.

[00:18:20] Um, and then, um, and then the, the final compilation of that report was a step of actually picking up the latest versions of the copying them into the report. Um, but ends, but basically what we had, we had everything set up. So that, that the. And the automatically produced word document was essentially identical to what was in the manually straightforward. Didn't we, we didn't go as, as getting programmers to try and write into type environment. Um, that's that was sort of a step too far. So, uh, there were other, other things that were produced that were fully. Sort of down based, um, kinda on a less, um, frequent basis, but those, that one in particular, there was always kind of narrative, um, commentary that went with it and there were other people writing that.

[00:19:18] So,

[00:19:19] David Keyes: Okay.

[00:19:20] Chris Knox: um, yeah, and then the 1:00 PM was more of a web update. Um, so then that was based on, um, we basically had a set of, of, um, Template web table templates and we'd update those templates based using so the final output of the final output of that daily

[00:19:47] David Keyes: They copy it into.

[00:19:49] Chris Knox: into the

[00:19:50] yeah.

[00:19:51] David Keyes: And so, so all the steps, if I understand correctly, were you. Import data, bring in data from the database, do that deduplication figure out the community cases versus the border cases. Tell folks if there's cases that they should be, you know, cases of concern to be looking into, make a summary, to send off to high level folks at 9:00 AM.

[00:20:17] Do another deeper summary that went off at 11:00 AM, um, that was done with our markdown, uh, knitted to word, and then a media summary, um, both in our markdown to word as well as, um, HTML given to the web team to put online. Are those all the steps? Is there anything that we've

[00:20:39] Chris Knox: That's that's based. Yep. Yep. I mean, no, those are all the steps. Um, it didn't always look exactly like that, but yeah.

[00:20:47] Um, but, and then, and I guess the other thing is that, that, that border and case of concern decision making. Sort of steps involved, accessing a whole other set of S it'd

[00:21:11] decision.

[00:21:14] David Keyes: Gotcha. Yeah. And how, I guess I, I should have asked this before, but how, how big was the team that was working on this?

[00:21:24] Chris Knox: Um, it varied a bit over time, but probably averaged about five or six people.

[00:21:29] David Keyes: Okay. Um, I mean, it's kind of amazing actually, when you think about, you know, a relatively small team like that, being able to. Um, automate all of this so that you can, you know, do produce this much output every day. Um, and I, I also actually liked what you were saying about how our, you know, despite the fact that there was some turnover, our kind of kept things going by having everything in a code based, um, Environment.

[00:21:56] So people could, you know, just pick up other people's code and run it as opposed to, you know, if Larry who does all the analysis and Excel leaves, then you know, you're, you're kind of screwed at that point cuz you don't have it written down as to what he

[00:22:12] Chris Knox: Yeah,

[00:22:14] David Keyes: Um,

[00:22:15] Chris Knox: the, the kind of most overlooked, like, like. There's obviously an like setting up, setting up Larry's analysis as Excel is usually faster than writing it up in code,

[00:22:30] and, it often feels like it's harder to onboard people into an R environment, but actually if you've gotta just sit down, run this look for error messages, almost anyone can do that, you know?

[00:22:43] And then, and so we had some people come in with essentially no, R experience.

[00:22:47] at all

[00:22:47] David Keyes: Oh,

[00:22:48] Chris Knox: they'd be, they'd be dropped into sort of a week of shadowing someone doing it. Uh, and then, um, then they could, you know, and obviously they were there for their, you know, they came in because they understood data.

[00:23:04] And so they were there for that since checking, um, Not, not for, they weren't there for their, R skills cause they didn't have them, but they were learning them fast. Uh, and, and obviously if something went wrong, then they had to get, go and talk to someone who knew what was going on, but we could, it at least meant that, you know, relatively junior people could pick up the reporting workflow.

[00:23:26] As long as someone who knew what was around was around to solve problems when it happened.

[00:23:31] David Keyes: Huh, that's interesting. I didn't realize that people were coming in and working on this so quickly after, after starting, uh, working with R that's. Interesting. Um, what about you, you talked about, you know, at some point you moved to GitLab, um, and collaborating that way. How were you collaborating on code prior to that?

[00:23:51] Chris Knox: The worst way. Um, so was, I mean, so initially, um, the code was just on the drive, which

[00:24:01] David Keyes: Okay. That's what I figured.

[00:24:03] Chris Knox: it was a, um, Collaboration was, are you in that file?

[00:24:08] David Keyes: Right, right,

[00:24:13] Chris Knox: that's ways to,

[00:24:18] David Keyes: right.

[00:24:19] right.

[00:24:21] Chris Knox: but.

[00:24:22] David Keyes: that's, I mean, that's amazing to me. You know, you both had people who were relatively new or, or totally new to our who were onboarded relatively quickly. You weren't using, you know, version control. Um, and. At the same time you were dealing with this huge, you know, world changing event, you were able to bring people onto our change, your team's workflow, um, and make it all work.

[00:24:50] Um, do you have any tips or thoughts as to, to what made that possible?

[00:24:57] Chris Knox: Um, I don't know. Um, I mean, everyone was highly motivated. I've never worked in the team that worked so well. Um, and I think that was, you know, and I think that sort of the sense of urgency that something like gives you just makes people more willing to, to just be uncomfortable, you know, like it was tough, you know, and actually the, the onboarding people into gets was much hard. In the end, then onboarding people into an R reporting workflow. Um, but that everyone we had relatively quickly picked up the value of having kinda an R based, um, reporting. I think that sort of spoke for itself, but the abstractions needed to sort of get into version control were a lot harder for people to pick up, uh, if they were coming from that sort of software background. And I guess, so we didn't initially have, because like all of this is also going on within a background, um, a lot of fantastic work was done before I got there, basically to, push the ministry to be quite innovative in this space. Um, you know, the, health laptops are just the whole computing system is phenomenally locked down. Uh, you know, Like coming from a sort of a journalism background where it's pretty open and just whatever source tool

[00:26:29] David Keyes: right. Sure.

[00:26:34] Chris Knox: Health's restrictions with phenomenal amounts of, of kinda identifiable information about people. And so. You know, you, you've gotta be a lot more locked down and, and, and, um, New Zealand, one of new Zealand's big hospitals had just been shut down by a cyber attack, um,

[00:26:55] David Keyes: Oh,

[00:26:56] Chris Knox: and had to kind of completely rebuild their system so that There's a lot of reasons to be very, very careful.

[00:27:02] Uh, and so just walking in and saying, I need this open source tool people are like,

[00:27:07] David Keyes: Yeah,

[00:27:08] Chris Knox: so, the initial reason there was, was wasn't option.

[00:27:13] David Keyes: I.

[00:27:15] Chris Knox: Um, but there was a software development team that had started using GitLab and we were able. to kind of, um, once it had been approved for them, we were able to make the argument that it was really valuable for data analysis

[00:27:30] David Keyes: Interesting.

[00:27:31] Chris Knox: Um, and, but also that took like that was quite a, the, the more software focused people were like, why, why, why analysts need version control? So it took quite a bit of, you know, um, took quite a bit of convincing, reduce the, sort of get based reproduce. Ideas. Um, and then the other thing that, that, um, worked for us was that the ministry had set up a, um, a big hosted internal R studio server instance.

[00:28:03] So everyone was ultimately everyone was running on their own on, on this big, uh, so, and, and by the time I left, I think we were close to 300, uh, users

[00:28:15] David Keyes: Wow.

[00:28:17] Chris Knox: So, yeah, so the ministry of health has become a, um, a massive user of, uh, and so kinda, and that was that's all. So within the COVID response, we were probably, there was our team, there were probably 20 or 30 analysts working in our, and then the, um, Then then I was also used heavily by the team that was managing the vaccine rollout and the rollout reporting as well.

[00:28:51] David Keyes: Okay.

[00:28:52] Chris Knox: It's definitely the tool of choice for, um, for analyst, for, for kind of health

[00:29:00] David Keyes: Yeah.

[00:29:00] Chris Knox: in New Zealand.

[00:29:02] David Keyes: Why has it, why do you think it's become so, um, popular among folks at the administrative.

[00:29:10] Chris Knox: Um, I think , I mean, because, well, I guess one, the, the, the lack of a license, significant licensing fee means that, you know, you're not, um, the costs of some of the Oracle systems and that being used before and other like that were prohibitive to expanding people's teams. Uh, and so. Um, yeah, I think it, that you can kind of just drop anyone in.

[00:29:37] Uh, and, and also I think just the very, relatively easy way people could pick up and start doing simple tasks. Uh, and, and, and then, um, and I guess just a increasing recognition of the need for sort of reproducible pipelines. And that sort of thing, and I guess is becoming, you know, it's, it's, I guess it's, it's become much more popular in New Zealand, um, which is ironic.

[00:30:07] Like it, it

[00:30:09] David Keyes: I know

[00:30:10] Chris Knox: here, but, um, but sort of didn't have a big, you know, was, was very statsy was, was I guess it was popular and, and, and, you know, academic statistics. Environments, but hadn't, didn't sort of propagate out into the rest of, you know, the rest of analytics, but now that now there's pretty in.

[00:30:34] David Keyes: Right.

[00:30:35] Chris Knox: And, and so, yeah, so, and I just, the, that you can get three to,

[00:30:42] David Keyes: right.

[00:30:43] Chris Knox: um, you know, there's, there's, there are now a good pool of people that.

[00:30:49] With those skills or, or, and now they're all your, or your people are like, you know, I've been doing this type of analytics. What I

[00:31:00] David Keyes: Hmm.

[00:31:02] Chris Knox: increase their employability. Whereas, you know, maybe, maybe not that long ago, you'd be like, well, I need you to learn R and people would be like, what?

[00:31:14] David Keyes: right, right.

[00:31:16] Chris Knox: Um,

[00:31:17] David Keyes: Okay.

[00:31:18] Chris Knox: Sorry so the, the person who set up their original pipeline, um, prior to my joining, he. Came from a similar view of things to me. Uh, and so I think having the key thing is just having this clear vision that we want to have a, an R based pipeline. That's almost entirely reproducible and we're just gonna make it work.

[00:31:45] And I think that apart from the fact that the benefits that we've talked about, I think that, that, that it's a relatively simple mental model. And so, um, so it's something that kind of everyone can be, can be working towards and, and, but also something that people can be picking off small pieces of.

[00:32:04] So it's not like, I think if you have a BI pipeline, it requires database, you know, um, DB admins and, and, and all kinds of like specialist skills to get it to work. And it also is quite monolith. Whereas we actually had something that I nicknamed the spreadsheet of doom and it was just, it was a, I mean, so I'm poor user of spreadsheets. for a long time, I only ever had a read only version of, of work of Excel. Um, and I would use it to and, and have

[00:32:42] David Keyes: Yeah.

[00:32:45] Chris Knox: there's

[00:32:49] spreadsheets. We, we had a, this spreadsheet that basically served us the error checking.

[00:32:57] David Keyes: Hm.

[00:32:58] Chris Knox: For for the pipe. So we we'd put in some, we'd get in a few kind numbers from, from a different source and they'd go into the spreadsheet and then the pipeline would run and if the spreadsheet and the pipeline, um, and if they didn't, we were not, we not good to go.

[00:33:15] And so, um, but it got, it got very complicated. But one of my key triumphs was to eliminate that spreadsheet and build the error checking in, another place. within R But also I think that's the flexibility that, that's what was great about this kinda hyper pipeline is, is that we could actually be like, this is what we are working towards, but we can't do this bit.

[00:33:37] We can't automate this bit now, so let's just do it in a spreadsheet. And then we had a couple of people set. that up And so I think that approach was really empowering for people so that, um, we weren't, you know, we didn't have analysts sitting around waiting for, for people to finish, setting up things in databases, you know, that sort of thing.

[00:33:58] People could actually just get it and do stuff and it would be, um, you could see the impact of it. You know, people would start work on Monday. and By sort of Wednesday they'd, they'd see the work work they did in the media, you know, like it's quite a

[00:34:13] David Keyes: right.

[00:34:13] Chris Knox: it's, it's, it's it's a pace that you don't normally get, um, you know, under a normal analytics type of situation you'll and do something for a few years, then it

[00:34:24] David Keyes: Right,

[00:34:25] Chris Knox: somewhere, you know?

[00:34:25] I'm evangelizing this, our pipeline pretty heavily, but I do think that it, um, yeah, as I technical,

[00:34:41] David Keyes: Huh.

[00:34:43] Chris Knox: I think that it's an abstraction that That people can kind of. Work with and be empowered by, um, and then also, cuz then, then you can be like, okay, you're new, but here's a little bit, we need to tidy up, go and understand how that works, tidy it up, do this one piece. Um, and you, and then, then, then that person becomes the expert on that one piece.

[00:35:04] David Keyes: Interesting. So you're saying it it's like it actually facilitated collaboration in ways that might not have been possible with another tool. Am I understanding you

[00:35:12] Chris Knox: I think so. Yeah. Yeah. Basically. Um, because yeah, it helps ensure that you're not too siloed.

[00:35:18] David Keyes: Hmm. That's interesting.

[00:35:21] Chris Knox: so.

[00:35:22] David Keyes: Yeah. Um, that's I haven't thought about R in that way, but that, that absolutely makes sense. So, um, well let me ask you one last question. Uh, cuz I see we're I, I wanna be respectful of your time. Um, If you were advising the government of New Zealand or, or indeed any government, I'm curious if you would have any thoughts for them with regard to R that might be things they should be thinking about to help prepare for the next COVID.

[00:35:50] Chris Knox: Yeah, just, I was think, actually thinking about this yesterday, ironically. It was good luck that we're able to stand up that our studio server environment, other people been pushing for that and aligned with the response as, but I think that it's almost like something like someone like the government needs a bunch of those systems ready to go,

[00:36:13] David Keyes: Hmm.

[00:36:13] Chris Knox: uh, and, and actually needs a central strategy. So, so like, I think what, what we did was great, but it would almost be like, I think that if we could have a, a, um, a government wide strategy around response analytics, like this is what it's gonna look like. This is how it's gonna work. These are the resources in place. Um, obviously that's. It's much harder to do something like that at sort of an all of government level. Um, but I think at a lower level, it's recognizing that a crisis, like, COVID you dunno what you're gonna need to do.

[00:36:50] Essentially, the way to plan for it is to set up systems that are flexible. You know, you, you could build a fancy software system and, and a dashboard and all sorts of things and, and be ready to go, but it wouldn't be right.

[00:37:05] David Keyes: sure.

[00:37:06] Chris Knox: So, so basically invest having the investment and the people and the tools in place.

[00:37:11] So that, um, and then I guess the thing where I think that where we. Our biggest, the biggest weakness of our response was, was the way that we published data. Um, so another thing I think that would help for this sort of thing was having in place a much better open data. Kind of culture and platform and structure.

[00:37:37] So that then if, if that had been in place, then we could have basically plugged into that, uh, and, and done it. So the way that we, it wasn't ideal, um, was very web table based. Wasn't very friendly to machine reading and that sort of thing.

[00:37:55] David Keyes: Mm.

[00:37:56] Chris Knox: Um, there, there there's a lot that could be done to improve that.

[00:37:59] Um, but trying to make those improvements while responding to a crisis is tricky.

[00:38:04] David Keyes: Yes. I can imagine that being the case. Um, great. Well, Chris, um, thank you so much for speaking with me today. This has been really interesting to learn about the role that R played in new Zealand's COVID response. So, yeah. Again, thank you very much.

[00:38:19] Chris Knox: Oh, thanks for having me on.

[00:38:20] Thanks again for listening. I hope you found this conversation. Interesting. If you have any feedback, I'd love to hear it, David, at our, for the rest of us.com. Thanks.