DateApril 14, 2021
Published byNIcole Arnett Phillips
Let’s get one thing out of the way: this is a pilot.
We are brand-new at evaluating and archiving work, so we began by
looking to other organisations to see how their assessment process
worked. Where we diverged from assessment
norms, we did so as we wanted to create an inclusive, diverse and
representative resource in service to our community as a whole.
Our intent is the body of work submitted to Kātoitoi is not judged; This archive is not an award program. We looked to some of the distinctive aspects of Aotearoa Design, both our output and our practice, and Te Ao Māori to see what we could learn about sharing, inclusivity, accessibility, and impact that shaped our unique approach.
Review and response are integral to the Kātoitoi kaupapa.
We invite broad participation and discussion from the entire Aotearoa design community.
Accessibility and inclusivity are key.
Kātoitoi is a diverse representation and discussion of what our community is creating at a given point in time. We examined roadblocks to participation. Every creative person has probably encountered a barrier to entry at some point in their design path—acknowledging and removing the obstacles designers experience in sharing and talking about their work with their peers set the foundation for this process.
This is not a showcase...
We aim to collect work that says something about our practice, who we are (both as a design community and as a
nation), the ideas, movements and distinctive aspects of Aotearoa design
today. The archive charts the role and impact of design on our society
and culture through interviews, critical essays and artworks as we
collectively examine, discuss, and celebrate our body of work.
We invited a gender and culturally diverse panel of design leaders to participate in our evaluation process. The panel was selected based not only on their area of expertise, scale of practice but also their ability to bring an individual perspective to the evaluation process. They were asked to assess the content in line with the archives Kaupapa.
Ideas and values are as important as execution.
We designed a scoring interface to place equal importance on the
values as on visual presentation. With two assessment criteria
addressing the design innovation and craft. And two addressing the
impact and intent.
The reviewers considered;
- Does this project demonstrate mastery or innovation of craft in the chosen outcome?
- Does this project advance our design practice or discourse?
- Does this project have influence and or impact outside of the design community?
- Does this project deliver against the selected Kaupapa?
We used a numeric scoring system from 1 to
10. A score of 1 represents a strong “No,”
while 10 represents a strong “Yes.” Scores of 2 - 9 represent the spectrum of assessments somewhere in between. Each submission required four scores — one for each question/metric.
Our team of reviewers each assessed three categories. (With 3-5
reviewers looking at each output category).
Reviewers were asked to evaluate each project objectively and equitably.
They worked remotely online, independently and without having the
chance to influence each other.
We also asked the review panel to consider;
- - What we strive to achieve with Kātoitoi (learn more about the Archives vision and values).
- - A broad definition of each kaupapa category work was submitted against.
- - That English may not be the designer’s native language. Asking reviewers to forgive grammatical or spelling errors. To evaluate the substance of what the designer is describing, rather than the writing quality.
- - Not all designers or studios have the ability to fund professional photography when presenting their work.
- - We aim to be a friendly, encouraging and constructive platform, so we asked participating reviewers to extend generosity while discussing the work.
- - Reviewers abstained from scoring projects where they had a conflict of interest (with 3-5 reviewers assessing each category abstaining did not negatively impact the mahi).
We encouraged our review panel to discuss the work not just to assign metrics. In the first instance, through written feedback input into the assessment scales (insights of which are archived alongside the artefacts).
Thanks to the support of Creative New Zealand we are also about to embark on a series of interviews with the reviewers where they will talk about the projects, key learnings, themes and observations that emerged within the categories they reviewed. And we commissioned a series of
writers, academics and artists to participate in the response. They are looking to the
Kaupapa categories for a
broad view of the work submitted.
These responses from the contributing designers, writers, academics and artists will be published throughout April, May and June, to contextualise the work in our inaugural collection.
Before the assessment, the DA team have manually weeded out incomplete submissions where work was submitted either without text or without images. We also resolved or removed glitch, duplicate and ineligible (based on date or location) submissions. Throughout this process, we took a view of inclusion over exclusivity.
Our review panel, writers and artists independently assessed and
responded to 216 projects. The Design Assembly team then collated and
moderated the reviews. This blind moderation process involved compiling
the individual assessments to generate a mean score for each project.
The 100 submissions with the highest cumulative means in each category
were selected to appear in the 2020 Kātoitoi collection.
Today we proudly share with you those 100 submissions.
Over the next 12 weeks, the reviewers, writers and artists, responses to the work will be published on katoitoi.nz. You can join the kōrero by creating an account and discussing the work there too; we would love to have you take part.
Louise and Nicole (Kātoitoi co-founders)