My time as a MFA candidate in the
Design and Technology program at
Parsons The New School for Design.

The latest Shadow En(Counter) prototype…

This prototype responds to the feedback that I received about dynamically aligning the shadows to the objects on the countertop. It consists of a “countertop” and an administrative interface. The admin interface displays a live video feed of the countertop. On top of this video feed are colored shapes, the forms of which correspond to the contours of the objects on the counter. When the shapes are mouse clicked, a shadow of the object is generated on the counter surface.

Technically, I’m using a video camera, OpenCV, and blob detection to detect the contours of the objects on the counter. When an object is selected from within an admin interface, the vertices of the associated blob are then used to create a shadow for the object.

This prototype shows two things: 1) proof of concept for dynamically generated and positioned shadows; and 2) the beginning of an “administrative” tool for the counter interface. While the proof of concept is not terribly complicated technically, it does allow for a new thread of ideation and evaluation. (Where can we go from here?)

I find the admin interface particularly interesting. It began as a side effect of trying to create the dynamic shadows. However, as I built it, I began to imagine how this kind of customization would be useful for someone assessing the appropriateness of an interface like this. Shadows could be created, adjusted and repositioned live as a person with dementia used the interface. Fine grain adjustments could be made according to the abilities of the individual.

I have received some great feedback on the Shadow En(Counter) prototype. The most important question so far is: how will these “shadow” images be received by individuals with dementia? That is a great question and the answer won’t be known until testing is conducted. I, however, think that conducting testing such as this is a little too far afield from my area of expertise. For these tests to be efficacious I would certainly need a good deal of expert input from fields such as occupational therapy, psychology, and medical engineering.

This speaks to a larger point that my thesis instructor has raised: perhaps the “users” I’m designing for are not individuals with cognitive disabilities. Perhaps I could be designing for individuals who are conducting research into cognitive disabilities. Perhaps there are platforms, systems, or similar that people who work in applied cognitive psychology, occupational therapy, and similar could benefit from. It is certainly something to consider.

Another piece of feedback that I received was about the relationship of the subsequent task performance and the recorded video. Professor Jesse Hoey (University of Waterloo) asked about the possibility of adjusting the projected image during playback so that the shadows dynamically align with the actual positions of task artifacts (teapot, teacup, etc). This raises some very interesting questions.

What becomes of the playback of prerecorded video?

One serious constraint of the system as presented last time is that it could only playback video that had actually been recorded by the system. A “correct” instance of the user executing a task would be video recorded and marked as exemplary or correct. Then, when that task was to be performed again, the exemplary video would be viewed. This means that the spatial and temporal relationships between user movements, procedural steps, and task artifacts are limited to what they were at the time of recording. Certainly the teapot doesn’t need to be in the exact position on the counter every time the task is executed. Certainly there are multiple “correct” procedural flows through task processes. Is it any less “correct” to place the teabag in a cup before adding water rather than afterwards? (Japanese tea ceremonies aside.) The system would benefit from being more flexible than previously outlined.

What of the “shadows”?

There were a couple of ideas behind the shadow treatment. The first was that a person’s shadow is a personal mark on the environment much in the same way that a paint stroke is a recording of the brush across a canvas. The conceit of the prototype is to decouple a person’s shadow from time. The record of someone’s mark on their environment can now be replayed. I think this has a strong aesthetic value, but the question above remains: How will someone with dementia react to the presence of these “shadows”? Will they be understood as visual cues and prompts to be followed? Could they be learned to be this? Could they benefit from being bolstered by additional modes of feedback?

Another reason behind the shadows was that they could potentially be recognized as both a more “embedded” and “human” embedded visual display. It seems that quite often, visual displays (such as projector streams) embedded into domestic spaces are far from “calm technology”. At worst there are examples of an existing GUI being projected on the wall or floor of a home. As if Windows 7 weren’t difficult enough to grapple with on a computer monitor, now the desktop metaphor is being inserted into the domestic space. The best examples I’ve run across are from the MIT Counter Intelligence project, and specifically from their Augmented Reality Kitchen.

So there are some thoughts on the prototype. I will be addressing Prof Hoey’s note about dynamism in another prototype to be posted soon.

I want to make sure one thing is clear about my Shadow En(Counter) prototype. This is more of a “look and feel” prototype rather than a technical one.

The video was actually taken from above the activity and then doctored to achieve the “shadow” effect. The props used were spray painted black to achieve a better look. I included this photo in the last post but didn’t explain it all that well. Here are the painted props I used:

The assumption is that I could achieve a shadow effect through automated filtering or camera techniques. Were the activity actually shot through a translucent plexiglas surface, the raw footage would already appear more shadow like. Regarding computerized filters, here is a simple demonstration of applying video filters post fact…

I am looking at creating an interface that would be incorporated into a “smart home” environment like Ambient Kitchen and leverage the tracking model in a project like COACH.

The COACH project is a prompting system that helps people with dementia accomplish activities of daily living. Here is a User Persona of someone who might use such a system:

I am proposing a countertop interface that would act as a display surface in a system like COACH. Here is one possible setup:

The user’s movements and interactions with objects on and above the counter are recorded by the video camera. Once a recording has been determined to be of a “correct” execution of a task, it can then be played back later as a “shadow” – a guide for the user to follow. Here is a rough prototype:

I just like this image of props from the prototype.

The deck from my studio presentation has some additional information:

Ubiquitous computing interface

The Attention-Span Myth in the NY Times. While it could benefit from some more support material and it has an reliance on references to literature, I enjoyed its devil’s advocate stance.

Here I hazard some connections and deductions on dicey support…

What does it mean to remember?

I was listening to the “Eternal Sunshine of the Spotless Rat” segment on Radiolab. In it they mention that a memory takes on physicality as a sequence of proteins in the brain. Tested in rats, if the process of forming this sequence is prevented (chemically) then a memory is not formed. Additionally, if the same chemical intervention is performed at the act of remembering, the memory is erased – the proteins are dissolved, scrambled, whatever.

So the act of remembering is an act of re-creating memory.

In light of this, what are the goals of cognitive prosthetics and memory augmentation? (Perhaps the conflation of the two is not judicious.)

One goal is the supplementation of human memory. My address book – digital or not – assists me in recalling a volume of information that I would have great difficulty recalling otherwise. I enter names and contact info into my address book so that I don’t have to rely on keeping that info in my brain. (This is a fundamental facet of of writing, right? It is a supplement to human memory.)

So memory augmentation as memory supplement is a giant bucket that we can pour our memories into. Recently we’ve seen some writing (Is Google Making Us Stupid) and artwork (If Search is our memory) address the large digital bucket – the internet – and how it affects our native ability to remember and the meaning of remembering. The thesis of many of these musings is that digital memory augmentation at large scale is bad for our innate abilities to remember.

Let’s assume that this is true – the crutch of digital memory augmentation is eroding our human abilities to remember. For cognitive prostheses then, what about the goal of augmenting not the volume of our memory but our ability to remember? I cannot provide citation of the top of my head, but it feels like I’ve read that memory can be exercised and kept fit. Use engenders ability.

Supplements discourage exercise. Exercise encourages remembering. Do memory exercises help prevent or allay conditions such as senility or dementia? What technologies do we have to support the exercise of memory?

End thought. I’m tired.

Will the oracle of craigslist yield anything interesting?

Okay, so it’s been a while since I last posted anything on here. It’s been a hard row. Obviously, I’ve posted a few of my class presentations. However, my slide decks are made to accompany a live presentation so they are probably a bit unclear on their own.

I think the big thing that I’ve been wrestling with this semester is actually finding a working process. And by that I mean, a process of working. I have dropped the pursuit of artificial limbs for now. (This occurred many weeks ago.) I felt that it would be too difficult to approach with no prior skills in that area. I was also working under the assumption that I would be working with amputees. This issue of “finding a user group” has been a real trial for the past few months.

Looking back, I think that what interested me most from my thesis statement at the beginning of the year was the idea of how designed objects can appear to have autonomous selves. Things that act in ways that we do not fully understand and in ways that we cannot predict can be understood as having a sense of idiosyncrasy, like people or pets. I am not speaking of humanoid robots. This is about behavior, not mimesis.

I think that I have at times been afraid of having “a solution looking for a problem”. Thus the extreme emphasis on dialing in on a very specific user group and problem. This has prevented me from full exploration. I feel I need to open up again to full exploration.

I do not mean that want to ditch my current topic. I am very interested in cognition and digital interfaces. (By the way, gentle reader, I have been researching cognition and digital interfaces. Text message break up.) How interfaces to digital systems work (or don’t) for those of us with the shorter end of the memory stick (triple entendre???) seems a fecund area of research.

So who are the those of us? Really could be anyone. “Cognitive prosthesis” has a medical scent to it. It’s a term that I’ve been using. If I operate under that banner, then I’m speaking about people with memory deficits that are considered abnormal or non-normative. I’ve identified a few major groups of people:

1. The elderly – whether it’s memory loss attributed to the natural process of aging or associated with a progressive disease like Alzheimer’s
2. Those with brain trauma – people in car accidents, those injured in war, etc
3. Victims of stroke
4. Those with developmental disabilities – people with cerebral palsy or similar conditions

But one could classify any device or system that augments an individual’s ability to remember as a cognitive prosthetic. It depends where the the definition of “prosthesis” should begin and end. This is a common issue in some of the readings that I’ve done.

Okay, that’s all I have for the moment. I had to write something to get back in the game. More soon.