Thesis: Master of Engineering Science (Computer):
Goschnick, S.B. (2001). Shadowboard: an agent architecture for enacting a sophisticated digital self, The University of Melbourne (Department of Computer Science & Software Engineering), 199 pages, Sep. 2001.

Supervisor: Professor Leon Sterling

Abstract:

In recent years many people have built Personal Assistant Agents, Information Agents and the like, and have simply added them to the operating system as auxiliary applications, without regard to architecture. This thesis argues that an agent architecture, one designed as a sophisticated representation of an individual user, should be embedded deep in the device system software, with at least equal status to the GUI - the graphical user interface. A sophisticated model of the user is then built, drawing upon contemporary Analytical Psychology - the Psychology of Subselves. The Shadowboard Agent architecture is then built upon that user model, drawing both structural and computational implications from the underlying psychology. An XML DTD file named Shadowboard.dtd is declared as a practical manifestation of the semantics of Shadowboard. An implementation of the Shadowboard system is mapped out, via a planned conversion of two existing integrated systems: SlimWinX, an event-driven GUI system; and XSpaces.



I have mailed this interesting person asking if I can see his paper.

ooh! my blog's back at the http address! don't know what happened to that other site that was there. good. hmmph. etc.

yes, I'm pinching Meredith's bandwidth. hopefully I'll see her tomorrow night and buy her a drink in compensation.

this linking thing, it's all about expressing one's take on the world. Jill says it's partly a kudos/political thing, and it is; it's also a chance to "frame" (there's those quote marks again!) something else and comment on it. and it is useful to be able to read other people's thoughts on the topic at hand. not necessarily a staff room; more like eavesdropping on them thinking aloud in their study.

I've decided my ratava blog really should be over here with all the other online stuff. not sure what to do; whether to just point it here or to repost some of it. there's not much there.

an rmit media studies student's blog

dammit! a day of editing and I've only succeeded in making the essay 3700 words. which is 700 too long. and it still needs more explanations.
think I'll copy it and do a vicious cut on the copy, see how it looks.

jill's talking about nomadic publishing, too.
I now have an iPaq (well, it's my husband's but he's not getting it back).
now I need to find an app that lets me at least post while on buses for later automatic publication when I synchronise the iPaq. that is, once I learn to do that.

not to mention Jill, who is much cleverer than me. it's all her fault anyway, this blogging thing.

and she's picked up grumpygirl's cartoon-debate meme. it's cute. wish I could cartoon.

(you can tell I'm procrastinating, huh?)

see, things like this essay on New York and the Internet and communication are surely more interesting.

oh dear. this blog seems to have developed readers. I should point out that really, it's just a handy place to bung my study stuff, from uni notes to links to the uni library to essay drafts, plus a few links to sites that are actually interesting
you're welcome, but don't believe a word of any of it.

too weird. the address http://flaneur.blogspot.com brings up a site in spanish or portugese.
whereas http://www.flaneur.blogspot.com is this site, me. is this possible?

have realised I use too many "quote marks", as in "the idea of "virtual" reality..."

when I should say "so-called virtual reality" or "what is known as" or tease out the special meaning of the word in quotes some other way.discovered this by re-reading my first essay, and now I'm trawling the draft of this one for quote marks.

have also discovered that I am a compulsive quoter of Descartes and Heidegger. but the mind-body question and the phenomenology vs existentialism question are so central to the questions about new ways of being, aren't they? so will not excise those.

one day to deadline. starting to get serious about actually looking things up and inserting them, rather than just putting in a vague note to "add later". I did an "add later" on the whole of Heidegger's concept of Dasein, for goodness sake! may as well have written: "essay; add later."

writing essays in a houseful of people is not easy. when you haven't touched something for 5-6 days and have to get your head back into it, it doesn't help when people wander in and out chatting about this and that. they probably think they're only interrupting for a second. but my brain is so useless these days it's like getting a big truck into gear and it's not so easy to start up again. sigh.

will delete this later, but it's quicker and safer than finding the floppy drive. and after about three hours' work, I don't want to lose it:
??mental interactivity ? essay framework

The thesis of this essay is that as a medium, the Internet-computer network is a facilitator of experiences, which while not new to the human condition, were unavailable in previous media.
Those experiences make up a sense of reality unachievable in any other medium, and it is that sense of reality which will define the network’s unique uses.
I want to explore the definitions of “reality” currently in use in computing and cyberspace studies, and show that some foci of virtual reality (VR) research, and some of so-called interactive narrative are based on mistaken assumptions about what underpins the sense of reality for humans.
The essential factors in the Internet operating as it does are twofold: the networked, multidirectional nature of its communication flows and the nature of the devices that are connected to it, i.e. computers, which are machines for imitating other machines (and sometimes imitating humans).
Other devices, such as Webcams, streaming video, Internet telephony and so on, are examples of a new medium’s content being the media before it, as McLuhan said (*) but if these were set aside, the new experiences made possible via the Internet would still exist.
The new experiences which I want to investigate are, in a sense, old experiences reframed in a new medium. They stem from the uniquely human sense of being in a world, and of sharing a world.
I will approach the way the Internet, connected to complex computers, can appeal to that sense of being via the questions of interactivity, telepresence and virtual reality.
Interactivity, it seems to me, is a concept that needs to be redefined every time it is discussed. To some, like (*example from Bolter*), interactivity can be achieved in an action as simple as a reader choosing one of two paths in a text. This sense of interactivity is dependent on some action by the reader, and it finds expression in “interactive fictions” like Michael Joyce’s Afternoon, which attempts to make “real” (albeit in a virtual environment) the various strands and points of view and approaches to given fictional events. (quote from Bolter*)

Some theorists would take interactivity to an even more basic level, suggesting that the act of reading even the most linear and traditional text is “interactive” in that the reader construes and constructs meaning from the fixed words on the page (or screen.)

Others suggest that interactivity exists when the “reader” has an observable effect on the text itself, by moving objects, co-writing or otherwise leaving a trace of their interaction with the text. A recorded “reading” of Afternoon may fit into this category.

Recent commercial developments, as in the field of educational software and commercial Web page design, suggest that “interactivity” is anything that elicits an action from the reader. * give an example/quote from a page*)

I believe that what an “interactive” reading or experience does is create a sense of intersubjectivity, or that “other minds” are involved. In addition to this, some texts can create a strong sense of being the product of intentionality, meaning that they imply to the reader or participant that another mind has been at work.

This kind of “aboutness” and the question of the existence of other minds has been the topic of numerous philosophical books and arguments.

To Karl Popper in The Self and Its Brain *

The sense of intersubjectivity is part, but not all, of one of the Internet’s most interesting effects, that of telepresence.
I’d define telepresence as occurring in two main ways.
First, I can feel myself to be elsewhere (the medium may range from a phone call to sophisticated telemedicine technologies that allow surgeons to operate on human bodies at a distance *ref) Whether the “elsewhere” is a real (material) place or not is irrelevant to this feeling.
Second, another person can feel I am in their space when I am physically absent. Again, the medium might be a phone call or videoconference, or it might be the actual manipulation of a surgical implement in/on their body. It might also be my avatar suddenly appearing from behind a virtual pillar in a game of *…?* and shooting their avatar dead, regardless of my physical location in Melbourne and the other person’s physical location in Tokyo.
In either situation, I have a physical place of being in the real world where my body (sometimes known as a ratava, from avatar) is, as well as a non-physical place of being in the virtual or physical world. “Avatar” is normally restricted to virtual being in the virtual world, but I believe it can be usefully extended to virtual presence in the virtual world. For example, when Derrida quotes James Joyce in…, he is discussing how Bloom’s avatar enters the newspaper office by being “at the phone” that in in the office.
Where “is” the space of telepresence?
By extension from Popper, the “reality” of a place is not so much about its atomic existence as the effect is has on people and objects.
To him “real” is that which has an effect on one of his three worlds, and virtual reality is in this sense real reality. The more physical industrial designs, bombs and medical procedures are modelled inside computers before being used in real space, the clearer it becomes that his “third world” is a useful and workable model for dealing with the paradox of ideas that have no material being, yet massive power to affect the material world.
Gibson’s cyberspace is a consensual hallucination, but it is one without a location other than its inhabitant’s minds. In this sense, if no one is there, there is no cyberspace at that moment.
For these kinds of telepresence, human intersubjectivity is crucial.
But telepresence is not only about “with-ness”. It can exist without a sense of human interaction, say when an astronaut uses a remote robot to make repairs on the outside of the spacecraft. In this sense, cyberspace (a virtual representation of a real space or a constructed space like, say, a 3-D building plan) exists ready to be found, even if no one is “there.”
I’d call this representational virtual reality. It is weak in intersubjectivity, but strong in sensory inputs. It is an attempt to recreate Popper’s World One, and while computing is very good at this, its uses remain practical while sensory inputs are the only?? ….

So the question arises which matters more in virtual reality and telepresence: to seem physically real, or to have readers/users mentally and imaginatively engaged in the ideas and concepts that form the structure of the virtual world? (To further clarify my meaning, I’d appeal to Heidegger’s concept of humans “having” a world, rather than “being in” a world.)
This is really the argument being worked out at the moment in cinema between different schools of animation, and to some extent video game producers.
One side holds that the aim of animation techniques are to make the images appear “really real” – they may be fictional creatures or human, but the idea is that a viewer not be able to see the seams and stitches, so to speak. The movie Final Fantasy,* marketed as “* quote), represents a statement by this side.
The other side, represented by less realistic but more popular animations like Shrek * and Monsters Inc.*, chooses to concentrate on characterisation and storyline at the expense of realistic appearance.

and (reference to movie-theatre theory guy) that belief or hallucination is not dependent on the quality of the graphics, but the quality of the concepts involved.

Forced to choose a theory, I would side with the non-representationalists (and for entertainment I personally choose a good storyline to a technically brilliant animation).
Dennett summarises extensive research into perception in Consciousness Explained, investigating the nature of vision and illusion. The key point for my purposes can be summarised as “people see what they expect to see.”
To me, this means that it doesn’t really matter if every leaf on a tree is different and outlined perfectly, as it didn’t matter to Claude Monet. What matters is the feeling of being outside on a summer’s day in Giverny.
And that feeling is something that can only be shared with and in a sense verified by another mind.
If intentionality is aboutness – the ability to think about a tree – intersubjectivity is the confidence that another person also thinks about a tree. Heidegger ….


In trying to draw a diagram of these features of telepresence, interactivity and VR, I found I needed 3-D modelling expertise I don’t have. But to briefly recap:
Telepresence can mean the “other” being here, or the self being “there”.
The sense of telepresence is dependent on two forms of “virtual reality”. these can be mental – the degree of belief in interaction with another mind and its interntionality (mental VR) – or sensory – the degree of belief in the realness of the sights, sounds, haptic stimuli, smells etc being presented (representational VR)
both kinds of VR are affected by the degree of interactivity, but representational VR is possible without interactivity, while mental VR is dependent on interactivity, which need not be representationally based.
(A parallel might be the … kitten experience. The kitten in the gondola has a representational VR ” experience in that it sees everything the other kitten sees. But its lack of interactivity – ability to affect its surroundings - prevents it from having a mental VR experience, as shown by its lack of learning from the experience. Humans, as Heidegger pointed out, have a world that goes beyond the physical into questions of being and purpose, and we require interaction with other minds to experience a full world.
This is why teachers get students to think theories through for themselves rather than simply telling them what to believe)
Ultimately, I believe, intersubjectivity is essential to the unique features of the Internet. It’s not that online interactivity isn’t possible without it, but that without it, the kinds of interactivity available are limited, trivial and available through other media, or indeed by simply moving objects about in the “real” world.
Comparing this with the various definitions of interactivity, we can see that again, we are asking whether interacting things or ideas matter most.
This argument could be reframed in terms of Descartes’ mind-body split, where the representative “virtual reality” is the body, and the intersubjective/intentional (mental VR) experience is the mind. Gilbert Ryle *ref * would probably say that the argument is absurd, that they are both ways of being, but …..
I’d like to return to the question of degrees of interactivity (setting aside representational VR, and focussing for the moment on mental VR and the intersubjectivity required to achieve it, I believe we can see degrees of interactivity.
So, a short aside on interpretation.
Most semiologists would say that some interactivity exists in even the smallest act of interpretation. perfect “communication” can only exist in a noiseless environment. computers may be capable of it, within tightly defined parameters, but people aren’t.
so interactivity is integral to human communication. we can’t filter out noise, or define our terms as closely as computers do. instead, we set up feedback loops to maximise the level of shared understanding.
this is where intersubjectivity comes in. we can’t know for sure there are other minds, but we can act as if there are (see Andrew’s mind book again for this one)

by cheerfully assuming other people are real, we open the way to discussions about the world we “have”, and have jointly. Heidegger said this tendency to be aware of our existence made us “dasein”, that for which its existence was an issue.
by doing this, unlike the phenomenologists he refocussed the question away from what the world objectively was, to how we actually interact with it, what we are within it.
the computer-mediated world is no different. we, in our basic existence, are no different when we’re interacting with a moving cursor to when we’re walking down the street. The being that “has”
to go back to the earlier literary theories of how the user (reader) “interacts” or “produces” the text (is this Derrida) – we have to ask if there’s a line that’s crossed, or if it’s all a continuum of the same thing.
The factors that might exist in an interactive “experience”, to greater or lesser degree, include the control you have over:
-what actions you can take (in a modernist novel, none; you’re supposed to read from front to end. in a massively multiplayer game, if you do nothing, nothing happens)
-your world (in a modernist novel, a little, which is interpretation that you imagine the setting. in a “realistic” animated movie, like Final Fantasy, almost none)
-other actors: that might appear to be sentient, or might not. (in a computer game)
-your avatar, if you have one.
Texts we can interpret and hence perhaps interact with range from a manual on how to assemble a piece of furniture (which is supposed to provide almost pure information subject to no interpretation whatsoever), through to a modernist novel like … that claims to completely “tell” the story, to a self-conscious and knowing “interactive” book like Pale Fire that encourages the reader to skip around, to read the book out of “order” and to actively question the truth of what is on the page – in effect, to construct their own meaning, although itself a carefully predetermined one, to the “interactive” cartoon that lets you choose which character’s perspective you see the action from (reactive.com.au) to say a TV show that takes “votes” from viewers to create the impression they are participants, to a text-based environment where they write the spaces, to a massively multiplayer gaming environment where a player takes on the role of a soldier on a battlefield - all these are interactive.
The common factor, it seems to me, is that the person taking part makes certain assumptions or accepts certain “hallucinations” about the world of the book/tv show/game.
Therefore, the interactivity subsists (??) in the engagement of a human being with a world.
I’d suggest that the degree of representational VR is peripheral to how strongly we feel ourselves to be in the presence of other minds.
This is why we accept written letters or telephone conversations, both very “low bandwith” media, as evidence of a human speaking to us.
This is also why we continue to have a fascination with what are known as bots, or chatbots. These are computer programs that will answer textual questions
The Turing Test only calls for a computer to communicate via typed responses in its attempts to convince judges it is a human being. (the original was a bit more complicated, but this description suffices)
These efforts have, so far, been much more convincing than the attempts of modern robotics to simulate a human being. It’s only been in the past two years that a humanoid robot that can walk on two legs has been developed, and the problems of vision and other human-style sensory input are proving extremely slow and complex to solve. (see * ref.
Tthis is why the Turing test is of enduring interest. it strips “human” down to the condensed or mediated content of a mind, via words.
Arobot or wax figure that looked and moved exactly like a human is entirely unconvincing if it can’t have an argument about the football. Alice (reference) is interesting because she pretends to live in the same world as us.
A final, basic example: Hairy Mclairy from Donaldson’s Dairy * ref engages kids more than a photograph of a real dog, because he has a name, personal characteristics and a storyline, a world, which children can enter into an share.
So if an impression of another mind is sufficient for us to enter into virtual reality and telepresence, to feel that we are “interacting”, how is the Internet unique?
Computers running programs like Alice, or games that involve complex and responsive characters (like *war game or The Sims ref ), are available offline.
What the Internet makes possible is feedback and more of a projection of ourselves into cyberspace. It is the Internet’s combination of telepresence (of the physical and mental types) with machines that can increasingly mimic human minds, as well as its ability to “project” us into distant real environments or computer-generated or human-drawn ones.
Without the computer, basically, the Internet would be a jumped-up telephone network. Without the Internet, the computer would still be powerful, but it would always be restricted to what it could hold in its individual memory. Combined, they make a “virtual” world that does more than stand alongside the “real” world – it profoundly affects it. ????
Interactivity and intersubjectivity form a loop on the Internet. Once you believe in one, it creates and strengthens the illusion of the other.

Mental interactivity in my definition is about doing something to someone, and being “done-to” by a real or imagined someone.

Where old media are simply channelled by the new one, nothing new really happens, and the essential features of the new medium cannot be seen. So if writting and reading are “cool” and set us back into ourselves. so reading a book online is really not new, where killing someone’s game avatar online is.
Therefore, interactivity in the non-intersubjective sense is not essential to what the web is. The Internet-computer medium still exists in a site that is barely interactive, where all the reader does is call up the page. This would indded be just a new medium channelling and old one (reference: mcluhan?)
It’s not doing anything interesting.
A quick examination of things that have “worked” online: sexually explicit content, particularly that which offers connections to real people, usually known as camsites. Communities “work” though don’t always persist. For example, Salon’s Table Talk area contained hundreds of highly active discussions where users revelled in their sense of intersubjectivity and shared ideas for several years. Some shopping sites work and these are interactive in the old physical sense – you can make something happen, ie an object come to your door, but again, this is simply an extension of catalogue sites.
Hypertext writing, online or off, has not been generally embraced outside those interested in its theoretical implications.
I believe this is because link-based or topographic writing as outlined by Bolter * is not really doing anything for the reader that “old” printed fiction couldn’t.
Link-based writing designed for readers (hypertext) is as predetermined as Pale Fire. it’s not doing anything new for the reader. Bolter claims (in the … edition, and I acknowledge the new edition may change this) that
Link-based writing designed for writers, however, is a form of “reading” that allows us to create and control our own virtual world, defined by what we link to. In this category I’d point to weblogs, or blogs, which, once they were made technically simple, became….
Many blogs do follow the old writing for the audience model, but many more seem to me to be manifestos, personal attempts to define the self in an environment where one’s electronic encoding in databases has been largely invisible yet powerful (see my essay at *….)
In that sense, online self-publishing is a form of interactivity that creates mental virtual reality, in that it lets you control your world and define its boundaries for other people to share.
So is online game-playing, because of the sophistication of the computerised opponents, or more often the fact that players are competing with other real people.

Because the web is a set of computers, and because computers are virtuality machines, it’s hard to see what it really does.
It’s highly transparent. at the moment we can say it’s something we access on PCs, but even PCs are not perfect computers, virtuality machines.
(* take issue with person who said its possibilities were limitless)
So far, this discussion has taken text as its starting point, with some diversions into cinema, which is still largely a medium divided into creator and audience. The Internet-computer medium can be made accessible to people via an almost infinite number of interfaces, most still purely theoretical or unthought of.
Some early examples are haptic interfaces (w ,
The … issue of wired magazine outlines very primitive devices for directly stimulating the optic nerve, raising the possibility of
Analagous “bionic” devices for direct output from human minds and bodies to real or virtual devices are not yet in wide use, but may follow.
as the interface disappears, we’ll be able to see more clearly what the Web is actually for.

the real question is: is it new? if interactivity and intersubjectivity, the illusion of realness, is what it does, and we already have that in RL, then it’s a kind of parallel, cyber-interactivity. what will we use that for? to be the same humans, just in a superficially different world?

finding it hard to work out whether I'm looking at electronic writing or the wider medium. I think I'm starting at the former and working my way to the latter; but then couldn't I/shouldn't I be looking at film antecedents etc? textuality is a funny thing. do you read it or are you in it? and where is the line drawn on the Web?
if I put a page of text online, is it more interactive? what if you create a link from your blog to the page? what about when I put hypertext links in it? when I split it up into many interlinked lexia? when you can record your reading? add text? when I replace the text with a chatterbot designed to telll the story no matter what you ask? when the text becomes a game you can play? against other people?
when does the text become a world?

sure, I can ask questions. that's the easy bit.

it's a bona fide Perfect Day out there.
and I'm locked inside with Karl Popper.
who says that reality can be defined as anything which causes effects on things. so "world 3" events, like a scientific theory, are demonstrably real.
hmm.
he also says the mind-body problem can be redefined as the mind-brain problem (p 37). I. Don't. Think. So.
and I think I"m getting further away from my actual topic and caught up in the what-makes-good-vr question again.

meredith can see sense in what I think is a confusing list of assorted junk and random readings. I, too, will be very interested to see what comes out the other end of all this. and I'm jealous of her cartooning ability. hope to meet her at blogmeet next Wednesday.

have re-enrolled for next year, provisionally, but very disappointed that Dean is not teaching Electronic Writing next year. and have put down thesis for second semester, which they won't allow. but there's nothing I want to do in semester two anyway.

a thought: is all my focus on the difference between technical VR and the experience of mutual subjectivity just a new version of Descartes' mind/body split? can we really have one without the other? I mean, text environments aren't really environments, are they? and VR is more fun with other people in it - just ask my network wargame-addicted nephew.

geert lovinkis a net theorist who lives in Canberra. this is where he keeps his archives.

some lanier stuff

and their Web catalogue won't let you "hold" things that are on the shelves. so I can't hold the books I find until I get there this afternoon.

lame as: Swinburne has scanned some reserve titles and put them on the Web. sideways. how you're supposed to read that without printing it out I don't know.
sigh.
and yes I'm annoyed today. just want things to work properly

slightly annoyed to find that Dean Kiley may not, in fact, be teaching the subject I want to do next year. don't know yet how that affects my plans; haven't yet re-enrolled, or formally had permission to work 3 days/week so I can do two subjects.
but must check.