-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathp5.html
434 lines (390 loc) · 18.9 KB
/
p5.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
<!DOCTYPE html>
<html lang="en">
<head>
<title>Code 2 | Projects</title>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<!-- import the webpage's stylesheet -->
<link rel="stylesheet" href="style.css" />
<!-- import the webpage's javascript file -->
<script src="/script.js" defer></script>
<script src="jquery-3.4.1.min.js"></script>
</head>
<body>
<div class="nav">
<h1>Code 2</h1>
<ul>
<li>
<a
href="https://docs.google.com/document/d/16yNNHgZIqe3B8W9TLSEBrOSo2KsAF5qVFqkZ8sEh2mc/edit?usp=sharing"
target="blank"
>Syllabus</a
>
</li>
<li>
Projects <a href="p1.html">#1</a> <a href="p2.html">#2</a>
<a href="p3.html">#3</a> <a href="p4.html">#4</a>
<a href="p5.html">#5</a>
</li>
<li><a href="events.html">Public Events</a></li>
<li><a href="readings.html">References</a></li>
<!-- <li><a href="people.html">People</a></li> -->
</ul>
</div>
<div class="content">
<h1>Project #5 Artificial Companion</h1>
<div class="full-width-box">
<img
src="https://freight.cargo.site/w/540/q/94/i/616fe7e44404ec22ba386fd08de53fa50508c7b11ffbda3ad9257a42dad10d8a/gif2.gif"
/>
</div>
<p>
<a href="https://lauren-mccarthy.com/LAUREN">LAUREN</a>, Lauren McCarthy
</p>
<h2>Description</h2>
<p>
The relationship between human and robot has been one of the most
important topics since the industrial revolution in the 18th century.
Western fantasies like to anthropomorphize robots out of the fascination
and fear of being replaced. We've all have seen movies, TV shows, or
click-bait articles positioning human as "us" and robots as "them"
— will they steal our jobs? Will they guard our home and provide
emotional and care labor to our children and the elderly? Will they
realized their oppressed position one day and take revenge? Will they
live on when we are gone?
<br /><br />
In reality, as AI and machine learning are being rapidly developed to
make robots "smarter", we need to be asking better questions not just on
the power relations between human to robots, but also human to human.
Does Alexa really cares whether someone says thank you to them? If
robots are taking your job away, who is designing those robots and who
should really be held accountable for your lost job? Can we imagine new
relationships between human and robots that goes beyond the fantasies of
the dominant - surbordinate?
<br /><br />
For Project #5 you will imagine a new human-robot relationship by
designing an artificial companion. This is the final project for Code 2,
which means you now have the option to draw from your ability to
generate dialogue, use the camera and include facial recognition, add a
DOM interface, and process images in surprising ways. For the remainig
of the semester we will cover p5.speech, the Teachable Machine, and a
few additional libraries listed under
<i>Resources</i>. In essence you're creating a communication software
that may or may not have an anthropomorphic appearance. Returning to the
conversation we had in our very first class on <i>input</i> and
<i>output</i>, the kinds of cues and expressions you end up designing
will have a great impact on the meaning of your relationship.
</p>
<h2>Timeline & Deliverables</h2>
<p>
April 22: Submit an illustration + written proposal for your artificial
companion to Canvas — be very specific about what your companion
can do and cannot do.
<br />
April 29: Complete white-boxed version of the project.
<br />
May 6: Project due, present during class.
</p>
<h2>Submission Guidelines</h2>
<p>
Submit proposal and white-boxed version of your project to Canvas
<br />
Add your final project to your Glitch portfolio, and submit your project
link to Canvas
</p>
<h2>Design Constraints</h2>
<p>
(1) Your can determine the size of your canvas, but please be very
intentional about the size you choose <br />
(2) Your project should use p5.speech of the Teachable Machine
</p>
<h2>
Required Readings
</h2>
<p>
(1)
<a href="https://logicmag.io/failure/the-automation-charade/"
>The Automation Charade</a
>, Astra Taylor
</p>
<h2>
Further Readings
</h2>
<p>
(1)
<a
href="http://magazine.art21.org/2017/11/07/robots-race-and-algorithms-stephanie-dinkins-at-recess-assembly/#.Xpc2flNKjVo"
>Robots, Race, and Algorithms: Stephanie Dinkins at Recess Assembly</a
>, Jacquelyn Gleisner
<br />
(2)
<a href="https://www.csee.umbc.edu/courses/471/papers/turing.pdf"
>Computing Machines and Intellegence</a
>, A.M. Turing
<br />
(3)
<a
href="http://leetusman.com/nosebook/programming/art/teaching/2018/02/17/chatbots-a-reading-list.html"
>Chatbots: Principles, Methods, Ethics - a Noncomprehensive Reading
List</a
>, Lee Tusman
</p>
<h2>Resources</h2>
<p>
(1)
<a href="https://idmnyu.github.io/p5.js-speech/">p5.speech</a>, a p5
library that translates speech to text and text to speech
<br />
(2)
<a href="https://teachablemachine.withgoogle.com/"
>The Teachable Machine</a
>, google creative lab
<br />
(3)
<a href="https://ml5js.org/">ml5.js</a>, friendly machine learning for
the web
<br />
(4)
<a href="https://voice-commands-starter.glitch.me/"
>Voice Commands Starter Template</a
>, Lee Tusman
<br />
(5)
<a href="https://www.tetoki.eu/vida/">Vida</a>, a simple library that
adds motion detection and blob tracking to p5.js.
<br />
(6)
<a href="https://ml4a.github.io/demos/">Machine Learning for Artsits</a
>, Gene Kogan
</p>
<h2>References</h2>
<p>
(1)
<a
href="https://www.stephaniedinkins.com/conversations-with-bina48.html"
>Conversations with Bina48</a
>, Stephanie Dinkins
<br />
(2)
<a href="https://www.stephaniedinkins.com/ntoo.html">Not the Only One</a
>, Stephanie Dinkins
<br />
(3)
<a href="https://www.nytimes.com/video/robotica"
>Robotica video series</a
>, New York Times
<br />
(4)
<a href="https://rkuo.net/Faith">Faith</a> and
<a href="https://faith.rhizome.org/">Baby Faith</a>, Ryan Kuo
<br />
(5)
<a href="https://lauren-mccarthy.com/LAUREN">LAUREN</a>, Lauren McCarthy
<br />
(6)
<a href="http://www.trytobegood.com/projects/build-the-love-you-deserve"
>Build the Love You Deserve</a
>, Fei Liu
<br />
(7)
<a href="http://www.galaxykate.com/">Bot projects</a>
by Kate Compton
<br />
</p>
<h1>Study Guide</h1>
<h2>
Table of Content
</h2>
<p>
(1) <a href="#speech">p5.speech</a><br />
(2) <a href="#lowercase">JavaScript toLowerCase()</a><br />
(3) <a href="#find">JavaScript find()</a><br />
(4) <a href="#switch">switch Statement Pt.2</a><br />
(5) <a href="#ml5">Teachable Machine</a><br />
</p>
<div id="speech"><h2>p5.speech</h2></div>
<p>
<a href="https://idmnyu.github.io/p5.js-speech/">p5.speech</a> is a p5
library initiated by Luke DuBois. It enables you to add text-to-speech
and speech-to-text features into your software, which could come in
handy for building an artificial companion! Unlike previous libraries
we've worked with, p5.speech isn't hosted on a CDN. This means you need
to
<a
href="https://raw.githubusercontent.com/IDMNYU/p5.js-speech/master/lib/p5.speech.js"
>manually download</a
>
the p5.speech.js file, upload it to your sketch folder, and add a link
to the file inside your index.html <head> tag.
</p>
<h3>
Speech Synthesis
</h3>
<p>
Speech synthesis is a text-to-speech feature that translates a string
into spoken words. You have a list of different voices to choose from,
and some of them sound a lot more robotic than the others.
</p>
<div class="full-width-box">
<iframe src="https://editor.p5js.org/xinxin/embed/Ba5OLrIXt"></iframe>
<!-- <p>↳ the speech synthesis wasn't working on chrome for me as of May 4 :( Though it's still working on firefox</p> -->
</div>
<p>
In order to initiate the speech synthesis in your sketch, you would need
to create a new p5.Speech object at the start of your sketch. And from
there you could load a series of methods, such as
<a href="https://idmnyu.github.io/p5.js-speech/">speak()</a> —
</p>
<div class="full-width-box">
<script src="https://gist.github.com/xinemata/756f81aaeb4c1ab0a1bbb5c41003dfa0.js"></script>
<a href="https://editor.p5js.org/xinxin/sketches/Ba5OLrIXt"
>Source Code</a
>
</div>
<p>
Next, you could use
<a href="https://idmnyu.github.io/p5.js-speech/">listVoices()</a> to
check out a list of possible voices you could use in your browser, and
use <a href="https://idmnyu.github.io/p5.js-speech/">setVoice()</a> to
select a voice.
</p>
<h3>
Speech Recognition
</h3>
<p>
Opposite from speech synthesis, speech recognition is a speech-to-text
feature that translates your spoken words into strings. In order to
initiate the speech recognition in your sketch, you have to create a new
p5.SpeechRec object at the start of your sketch, and use
<a href="https://idmnyu.github.io/p5.js-speech/">start()</a> to begin
speech recognition, and
<a href="https://idmnyu.github.io/p5.js-speech/">resultString</a> to
fetch the most recent detected speech —
</p>
<script src="https://gist.github.com/xinemata/32950cf14e74531e52d5da23f7e44cd5.js"></script>
<p>
Now there are two settings you could add to your speech recognition
device. One is called
<a href="https://idmnyu.github.io/p5.js-speech/">continuous</a>, which
sets up a continuous stream between the browser and Google's cloud
service. Setting this to true can cause some serious privacy concerns,
nevertheless it could also increase stability of your software. I guess
in this case you can't have your cake and eat it too. Another feature to pay attention to is <a href="https://idmnyu.github.io/p5.js-speech/">intrimResults</a>, which decides whether the speech rec would print out a partial, incomplete reading of the speech —
</p>
<script src="https://gist.github.com/xinemata/8d8cbcc7b8eae68fca591ce7494020aa.js"></script>
<p>
Now that you have the foundation for speech recognition set up, we can dive into using the resultString to trigger some actions!
</p>
<script src="https://gist.github.com/xinemata/ed1584509d72a29b2f8df34f719b9327.js"></script>
<a href="https://editor.p5js.org/xinxin/sketches/BBRBr1p87">Source Code</a>
<div id="lowercase"><h2>JavaScript toLowerCase()</h2></div>
<p>
Sometimes p5.speech voice recogintion would interprets something you've
said with a capitalized first letter. For instance, the speech recognition example above prints the word
"circle" as "Circle". And since JavaScript is a case-sensitive language,
this could cause some of your IF statements to fail. Luckily you could
use
<a href="https://www.w3schools.com/jsref/jsref_tolowercase.asp"
>toLowerCase()</a
>, a JavaScript method that turns your string into all lowercase —
</p>
<div class="full-width-box">
<iframe src="https://editor.p5js.org/xinxin/embed/kie0u6wTy"></iframe>
</div>
<div class="full-width-box">
<script src="https://gist.github.com/xinemata/57633d0c61fac65b549997cf0441fa57.js"></script>
</div>
<p>
As you can imagine, if there's a toLowerCase() there might also be a
<a href="https://www.w3schools.com/jsref/jsref_touppercase.asp"
>toUpperCase()</a
>. This method isn't immediately useful for this project, just something
to keep in mind if you ever need it in the future.
</p>
<div id="find"><h2>JavaScript find()</h2></div>
<p>
Next, let's move onto a more complex step and integrate a keyword
searching feature for your artificial companion. For instance, instead
of only saying "circle", I could say "could you draw a circle for me
please" and still have my companion recognize the keyword "circle" and
understands that it should be drawing an ellipse on the screen. There
are many different ways to do a keyword search in JavaScript, in this
example we are going to focus on using
<a href="https://www.geeksforgeeks.org/javascript-array-find-method/"
>find()</a
>, a method that searches for a matching element inside the array.
</p>
<p>
However, the first step we need to take here is to split a string into
an array of individual words. To do this we could use the JavaScript
<a href="https://www.w3schools.com/jsref/jsref_split.asp">split()</a>
—
</p>
<script src="https://gist.github.com/xinemata/43d317b7b1ee6642546478e15186934b.js"></script>
<p>
Our next step is to use find() to search for the keyword "circle" inside
of the <i>words</i> array. Let's first create a local variable
<i>result</i>, which would store the outcome of the search, followed by
the ideocyncratic syntax of the find() method —
</p>
<script src="https://gist.github.com/xinemata/a900d014c00481eab56cf9157a805686.js"></script>
<a href="https://editor.p5js.org/xinxin/sketches/UMn7VipLd"
>Source Code</a
>
<p>
What is happening here is that there is an argument function built
inside of find(), and it's called to check every single element of the
array until it finds a matching keyword. You can think of this argument
function working like a boomrang — it gets thrown out towards the
first element of the array, returns when it finds nothing, and gets
thrown out again to look for the next element of the array. The search
ends when it returns a value that matches the keyword. This means even
if you've said "circle" twice inside a sentence, find() would only find
the first one and the search would end.
</p>
<div class="full-width-box">
<img
src="https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/de2c016c-7a43-49c6-82f2-607aede2e195/dcvxtzu-35a911cc-8420-449b-81d5-c45478621275.gif?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7InBhdGgiOiJcL2ZcL2RlMmMwMTZjLTdhNDMtNDljNi04MmYyLTYwN2FlZGUyZTE5NVwvZGN2eHR6dS0zNWE5MTFjYy04NDIwLTQ0OWItODFkNS1jNDU0Nzg2MjEyNzUuZ2lmIn1dXSwiYXVkIjpbInVybjpzZXJ2aWNlOmZpbGUuZG93bmxvYWQiXX0.51LT1wkOuCW9tJYH4A3Xs_P4fvNt2DB6zvA485LbBUc"
/>
</div>
<p>
Besides find(), there are other ways to search and sort words inside an array in JavaScript. Here is a summary borrowed from Mozilla's MDN web docs —
</p>
<ul>
<li class="bullet">If you need the index of the found element in the array, use <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/findIndex">findIndex()</a></li>
<li class="bullet">If you need to find the index of a value, use <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/indexOf">Array.prototype.indexOf()</a> (It’s similar to <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/findIndex">findIndex()</a>, but checks each element for equality with the value instead of using a testing function)</li>
<li class="bullet">If you need to find if a value exists in an array, use <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/includes">Array.prototype.includes()</a></li>
<li class="bullet">If you need to find if any element satisfies the provided testing function, use <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/some">Array.prototype.some()</a></li>
</ul>
<p>
As you can see, the journey of JavaScript is fruitful and filled with adventures. Proceed safely and have fun!
</p>
<div id ="switch">
<h2>
switch Statement Pt.2
</h2>
</div>
<p>
One of the common issue in bot creation is that you have to teach the bot to understand that there are many ways to say the same thing. For instance, to greet someone I could say "hi", "hello", "hey", "howdy", and ideally the bot should understand those different words as having the same meaning and respond accordingly.
</p>
<p>
This is where switch statement can come into play! When we omit "break" inside the switch statement, it would basically work like a waterfall, so the detected string would get compared to multiple different cases until it finds a match —
</p>
<script src="https://gist.github.com/xinemata/b7e2877dfb2e0eee9aaf2ac30bb5a91f.js"></script>
<p>
Adding an additional feature here so that the bot would also be able to tell the time ^_^
</p>
<script src="https://gist.github.com/xinemata/992aeacd4d130cfdb65eb5d69b4b5e36.js"></script>
<a href="https://editor.p5js.org/xinxin/sketches/jVjCX67JD">Source Code</a>
<div id ="ml5">
<h2>
Teachable Machine
</h2>
</div>
<p>
For the Teahchable Machine tutorials, please refer to this <a href="https://www.youtube.com/playlist?list=PLRqwX-V7Uu6aJwX0rFP-7ccA6ivsPDsK5">playlist</a>.
</p>
</div>
</body>
</html>