HumanMotion-Khatib.pdf: Human Motion Reconstruction and Multi-Modal Feedback for Pathology Evaluation, Therapy Delivery, and Athlete Training
Became a test subject for Karlin (got to experience the benchtop skin stretch device first-hand). Given a series of three stretches, one of which is greater in angle of stretch than the others. (10, slow). Trying to find the JND (just noticeable difference), ie to be able to map the resolution of skin stretch as haptic feedback.
24th June
Worked on designing casing/housing for motor driver
Learnt a little about the controls that we're using
Started working on Solidworks
Karlin's experiment part 2 (30, slow). Learning more about her experiments
Met the Haas
25th June
Modeled the motor driver board on SW
Cleanup - discovering where some things in the lab are
Karlin's experiment part 3 (10, fast)
Witnessed the motor driver heating up considerably on portable device. Possibly due to controller effects. Considered adding metal parts/have a casing that has some heat sink elements
Started modeling the casing on SW
Started reading chapter on Haptics in new Springer Handbook of Robotics
26th June
Finished chapter on Haptics (Springer) - now some of the things make more sense.
Learnt a little about controls
Almost done modeling cover. Went with a (Stanford) block S-shaped hole on the cover.
Karlin's experiment part 4 (30, fast), the last one in the series. Some things to think about:
environment - noisy vs absolute silence vs quiet background noise, effects on concentration
age of subject - are younger people more accustomed to distractions and able to tune them out? boredom factor?
recruiting subjects/volunteers
27th June
Listened in on morning discussion about haptics progress: observations from current experiments, direction of future experiments, direction of focus for applications, implications
Learnt more about designing Haptics experiments:
not highly useful: comparing 1)haptic feedback, 2)no feedback (hopefully 1 is better than 2)
more useful, and the general haptics paper style: 1)visual and haptic feedback, 2)visual only, 3)haptic feedback only, 4)nothing (ideally in that order, with 1 usually slightly better than 2, which is much better than 3, which is considerably better than 4)
a little different: comparing different types of feedback.. vibration vs skin stretch?
Learnt a couple new and useful things on SW (convert, how to use the mirror function properly, creating drawings)
Motor driver casing update: going to get the clip first, attach it to a thin piece of acrylic, so that it can be screwed onto either motor driver. Also having an extended portion to attach the cable to.
Things to start thinking about:
motor driver improvements, like a switch
designing experiments/what experiments: there are so many directions.. so also need to think of what kinds of applications (especially ones where visuals are not a huge component, since they tend to override everything else)
end-effectors: more effective ones for the portable, and improving the existing ones on the benchtop (torsion effects)
building the next portable: improvements to be made
Week 2
30th June
Haptics meeting: Karlin presented
circumstances under which device is used (what would the real situation be?)
concern: one-sided references. Carry out some bimodal experiments (higher and lower stimuli), 2-sided JND
portable device - what can we expect?
2 ideas for next experiments: mapping/tracking, 'therapist'
skin stretch as guiding or error-reporting? Original: former. How to best to exploit the usefulness of skin stretch (motion and velocity)
taking away visual cues/scenarios where vision is not key
human factor: humans tend to want to do well
portable device design:
end effector length (Jason's)
torsional spring
comfort
helped clean fish tank!
Katri's experiment part 1
modified end effector: modeled longer end effector for the portable device for Jason's experiment
fastened cables on motor driver
got clip for motor driver casing
1 July
SDM of longer end effector - til pouring. Something weird with the cutting.. need to check model again
Katri's experiment part 2
searched for articles related to motion training, ideas for good experiments
2 July
finished SDM of longer end effector (extraction)
end effector rotation:
experienced it on the benchtop
thought of ways to have the effect we want: stiff at first, then able to rotate easier and easier (transition at maybe around 20-30 degrees)
main ideas: torsional spring, 4-bar linkage (string/ribbon on fixed point idea). Joystick inspiration for torsional spring. Made rough foamcore model.
3 July
load-sharing in stickybot/spinybot brainstorm session
made Lego model of torisonal spring concept:
discussed next steps, now that the prototype seems to show that it'll work:
how to get free rotation of end effectors - possibly ball bearings. Look for small ball bearings, shafts, pins etc
how to shrink things to fit in the end effector and body - designing the layout (make SW model next week)
Week 3
7 July
Haptics meeting:
Yong Lae presented on dynamic fiber optic force sensing in ablation catheters
Skin stretch items:
need to start portable device testing soon
torsional end-effector soon
hysteresis - find out more, what to do about it, possibly tattoo thing again (using ink stamp idea)
found some articles on hysteresis, put it in a new folder on refworks
ordered samples of small torsional springs
created SW drawing of end effector to help as we work on making it
8 July
helped Karlin to try getting the portable ready and working for the JND test: getting familiar with simulink, labview, how some of the controls work, calibrating the Hall effect sensor. Made a more rigid coupler (for the encoder to the portable to make sure that the results weren't affected by any torsion of the rubber tubing that was originally connecting the two)
dug around and found small torsional springs that are a good size (work nicely over a 2mm shaft) and exert about the right amount of force. Worked on placement of parts, designing layout.
9 July
more work on the portable device. Made another, better coupler. Can run the calibration and testing on my own. Tried switching motor drivers, both directions, tweaking gains, size of deadband, and matlab-generated polynomial for the Hall sensor calibration curve, but error is still up to 0.7 degrees (encoder and hallsensor/desiredposition). Certain angles (specified desired positions) had very small errors (like 36deg counter clockwise had 0) but some had significant errors (like 10deg had 0.7deg or so error). Need to get better resolution for the JND tests (the benchtop JND tests used increments of 0.25deg).
10 July
more portable device work. Not making use of full range previously (+- 10V) since only 0-5V is what we use, so found switch on card to change range. Trying to create a lookup table instead of relying on the matlab polyfit (which seem to always be slightly out of phase or off, even at high orders). Attached rubber tubing directly to motor (wondering if gear ratio had something to do with it) to test, but still off.
learnt about (optical) encoders (like the ones used in the skin stretch devices), quadrature as we were trying to get better resolution
11 July
worked on torsional end effector - looking up parts on mcmaster, sizing, building SW models
Week 4
14 July
Haptics meeting:
Skin stretch:
Karlin to start talking with Sanjay (muscle actuators)
Pete building model of skin hysteresis
optical mouse (hp) to calibrate?
torsional end effectors will be made soon
Katri presented on her tactile stimulus experiments
Finished modeling assembly for torsional end effectors
Kaust meeting
15 July
torsional end effector - ordered parts (and also a metric drill set). Finished SW stuff (had to make separate file for the holes for the M1 pins because of tool size constraints). Started SDM but got stuck at unigraphics because the server is down ):
visited Thor Besier's lab (human performance, in Arillaga)
torsional end effector - realized something had to be modified
parts came
learned more about piezomotors, especially the ultrasonic one that the skin stretch devices use
17 July
Unigraphics back up again!
torsional stuff - unigraphics processing etc, got as far as machining wax block (left it running overnight because of the tiny tool and long toolpath), pretty much on my own.
started calibrating hall effect sensor again, but...
18 July
poured task 3 in the morning (stronger than task 9, but opaque..)
things realized:
should make another set (multiple copies, since it takes a long time)
didn't check actual tool depth properly, so there's a slight chamfer-like feature on the edge of the end effector caps. But it might not affect the rest of the assembly (the thing I worry about is the possibly interference with the middle thickness block on the body). Probably it will just result in a slightly larger contact area on the caps.
looked for long, mini end mills
other ways to make? flip cap orientation (face up instead of down), modification to design.. thinking of ways
read some things about miniature ball bearings (most interesting and useful thing I read: axial play is usually 8-10 times the radial)
(Jason's setup) tried out end effectors spaced further apart (put some old spare parts together)
realized a few other hardware design limitations in current portable because of the above: there is a ledge/lip feature that limits the distance that the end effectors can be apart. Again, issue of how would it be possible to maintain contact that is secure, but also compliant such that it is always in normal contact, and ideally equally for each end effector.
Week 5
21 July
torsional end eff -
ran rest of SDM, took a long time to gouge out parts from wax (in particular, the caps).. time to look for Bioact 280, a solvent that's supposed to make high melting point waxes (like the blue stuff we use) soften so it'd be easier to remove small parts
started putting parts together, experimenting with modifying the torsional springs, primarily their legs. Thinking of adding spacers (thin rings of plastic tubing, adding a bolt at the end of the set screw (which acts as the main shaft) etc:
springs and parts:
with bolt:
haptics meeting:
Jason - problems with portable, noted vertical gear play (possibly affecting hall effect). Positioning (upper arm above elbow vs forearm).
Pete - hysteresis update. Seems to be time-based rather than position-based. Checking out/possibly create skin model.
did a little googlesearch on wearable (medical) devices, to look for ideas on how to keep the portable on and the end effectors in normal contact.
Karlin tried the benchtop on upper arm vs on forearm (normal position). Seems that there is a greater 'dead' zone, but then past that zone, a further small rotation is felt more than if it were on the forearm. It was just one configuration though.
22 July
worked on assembly of torsional end effectors:
ended up not using the bolt because of a tricky pin (the one sticking out is the culprit)
problem with there being a dead region of free rotation before the spring legs would catch onto the pin/jut in cap.
preload of spring - uncooperative wire, contributing to above problem
issue with friction/not perfectly smooth rotation
ended up tweaking and cutting and bending (lots of plier work) til it worked. The second end effector was considerably easier to get to work than the first, except for a couple complications with the tricky pin (a very very tight press fit)
notice the bent legs of the spring:
tricky pin:
it works! (: it still needs to be improved though. Main thing is that Karlin and I tried it out on the benchtop, and it needs more torque. Ordered another bunch of torsional springs from mcmaster, which hopefully will have a bit more torque (and allow some experimenting with spring leg configurations). It's not possible to preload the current springs additionally I think, although I am thinking still..
final shot:
23 July
Office of Technology Licensing people came by the lab to talk about the disclosure for the skin stretch devices, detailing the process of a patent application (emphasizing that they look for something novel and non-obvious, with the latter being increasingly important these days) and their role in 'marketing' it. Interesting to hear about it - I think the last time I learned about intellectual property stuff was in me21n, when we talked about Renaissance ways of protecting intellectual property...
thought of torsional end effector version 2 - what to change/add/modify/how to. Some ideas include:
adding small lip (0.5mm) on top of body to act as spacer
try making caps the other way? Tool reach (or rather, the limit of the tool reach) was a problem in the sdm-ing
modify body to be able to mount force sensor (do we want a force sensor always mounted? Benchtop and portable?)
change order of assembly - remember to drill set screw holes etc first before assembling other parts
task 9 (less opaque than current task 3) caps so we can see what is going on in the caps?
use ball end mill to get slight chamfer on cap ends? Maybe explore different cap end edges since it appears that the chamfered edge feels a lot better (less poky/painful, more pleasant than regular urethane cylindrical ends)
change height of middle reinforcement section on body? For comparisons with other types of end effector, how does the amount of space between the contacts and the middle reinforcement section change the feel/effectiveness of the effector, since it's usually pressed into the skin some amount?
I guess the question is, what sort of experiments are we aiming to do with this?
torsional springs from mcmaster aren't any better
Joan found an encoder to fix onto the portable directly (no more hall effect sensor issues with noise etc)
24 July
worked on making piece so that the torsional end effector can be attached to the force sensor: finished SW models but then Joan had some M5 bar stock that I could use so I made a temporary piece
Karlin's JND test, but with the portable (10 and 30 deg, fast). Turns out that it's no worse than the benchtop. On the portable, Karlin had a lower JND for both 10 and 30 deg, Jason had a slightly higher JND at 10 deg and a lower JND at 30 deg, and I had an about the same JND at 10 deg and a lower JND at 30 deg. Possible reasons (there are so many differences between the benchtop set up and the portable set up, but main ones might be): the constraints on the forearm (squished against the baseboard versus free but with two straps around the forearm), the encoder (500 counts for the benchtop one versus 1000 counts per revolution for the portable one, gear ratio), we might just be getting better at rotational skin stretch (though not as likely)..
prepared for hosting the MERL BBQ: planned with Greg, Ivan and Taylor, made witty flyers and posted them, went on a costco sausage-hunt etc
25 July
Jason's: portable acted up in the morning, tried to figure out what was wrong. He did some debugging.
ideas for experiments with torsional end effector
force vs position question again.. what are we really sensing?
starting to try making own torsional springs/thinking of alternatives to wire springs
hosted BBQ, pretty successfully
tested out torsional end eff (in comparison with free rotating ends and fixed ends):
note: the vertical axis is degrees commanded/torque in mNm; horizontal axis is time in s; commanded is the ramp commanded (velocity of 10 deg/s, 0 10 0 20 0 30 0 20 0 10 0 deg, with 2s pause at each stage)
the plot indicates that the torsional end effector is behaving much like the free rotating one, whereas we would like to see it closely following the behaviour of the free only at lower angles, and then behave more like the fixed end effector at higher angles. Need more torque!
also interesting is that you can see the hysteresis - the peaks on the ramp down are noticeably lower, though as a test subject, I didn't/couldn't sense such a difference
another observation: negative torques were recorded (look at the plots of the Free).
Week 6
28 July
meeting with Karlin, Jason, Pete. Hysteresis update:
interesting observations, including a 'resonant' frequency of sinusoidal commanded positions (deg) of 0.75 Hz for Pete's skin, ie at 0.75 Hz there was no hysteresis
above 0.75 Hz, 'backward' hysteresis observed. rather unexpected.
torque continued to increase even though measured position decreased
brought up the question of how to model the skin - series of springs and dampers?
does force take time to build up in the skin?
tried to make torsional springs: got various wires, steel spring bands from lab. Read up more on torsion springs, attempted (but no success in) making springs.
29 July
figured out a way to make torsion springs that work (: made a wire guide and found a suitable arbor to coil wire around. Successfully made a spring that feels like it will be the right amount of preload. Also figured out how to adjust the legs of the springs to get rid of slack.
started thinking of experiments/how to test torsional end eff?
experimented with new springs:
tested out new torsion springs with same ramp as before (see 25 July):
this is more like what we are looking for. The torsional end eff behaves more like the fixed at small degrees, then behaves more like the free at larger degrees.
30 July
tweaked the legs of the spring to reduce slack in the torsional end eff to nearly zero
read up on torsion springs, did some rough calculations: for the spring currently in use (Joan's wire), Rate (torque in in.lb/angular deflection in revoultions)= 0.4464 in.lb = 0.05044 Nm per revolution.
started working on what we're calling the knob experiment, in which we are hoping to show how the torsional end eff is better, in terms of allowing a larger comfortable range of rotation. The idea is to allow people to play with the knob, and see how far they are comfortable in rotating their own skin. Our hypothesis is that the torsional end eff will result in a greater range of comfortable motion. :
created code for knob experiment
setup is on the benchtop device, testing in order: fixed end eff (fix), torsional end eff (tor) and free rotating end eff (free). Sufficient time was given in between end eff tests to allow skin to 'reset' (from hysteresis effects) to normal.
Karlin and I were the first two test subjects. Qualitatively, the results seem to support the hypothesis. However we need to think more about the experiment (what to tell test subjects to do? how to analyse the data? how best to position the knob?). We will try it on someone new to skin stretch, and see if their comfortable range is smaller.
31 July
Pete started a discussion on experiments (+ Jason, Karlin, me) - challenges of haptics experiments in general (dealing with humans especially), concerns about skin stretch experiments in particular, refining his idea for the experiments he is interested in running etc. Pete's essential question he hopes to answer: How do humans map positions? Pete's hypothesis is that it's by the torques. After that, the next question is: How do humans map velocity?
talked with Karlin and Jason about what needs to go on the soon-to-be new skin stretch poster. Briefly:
background: on mechanoreceptors in skin. pics:skin, small plots/tables (see old poster/past papers)
rotational skin stretch: why rotational, what's good about it. pics: tattoo/strain plots with arrows
experiments: what they were, how they were done etc, results. pics: experimental setup (benchtop), plots of velocity and error
read a couple articles while trying to find some literature on a JND for time - what is the time discrimination threshold that people can sense through skin/tactile/proprioception? Would be useful to know a rough estimate to help in designing experiments.
measured spring force (in torsion spring in the new end eff), spring was nice and linear.
1 August
checked loads on force sensor since we hadn't really made sure we were operating within the recommended range. Forces and torques in the x and y directions were much larger than anticipated, while those in the z direction were fine (and we usually only care about the z in the skin stretch experiments). Specs on the manufacturer's website say that the max loads range from 4.5-19.4 times the rated max. Checked the force sensor to make sure it was working accurately (used a spring scale and applied 5N of force, checked that it was about right), calculated torques, but it seems like the forces are actually that large. Possible reasons include the slant of the arm (it not being normal to the end eff creates additional Tx and Ty), skin deformation at higher angles pressing against the sides of the end eff.
started working on haptics poster
searched for literature on mechanoreceptors in upper arm (need to find 'sweet spot' for portable on upper arm..)
Week 7
4 August
Haptics meeting:
Jason presented EMG experiments, results from testing on himself. He's ready to start testing a few of us on the visual demand/EMG tests. Also thinking about non-linear mapping, and modifying the continuous tracking trial.
Karlin presented the latest JND test update, and also torque data from the last subject - the torque about the z axis appears to change (decrease) significantly over time.
Mark: maybe need to go back to: do what people perceive match up with strain in skin? Possible that even if torques are different, strain is the same.
Is there a different strain regime that the skin moves into? Do people with higher JND have lower torques?
Comparing the 3 end effectors (fix, tor, free):
subjects played with the knob to find comfortable range
Benchtop: Greg and Jason (but incomplete data in Greg's case due to losing a spring, and poor data due to major slipping in Jason's case)
Portable: Karlin. Seems to be as expected.
need to come up with a good experiment, since it seems worth pursuing. Will use the portable, because it makes for better attachment normal to the skin.
5 August
worked on poster
was a subject for Jason's experiments (2)
searched for articles for ideas on what kind of experiment to test the torsional end effector
decided with Karlin on designing an experiment similar to the open-scale test they first did on rotational skin stretch:
compare fix, tor and free
flow of experiment:
trial: skin stretch is applied (commanded to a random position, at a random velocity)
subjects report their perceived amount of skin stretch by turning a knob that controls a virtual end effector image
trial repeats, and experiment repeats for fix, tor, free
some considerations in designing experiment:
callibrate to each subject, based on their self-defined comfortable range of skin stretch
put in zero line?
vary velocities, not starting position (always start from zero)
slipping?
discussion on how to measure strain in skin:
back to original tattoo? Preferably not, because it is inaccurate (human error in selecting points on image), and also that it is 2D mapping of 3D strain (doesn't take into account the z direction since the camera only takes the top view image).
motion capture: HPL, Delp's lab? Need to find out more, and if the resolution can be scaled to what we want.. also the logistics (time taken, positioning of cameras,
3D laser scanning?
did some searching on Vicon system, retroreflective markers etc. The resolution is supposed to scale linearly. Retroreflective markers for face motion capture are available in diameter 1.5mm.
6 August
Code for knob experiment (aka end effector testing):
first callibrate - subject gets to play with knob to find maximum comfortable range (if slip occurs, considered too far even if not painful)
then trials begin - positions commanded lie within their comfortable range
more knob experiment considerations:
don't want to influence subject's perception of commanded position, but give them a reference line - so we decided on a vertical axis and a horizontal (zero) axis only on the figure that the subject sees when reporting their perceived amount of skin stretch. Any more axes/lines might influence subject to report increments of degrees based on the lines.
as before, leave reporting units ambiguous - tell subject to report how far they felt it went, using the knob, instead of reporting degrees or angles or specified units.
tested out first subject (Karlin), Fix and Tor:
Plot of Perceived(Reported) position vs Commanded position:
blue = Fix, red = Tor
comfortable range for Tor was twice that for Fix
for both end eff, accuracy (closeness of perceived position to commanded actual position) was good for the smallest positions. Thereafter the perceived position for Fix was increasingly higher. Whereas for Tor, although the perceived position was lower, it didn't seem to deviate more and more.
spread of points was large - might need to change number of repetitions of trial
other interesting observation:
Plot of Perceived(Reported) position vs Commanded position, with Tor scaled (Commanded position/2):
blue = Fix, red = Tor
almost the same?? does this then mean that the the increased range of Tor is at the direct expense (linearly scaled) of the precision?
7 August
knob experiment:
tested 2 more subjects (myself and Jason), Fix, Tor and Free
modified m-file slightly
made another stiffer spring because we noticed that the angle at which the end effector cap starts rotating is 5-10 degrees, when ideally we want 15 degrees. Used music wire (from the box of spring steel in lab). However with the thicker diameter music wire, the spring plastically deforms instead
8 August
read "Psychophysical Characterization and Testbed Validation of a Wearable Vibrotactile Glove for Telemanipulation" (Anne M. Murray, Roberta L. Klatzky, Pradeep K. Khosla. Some notes on what they did that might be useful for us:
in their first experiment (magnitude estimation with either AM or FM):
each subject's response was normalized by dividing by the subject's mean in the given condition (AM or FM), then multiplying by the grand mean across the condition for all subjects
to create a more linear scale, they used log scales
finally, the 3 observations for each stimulus within a given subject were averaged
Steven's power law suggests some sort of exponential relationship between the perceived magnitude of a stimulus and the actual intensity
found a handy online spring calculator/designer that calculates torque, strain etc when you give it some inputs. Assuming the music wire and Joan's wire are the same material, the music (bigger diameter) wire with 3 coils should still be stiffer than Joan's wire (current ones have 3 coils as well). But after making some springs with the bigger wire and trying them out, they still plastically deformed quite a lot.
in light of the above, Joan and I talked about some ideas for making stiffer springs/something that would be similarly compliant
Week 8
11 August
Haptics meeting:
talked about the knob experiment:
instead of fixed increments, have increments of maxrange/n, so get same number of points for each condition (Fix Tor Free)
poster: draft by end of the week
knob experiment:
tested Joan, who suggested showing the maxrange as a reference line on the screen image, in addition to the zero and 90deg lines.
tested Seok Chang, realized some problems/limitations of the current experiment. The above idea would help.
12 August
worked on poster
looked at results of knob experiment. I think that maybe we could have the knob image appear during the callibration step, in order for subject to build, whether consciously or not, a mental map of the movement (rather than having a reference line, which would be going back to biasing the reporting in a certain way..)
13 August
Regroup meeting with Jason, Joan, Karlin (she was away). Updates on experiments, and positive news from Delp's lab.
knob experiment:
made some changes:
added figure on screen during callibration, so subject can see what their representation max range looks like as they are feeling the corresponding maxrange deg
instead of fixed increments, set number of points = 6, so maxrange/6 is the increment size
captures true maxrange, whereas previously, eg 40-49 degrees would be rounded down to 40
now, fixed number of points, same number of trials (=36)
normalize subject's responses afterwards: divide by maxrange to get all three conditions (Fix Tor Free) on a 0-1 scale.
tested Karlin. Plots of the different end effectors are pretty similar, and linear.
next steps:
add practice trials before each of Fix, Tor Free? Had 16 practice trials for Seok Chang..
randomize order of end effectors used
test a couple more subjects
what if we ran experiments the opposite way? As in, show a static position on screen, then ask people to control skin stretch to match what they think is that position. Takes away velocity..
14 August
knob experiment:
tested Joan, myself, Pete, Jason. Different orders of Fix Tor Free.
took some photos of portable
15 August
worked on haptics poster. Ideas/thoughts from Jason, Joan, Karlin:
flow diagram? To show use of portable device as feedback
what is the focus of the poster? Rather, what do we want the focus to be?
highlight the portable device somehow
pretty much figured out the poster content, now starting to focus on placing elements, organizing etc
Week 9
18 August
Mini-haptics meeting with Jason, Joan, Karlin, Pete. Discussion on writing a paper/papers, what will go in them, organizing/what storyline to weave. Journal vs conference paper. How to highlight 2 point rotational skin stretch on non-glabrous skin.
discussed and explored motion capture options for figuring out skin strain
figuring out significance of results from knob experiment
19 August
worked on poster - pretty much settled on using a picture of Santhi's arm (thanks Santhi!) in the portable during an EMG experiment:
planned what to write about the end effectors (what might be a paragraph in a paper/for SURI)
tried out another idea for making and using torsion springs that will make for a slightly stiffer spring (but not sure how much stiffer, if significant). Involves loading the torsion spring in the more usual way, so the spring legs cross each other between the set screw-shaft and the first (fixed) pin. Worked with Joan's wire, slightly stiffer than the