Previously in this series - finetuning 117M, finetuning 345M

OpenAI’s pretrained Gpt2 models have been all the rage in nlp model finetunning. Sub Simulators, AI Dungeons, Poetry, Chess, Cards Against Humanity - you name it.

My particular interest has been in applying it to my personal chat data, so I can talk to arbitrary friends and more importantly myself whenever I want to. Why? Well, sometimes if you don’t hear voices you have to go and program the voices into being. More seriously, I wished to be able to talk with a version of myself so I can get an outside view of how I think, for analysing my tendencies and problems, rubber ducking, self-therapy and - let’s face it - most importantly for fun.

I won’t go into detail describing the Colab with Facebook data, as it is almost the same as the one used with the 117M version which I describe here and here. We are again using nsheppered’s GPT training code as a base with shawwwn’s fork allowing us to train the whole of gpt-1.5b on a TPU - which means we can do so for free on Colab again! I’ve only added a little bit more so we can talk interactively with gpt2, have it respond to us as a specific person and go through the whole general back and forth which we typically call ‘having a conversation’.

You can train your own in the new GPT2-1.5b collab here. Make sure to click Runtime> Change Runtime type> TPU

Note: It is a lot easier to train if you pay a few bucks for Google One so you can fit it all easily on Drive. A checkpoint of the model comes to ~6 gb, so if you only keep one checkpoint you can do it without paying - it is just more hassle.
Note 2: It took me over a month of training to get it here using just Colab

Self-Chat Samples

Motivation

The idea of modelling a person fascinates me, and as language models have been becoming better and better it genuinely seems like they capture more and more of a person’s style and tendencies. Honestly, I don’t even need more motivation than that to want to do it - the only real question is who to model as better and bigger models come out?

Now, I have to admit - my first choice would be someone like Von Neumann but that’s not someone on whom I have the right type of conversational data to do well (although there are ways to tackle the problem). The person who I am guaranteed to have data on, AND who is still very useful and interesting subject of study for myself is, you guessed it, myself.

Is it useful though? Don’t I already have access to a pretty good, some might say perfect simulation of myself from the get-go? I do. However, if I can have a second one of me to bounce ideas with I’d take it. If I can analyse my problems from the outside rather than from the inside, I’d take that, too.

After playing with the results, and extrapolating to what the next generation of models will be able to do (gpt-2 is already not the newest and best architecture), I think a lot of people might at some point want a version of themselves to talk to, bounce ideas with, get help from and so on.

A Few More Words

I’ve spent some time talking to the irc chat bots of nshepperd and shawwn (both trained on the same data) and had a lot of fun with it. Talking to myself though has been highly addictive, and I’ve been running a Colab where I occasionally shoot myself a question or go into a deep conversation with my gpt2 clone for days now. It’s …interesting.

One thing I didn’t realize though was how personal it is - it’s all trained on my facebook messenger data, where I often talk with my closest friends. It takes very little for gpt to start talking about intimate topics which I’d neither want to share nor would they be as meaningful to anyone else.

Having said that the next are my last 7 unfiltered (I’ve only changed the names) conversations with it, where I am actively trying to avoid sensitive topics (I still had to restart it twice due to sensitive subjects).

Results

The end loss I reached (for whatever that is worth) was just over 0.6, substantially lower than with the smaller models. My dataset is still pretty small - 14mb, and the next step would be to add my personal data from other sources. Yet, it does quite well, though perhaps it wouldn’t seem as exciting to others.

Facebook

The way it works is - I write a sentence under the name of a friend from the dataset but here labeled as ‘Me’, pass it to the model while prompting it to start typing using my name - ‘Svilen Todorov’, (here labeled as ‘Gpt Me’). When it starts outputting something not by ‘Svilen Todorov’ I cut it off, and write myself. Then I feed that + the history so far to the model, which allows us to have an arbitrarily long coherent conversation.

Chat Samples:

Example 1
Me: How's work?
GPT Me: ive been working a bit but still not in the office
Me: I've been working, too. Can't solve this damn bug I have.
GPT Me: ive spent hours on it
GPT Me: but to no avail
GPT Me: im spending all my time on it
Me: I checked everything, yet it runs out of memory despite obvious leaks. hoooow
GPT Me: ive been wasting so much of my time on it tho
GPT Me: 16gb memory, 16gb video ram, 60 hours a week for 3 years
GPT Me: barely have anything open other than youtube and facebook
Me: oh it's a browser issue? man the web sure sounds bloated nowadays
GPT Me: ive tried everything
Me: did you try turning it on and off?
GPT Me: haha
GPT Me: ive changed it between view and off in the dropbox since the beginning, should be the same now as in your case and I doesnt seem to be working
GPT Me: and tried a few things
GPT Me: nothing seems to be working
GPT Me: i uploaded a new one though when you did
GPT Me: but pointless to try it now
Me: don't give up
GPT Me: ive uploaded a new one
GPT Me: doesnt run properly
GPT Me: etc.
GPT Me: still nothing
GPT Me: maybe in a few days when i have a few gigs under my belt it will run a bit better
GPT Me: and then in the mean time i can send you what i have
GPT Me: to use as you like
GPT Me: until then use whatever is in my dropbox
GPT Me: if you want i can even send you my complete open source build, so you can build it and send me what i use
Me: ok, let's do that
GPT Me: ive only sent you a part of the source so i can get the full thing
GPT Me: https://github.com/p5pit/pciusb
Me: I can't access it. Can you resend the link?
GPT Me: ive just changed the link in the source
Me: what's the new url?
GPT Me: ive just changed the name aboven you send it
GPT Me: https://github.com/p5pit/pciusb/tree/master/libs
GPT Me: but you need to update the links in the 'message' part
GPT Me: anyway whats up with you
Me: Not much, just reading
GPT Me: ive been at lectures 4-7
GPT Me: im helping a mate with some maths problems
GPT Me: and trying to read a paper since 5
Me: ok cool
GPT Me: ive not sure what it was on
GPT Me: Lyndon had problems with reading just finding the right words in random papers to use them to make a point
GPT Me: and it was on whether certain types of choice can build a successful models to fit the data to believe


Example 2
Me: hey how are you
GPT Me: ive quit and you know where to find me
Me: what did you quit
GPT Me: my job, ive given a few interviews here and there
GPT Me: and my boss is giving me shit for not doing more work during the day
GPT Me: but ive quit
GPT Me: he's so annoyed at me
GPT Me: thats a bit of an overreaction
Me: yeah you shouldnt have to work
GPT Me: ive been working since 8 am today
Me: oh, you quit your job and are working on getting a new one?
GPT Me: ive been trying to for a while now
GPT Me: and the previous one ends in oct
Me: what job are you looking for?
GPT Me: ive been searching for 'concierge' level stuff
GPT Me: i like working on cutting edge stuff
Me: so something in AI again?
GPT Me: ive been looking at it for 8 months at least
GPT Me: and they want you to start at 17 / 20 euro a day
GPT Me: with no holiday
GPT Me: which is pretty ridiculous
Me: that does seem like a ridiculously low salary
GPT Me: ive talked to people at home and abroad about it
GPT Me: and they were all 'wtf'
GPT Me: i havent even called in to my third favourite coffee place to try to get them to put me as a regular
Me: interesting
GPT Me: ive gone to application after meeting with them
GPT Me: :D
GPT Me: coffee and biscuits*
Me: how did it go
GPT Me: ive only went for a small place so far, startup
GPT Me: and they wanted me to make a profile there
GPT Me: and to tell them about the personal stuff like my salary and social security
GPT Me: and if i want it there it has to be on the phone
GPT Me: and so on


Example 3
Me: whats up
GPT Me: ive been having a lot of success with my programming but dunno if Ive succeed
GPT Me: i have a lot of stuff
GPT Me: not sure how many hours Im going to spend a day on it
Me: you are not working too much and getting burnt out are you?
GPT Me: ive had years with zero effort and amazing results
GPT Me: - especially when compared to other programmers
GPT Me: so yeah, probably a bit
Me: well, take it easy man
GPT Me: ive been doing it for a few hours a day for a few years now
GPT Me: i really want to move to machine learning though
GPT Me: and Ive started on that over a year ago
Me: what ML stuff do you want to do
GPT Me: ive done a little in R, but want to move to using things like neural nets for a lot more
GPT Me: ideally something which is more general
GPT Me: and less focused on just driving
Me: but what do you want to work on specifically
GPT Me: ive helped people a lot with getting them to focus on a specific problem
GPT Me: at this point
GPT Me: :D
Me: so what problem are YOU focusing on
GPT Me: ive helped a lot with getting them to focus on adding simple analytics
GPT Me: to their sites
GPT Me: and getting them to focus on just a few things
GPT Me: rather than whole things
GPT Me: but im moving more towards the latter
Me: hm
GPT Me: ive done a lot of the heavy lifting for them, by getting them to focus on a task that is either too hard or overly boring for them
GPT Me: but thats a bit of an unfair comparison
GPT Me: they have their own stuff
Me: and what do you have
GPT Me: ive done a lot, but they have their stuff
GPT Me: i have to move away from that
GPT Me: in a few months
GPT Me: and program something better
Me: like what
GPT Me: something general
GPT Me: agi


Example 4
GPT Me: ive done fuck all for an hour now
GPT Me: nothing
GPT Me: i shouldnt be working today
GPT Me: i was supposed to be working yesterday too but cba
GPT Me: just tired
Me: you seem to be tired all the time, man. everything ok?
GPT Me: ive been sleeping since 9
GPT Me: no, not really
GPT Me: my back hurts
GPT Me: and i hurt my back peacing out
GPT Me: but other than that im ok
GPT Me: i can do little but write
GPT Me: but i cant read and i havent done anything in days
GPT Me: i am so bored
GPT Me: my head is so bad
GPT Me: :/
Me: it happens - maybe take a break until you feel better
GPT Me: ive never been so annoyed by my head
GPT Me: i can barely use the computer
Me: damn that sucks
GPT Me: ive been worse
GPT Me: but i dont think its as bad as my back
GPT Me: which cant even move
Me: do you need any help?
GPT Me: ive been asking people to do shit for me
GPT Me: for a long time now
GPT Me: no help
GPT Me: people give me shit for not doing more
GPT Me: havent had a break until now
GPT Me: i feel so bad for not being able to pass out
Me: what have you tried for your sleeping problems so far?
GPT Me: ive been trying for a long time an antihistamines
GPT Me:beta-blockers
GPT Me:xanax
GPT Me:histamine
GPT Me:hainamines
GPT Me:hordamances
GPT Me:hordana
GPT Me:hordano
GPT Me:shrooms
GPT Me:whatever
GPT Me: not even going to read now
GPT Me: too tired
GPT Me: anything else
GPT Me: slowing myself down
GPT Me: exhausted
GPT Me: read the whole thing now
Me: I did but I am confused. And you shouldn't take shrooms to sleep.
GPT Me: i didnt really
GPT Me: its a whole thing


Example 5
Me: want to listen to something cool. send me a track
GPT Me: ive been putting tracks from acts I like on and off
GPT Me: into my library
Me: got any links to share?
GPT Me: ive been pushing some friends to listen to more stuff
GPT Me: so you can do that too now :D
GPT Me: https://www.youtube.com/watch?v=hfTPiS7Nmw
GPT Me: https://www.youtube.com/watch?v=QsXgEzx_GGk
GPT Me: and this other one is one of the earlier songs I liked from them
GPT Me: https://www.youtube.com/watch?v=Iqz02saPcoM
Me: cool! I like the cramps. Got any techno tho?
GPT Me: ive been trying to get people to listen to some of the stuff
Me: which stuff? what artists or labels?
GPT Me: ive been pushing mostly labelless stuff which are usually quite good
GPT Me: stuff by bez batteries and their cousin or whatever
GPT Me: and stuff by dvs1 and his cousin
GPT Me: or vatican shadow or whatever
GPT Me: ive sent you the first two
Me: cool


Example 6
Me: hey man, what are you up to this weekend?
GPT Me: ive tickets for a party on friday
GPT Me: wasnt expecting to go out this early
GPT Me: but got tickets during the sale
GPT Me: so im waking up early i guess
Me: cool, what's the party?
GPT Me: ive told people and they havent been able to guess
GPT Me: https://www.facebook.com/events/553455714529578/
GPT Me: its a bit weird
GPT Me: i have friends in charge
GPT Me: who i trust more than me
GPT Me: to tell me if i have a ticket
Me: looks good. who's playing?
GPT Me: ive asked them
GPT Me: they said they dont know
GPT Me: i hope i do
GPT Me: but they were saying it in a way that it sounds bad
GPT Me: :D
GPT Me: aufnahme'samenere
GPT Me: its a bit weird
GPT Me: the only guy that ive met like him is berghainarian
GPT Me: no wait that was mike
GPT Me: still ja
GPT Me: and its this group that does weird stuff
GPT Me: that guy does the sound, and looks kind of weird
Me: sounds pretty interesting really
GPT Me: ive read a little about him on wiki but thats it
GPT Me: he is apparently a maker
GPT Me: but doesnt really know how the word works
Me: do I know him?
GPT Me: ive met him and his brother at makerfaire
GPT Me: very briefly
GPT Me: but they were really high
GPT Me: and i dont really remember them
GPT Me: thats 2 months ago
GPT Me: :D
Me: fair enough


Example 7
Me: how would you describe yourself
GPT Me: ive been working in the same job for 3 and a half years now
GPT Me: relatively new to berlin
GPT Me: 25
GPT Me: 6 years as a programmer/machine learning person
GPT Me: pretty much all my time for the last 5 years has been spent on just learning machine learning and not programming really
GPT Me: and Im pretty sure that even if I was to move elsewhere and work with people from different fields, I would still not be able to keep up
GPT Me: I would spend most of my time on just learning the ropes and whatever else they often have free or cheap classes for everyone
GPT Me: that seems to be the general approach in many places
GPT Me: but I havent seen that approach work for me
GPT Me: i spend literally every day with people and projects that dont allow me to do more than that
Me: and what about your personal life and hobbies?
GPT Me: ive just been doing the same stuff ever since I moved to berlin a year ago
GPT Me: didnt really do much else than that and the online courses to where I am now
GPT Me: I used to spend my time more on random stuff
GPT Me: like programming, music, reading things like that
Me: what about going out
GPT Me: and its rained a lot the last few days which doesnt help either
GPT Me: https://www.facebook.com/photo.php?fbid=324979844210832&set=a.324977827544367.75530.14546_1858271655695&type=3&theater
Me: sounds fun


Conclusion

The conversations aren’t ideal but it’s a lot fun playing with it. Sadly the stuff I talk about on facebook is either too personal or generic ( e.g. arranging to meet people) so I am unsure how well I’ve managed to convey the fun of talking to it. Perhaps another sample of gpt2 talking as me, but with the irc bot mentioned earlier (and ran by nshepperd), might convey it. You can find some more of those samples on Shawn’s twitter + a lot more gpt fun from his collaborations with gwern.

I genuinely think that even with gpt2, let alone bigger and better models you can get a lot of use out of having a model of yourself you can talk to - and not just yourself. Playing with this has really made me realise the value of a service that can create a model of dead or dying relatives, and I want to eventually set up something for my loved ones - at minimum the data collection part.

It will be unfair to say that it is always great though - sometimes it talks about whatever it wants to and the conversation can be harder to steer, otherwise it follows my lead easily. From what I can see in my actual messages I do this a lot, so maybe it is just copying me. Additionally it sometimes talks about things that aren’t really True - like the back pain in Example 1, and if you play with the different parameters (top_k/top_p and temperature mainly) you can force it to go on long tirades which eventually become nonsensical. It’s definitely not perfect.. but it sure is fascinating to me.

Fun fact - there is a sentence in this post written entirely by the GPT version of Me. I wonder how easy it is to spot.