I used to get in trouble talking to a friend of mine. I'd start a sentence with of course, and she'd ask me "What do you mean, of course? It's debatable." She was taking of course to mean I thought what I was saying was self-evident. But I only meant it as a way to signal additional information.
For example, if I said we're going on a picnic and then finished with "of course, there will be ants," she might counter that ants are an easily avoidable problem—missing the fact that I was joking, and that in this context of course was intended to signal the change from serious to humorous.
Words that talk about the talking (instead of what's being talked about) are called discourse markers. A discourse marker is a particle (a small, uninflected word) that directs conversational traffic instead of adding meaning. These little words carry a lot of information about the conversation, the relationship between the speakers, and how they feel about what they're saying.
When you start a sentence with so or anyway,1 you are signaling a change in topic. When you start with well, you are directing the listener's focus to a specific subtopic. Informal particles such as oh and you know redirect the conversational flow without adding much additional meaning.
One unfairly maligned discourse marker is like, which is useful in several ways. It can mean approximately (I got, like, three hours of sleep) or it can signal a significant change in tone, subject, or emphasis: Like, what are you doing? A lone, trailing "Like…" can signal desperation, or a plea for action. An important use for like is when you want to convey the impression of what someone else without quoting them directly. If you don’t remember the words well enough to say this is what she said, you can fall back on she was, like.
One of my favorite discourse markers is the dangling so. Left at the end of a sentence, the word so asks the listeners to fill in the meaning themselves, without the speaker having to be explicit. There's a big difference between I like prunes and I like prunes, so. The latter sentence could be a complaint, a humble brag, or an invitation to imagine the result of eating all those prunes.
Or so, anyway.
So, in the past twenty years or so it became a thing to start a conversation or a fairly formal speech with “So,” Oxford comma or no.
And another thing (terribly OT) even Obama would repeat “is” as in “the thing is, is...” didn’t he?
While what you are saying is not wrong, it is only part of the story. You are assuming some sort of intention, however opaque, on the part of the speaker, when some speakers actually use these interjections and hesitations as a way to stall for time, apparently because they don't really know what they're going to say next. It particularly annoys me to hear this in podcasts or pre-recorded radio shows, and I have been known to write to the producer(s) to ask them to work with the (sometimes nervous or inexperienced) broadcasters to overcome their distracting verbal tics and get to their point, or if that is not possible, to excise these "fillers" from the recording before it airs. (Somewhat to my surprise, I have actually noticed improvements after sending these messages!)