ouch...there is no reason for you to edit core...seriously...totally wrong place to edit.
this is my super fast way of explaining this...you will need to tell me the parts you don't understand. but first off, there is a basic plugin tutorial in the help/tutorials section.
make a file and call it 'plugin.myflv.php'...
in this file do something like...
function myflv($filename, $width='400', $height='300', $prv_img='')
{
    $prv_img = ($prv_img != '') ? $prv_img : '';
    $no = rand(1, 1000);    
    $out = ($wrap == 'inline') ? "<div id="player-$no" class='flash-flv'></div>n" : "<p id="player-$no"></p>n";
    $out .= "<script type="text/javascript">n";
    $out .= "var so = new SWFObject('" . BASEURL . "/files/player.swf', 'mpl', '$width', '$height', '7');n";
    $out .= "so.addParam('allowfullscreen', 'true');n";
    $out .= "so.addParam('allowscriptaccess', 'always');n";
    $out .= "so.addVariable('displayheight', '$height');n";
    $out .= "so.addVariable('file', '" . BASEURL . "/files/$filename');n";
    $out .= "so.addVariable('height', '$height');n";
    $out .= "so.addVariable('image', '" . BASEURL . "/files/$prv_img');n";
    $out .= "so.addVariable('width', '$width');n";
    $out .= "so.write('player-$no');n";
    $out .= "</script>n";
    return $out;
}
this is assuming you are using the JW Player...and swfobject but the idea is the same for anything else.
in your textarea then you would call this up as...
<plug:myflv 'filename.mov', '600', '400', 'preview_image.jpg' />
;)