Step 1.

 

Admin CP > Look & Feel > Select Template > skin_shoutbox_hooks > hookGlobalShoutbox

Find:

<input type='button' id='shoutbox-myprefs-button' value='{$this->lang->words['my_prefs']}' class='input_submit alt' />

Add below:

<input type='button' id='shoutbox-rules-button' value='{$this->lang->words['board_rules']}' class='right input_submit alt' />

Step 2.

 

Find:

<script type='text/javascript'>
//<![CDATA[
{$d['js']}
//]]>
</script>

Add below:

<div id='shoutbox-rules' style='display: none; line-height: 1.5;'>
 <h3>{$this->lang->words['board_rules']}</h3>
 <div class='ipsBox'>
   <div class='ipsBox_container ipsPad'>
     <p>Your rules text</p>
   </div>
 </div>
</div>

<script type='text/javascript'>
 $('shoutbox-rules-button').observe( 'click', function(e) {
   _var = new ipb.Popup( 'booboo', { type: 'pane',
                           initial: $('shoutbox-rules').innerHTML,
                           hideAtStart: false} );

   Event.stop(e);

   return false;
 });
</script>

Poza: