Here is what you need to style your category div after moving it of course :

First we take off the margin off the div container:

/*Added after moving*/
#categorydiv.postbox
{
margin:0;
}
/*Added after moving*/
#categorydiv div.ui-tabs-panel
{
margin:0;
clear:both;
}

Next take off the float and margin of the category selection box and change the style of the links above so they look like tabs:

#categorydiv ul#category-tabs {
/*float: left; removed after moving*/
width:200px;
text-align: right;
/* Negative margin for the sake of those without JS: all tabs display */
/*margin: 0 -120px 0 0; removed after moving*/
padding: 0;
}
ul#category-tabs li {
padding: 8px;
float:left;/*added after moving*/
margin-bottom:0;/*added after moving*/
}


And them finish with some changes on the Add category link:

#categorydiv #category-adder {
/*margin-left: 120px; removed after moving*/
padding: 4px 0;
}
#category-add input, #category-add select {
width: 180px;/*modified after moving*/
}
#category-add input#category-add-sumbit {
width: auto;
}

All in all we get this:

/* Categories */
/*Added after moving*/
#categorydiv.postbox
{
margin:0;
}
/*Added after moving*/
#categorydiv div.ui-tabs-panel
{
margin:0;
clear:both;
}
#categorydiv #category-adder {
/*margin-left: 120px; removed after moving*/
padding: 4px 0;
}
#category-add input, #category-add select {
width: 180px;/*modified after mmoving*/
}
#category-add input#category-add-sumbit {
width: auto;
}
#categorydiv ul#category-tabs {
/*float: left; removed after moving*/
width:200px;
text-align: right;
/* Negative margin for the sake of those without JS: all tabs display */
/*margin: 0 -120px 0 0; removed after moving*/
padding: 0;
}
ul#category-tabs li {
padding: 8px;
float:left;/*added after moving*/
margin-bottom:0;/*added after moving*/
}


And the result: