I'm on a project which ran into the exact same issue.
If you're using IE on a pc with MS Office installed, IE will automatically try to open the spreadsheet. Problem is if you have any custom buttons with code it most likely doesn't work in the IE window.
Our solution was to put an autorun macro in the spreadsheet which checks for "http" in the spreadsheet name (opened in IE) and if so, saves a copy of itself locally, copies the local copy and opens the copy with Excel.
Private Sub CreateNewInstance()
'Saves the copy of the workbook to a local directory,
'opens a new Excel instance and opens the saved copy
'in this new Excel instance.
On Error Resume Next
'Checking to see if the workbook is accessed from the Web Server
If Left(Application.ActiveWorkbook.FullName, 4) = "http" Then
'Use a timestamp to generate a unique filename
Dim currDate As Double, currTime As Double
Dim timeStamp As String
currDate = DateTime.Date
currTime = DateTime.Time
timeStamp = currDate & currTime
'Generate the unique file name and default file path for the workbook to be saved to
Dim strLocationFirst
strLocationFirst = Application.DefaultFilePath & Application.PathSeparator & _
"~webbook" & timeStamp & ".xls"
'Save the workbook to the default file directory
'This is necessary so that the workbook displayed in the browser now references
'the locally saved workbook
ActiveWorkbook.SaveAs strLocationFirst
'Generate the unique file name and default file path for the saved workbook to be copied to
Dim strLocationSecond
strLocationSecond = Application.DefaultFilePath & Application.PathSeparator & _
"~webbookcopy" & timeStamp & ".xls"
'Save the workbook to the default file directory
ActiveWorkbook.SaveCopyAs strLocationSecond
'Open the Excel Application and make it visible
Dim objExcel
Set objExcel = CreateObject("Excel.Application"

objExcel.Visible = True
'Instruct the application to open the saved workbook
objExcel.Workbooks.Open strLocationSecond
ActiveWorkbook.RunAutoMacros xlAutoOpen
End If
End Sub
Not very elegant and can probably be simplified but it works.