I just started playing X3 about a month ago. I bought it way back when, didn't like it for the same reasons as you, just didn't have any clue about what the hell is going on, so I uninstalled and shelved it. But then I heard so many good things about it once you get over the learning curve, so I tried again a month ago.
Now I feel sad that I gave up on it so early. Its really a good game, but it does have its flaws.
I watched some tutorial videos about the UI. There was a 3 parter someone wrote up, but I do not see them on Youtube, try searching egosoft site. It helped alot.
Here are my pointers to help you get on your feet:
1) If you want a story (well something to do) you have to play with storymode enabled. When you start a new game, it lists storymode or not. Keep in mind, even with storymode enabled, it's hard to know exactly whats going on. I pretty much beat the game before restarting (didnt get to the end but close), and I really still have no idea what happened.
From what I believe happened: You start the game and are delivering a crystal so some goner guys, and in the process of delivering it, it gets stolen by a pirate. You have to then go beat that pirate up and get info out of him, and found out it was stolen by some Yaki (a race of pirates rather than random pirates) and they are building a gate (warp gate) to earth. I won't give out anymore, but as you play with story mode, just keep that in mind.
2) The controls aren't really all that hard once you figure it out. The hardest thing to get over the learning curve is to realize theres 2 modes with the mouse. "Flight mode" and "Cursor mode" you right click between the two. Once you get into cursor mode, you can click on things on the screen, like your target, and click little icons on the screen, such as dock. Then it goes and docks you at the target. There is also a little icon on the bottom right of the screen that looks like a "<" or email icon. You can click that and it brings up a list of menus or other screens. The main one being is "Command console".
3) Remember that in flight mode, you are the pilot of the space craft. Using the command console is like being a captain on star trek. You give orders to your ship or your crew and they execute the plan. If you go to command console with your ship, you can order it to dock at a certain place and then the computer/ship takes control with autopilot and does the operation. You can do many things with the command console. If you visit the equipment docks, and other places, you can buy ship upgrades, this is basically software you install on your ship to expand the command console to do more things.
4) If you click through that lower right icon and find the menu, the first one has things like "pilot", "property", "ship". Thats the main thing to look up info about you, the things you own, and your ship, obviously. Pillot lets you see your stats, your factions with other races, reading your email. The property ill get into in a minute. Ship lets you look at what weapons are installed, whats in your cargo bay, and lets you eject whatever.
5) Property or the "r" key on the keyboard lets you bring up property screen. Thats going to be your ship, or any other things you own. If you click on these, you can bring up the command console, or other things and is probably the main screen I have up at all times when having a fleet. Keep in mind, this is an empire building game first, a real time strategy, and a space "shoot em up" 2nd. But that was more of my preference, you could focus on the shootem up aspect entirely. Usually after I get about 5 ships, I usually just take my ship near a space station and bark orders the rest of my gametime, and fly around very little.
6) Goal of the game: To make money, build up a fleet, take over the world. Or whatever you want it to be. If you want to beat the story only, then play story and forget the rest. If you want to make a billion credits, then make that your objective. Want to kill all enemies or have a fleet of destroyers, thats up to you. You get out of the game what you put into it. There are some cool missions with the storemode mode enabled. Like sneaking through an asteroid in your spacesuit avoiding security monitors to find objects, or to fly a spaceship through a big spacestation insides to blow up its reactor (think star wars death stars fighting xwing through the thing and going through tunnels and crap). So, it can be pretty cool at times.
7) When you first start the world, every race has a ton of ships, pirates, bad guys, good guys. And they are placed accordingly. But the pirates will go into good sectors and get destroyed, so it seems after time the amount of "easy pickings" becomes harder. You won't find pirates in good sectors very much later on because they've all been killed. So the first thing to do is this: Fly around good sectors that are "border" worlds. (hitting the , (comma) key will bring up the world map and tell you if its border security or not). It only lists where you've been so you have to fly through jump gates. Find some pirates or bad guys. And shoot em up. Certain races are more "wimps" than others. For example, boron pirates will likely bail out of their ship alot more likely than teladi. Once they bail, steal their ship and sell it. You can't really tell which race is what, but when you fight them they usually say something like "Now you know what it will be like to face a boron in battle." Once you get them down on shields and start killing their hull, they are more likely to bail than the others, so start taking small shots at them, instead of unloading. Because if they bail you might blow up their ship anyways. They usually tell you when they bail "Take my ship, leave me alone!" and they eject so stop firing.
8) To claim a ship as yours once they eject, you have to bring up the little menu in the bottom right, and click eject. That puts you in the spacesuit. Fly within 30 meters of the ship, click the button right button again, click target, and then click claim. Now fly back to your ship and re-enter it. Now you can command it with teh command console and the property screen.
9) Shields are important and probably the most important thing to get for your ship to live. You need about 50mj worth of shields to take on most groups of pirates. You won't get that with normal sized fighter, only about 15 mj, so its likely until you get a bigger ship or more shields, you won't be able to take on very much. For this reason I usually start on the mode "bankrupt assassin" which is a non story mode game mode. They give you a bigger fighter to start off with.
10) Ship types. If you click on a ship on the screen and hit "i" for information, it will list the ship type.
TS is transport small
TL is transport large
M5 scout ships usually about 5mj shields at most and a small laser but very fast.
M4 normal fighter about 15 mj shields and a few lasers.
M3 a large fighter, 50-75mj shields and 8 lasers, maybe a turret (rear gunner). (Think Defiant in Star Trek)
M6 a small capital ship. 125 mj shields, usually 3 turrets. (Think Enterprise in star trek TNG)
M2 is a carrier, has hanger bays for M5-M3's I think.
M1 is a destroyer, lots of guns.
M0 is a battleship.
If you are an M4, you can usually take on a lone M3 provided it does not have a turret, and manuouver around it and take it out. I usually hunt these when I start because they sell for alot. Once you claim it, order it to a shipyard and sell it.
11) Remote control trading. Once you have something to trade with (another ship, hopefully TS), you have to install "Trade system extension" on the ship you are flying. This allows you to bring up the property screen and right below "command console" you can select "trade with station" That way you can trade remotely from another ship and have them buy things as you would in your own ship.
Thats enough for now! But with those 11 things you should get through most things and find more happiness within the game.